Applied Psychological Measurement

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Free Access - Register Here

Click here for more information

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Lam, T. C. M.
Right arrow Articles by Kolic, M.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
Applied Psychological Measurement, Vol. 32, No. 3, 248-260 (2008)
DOI: 10.1177/0146621607301094

Effects of Semantic Incompatibility on Rating Response

Tony C. M. Lam

Ontario Institute for Studies in Education of the University of Toronto, tlam{at}oise.utoronto.ca

Mary Kolic

Ontario Institute for Studies in Education of the University of Toronto

Semantic incompatibility, an error in constructing measuring instruments for rating oneself, others, or objects, refers to the extent to which item wordings are incongruent with, and hence inappropriate for, scale labels and vice versa. This study examines the effects of semantic incompatibility on rating responses. Using a 2 x 2 factorial design with semantic compatibility and scale packedness as the between-subjects factors, 160 university students were randomly assigned to four treatment conditions. Analysis of their responses to a 10-item academic ability self-assessment rating scale shows a significant difference in means between positive-packed and equal-interval conditions when item wordings and scale labels are semantically compatible. The semantically compatible conditions also show smaller variability and a slightly higher internal consistency of responses than the semantically incompatible conditions. The authors conclude that when rating scales are semantically incompatible, respondents tend to ignore the scale labels, use a greater variety of strategies to generate responses, and produce less reliable responses.

Key Words: rating scale construction • attitude scaling • survey research • self-reporting • bias in rating

References

  • Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160.[CrossRef][ISI]
  • Bendig, A.W. (1953). The reliability of self-ratings as a function of the amount of verbal anchoring and of the number of categories on the scale. Journal of Applied Psychology, 37, 38-41.[CrossRef]
  • Bendig, A.W. (1954a). Reliability and the number of rating scale categories. Journal of Applied Psychology, 38, 38-40.[CrossRef]
  • Bendig, A.W. (1954b). Reliability of short rating scales and the heterogeneity of the rated stimuli. Journal of Applied Psychology, 38, 167-170.[CrossRef]
  • Bendig, A.W. (1955). Rater reliability and the heterogeneity of the scale anchors. Journal of Applied Psychology, 39, 37-39.[Medline] [Order article via Infotrieve]
  • Chang, L. (1997). Dependability of anchoring labels of Likert-type scales. Educational & Psychological Measurement, 57, 800-808.[CrossRef]
  • Childers, T.L., Houston, M.J., & Heckler, S.E. (1985). Measurement of individual differences in visual versus verbal information processing. Journal of Consumer Research, 12, 125-134.[CrossRef][ISI]
  • Cox, J. (1996). Your opinion, please!: How to build the best questionnaires in the field of education. Thousand Oaks, CA: Corwin Press.
  • Desimone, L.M., & Floch, D.C.L. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26, 1-22.[Abstract/Free Full Text]
  • Dixon, P.N., BoBo, M., & Stevick, R.A. (1984). Response differences and preferences for all-category-defined and end-defined Likert formats. Educational and Psychological Measurement, 44, 61-66.[Abstract]
  • Dunham, T.C., & Davison, M.L. (1991). Effects of scale anchors on student ratings of instructors. Applied Measurement in Education, 4, 23-35.[CrossRef]
  • Feldt, L.S. (1969). A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363-373.[CrossRef][ISI]
  • Fowler, F.J. (1992). How unclear terms can affect survey data. Public Opinion Quarterly, 56, 218-231.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Fowler, F.J. (1995). Improving survey questions: Design and evaluation (Vol. 38). Thousand Oaks, CA: Sage.
  • French-Lazovik, G., & Gibson, C.L. (1984). Effects of verbally labeled anchor points on the distributional parameters of rating measures. Applied Psychological Measurement, 2, 49-57.
  • Frey, B.F., & Edwards, L.M. (2001, April). Strong words or moderate words: A comparison of the reliability and validity of responses on attitude scales. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
  • Furuhata, T. (2003). The influence of labels and positions in rating scales of Japanese students. Unpublished doctoral dissertation, University of Washington.
  • Gable, R.W., & Wolf, M.B. (1993). Instrument development in the affective domain (2nd ed.). Boston: Kluwer.
  • Graesser, A.C., Bommareddy, S., Swamer, S., & Golding, J.M. (1996). Integrating questionnaire design with a cognitive computational model of human question answering. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 143-174). San Francisco: Jossey-Bass.
  • Hancock, G.R., & Klockars, A.J. (1991). The effect of scale manipulations on validity: Targeting frequency rating scales for anticipated performance levels. Applied Ergonomics, 22, 147-154.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Hippler, H.-J., Schwartz, N., & Sudman, S. (1988). Social information processing and survey methodology. New York: Springer-Verlag.
  • Holleman, B. (1999). Wording effects in survey research: Using meta-analysis to explain the forbid/allow asymmetry. Journal of Quantitative Linguistics, 6, 29-40.[CrossRef]
  • Jenkins, G.D., Jr., & Taber, T.D. (1977). A Monte Carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 63, 392-398.
  • Jobe, J., & Mingay, D.J. (1991). Cognition and survey measurement: History and overview. Applied Cognitive Psychology, 54, 940-952.
  • Jobe, J.B., & Mingay, D.J. (1989). Cognitive research improved questionnaires. American Journal of Public Health, 79, 1053-1055.[Free Full Text]
  • Klockars, A.J., & Hancock, G.R. (1993). Manipulations of evaluative rating scales to increase validity. Psychological Reports, 73, 1059-1066.[ISI]
  • Klockars, A.J., & Yamagishi, M. (1988). The influence of labels and positions in rating scales. Journal of Educational Measurement, 25, 85-96.[CrossRef][ISI]
  • Krosnick, J.A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Krosnick, J.A., Narayan, S., & Smith, W.R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, 29-44.
  • Lam, T.C.M. (1997). Issues in rating scale construction. Unpublished manuscript. Toronto, Canada: University of Toronto.
  • Lam, T.C.M., & Klockars, A.J. (1982). Anchor point effects on the equivalence of questionnaire items. Journal of Educational Measurement, 19, 317-322.[CrossRef][ISI]
  • Lam, T.C.M., & Stevens, J.J. (1994). Effects of content polarization, item wording, and rating scale width on rating response. Applied Measurement in Education, 7, 141-158.[CrossRef]
  • Likert, R.A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 40-52.
  • Lissitz, R.W., & Green, S.B. (1975). Effects of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.[CrossRef][ISI]
  • Newstead, S.E., & Arnold, J. (1989). The effect of response format on ratings of teaching. Educational and Psychological Measurement, 49, 33-43.[Abstract]
  • Nunnally, J.C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
  • Ostrom, T.M., & Gannon, K.M. (1996). Exemplar generation. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 293-318). San Francisco: Jossey-Bass.
  • Peterson, R.A. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage.
  • Rotter, G.S. (1972). Attitudinal points of agreement and disagreement. Journal of Social Psychology, 86, 211-218.[ISI]
  • Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York: Academic.
  • Schwartz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.[CrossRef]
  • Schwarz, N., Knauper, B., Hippler, H.-J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570-582.[Abstract]
  • Schwartz, N., & Sudman, S. (1996). Answering questions: Methodology for determining cognitive and communicative processes in survey research. San Francisco: Jossey-Bass.
  • Sirken, M.G., Herrmann, D.J., Schechter, S., Schwarz, N., Tanur, J.M., & Tourangeau, R. (1999). Cognition and survey research. New York: John Wiley.
  • Smith, T.W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns. Public Opinion Quarterly, 51, 75-83.[Abstract]
  • Spector, P.E. (1976). Choosing response categories for summated rating scales. Journal of Applied Psychology, 61, 374-375.[CrossRef][ISI]
  • Spector, P.E. (1980). Ratings of equal and unequal response choice intervals. Journal of Social Psychology, 112, 115-119.[ISI]
  • Spector, P.E. (1992). Summated rating scale construction: An introduction. Newbury Park, CA: Sage.
  • Tanur, J.M. (1992). Questions about questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation.
  • Thomas, S.J. (2004). Using Web and paper questionnaires for data-based decision making: From design to interpretation of the results. Thousand Oaks, CA: Corwin Press.
  • Tourangeau, R., & Rasinski, K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314.[CrossRef][ISI]
  • Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
  • Viswanathan, M. (1993). Measurement of individual differences in preference for numerical information. Journal of Applied Psychology, 78, 741-752.[CrossRef][ISI]
  • Weng, L. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-972.[Abstract]
  • Willis, G.B. (1999). Cognitive interviewing: A ``how to'' guide. Retrieved July 29, 2005, from http://www.appliedresearch.cancer.gov/areas/cognitive/interview.pdf
  • Willis, G.B. (2004). Cognitive interviewing revisited: A useful technique, in theory? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, et al. (Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). New Jersey: John Wiley.
  • Willis, G.B. (2005). Cognitive interviews: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
  • Wyatt, R.C., & Meyers, L.S. (1987). Psychometric properties of four 5-point Likert-type response scales. Educational and Psychological Measurement, 47, 27-35.[Abstract]

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Lam, T. C. M.
Right arrow Articles by Kolic, M.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?