Applied Psychological Measurement

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Free Access - Register Here

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Waller, N. G.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
This version was published on May 1, 2008
Applied Psychological Measurement, Vol. 32, No. 3, 211-223 (2008)
DOI: 10.1177/0146621607300860

Commingled Samples: A Neglected Source of Bias in Reliability Analysis

Niels G. Waller

University of Minnesota, nwaller{at}umn.edu

Reliability is a property of test scores from individuals who have been sampled from a well-defined population. Reliability indices, such as coefficient {alpha} and related formulas for internal consistency reliability (KR-20, Hoyt's reliability), yield lower bound reliability estimates when (a) subjects have been sampled from a single population and when (b) test items are congeneric (i.e., when items are sampled from a single latent dimension). However, when samples are commingled—that is, when they are composed of scores that are drawn from multiple populations— coefficient {alpha} and related indices can be severely biased. In most cases the bias inflates {alpha}; in other cases {alpha} is attenuated. Equations are derived for elucidating this bias in two-group mixture distributions.

Key Words: coefficient alpha • reliability • measurement bias

References

  • Ayers, M. (1981). Locke versus Aristotle on natural kinds. Journal of Philosophy, 78, 247-272.[CrossRef][ISI]
  • Barchard, K.A., & Hakstian, R. (1997). The effects of sampling model on inference with coefficient alpha. Educational and Psychological Measurement, 57, 893-905.[Abstract]
  • Bendig, A.W. (1953). The reliability of self-ratings as a function of amount of verbal anchoring and of the number of categories on the scale. Journal of Applied Psychology, 37, 38-41.[CrossRef]
  • Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305-314.[CrossRef][ISI]
  • Boyd, R. (1991). Realism, anti-foundationalism and the enthusiasm for natural kinds. Philosophical Studies, 61, 127-148.[CrossRef][ISI]
  • Briel, J. B., O'Neill, K. A., & Scheuneman, J. D. (Eds.). (1993). GRE technical manual: Test development, score interpretation, and research for the Graduate Record Examinations Program. Princeton, NJ: Educational Testing Service.
  • Carlson, R. (1971). Where is the person in personality research? Psychological Bulletin, 75, 212.
  • Cattell, R.B. (1965). The scientific analysis of personality. New York: Penguin.
  • Cleveland, W.S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74, 829-836.[CrossRef][ISI]
  • Cochran, W.G. (1977). Sampling techniques (3rd ed.). New York: John Wiley.
  • Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and application. Journal of Applied Psychology, 78, 98-104.[CrossRef][ISI]
  • Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.[CrossRef][ISI]
  • Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.
  • Duhachek, A., & Iacobucci, D. (2004). Alpha's standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 89, 792-808.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Durvasula, S., Netemeyer, R.G., Andrews, J.C., & Lysonski, S. (2006). Examining the cross-national applicability of multi-item, multidimensional measures using generalizability theory. Journal of International Business Studies, 37, 469-483.[CrossRef][ISI]
  • Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
  • Everitt, B.S., & Hand, D.J. (1981). Finite mixture distributions. London: Chapman and Hall.
  • Feldt, L.S. (1969). A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363-373.[CrossRef][ISI]
  • Feldt, L.S. (1980). A test of the hypothesis that Cronbach's alpha reliability coefficient is the same for two tests administered to the same sample. Psychometrika, 45, 99-105.[CrossRef][ISI]
  • Feldt, L.S. (2002). Estimating the internal consistency reliability of tests composed of testlets varying in length. Applied Measurement in Education, 15, 33-48.[CrossRef][ISI]
  • Feldt, L.S., Woodruffe, D.J., & Salih, F.A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11, 93-103.[Abstract/Free Full Text]
  • Green, S.B., & Hershberger, S.L. (2000). Correlated errors in true score models and their effect on coefficient alpha. Structural Equation Modeling, 7, 251-270.[CrossRef]
  • Green, S.B., Lissitz, R.W., & Mulaik, S.A. (1977) Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827-838.[Abstract]
  • Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10, 255-282.[CrossRef][ISI]
  • Guttman, L. (1953). Reliability formulas that do not assume experimental independence. Psychometrika, 18, 225-239.[CrossRef][ISI]
  • Hacking, I. (1991). A tradition of natural kinds. Philosophical Studies, 61, 109-126.[CrossRef][ISI]
  • Hakstian, A.R., & Barchard, K.A. (2000). Toward more robust inferential procedures for coefficient alpha under sampling of both subjects and conditions. Multivariate Behavioral Research, 35, 427-456.[ISI][Medline] [Order article via Infotrieve]
  • Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological measurement, 9, 139-164.[Abstract]
  • Hogan, T.P., Benjamin, A., & Brezinski, K.L. (2000). Reliability methods: A note on the frequency of us of various types. Educational and Psychological Measurement, 60, 523-531.[Abstract/Free Full Text]
  • Horst, P. (1953). Correcting the Kuder-Richardson reliability for dispersion of item difficulties. Psychological Bulletin, 50, 371-374.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Hoyt, C. (1941). Test reliability estimated by analysis of variance. Psychometrika, 6, 153-160.[CrossRef]
  • Kalton, G. (1983). Introduction to survey sampling (Sage University Paper Series on Quantitative Applications in the Social Sciences). Beverly Hills, CA: Sage.
  • Kish, L. (1965). Survey sampling. New York: John Wiley.
  • Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalence and uncorrelated error on coefficient alpha. Applied Psychological Measurement, 21, 337-348.[Abstract/Free Full Text]
  • Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25, 897-995.[CrossRef][ISI]
  • Korfine, L., & Lenzenweger, M.F. (1995). The taxonicity of schizotypy: A replication. Journal of Abnormal Psychology, 104, 26-31.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Kuder, G.F., & Richardson, M.W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151-160.[CrossRef]
  • Lazarsfeld, P.F., & Henry, N.W. (1968). Latent structure analysis. Boston: Houghton Mifflin.
  • Lenzenweger, M.F. (1999). Deeper into the schizotypy taxon: On the robust nature of maximum covariance analysis. Journal of Abnormal Psychology, 108, 182-187.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Lenzenweger, M.F., & Korfine, L. (1992). Confirming the latent structure and base rate of schizotypy: A taxometric analysis. Journal of Abnormal Psychology, 101, 567-571.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Lissitz, R.W., & Green, S.B. (1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.[CrossRef][ISI]
  • Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
  • Lucke, J.F. (2005). ``Rassling the hog'': The influence of correlated item error on internal consistency, classical reliability, and congeneric reliability. Applied Psychological Measurement, 29, 106-125.[CrossRef][ISI]
  • McDonald, R.P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
  • McLachlan, G.J., & Basford K.E. (1988). Mixture models: Inference and applications to clustering. New York: Marcel Dekker.
  • Meehl, P.E. (1962). Schizotaxia, schizotypy, schizophrenia. American Psychologist, 17, 827-838.[CrossRef]
  • Meehl, P.E. (1973). MAXCOV-HITMAX: A taxonomic search method for loose genetic syndromes. In P. E. Meehl, Psychodiagnosis: Selected papers (pp. 200-224). Minneapolis: University of Minnesota Press.
  • Meehl, P.E. (1990). Schizotaxia as an open concept. In A. I. Rabin, R. Zucker, R. Emmons, & S. Frank (Eds.), Studying persons and lives (pp. 248-303). New York: Springer.
  • Meehl, P.E. (1992) Factors and taxa, traits and types, differences of degree and differences in kind. Journal of Personality, 60, 117-174.[CrossRef][ISI]
  • Mendoza, J.L., Stafford, K.L., & Stauffer, J.M. (2000). Large-sample confidence intervals for validity and reliability coefficients. Psychological Methods, 5, 356-369.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Millsap, R.E., & Everson, H.T. (1993). Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17, 297-334.[Abstract/Free Full Text]
  • Novick, M.R., & Lewis, C. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Osburn, H.G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5, 343-355.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Pap, A. (1953). Reduction sentences and open concepts. Methodos, 5, 3-30.
  • Peterson, R.A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 250-261.
  • Raju, N.S. (1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.[CrossRef][ISI]
  • Raykov, T. (2001). Bias of coefficient {alpha} for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69-76.[Abstract/Free Full Text]
  • Reichenbach, H. (1938). Experience and prediction: An analysis of the foundations and the structure of knowledge. Chicago: University of Chicago Press.
  • Sears, D.O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515-530.[CrossRef][ISI]
  • Sharma, S., & Weathers, D. (2003). Assessing generalizability of scales used in cross-national research. International Journal of Research in Marketing, 20, 287-295.[CrossRef][ISI]
  • Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
  • Streiner, D.L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99-103.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Symonds, P.M. (1924). On the loss of reliability in rating due to coarseness of the scale. Journal of Experimental Psychology, 7, 456-461.[CrossRef][ISI]
  • Traub, R.E. (1994). Reliability for the social sciences: Theory and applications. Thousand Oaks, CA: Sage.
  • van Zyle, S.M., Heinz, N., & Nel, D.G. (2000). On the distribution of the maximum likelihood estimator of Cronbach's alpha. Psychometrika, 65, 271-280.[CrossRef][ISI]
  • Waller, N.G., & Meehl, P.E. (1998). Multivariate taxometric procedures: Distinguishing types from continua. Thousand Oaks, CA: Sage.
  • Waller, N.G., Putnam, F.W., & Carlson, E.B. (1996). Types of dissociation and dissociative types: A taxometric analysis of dissociative experiences. Psychological Methods, 3, 300-321.
  • Waller, N.G., Thompson, J., & Wenk, E. (2000). Black-white differences on the MMPI: Using IRT to separate measurement bias from true group differences on homogeneous and heterogeneous scales. Psychological Methods, 5, 125-146.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Wasserman, J.D. (2003). Assessment of intellectual function. In J. R. Graham & J. A. Naglieri (Eds.), Handbook of psychology. New York: John Wiley.
  • Wilkinson, L., & the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals. Guidelines and explanations. American Psychologist, 54, 594-604.[CrossRef]
  • Yuan, K.H., & Bentler, P.M. (2002). On robustness of the normal-theory cased asymptotic distributions of three reliability coefficient estimates. Psychometrika, 67, 251-259.[CrossRef][ISI]
  • Zimmerman, D.W., Zumbo, B.D., & Lalonde, C. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33-49.[Abstract]
  • Zinbarg, R.E., Revelle, W., Yovel, I., & Li. W. (2005). Cronbach's {alpha}, Revelle's β, McDonald's {omega}H: Their relations with each and two alternative conceptualizations of reliability. Psychometrika, 70, 123-133.[CrossRef][ISI]

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Waller, N. G.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?