This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Lindell, M. K. Articles by Whitney, D. J. Search for Related Content Social Bookmarking                         What's this?

A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target

Texas A&M Universitymlindell{at}archone.tamu.edu

Christina J. Brandt

Michigan State University

David J. Whitney

California State University, Long Beach

The commonly used form of r_{wg. (J)} can display irregular behavior, so four variants of this index were examined. An alternative index, r*_{wg. J}, is recommended. This index is an inverse linear function of the ratio of the average obtained variance to the variance of uniformly distributed random error. r*_wg.Jis superficially similar to Cronbach’s , but careful examination confirms that r*_wg.Jis an index of agreement, not reliability. Based on an examination of the small-sample behavior of r_wgand r*_wg.J, sample sizes of 10 or more raters are recommended.

Applied Psychological Measurement, Vol. 23, No. 2, 127-135 (1999)
DOI: 10.1177/01466219922031257


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				W. Zhu, B. J. Avolio, and F. O. Walumbwa Moderating Role of Follower Characteristics With Transformational Leadership and Follower Work Engagement Group Organization Management, October 1, 2009; 34(5): 590 - 619. [Abstract] [PDF]

				O. Ludtke and A. Robitzsch Assessing Within-Group Agreement: A Critical Examination of a Random-Group Resampling Approach Organizational Research Methods, July 1, 2009; 12(3): 461 - 487. [Abstract] [PDF]

				D. J. Pasisz and G. M. Hurtz Testing for Between-Group Differences in Within-Group Interrater Agreement Organizational Research Methods, July 1, 2009; 12(3): 590 - 613. [Abstract] [PDF]

				H. van Mierlo, J. K. Vermunt, and C. G. Rutte Composing Group-Level Constructs From Individual-Level Survey Data Organizational Research Methods, April 1, 2009; 12(2): 368 - 392. [Abstract] [PDF]

				A. Cohen, E. Doveh, and I. Nahum-Shani Testing Agreement for Multi-Item Scales With the IndicesrWG(J) and AD M(J) Organizational Research Methods, January 1, 2009; 12(1): 148 - 164. [Abstract] [PDF]

				D. A. Newman and H.-P. Sin How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse Organizational Research Methods, January 1, 2009; 12(1): 113 - 147. [Abstract] [PDF]

				J. M. LeBreton and J. L. Senter Answers to 20 Questions About Interrater Reliability and Interrater Agreement Organizational Research Methods, October 1, 2008; 11(4): 815 - 852. [Abstract] [PDF]

				Q. M. Roberson, M. C. Sturman, and T. L. Simons Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes Organizational Research Methods, October 1, 2007; 10(4): 564 - 588. [Abstract] [PDF]

				A. W. Meade and L. T. Eby Using Indices of Group Agreement in Multilevel Construct Validation Organizational Research Methods, January 1, 2007; 10(1): 75 - 96. [Abstract] [PDF]

				S. Geister, U. Konradt, and G. Hertel Effects of Process Feedback on Motivation, Satisfaction, and Performance in Virtual Teams Small Group Research, October 1, 2006; 37(5): 459 - 489. [Abstract] [PDF]

				C. E. Lance, M. M. Butts, and L. C. Michels The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say? Organizational Research Methods, April 1, 2006; 9(2): 202 - 220. [Abstract] [PDF]

				R. D. Brown and N. M. A. Hauenstein Interrater Agreement Reconsidered: An Alternative to the rwg Indices Organizational Research Methods, April 1, 2005; 8(2): 165 - 184. [Abstract] [PDF]

				J. M. LeBreton, L. R. James, and M. K. Lindell Recent Issues Regarding rWG, rWG, rWG(J), and rWG(J) Organizational Research Methods, January 1, 2005; 8(1): 128 - 138. [Abstract] [PDF]

				M. Bayazit and E. A. Mannix Should I Stay or Should I Go?: Predicting Team Members' Intent to Remain in the Team Small Group Research, June 1, 2003; 34(3): 290 - 321. [Abstract] [PDF]

				M. J. Burke and W. P. Dunlap Estimating Interrater Agreement with the Average Deviation Index: A User's Guide Organizational Research Methods, April 1, 2002; 5(2): 159 - 172. [Abstract] [PDF]

				W.-C. Tsai Determinants and consequences of employee displayed positive emotions Journal of Management, August 1, 2001; 27(4): 497 - 512. [Abstract] [PDF]

				S. S. Webber, G. Chen, S. C. Payne, S. M. Marsh, and S. J. Zaccaro Enhancing Team Mental Model Measurement with Performance Appraisal Practices Organizational Research Methods, October 1, 2000; 3(4): 307 - 322. [Abstract] [PDF]

				L. T. Eby, A. W. Meade, A. G. Parisi, and S. S. Douthitt The Development of an Individual-Level Teamwork Expectations Measure and the Application of a Within-Group Agreement Statistic to Assess Shared Expectations for Teamwork Organizational Research Methods, October 1, 1999; 2(4): 366 - 394. [Abstract] [PDF]