This Article Free Full Text (Free PDF) Free References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Wilson, M. Articles by Wang, W.-c. Search for Related Content Social Bookmarking                   What's this?

Complex Composites: Issues That Arise in Combining Different Modes of Assessment

University of California, Berkeley

Wen-chung Wang

National Taiwan University

Data from the California Learning Assessment System are used to examine certain characteristics of tests designed as the composites of items of different modes. The characteristics include rater severity, test information, and definition of the latent variable. Three different assessment modes-multiple-choice, open-ended, and investigation items (the latter two are referred to as performance-based modes)-were combined in a test across three different test forms. Rater severity was investigated by incorporating a rater parameter for each rater in an item response model that then was used to analyze the data. Some rater severities were found to be quite extreme, and the impact of this variation in rater severities on both total scores and trait level estimates was examined. Within-rater variation in rater severity also was examined and was found to have significant variation. The information contribution of the three modes was compared. Performance-based items provided more information than multiple-choice items and also provided greatest precision for higher levels of the latent variable. A projection-like method was applied to investigate the effects of assessment mode on the definition of the latent variable. The multiple-choice items added information to the performance-based variable. The results of the analysis also showed that the projection-like method did not practically differ from the results when the latent trait was defined jointly by both the multiple-choice and the performance-based items. Index terms: equating, linking, multiple assessment modes, polytomous item response models, rater effects.

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

				R. J. Patz, B. W. Junker, M. S. Johnson, and L. T. Mariano The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data Journal of Educational and Behavioral Statistics, January 1, 2002; 27(4): 341 - 384. [Abstract] [PDF]

				M. Wilson and M. Hoskens The Rater Bundle Model Journal of Educational and Behavioral Statistics, January 1, 2001; 26(3): 283 - 306. [Abstract] [PDF]

				D. Southworth Using Statistical-Based Rating Sheets to Measure Oral Test Inter-Rater Reliability Review of Public Personnel Administration, July 1, 2000; 20(3): 43 - 57. [PDF]

				R. J. Patz and B. W. Junker Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses Journal of Educational and Behavioral Statistics, January 1, 1999; 24(4): 342 - 366. [Abstract] [PDF]