|
Sign In to gain access to subscriptions and/or personal tools.
|
Applied Psychological Measurement, Vol. 32, No. 2,
119-137 (2008)
DOI: 10.1177/0146621606297308
A Monte Carlo Approach to the Design, Assembly, and Evaluation of Multistage Adaptive Tests
Dmitry I. Belov
Law School Admission Council, dbelov{at}lsac.org, belovd{at}mail.ru
Ronald D. Armstrong
Rutgers, The State University of New Jersey
This article presents an application of Monte Carlo methods for developing and assembling multistage adaptive tests (MSTs). A major advantage of the Monte Carlo assembly over other approaches (e.g., integer programming or enumerative heuristics) is that it provides a uniform sampling from all MSTs (or MST paths) available from a given item pool. The uniform sampling allows a statistically valid analysis for MST design and evaluation. Given an item pool, MST model, and content constraints for test assembly, three problems are addressed in this study. They are (a) the construction of item response theory (IRT) targets for each MST path, (b) the assembly of an MST such that each path satisfies content constraints and IRT constraints, and (c) an analysis of the pool and constraints to increase the number of nonoverlapping MSTs that can be assembled from the pool. The primary intent is to produce reliable measurements and enhance pool utilization.
Key Words: computer adaptive testing • test assembly • Monte Carlo methods • item response theory • testlet • automated test assembly • test construction
References
- Ahuja, R.K., Magnanti, T.L., & Orlin, J.B. (1993). Network flows: Theory, algorithms, and applications. Englewood Cliffs, NJ: Prentice Hall.
- Armstrong, R.D., Jones, D.H., Koppel, N.B., & Pashley, P.J. (2004). Computerized adaptive testing with multiple form structures. Applied Psychological Measurement, 28, 147-164.[Abstract]
- Armstrong, R.D., & Roussos, L. (2003). A method to determine targets for multi-stage adaptive tests (Computerized Testing Report No. 02-07). Newtown, PA: Law School Admission Council.
- Belov, D.I., & Armstrong, R.D. (2005). Monte Carlo test assembly for item pool analysis and extension. Applied Psychological Measurement, 29, 239-261.[Abstract]
- Boyd, A.M., Dodd, B.G., & Fitzpatrick, S.J. (2003, April). A comparison of exposure control procedures in CAT systems based on different measurement models for testlets using the verbal reasoning section of the MCAT. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
- Breithaupt, K., Ariel, A., & Veldkamp, B. (2004, April). Balancing item exposure and optimality in automated assembly for multistage testing. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
- Edwards, M.C., & Thissen, D. (2004, April). Multistage computerized adaptive testing: Using item response theory for design selection. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
- Ilog, Inc. (2003). CPLEX 9.0 [Computer software and manual]. Mountain View, CA: Author. Available from http://www.ilog.com
- Lord, F. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
- Luecht, R.M., & Burgin, W. (2003, April). Test information targeting strategies for adaptive multistage testing designs. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
- Luecht, R.M., & Nungester, R.J. (1998). Some practical examples of computer-adaptive sequential testing. Journal of Educational Measurement, 35, 229-247.[CrossRef][ISI]
- Luecht, R.M., & Nungester, R.J. (2000). Computer-adaptive sequential testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 117-128). Boston: Kluwer Academic.
- Patsula, L.N. (1999). A comparison of computerized adaptive testing and multi-stage testing. Unpublished doctoral dissertation, University of Massachusetts, Amherst.
- Schrage, L. (2002). Optimization modeling with LINGO (4th ed.). Chicago, IL: LINDO Systems.
- Wang, X., Bradlow, E.T., & Wainer, H. (2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement, 26, 109-128.[Abstract/Free Full Text]
- Xing, D., & Hambleton, R. (2004). Impact of test design, item quality, and item bank size on the psychometric properties of computer-based credentialing examinations. Educational and Psychological Measurement, 64, 5-21.[Abstract/Free Full Text]
- Zenisky, A., & Hambleton, R. (2004, April). Effects of selected multi-stage design alternatives on credentialing examination outcomes. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
 CiteULike  Connotea  Del.icio.us  Digg  Reddit  Technorati What's this?
|
