Applied Psychological Measurement

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Free Access - Register Here

Click here for more information

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Freund, P. A.
Right arrow Articles by Holling, H.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
Applied Psychological Measurement, Vol. 32, No. 3, 195-210 (2008)
DOI: 10.1177/0146621607306972

Explaining and Controlling for the Psychometric Properties of Computer-Generated Figural Matrix Items

Philipp Alexander Freund

Westfälische Wilhelms-Universität Münster, Germany, pafreund{at}uni-muenster.de

Stefan Hofer

Bundesagentur für Arbeit, Nürnberg, Germany

Heinz Holling

Westfälische Wilhelms-Universität Münster, Germany

Figural matrix items are a popular task type for assessing general intelligence (Spearman's g). Items of this kind can be constructed rationally, allowing the implementation of computerized generation algorithms. In this study, the influence of different task parameters on the degree of difficulty in matrix items was investigated. A sample of N = 169 participants (all age groups) completed a set of 25 automatically generated 4 x 4 matrix items. Data collection was conducted through the World Wide Web. All items showed a good fit with the Rasch model, and item difficulty could be explained reasonably well through the implemented task parameters. The research indicated that matrix items can easily be generated using well-defined computerized algorithms. Their composite character explains item difficulty to a satisfactory degree and enables researchers to construct items with anticipated psychometric properties and Rasch model conformity. Practical advantages of these findings are pointed out.

Key Words: automatic item generation • figural matrix items • item task parameters • LLTM • Rasch model

References

  • Andersen, E.B. (1973). A goodness of fit test for the Rasch model. Psychometrika, 38, 123-140.[CrossRef][ISI]
  • Arendasy, M. (2002). GeomGen-Ein Itemgenerator für Matrizentestaufgaben [GeomGen-An item generator for matrices]. Wien: Eigenverlag.
  • Arendasy, M., & Gittler, G. (2003). IRT-basierter Vergleich zweier Varianten automatisiert erstellter Matrizentestaufgaben [IRT-based comparison of automatically generated figural matrix items]. Zeitschrift für Differentielle und Diagnostische Psychologie, 24, 261-275.[CrossRef]
  • Arendasy, M., & Sommer, M. (2005). The effect of different types of perceptual manipulations on the dimensionality of automatically generated figural matrices. Intelligence, 33, 307-324.[CrossRef][ISI]
  • Arendasy, M., & Sommer, M. (in press). Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items. Learning and Individual Differences.
  • Bejar, I.I. (2002). Generative testing: From conception to implementation. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 199-217). Mahwah, NJ: Lawrence Erlbaum.
  • Bors, D.A., & Stokes, T.L. (1998). Raven's Advanced Progressive Matrices: Norms for firstyear university students and the development of a short form. Educational and Psychological Measurement, 58, 382-398.[Abstract]
  • Carpenter, P.A., Just, M.A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices test. Psychological Review, 97, 404-431.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Dennis, I., Handley, S., Bradon, P., Evans, J., & Newstead, S. (2002). Approaches to modeling item-generative tests. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 53-71). Mahwah, NJ: Lawrence Erlbaum.
  • Dillon, R.F., Pohlmann, J.T., & Lohman, D.F. (1981). A factor analysis of Raven's Advanced Progressive Matrices freed of difficulty factors. Educational and Psychological Measurement, 41, 1295-1302.[ISI]
  • Embretson, S.E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380-396.[CrossRef][ISI]
  • Embretson, S.E. (1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64, 407-433.[CrossRef][ISI]
  • Embretson, S.E. (2002). Generating abstract reasoning items with cognitive theory. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 219-250). Mahwah, NJ: Lawrence Erlbaum.
  • Enright, M.K., & Sheehan, K.M. (2002). Modeling the difficulty of quantitative reasoning items: Implications for item generation. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 129-157). Mahwah, NJ: Lawrence Erlbaum.
  • Fischer, G.H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359-374.[CrossRef]
  • Fischer, G.H. (1995a). Derivations of the Rasch model. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 15-38). New York: Springer.
  • Fischer, G.H. (1995b). The linear logistic test model. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 157-180). New York: Springer.
  • Fischer, G.H., & Ponocny-Seliger, E. (1998). LPCM-Win 1.0. Structural Rasch modeling [Computer software]. Groningen, Netherlands: ProGamma.
  • Formann, W., & Piswanger, J. (1979). Wiener Matrizentest (WMT) [Wiener Matrices Test]. Weinheim, Germany: Beltz.
  • Gorin, J.S. (2005). Manipulating processing difficulty of reading comprehension questions: The feasibility of verbal item generation. Journal of Educational Measurement, 42, 351-373.[CrossRef][ISI]
  • Gorin, J.S., & Embretson, S.E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30, 394-411.[Abstract/Free Full Text]
  • Green, K.E., & Kluever, R.C. (2001). Components of item difficulty of Raven's matrices. Journal of General Psychology, 119, 189-199.
  • Hofer, S. (2004). MatrixDeveloper [Unpublished computer software]. Münster, Germany: Psychological Institute IV, Westfälische Wilhelms-Universität.
  • Hornke, L.F., & Habon, M.W. (1986). Rule-based item bank construction and evaluation within the linear logistic framework. Applied Psychological Measurement, 10, 369-380.[Abstract]
  • Irvine, S.H., & Kyllonen, P.C. (2002). Item generation for test development. Mahwah, NJ: Lawrence Erlbaum.
  • Kyllonen, P.C. (2002). Item generation for repeated testing of human performance. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 251-275). Mahwah, NJ: Lawrence Erlbaum.
  • Kyllonen, P.C., & Christal, R.E. (1990). Reasoning ability is (little more than) working memory capacity?! Intelligence, 14, 389-433.[CrossRef][ISI]
  • Martin-Löf, P. (1973). Statistika modeler: Anteckningar fran seminarier lasåret 1969-1970, utarbetade av Rolf Sundberg. Obetydligt ändrat nytryk, Oktober 1973 [Statistical models: Notes from seminars 1969-1970, prepared by Rolf Sundberg]. Stockholm, Sweden: Institutet för Försäkringsmatematik och Matematisk Statistisk vid Stockhokms Universitet.
  • Meo, M., Roberts, M.J., & Marucci, F.S. (2007). Element salience as a predictor of item difficulty for Raven's Progressive Matrices. Intelligence, 35, 359-368.[CrossRef][ISI]
  • Müller, A. (2001). Variation Einfachbezug versus Mehrfachbezug und Training versus kein Training bei regelgeleitet konstruierten Matrizenaufgaben auf die Ergebnisleistung [Variation of single versus multiple rules per element and training versus no training in rule-based generated matrix items and performance]. Unpublished master's thesis, University of Münster, Germany.
  • Nevo, B. (1976). The effects of general practice, specific practice, and item familiarization on change in aptitude test scores. Measurement and Evaluation in Guidance, 9, 16-20.[ISI]
  • Preckel, F. (2003). Diagnostik intellektueller Hochbegabung. Testentwicklung zur Erfassung der fluiden Intelligenz [Assessment of intellectual giftedness: Test development for the assessment of fluid intelligence]. Göttingen, Germany: Hogrefe.
  • Primi, R. (2001). Complexity of geometric inductive reasoning tasks: Contribution to the understanding of fluid intelligence. Intelligence, 30, 41-70.[CrossRef][ISI]
  • Raven, J.C. (1962). Advanced Progressive Matrices. London: H. K. Lewis.
  • Raven, J.C. (1965). Guide to using the Coloured Progressive Matrices Sets A, Ab, B. Dumfries, Scotland: Grieve.
  • Raven, J.C. (1985). Standard Progressive Matrices Sets A, B, C, D, and E. London: H. K. Lewis.
  • Rost, J., & von Davier, M. (1994). A conditional item fit index for Rasch models. Applied Psychological Measurement, 18, 171-182.[Abstract/Free Full Text]
  • van der Linden, W.J. (2005). Linear models for optimal test design. New York: Springer.
  • van der Ven, A.H.G.S., & Ellis, J.L. (2000). A Rasch analysis of Raven's Standard Progressive Matrices. Personality and Individual Differences, 29, 45-64.[CrossRef][ISI]
  • Vigneau, F., & Bors, D.A. (2005). Items in context: Assessing the dimensionality of Raven's Advanced Progressive Matrices. Educational and Psychological Measurement, 65, 109-123.[Abstract]

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?



This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Google Scholar
Right arrow Articles by Freund, P. A.
Right arrow Articles by Holling, H.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?