Applied Psychological Measurement current issue

Testing Person Fit in Cognitive Diagnosis

Liu, Y., Douglas, J. A., Henson, R. A. — Thu, 22 Oct 2009 16:21:44 PDT

In cognitive diagnosis, the test-taking behavior of some examinees may be idiosyncratic so that their test scores may not reflect their true cognitive abilities as much as that of more typical examinees. Statistical tests are developed to recognize the following: (a) nonmasters of the required attributes who correctly answer the item (spuriously high scores) and (b) masters of the attributes who fail to correctly answer the item (spuriously low scores). For a person, nonzero probability of aberrant behavior is tested as the alternative hypothesis, against normal behavior as the null hypothesis. The two generalized likelihood ratio test statistics used, with the null hypothesis parameter on the boundary of the parameter space in each, have asymptotic distributions of a 50:50 mixture of a chi-square distribution with one degree of freedom and a degenerate distribution that is a constant of 0 under the null hypothesis. Simulation results, primarily based on the DINA model (deterministic inputs, noisy ‘‘AND’’ gate), are used to investigate the following: (a) how accurately the statistical tests identify normal/aberrant behaviors, (b) how the power of the tests depends on the length of the cognitive exam and the degree of the inclination toward aberrance, and (c) how sensitive the tests are to inaccurate estimation of model parameters.

Detection and Diagnosis of Person Misfit From Patterns of Summed Polytomous Item Scores

Emons, W. H. M. — Thu, 22 Oct 2009 16:21:44 PDT

For valid decision making, it is essential to both the person being measured and the person or organization that is having the person measured that the observed scores adequately represent the underlying trait. This study deals with person-fit analysis of polytomous item scores to detect unusual patterns of sum scores on subsets of items. This approach has the advantage that it allows for a diagnostic approach in which specific hypotheses of person misfit can be tested. In a simulation study, the false-positive and detection rates have been investigated under varying test and item characteristics and different types and levels of aberrant response behavior. The performance of the sum-score—based approach is compared with the perforance of the l^P_z person-fit measure. The simulations show that the person-fit analysis based on sum-score patterns is useful and performs best for detecting aberrant response behavior that manifests itself locally in the pattern, such as careless responding to reverse-worded tems. Statistic l^P_z performs better for detecting aberrant response behavior that affects the responses globally, such as a tendency to choose extreme response options. The person-fit measures discussed are illustrated using real data from the Neuroticism—Extraversion—Openness Personality Inventory—Revised.

Simultaneous Estimation of Overall and Domain Abilities: A Higher-Order IRT Model Approach

de la Torre, J., Song, H. — Thu, 22 Oct 2009 16:21:44 PDT

Assessments consisting of different domains (e.g., content areas, objectives) are typically multidimensional in nature but are commonly assumed to be unidimensional for estimation purposes. The different domains of these assessments are further treated as multi-unidimensional tests for the purpose of obtaining diagnostic information. However, when the domains are disparate, assuming a single underlying ability across the domains is not tenable. Moreover, estimating domain proficiencies based on short tests can result in unreliable scores. This article presents a higher-order item response theory framework where an overall and multiple domain abilities are specified in the same model. Using a Markov chain Monte Carlo method in a hierarchical Bayesian framework, the overall and domain-specific abilities, and their correlations, are estimated simultaneously. The feasibility and effectiveness of the proposed model are investigated under varied conditions in a simulation study and illustrated using actual assessment data. Implications of the model for future test analysis and ability estimation are also discussed. Index terms: higher-order ability estimation, item response theory, multidimensionality, domain scoring, Markov chain Monte Carlo

Book Review: by Samuel A. Livingston. Equating Test Scores (Without IRT). Princeton, NJ: Educational Testing Service, 2004, 68 pp. (paperback)

Puhan, G. — Thu, 22 Oct 2009 16:21:44 PDT

FreeIAT: An Open-Source Program to Administer the Implicit Association Test

Meade, A. W. — Thu, 22 Oct 2009 16:21:44 PDT

Firestar: Computerized Adaptive Testing Simulation Program for Polytomous Item Response Theory Models

Choi, S. W. — Thu, 22 Oct 2009 16:21:44 PDT

Applied Psychological Measurement Manuscript Submission Guidelines

Thu, 22 Oct 2009 16:21:44 PDT