Applied Psychological Measurement recent issues

Testing Person Fit in Cognitive Diagnosis

Liu, Y., Douglas, J. A., Henson, R. A. — Thu, 22 Oct 2009 16:21:44 PDT

In cognitive diagnosis, the test-taking behavior of some examinees may be idiosyncratic so that their test scores may not reflect their true cognitive abilities as much as that of more typical examinees. Statistical tests are developed to recognize the following: (a) nonmasters of the required attributes who correctly answer the item (spuriously high scores) and (b) masters of the attributes who fail to correctly answer the item (spuriously low scores). For a person, nonzero probability of aberrant behavior is tested as the alternative hypothesis, against normal behavior as the null hypothesis. The two generalized likelihood ratio test statistics used, with the null hypothesis parameter on the boundary of the parameter space in each, have asymptotic distributions of a 50:50 mixture of a chi-square distribution with one degree of freedom and a degenerate distribution that is a constant of 0 under the null hypothesis. Simulation results, primarily based on the DINA model (deterministic inputs, noisy ‘‘AND’’ gate), are used to investigate the following: (a) how accurately the statistical tests identify normal/aberrant behaviors, (b) how the power of the tests depends on the length of the cognitive exam and the degree of the inclination toward aberrance, and (c) how sensitive the tests are to inaccurate estimation of model parameters.

Detection and Diagnosis of Person Misfit From Patterns of Summed Polytomous Item Scores

Emons, W. H. M. — Thu, 22 Oct 2009 16:21:44 PDT

For valid decision making, it is essential to both the person being measured and the person or organization that is having the person measured that the observed scores adequately represent the underlying trait. This study deals with person-fit analysis of polytomous item scores to detect unusual patterns of sum scores on subsets of items. This approach has the advantage that it allows for a diagnostic approach in which specific hypotheses of person misfit can be tested. In a simulation study, the false-positive and detection rates have been investigated under varying test and item characteristics and different types and levels of aberrant response behavior. The performance of the sum-score—based approach is compared with the perforance of the l^P_z person-fit measure. The simulations show that the person-fit analysis based on sum-score patterns is useful and performs best for detecting aberrant response behavior that manifests itself locally in the pattern, such as careless responding to reverse-worded tems. Statistic l^P_z performs better for detecting aberrant response behavior that affects the responses globally, such as a tendency to choose extreme response options. The person-fit measures discussed are illustrated using real data from the Neuroticism—Extraversion—Openness Personality Inventory—Revised.

Simultaneous Estimation of Overall and Domain Abilities: A Higher-Order IRT Model Approach

de la Torre, J., Song, H. — Thu, 22 Oct 2009 16:21:44 PDT

Assessments consisting of different domains (e.g., content areas, objectives) are typically multidimensional in nature but are commonly assumed to be unidimensional for estimation purposes. The different domains of these assessments are further treated as multi-unidimensional tests for the purpose of obtaining diagnostic information. However, when the domains are disparate, assuming a single underlying ability across the domains is not tenable. Moreover, estimating domain proficiencies based on short tests can result in unreliable scores. This article presents a higher-order item response theory framework where an overall and multiple domain abilities are specified in the same model. Using a Markov chain Monte Carlo method in a hierarchical Bayesian framework, the overall and domain-specific abilities, and their correlations, are estimated simultaneously. The feasibility and effectiveness of the proposed model are investigated under varied conditions in a simulation study and illustrated using actual assessment data. Implications of the model for future test analysis and ability estimation are also discussed. Index terms: higher-order ability estimation, item response theory, multidimensionality, domain scoring, Markov chain Monte Carlo

Book Review: by Samuel A. Livingston. Equating Test Scores (Without IRT). Princeton, NJ: Educational Testing Service, 2004, 68 pp. (paperback)

Puhan, G. — Thu, 22 Oct 2009 16:21:44 PDT

FreeIAT: An Open-Source Program to Administer the Implicit Association Test

Meade, A. W. — Thu, 22 Oct 2009 16:21:44 PDT

Firestar: Computerized Adaptive Testing Simulation Program for Polytomous Item Response Theory Models

Choi, S. W. — Thu, 22 Oct 2009 16:21:44 PDT

Applied Psychological Measurement Manuscript Submission Guidelines

Thu, 22 Oct 2009 16:21:44 PDT

Model Selection Indices for Polytomous Items

Kang, T., Cohen, A. S., Sung, H.-J. — Sun, 27 Sep 2009 22:12:01 PDT

This study examines the utility of four indices for use in model selection with nested and nonnested polytomous item response theory (IRT) models: a cross-validation index and three information-based indices. Four commonly used polytomous IRT models are considered: the graded response model, the generalized partial credit model, the partial credit model, and the rating scale model. In a simulation study, comparisons among the four indices suggest that model selection is dependent to some extent on the particular conditions simulated. Overall, the Bayesian information criterion index appears to be most accurate in selecting the correct polytomous IRT model. Results are presented from analysis of a real data set to illustrate the use of the four indices for selecting an appropriate model.

Posterior Predictive Model Checking for Multidimensionality in Item Response Theory

Levy, R., Mislevy, R. J., Sinharay, S. — Sun, 27 Sep 2009 22:12:01 PDT

If data exhibit multidimensionality, key conditional independence assumptions of unidimensional models do not hold. The current work pursues posterior predictive model checking, a flexible family of model-checking procedures, as a tool for criticizing models due to unaccounted for dimensions in the context of item response theory. Factors hypothesized to influence dimensionality and dimensionality assessment are couched in conditional covariance theory and conveyed via geometric representations of multidimensionality. A simulation study investigates the performance of the model-checking tools for dichotomous observables. Key findings include support for the hypothesized effects of the manipulated factors with regard to their influence on dimensionality assessment and the superiority of certain discrepancy measures for conducting posterior predictive model checking for dimensionality assessment.

Testing for Differential Item Functioning With Measures of Partial Association

Woods, C. M. — Sun, 27 Sep 2009 22:12:01 PDT

Differential item functioning (DIF) occurs when an item on a test or questionnaire has different measurement properties for one group of people versus another, irrespective of mean differences on the construct. There are many methods available for DIF assessment. The present article is focused on indices of partial association. A family of average conditional ordinal association measures described by Quade is described and empirically compared to the Mantel-Haenszel (MH) test, the partial Pearson correlation pr, and the Spearman rank correlation pr_s. Because coefficients of linear association are not meaningful for the binary and ordinal variables usually used in DIF applications, practitioners are urged to seek alternatives to pr and pr_s. Some of the Quade-family measures are viable alternatives and performed as well as, or better than, the established MH test. Computer code for calculating the average conditional ordinal measures using the free R program is given.

Locally Dependent Linear Logistic Test Model With Person Covariates

Ip, E. H., Smits, D. J. M., De Boeck, P. — Sun, 27 Sep 2009 22:12:01 PDT

The article proposes a family of item-response models that allow the separate and independent specification of three orthogonal components: item attribute, person covariate, and local item dependence. Special interest lies in extending the linear logistic test model, which is commonly used to measure item attributes, to tests with embedded item clusters. The problem of local item dependence arises in item clusters. Existing methods for handling such dependence, however, often fail to satisfy the property of invariant marginal interpretation of the item attribute parameters. Although such a property may not be necessary for applications that focus on predictive analysis, it is critical for linear logistic test models. To achieve the marginal property, we implement an iterative estimation method, which is illustrated using data collected from an inventory on verbal aggressiveness.

BMDS: A Collection of R Functions for Bayesian Multidimensional Scaling

Okada, K., Shigemasu, K. — Sun, 27 Sep 2009 22:12:01 PDT

Applied Psychological Measurement Manuscript Submission Guidelines

Sun, 27 Sep 2009 22:12:01 PDT

Comparison of CAT Item Selection Criteria for Polytomous Items

Choi, S. W., Swartz, R. J. — Mon, 10 Aug 2009 11:22:45 PDT

Item selection is a core component in computerized adaptive testing (CAT). Several studies have evaluated new and classical selection methods; however, the few that have applied such methods to the use of polytomous items have reported conflicting results. To clarify these discrepancies and further investigate selection method properties, six different selection methods are compared systematically. The results showed no clear benefit from more sophisticated selection criteria and showed one method previously believed to be superior—the maximum expected posterior weighted information (MEPWI)—to be mathematically equivalent to a simpler method, the maximum posterior weighted information (MPWI).

The Impact of Multidimensionality on the Detection of Differential Bundle Functioning Using Simultaneous Item Bias Test

Furlow, C. F., Raiford Ross, T., Gagne, P. — Mon, 10 Aug 2009 11:22:45 PDT

Douglas, Roussos, and Stout introduced the concept of differential bundle functioning (DBF) for identifying the underlying causes of differential item functioning (DIF). In this study, reference group was simulated to have higher mean ability than the focal group on a nuisance dimension, resulting in DIF for each of the multidimensional items that, when examined together, produced DBF. The empirical power and the Type I error of the Simultaneous Item Bias Test for DBF analysis were examined under various sample sizes, ratios of reference to focal group sizes, correlations between target and nuisance dimensions, magnitudes of DIF/ DBF, test lengths, percentages of test items in the bundle, and item discriminations. Power was generally high in cells with larger DIF magnitudes, higher percentages of items in the bundle, larger sample sizes, and with the nuisance dimension having a higher discrimination than the target dimension. Type I error rates approximated the nominal alpha rate for all conditions.

Improving the Quality of Ability Estimates Through Multidimensional Scoring and Incorporation of Ancillary Variables

de la Torre, J. — Mon, 10 Aug 2009 11:22:45 PDT

For one reason or another, various sources of information, namely, ancillary variables and correlational structure of the latent abilities, which are usually available in most testing situations, are ignored in ability estimation. A general model that incorporates these sources of information is proposed in this article. The model has a general formulation that allows incorporation of either source or both sources of information in scoring the examinees using various item response models and subsumes the traditional method of expected a posteriori as a special case. Results show that using the different sources of information singly or simultaneously provides better ability estimates (i.e., higher correlation with the true abilities and smaller posterior variance and mean squared error). The optimal condition occurs when several short tests measuring highly correlated abilities that also correlate highly with the covariates are used. Markov chain Monte Carlo parameter estimation algorithms corresponding to the different model formulations are also developed. Simulated and actual data are analyzed to establish the usefulness and feasibility of the proposed models. Several practical considerations in using these models are also discussed.

A Comment on ''The J Index as a Measure of Nominal Scale Response Agreement''

Warrens, M. J. — Mon, 10 Aug 2009 11:22:45 PDT

A Clarification of the Effects of Rapid Guessing on Coefficient {alpha}: A Note on Attali's ''Reliability of Speeded Number-Right Multiple-Choice Tests''

Wise, S. L., DeMars, C. E. — Mon, 10 Aug 2009 11:22:46 PDT

IRTEQ: Windows Application That Implements Item Response Theory Scaling and Equating

Han, K. T. — Mon, 10 Aug 2009 11:22:46 PDT

Addressing Score Bias and Differential Item Functioning Due to Individual Differences in Response Style

Bolt, D. M., Johnson, T. R. — Wed, 17 Jun 2009 10:35:45 PDT

A multidimensional item response theory model that accounts for response style factors is presented. The model, a multidimensional extension of Bock's nominal response model, is shown to allow for the study and control of response style effects in ordered rating scale data so as to reduce bias in measurement of the intended trait. In the current application, the model is also used to investigate response style as an underlying cause of differential item functioning. The approach is illustrated using the item responses of cigarette smokers to the Wisconsin Inventory of Smoking Dependence Motives, a self-report measure of tobacco dependence.

Model Selection Methods for Mixture Dichotomous IRT Models

Li, F., Cohen, A. S., Kim, S.-H., Cho, S.-J. — Wed, 17 Jun 2009 10:35:45 PDT

This study examines model selection indices for use with dichotomous mixture item response theory (IRT) models. Five indices are considered: Akaike's information coefficient (AIC), Bayesian information coefficient (BIC), deviance information coefficient (DIC), pseudo-Bayes factor (PsBF), and posterior predictive model checks (PPMC). The five indices provide somewhat different recommendations for a set of real data. Results from a simulation study indicate that BIC selects the correct (i.e., the generating) model well under most conditions simulated and for all three of the dichotomous mixture IRT models considered. PsBF is almost as effective. AIC and PPMC tend to select the more complex model under some conditions. DIC is least effective for this use.

Classification Consistency and Accuracy for Complex Assessments Under the Compound Multinomial Model

Lee, W.-C., Brennan, R. L., Wan, L. — Wed, 17 Jun 2009 10:35:45 PDT

For a test that consists of dichotomously scored items, several approaches have been reported in the literature for estimating classification consistency and accuracy indices based on a single administration of a test. Classification consistency and accuracy have not been studied much, however, for ``complex'' assessments—for example, those that involve polytomously scored items or mixtures of different types of items. This article describes procedures for estimating various types of single-administration classification consistency and accuracy indices using the multinomial and compound multinomial models for individuals and groups who take a single administration of a complex assessment. An alternative procedure using the bootstrap is also proposed. These procedures are illustrated using real data sets obtained from tests consisting of both polytomous and dichotomous items.

A Parametric Cumulative Sum Statistic for Person Fit

Armstrong, R. D., Shi, M. — Wed, 17 Jun 2009 10:35:45 PDT

This article develops a new cumulative sum (CUSUM) statistic to detect aberrant item response behavior. Shifts in behavior are modeled with quadratic functions and a series of likelihood ratio tests are used to detect aberrancy. The new CUSUM statistic is compared against another CUSUM approach as well as traditional person-fit statistics. A simulation study demonstrates the advantage of the proposed method. Also, the person-fit methods are applied to real response data from the administration of a high-stakes exam. The use of CUSUM charts to help visually identify types of aberrant behavior is demonstrated.

ResidPlots-2: Computer Software for IRT Graphical Residual Analyses

Liang, T., Han, K. T., Hambleton, R. K. — Wed, 17 Jun 2009 10:35:45 PDT