This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Kluge, A. Search for Related Content Social Bookmarking                   What's this?

Performance Assessments With Microworlds and Their Difficulty

Universität St. Gallen, Switzerland, annette.kluge{at}unisg.ch

The use of microworlds (MWs), or complex dynamic systems, in educational testing and personnel selection is hampered by systematic measurement errors because these new and innovative item formats are not adequately controlled for their difficulty. This empirical study introduces a way to operationalize an MW's difficulty and demonstrates the strong effects of variations in it. A rationale for predicting an MW's difficulty (number and quality of relationships between the same number of variables) is tested for difficulty's impact on assessed performance. As the number of relationships increases, control performance, acquired knowledge, and self-efficacy as moderating variables decrease. Analysis of knowledge types (qualitative and quantitative) shows that system difficulty affects quantitative knowledge more than it does qualitative knowledge.

Key Words: microworlds • performance assessment • difficulty • control performance • knowledge • self-efficacy

References

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
• Azen, R., & Budescu, D.V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129-148.[CrossRef][ISI][Medline] [Order article via Infotrieve]
• Badke-Schaub, P., & Strohschneider, S. (1998). Complex problem solving in the cultural context. Le Travail Humain, 61, 1-28.
• Bandura, A., & Wood, R.E. (1989). Effect of perceived controllability and performance standards on self-regulation of complex decision making. Journal of Applied Psychology, 56, 805-814.
• Beauducel, A., & Kersting, M. (2002). Fluid and crystallized intelligence and the BIS. European Journal of Psychological Assessment, 18, 97-112.[CrossRef][ISI]
• Bennett, R.E., Morley, M., & Quardt, D. (2000). Three response types for broadening the conception of mathematical problem solving in computerized tests. Applied Psychological Measurement, 24, 294-310.[Abstract/Free Full Text]
• Blech, C., & Funke, J. (2005). DYNAMIS review: An overview about applications of the DYNAMIS approach in cognitive psychology. Heidelberg, Germany: University of Heidelberg, Department of Psychology.
• Brehmer, B., & Doerner, D. (1993). Experiments with computer-simulated microworlds: Escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Computers in Human Behavior, 9, 171-184.[CrossRef][ISI]
• Buchner, A. (1995). Basic topics and approaches to the study of complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 29-63). Hillsdale, NJ: Lawrence Erlbaum.
• Clauser, B.E. (2000). Recent issues and recent advances in scoring performance assessments. Applied Psychological Measurement, 24, 310-324.[Abstract/Free Full Text]
• Debowski, S., Wood, R.E., & Bandura, A. (2001). Impact of guided exploration and enactive exploration on self-regulatory mechanisms and information acquisition through electronic search. Journal of Applied Psychology, 86, 1129-1141.[CrossRef][ISI][Medline] [Order article via Infotrieve]
• Dörner, D., Kreuzig, H.W., Reither, F., & Stäudel, T. (1994). Lohhausen. Vom Umgang mit Unbestimmtheit und Komplexität [Lohhausen: About handling uncertainty and complexity]. Bern, Switzerland: Huber. (Original work published 1983)
• Dörner, D., & Preussler, W. (1990). Die Kontrolle eines einfachen ökologischen Systems [Controlling a simple ecological system]. Sprache und Kognition, 4, 205-217.
• Frensch, P.A., & Funke, J. (1995). Definitions, traditions, and a general framework for understanding complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 3-25). Hillsdale, NJ: Lawrence Erlbaum.
• Funke, J. (1985). Steuerung dynamischer Systeme durch Aufbau und Anwendung subjektiver Kausalmodelle [Controlling dynamic systems through acquisition and application of individual mental models]. Zeitschrift für Psychologie, 193, 443-465.
• Funke, J. (1992). Wissen über dynamische Systeme: Erwerb, Repräsentation und Anwendung [Knowledge about dynamic systems: Acquisition, representation, and application]. Berlin, Germany: Springer.
• Funke, J. (1993). Microworlds based on linear equation-systems: A new approach to complex problem solving and experimental results. In G. Strube & K. F. Wender (Eds.), The cognitive psychology of knowledge (pp. 313-330). Amsterdam: Elsevier Science.
• Funke, J. (1998). Computer-based testing and training with scenarios from complex problem-solving research: Advantages and disadvantages. International Journal of Selection and Assessment, 6, 90-96.[CrossRef][ISI]
• Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking and Reasoning, 7, 69-89.[CrossRef]
• Funke, U. (1995). Szenarien in der Eignungsdiagnostik um im Personaltraining. In B. Strauss & M. Kleinmann (Eds.), Computersimulierte Szenarien in der Personalarbeit [Computer simulated scenarios for personnel work] (pp. 145-216). Göttingen, Germany: Hogrefe.
• Gray, W.D. (2002). Simulated task environments: The role of high-fidelity simulations, scaled worlds, synthetic environments, and laboratory tasks in basic and applied cognitive research. Cognitive Science Quarterly, 2, 205-227.
• Hambleton, R.K. (2000). Advances in performance assessment methodology. Applied Psychological Measurement, 24, 291-294.[Free Full Text]
• Hambleton, R.K., Jaeger, R.M., Plake, B.S., & Mills, C. (2000). Setting performance standards on complex educational assessments. Applied Psychological Measurement, 24, 355-366.[Abstract/Free Full Text]
• Heineken, E., Arnold, H.-J., Kopp, A., & Soltysiak, R. (1992). Strategien des Denkens bei der Regelung eines einfachen dynamischen Systems unter verschiedenen Totzeitbedingungen [Thinking strategies to control a simple dynamic system under different conditions of feedback delay]. Sprache & Kognition, 11, 136-148.
• Hochberg, Y., & Tamhane, A. (1987). Multiple comparison procedures. New York: John Wiley.
• Hornke, L.F., & Kersting, M. (2006). Optimizing quality in the use of Web-based and computer-based testing for personnel selection. In D. Bartram & R. K. Hambleton (Eds.), Computer-based testing and the Internet: Issues and advances (pp. 149-162). New York: John Wiley.
• Kersting, M. (1998). Differentielle Aspekte der sozialen Akzeptanz von Intelligenztests und Problemlöseszenarien als Personalauswahlverfahren [Differential psychological aspects of applicants' acceptance of intelligence test and problem solving scenarios as diagnostic tools for personnel selection]. Zeitschrift für Arbeits- und Organisationspsychologie, 42, 61-75.[ISI]
• Kersting, M. (1999). Diagnostik und Personalauswahl mit computergestützten Problemlöseszenarien? Zur Kriteriumsvalidität mit Problemlöseszenarien und Intelligenztests [Assessment and personnel selection through computer-based scenarios for problem solving: On criterion validity with problem solving scenarios and intelligence tests]. Göttingen, Germany: Hogrefe.
• Kluge, A. (2004). Wissenserwerb für das Steuern komplexer Systeme [Knowledge acquisition for controlling complex systems]. Lengerich, Germany: Pabst Publishers.
• Kröner, S. (2001). Multiflux—Entwicklung und Validierung eines simulationsbasierten Intelligenzdiagnostikums [Multiflux—Development and validation of a simulation-based assessment tool for testing intelligence]. Münster, Germany: Waxmann.
• Macan, T.H., Avedon, M., Paese, M., & Smith, D.E. (1994). The effects of applicants' reactions to cognitive ability tests and an assessment center. Personnel Psychology, 47, 715-738.[CrossRef][ISI]
• Obermann, C. (1991). AIRPORT. Problemlösesimulation 2.2 [AIRPORT problem solving simulation]. Göttingen, Germany: Hogrefe.
• Preussler, W. (1998). Strukturwissen als Voraussetzung für die Steuerung komplexer dynamischer Systeme [Structural knowledge as a prerequisite for controlling complex and dynamic systems]. Zeitschrift für Experimentelle Psychologie, 45, 218-240.[ISI]
• Putz-Osterloh, W. (1993). Strategies for knowledge acquisition and transfer of knowledge in dynamic tasks. In G. Strube & K. F. Wender (Eds.), The cognitive psychology of knowledge (pp. 331-350). Amsterdam: Elsevier Science.
• Schuler, H., Frier, D., & Kaufmann, M. (1993). Personalauswahl im eurpoäischen Vergleich [A European comparison of personnel selection techniques]. Göttingen, Germany: Hrogrefe.
• Shotland, A., Alliger, G.M., & Sales, T. (1998). Face validity in the context of personnel selection: A multimedia approach. International Journal of Selection and Assessment, 6, 124-130.[CrossRef][ISI]
• Simon, H. (1978). Information-processing theory of human problem solving. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (pp. 271-295). Hillsdale, NJ: Lawrence Erlbaum.
• Strauss, B. (1995). Zur Operationalisierung der Komplexität in dynamischen Systemen [On operationalizing complexity in dynamic systems]. Zeitschrift für Psychologie, 20, 73-99.
• Vollmeyer, R., Burns, B.D., & Holyoak, K.J. (1996). The impact of goal specificity on strategy use and the acquisition of problem structure. Cognitive Science, 20, 75-100.[CrossRef][ISI]
• Wagener, D. (2001). Psychologische Diagnostik mit komplexen Szenarios. Taxonomie, Entwicklung, Evaluation [Psychological assessment based on complex scenarios: Taxonomy, development, evaluation]. Lengerich, Germany: Pabst Science.
• Wittmann, W.W., & Hattrup, K. (2004). The relationship between performance in dynamic systems and intelligence. Systems Research and Behavioral Sciences, 21, 393-409.[CrossRef]

CiteULike

Connotea

Del.icio.us

Digg

Reddit

Technorati

This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Kluge, A. Search for Related Content Social Bookmarking                   What's this?