A Procedure for Diagnostically Modeling Extant Large-Scale Assessment Data: The Case of the Programme for International Student Assessment in Reading — Oak Academic Publishing
Research ArticleOpen AccessGoogle Scholar indexed
A Procedure for Diagnostically Modeling Extant Large-Scale Assessment Data: The Case of the Programme for International Student Assessment in Reading
Department of Psychology, Sun Yat-sen University, Guangzhou, China
,
Department of Educational Psychology, Rutgers, The State University of New Jersey, New Brunswick, USA
1 Department of Psychology, Sun Yat-sen University, Guangzhou, China
2 Department of Educational Psychology, Rutgers, The State University of New Jersey, New Brunswick, USA
Cognitive diagnosis models (CDMs) are psychometric models developed mainly to assess examinees’ specific strengths and weaknesses of a set of skills or attributes within a domain. Recently, several methodological developments have been added to the CDM literature, which include the development of general and reduced CDMs, various absolute and relative fit measures at both the test and item levels, and a general Q-matrix validation procedure. Building on these developments, this research proposes a systematic procedure to diagnostically model extant large-scale assessment data. The procedure can be divided into four phases: construction of initial attributes and Q-matrices, construction of final attributes and Q-matrix, evaluation of reduced CDMs, and crossvalidation of the selected model. Working with language experts, we use data from the PISA 2000 reading assessment to illustrate the procedure.
Akaike, H. (1974). A New Look at the Statistical Identification Model. IEEE Transactions on Automated Control, 19, 716-723. http://dx.doi.org/10.1109/TAC.1974.1100705
Chen, J., & de la Torre, J. (2012). An Extension of the G-DINA Model for Polytomous Attributes. Paper Presented at the Annual Meeting of American Educational Research Association, Vancouver.
Chen, J., & de la Torre, J. (2013). A General Cognitive Diagnosis Model for Expert-Defined Polytomous Attributes. Applied Psychological Measurement, 37, 419-437. http://dx.doi.org/10.1177/0146621613479818
Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and Absolute Fit Evaluation in Cognitive Diagnosis Modeling. Journal of Educational Measurement, 50, 123-140. http://dx.doi.org/10.1111/j.1745-3984.2012.00185.x
de la Torre, J. (2008). An Empirically-Based Method of Q-Matrix Validation for the DINA Model: Development and Applications. Journal of Educational Measurement, 45, 343-362. http://dx.doi.org/10.1111/j.1745-3984.2008.00069.x
de la Torre, J. (2011). The Generalized DINA Model Framework. Psychometrika, 76, 179-199. http://dx.doi.org/10.1007/s11336-011-9207-7
de la Torre, J., & Chen, J. (2011). Estimating Different Reduced Cognitive Diagnosis Models Using a General Framework. Paper Presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans.
de la Torre, J., & Chiu, C.-Y. (2010). A General Method of Empirical Q-Matrix Validation Using the G-DINA Model Discrimination Index. Paper Presented at the Annual Meeting of the National Council on Measurement in Education, Denver.
de la Torre, J., & Lee, Y.-S. (2010). Item-Level Comparison of Saturated and Reduced Cognitive Diagnosis Models. Paper Presented at the Annual Meeting of the National Council on Measurement in Education, Denver.
DiBello, L., Roussos, L., & Stout, W. (2007). Cognitive Diagnosis Part I. In C. R. Rao, & S. Sinharay (Eds.), Handbook of Statistics (Vol. 26): Psychometrics (pp. 979-1030). Amsterdam: Elsevier.
Haertel, E. H. (1989). Using Restricted Latent Class Models to Map the Skill Structure of Achievement Items. Journal of Educational Measurement, 26, 301-321. http://dx.doi.org/10.1111/j.1745-3984.1989.tb00336.x
Hartz, S. (2002). A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality. Unpublished Doctoral Dissertation, University of Illinois at Urbana-Champaign.
Henson, R. A., Templin, J., & Willse, J. (2009). Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables. Psychometrika, 74, 191-210. http://dx.doi.org/10.1007/s11336-008-9089-5
Jang, E. E. (2009). Cognitive Diagnostics Assessment of L2 Reading Comprehension Ability: Validity Arguments for Fusion Model Application to Langu Edge Assessment. Language Testing, 26, 31-73. http://dx.doi.org/10.1177/0265532208097336
Junker, B. W., & Sijtsma, K. (2001). Cognitive Assessment Models with Few Assumptions, and Connections with Non-Parametric Item Response Theory. Applied Psychological Measurement, 25, 258-272. http://dx.doi.org/10.1177/01466210122032064
Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The Impact of Model Misspecification on Parameter Estimation and Item-Fit Assessment in Log-Linear Diagnostic Classification Models. Journal of Educational Measurement, 49, 59-81. http://dx.doi.org/10.1111/j.1745-3984.2011.00160.x
Lee, Y., & Sawaki, Y. (2009). Application of Three Cognitive Diagnosis Models to ESL Reading and Listening Assessments. Language Assessment Quarterly, 6, 239-263. http://dx.doi.org/10.1080/15434300903079562
Maris, E. (1999). Estimating Multiple Classification Latent Class Models. Psychometrika, 64, 187-212. http://dx.doi.org/10.1007/BF02294535
OECD (1999). Measuring Student Knowledge and Skills: A New Framework for Assessment. Paris: Organization for Economic Cooperation and Development.
OECD (2006a). Assessing Scientific, Reading and Mathematical Literacy: A Framework for PISA 2006. Paris: Organization for Economic Cooperation and Development.
OECD (2006b). PISA Released Items: Reading. http://www.oecd.org/pisa/38709396.pdf
Schwarzer, G. (1976). Estimating the Dimension of a Model. Annals of Statistics, 6, 461-464. http://dx.doi.org/10.1214/aos/1176344136
Tatsuoka, K. K. (1983). Rule Space: An Approach for Dealing with Misconceptions Based on Item Response Theory. Journal of Educational Measurement, 20, 345-354. http://dx.doi.org/10.1111/j.1745-3984.1983.tb00212.x
Tatsuoka, K. K. (1990). Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Diagnostic Monitoring Skills and Knowledge Acquisition (pp. 453-488). Hillsdale, NJ: Erlbaum.
Templin, J., & Henson, R. A. (2006). Measurement of Psychological Disorders Using Cognitive Diagnosis Models. Psychological Methods, 11, 287-305. http://dx.doi.org/10.1037/1082-989X.11.3.287
von Davier, M. (2008). A General Diagnostic Model Applied to Language Testing Data. British Journal of Mathematical and Statistical Psychology, 61, 287-307. http://dx.doi.org/10.1348/000711007X193957