Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analysis of a Reading Comprehension Test Battery

Document Type : Research Paper

Authors

1 Associate Professor, University of Tehran

2 PhD Candidate, University of Tehran

Abstract

Of paramount importance in the study of cognitive diagnostic assessment (CDA) is the absence of tests developed for small-scale diagnostic purposes. Currently, much of the research carried out has been mainly on large-scale tests, e.g., TOEFL, MELAB, IELTS, etc. Even so, formative language assessment with a focus on informing instruction and engaging in identification of student’s strengths and weaknesses to guide instruction has not been conducted in the Iranian English language learning context. In an attempt to respond to the call for developing diagnostic tests, this study explored developing a cognitive diagnostic reading comprehension test for CDA purposes. To achieve this, initially, a list of reading attributes was prepared based on the literature and then the attributes were used to construct 20 reading comprehension items. Then seven content raters were asked to identify the attributes of each item of the test. To obtain quantitative data for Q-matrix construction, the test battery was administered to 1986 students of a General English Language Course at the University of Tehran, Iran. In addition, 13 students were recruited to participate in think-aloud verbal protocols. On the basis of the overall agreement of the content raters’ judgements concerning the choices of attributes and results of think-aloud verbal protocol analysis, a Q-matrix that specified the relationships between test items and target attributes was developed. Finally, to examine the CDA of the test, the Fusion Model, a type of cognitive diagnostic model (CDM), was used for diagnosing the participants' strengths and weaknesses. Results suggest that nine major reading attributes are involved in these reading comprehension test items. The results obtained from such cognitive diagnostic analyses could be beneficial for both teachers and curriculum developers to prepare instructional materials that target specific weaknesses and inform them of the more problematic areas to focus on in class in order to plan for better instruction.

Keywords


Article Title [فارسی]

ساخت و اعتباربخشی کیو ماتریس در تجزیه و تحلیل تشخیصی‌شناختی مجموعه آزمون خواندن و درک مفاهیم

Authors [فارسی]

  • محمد علوی 1
  • فاطمه رنجبران 2
1 دانشیار دانشگاه تهران
2 دانشجوی دکترا، دانشگاه تهران
Abstract [فارسی]

یکی از موضوعات حائز اهمیت در تجزیه و تحلیل تشخیصی‌شناختی فقدان مطالعاتی در زمینه ایجاد آزمون‌های تشخیصی به منظور پیشبرد اهداف تشخیصی‌شناختی است. طی سالهای اخیر مطالعات در زمینه آزمون‌سازی و اعتباربخشی به آزمون سمت و سویی دیگر پیدا کرده و محققین در تلاش هستند تا نتایج آزمون را به صورت کاربردی‌تر برای دانشجو و معلمان فراهم کنند. منظور از کاربردی کردن نتایج آزمون ارائه نتایج فراتر از نمره میانگین یا رتبه‌بندی کلی همانند آزمون‌های کلاسی یا کنکور سازمان سنجش در ایران و تافل و آیلتس در سطح بین‌المللی است تا دانشجو بتواند به وضعیت درسی خود بهبود بخشد. در این مطالعه یک آزمون خواندن و درک مفاهیم شامل بیست سوال بر اساس چهارچوب شناختی طراحی شده است. به همین منظور ابتدا یک لیست از مهارت‌های لازم برای پاسخ دهی به هر سوال فراهم و سپس یک کیو ماتریس برای شناسایی رابطه بین هر سوال و مهارت‌های مورد نیاز ساخته شده است. سپس هفت متخصص در زمینه آزمون سازی مهارت‌های لازم برای پاسخ به این سوالات را مشخص کردند تا بر اساس نتایج به دست آمده کیو ماتریس تصحیح گردد. در نهایت این آزمون برای تشخیص نقاط قوت و ضعف ۱۹۸۶ دانشجو در کلاس‌های زبان عمومی دانشگاه تهران برگزار شد. برای تجزیه و تحلیل تشخیصی‌شناختی آزمون از مدل فیوژن استفاده شد. این روش در واقع یک مرحله فراتر از ارائه پاسخ‌های درست و غلط است و مشخص کردن تعداد مهارت‌هایی است که در هر سؤال آزمون به کار رفته است. نتایج این مطالعه می‌تواند برای معلمان در طراحی برنامه درسی و انتخاب منابع متناسب با نقاط قوت و ضعف دانشجوها مفید واقع شود.

Keywords [فارسی]

  • مهارت‌ها
  • تجزیه تحلیل تشخیصی‌شناختی
  • مدل فیوژن
  • ‌ کیو ماتریس
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
Alderson, J. C. (2010). “Cognitive diagnosis and Q-Matrices in language assessment”: A Commentary, 96-103.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford: London.
Birch, B. M. (2002). English L2 reading: Getting to the bottom. Routledge.
Birenbaum, M., Kelly, A. E., & Tatsuoka, K. K. (1993). Diagnosing knowledge states in algebra using the rule-space model. Journal of Research in Mathematics Education, 24, 442-459.
Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: examining attributes of a free response listening test. Language Testing, 15(2), 119–157.
Cohen, A. D., & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph No. MS-33). Princeton, NJ: ETS.
De la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163-183.
DiBello, L. V., Stout, W. F., & Roussos, L. (1995). Unified cognitive psychometric assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ: Erlbaum.
DiBello, L., & Stout, W. (2008). Arpeggio documentation and analyst manual. Chicago: Applied informative assessment research enterprises (AIARE)—LLC.
DiBello, L., & Stout, W. (2008). Arpeggio suite, version 3.1. 001 [Computer program]. Chicago: Applied informative assessment research enterprises (AIARE)—LLC.
Embretson, S.E., Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38(4), 343-368.  
Fletcher, J. M. (2006). Measuring reading comprehension. Scientific Studies of Reading, 10(3), 323-330.
Francis, D.J., Snow, C.E., August, D., Carlson, C.D., Miller, J., & Iglesias, A. (2006). Measures of reading comprehension: A latent variable analysis of the diagnostic assessment of reading comprehension, Scientific Studies of Reading, 10(3), 301-322.
Goodman, D. P., & Hambleton, R. K. (2004). Student test score reports and interpretive guides: Review of current practices and suggestions for future research.  Applied Measurement in Education, 7(2), 145-220.
Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63(2-B), 864.
Hartman, H. J. (2001). Developing students’ metacognitive knowledge and skills. In H. J. Hartman (Ed.). Metacognition in learning and instruction: Theory, Research and Practice (pp. 33-68). Dordrecht, the Netherlands: Kluwer.
Huang, T.W., & Wu, P.C. (2013). Classroom-based cognitive diagnostic model for a teacher-made fraction-decimal Test. Educational Technology & Society, 16 (3), 347–361.
Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Available from ProQuest Dissertations and Theses database. (AAT 3182288)
Jang, E. E. (2009). Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Language Assessment Quarterly, 6(3), 210-238.
Kim, A. Y. (2011). Examining second language reading components in relation to reading test performance for diagnostic purposes: A Fusion model approach (Doctoral dissertation, Teachers College, Columbia University).
Kim, A. Y. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing. [0265532214558457].
Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239-263.
Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka's Rule‐Space approach. Journal of Educational Measurement, 41(3), 205-237.
Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.
Li, H. (2011). Evaluating language group differences in the subskills of reading using a cognitive diagnostic modeling and differential skill functioning approach [Doctoral dissertation, The Pennsylvania State University].
Li, H., & Suen, H. K. (2013). Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analyses of a Reading Test. Educational Assessment, 18(1), 1-25.
Lumley, T. (1993). The notion of sub-skills in reading comprehension test: An EAP example. Language Testing, 10(3), 211-234.
Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33(4), 379-416.
Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439–483.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). Design and analysis in task-based language assessment. Language Testing. Special Issue: Interpretations, Intended Uses, and Designs in Task-based Language, 19(4), 477–496.
Mislevy, R.J. (2006). Cognitive psychology and educational assessment. Educational measurement, 4, 257-305.
Patz, R. J., & Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342–366.
Pellegrino, J. C., & Chudowsky, N. N. & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. (National research council’s committee on the foundation of assessment) National Academy Press; Washington D.C.
Roussos, L. A., DiBello, L. V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. H. (2007). The fusion model skills diagnostic system. In J. Leighton & M. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 275–318). New York, NY: Cambridge University Press.
Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441-474.
Rupp, A. A., Templin, J., & Henson, R. A. (2012). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190-209.
Sheehan, K., & Mislevy, R. (1990). Integrating cognitive and psychometric models to measure document literacy. Journal of Educational Measurement, 27, 255-272.
Snow, R. E., & Lohman, D. F. (1989). Implications of cognitive psychology for educational measurement. American Council on Education.
Stiggins, R., Arter, J., & Chappuis, S. (2004). Classroom assessment for student learning: Doing it rightusing it well. Dover, NH: Assessment Training Institute.
Svetina, D., Gorin, J.S. & Tatsuoka, K.K. (2011) Defining and comparing the reading comprehension construct: A cognitive-psychometric modelling approach, International Journal of Testing, 11:1, 1-23.
Tatsuoka, K. K. (1995). Architecture of knowledge structure and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, and R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 327-361). Hillsdale, NJ: Lawrence Erlbaum Associates.
Templin, J. L. (2004). Generalized linear mixed proficiency models for cognitive diagnosis. [Available from ProQuest Dissertations and Theses database. (AAT3160960)]
Urquhart, S., & Weir, C. J. (1998). Reading in a second language: Process, product and practice. New York: Longman.