Investigating the Reliability of the Reading Module of the Iranian Ministry of Science, Research, and Technology’s English Proficiency Test

Document Type : Research Paper

Authors

1 Assistant Professor, English Department, Science and Arts University of Yazd, Yazd, Iran

2 MA. in ELT, Science and Arts University of Yazd, Yazd, Iran

Abstract

As a national high-stakes test of English proficiency, MSRT needs further scrutiny of reliability. Thus, the present study aimed to investigate different sources of variation that may impact the MSRT test-takers’ reading comprehension. Accordingly, some factors including the reading topics, item types, and participants’ general proficiency were delved into based on the scores obtained from 60 MSRT prospective candidates. Upon administering a sample of the reading subtest taken from a recent version of MSRT, the collected data was dichotomously scored and then analyzed in terms of internal consistency, inter-correlations, and causal patterns. The yielded results showed an overall reliability of 0.86 for the reading module, while a moderate interrelationship was obtained amongst the passages (r = 0.47) as well as the item types (r = 0.44). Furthermore, the Mixed ANOVA results demonstrated that topic and item type significantly affected the reading performance, whereas the proficiency factor did not play a conspicuous role in distinguishing the participants’ reading accomplishment. Both theoretically and operationally, the results reported through this study may stress the need for reconsidering the influence of such test-method facets as topic and item type in the MSRT reading subtest to upgrade the test’s unidimensionality and fairness. 

Keywords

Main Subjects

References

Ahmadi Shirazi, M., Alavi, S. M., & Salarian, H. (2019). An investigation into item types and text types of reading comprehension section of Iranian Ph.D. entrance exams using G-theory. Journal of Modern Research in English Language Studies, 6(1), 1-29. https://doi.org/10.30479/jmrels.2019.10591.1326
Ary, D., Jacobs, L. C., Irvine, C. K. S., & Walker, D. (2019). Introduction to research in education. Cengage Learning.
Ashrafzadeh, A., Don, Z. M., & Meshkat, M. (2015). The effect of familiarity with content knowledge on Iranian medical students' performance in reading comprehension texts: A comparative study of medical and TEFL students. Journal of Language Teaching and Research, 6(3), 524-534. https://doi.org/10.17507/jltr.0603.07
Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford University Press.
Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice. Oxford University Press.
Best, J. W., & Khan, J. V. (2016). Research in education. United Kingdom: Pearson.
Brown, H. D., & Abeywickrama, P. (2018). Language assessment: Principles and classroom practices (3rd ed.). Pearson Education ESL.
Chalhoub-Deville, M., & Turner, C. E. (2000). What to look for in ESL admission tests: Cambridge certificate exams, IELTS, and TOEFL. System, 28(4), 523-539.https://doi.org/10.1016/S0346-251X (00)00036-1
Dewi, H., Damio, S., & Sukarno, S. (2023). Item analysis of reading comprehension questions for English proficiency test using Rasch model. REID (Research and Evaluation in Education), 9(1), 24-36. https://doi.org/10.21831/reid.v9i1.53514
Fallahian, E. & Tabatabaei, O. (2015). Construct validity of MSRT reading comprehension module in Iranian context. English Language Teaching, 8(9), 173-186. https://doi.org/10.5539/elt. v8n9p173
Field, A. (2024). Discovering statistics using IBM SPSS statistics. Sage Publications Limited.
Fulcher, G. (2010). Practical language testing. London: Hodder Education.
Ghahraki, S., Tavakoli, M., Ketabi, S. (2022). Applying a two-parameter item response model to explore the psychometric properties: The case of the ministry of Science, Research and Technology (MSRT) high-stakes English Language Proficiency test. Journal of English Language Teaching and Learning, 14(29), 1-26. https://doi.org/10.22034/ELT.2021.46325.2396
Hair Jr, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2022). A primer on partial least squares structural equation modeling (PLS-SEM) (3rd ed.). Sage Publications, Inc.
Hashemi, A., & Daneshfar, S. (2018). A review of the IELTS test: Focus on validity, reliability, and washback. IJELTAL (Indonesian Journal of English Language Teaching and Applied Linguistics), 3(1), 39-52. https://doi.org/10.21093/ijeltal. v3i1.123
Hill, Y. Z., & Liu, O. L. (2012). Is there any interaction between background knowledge and language proficiency that affects TOEFL iBT® reading performance? ETS Research Report Series, 2012(2), 1-34. https://doi.org/10.1002/j.2333-8504.2012.tb02304.x
Karami, H. (2012). The relative impact of persons, items, subtests, and academic background on performance on a language proficiency test. Psychological Test and Assessment Modeling, 54(3), 211-226.
Khodi, A., Ponniah, L. S., Farrokhi, A. H., & Sadeghi, F. (2024). Test review of Iranian English language proficiency test: MSRT test. Language Testing in Asia, 14(4), 1-11. https://doi.org/10.1186/s40468-023-00270-0
Kiani, R., & Haghighi, M. (2006). The investigation of the TMU English proficiency test: Reliability related issues. Quarterly Journal of Humanities, 16(58), 55-73.
Mackey, A., & Gass, S. M. (2022). Second language research: Methodology and design (3rd ed.). Taylor & Francis.
Mehrens, W. A., & Lehmann, I. J. (1991). Measurement and evaluation in education and psychology (4th ed). Belmont, CA: Wadsworth. Thomson Learning.
Noori, M. & Hosseini Zadeh, S. (2017). The English proficiency test of the Iranian ministry of science, research, and technology: A review. International Journal of English Language & Translation Studies, 5(3), 21-26.
Pallant, J. (2020). SPSS survival manual: A step-by-step guide to data analysis using IBM SPSS (7th ed.). Routledge.
Rashvand Semiyari, S., & Ahangari, S. (2022). Examining differential item functioning (DIF) for Iranian EFL test takers with different fields of study. Research in English Language Pedagogy, 10(1), 169-190. https://doi.org/10.30486/RELP.2021.1935588.1295
Sahrai, R. M., & Mamaghani, H. (2013). An assessment of reliability and validity of MSRT test. Quarterly of Educational Measurement, 3(10), 1-20.
Tabachnick, B. G., Fidell, L. S., & Ullman, J. B. (2013). Using multivariate statistics (Vol. 6, pp. 497-516). Boston, MA: Pearson.

Files

Share

How to cite

Statistics