The Impact of Experience and Academic Degree on EFL Test-Takers’ Perception of Computer-Assisted Language Testing.

Document Type : Research Paper

Authors

1 English Language Department, Razi University, Kermashan, Iran

2 Department of English, Faculty of Literature, Alzahra University, Tehran, Iran.

3 Department of English, Faculty of Literature, Alzahra University, Tehran, Iran

Abstract

Assessment plays a vital role in educational systems. With the emergence of computer-based teaching, global attention has shifted from traditional to online teaching, but not necessarily to technology-based testing. As a new form, testing through technology requires careful consideration, as decisions based on test results may significantly impact examinees’ futures. An exploratory mixed-methods design was used to examine examinees’ perceptions of computer-assisted language testing (CALT) and its relationship to test-takers’ academic degrees and experience. To this end, 162 Iranian students studying translation, teaching, linguistics, and English literature were recruited to complete an existing questionnaire, and 11 participants were selected from the 165 to participate in interviews. The data were gathered at Payam Nour University in Iran through an adapted questionnaire. A two-way ANOVA was employed to examine the effects of experience with technology-based testing and academic degree on testing results. Regarding the qualitative part, we used observation and semi-structured interviews to collect in-depth data, drawing on interpretation and reflection to meaningfully code and categorize the data. Finally, three themes emerged from the data through in vivo and open coding, using a bottom-up coding scheme. The results revealed that prior exposure to this technology significantly affected examinees’ perceptions, regardless of their academic degree. The findings of this study can be beneficial to policymakers, administrators, and test developers in making informed decisions about whether and how to utilize this technology.

Keywords

Main Subjects

References

Akdemir, O., & Oguz, A. (2008). Computer-based testing: An alternative for the assessment of Turkish undergraduate students. Computers & Education, 51(3), 1198-1204. https://doi.org/10.1016/j.compedu.2007.11.007
Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14(2), 115–129. 
Alrob, M., Abu, M., Asad, N. A., & Daqar, M. A. (2019). Attitudes toward and Implications of the Computer-Based Exams at Arab American University of Palestine. Journal of Education and Learning, 8(1), 196-205. https://10.5539/jel.v8n1p196
Bachman, L. F. (1990). Fundamental considerations in language testing: Oxford University Press.
Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice: Developing language assessments and justifying their use in the real world. Oxford University Press Oxford.
Bachman, L. F., & Palmer, A. S. (1982). The construct validation of some components of communicative proficiency. TESOL Quarterly, 16(4), 449–465.
Bahari, A. (2021). Computer-assisted language proficiency assessment tools and strategies. Open Learning: The Journal of Open, Distance and e-Learning, 36(1), 61-87. https://doi.org/10.1080/02680513.2020.1726738
Balogun, A. G., & Olanrewaju, A. S. (2016). Role of computer self-efficacy and gender in computer-based test anxiety among undergraduates in Nigeria. Psychological Thought, 9(1), 58-66. https://doi.org/10.5964/psyct.v9i1.160
Brown, G. T. (2019). Technologies and infrastructure: costs and obstacles in developing large-scale computer-based testing. Education Inquiry, 10(1), 4–20. https://doi.org/10.1080/20004508.2018.1529528
Bugbee, A. C. Jr. (1996). The equivalence of paper-and-pencil and computer-based testing. Journal of research on computing in education, 28(3), 282-299. https://doi.org/10.1080/08886504.1996.10782166
Burgmanis, Ģ., Mikīte, M., France, I., & Namsone, D. (2023). Development of computer-based diagnostic assessment system: case study of the equivalence of paper-and-pencil and computer-based testing. In Society Integration Education Proceedings of the International Scientific Conference (Vol. 1, pp. 613-622).
Clariana, R., & Wallace, P. (2002). Paper–based versus computer–based assessment: Key factors associated with the test mode effect. British Journal of Educational Technology, 33(5), 593–602.
Chan, S., Bax, S., & Weir, C. (2018). Researching the comparability of paper-based and computer-based delivery in a high-stakes writing test. Assessing Writing, 36, 32-48. https://doi.org/10.1016/j.asw.2018.03.008
Chapelle, C. A. (2001). Computer applications in second language acquisition. Cambridge University Press.
Chapelle, C. A. (2003). English language learning and technology: Lectures on applied linguistics in the age of information and communication technology (Vol. 7). John Benjamins Publishing.
Chapelle, C. A., & Douglas, D. (2006). Assessing language through computer technology. Ernst Klett Sprachen.
Chapelle, C. A., & Sauro, S. (2017). Introduction to the handbook of technology and second language teaching and learning. Wiley Online Library.
Chapelle, C. A., & Voss, E. (2016). Twenty years of technology and language assessment in Language Learning and technology. Language Learning & Technology, 20(2), 116-128. https://www.learntechlib.org/p/176101/.
Dadey, N., Lyons, S., & DePascale, C. (2018). The comparability of scores from different digital devices: A literature review and synthesis with recommendations for practice. Applied Measurement in Education, 31(1), 30. https://doi.org/10.1080/08957347.2017.1391262
Daniels, H. (2015). Mediation: An expansion of the socio-cultural gaze. History of the Human Sciences, 28(2), 34-50. https://doi.org/10.1177%2F0952695114559994
Daniels, L. M., & Gierl, M. J. (2017). The impact of immediate test score reporting on university students’ achievement emotions in the context of computer-based multiple-choice exams. Learning and Instruction, 52, 27-35. https://doi.org/10.1016/j.learninstruc.2017.04.001
Dewi, A. T. A., Rofik, M., Holidi, M., Najiburahman, N., Baharun, H., Zaini, A. W., & Sanjani, M. A. F. (2025). CBT exam implementation guidance for students in facing digital-based assessments. Communautaire: Journal of Community Service, 4(3), 297-312.
Douglas, D. (2012). Technology and language testing. The encyclopedia of applied linguistics. https://doi.org/10.1002/9781405198431.wbeal1182
Ebadi, S., & Rahimi, M. (2019). Mediating EFL learners’ academic writing skills in online dynamic assessment using Google Docs. Computer Assisted Language Learning, 32(5-6), 527-555. https://doi.org/10.1080/09588221.2018.1527362
Ebadi, S., & Bashir, S. (2020). An exploration into EFL learners’ writing skills via mobile-based dynamic assessment. Education and Information Technologies, 1-22.
Gipps, C., & Stobart, G. (2010). Alternative assessment. In International Encyclopedia of Education (3rd ed.). Elsevier.
Golonka, E. M., Bowles, A. R., Frank, V. M., Richardson, D. L., & Freynik, S. (2014). Technologies for foreign language learning: A review of technology types and their effectiveness. Computer Assisted Language Learning, 27(1), 70–105.
Habibi, H. (2017). Pathology of electronic tests in Payame Noor University. Educational Measurement and Evaluation Studies, 6(16), 43–66.
Heigham, J., & Croker, R. (2009). Qualitative research in applied linguistics: A practical introduction. Springer.
Hinton, P., Brownlow, C., & McMurray, I. (2004). I., & Cozens, B.(2004). SPSS explained.
Hosseini, M., Abidin, M. J. Z., & Baghdarnia, M. (2014). Comparability of test results of computer based tests (CBT) and paper and pencil tests (PPT) among English language learners in Iran. Procedia-Social and Behavioral Sciences, 98, 659-667. https://doi.org/10.1016/j.sbspro.2014.03.465
Jimoh, R., Shittu, A. J. K., & Kawu, Y. K. (2012). Students’ perception of computer based test (CBT) for examining undergraduate chemistry courses. Journal of Emerging Trends in Computing and Information Sciences, 3(2), 125-134. https://10.14738/abr.46.2325
Jin, Y., & Yan, M. (2017). Computer literacy and the construct validity of a high-stakes computer-based writing assessment. Language Assessment Quarterly, 14(2), 101-119. https://doi.org/10.1080/15434303.2016.1261293
Khoshsima, H., & Toroujeni, S. M. H. (2017). Computer-Based Testing: Score Equivalence and Testing Administration Mode Preference in a Comparative Evaluation Study. International Journal of Emerging Technologies in Learning, 12(10).
Khoshsima, H., Hashemi Toroujeni, S. M., Thompson, N., & Reza Ebrahimi, M. (2019). Computer-Based (CBT) vs. Paper-Based (PBT) Testing: Mode Effect, Relationship between Computer Familiarity, Attitudes, Aversion and Mode Preference with CBT Test Scores in an Asian Private EFL Context. Teaching English with Technology, 19(1), 86-101.
Lantolf, J. P., Thorne, S. L., & Poehner, M. E. (2015). Sociocultural theory and second language development. Theories in second language acquisition: An introduction, 1, 207-226.
Larson-Hall, J. (2015). A guide to doing statistics in second language research using SPSS and R: Routledge.
Lee, J. A. (1986). The effects of past computer experience on computerised aptitude test performance. Educational and Psychological Measurement, 46(3), 727-733. https://doi.org/10.1177%2F0013164486463030
Lee, C. L., & Huang, M. K. (2014). The influence of computer literacy and computer anxiety on computer self-efficacy: the moderating effect of gender. Cyberpsychology, Behavior, and Social Networking, 17(3), 172-180. https://doi.org/10.1089/cyber.2012.0029
Merriam, S. B., & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation (4th ed.). Jossey-Bass.
Nicholson, J. A., Hardin, A. M., & Nicholson, D. B. (2003). Test performance and the medium: Unearthing differences that make a difference [Conference presentation]. Americas Conference on Information Systems.
Ockey, G. J. (2009). Developments and challenges in the use of computer‐based testing for assessing second language ability. The Modern Language Journal, 93, 836-847. https://doi.org/10.1111/j.1540-4781.2009.00976.x
Olufemi, O. A., & Oluwatayo, O. J. (2014). Computer anxiety and computer knowledge as determinants of candidates’ performance in computer-based test in Nigeria. Journal of Education, Society and Behavioural Science, 495-507. https://doi.org/10.9734/BJESBS/2014/6632
Pallant, J. (2011). Survival manual. A step by step guide to data analysis using SPSS.
Powell, A. L. (2013). Computer anxiety: Comparison of research from the 1990s and 2000s. Computers in Human Behavior, 29(6), 2337-2381. https://doi.org/10.1016/j.chb.2013.05.012
Richards, L. (2014). Handling qualitative data: A practical guide (3rd ed.). SAGE Publications.
Roever, C., & McNamara, T. (2006). Language testing: The social dimension. Blackwell Publishing
Schulenberg, S. E., & Melton, A. M. (2008). The computer aversion, attitudes, and familiarity index (CAAFI): A validity study. Computers in Human Behavior, 24(6), 2620-2638. https://doi.org/10.1016/j.chb.2008.03.002
Suvorov, R., & Hegelheimer, V. (2014). Computer-assisted language testing (w) Kunnan AJ (red.), The Companion to Language Assessment (Vol. 2, Chap. 36). In: London: John Wiley & Sons.
Taylor, C., Kirsch, I., Jamieson, J., & Eignor, D. (1999). Examining the relationship between computer familiarity and performance on computer-based language tasks. Language Learning, 49(2), 219-274. https://doi.org/10.1111/0023-8333.00088
Tella, A., & Bashorun, M. (2012). Attitude of undergraduate students towards computer-based test (cbt): a case study of the university of Ilorin, Nigeria. International Journal of Information and Communication Technology Education (IJICTE), 8(2), 33-45. https://doi.org/10.4018/jicte.2012040103
Voss, E. (2018). Technology and Assessment. The TESOL Encyclopedia of English Language Teaching, 1-7.
Winke, P. M., & Isbell, D. R. (2018). Construct of listening. The TESOL Encyclopedia of English Language Teaching, 1-6. https://doi.org/10.1002/9781118784235.eelt0618
Wise, S. L. (2019). Controlling construct-irrelevant factors through computer-based testing: disengagement, anxiety, & cheating. Education Inquiry, 10(1), 21-33. https://doi.org/10.1080/20004508.2018.1490127
Yeom, S., & Jun, H. (2020). Young Korean EFL Learners’ Reading and Test-Taking Strategies in a Paper and a Computer-Based Reading Comprehension Tests. Language Assessment Quarterly, 1-18. https://doi.org/10.1080/15434303.2020.1731753
Zhang, J. M., & Abdul Razak, R. B. (2026). Literature Review: Analysing the Factors Affecting College Students’ Digital Language Tests. Africa Education Review, 1-36.

Files

Share

How to cite

Statistics