Automated feedback for speaking and writing skills: Deep learning in English language assessment

Andi Asrifan, Luís Miguel Oliveira de Barros Cardoso, K.J. Vargheese

Abstract


The incorporation of Artificial Intelligence (AI) into language evaluation has revolutionized how learners receive feedback on their speaking and writing abilities. Nevertheless, empirical information about the precision and educational efficacy of AI-generated feedback—especially in advanced language competencies—continues to be scarce. This study seeks to evaluate the efficacy of deep learning–driven automated feedback systems in enhancing English learners' speaking and writing skills. The study utilized a mixed-methods research approach and included 100 undergraduate students participating in an English for Academic Purposes course, focusing on English as a Foreign Language (EFL). Quantitative data were gathered via pre-test and post-test writing and speaking activities evaluated using AI tools (Grammarly, ETS e-rater, and Google Automatic Speech Recognition), whilst qualitative data were derived from surveys and interviews to capture learners' impressions. The findings demonstrate statistically significant enhancements in grammatical accuracy, lexical diversity, coherence, fluency, pronunciation, and intelligibility following exposure to AI-generated feedback. However, inconsistencies were identified between AI and human assessments regarding speech coherence and contextual relevance. The results indicate that AI-generated feedback serves as an excellent additional evaluation instrument, especially for form-focused linguistic elements, however it is constrained in its ability to measure higher-order communication competencies. This study underscores the significance of amalgamating AI-driven feedback with human discernment to establish a more holistic and pedagogically robust language assessment framework.


Keywords


AI-generated feedback; deep learning; EFL learning; language assessment; speaking and writing skills

Full Text:

PDF

References


Alazemi, A. F. T. (2024). Formative assessment in artificial integrated instruction: delving into the effects on reading comprehension progress, online academic enjoyment, personal best goals, and academic mindfulness. In Language Testing in Asia (Vol. 14, Issue 1). Springer Science+Business Media. https://doi.org/10.1186/s40468-024-00319-8

Alnufaie, M. R., & Alzahrani, I. H. (2024). EFL Grammar Learning Strategy Use: Utilizing Grammar Learning Strategy Inventory in an Arabic Context. In Teaching English as a Second or Foreign Language--TESL-EJ (Vol. 27, Issue 4). https://doi.org/10.55593/ej.27108a6

Altamimi, O. A., & Masood, M. (2021). Teacher Electronic Written Corrective Feedback, Trends and Future Directions. In Arab World English Journal (Vol. 12, Issue 3, p. 308). Arab World English Journal. https://doi.org/10.24093/awej/vol12no3.21

AlTwijri, L., & Alghizzi, T. M. (2024). Investigating the integration of artificial intelligence in English as foreign language classes for enhancing learners’ affective factors: A systematic review [Review of Investigating the integration of artificial intelligence in English as foreign language classes for enhancing learners’ affective factors: A systematic review]. Heliyon, 10(10). Elsevier BV. https://doi.org/10.1016/j.heliyon.2024.e31053

Bai, B., & Wang, J. (2020). The role of growth mindset, self-efficacy and intrinsic value in self-regulated learning and English language learning achievements. In Language Teaching Research (Vol. 27, Issue 1, p. 207). SAGE Publishing. https://doi.org/10.1177/1362168820933190

Biju, N., Abdelrasheed, N. S. G., Bakiyeva, K., Prasad, K. D. V., & Jember, B. (2024). Which one? AI-assisted language assessment or paper format: an exploration of the impacts on foreign language anxiety, learning attitudes, motivation, and writing performance. In Language Testing in Asia (Vol. 14, Issue 1). Springer Science+Business Media. https://doi.org/10.1186/s40468-024-00322-z

Bulut, O., & Beiting-Parrish, M. (2024). The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges. In Chinese/English Journal of Educational Measurement and Evaluation (Vol. 5, Issue 3). https://doi.org/10.59863/miql7785

Campino, J. (2024). Unleashing the transformers: NLP models detect AI writing in education. In Journal of Computers in Education. Springer Science+Business Media. https://doi.org/10.1007/s40692-024-00325-y

Cotton, D., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. In Innovations in Education and Teaching International (Vol. 61, Issue 2, p. 228). Taylor & Francis. https://doi.org/10.1080/14703297.2023.2190148

D’Esposito, M. E. W., & Gatner, S. (2024). Inteligência artificial no ensino-aprendizagem de línguas. In The ESPecialist (Vol. 45, Issue 3, p. 134). Pontifícia Universidade Católica de São Paulo. https://doi.org/10.23925/2318-7115.2024v45i3e63941

Dong, Y. (2023). Revolutionizing Academic English Writing through AI-Powered Pedagogy: Practical Exploration of Teaching Process and Assessment. In Journal of Higher Education Research (Vol. 4, Issue 2, p. 52). https://doi.org/10.32629/jher.v4i2.1188

Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023). Automated feedback and writing: a multi-level meta-analysis of effects on students’ performance [Review of Automated feedback and writing: a multi-level meta-analysis of effects on students’ performance]. Frontiers in Artificial Intelligence, 6. Frontiers Media. https://doi.org/10.3389/frai.2023.1162454

Gao, R., Merzdorf, H., Anwar, S., Hipwell, M. C., & Srinivasa, A. R. (2024). Automatic assessment of text-based responses in post-secondary education: A systematic review [Review of Automatic assessment of text-based responses in post-secondary education: A systematic review]. Computers and Education Artificial Intelligence, 6, 100206. Elsevier BV. https://doi.org/10.1016/j.caeai.2024.100206

Getchell, K., Carradını, S., Cardon, P. W., Fleischmann, C., Ma, H., Aritz, J., & Stapp, J. (2022). Artificial Intelligence in Business Communication: The Changing Landscape of Research and Teaching. In Business and Professional Communication Quarterly (Vol. 85, Issue 1, p. 7). SAGE Publishing. https://doi.org/10.1177/23294906221074311

Giglio, A. del, & Costa, M. U. P. da. (2023). The use of artificial intelligence to improve the scientific writing of non-native english speakers [Review of The use of artificial intelligence to improve the scientific writing of non-native english speakers]. Revista Da Associação Médica Brasileira, 69(9). Brazilian Medical Association. https://doi.org/10.1590/1806-9282.20230560

Grájeda, A., Burgos, J., Olivera, P. C., & Sanjinés, A. (2023). Assessing student-perceived impact of using artificial intelligence tools: Construction of a synthetic index of application in higher education. In Cogent Education (Vol. 11, Issue 1). Taylor & Francis. https://doi.org/10.1080/2331186x.2023.2287917

Gu, L., Davis, L., Tao, J., & Zechner, K. (2020). Using spoken language technology for generating feedback to prepare for the TOEFL iBT® test: a user perception study. In Assessment in Education Principles Policy and Practice (Vol. 28, Issue 1, p. 58). Routledge. https://doi.org/10.1080/0969594x.2020.1735995

Guo, Q., Feng, R., & Hua, Y. (2021). How effectively can EFL students use automated written corrective feedback (AWCF) in research writing? In Computer Assisted Language Learning (Vol. 35, Issue 9, p. 2312). Routledge. https://doi.org/10.1080/09588221.2021.1879161

Guzik, E. E., Byrge, C., & Gilde, C. (2023). The originality of machines: AI takes the Torrance Test. In Journal of Creativity (Vol. 33, Issue 3, p. 100065). Elsevier BV. https://doi.org/10.1016/j.yjoc.2023.100065

Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023). Is GPT-4 a reliable rater? Evaluating consistency in GPT-4’s text ratings. In Frontiers in Education (Vol. 8). Frontiers Media. https://doi.org/10.3389/feduc.2023.1272229

Hohenstein, J., Kizilcec, R. F., DiFranzo, D., Aghajari, Z., Mieczkowski, H., Levy, K., Naaman, M., Hancock, J. T., & Jung, M. (2023). Artificial intelligence in communication impacts language and social relationships. In Scientific Reports (Vol. 13, Issue 1). Nature Portfolio. https://doi.org/10.1038/s41598-023-30938-9

Hopfenbeck, T. N., Zhang, Z., Sun, S. Z., Robertson, P., & McGrane, J. A. (2023). Challenges and opportunities for classroom-based formative assessment and AI: a perspective article. In Frontiers in Education (Vol. 8). Frontiers Media. https://doi.org/10.3389/feduc.2023.1270700

Jain, R., & Jain, A. (2024). Generative AI in Writing Research Papers: A New Type of Algorithmic Bias and Uncertainty in Scholarly Work. In Lecture notes in networks and systems (p. 656). Springer International Publishing. https://doi.org/10.1007/978-3-031-66329-1_42

Jamshed, M., Ahmed, A. S. M. M., Sarfaraj, M., & Warda, W. U. (2024). The Impact of ChatGPT on English Language Learners’ Writing Skills. In International Journal of Interactive Mobile Technologies (iJIM) (Vol. 18, Issue 19, p. 18). kassel university press. https://doi.org/10.3991/ijim.v18i19.50361

Khoiriyah, K. (2021). Flipping the classroom to enhance EFL students’ listening skill. In Journal on English as a Foreign Language (Vol. 11, Issue 1, p. 21). Institut Agama Islam Negeri (IAIN) Palangka Raya. https://doi.org/10.23971/jefl.v11i1.2010

Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., Leon, L. D., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. In PLOS Digital Health (Vol. 2, Issue 2). Public Library of Science. https://doi.org/10.1371/journal.pdig.0000198

Marzuki, M., Widiati, U., Rusdin, D., Darwin, D., & Indrawati, I. (2023). The impact of AI writing tools on the content and organization of students’ writing: EFL teachers’ perspective. In Cogent Education (Vol. 10, Issue 2). Taylor & Francis. https://doi.org/10.1080/2331186x.2023.2236469

Nazari, N., Shabbir, M. S., & Setiawan, R. (2021). Application of Artificial Intelligence powered digital writing assistant in higher education: randomized controlled trial. In Heliyon (Vol. 7, Issue 5). Elsevier BV. https://doi.org/10.1016/j.heliyon.2021.e07014

Oxford, R. L., Gu, Y., Gunning, P., & Hernández-González, T. (2024). Considerations in Designing and Validating the Diagnostic Inventory for Self-Regulated Language Learning (DISLL): Status of the Process. In Language Teaching Research Quarterly (Vol. 41, p. 4). https://doi.org/10.32038/ltrq.2024.41.02

Peters, T. M., & Visser, R. W. (2023). The Importance of Distrust in AI. In Communications in computer and information science (p. 301). Springer Science+Business Media. https://doi.org/10.1007/978-3-031-44070-0_15

Roshanaei, M., Olivares, H., & Lopez, R. R. (2023). Harnessing AI to Foster Equity in Education: Opportunities, Challenges, and Emerging Strategies. In Journal of Intelligent Learning Systems and Applications (Vol. 15, Issue 4, p. 123). Scientific Research Publishing. https://doi.org/10.4236/jilsa.2023.154009

Rukiati, E., Wicaksono, J. A., Taufan, G. T., & Suharsono, D. D. (2023). AI on Learning English: Application, Benefit, and Threat. In Journal of Language Communication and Tourism (Vol. 1, Issue 2, p. 32). https://doi.org/10.25047/jlct.v1i2.3967

Rusmiyanto, R., Huriati, N., Fitriani, N., Tyas, N. K., Rofi’i, A., & Sari, M. N. (2023). The Role Of Artificial Intelligence (AI) In Developing English Language Learner’s Communication Skills. In Journal on Education (Vol. 6, Issue 1, p. 750). Universitas Pahlawan Tuanku Tambusai. https://doi.org/10.31004/joe.v6i1.2990

Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. In Frontiers in Psychology (Vol. 14). Frontiers Media. https://doi.org/10.3389/fpsyg.2023.1260843

Steiss, J., Tate, T., Graham, S., Cruz, J., Hébert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. In Learning and Instruction (Vol. 91, p. 101894). Elsevier BV. https://doi.org/10.1016/j.learninstruc.2024.101894

Sumarni, B., Bhatta, D. D., & Kho, S. F.-C. (2022). The Use of Total Physical Response in Teaching Vocabulary Integrated with Meaningful Classroom Interaction. In Journal of Language and Literature Studies (Vol. 2, Issue 1, p. 23). https://doi.org/10.36312/jolls.v2i1.710

Syafryadin, S. (2022). Actualizing Learning English Grammar Strategy Using New Inventory for University Students. In PAROLE Journal of Linguistics and Education (Vol. 12, Issue 1, p. 25). Diponegoro University. https://doi.org/10.14710/parole.v12i1.25-35

Tang, X., Chen, H., Lin, D., & Li, K. (2024). Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments. In Heliyon (Vol. 10, Issue 14). Elsevier BV. https://doi.org/10.1016/j.heliyon.2024.e34262

The Role of Artificial Intelligence Technology on English Language Learning: A Literature Review. (2023). Canadian Journal of Language and Literature Studies, 3(2). https://doi.org/10.53103/cjlls.v3i2.87

Vasconcelos, M. A. R., & Santos, R. P. dos. (2023). Enhancing STEM learning with ChatGPT and Bing Chat as objects to think with: A case study. In Eurasia Journal of Mathematics Science and Technology Education (Vol. 19, Issue 7). Modestum Limited. https://doi.org/10.29333/ejmste/13313

Vasconcelos, M. A. R., & Santos, R. P. dos. (2023). Enhancing STEM learning with ChatGPT and Bing Chat as objects to think with: A case study. In Eurasia Journal of Mathematics Science and Technology Education (Vol. 19, Issue 7). Modestum Limited. https://doi.org/10.29333/ejmste/13313

Woo, J. H., & Choi, H. (2021). Systematic Review for AI-based Language Learning Tools. In Journal of Digital Contents Society (Vol. 22, Issue 11, p. 1783). Digital Contents Society. https://doi.org/10.9728/dcs.2021.22.11.1783

Xi, X. (2023). Advancing Language Assessment with AI and ML–Leaning into AI is Inevitable, but Can Theory Keep Up? In Language Assessment Quarterly (Vol. 20, p. 357). Taylor & Francis. https://doi.org/10.1080/15434303.2023.2291488

Xie, Y., Liu, Y., Zhang, F., & Ping, Z. (2022). Virtual Reality-Integrated Immersion-Based Teaching to English Language Learning Outcome. In Frontiers in Psychology (Vol. 12). Frontiers Media. https://doi.org/10.3389/fpsyg.2021.767363

Yong, Q. (2020). Application Analysis of Artificial Intelligence in Oral English Assessment. In Journal of Physics Conference Series (Vol. 1533, Issue 3, p. 32028). IOP Publishing. https://doi.org/10.1088/1742-6596/1533/3/032028

Zhuk, A. (2023). Navigating the legal landscape of AI copyright: a comparative analysis of EU, US, and Chinese approaches. In AI and Ethics (Vol. 4, Issue 4, p. 1299). Springer Nature. https://doi.org/10.1007/s43681-023-00299-0




DOI: http://dx.doi.org/10.30659/e.11.1.67-85

Refbacks

  • There are currently no refbacks.


Copyright (c) 2026 Andi Asrifan, Luís Miguel Oliveira de Barros Cardoso, K.J. Vargheese

License URL: https://creativecommons.org/licenses/by/4.0/