AI-assisted learning and the illusion of competence: Measuring the AI-learning gap through random knowledge verification

Đức Minh

doi:10.32674/nrmwyw65

Authors

Đức Minh Thu Dau Mot University

DOI:

https://doi.org/10.32674/nrmwyw65

Keywords:

AI–Learning Gap, assessment, generative artificial intelligence, higher education, knowledge mastery

Abstract

This study examines the AI–Learning Gap (ALG), defined as the difference between the quality of AI-assisted academic work and students’ independently demonstrated knowledge mastery. Using data from 1,498 undergraduate students across 38 classes at a Vietnamese public university, the study developed a random knowledge verification (RKV) approach combining oral questioning and written recall tasks under device-restricted conditions. The results showed a substantial gap between assignment quality (M = 7.62) and knowledge mastery (M = 5.55), producing an average ALG of 2.07. The findings suggest that AI-assisted tasks may improve the quality of academic outputs without necessarily strengthening conceptual understanding. The study proposes RKV as a practical assessment approach for evaluating learning in AI-integrated higher education.

References

Adeshola, I., & Adepoju, A. P. (2024). The opportunities and challenges of ChatGPT in education. Interactive Learning Environments, 32(10), 6159–6172. https://doi.org/10.1080/10494820.2023.2253858

Albadarin, Y., Saqr, M., Pope, N., & Tukiainen, M. (2024). A systematic literature review of empirical research on ChatGPT in education. Discover Education, 3(1), 60. https://doi.org/10.1007/s44217-024-00138-2

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. AERA.

Association for Writing Across the Curriculum. (2025). AWAC statement on AI and writing across the curriculum. https://wacassociation.org/statement-on-ai-writing-tools-in-wac/

Baek, C., Tate, T., & Warschauer, M. (2024). ChatGPT seems too good to be true: College students’ use and perceptions of generative AI. Computers and Education: Artificial Intelligence, 6, Article 100294. https://doi.org/10.1016/j.caeai.2024.100294

Baig, M. I., & Yadegaridehkordi, E. (2024). ChatGPT in higher education: A systematic literature review and research challenges. International Journal of Educational Research, 127, 102411. https://doi.org/10.1016/j.ijer.2024.102411

Belland, B. R. (2013). Scaffolding: Definition, current debates, and future directions. In M. J. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 505–518). Springer. https://doi.org/10.1007/978-1-4614-3185-5_39

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. https://doi.org/10.1146/annurev-psych-113011-143823

Bobula, M. (2024). Generative artificial intelligence (AI) in higher education: A comprehensive review of challenges, opportunities, and implications. Journal of Learning Development in Higher Education, (30), Article 1137. https://doi.org/10.47408/jldhe.vi30.1137

Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20, Article 38. https://doi.org/10.1186/s41239-023-00408-3

Chauncey, S. A., & McKenna, H. P. (2023). A framework and exemplars for ethical and responsible use of AI chatbot technology to support teaching and learning. Computers and Education: Artificial Intelligence, 5, Article 100182. https://doi.org/10.1016/j.caeai.2023.100182

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148

Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assistance on student agency. Computers and Education, 210, Article 104967. https://doi.org/10.1016/j.compedu.2023.104967

Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, 1206936. https://doi.org/10.3389/feduc.2023.1206936

Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE Publications.

Fiorella, L., & Mayer, R. E. (2015). Learning as a generative activity: Eight learning strategies that promote understanding. Cambridge University Press. https://doi.org/10.1017/CBO9781107707085

Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846

Graham, S. (2019). Changing how writing is taught. Review of Research in Education, 43(1), 277–303. https://doi.org/10.3102/0091732X18821125

Graham, S., & Harris, K. R. (2006). Strategy instruction and the teaching of writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 187–207). Guilford Press.

Graham, S., MacArthur, C. A., & Hebert, M. (Eds.). (2013). Best practices in writing instruction (3rd ed.). Guilford Press.

Hodges, T. S., Wright, K. L., Wind, S. A., Matthews, S. D., Zimmer, W. K., & McTigue, E. (2019). Developing and examining validity evidence for the writing rubric to inform teacher educators (WRITE). Assessing Writing, 40, 1–13. https://doi.org/10.1016/j.asw.2019.03.001

Holmes, W., & Miao, F. (2023). Guidance for generative AI in education and research. UNESCO Publishing. https://doi.org/10.54675/EWZM9535

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274

Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973–1018. https://doi.org/10.1007/s10639-022-11177-3

Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1), 56. https://doi.org/10.1186/s41239-023-00426-1

Langer, J. A., & Applebee, A. N. (1987). How writing shapes thinking: A study of teaching and learning (NCTE Research Report No. 22). ERIC.

Levine, S., Beck, S. W., Mah, C., Phalen, L., & Pittman, J. (2024). How do students use ChatGPT as a writing support? Journal of Adolescent & Adult Literacy. Advance online publication. https://doi.org/10.1002/jaal.1373

Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779

Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727

Mai, D. T. T., Da, C. V., & Hanh, N. V. (2024). The use of ChatGPT in teaching and learning: A systematic review through SWOT analysis approach. Frontiers in Education, 9, 1328769. https://doi.org/10.3389/feduc.2024.1328769

Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D., Thierry-Aguilera, R., & Gerardou, F. (2023). Challenges and opportunities of generative AI for higher education as explained by ChatGPT. Education Sciences, 13(856), 1–18. https://doi.org/10.3390/educsci13090856

Modern Language Association and Conference on College Composition and Communication. (2024). Generative AI and policy development: Guidance from the MLA-CCCC task force. https://cccc.ncte.org/mla-cccc-joint-task-force-on-writing-and-ai

Mollick, E. (2024). Co-intelligence: Living and working with AI. Random House UK.

Mollick, E., & Mollick, L. (2023). Assigning AI: Seven approaches for students, with prompts. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4475995

Molenaar, I. (2022). Towards hybrid human–AI learning technologies. European Journal of Education, 57(4), 632–645. https://doi.org/10.1111/ejed.12527

Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023). Impact of the implementation of ChatGPT in education: A systematic review. Computers, 12(8), 153. https://doi.org/10.3390/computers12080153

Munaye, Y. Y., Admass, W., Belayneh, Y., Molla, A., & Asmare, M. (2025). ChatGPT in education: A systematic review on opportunities, challenges, and future directions. Algorithms, 18(6), 352. https://doi.org/10.3390/a18060352

Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, 100041. https://doi.org/10.1016/j.caeai.2021.100041

Nückles, M., Hübner, S., & Renkl, A. (2020). The self-regulation view in writing-to-learn: Using journal writing to optimize cognitive load in self-regulated learning. Educational Psychology Review, 32(3), 753–778. https://doi.org/10.1007/s10648-020-09528-1

Poe, M., & Elliot, N. (2019). Evidence of fairness: Twenty-five years of research in Assessing Writing. Assessing Writing, 42, 100418. https://doi.org/10.1016/j.asw.2019.100418

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage Publications.

Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002

Selwyn, N. (2024). On the limits of artificial intelligence (AI) in education. Nordisk Tidsskrift for Pedagogikk og Kritikk, 10(1), 3–14. https://doi.org/10.23865/ntpk.v10.6062

Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). SAGE Publications.

Tillmanns, T., Salomão Filho, A., Rudra, S., Weber, P., Dawitz, J., Wiersma, E., Dudenaite, D., & Reynolds, S. (2025). Mapping tomorrow’s teaching and learning spaces: A systematic review on GenAI in higher education. Trends in Higher Education, 4(1), 2. https://doi.org/10.3390/higheredu4010002

Van de Pol, J., Volman, M., & Beishuizen, J. (2010). Scaffolding in teacher–student interaction: A decade of research. Educational Psychology Review, 22(3), 271–296. https://doi.org/10.1007/s10648-010-9127-6

Walter, Y. (2024). Embracing the future of artificial intelligence in the classroom: The relevance of AI literacy, prompt engineering, and critical thinking in modern education. International Journal of Educational Technology in Higher Education, 21(1), 15. https://doi.org/10.1186/s41239-024-00448-3

Walters, W. H. (2023). The effectiveness of software designed to detect AI-generated writing: A comparison of 16 AI text detectors. Open Information Science, 7(1), 20220158. https://doi.org/10.1515/opis-2022-0158

Wang, C., & Tian, Z. (2025). Rethinking writing education in the age of generative AI. Taylor & Francis. https://doi.org/10.4324/9781003426936

Wang, C. Z., Aguilar, S. J., Bankard, J. S., Bui, E., & Nye, B. (2024). Writing with AI: What college students learned from utilizing ChatGPT for a writing assignment. Education Sciences, 14(9), Article 976. https://doi.org/10.3390/educsci14090976

Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13.

Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x