A Meta-Analysis of ChatGPT's Influence on Learning Achievement

Min Young Doo; Yeonjeong Park

doi:10.19173/irrodl.v27i1.8775

Authors

Min Young Doo Department of Education, College of Education, Kangwon National University https://orcid.org/0000-0003-3565-2159
Yeonjeong Park Department of Education, College of Education, Jeonbuk National University https://orcid.org/0000-0003-4960-8300

DOI:

https://doi.org/10.19173/irrodl.v27i1.8775

Keywords:

ChatGPT, learning achievement, generative AI, AIED, artificial intelligence in education

Abstract

This meta-analysis synthesized empirical findings on the influence of ChatGPT on learning achievement. An electronic database search using PRISMA guidelines was conducted with relevant keywords to identify eligible research studies published between November 2022 and December 2024. A total of 22 eligible publications that met our inclusion criteria were reviewed. The overall effect size of ChatGPT's influence on learning achievement was moderate (g= 0.573), suggesting that ChatGPT has the potential to improve learning outcomes. Most participants in the studies were undergraduates (70.9%). However, subgroup analysis revealed that the effect size for middle and high school students (g= 0.928) was larger than that for undergraduates (g= 0.538), although the difference was not statistically significant. This finding highlights the importance for instructors and educational practitioners to consider the applications of ChatGPT in middle and high school settings. No significant statistical differences were found among the three learning domains: cognitive (g= 0.612), affective (g= 0.481), and metacognitive (g= 0.619). Given that nearly half of the studies focused on the cognitive domain, it is important to diversify the application of generative AI across a variety of subjects in different learning domains. The most frequently used instructional approaches with ChatGPT applications were lectures (22.1%) and self-regulated learning (16.3%). The largest effect sizes were observed for self-regulated learning (g= 1.115) and case-based learning (g= 0.836), while the smallest effect size was for game-based learning (g= 0.092, ns). This study was conducted within two years of ChatGPT's emergence, limiting in our ability to analyze a large number of publications. Nevertheless, this study offers meaningful implications for future research on the application of ChatGPT for educational purposes.

Author Biographies

Min Young Doo, Department of Education, College of Education, Kangwon National University

Min Young Doo is an associate professor in the Department of Education at Kangwon National University, South Korea. Her research interests include online learning, AI in education (AIED), learning engagement, and research methodologies such as meta-analysis and structural equation modeling.

Yeonjeong Park, Department of Education, College of Education, Jeonbuk National University

Yeonjeong Park is an associate professor in the Department of Education at Jeonbuk National University, South Korea. Her research interests are social theories of learning, mobile learning, educational data mining, learning analytics, and AI in education.

References

Alneyadi, S., & Wardat, Y. (2023). ChatGPT: Revolutionizing student achievement in the electronic magnetism unit for eleventh-grade students in Emirates schools. Contemporary Educational Technology, 15(4), Article ep448. https://doi.org/10.30935/cedtech/13417

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives: Complete edition. Addison Wesley Longman, Inc.

Becker, E., Wünsche, J., Veith, J. M., Schrader, J., Bitzenbauer, P. (2025). From cognitive relief to affective engagement: An empirical comparison of AI chatbots and instructional scaffolding in physics education. arXiv. https://doi.org/10.48550/arXiv.2508.06254

Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York, NY: David McKay.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons.

Buchner, J., & Kerres, M. (2023). Media comparison studies dominate comparative research on augmented reality in education. Computers & Education, 195, Article 104711. https://doi.org/10.1016/j.compedu.2022.104711

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Rand-McNally.

Chang, C.-C., & Hwang, G.-H. (2024). Promoting professional trainers’ teaching and feedback competences with ChatGPT: A question-exploration-evaluation training mode. Educational Technology & Society, 27(2), 405–421. https://doi.org/10.30191/ETS.202404_27(2).TP06

Chang, C.-C., & Hwang, G.-J. (2024). ChatGPT facilitated professional development: Evidence from professional trainers’ learning achievements, self-worth, and self-confidence. Interactive Learning Environments, 33(1), 883–900. https://doi.org/10.1080/10494820.2024.2362798

Chang, C.-Y., Yang, C.-L., Jen, H.-J., Ogata, H., & Hwang, G.-H. (2024). Facilitating nursing and health education by incorporating ChatGPT into learning designs. Educational Technology & Society, 27(1), 215–230. https://doi.org/10.30191/ETS.202401_27(1).TP02

Chen, Y.-C., & Hou, H.-T. (2024). A mobile contextualized educational game framework with ChatGPT interactive scaffolding for employee ethics training. Journal of Educational Computing Research, 62(7), 1737–1762. https://doi.org/10.1177/07356331241268505

Coban, M., Bolat, Y. I., & Goksu, I. (2022). The potential immersive virtual reality to enhance learning: A meta-analysis. Educational Research Review, 36, Article 100452. https://doi.org/10.1016/j.edurev.2022.100452

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Council of Europe. (2022). Artificial intelligence and education. https://www.coe.int/en/web/education/artificial-intelligence

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.

Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20, Article 22. https://doi.org/10.1186/s41239-023-00392-8

Dalgıç, A., Yaşar, E., & Demir, M. (2024). ChatGPT and learning outcomes in tourism education: The role of digital literacy and individualized learning. Journal of Hospitality, Leisure, Sport & Tourism Education, 34, Article 100481. https://doi.org/10.1016/j.jhlste.2024.100481

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 227, Article 105224. https://doi.org/10.1016/j.compedu.2024.105224.

Díaz, B., & Nussbaum, M. (2024). Artificial intelligence for teaching and learning in schools: The need for pedagogical intelligence. Computers & Education, 217, Article 105071. https://doi.org/10.1016/j.compedu.2024.105071

Dikilitaş, K., Klippen, M. I. F., & Keles, S. (2024). A systematic rapid review of empirical research on students’ use of ChatGPT in higher education. Nordic Journal of Systematic Reviews in Education, 2, 103–125.

Doo, M. Y., Zhu, M., & Bonk, C. J. (2023). MOOC learners’ self-directed learning and learning outcomes: A meta-analysis. Distance Education, 44(1), 86-105. https://doi.org/10.1080/01587919.2022.2155618

Driscoll, M. P. (1993). Psychology of learning for instruction (2nd ed.). Allyn & Bacon.

Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ (Clinical Research Ed.), 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629

Essel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024). ChatGPT effects on cognitive skills of undergraduate students: Receiving instant responses from AI-based conversational large language models (LLMs). Computers and Education: Artificial Intelligence, 6, Article 100198. https://doi.org/10.1016/j.caeai.2023.100198

Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2024). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544

Gagne, R. M. & Briggs, L. J. (1984). Principles of instructional design. Holt, Rinehart, and Winston.

Garcia-Varela, F., Beckerman, Z., Nussbuam, M., & Mendoza, M. (2025). Reducing interpretative ambiguity in an educational environment with ChatGPT. Computers & Education, 225, Article 105182. https://doi.org/10.1016/j.compedu.2024.105182

Garger, J., Jacques, P. H., Gastle, B. W., & Connolly, C. M. (2019). Threats of common method variance in student assessment of instruction instruments. Higher Education Evaluation and Development, 13(1), 2–17. https://doi.org/10.1108/HEED-05-2018-0012

Gunawardena, M., Bishop, P., & Aviruppola, K. (2024). Personalized learning: The simple, the complicated, the complex and the chaotic. Teaching and Teacher Education, 139, Article 104429. https://doi.org/10.1016/j.tate.2023.104429

Guo, K., Zhong, Y., Li, D., & Chu, S. K. W. (2023). Effects of chatbot-assisted in-class debates on students’ argumentation skills and task motivation. Computers & Education, 203, Article 104862. https://doi.org/10.1016/j.compedu.2023.104862

Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186

Hsu, M.-H. (2024). Mastering medical terminology with ChatGPT and Termbot. Health Education Journal, 83(4), 352–358. https://doi.org/10.1177/0017896923119737

Huesca, G., Martínez-Trevino, Y., Molina-Espinosa, J. M., Sanromán-Calleros, A. R., Martínez-Román, R., Cendejas-Castro, E., & Bustos, R. (2024). Effectiveness of using ChatGPT as a tool to strengthen benefits of the flipped learning strategy. Educational Science, 14, Article 660. https://doi.org/10.3390/educsci14060660

Jeon, J., Lee, S. & Choe, H. (2023). Beyond ChatGPT: A conceptual framework and systematic review of speech-recognition chatbots for language learning. Computers and Education, 206, Article 104898. https://doi.org/10.1016/j.compedu.2023.104898

Lee, H.-Y., Chen, P.-H., Wang, W.-S., Huang, Y.-M., & Wu, T.-T. (2024). Empowering ChatGPT with guidance mechanism in blended learning: Effect of self-regulated learning, higher-order thinking skills, and knowledge construction. International Journal of Educational Technology in Higher Education, 21, Article 16. https://doi.org/10.1186/s41239-024-00447-4

Lee, S. S., & Moore, R. L. (2024). Harnessing generative AI (GenAI) for automated feedback in higher education: A systematic review. Online Learning, 28(3), 82–106. https://doi.org/10.24059/olj.v28i3.4593

Li, H.-F. (2023). Effects of a ChatGPT-based flipped learning guiding approach on learners’ courseware project performances and perceptions. Australasian Journal of Educational Technology, 39(5), 40–58. https://doi.org/10.14742/ajet.8923

Liang, H.-Y., Hwang, G.-J., Hsu, T.-Y., & Yeh, J.-Y. (2024). Effect of an AI-based chatbot on students’ learning performance in alternate reality game-based museum learning. British Journal of Educational Technology, 55(5). 2315–2338. https://doi.org/10.1111/bjet.13448

Liao, X., Zhang, X., Wang, Z., & Luo, H. (2024). Design and implementation of an AI-enabled visual report tool as formative assessment to promote learning achievement and self-regulated learning: An experimental study. British Journal of Educational Technology, 55(3), 1253–1276. https://doi.org/10.1111/bjet.13424

Lo, C. K., Hew, K. F., & Jong, M. S.-y. (2024). The influence of ChatGPT on student engagement: A systematic review and future research agenda. Computers & Education, 219, Article 105100. https://doi.org/10.1016/j.compedu.2024.105100

Martin, A. J., Collie, R. J., Kennett, R., Liu, D., Ginns, P., Sudimantara, L. B., ... & Rüschenpöhler, L. G. (2025). Integrating generative AI and load reduction instruction to individualize and optimize students' learning. Learning and Individual Differences, 121, 102723. https://doi.org/10.1016/j.lindif.2025.102723

Ng, D. T. K., Tan, C. W., & Leung, J. K. L. (2024). Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study. British Journal of Educational Technology, 55(4), 1328–1353. https://doi.org/10.1111/bjet.13454

Nwana, H. S. (1990). Intelligent tutoring systems: An overview. Artificial Intelligence Review, 4, 251–277. https://doi.org/10.1007/BF00168958

Patac, L. P., & Patac Jr, A. V. (2025). Using ChatGPT for academic support: Managing cognitive load and enhancing learning efficiency–A phenomenological approach. Social Sciences & Humanities Open, 11, 101301. https://doi.org/10.1016/j.ssaho.2025.101301

Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12, Article 22. https://doi.org/10.1186/s41039-017-0062-8

PRISMA. (2020). PRISMA flow diagram. https://www.prisma-statement.org/prisma-2020-flow-diagram

Reiser, R. A. (2001). A history of instructional design and technology: Part I: A history of instructional media. Educational Technology Research and Development, 49(1), 53–64. https://doi.org/10.1007/BF02504506

Reuters. (2024, August 30). OpenAI says ChatGPT’s weekly users have grown to 200 million. https://www.reuters.com/technology/artificial-intelligence/openai-says-chatgpts-weekly-users-have-grown-200-million-2024-08-29/

Rospigliosi, P. (2023). Artificial intelligence in teaching and learning: What questions should we ask of ChatGPT? Interactive Learning Environments, 31(1), 1–3. https://doi.org/10.1080/10494820.2023.2180191

Sabzalieva, E., & Valentini, A. (2023). ChatGPT and artificial intelligence in higher education: Quick start guide. UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000385146

Saettler, P. (1990). The evolution of American educational technology. Libraries Unlimited.

Schoenherr, J., Strohmaier, A. R, & Schukajlow, S. (2024). Learning with visualizations helps: A meta-analysis of visualization intervention in mathematics education. Educational Research Review, 45, Article 100639. https://doi.org/10.1016/j.edurev.2024.100639

Smith, P. L., & Ragan, T. J. (2005). Instructional design (3rd ed.). Wiley.

Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347. https://doi.org/10.1037/a0034752

Sun, L., & Zhou, L. (2024). Does generative artificial intelligence improve the academic achievement of colleges? A meta-analysis. Journal of Educational Computing Research, 62(7), 1673-1713. https://doi.org/10.1177/07356331241277937

Svendsen, K., Askar, M., Umer, D., & Halvorsen, K. H. (2024). Short-term learning effect of ChatGPT on pharmacy students’ learning. Exploratory Research in Clinical and Social Pharmacy, 15, Article 100478. https://doi.org/10.1016/j.rcsop.2024.100478

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4

Teng, M. F. (2024). “ChatGPT is the companion, not enemies”: EFL learners’ perceptions and experiences in using ChatGPT for feedback in writing. Computers & Education: Artificial Intelligence, 7, Article 100270. https://doi.org/10.1016/j.caeai.2024.100270

Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT improves creative problem-solving performance in university students: An experimental study. Computers & Education, 215, Article 105031. https://doi.org/10.1016/j.compedu.2024.105031

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.

Warnick, B. R., & Burbules, N. C. (2007). Media comparison studies: Problems and possibilities. Teachers College Record, 109(11), 2483–2510. https://eric.ed.gov/?id=EJ820500

Wu, R., & Yu, Z. (2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55(1), 10–33. https://doi.org/10.1111/bjet.13334

Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024). Does ChatGPT help with introductory programming? An experiment of using students ChatGPT in CS1. In A. Paiva, R. Abreu, K. Gama, & J. Siegmund (Chairs), ICSE-SEET '24: Proceedings of the 46th international conference on software engineering: Software engineering education and training (pp. 331–341). ACM. https://doi.org/10.1145/3639474.3640076

Yilmaz, R., & Yilmaz. F. G. K. (2023). The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers & Education: Artificial Intelligence, 4, Article 100147. https://doi.org/10.1016/j.caeai.2023.100147

Zhang, K., & Aslan, A. B. (2021). AI technologies for education: Recent research and future directions. Computers and Education: Artificial Intelligence, 2, Article 100025. https://doi.org/10.1016/j.caeai.2021.100025

Zhou, W., & Kim, Y. (2024). Innovative music education: An empirical assessment of ChatGPT-4’s impact on student learning experiences. Education and Information Technology, 29, 20855–2088. https://doi.org/10.1007/s10639-024-12705-z