Assessment of Learner Engagement and Expert Evaluations of AI-Generated Versus Human-Created Interactive Content in an Online Course

Hamza Aydemir; Şeyda Kır

doi:10.19173/irrodl.v26i4.8608

Authors

Hamza Aydemir Faculty of Communication, Kahramanmaraş İstiklal University, Kahramanmaraş, Türkiye https://orcid.org/0000-0002-2657-3195
Şeyda Kır Technology Transfer Office, Yozgat Bozok University, Yozgat, Türkiye https://orcid.org/0000-0002-6270-8034

DOI:

https://doi.org/10.19173/irrodl.v26i4.8608

Keywords:

Generative AI, online education, learner engagement, higher education, case study

Abstract

Generative artificial intelligence (GenAI) has introduced a novel aspect to educational methodologies and sparked fresh dialogues regarding the creation and evaluation of instructional resources. This project seeks to investigate the impact of GenAI on the development and assessment of online course materials and learners’ engagement with these materials in the online learning environment. The study analyzed GenAI-generated multiple-choice questions, fill-in-the-blank exercises, and true-false activities during 3 weeks of a 14-week online course. Subject matter experts assessed these documents in regards to content, relevance, and clarity. Data was collected through an online form with open-ended questions. The interactions of learners with the GenAI-created learning activities were analyzed using log records of the learning management system and compared to the content provided by the course instructor regarding interaction levels. The study’s conclusions elucidate the capability of GenAI technologies to produce course-specific content and their efficacy in education. We stress that human specialists’ critical evaluations play a crucial part in improving the pedagogical validity of GenAI-powered learning materials. Further research into topics including the ethical dimension, the effect on academic achievement, and student motivation is recommended.

Author Biographies

Hamza Aydemir, Faculty of Communication, Kahramanmaraş İstiklal University, Kahramanmaraş, Türkiye

Hamza Aydemir is a lecturer of the Digital Game Design Department at Kahramanmaraş İstiklal University. His research explores the integration of new technologies into learning, focusing on educational technology, game design and development, and the applications of artificial intelligence. He investigates how these technological interventions impact student motivation and academic achievement.

Şeyda Kır, Technology Transfer Office, Yozgat Bozok University, Yozgat, Türkiye

Şeyda Kır is an instructor at Yozgat Bozok University. She graduated from Anadolu University, Distance Education Department Master’s Program in 2019. Her research interests are MOOCs, OERs, micro-credentials, open pedagogy, lifelong learning, adult education, and open and distance learning.

References

Alasadi, E. A., & Baiz, C. R. (2023). Generative AI in education and research: Opportunities, concerns, and solutions. Journal of Chemical Education, 100(8), 2965–2971. https://doi.org/10.1021/acs.jchemed.3c00323

Ali, F., & OpenAI, Inc., C. (2023). Let the devil speak for itself: Should ChatGPT be allowed or banned in hospitality and tourism schools? Journal of Global Hospitality and Tourism, 2(1), 1–6. https://doi.org/10.5038/2771-5957.2.1.1016

Anderson, T. (2003). Getting the mix right again: An updated and theoretical rationale for interaction. The International Review of Research in Open and Distributed Learning, 4(2). https://doi.org/10.19173/irrodl.v4i2.149

Aydemir, H., & Seferoğlu, S. S. (2024). Investigation of users’ opinions, perceptions and attitudes towards the use of artificial intelligence in online learning environments. European Journal of Open, Distance and E-Learning, 26(1), 70–92. https://doi.org/10.2478/eurodl-2024-0007

Bachiri, Y.-A., & Mouncif, H. (2023). Artificial intelligence system in aid of pedagogical engineering for knowledge assessment on MOOC platforms: Open EdX and Moodle. International Journal of Emerging Technologies in Learning (iJET), 18(05), 144–160. https://doi.org/10.3991/ijet.v18i05.36589

Bachiri, Y.-A., Mouncif, H., & Bouikhalene, B. (2023). Artificial intelligence empowers gamification: Optimizing student engagement and learning outcomes in e-learning and MOOCs. International Journal of Engineering Pedagogy (iJEP), 13(8), 4–19. https://doi.org/10.3991/ijep.v13i8.40853

Baidoo-anu, D., & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52–62. https://doi.org/10.61969/jai.1337500

Bozkurt, A. (2023). ChatGPT, üretken yapay zekâ ve algoritmik paradigma değişikliği [ChatGPT, generative artificial intelligence and algorithmic paradigm shift]. Alanyazın 4(1), 63–72. https://doi.org/10.59320/alanyazin.1283282

Chen, X., Zou, D., Xie, H., Cheng, G., & Liu, C. (2022). Two decades of artificial intelligence in education. Educational Technology & Society, 25(1), 28–47. https://www.jstor.org/stable/48647028

Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Sage.

DaCosta, B., & Kinsell, C. (2024). Investigating media selection through ChatGPT: An exploratory study on generative artificial intelligence in the aid of instructional design. Open Journal of Social Sciences, 12(4), 187–227. https://doi.org/10.4236/jss.2024.124014

Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. Sci, 6(1), Article 3. https://doi.org/10.3390/sci6010003

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6

Guo, Z., Jing, T., Zhu, H., & Xu, X. (2023). Analysis of the impact of artificial intelligence on college students’ learning. The Frontiers of Society, Science and Technology, 5(13), 110–116. https://doi.org/10.25236/fsst.2023.051318

Haleem, A., Javaid, M., & Singh, R. P. (2022). An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2(4), Article 100089. https://doi.org/10.1016/j.tbench.2023.100089

Huang, J. Shen, G., & Ren, X. (2021). Connotation analysis and paradigm shift of teaching design under artificial intelligence technology. International Journal of Emerging Technologies in Learning (iJET), 16(05), 73–86. https://doi.org/10.3991/ijet.v16i05.20287

Jamilah, Jayadi, K., Yatim, H., Sahnir, N., Djirong, A., & Abduh, A. (2024). The integration of local cultural arts in the context of teaching materials on the implementation of the Merdeka Belajar curriculum. Journal of Education Research and Evaluation, 8(2), 404–413. https://doi.org/10.23887/jere.v8i2.78359

Jouault, C., Seta, K., & Hayashi, Y. (2016). Content-dependent question generation using LOD for history learning in open learning space. New Generation Computing, 34(4), 367–394. https://doi.org/10.1007/s00354-016-0404-x

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., ... Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274

Kır, Ş., & Şenocak, D. (2022). Açık ve uzaktan öğrenme sistemlerinde yapay zekânın öğrenen destek hizmeti bağlamında kullanımı [The use of artificial intelligence in the context of learner support services in open and distance learning systems]. Dijital Teknolojiler Ve Eğitim Dergisi, 1(1), 39–56. https://doi.org/10.5281/zenodo.6647642

Kriangchaivech, K., & Wangperawong, A. (2019). Question generation by transformers. ArXiv. https://doi.org/10.48550/arxiv.1909.05017

Lee , E. (2023). Is ChatGPT a false promise? Berkeley Blog. https://coilink.org/20.500.12592/hzdz6k

Liang, Z., Zhao, Q., Zhou, Z., Yu, Q., Li, S., & Chen, S. (2020). The effect of “novelty input” and “novelty output” on boredom during home quarantine in the COVID-19 pandemic: The moderating effects of trait creativity. Frontiers in Psychology, 11, Article 601548. https://doi.org/10.3389/fpsyg.2020.601548

Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2), Article 100790. https://doi.org/10.1016/j.ijme.2023.100790

Lodge, J. M., Thompson, K., & Corrin, L. (2023). Mapping out a research agenda for generative artificial intelligence in tertiary education. Australasian Journal of Educational Technology, 39(1), 1–8. https://doi.org/10.14742/ajet.8695

Nasution, N. E. A. (2023). Using artificial intelligence to create biology multiple choice questions for higher education. Agricultural and Environmental Education, 2(1), Article em002. https://doi.org/10.29333/agrenvedu/13071

Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B.-P. T. (2023). Ethical principles for artificial intelligence in education. Education and Information Technologies, 28, 4221–4241. https://doi.org/10.1007/s10639-022-11316-w

Sihite, M. R., Meisuri, M., & Sibarani, B. (2023). Examining the validity and reliability of ChatGPT 3.5-generated reading comprehension questions for academic texts. Randwick International of Education and Linguistics Science Journal, 4(4), 937–944. https://doi.org/10.47175/rielsj.v4i4.835

Skanda, V. C., Jayaram, R., Bukitagar, V. C., & Kumar, N. S. (2020). Automatic questionnaire and interactive session generation from videos. In A. Chandrabose, U. Furbach, A. Ghosh, & M. A. Kumar (Eds.), IFIP advances in information and communication technology (Vol. 578; pp. 205–212). Springer. https://doi.org/10.1007/978-3-030-63467-4_16

Sun, Z. (2010). Language teaching materials and learner motivation. Journal of Language Teaching and Research, 1(6), 889–892. https://doi.org/10.4304/jltr.1.6.889-892

Sundari, W., Siahaan, S. M., & Ismet, I. (2023). Digital teaching materials based on attention relevance confidence satisfaction substance pressure material. Journal of Curriculum Indonesia, 6(1), 69–75. https://doi.org/10.46680/jci.v6i1.77

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4

Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), Article 15. https://doi.org/10.1186/s40561-023-00237-x

Tran, S., Krishna, P., Pakuwal, I., Kafle, P., Singh, N., Lynch, J., & Drori, I. (2021). Solving machine learning problems. ArXiv. https://doi.org/10.48550/arxiv.2107.01238

Wood, D., & Moss, S. H. (2024). Evaluating the impact of students’ generative AI use in educational contexts. Journal of Research in Innovative Teaching & Learning, 17(2), 152–167. https://doi.org/10.1108/jrit-06-2024-0151

Xu, B., & Ismail, H. H. (2024). The impact of artificial intelligence-assisted learning applications on oral English ability: A literature review. International Journal of Academic Research in Progressive Education and Development, 13(4), 1118–1134. https://doi.org/10.6007/ijarped/v13-i4/23352

Yin, R. K. (2014). Case study research design and methods. Sage Publications.

Zabala-Vargas, S., García-Mora, L. H., Arciniegas-Hernández, E., Reina-Medrano, J. I., de Benito-Crosetti, B., & Darder-Mesquida, A. (2021). Strengthening motivation in the mathematical engineering teaching processes—A proposal from gamification and game-based learning. International Journal of Emerging Technologies in Learning (iJET), 16(06), 4–19. https://doi.org/10.3991/ijet.v16i06.16163

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education—Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0