MOOC Evaluation System Based on Deep Learning
DOI:
https://doi.org/10.19173/irrodl.v22i4.5417Keywords:
MOOC, deep learning, learner satisfaction, learning analyticsAbstract
Massive open online courses (MOOCs) are open access, Web-based courses that enroll thousands of students. MOOCs deliver content through recorded video lectures, online readings, assessments, and both student–student and student–instructor interactions. Course designers have attempted to evaluate the experiences of MOOC participants, though due to large class sizes, have had difficulty tracking and analyzing the online actions and interactions of students. Within the broader context of the discourse surrounding big data, educational providers are increasingly collecting, analyzing, and utilizing student information. Additionally, big data and artificial intelligence (AI) technology have been applied to better understand students’ learning processes. Questionnaire response rates are also too low for MOOCs to be credibly evaluated. This study explored the use of deep learning techniques to assess MOOC student experiences. We analyzed students’ learning behavior and constructed a deep learning model that predicted student course satisfaction scores. The results indicated that this approach yielded reliable predictions. In conclusion, our system can accurately predict student satisfaction even when questionnaire response rates are low. Accordingly, teachers could use this system to better understand student satisfaction both during and after the course.
References
Adamopoulos, P. (2013). What makes a great MOOC? An interdisciplinary analysis of student retention in online courses. In R. Baskerville & M. Chau (Eds.), Proceedings of the 34th International Conference on Information Systems (ICIS 2013). http://aisel.aisnet.org/icis2013/proceedings/BreakthroughIdeas/13/
Alario-Hoyos, C., Estévez-Ayres, I., Pérez-Sanagustín, M., Delgado Kloos, C., & Fernández-Panadero, C. (2017). Understanding learners’ motivation and learning strategies in MOOCs. The International Review of Research in Open and Distributed Learning, 18(3), 120–-137. https://doi.org/10.19173/irrodl.v18i3.2996
Baxter Magolda, M. B. (1993). What doesn’t matter in college? [Review of the book What matters in college: Four critical years revisited, by A. W. Astin]. Journal of Higher Education 22(8), 32–34. https://doi.org/10.2307/1176821
Bean, J., & Bradley, R. (1986). Untangling the satisfaction-performance relationship for college students. The Journal of Higher Education, 57(4), 393–-412. https://doi.org/10.2307/1980994
Boulay, B. d. (2016). Artificial intelligence as an effective classroom assistant. IEEE Intelligent Systems, 31(6), 76-81. http://users.sussex.ac.uk/~bend/papers/meta-reviewsIEEEV5.pdf
Cakmak, M. (2011). Changing roles of teachers: Prospective teachers’ thoughts. Egitim ve Bilim, 36, 14–-24. https://www.webofscience.com/wos/woscc/full-record/WOS:000286914500002?SID=E3dPJYYg8zalgwNi8L7 http://egitimvebilim.ted.org.tr/index.php/EB/article/download/173/235
Chen, X., Xie, H., Zou, D., & Hwang, G. J. (2020). Application and theory gaps during the rise of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1, 100002. https://doi.org/10.1016/j.caeai.2020.100002
Dohaiha, H., Prasad, P. W. C., Maag, A., & Alsadoon, A. (2018). Deep learning for aspect-based sentiment analysis: A comparative review. Expert Systems with Applications, 118, 272–-299. https://doi.org/10.1016/j.eswa.2018.10.003
Donath, J. (1996). Identity and deception in the virtual community. In M. H. Smith & P. Kollock (Eds.), Communities in cyberspace (pp. 29–59). Routledge. https://smg.media.mit.edu/people/judith/Identity/IdentityDeception.html
Donohue, T. L., & Wong, E. H. (1997). Achievement motivation and college satisfaction in traditional and nontraditional students. Education, 118(2), 237–-243. https://go.gale.com/ps/anonymous?id=GALE%7CA20479498&sid=googleScholar&v=2.1&it=r&linkaccess=abs&issn=00131172&p=AONE&sw=w
Du Boulay, B. (2016). Artificial intelligence as an effective classroom assistant. IEEE Intelligent Systems, 31, 76–-81. https://doi.org/10.1109/MIS.2016.93https://www.computer.org/csdl/magazine/ex/2016/06/mex2016060076/13rRUxlgxVq
Durksen, T., Chu, M.-W., Ahmad, Z. F., Radil, A., & Daniels, L. (2016). Motivation in a MOOC: A probabilistic analysis of online learners’ basic psychological needs. Social Psychology of Education, 19, 241–260. https://doi.org/10.1007/s11218-015-9331-9
Fauvel, S., Yu, H., Miao, C., Cui, L., Song, H., Zhang, L., Li, X., & Leung, C. (2018). Artificial Intelligence Powered MOOCs: A Brief Survey [Paper presentation]. IEEE International Conference on Agents. Singapore. https://doi.org/10.1109/AGENTS.2018.8460059https://ieeexplore.ieee.org/document/8460059
Gameel, B. (2017). Learner satisfaction with massive open online courses. American Journal of Distance Education, 31(2), 98–111. https://doi.org/10.1080/08923647.2017.1300462
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x
Gonçalves, G. (2014). Studying public health. Results from a questionnaire to estimate medical students’ workload. Procedia - Social and Behavioral Sciences, 116, 2915–2919. https://doi.org/10.1016/j.sbspro.2014.01.679
Hew, K. F., Hu, X., Qiao, C., & Tang, Y. (2020). What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach. Computers & Education, 145, 103724. https://doi.org/10.1016/j.compedu.2019.103724
Hu, H., Zhang, G., Gao, W., & Wang, M. (2020). Big data analytics for MOOC video watching behavior based on Spark. Neural Computing and Applications, 32(11), 6481-6489. https://doi.org/10.1007/s00521-018-03983-z
Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. The International Review of Research in Open and Distributed Learning, 16(3). https://doi.org/10/f5vw
Kastrati, Z., Imran, A., & Kurti, A. (2020). Weakly supervised framework for aspect-based sentiment analysis on students’ reviews of MOOCs. IEEE Access, 8, 106799–106810. https://doi.org/10.1109/ACCESS.2020.3000739https://ieeexplore.ieee.org/document/9110884
Kay, J. (2012). AI and education: Grand challenges. Intelligent Systems, IEEE, 27(5), 66–-69. https://doi.org/10.1109/MIS.2012.92 https://www.researchgate.net/publication/260330353_AI_and_Education_Grand_Challenges
Klobas, J. E., Mackintosh, B., & Murphy, J. (2014). The anatomy of MOOCs. In P. Kim (Ed.), Massive open online courses: The MOOC revolution (pp. 1–-22). Routledge. https://www.researchgate.net/publication/272427367_The_anatomy_of_MOOCs
Kop, R. (2011). The challenges to connectivist learning on open online networks: Learning experiences during a massive open online course. The International Review of Research in Open and Distributed Learning, 12(3), 19–38. https://doi.org/10.19173/irrodl.v12i3.882
Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distributed Learning, 12(7), 74–93. https://doi.org/10.19173/irrodl.v12i7.1041
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education: A literature review. In A. Peña-Ayala (Ed.), Learning analytics: Fundamentals, applications, and trends (pp. 1–-23). Springer International Publishing AG. https://link.springer.com/chapter/10.1007/978-3-319-52977-6_1
Liu, M., Kang, J., Cao, M., Lim, M., Ko, Y., Myers, R., & Schmitz Weiss, A. (2014). Understanding MOOCs as an emerging online learning tool: Perspectives from the students. American Journal of Distance Education, 28(3), 147–159. https://doi.org/10.1080/08923647.2014.926145
Lundqvist, K., Liyanagunawardena, T., & Starkey, L. (2020). Evaluation of student feedback within a MOOC using sentiment analysis and target groups. The International Review of Research in Open and Distributed Learning, 21(3), 140–156. https://doi.org/10.19173/irrodl.v21i3.4783
Marciniak, R. (2018). Quality assurance for online higher education programmes: Design and validation of an integrative assessment model Applicable to Spanish universities. The International Review of Research in Open and Distributed Learning, 19(2), 127–154. https://doi.org/10.19173/irrodl.v19i2.3443
Marin, V., Pereira, T., Sridharan, S., & Rivero, C. (2017). Automated personalized feedback in Introduction Java Programming MOOCs. 2017 IEEE 33rd International Conference on Data Engineering (ICDE; pp. 1259–1270). https://doi.org/10.1109/ICDE.2017.169 https://ieeexplore.ieee.org/document/7930065
McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). In the open: The MOOC model for digital practice (Knowledge Synthesis Grants on the Digital Economy) [Grant Report]. Social Sciences & Humanities Research Council of Canada. https://www.oerknowledgecloud.org/archive/MOOC_Final.pdf
McNamara, D., Baker, R., Wang, Y., Paquette, L., Barnes, T., & Bergner, Y. (2015). Language to completion: Success in an educational data mining massive open online class. Proceedings of the 8th International Educational Data Mining Society (pp., 388–391). https://www.researchgate.net/publication/319764516_Language_to_Completion_Success_in_an_Educational_Data_Mining_Massive_Open_Online_Class
Moreno-Marcos, P., Alario-Hoyos, C., Munoz-Merino, P., Estévez-Ayres, I., & Delgado-Kloos, C. (2018). Sentiment analysis in MOOCs: A case study. IEEE Global Engineering Education Conference (EDUCON; pp. 1489–1496). https://ieeexplore.ieee.org/document/8363409
Ochando, H. M. P. (2017). Transcultural validation within the English scope of the questionnaire evaluation of variables moderating style of teaching in higher education. CEMEDEPU. Procedia-Social and Behavioral Sciences, 237, 1208–1215. https://doi.org/10.1016/j.sbspro.2017.02.191
Onah, D., Sinclair, J., & Boyatt, R. (2014). Dropout rates of massive open online courses: Behavioural patterns. Proceedings of the 6th International Conference on Education and New Learning Technologies (EDULEARN14). https://doi.org/0.13140/RG.2.1.2402.0009 https://www.researchgate.net/publication/273777281_Dropout_Rates_of_Massive_Open_Online_Courses_Behavioural_Patterns
Papamitsiou, Z., & Economides, A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64. https://www.j-ets.net/collection/published-issues/17_4 https://www.researchgate.net/publication/267510046_Learning_Analytics_and_Educational_Data_Mining_in_Practice_A_Systematic_Literature_Review_of_Empirical_Evidence
Pérez, R. C., Jurado, F., & Villen, A. (2019). Moods in MOOCs: Analyzing emotions in the content of online courses with edX-CAS. IEEE Global Engineering Education Conference (EDUCON; pp. 1467–1474). https://doi.org//10.1109/EDUCON.2019.8725107ieeexplore.ieee.org/document/8725107
Shatnawi S., Gaber M. M., & Cocea M. (2014) Automatic content related feedback for MOOCs based on course domain ontology. In E. Corchado, J. A. Lozano, H. Quintián, & H. Yin (Eds.) Intelligent data engineering and automated learning, IDEAL 2014 (Lecture Notes in Computer Science, Vol. 8669). Springer. https://doi.org/10.1007/978-3-319-10840-7_4
Shatnawi, S., Gaber, M., & Haig, E. (2014). Automatic content related feedback for MOOCs based on course domain ontology (Vol. 8669). International Conference on Intelligent Data Engineering and Automated Learning. https://www.researchgate.net/publication/265132990_Automatic_Content_Related_Feedback_for_MOOCs_Based_on_Course_Domain_Ontology
Sindhu, I., Daudpota, S., Badar, K., Bakhtyar, M., Baber, J., & Nurunnabi, M. (2019). Aspect based opinion mining on student’s feedback for faculty teaching performance evaluation. IEEE Access, 7, 108729–108741. https://doi.org/10.1109/ACCESS.2019.2928872
Su, Y. S., & Lai, C. F. (2021). Applying educational data mining to explore viewing behaviors and performance with flipped classrooms on the social media platform Facebook. Frontiers in Psychology, 12, 653018. https://doi.org/10.3389/fpsyg.2021.653018
Su, Y. S., & Wu, S. Y. (2021). Applying data mining techniques to explore users behaviors and viewing video patterns in converged it environments. Journal of Ambient Intelligence and Humanized Computing. https://doi.org/10.1007/s12652-020-02712-6
Su, Y. S., Ding, T. J., & Chen, M. Y. (2021). Deep learning methods in internet of medical things for valvular heart disease screening system. IEEE Internet of Things Journal. https://ieeexplore.ieee.org/document/9330761
Swinnerton, B., Morris, N., Hotchkiss, S., & Pickering, J. (2016). The integration of an anatomy massive open online course (MOOC) into a medical anatomy curriculum: Anatomy MOOC. Anatomical Sciences Education, 10(1), 53–-67. https://doi.org/10.1002/ase.1625https://pubmed.ncbi.nlm.nih.gov/27315160/
Tekin, C., Braun, J., & van der Schaar, M. v. d. (2015). eTutor: Online learning for personalized education [Paper presentation]. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brisbane, Australia. https://arxiv.org/abs/1410.3617
Veletsianos, G. (2013). Learner experiences with moocs and open online learning. Hybrid Pedagogy Publica. http://learnerexperiences.hybridpedagogy.com
Wen, M., Yang, D., & Rose, C. (2014). Sentiment analysis in MOOC discussion forums: What does it tell us? In J. Stamper, Z. Pardos, M. Mavrikis, & B. M. McLaren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014; pp. 130–137). https://www.dropbox.com/s/crr6y6fx31f36e0/EDM%202014%20Full%20Proceedings.pdf
Xing, W., & Du, D. (2019). Dropout prediction in MOOCs: Using deep learning for personalized intervention. Journal of Educational Computing Research, 57(3), 547-570. https://doi.org/10.1177/0735633118757015
Yang, S. J. H., Lu, O. H. T., Huang, A. Y. Q., Huang, J. C. H., Ogata, H., & Lin, A. J. Q. (2018). Predicting students’ academic performance using multiple linear regression and principal component analysis. Journal of Information Processing, 26(2), 170-176. https://doi.org/10.2197/ipsjjip.26.170
Yang, T. Y., Brinton, C. G., Joe-Wong, C., & Chiang, M. (2017). Behavior-based grade prediction for MOOCs via time series neural networks. IEEE Journal of Selected Topics in Signal Processing, 11(5), 716-728. https://doi.org/10.1109/JSTSP.2017.2700227
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International Licence. The copyright of all content published in IRRODL is retained by the authors.
This copyright agreement and use license ensures, among other things, that an article will be as widely distributed as possible and that the article can be included in any scientific and/or scholarly archive.
You are free to
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms below:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
