A Systematic Review of Questionnaire-Based Quantitative Research on MOOCs

Authors

  • Mingxiao Lu Nankai University
  • Tianyi Cui Nankai University
  • Zhenyu Huang Central Michigan University
  • Hong Zhao Nankai University
  • Tao Li Nankai University
  • Kai Wang Nankai University

DOI:

https://doi.org/10.19173/irrodl.v22i2.5208

Keywords:

MOOC, factors-goals graph (F-G graph), questionnaire-based survey, quantitative analysis, research topics

Abstract

Massive open online courses (MOOCs) have attracted much interest from educational researchers and practitioners around the world. There has been an increase in empirical studies about MOOCs in recent years, most of which used questionnaire surveys and quantitative methods to collect and analyze data. This study explored the research topics and paradigms of questionnaire-based quantitative research on MOOCs by reviewing 126 articles available in the Science Citation Index (SCI) and Social Sciences Citation Index (SSCI) databases from January 2015 to August 2020. This comprehensive overview showed that: (a) the top three MOOC research topics were the factors influencing learners’ performance, dropout rates and continuance intention to use MOOCs, and assessing MOOCs; (b) for these three topics, many studies designed questionnaires by adding new factors or adjustments to extant theoretical models or survey instruments; and (c) most researchers used descriptive statistics to analyze data, followed by the structural equation model, and reliability and validity analysis. This study elaborated on the relationship of research topics and key factors in the research models by building factors-goals (F-G) graphs. Finally, we proposed some directions and recommendations for future research on MOOCs.

Author Biographies

Mingxiao Lu, Nankai University

Mingxiao Lu is an experimentalist of College of Computer Science, Nankai University in China. She has more than 3 years of experience teaching students from multiple countries. She has been involved in the creation and maintenance of three MOOCs. She has published nine articles in International academic conferences. Her research interests include Computer Education, MOOC Research, and Data Analysis. E-mail: lumx@nankai.edu.cn

Tianyi Cui, Nankai University

Tianyi Cui is an undergraduate at Nankai University. Her research interests include financial management, securities markets and investing, and data modeling. E-mail: nku_cuitianyi@126.com

Zhenyu Huang, Central Michigan University

Zhenyu Huang is a professor of Information Systems at Central Michigan University in the College of Business Administration. His research interests include Business Intelligence, E-government, Knowledge Management, ERP, System Usability, Software Design, and Gaming for Learning. His research results have appeared in MIS journals such as the European Journal of Information Systems, Information and Management, Journal of Computer Information Systems, and Information Systems Management, and MIS conference proceedings. E-mail: Huang1z@cmich.edu

Hong Zhao, Nankai University

Hong Zhao is an associate professor of College of Computer Science, Nankai University in China. Her research interests include Big Data, Service Learning, Instructional Design, and MOOC. E-mail: zhaoh@nankai.edu.cn

Tao Li, Nankai University

Tao Li is a professor of College of Computer Science, Nankai University in China. His research interests include Heterogeneous Computing, Intelligent Internet of Things, Blockchain, and Instructional Design. E-mail: litao@nankai.edu.cn

Kai Wang, Nankai University

Kai Wang is an associate professor at Nankai University. His research interests include Computer Vision, Artificial Intelligence, Software Engineering, Data Modelling, Parallel Computing, Business Intelligence, Instructional Design, MOOC, and Case-based Learning. E-mail: wangk@nankai.edu.cn       

References

Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. In J. Kuhl & J. Beckmann (Eds.), Action control: From cognition to behavior (pp. 11–39). Springer.

Albert, S., Mercedes, G. S., & Terry, A. (2015). Meta-Analysis of the research about MOOC during 2013-2014. Educacion Xx1, 18(2). https://doi.org/10.5944/educxx1.14808

Ang, K. L. M., Ge, F. L., & Seng, K. P. (2020). Big educational data & analytics: Survey, architecture and challenges. IEEE Access, 8, 116392–116414. https://doi.org/10.1109/ACCESS.2020.2994561

Babori, A., Abdelkarim, Z., & Fassi, H. F. (2019). Research on MOOCs in major referred journals: The role and place of content. International Review of Research in Open and Distributed Learning, 20(3). https://doi.org/10.19173/irrodl.v20i4.4385

Blum-Smith, S., Yurkofsky, M. M., & Brennan, K. (2021). Stepping back and stepping in: Facilitating learner-centered experiences in MOOCs. Computers & Education, 160, 104042. https://doi.org/10.1016/j.compedu.2020.104042

Botero, G. G., Questier, F., Cincinnato, S., He, T., & Zhu, C. (2018). Acceptance and usage of mobile assisted language learning by higher education students. Journal of Computing in Higher Education, 5, 1–26. https://doi.org/10.1007/s12528-018-9177-1

Bozkurt, A., Akgun-Ozbek, E., & Zawacki-Richter, O. (2017). Trends and patterns in massive open online courses: Review and content analysis of research on MOOCs (2008–2015). International Review of Research in Open and Distributed Learning, 18(5), 118–147. http://doi.org/10.19173/irrodl.v18i5.3080

Cagiltay, N. E., Cagiltay, K., & Celik, B. (2020). An analysis of course characteristics, learner characteristics, and certification rates in MITx MOOCs. International Review of Research in Open and Distributed Learning, 21(3), 121–139. http://doi.org/10.19173/irrodl.v21i3.4698

Carlos, A. H., Iria, E. A., Mar, P. S., Carlos, D. K., & Carmen, F. P. (2017). Understanding learners’ motivation and learning strategies in MOOCs. International Review of Research in Open and Distributed Learning, 18(3), 119–137. http://doi.org/10.19173/irrodl.v18i3.2996

Castano-Munoz, J., Kreijns, K., Kalz, M., & Punie, Y. (2017). Does digital competence and occupational setting influence MOOC participation? Evidence from a cross-course survey. Journal of Computing in Higher Education, 29(1), 28–46. http://doi.org/10.1007/s12528-016-9123-z

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-339. http://doi.org/10.2307/249008

de Barba, P. G., Kennedy, G. E., & Ainley, M. D. (2016). The role of students’ motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3), 218–231. http://doi.org/10.1111/jcal.12130

Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125, 627–626. http://doi.org/10.1037/0033-2909.125.6.627

Ding, Y., & Zhao, T. (2020). Emotions, engagement, and self-perceived achievement in a small private online course. Journal of Computer Assisted Learning, 36(4), 449–457. https://doi.org/10.1111/jcal.12410

Donitsa-Schmidt, S., & Topaz, B. (2018). Massive open online courses as a knowledge base for teachers. Journal of Education for Teaching, 44(5), 608–620. https://doi.org/10.1080/02607476.2018.1516350

Durksen, T. L., Chu, M. W., Ahmad, Z. F., Radil, A. I., & Daniels, L. M. (2016). Motivation in a MOOC: A probabilistic analysis of online learners’ basic psychological needs. Social Psychology of Education, 19(2), 241–260. https://doi.org/10.1007/s11218-015-9331-9

Fang, J. W., Hwang, G. J., & Chang, C. Y. (2019). Advancement and the foci of investigation of MOOCs and open online courses for language learning: A review of journal publications from 2009 to 2018. Interactive Learning Environments, 28, 1–19. http://doi.org/10.1080/10494820.2019.1703011

Farhan, W., Razmak, J., Demers, S., & Laflamme, S. (2019). E-learning systems versus instructional communication tools: Developing and testing a new e-learning user interface from the perspectives of teachers and students. Technology in Society, 59, 101192. http://doi.org/10.1016/j.techsoc.2019.101192

Fryer, L. K., & Bovee, H. N. (2018). Staying motivated to e-learn: Person- and variable-centred perspectives on the longitudinal risks and support. Computers & Education, 120, 227–240. http://doi.org/10.1016/j.compedu.2018.01.006

Gallagher, S. E., & Savage, T. (2016). Comparing learner community behavior in multiple presentations of a massive open online course. Journal of Computing in Higher Education, 28(3), 358–369. http://doi.org/10.1007/s12528-016-9124-y

Gan, T. (2018). Construction of security system of flipped classroom based on MOOC in teaching quality control. Educational Sciences: Theory & Practice, 18(6), 2707–2717. http://doi.org/10.12738/estp.2018.6.170

Geng, S., Niu, B., Feng, Y. Y., & Huang, M. J. (2020). Understanding the focal points and sentiment of learners in MOOC reviews: A machine learning and SC-LIWC-based approach. British Journal of Educational Technology, 51(5), 1785–1803. http://doi.org/10.1111/bjet.12999

Goodhue, D. L., Klein, B. D., & March, S. T. (2000). User evaluations of IS as surrogates for objective performance. Information and Management, 38(2), 87–101. http://doi.org/10.1016/S0378-7206(00)00057-4

Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98, 157–168. http://doi.org/10.1016/j.compedu.2016.03.016

Hung, C. Y., Sun, J. C. Y., & Liu, J. Y. (2019). Effects of flipped classrooms integrated with MOOCs and game-based learning on the learning motivation and outcomes of students from different backgrounds. Interactive Learning Environments, 27(8), 1028–1046. http://doi.org/10.1080/10494820.2018.1481103

Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self-regulated online learning questionnaire. Journal of Computing in Higher Education, 29(1), 6–27. http://doi.org/10.1007/s12528-016-9125-x

Jo, D. H. (2018). Exploring the determinants of MOOCs continuance intention. Ksii Transactions on Internet and Information Systems, 12(8), 3992–4005. http://doi.org/10.3837/tiis.2018.08.024

Joo, Y. J., So, H. J., & Kim, N. H. (2018). Examination of relationships among students’ self-determination, technology acceptance, satisfaction, and continuance intention to use k-MOOCs. Computers & Education, 122, 260–272. http://doi.org/10.1016/j.compedu.2018.01.003

Jung, E., Kim, D., Yoon, M., Park, S. H., & Oakley, B. (2019). The influence of instructional design on learner control, sense of achievement, and perceived effectiveness in a supersize MOOC course. Computers & Education, 128, 377–388. https://doi.org/10.1016/j.compedu.2018.10.001

Jung, Y., & Lee, J. (2018). Learning engagement and persistence in massive open online courses (MOOCs). Computers & Education, 122, 9–22. https://doi.org/10.1016/j.compedu.2018.02.013

Kahan, T., Soffer, T., & Nachmias, R. (2017). Types of participant behavior in a massive open online course. International Review of Research in Open and Distributed Learning, 18(6). https://doi.org/10.19173/irrodl.v18i6.3087

Khan, I. U., Hameed, Z., Yu, Y., Islam, T., & Khan, S. U. (2018). Predicting the acceptance of MOOCs in a developing country: Application of task-technology fit model, social motivation, and self-determination theory. Telematics and Informatics, 35(4), 964–978. https://doi.org/10.1016/j.tele.2017.09.009

Kizilcec, R. F., Perez-Sanagustin, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in massive open online courses. Computers & Education, 104, 18–33. https://doi.org/10.1016/j.compedu.2016.10.001

Kormos, J., & Nijakowska, J. (2017). Inclusive practices in teaching students with dyslexia: Second language teachers’ concerns, attitudes and self-efficacy beliefs on a massive open online learning course. Teaching & Teacher Education, 68, 30–41. https://doi.org/10.1016/j.tate.2017.08.005

Kovanovic, V., Joksimovic, S., Poquet, O., Hennis, T., & Gasevic, D. (2017). Exploring communities of inquiry in massive open online courses. Computers & Education, 119, 44–58. https://doi.org/10.1016/j.compedu.2017.11.010

Lee, D., Watson, S. L., & Watson, W. R. (2020). The relationships between self-efficacy, task value, and self-regulated learning strategies in massive open online courses. International Review of Research in Open and Distributed Learning, 21(1), 23–39. https://doi.org/10.19173/irrodl.v20i5.4389

Li, B., Wang, X., & Tan, S. C. (2018). What makes MOOC users persist in completing MOOCs? A perspective from network externalities and human factors. Computers in Human Behavior, 85, 385–395. https://doi.org/10.1016/j.chb.2018.04.028

Lowenthal, P., Snelson, C., & Perkins, R. (2018). Teaching massive, open, online, courses (MOOCs): Tales from the front line. International Review of Research in Open and Distributed Learning, 19(3), 1–19. https://doi.org/10.19173/irrodl.v19i3.3505

Luik, P., Suviste, R., Lepp, M., Palts, T., Tonisson, E., Sade, M., & Papli, K. (2019). What motivates enrolment in programming MOOCs? British Journal of Educational Technology, 50(1), 153–165. https://doi.org/10.1111/bjet.12600

Maldonado-Mahauad, J., Perez-Sanagustin, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from big data: Identifying self-regulated learning strategies in massive open online courses. Computers in Human Behavior, 80, 179–196. https://doi.org/10.1016/j.chb.2017.11.011

Marta-Lazo, C., Frau-Meigs, D., & Osuna-Acedo, S. (2019). A collaborative digital pedagogy experience in the tMOOC “step by step.” Australasian Journal of Educational Technology, 35(5), 111–127. https://doi.org/10.14742/ajet.4215

Martin Nunez, J. L., Tovar Caro, E., & Hilera Gonzalez, J. R. (2017). From higher education to open education: Challenges in the transformation of an online traditional course. IEEE Transactions on Education, 60(2), 134–142. https://doi.org/10.1109/TE.2016.2607693

Martinez-Lopez, R., Yot, C., Tuovila, I., & Perera-Rodriguez, V. H. (2017). Online self-regulated learning questionnaire in a Russian MOOC. Computers in Human Behavior, 75, 966–974. https://doi.org/10.1016/j.chb.2017.06.015

Meinert, E., Alturkistani, A., Brindley, D., Carter, A., Wells, G., & Car, J. (2018). Protocol for a mixed-methods evaluation of a massive open online course on real world evidence. BMJ Open, 8(8), e025188. https://doi.org/10.1136/bmjopen-2018-025188

Montes-Rodriguez, R., Martinez-Rodriguez, J. B., & Ocana-Fernandez, A. (2019). Case study as a research method for analyzing MOOCs: Presence and characteristics of those case studies in the main scientific databases. International Review of Research in Open and Distributed Learning, 20(3), 59–79. https://doi.org/10.19173/irrodl.v20i4.4299

Peral, J., Maté, A., & Marco, M. (2017). Application of data mining techniques to identify relevant key performance indicators. Computer Standards & Interfaces, 50, 55–64. https://doi.org/10.1016/j.csi.2016.09.009

Pintrich, P. R., Smith, D. A. F., Duncan, T., & Mckeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). University of Michigan.

Rasheed, R. A., Kamsin, A., Abdullah, N. A., Zakari, A., & Haruna, K. (2019). A systematic mapping study of the empirical MOOC literature. IEEE Access, 7, 124809–124827. https://doi.org/10.1109/ACCESS.2019.2938561

Robinson, R. (2016). Delivering a medical school elective with massive open online course (MOOC) technology. Peerj, 4(1), e2343. https://doi.org/10.7717/peerj.2343

Ruiz-Palmero, J., Lopez-Alvarez, D., Sanchez-Rivas, E., & Sanchez-Rodriguez, J. (2019). An analysis of the profiles and the opinion of students enrolled on xMOOCs at the University of Malaga. Sustainability, 11(24). https://doi.org/10.3390/su11246910

Sablina, S., Kapliy, N., Trusevich, A., & Kostikova, S. (2018). How MOOC-takers estimate learning success: Retrospective reflection of perceived benefits. International Review of Research in Open and Distributed Learning, 19(5). https://doi.org/10.19173/irrodl.v19i5.3768

Sanchez-Gordon, S., & Lujan-Mora, S. (2017). Research challenges in accessible MOOCs: A systematic literature review 2008–2016. Universal Access in the Information Society, 17(4), 775–789. http://doi.org/10.1007/s10209-017-0531-2

Sari, A. R., Bonk, C. J., & Zhu, M. (2020). MOOC instructor designs and challenges: What can be learned from existing MOOCs in Indonesia and Malaysia? Asia Pacific Education Review, 21(1), 143–166. https://doi.org/10.1007/s12564-019-09618-9

Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19(4), 460–475. https://doi.org/10.1006/ceps.1994.1033

Seddon, P. (1997). A respecification and extension of the Delone and McLean model of IS success. Information Systems Research, 8, 240–253. https://doi.org/10.1287/isre.8.3.240

Shahzad, F., Xiu, G., Khan, I., Shahbaz, M., & Abbas, A. (2020). The moderating role of intrinsic motivation in cloud computing adoption in online education in a developing country: A structural equation model. Asia Pacific Education Review, 21(1), 121–141. https://doi.org/10.1007/s12564-019-09611-2

Shao, Z. (2018). Examining the impact mechanism of social psychological motivations on individuals’ continuance intention of MOOCs. Internet Research, 28(1), 232–250. https://doi.org/10.1108/IntR-11-2016-0335

Shirky, C. (2013, July 8). MOOCs and economic reality. Chronicle of Higher Education, 59(42), B2.

Sneddon J., Barlow G., Bradley S., Brink A., Chandy S. J., & Nathwani D. (2018). Development and impact of a massive open online course (MOOC) for antimicrobial stewardship. Journal of Antimicrobial Chemotherapy, 73(4), 1091–1097. https://doi.org/10.1093/jac/dkx493

Soffer, T., & Cohen, A. (2015). Implementation of Tel Aviv University MOOCs in academic curriculum: A pilot study. International Review of Research in Open and Distributed Learning, 16(1), 80–97. https://doi.org/10.19173/irrodl.v16i1.2031

Sun, Y., Ni, L., Zhao, Y., Shen, X.-L., & Wang, N. (2019). Understanding students’ engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 50(6), 3156–3174. https://doi.org/10.1111/bjet.12724

Tao, D., Fu, P., Wang, Y., Zhang, T., & Qu, X. (2019). Key characteristics in designing massive open online courses (MOOCs) for user acceptance: An application of the extended technology acceptance model. Interactive Learning Environments, 44, 1–14. https://doi.org/10.1080/10494820.2019.1695214

Teo, T., & Dai, H. M. (2019). The role of time in the acceptance of MOOCs among Chinese university students. Interactive Learning Environments, 1–14. https://doi.org/10.1080/10494820.2019.1674889

Teresa Garcia-Alvarez, M., Novo-Corti, I., & Varela-Candamio, L. (2018). The effects of social networks on the assessment of virtual learning environments: A study for social sciences degrees. Telematics and Informatics, 35(4), 1005–1017. https://doi.org/10.1016/j.tele.2017.09.013

Tsai, Y. H., Lin, C. H., Hong, J. C., & Tai, K. H. (2018). The effects of metacognition on online learning interest and continuance to learn with MOOCs. Computers & Education, 121, 18–29. https://doi.org/10.1016/j.compedu.2018.02.011

Veletsianos, G., & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013–2015. International Review of Research in Open and Distributed Learning, 17(2). https://doi.org/10.19173/irrodl.v17i2.2448

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540

Wang, Y., & Baker, R. (2018). Grit and intention: Why do learners complete MOOCs? International Review of Research in Open and Distributed Learning, 19(3), 20-42. https://doi.org/10.19173/irrodl.v19i3.3393

Warr, P., & Downing, J. (2000). Learning strategies, learning anxiety and knowledge acquisition. British Journal of Psychology, 91(3), 311–333. https://doi.org/10.1348/000712600161853

Watson, S. L., Watson, W. R., & Tay, L. (2018). The development and validation of the attitudinal learning inventory (ALI): A measure of attitudinal learning and instruction. Educational Technology Research and Development, 66(6), 1601–1617. https://doi.org/10.1007/s11423-018-9625-7

Watson, W. R., Kim, W., & Watson, S. L. (2016). Learning outcomes of a MOOC designed for attitudinal change: A case study of an animal behavior and welfare MOOC. Computers & Education, 96, 83–93. https://doi.org/10.1016/j.compedu.2016.01.013

Watted, A., & Barak, M. (2018). Motivating factors of MOOC completers: Comparing between university-affiliated students and general participants. The Internet and Higher Education, 37, 11–20. https://doi.org/10.1016/j.iheduc.2017.12.001

Wu, B., & Chen, X. (2017). Continuance intention to use MOOCs: Integrating the technology acceptance model (TAM) and task technology fit (TTF) model. Computers in Human Behavior, 67, 221–232. https://doi.org/10.1016/j.chb.2016.10.028

Yang, H. H., & Su, C. H. (2017). Learner behaviour in a MOOC practice-oriented course: An empirical study integrating TAM and TPB. International Review of Research in Open and Distributed Learning, 18(5), 35–63. https://doi.org/10.19173/irrodl.v18i5.2991

Yang, M., Shao, Z., Liu, Q., & Liu, C. (2017). Understanding the quality factors that influence the continuance intention of students toward participation in MOOCs. Educational Technology Research and Development, 65(5), 1195–1214. https://doi.org/10.1007/s11423-017-9513-6

Zhang, J. (2016). Can MOOCs be interesting to students? An experimental investigation from regulatory focus perspective. Computers & Education, 95, 340–351. https://doi.org/10.1016/j.compedu.2016.02.003

Zhang, M., Yin, S., Luo, M., & Yan, W. (2016). Learner control, user characteristics, platform difference, and their role in adoption intention for MOOC learning in China. Australasian Journal of Educational Technology, 33(1), 114–133. https://doi.org/10.14742/ajet.2722

Zhao, Y., Wang, A., & Sun, Y. (2020). Technological environment, virtual experience, and MOOC continuance: A stimulus-organism-response perspective. Computers & Education, 144, 103721. https://doi.org/10.1016/j.compedu.2019.103721

Zhou, J. (2017). Exploring the factors affecting learners’ continuance intention of MOOCs for online collaborative learning: An extended ECM perspective. Australasian Journal of Educational Technology, 33(5), 123–135. http://doi.org/10.14742/ajet.2914

Zhou, M. (2016). Chinese university students’ acceptance of MOOCs: A self-determination perspective. Computers & Education, 92–93, 194–203. https://doi.org/10.1016/j.compedu.2015.10.012

Zhu, M., Sari, A., & Lee, M. M. (2018). A systematic review of research methods and topics of the empirical MOOC literature (2014–2016). The Internet and Higher Education, 37. https://doi.org/10.1016/j.iheduc.2018.01.002

Downloads

Published

2021-01-22

How to Cite

Lu, M., Cui, T., Huang, Z., Zhao, H., Li, T., & Wang, K. (2021). A Systematic Review of Questionnaire-Based Quantitative Research on MOOCs. The International Review of Research in Open and Distributed Learning, 22(2), 285–313. https://doi.org/10.19173/irrodl.v22i2.5208

Issue

Vol. 22 No. 2 (2021)

Section

Literature Reviews

License

This work is licensed under a Creative Commons Attribution 4.0 International Licence. The copyright of all content published in IRRODL is retained by the authors.

This copyright agreement and use license ensures, among other things, that an article will be as widely distributed as possible and that the article can be included in any scientific and/or scholarly archive.

You are free to

  • Share — copy and redistribute the material in any medium or format
  • Adapt — remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms below:

  • No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.