Scrutinizing Learning Management Systems in Practice: An Applied Time Series Research in Higher Education

Authors

DOI:

https://doi.org/10.19173/irrodl.v24i2.6905

Keywords:

learning management systems, e-learning, online exams, transactional distance theory

Abstract

This study examined the use of Advancity Learning Management Systems (ALMS) and the Moodle Learning Management Systems (LMS) in learning settings, as well as online exams, within the framework of Transactional Distance Theory. With 146 college students (nfemale = 102, nmale = 44) as voluntary participants, data was gathered through an online questionnaire. A time series design was used for two different LMS sessions, and participants who voluntarily participated in ALMS and Moodle LMS sessions were matched. The findings revealed that while Moodle and ALMS both receive relatively similar assessment ratings for online exams, Moodle scored better in terms of learning setting. When factors of the Learning Management Systems Evaluation Scale (LMSES) based on Transactional Distance Theory were compared, the dialogue and autonomy factors were significantly higher for Moodle LMS than for ALMS. When online exams in the LMS were compared, there was no significant difference between ALMS and Moodle LMS, and for both LMS, the reliability factor was a determinant indicator than the other factors. As a result, in assessing and using an LMS, choices should be based on how well the LMS characteristics address an institution’s demands.

Author Biography

Esra Barut Tuğtekin, Inonu University, Distance Education Application and Research Center, Malatya, Türkiye

Esra BARUT TUĞTEKİN is an Associate Professor, since 2022, in the Distance Education Application and Research Center at Inonu University, Türkiye. She has a M.A. in Computer Education and Instructional Technology at Suleyman Demirel University (2015) and has a PhD in Computer Education and Instructional Technology at Anadolu University (2020). Her studies and research interests are on instructional technology, multivariate statistics, media literacy, social networking, distance education, online learning, social media in education, multimedia learning, and virtual identity.

References

Abuhassna, H., Busalim, A. H., Mamman, B., Yahaya, N., Zakaria, M., Al-Maatouk, Q., & Awae, F. (2022). From student’s experience: Does e-learning course structure influenced by learner’s prior experience, background knowledge, autonomy, and dialogue. Contemporary Educational Technology, 14(1), ep338. https://doi.org/10.30935/cedtech/11386

Akar, E., & Mardikyan, S. (2014). Analyzing factors affecting users’ behavior intention to use social media: Twitter case. International Journal of Business and Social Science, 5(11), 85–95. http://ijbssnet.com/view.php?u=https://www.ijbssnet.com/journals/Vol_5_No_11_1_October_2014/9.pdf

Al-Fraihat, D., Joy, M., Masa’deh, R., & Sinclair, J. (2020). Evaluating e-learning systems success: An empirical study. Computers in Human Behavior, 102(1), 67–86. https://doi.org/10.1016/j.chb.2019.08.004

Alshurideh, M. T., Al Kurdi, B., AlHamad, A. Q., Salloum, S. A., Alkurdi, S., Dehghan, A., Abuhashesh, M., & Masa’deh, R. E. (2021). Factors affecting the use of smart mobile examination platforms by universities’ postgraduate students during the COVID-19 pandemic: An empirical study. Informatics, 8(2), 32. https://doi.org/10.3390/informatics8020032

Arnò, S., Galassi, A., Tommasi, M., Saggino, A., & Vittorini, P. (2021). State-of-the-art of commercial proctoring systems and their use in academic online exams. International Journal of Distance Education Technologies, 19(2), 55–76. https://doi.org/10.4018/IJDET.20210401.oa3

Arora, S., Chaudhary, P., & Singh, R. K. (2021). Impact of coronavirus and online exam anxiety on self-efficacy: The moderating role of coping strategy. Interactive Technology and Smart Education, 18(3), 475–492. https://doi.org/10.1108/ITSE-08-2020-0158

Barut Tuğtekin, E. (2021). Development of the learning management systems evaluation scale based on transactional distance theory. Journal of Educational Technology and Online Learning, 4(3), 503-515. https://doi.org/10.31681/jetol.943335

Baxto da Silva, W., Amaro, R., & Mattar, J. (2019). Distance education and the Open University of Brazil: History, structure, and challenges. International Review of Research in Open and Distributed Learning, 20(4), 99–115. https://doi.org/10.19173/irrodl.v20i4.4132

Bradley, V. M. (2021). Learning management system (LMS) use with online instruction. International Journal of Technology in Education, 4(1), 68–92. https://doi.org/10.46328/ijte.36

Cabi, E., & Ersoy, H. (2021). Technologies used in distance education during the COVID-19 global pandemic and investigation of the opinions of teachers: The case of Baskent University. Journal of Higher Education and Science, 12(1), 168–179.

Castañeda, L., & Selwyn, N. (2018). More than tools? Making sense of the ongoing digitizations of higher education. International Journal of Educational Technology in Higher Education, 15(1), 1–10. https://doi.org/10.1186/s41239-018-0109-y

Cohen, J. (1988). Statistical power analysis for the behavioural sciences. Earlbaum. https://doi.org/10.4324/9780203771587

Cooley, W. W., & Lohnes, P. R. (1971). Multivariate data analysis. Wiley. https://doi.org/10.1002/bimj.19730150413

Dadashzadeh, M. (2021). The online examination dilemma: To proctor or not to proctor? Journal of Instructional Pedagogies, 25, 1–11. https://eric.ed.gov/?id=EJ1294386

Dermo, J. (2009). e-Assessment and the student learning experience: A survey of student perceptions of e-assessment. British Journal of Educational Technology, 40(2), 203–214. https://doi.org/10.1111/j.1467-8535.2008.00915.x

Dias, S. B., & Diniz, J. A. (2014). Towards an enhanced learning in higher education incorporating distinct learner’s profiles. Educational Technology & Society, 17(1), 307–319. https://www.jstor.org/stable/10.2307/jeductechsoci.17.1.307

Elfeky, A. I. M., Masadeh, T. S. Y., & Elbyaly, M. Y. H. (2020). Advance organizers in flipped classroom via e-learning management system and the promotion of integrated science process skills. Thinking Skills and Creativity, 35, 100622. https://doi.org/10.1016/j.tsc.2019.100622

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2013). Multivariate data analysis. Pearson.

Hollister, K. K., & Berenson, M. L. (2009). Proctored versus unproctored online exams: Studying the impact of exam environment on student performance. Decision Sciences Journal of Innovative Education, 7(1), 271–294. https://doi.org/10.1111/j.1540-4609.2008.00220.x

Horzum, M. B. (2011). Developing transactional distance scale and examining transactional distance perception of blended learning students in terms of different variables. Educational Sciences: Theory & Practice, 11(3), 1571–1587. https://eric.ed.gov/?id=EJ936610

Huang, R. H., Liu, D. J., Tlili, A., Yang, J. F., & Wang, H. H. et al. (2020). Handbook on facilitating flexible learning during educational disruption: The Chinese experience in maintaining undisrupted learning in COVID-19 outbreak. Smart Learning Institute of Beijing Normal University. https://iite.unesco.org/wp-content/uploads/2020/03/Handbook-on-Facilitating-Flexible-Learning-in-COVID-19-Outbreak-SLIBNU-V1.2-20200315.pdf

Ilgaz, H., & Adanir, G. A. (2020). Providing online exams for online learners: Does it really matter for them? Education and Information Technologies, 25(2), 1255–1269. https://doi.org/10.1007/s10639-019-10020-6

Jaap, A., Dewar, A., Duncan, C., Fairhurst, K., Hope, D., & Kluth, D. (2021). Effect of remote online exam delivery on student experience and performance in applied knowledge tests. BMC Medical Education, 21(1), 1–7. https://doi.org/10.1186/s12909-021-02521-1

Jia, J., & He, Y. (2021). The design, implementation and pilot application of an intelligent online proctoring system for online exams. Interactive Technology and Smart Education, 19(1), 112–120. https://doi.org/10.1108/ITSE-12-2020-0246

Jorczak, R. (2014). Differences in classroom versus online exam performance due to asynchronous discussion. Online Learning Journal, 18(2), 1–9. https://www.learntechlib.org/p/183751/

Jung, S., & Huh, J. H. (2019). An efficient LMS platform and its test bed. Electronics, 8(2), 154. https://doi.org/10.3390/electronics8020154

Karadag, E., Ciftci, S. K., Gok, R., Su, A., Ergin-Kocaturk, H., & Ciftci, Ş. S. (2021). Distance education capacities of universities during the COVID-19 pandemic process. Journal of University Research, 4(1), 8–22.

Kasim, N. N. M., & Khalid, F. (2016). Choosing the right learning management system (LMS) for the higher education institution context: A systematic review. International Journal of Emerging Technologies in Learning, 11(6), 55–61. http://dx.doi.org/10.3991/ijet.v11i06.5644

Kehrwald, B. A., & Parker, B. (2019). Implementing online learning: Stories from the field. Journal of University Teaching & Learning Practice, 16(1), 1. https://doi.org/10.53761/1.16.1.1

Kemp, N., & Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates’ opinions and test performance in classroom vs. online learning. Frontiers in Psychology, 5, 1278. https://doi.org/10.3389/fpsyg.2014.01278

Khalaf, K., El-Kishawi, M., Moufti, M. A., & Al Kawas S. (2020). Introducing a comprehensive high-stake online exam to final-year dental students during the COVID-19 pandemic and evaluation of its effectiveness. Medical Education Online, 25(1), 1826861. https://doi.org/10.1080/10872981.2020.1826861

Kwon, S., Kim, W., Bae, C., Cho, M., Lee, S., & Dreamson, N. (2021). The identity changes in online learning and teaching: Instructors, learners, and learning management systems. International Journal of Educational Technology in Higher Education, 18(1), 1–18. https://doi.org/10.1186/s41239-021-00304-8

Lazorak, O., Belkina, O., & Yaroslavova, E. (2021). Changes in student autonomy via e-learning courses. International Journal of Emerging Technologies in Learning, 16(17), 209–225. https://www.learntechlib.org/p/220066/

Liu, Y., Han, S., & Li, H. (2010). Understanding the factors driving m-learning adoption: A literature review. Campus-Wide Information Systems, 27, 210–226. https://doi.org/10.1108/10650741011073761

Markets & Markets. (2022). Learning Management System (LMS) market by component (solutions and services), delivery mode (distance learning, instructor-led training, and blended learning), deployment, user type (academic and corporate), and region (2022–2026). https://www.marketsandmarkets.com/Market-Reports/learning-management-systems-market-1266.html?gclid=EAIaIQobChMItKT-oful3QIVjcqyCh1PcwdkEAAYASAAEgIfGvD_BwE

McLachlan, G. J. (1999). Mahalanobis distance. Resonance, 4(6), 20–26. https://www.ias.ac.in/article/fulltext/reso/004/06/0020-0026

Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), Theoretical principles of distance education (pp. 22–38). Routledge. http://www.c3l.uni-oldenburg.de/cde/found/moore93.pdf

Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view. Wadsworth Publishing.

Nasser, R., Cherif, M., & Romanowski, M. (2011). Factors that impact the usage of the learning management system in Qatari schools. The International Review of Research in Open and Distance Learning, 12(6), 39–62. https://doi.org/10.19173/irrodl.v12i6.985

Oakes, K. (2002). E-learning: LCMS, LMS-They’re not just acronyms but powerful systems for learning. Training & Development, 56(3), 73–75.

Raza, S. A., Qazi, W., Khan, K. A., & Salam, J. (2020). Social isolation and acceptance of the learning management system (LMS) in the time of COVID-19 pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 59(2), 1–26. https://doi.org/10.1177/0735633120960421

Research & Markets. (2022). Learning management system (LMS) global market report 2022: By component, delivery mode, deployment mode, end user, and covering. https://www.researchandmarkets.com/reports/5522323/learning-management-system-lms-global-market

Shraim, K. (2019). Online examination practices in higher education institutions: Learners’ perspectives. Turkish Online Journal of Distance Education, 20(4), 185–196. https://doi.org/10.17718/tojde.640588

Stowell, J. R., & Bennett, D. (2010). Effects of online testing on student exam performance and test anxiety. Journal of Educational Computing Research, 42(2), 161–171. https://doi.org/10.2190/EC.42.2.b

Turnbull, D., Chugh, R., & Luck, J. (2019). Learning management systems: An overview. In A. Tatnall (Ed.), Encyclopedia of education and information technologies (pp. 1-7). Springer Nature. https://doi.org/10.1007/978-3-030-10576-1_248

Turnbull, D., Chugh, R., & Luck, J. (2021). The use of case study design in learning management system research: A label of convenience? International Journal of Qualitative Methods, 20. https://doi.org/10.1177/16094069211004148

Watson, R., & Watson, S. (2007). An argument for clarity: What are learning management systems, what are they not, and what should they become? TechTrends, 51(2), 28–34. https://doi.org/10.1007/s11528-007-0023-y

Weber, J. M., & Lennon, R. (2007). Multi-course comparison of traditional versus Web-based course delivery systems. Journal of Educators Online, 4(2), 1–19. https://eric.ed.gov/?id=EJ907748

Woldeab, D., & Brothen, T. (2021). Video surveillance of online exam proctoring: Exam anxiety and student performance. International Journal of E-Learning & Distance Education, 36(1), 1–26. https://www.ijede.ca/index.php/jde/article/view/1204/1856

Wood, D., Kurtz-Costes, B., & Copping K. (2011). Gender differences in motivational pathways to college for middle class African-American youths. American Psychological Association, 47(4), 961–968. https://doi.org/10.1037/a0023745

Yilmaz, O. (2016). Online examination assessment survey. e-Kafkas Journal of Educational Research, 3(3), 26–33. https://dergipark.org.tr/en/pub/kafkasegt/issue/27919/296377

Yolsal, H., & Yorulmaz, O. (2022). The effect of COVID19 pandemic on the performance of higher education students. Kafkas University Economics and Administrative Sciences Faculty, 13(25), 441–472. https://doi.org/10.36543/kauiibfd.2022.019

Yoruk, T., Akar, N., & Erdogan, H. (2020). An analysis of the factors affecting use of learning management system in the framework of extended technology acceptance model with structural equation model. Eskisehir Osmangazi University Journal of Social Sciences, 21(2), 431–449. https://dergipark.org.tr/en/pub/ogusbd/issue/58568/808336

Published

2023-03-22

How to Cite

Barut Tuğtekin, E. (2023). Scrutinizing Learning Management Systems in Practice: An Applied Time Series Research in Higher Education. The International Review of Research in Open and Distributed Learning, 24(2), 53–71. https://doi.org/10.19173/irrodl.v24i2.6905

Issue

Section

Research Articles