Scrutinizing Learning Management Systems in Practice: An Applied Time Series Research in Higher Education
DOI:
https://doi.org/10.19173/irrodl.v24i2.6905Keywords:
learning management systems, e-learning, online exams, transactional distance theoryAbstract
This study examined the use of Advancity Learning Management Systems (ALMS) and the Moodle Learning Management Systems (LMS) in learning settings, as well as online exams, within the framework of Transactional Distance Theory. With 146 college students (nfemale = 102, nmale = 44) as voluntary participants, data was gathered through an online questionnaire. A time series design was used for two different LMS sessions, and participants who voluntarily participated in ALMS and Moodle LMS sessions were matched. The findings revealed that while Moodle and ALMS both receive relatively similar assessment ratings for online exams, Moodle scored better in terms of learning setting. When factors of the Learning Management Systems Evaluation Scale (LMSES) based on Transactional Distance Theory were compared, the dialogue and autonomy factors were significantly higher for Moodle LMS than for ALMS. When online exams in the LMS were compared, there was no significant difference between ALMS and Moodle LMS, and for both LMS, the reliability factor was a determinant indicator than the other factors. As a result, in assessing and using an LMS, choices should be based on how well the LMS characteristics address an institution’s demands.
References
Abuhassna, H., Busalim, A. H., Mamman, B., Yahaya, N., Zakaria, M., Al-Maatouk, Q., & Awae, F. (2022). From student’s experience: Does e-learning course structure influenced by learner’s prior experience, background knowledge, autonomy, and dialogue. Contemporary Educational Technology, 14(1), ep338. https://doi.org/10.30935/cedtech/11386
Akar, E., & Mardikyan, S. (2014). Analyzing factors affecting users’ behavior intention to use social media: Twitter case. International Journal of Business and Social Science, 5(11), 85–95. http://ijbssnet.com/view.php?u=https://www.ijbssnet.com/journals/Vol_5_No_11_1_October_2014/9.pdf
Al-Fraihat, D., Joy, M., Masa’deh, R., & Sinclair, J. (2020). Evaluating e-learning systems success: An empirical study. Computers in Human Behavior, 102(1), 67–86. https://doi.org/10.1016/j.chb.2019.08.004
Alshurideh, M. T., Al Kurdi, B., AlHamad, A. Q., Salloum, S. A., Alkurdi, S., Dehghan, A., Abuhashesh, M., & Masa’deh, R. E. (2021). Factors affecting the use of smart mobile examination platforms by universities’ postgraduate students during the COVID-19 pandemic: An empirical study. Informatics, 8(2), 32. https://doi.org/10.3390/informatics8020032
Arnò, S., Galassi, A., Tommasi, M., Saggino, A., & Vittorini, P. (2021). State-of-the-art of commercial proctoring systems and their use in academic online exams. International Journal of Distance Education Technologies, 19(2), 55–76. https://doi.org/10.4018/IJDET.20210401.oa3
Arora, S., Chaudhary, P., & Singh, R. K. (2021). Impact of coronavirus and online exam anxiety on self-efficacy: The moderating role of coping strategy. Interactive Technology and Smart Education, 18(3), 475–492. https://doi.org/10.1108/ITSE-08-2020-0158
Barut Tuğtekin, E. (2021). Development of the learning management systems evaluation scale based on transactional distance theory. Journal of Educational Technology and Online Learning, 4(3), 503-515. https://doi.org/10.31681/jetol.943335
Baxto da Silva, W., Amaro, R., & Mattar, J. (2019). Distance education and the Open University of Brazil: History, structure, and challenges. International Review of Research in Open and Distributed Learning, 20(4), 99–115. https://doi.org/10.19173/irrodl.v20i4.4132
Bradley, V. M. (2021). Learning management system (LMS) use with online instruction. International Journal of Technology in Education, 4(1), 68–92. https://doi.org/10.46328/ijte.36
Cabi, E., & Ersoy, H. (2021). Technologies used in distance education during the COVID-19 global pandemic and investigation of the opinions of teachers: The case of Baskent University. Journal of Higher Education and Science, 12(1), 168–179.
Castañeda, L., & Selwyn, N. (2018). More than tools? Making sense of the ongoing digitizations of higher education. International Journal of Educational Technology in Higher Education, 15(1), 1–10. https://doi.org/10.1186/s41239-018-0109-y
Cohen, J. (1988). Statistical power analysis for the behavioural sciences. Earlbaum. https://doi.org/10.4324/9780203771587
Cooley, W. W., & Lohnes, P. R. (1971). Multivariate data analysis. Wiley. https://doi.org/10.1002/bimj.19730150413
Dadashzadeh, M. (2021). The online examination dilemma: To proctor or not to proctor? Journal of Instructional Pedagogies, 25, 1–11. https://eric.ed.gov/?id=EJ1294386
Dermo, J. (2009). e-Assessment and the student learning experience: A survey of student perceptions of e-assessment. British Journal of Educational Technology, 40(2), 203–214. https://doi.org/10.1111/j.1467-8535.2008.00915.x
Dias, S. B., & Diniz, J. A. (2014). Towards an enhanced learning in higher education incorporating distinct learner’s profiles. Educational Technology & Society, 17(1), 307–319. https://www.jstor.org/stable/10.2307/jeductechsoci.17.1.307
Elfeky, A. I. M., Masadeh, T. S. Y., & Elbyaly, M. Y. H. (2020). Advance organizers in flipped classroom via e-learning management system and the promotion of integrated science process skills. Thinking Skills and Creativity, 35, 100622. https://doi.org/10.1016/j.tsc.2019.100622
Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2013). Multivariate data analysis. Pearson.
Hollister, K. K., & Berenson, M. L. (2009). Proctored versus unproctored online exams: Studying the impact of exam environment on student performance. Decision Sciences Journal of Innovative Education, 7(1), 271–294. https://doi.org/10.1111/j.1540-4609.2008.00220.x
Horzum, M. B. (2011). Developing transactional distance scale and examining transactional distance perception of blended learning students in terms of different variables. Educational Sciences: Theory & Practice, 11(3), 1571–1587. https://eric.ed.gov/?id=EJ936610
Huang, R. H., Liu, D. J., Tlili, A., Yang, J. F., & Wang, H. H. et al. (2020). Handbook on facilitating flexible learning during educational disruption: The Chinese experience in maintaining undisrupted learning in COVID-19 outbreak. Smart Learning Institute of Beijing Normal University. https://iite.unesco.org/wp-content/uploads/2020/03/Handbook-on-Facilitating-Flexible-Learning-in-COVID-19-Outbreak-SLIBNU-V1.2-20200315.pdf
Ilgaz, H., & Adanir, G. A. (2020). Providing online exams for online learners: Does it really matter for them? Education and Information Technologies, 25(2), 1255–1269. https://doi.org/10.1007/s10639-019-10020-6
Jaap, A., Dewar, A., Duncan, C., Fairhurst, K., Hope, D., & Kluth, D. (2021). Effect of remote online exam delivery on student experience and performance in applied knowledge tests. BMC Medical Education, 21(1), 1–7. https://doi.org/10.1186/s12909-021-02521-1
Jia, J., & He, Y. (2021). The design, implementation and pilot application of an intelligent online proctoring system for online exams. Interactive Technology and Smart Education, 19(1), 112–120. https://doi.org/10.1108/ITSE-12-2020-0246
Jorczak, R. (2014). Differences in classroom versus online exam performance due to asynchronous discussion. Online Learning Journal, 18(2), 1–9. https://www.learntechlib.org/p/183751/
Jung, S., & Huh, J. H. (2019). An efficient LMS platform and its test bed. Electronics, 8(2), 154. https://doi.org/10.3390/electronics8020154
Karadag, E., Ciftci, S. K., Gok, R., Su, A., Ergin-Kocaturk, H., & Ciftci, Ş. S. (2021). Distance education capacities of universities during the COVID-19 pandemic process. Journal of University Research, 4(1), 8–22.
Kasim, N. N. M., & Khalid, F. (2016). Choosing the right learning management system (LMS) for the higher education institution context: A systematic review. International Journal of Emerging Technologies in Learning, 11(6), 55–61. http://dx.doi.org/10.3991/ijet.v11i06.5644
Kehrwald, B. A., & Parker, B. (2019). Implementing online learning: Stories from the field. Journal of University Teaching & Learning Practice, 16(1), 1. https://doi.org/10.53761/1.16.1.1
Kemp, N., & Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates’ opinions and test performance in classroom vs. online learning. Frontiers in Psychology, 5, 1278. https://doi.org/10.3389/fpsyg.2014.01278
Khalaf, K., El-Kishawi, M., Moufti, M. A., & Al Kawas S. (2020). Introducing a comprehensive high-stake online exam to final-year dental students during the COVID-19 pandemic and evaluation of its effectiveness. Medical Education Online, 25(1), 1826861. https://doi.org/10.1080/10872981.2020.1826861
Kwon, S., Kim, W., Bae, C., Cho, M., Lee, S., & Dreamson, N. (2021). The identity changes in online learning and teaching: Instructors, learners, and learning management systems. International Journal of Educational Technology in Higher Education, 18(1), 1–18. https://doi.org/10.1186/s41239-021-00304-8
Lazorak, O., Belkina, O., & Yaroslavova, E. (2021). Changes in student autonomy via e-learning courses. International Journal of Emerging Technologies in Learning, 16(17), 209–225. https://www.learntechlib.org/p/220066/
Liu, Y., Han, S., & Li, H. (2010). Understanding the factors driving m-learning adoption: A literature review. Campus-Wide Information Systems, 27, 210–226. https://doi.org/10.1108/10650741011073761
Markets & Markets. (2022). Learning Management System (LMS) market by component (solutions and services), delivery mode (distance learning, instructor-led training, and blended learning), deployment, user type (academic and corporate), and region (2022–2026). https://www.marketsandmarkets.com/Market-Reports/learning-management-systems-market-1266.html?gclid=EAIaIQobChMItKT-oful3QIVjcqyCh1PcwdkEAAYASAAEgIfGvD_BwE
McLachlan, G. J. (1999). Mahalanobis distance. Resonance, 4(6), 20–26. https://www.ias.ac.in/article/fulltext/reso/004/06/0020-0026
Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), Theoretical principles of distance education (pp. 22–38). Routledge. http://www.c3l.uni-oldenburg.de/cde/found/moore93.pdf
Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view. Wadsworth Publishing.
Nasser, R., Cherif, M., & Romanowski, M. (2011). Factors that impact the usage of the learning management system in Qatari schools. The International Review of Research in Open and Distance Learning, 12(6), 39–62. https://doi.org/10.19173/irrodl.v12i6.985
Oakes, K. (2002). E-learning: LCMS, LMS-They’re not just acronyms but powerful systems for learning. Training & Development, 56(3), 73–75.
Raza, S. A., Qazi, W., Khan, K. A., & Salam, J. (2020). Social isolation and acceptance of the learning management system (LMS) in the time of COVID-19 pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 59(2), 1–26. https://doi.org/10.1177/0735633120960421
Research & Markets. (2022). Learning management system (LMS) global market report 2022: By component, delivery mode, deployment mode, end user, and covering. https://www.researchandmarkets.com/reports/5522323/learning-management-system-lms-global-market
Shraim, K. (2019). Online examination practices in higher education institutions: Learners’ perspectives. Turkish Online Journal of Distance Education, 20(4), 185–196. https://doi.org/10.17718/tojde.640588
Stowell, J. R., & Bennett, D. (2010). Effects of online testing on student exam performance and test anxiety. Journal of Educational Computing Research, 42(2), 161–171. https://doi.org/10.2190/EC.42.2.b
Turnbull, D., Chugh, R., & Luck, J. (2019). Learning management systems: An overview. In A. Tatnall (Ed.), Encyclopedia of education and information technologies (pp. 1-7). Springer Nature. https://doi.org/10.1007/978-3-030-10576-1_248
Turnbull, D., Chugh, R., & Luck, J. (2021). The use of case study design in learning management system research: A label of convenience? International Journal of Qualitative Methods, 20. https://doi.org/10.1177/16094069211004148
Watson, R., & Watson, S. (2007). An argument for clarity: What are learning management systems, what are they not, and what should they become? TechTrends, 51(2), 28–34. https://doi.org/10.1007/s11528-007-0023-y
Weber, J. M., & Lennon, R. (2007). Multi-course comparison of traditional versus Web-based course delivery systems. Journal of Educators Online, 4(2), 1–19. https://eric.ed.gov/?id=EJ907748
Woldeab, D., & Brothen, T. (2021). Video surveillance of online exam proctoring: Exam anxiety and student performance. International Journal of E-Learning & Distance Education, 36(1), 1–26. https://www.ijede.ca/index.php/jde/article/view/1204/1856
Wood, D., Kurtz-Costes, B., & Copping K. (2011). Gender differences in motivational pathways to college for middle class African-American youths. American Psychological Association, 47(4), 961–968. https://doi.org/10.1037/a0023745
Yilmaz, O. (2016). Online examination assessment survey. e-Kafkas Journal of Educational Research, 3(3), 26–33. https://dergipark.org.tr/en/pub/kafkasegt/issue/27919/296377
Yolsal, H., & Yorulmaz, O. (2022). The effect of COVID19 pandemic on the performance of higher education students. Kafkas University Economics and Administrative Sciences Faculty, 13(25), 441–472. https://doi.org/10.36543/kauiibfd.2022.019
Yoruk, T., Akar, N., & Erdogan, H. (2020). An analysis of the factors affecting use of learning management system in the framework of extended technology acceptance model with structural equation model. Eskisehir Osmangazi University Journal of Social Sciences, 21(2), 431–449. https://dergipark.org.tr/en/pub/ogusbd/issue/58568/808336
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International Licence. The copyright of all content published in IRRODL is retained by the authors.
This copyright agreement and use license ensures, among other things, that an article will be as widely distributed as possible and that the article can be included in any scientific and/or scholarly archive.
You are free to
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms below:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
