International Review of Research in Open and Distributed Learning Scrutinizing Learning Management Systems in Practice: An Applied Time Series Research in Higher Education

This study examined the use of Advancity Learning Management Systems (ALMS) and the Moodle Learning Management Systems (LMS) in learning settings, as well as online exams, within the framework of Transactional Distance Theory. With 146 college students (nfemale = 102, nmale = 44) as voluntary participants, data was gathered through an online questionnaire. A time series design was used for two different LMS sessions, and participants who voluntarily participated in ALMS and Moodle LMS sessions were matched. The findings revealed that while Moodle and ALMS both receive relatively similar assessment ratings for online exams


Introduction
Learning management systems (LMS) are used at most institutions throughout the world.Nearly half of university courses will likely be based on e-learning soon, while approximately 42% of Global Fortune 500 companies currently use educational technology tools like LMS to deliver in-service training to their staff (Research & Markets, 2022).Given the changes in learning methodologies and procedures in e-learning settings, there is a high demand for LMS, with the global market expected to reach $25.7 billion by 2025 (Markets & Markets, 2022).Considering that there are more than 1,000 LMS vendors in the e-learning market, choosing an appropriate LMS from the many available is very challenging.
Practical testing of different LMS and analyzing their outcomes will help identify the criteria necessary to support those selecting LMS.
Although LMS were first used primarily as supplemental learning tools, thanks to the incorporation of various structures, they have now evolved into a systematic learning environment.The term LMS now describes various software systems that provide learners, instructors, and administrators with synchronous or asynchronous educational services (Elfeky et al., 2020;Turnbull et al., 2019).LMS learning environments are most effective when they consistently provide users with a variety of activities (Jung & Huh, 2019).LMS assist learners by monitoring and recording the learning process, as well as performing various assessments while providing uploaded and requested information.
Additionally, they provide access to educational resources, promote tutoring, and monitor and store information on each learner's activities (Kehrwald & Parker, 2019).As a result, a variety of enhancements and constructivist arrangements may be produced on LMS in line with pedagogical objectives and educational goals, and depending on learners' problems and suggestions (Al-Fraihat et al., 2020).
The use of online learning environments for education and training has triggered and significantly enhanced the importance of LMS, particularly amid the COVID-19 pandemic (Huang et al., 2020;Kwon et al., 2021;Raza et al., 2020;Turnbull, 2021).Despite the rise in academic research on LMS, particularly amid the pandemic, most studies have focused on systematic literature reviews or assessing user attitudes.With little quantitative analysis of LMS use in the literature, empirical comparison is limited.Furthermore, institutions may find it difficult to select the LMS best suited to their institutional needs and goals from among the many available.Empirical comparisons of different LMS may provide essential data, guidance, and also serve as a reference for learners, instructors, and managers of institutions selecting and implementing suitable LMS.

Learning Management Systems and Conceptual Framework
LMS provide a highly inclusive environment for learning, including online collaborative learning groups, discussion activities, and frameworks that encourage learners to connect with content as well as other LMS stakeholders (Baxto da Silva et al., 2019;Dias & Diniz, 2014;Jung & Huh, 2019).Using the LMS is a crucial and key factor for learners' performance and academic achievement (Nasser et al., 2011).Learners are encouraged to be autonomous through the use of LMS in e-learning environments (Bradley, 2021;Nasser et al., 2011;Wood et al., 2011) and LMS can encourage learners' engagement since they allow users to monitor the learning process (Al-Fraihat et al., 2020).LMS serve as a multifaceted platform for distributing, sharing, supervising, and monitoring educational content (Watson & Watson, 2007).They also offer a range of options for learners to sign up for courses, monitor and assess their progress in those courses, and promote engagement (Al-Fraihat et al., 2020;Oakes, 2002).In e-learning environments, even though the learners and their instructors are physically separated, LMS make it possible to establish communication and overcome physical distance through Internet technology.Moore (1993), who concentrated on the concept of distance in distance learning, called attention to the social and psychological distance brought on by communication gaps.These types of distance might lead to misconceptions and impede the learning process.According to Moore's (1993) Transactional Distance Theory, the detrimental effects of distance may be reduced by influencing one another and developing recurring behavioral patterns (Moore & Kearsley, 1996).Transactional distance has been conceptualized as all kinds of distance that prevent individuals from interacting (Horzum, 2011) and consists of three factors, namely structure, dialogue, and autonomy (Moore, 1993).
Structure describes the combination of features that address learner needs during learner-content and learner-interface interaction, whereas the dialogue factor describes the two-way interactions labelled as learner-instructor and learner-learner.Learner autonomy addresses the issue of choosing learning strategies and how learners' tenets of their own experiences are about how the autonomy factor is managed by learners (Horzum, 2011).The constraints of structure may create an inflexible learning environment and frustrate learners' ability to learn.On the other hand, LMS with a well-developed dialogue factor increases the likelihood of achieving new learning outcomes.Furthermore, supporting the autonomy factor enables learners to freely guide their learning in the LMS.In brief, transactional distance theory recommends that when selecting an LMS, learning materials that improve learners' autonomy and discourse should be included, and the structure factor of the LMS should be regulated to provide a flexible learning experience.It is critical for institutions that will employ LMS to focus on their benefits by analyzing learners' performance throughout the course and the learning outcomes after the course is concluded.Evaluating, organizing, and improving LMS within the context of transactional distance theory will enhance learners' outcomes.In addition, tests-a key component of the learning process-are employed as online examinations in LMS, so it is crucial to consider the potential effects of online examinations on learners and assessment practices.Therefore, while assessing LMS, the course and test processes should be considered together, while the LMS-based online exam options should be evaluated independently.

Online Exams
To evaluate learners' education standing, tests in face-to-face classrooms are generally held synchronously, though with the options provided by distance education, exams can also be held online.
The primary distinction between a face-to-face classroom and an online exam is physical presence and synchrony (Jorczak, 2014).While learners take tests synchronously and face-to-face in a classroom setting, they can take online exams synchronously or asynchronously during the exam period designated on the LMS.While exam security for face-to-face tests can be ensured by a hall attendant, automated monitoring solutions are available for online exams if there is a requirement for an attendant (Arnò et al., 2021;Jia & He, 2021;Khalaf et al., 2020;Woldeab & Brothen, 2021).Even with controls using a camera, microphone, and Internet connection during online tests, it is very challenging to obtain the monitoring and evaluation effectiveness afforded by human surveillance.Therefore, an investigation of online exam dependability metrics is ongoing.Additionally, it has been reported that learners may experience varying degrees of exam anxiety due to computer-based exam activities (Jaap et al., 2021).Studies have indicated that students with significant face-to-face test anxiety had lower (Stowell & Bennett, 2010) or greater (Shraim, 2019) degrees of anxiety in online examinations, and there is a significant relationship between online test anxiety and test performance (Arora et al., 2021;Jaap et al., 2021;Stowell & Bennett, 2010).
Various studies on online tests have compared supervised and unsupervised exam results (Dadashzadeh, 2021;Hollister & Berenson, 2009), as well as face-to-face and online exam methods (Kemp & Grieve, 2014;Weber & Lennon, 2007).However, there have been only limited findings for different online exam environments without supervision.In this current study, both online test and exam activities created in different LMS systems were carried out unsupervised.More time was allotted for test participation than the exam's duration, and learners were permitted to take the exam online asynchronously within the time limitation.Evaluating online test apps across various LMS platforms will be useful step and a fruitful guide, as examinations are a crucial part of any learning setting.

Research Questions (RQs)
In the literature, there is a gap in both the practical and statistical examination of LMS.Thus, the purpose of the current research was to assess online exams as they have been used in these settings, and to compare Advancity Learning Management Systems (ALMS) and Moodle LMS within the context of transactional distance theory.Accordingly, the following RQs were developed: • What are the descriptive statistics of LMS use and online exam processes for ALMS and Moodle LMS from the students' perspectives?
• Is there a statistically significant difference in students' perspectives on the use of ALMS and Moodle LMS?
• Is there a statistically significant difference between students' evaluations of online exam processes in ALMS and Moodle LMS?

Method Participants
The subjects were college students from a state university's Faculty of Education.All students were given access to the data collecting tool through the LMS, and participation was voluntary.College students from 13 departments participated in the current study; of the 146 participants, 102 were females (69.9%) and 44 were males (30.1%).The age of the participants ranged from 18 to 33 years, with an average age of 21.66 (SD = 2.61).Being an experienced user of both ALMS and Moodle LMS was a criterion for inclusion in the current study.

Data Collection Tools
An online questionnaire was used to collect data.This questionnaire contained demographic profile items, the Learning Management Systems Evaluation Scale -LMSES (Barut Tuğtekin, 2021), and the Online Examination Assessment Scale -OEAS (Yilmaz, 2016).LMSES consisted of 19 items and 3 factors, with a 5-point Likert scale ranging from (1) strongly disagree to (5) completely agree.Because the LMSES had one reversed item, it was reverse scored for this study.According to the original form of the LMSES, the explained variances of the factors were 23.06% for dialogue, 25.74% for structure, and 14.93% for autonomy.The fit indices obtained from the LMSES (structure = 0.9, dialogue = 0.89, autonomy = 0.82; χ 2 = 252.78,df = 146, χ 2 /df =1.73; CFI = 0.95, NFI = 0.90, GFI = 0.89, AGFI = 0.85; SRMR = 0.06, RMSEA = 0.06; p < 0.001), and Cronbach's Alpha (α) reliability coefficients were at an acceptable bound (i.e., α > .70).The OEAS had 3 factors and 17 items, with a 5-point Likert scale ranging from (1) strongly disagree to (5) completely agree.Because the OEAS contained six reversed items, these items were reverse scored and included in the ongoing analyses.According to the original form of the OEAS, the practicality-suitability factor explained 36% of the variance, the affective factors about 17%, and the reliability factor approximately 9%.Cronbach's alpha reliability coefficients for factors were found to be (α= 0.89) for practicality-suitability, (α = 0.82) for affective, and (α = 0.82) for reliability.

Procedure
Moodle, ALMS, Canvas, and Blackboard are popular LMS and are often used in the region where the research was done.Both Moodle and Canvas are open source and free to use, while ALMS and Blackboard are commercial LMS with annual fees.Although Blackboard has been used throughout the world, ALMS was developed in Turkey by Advancity.It has become one of the most popular LMS there, even though it is not used extensively worldwide.Moodle has been used in over 70 higher education institutions, and ALMS has been used in close to 60 higher education institutions when comparing the most popular LMS in Turkey (Cabi & Ersoy, 2022;Karadag et al., 2021;Yolsal & Yorulmaz, 2022).This study examined the use of Moodle LMS and ALMS, among the most frequently used LMS in the region.request for any other identification information.The goal was to select the same participants who also took part in the subsequent Moodle LMS implementation.In the second stage of the study, college students from the Faculty of Education who also studied through Moodle LMS in the fall semester of 2021-2022 were offered an online questionnaire to evaluate Moodle at the end of the semester.As with the previous implementation, the participants' nicknames and e-mail addresses were gathered, and their participation status in former ALMS sessions was also checked and verified.Following the second implementation, one-to-one comparisons of nicknames and e-mail addresses were performed, and the learners who participated in both implementations were determined.These individuals comprised the sample for this study.Figure 1 depicts the complete research procedure.

Figure 1
Research Procedure tested.The results of the analyses showed that the homogeneity of equal variance assumption was not violated, and that Mauchly's test of sphericity significance value was above 0.05 (Cooley & Lohnes, 1971).Once the prerequisites were fulfilled, two-way repeated measures ANOVA was conducted.The average scores for all the scales and factors were calculated and analyzed, and the average scores were interpreted.

Findings
Table 2 presents the average LMSES and OEAS scores of participants for two distinct LMS environments.When the total mean scores for the scales were compared, the LMSES scores for Moodle LMS (Mean = 3.74; SD = 0.69) outperformed the ALMS (Mean = 3.46; SD = 0.72).When the OEAS scores used to assess the online tests are compared, the average scores of Moodle LMS and ALMS were quite close.Two-factor repeated measures ANOVA was conducted to scrutinize the differences between the LMSES factors for the ALMS and Moodle LMS within the context of the Transactional Distance Theory.The findings are summarized in Table 4. Based on the differences between the LMS type variable across the groups, findings in Table 4 revealed a significant result (F(1-145) = 8.982; p < 0.01; ηp 2 = 0.058).Additionally, the statistical power value was found to be 0.845.There were found to be statistically significant differences between the groups in the analysis of the LMSES factors (F(2-290) = 187.040;p < 0.001; ηp 2 = 0.563).As well, it was revealed that there was a statistically significant difference in the interaction of the LMS type and LMSES factors (F(2-290) = 29.218;p < 0.001; ηp 2 = 0.168).The power value of this result was found to be 1.00.Figure 2 depicts the variations of LMSES factors based on LMS type.

Changes in LMSES Factors According to LMS Type
Figure 2 shows that the dialogue factor, for which Moodle had a higher mean score, was where the two LMS differed most significantly.On the other hand, both LMS scored similarly on the structure factor.
To ascertain which LMSES factors varied in statistical significance, a straightforward main effect analysis was used and paired-samples t-tests were conducted.The results are shown in Table 5.There was a significant difference between LMS in terms of dialogue (t(145) = -5.570;p < 0.001) and autonomy (t(145) = -2.213;p < 0.05), both of which are factors of LMSES.Since the value calculated for the dialogue factor was larger than 0.14, it suggested a large effect size, and since the value computed for the autonomy factor was less than 0.06, it indicated a small effect size (Cohen, 1988).When the averages of the OEAS factors in Table 6 were evaluated, it was revealed that the reliability factor for Moodle LMS had the higher score, while the ALMS usability factor had the lowest.
A two-factor repeated measures ANOVA was conducted to assess OEAS factors to measure differences in online exams based on the type of LMS (i.e., ALMS and Moodle LMS).The findings are presented in Table 7. Prior to the related analysis, the prerequisites were checked, and the sphericity assumption was not violated.According to Table 7, the difference in terms of the LMS type variable was not statistically significant, however, there was a statistically significant difference between groups in the analysis of the OEAS factors (F(2-290) = 12.257; p < 0.001; ηp 2 = 0.996).It was determined that there was no statistically significant difference while evaluating the related outcomes for the interactions of the LMS type and OEAS factors (F(2-290) = 0.114; p > 0.05; ηp 2 = 0.001).However, Figure 3 illustrates the variations in OEAS by LMS type.

Average Online Exam Scores for OEAS Factors by LMS Type
As seen in Figure 3, the higher difference between the two distinct LMS is in the usability factor, with Moodle LMS scoring better.When the mean scores of the OEAS factors were examined, the higher means were found in the reliability factor.Furthermore, Moodle LMS had higher average OEAS scores than ALMS in each factor.In brief, even if there was no statistically significant difference (F(2-290) = 0.114; p > 0.05; ηp 2 = 0.001), Moodle LMS had higher average scores in online exam evaluation than did ALMS.

Discussion
Although there are many different approaches to implementing e-learning, LMS are one of the most effective platforms for carrying out educational activities efficiently, effectively, and systematically.
Because of this, educational institutions look for a LMS that can satisfy their e-learning requirements.
There are two main options when choosing an LMS to address institutional needs.
One is an open source LMS, while the other is a pay-for-use LMS that has been commercially developed.Open source LMS are free-to-use and may be customized to meet an institution's demands, but these come with a range of maintenance and development costs.While the costs of acquiring commercial LMS are substantial, such systems have been designed expressly for the institution and might be simpler to use.Therefore, when deciding between free-to-use and commercial LMS, it is essential to evaluate (a) the institution's demands; (b) LMS ease of use, as well as features that improve and support satisfaction, and (c) the potential resources necessary for LMS implementation (Kasim & Khalid, 2016).Participants in this study used both open source free-to-use Moodle LMS, and the commercial ALMS at different time periods.Comparisons were made between the two alternative LMS.Both Moodle LMS and ALMS were linked to other systems in the institution were fully ready to use.
The usefulness, efficiency, and usability of LMS can be affected by various factors.According to research on ALMS, usability, intention to use, and satisfaction levels have been directly influenced by the quality of the course material and user interface design (Yoruk et al., 2020).According to Alshurideh et al. (2021), the perceived usability and utility of e-learning systems have been significantly influenced by the quality of the content.Since this study examined two distinct kinds of LMS, it is possible that their particular interface designs led to differences in the LMS rating scores.Learners' use of particular LMS during different education terms may have resulted in a range of quality levels in the presentation of instructional information in various courses.
When LMSES scores were considered, the average for Moodle LMS was higher than for ALMS.As a result, it can be argued that Moodle LMS is a more practical and efficient LMS option than ALMS.When the interactions of the LMSES factors were examined while taking into consideration the different LMS types, there was no statistically significant difference in the structure factor of the LMSES, but there was a significant difference in the dialogue and autonomy factors.Additionally, as compared to ALMS, the Moodle LMS revealed a positive and statistically significant difference in the autonomy and dialogue factors.Thus, it may be claimed that Moodle LMS encourages learners to act more independently and that ALMS has a poorer capacity for dialogue.On the other hand, the fact that the structure factor of ALMS had a higher average score than did the other factors, indicates that the ALMS interface was well structured.In addition, course format affects learners' autonomy, as well as learner-learner and learnerinstructor communication (Abuhassna et al., 2022).When analyzing how LMS features encourage learners to act independently and participate in dialogue, it is important to consider the ways that instructors use these activities and how frequently.As well, even though learners' autonomy is seen as a crucial notion in e-learning environments (Castañeda & Selwyn, 2018), the use of educational technologies that reinforce learners' autonomy may trigger learner-centered research (Lazorak et al., 2021).Although the structural elements of LMS (i.e., interface and curriculum) were evaluated using Transactional Distance Theory, the methods and activities employed by instructors in relation to autonomy and dialogue factors can also play an essential role.Therefore, to improve autonomy and dialogue in the successful use of an LMS and manage structural aspects, the LMS interface, features, and ease of use, as well as instructional materials, coursework, and related instructional activities should be scrutinized.
Moodle LMS and ALMS had similar average scores in the overall comparison of online exams.When the OEAS factors for online exams used in LMS were assessed, the reliability factor of Moodle LMS had the highest average score, while the practically-suitability factor of ALMS had the lowest average score.
However, there was no statistically significant interaction between LMS type and OEAS factors in online exam evaluation.There was no statistically significant difference between online exams according to the type of LMS employed and the OEAS factors with which the online exam procedures were evaluated.
Even with no statistically significant difference, Moodle LMS outperformed ALMS in terms of average scores for each OEAS factor in the assessment of online exams.The average score for the reliability factor of both LMS was relatively higher in comparison to the other factors when Moodle and ALMS were compared using the framework of practically-suitability, affective factors, and reliability.However, because the structural relevance of the online exam questions is measured by the reliability factor of the OEAS, it may be concluded that instructors typically provide trustworthy online exam items.
On the other hand, the fact that both LMS platforms offer unsupervised online exams, and that most instructors favor multiple-choice exams, may have led to comparable experiences for learners during the online exam procedures.Online tests may be associated with a variety of security issues; it is recommended that they be used for formative rather than summative evaluation to ensure that assessments are accurate, dependable, and adaptable when used in distance learning (Shraim, 2019).
Considering the security issues with online examinations, formative evaluation targeted at enhancing learning may be a better option for online assessment rather than grading with summative assessment.
On the other hand, it would be difficult to provide a formative evaluation setting that delivers individual feedback in online exams when there are numerous participants (Ilgaz & Adanir, 2020).Furthermore, system quality has been cited as the most fundamental component influencing online exams, e-learning experience, mobile learning, and cloud services (Akar & Mardikyan, 2014).Therefore, improved system quality is likely to boost both LMS use and intentions to use (Alshurideh et al., 2021;Liu et al., 2010).
As Dermo (2009) has indicated from learners' assessment of online exams, it is crucial to improve exam procedures by addressing affective factors, validity, practical issues, reliability, security, as well as learning and teaching considerations.

Limitations
There were some limitations to this study that should be noted when interpreting the research findings.
First, this study was limited to evaluating college students' use of ALMS and Moodle LMS for a semester each.Second, since the institution managed the sequence in which the LMS used in this study were implemented, the inability to alter this sequence should be regarded as one of the crucial limitations.
Third, while the data instruments used in the LMS comparisons were validated, they were limited to LMSES and OEAS scales.Fourth, it was expected that instructors used LMS systems efficiently while creating and delivering online exams and related course materials.As well, was assumed that learners had a sufficient degree of expertise using the LMS since the institution provided user guides and support services.Finally, although participation in the LMS surveys was entirely optional, it was assumed that respondents provided honest evaluations.

Conclusion and Practical Implications
In the current study, the use of two LMS in the e-learning process was scrutinized using the time series approach and within the context of Transactional Distance Theory.Additionally, the effectiveness of online exam procedures in each LMS was assessed.The research findings indicated that online exams in Moodle LMS and ALMS both had similar assessment ratings, while Moodle had a higher evaluation score for the e-learning process.The findings obtained from the Transactional Distance Theory factors indicated that, despite ALMS's structural aspects being predominant, Moodle's strength was mostly tied to learners' autonomy.It revealed that when evaluated according to the LMSES factors, average scores of the dialogue and autonomy factors of Moodle LMS were significantly higher than for ALMS.
The reliability of both LMS was found to be a better indicator than other factors in comparing LMS online exams, where it was found that there was no statistically significant difference between ALMS and Moodle LMS.
We recommend that in selecting and using LMS, choices should be based on their specific characteristics in accordance with the demands of the institutions.Additionally, we believe that LMS may be used more effectively when e-learning instructors are offered specific training to improve their abilities to use LMS.However, we think that results are comparable when tests are given online in an unsupervised setting and are typically of a similar kind.Therefore, we recommend doing empirical comparisons of online exams in e-learning environments for various exam types (e.g., supervised vs. unsupervised, multiple choice vs. open-ended).

Table 1
compares some notable characteristics and attributions of the Moodle LMS and ALMS as used in the current research.
learners who had experienced both LMS.First, an online data collecting tool was made available to Faculty of Education students who were taking courses via ALMS during the spring semester of 2020-2021.This online survey collected the participants' nicknames and e-mail addresses only, with no direct

Table 2
Descriptive Statistics of ALMS and Moodle LMS for LMSES and OEAS

Table 3
presents the descriptive statistics for ALMS and Moodle LMS regarding LMSES factors based on Transactional Distance Theory.

Table 3
Descriptive Statistics of ALMS and Moodle LMS for LMSES Factors According to Table3, when the averages of the LMSES factors were examined, autonomy in the Moodle LMS had the greatest average score, and dialogue in ALMS had the lowest.Additionally, structure in ALMS had a higher average score than the other ALMS factors.

Table 6
presents the descriptive statistics of ALMS and Moodle LMS for OEAS variables, whereby online exams made in the two distinct LMS types were compared.

Table 6
Descriptive Statistics for Online Exams Via Distinct LMS

Table 7
ANOVA Results for Interactions of LMS Types and Online Exam Factors * p <0.001.