What Are the Indicators of Student Engagement in Learning Management Systems? A Systematized Review of the Literature

Student engagement has an important role in academic achievement in all learning contexts, including e-learning environments. The extent of monitoring and promoting student engagement in e-learning affects the quality of education and is a determining factor for ensuring student’s success. Log data of students’ activities recorded in a learning management system (LMS) can be used to measure their level of engagement in the online teaching–learning process. No previous studies have been found stating a consistent and systematically raised list of LMS-based student engagement indicators, so this systematized review aimed to fulfill this gap. The authors performed an advanced search in the PubMed, Ovid, Google Scholar, Scopus, Web of Science, ProQuest, Emerald, and ERIC databases to retrieve relevant original peer-reviewed articles published until the end of June 2021. Reviewing the 32 included articles resulted in 27 indicators that were categorized into three themes and six categories as follows: (a) log-in and usage (referring to LMS, access to course material), (b) student performance (assignments, assessments), and (c) communication (messaging, forum participation). Among the categories, access to course material and messaging were the most and the least mentioned, respectively.

course structure, and learner autonomy-are factors that affect students' feelings of transactional distance.He argues that instructors, learners, and educational organizations can use these elements to plan for effective and deliberate learning (Moore, 1993).This theory of transactional distance has been used as the theoretical framework for research in online and technology enhanced learning, in which the medium of learner-instructor communication provides the chance for higher levels of interaction and engagement (Moore, 2018).This is important because the extent of monitoring and promoting student engagement in e-learning affects the quality of education (Henrie et al., 2017) and is a determining factor in whether an e-learning strategy is productive for educational institutions through ensuring students' success (Meyer, 2014).Several studies have mentioned low levels of student engagement as the most important reason for student drop out in e-learning (Lee & Choi, 2011;Kim et al, 2017).Moreover, the level of student engagement shows the university's level of commitment to academic activities and active learning (Lee et al., 2019).Implementing strategies to replace an estudent's sense of isolation with relatedness and closeness facilitates their being more active in online courses, which results in the student's higher satisfaction as well (Young & Bruce, 2011).Moreover, monitoring student engagement in e-learning environments helps in improving instructional events through recognizing the learners that need more support for following their studies on the path toward success (Henrie et al., 2015).
Studies on student engagement focus on two main aspects: learning behaviors and feelings of emotional belonging.In contrast to emotional belonging, learning behaviors are often measured quantitatively based on generic indicators of student engagement, which assess, report, and value the university's performance.Such a qualitative analysis provides authorities and stakeholders with a feeling of certainty in understanding whether an educational process goes well or not (Zepke, 2015).Several studies have addressed the characteristics or constructs that make up student engagement assessment in a quantitative method, though a few have focused on the e-learning context (Lee et al., 2019).Most of these are cross-sectional studies, in which engagement self-reporting methods were used rather than continuous monitoring (Henrie et al., 2017).Meanwhile, an e-learning context with the possibility to instantly record indicators of student's behaviors and learning activities within a learning management systems (LMS) provides a valid and approximate measure for student engagement in courses (Henrie et al., 2017;Motz et al., 2019).
LMSs are Web-based software used mainly for asynchronous interaction between instructors and students by delivering a course's information, materials, and activities (Raza et al., 2021).Generated log data in LMS-based courses can be used as predictors of student achievement (Macfadyen & Dawson, 2010).Records of user's activities within a software are labeled as log data, which include items such as number of clicks or page views, time spent on an action, and results of performed tasks or activities.
Reviewing log data demonstrates a user's real-time interaction with the software and can be analyzed to understand any changes in a user's behavior (Henrie et al., 2017).The advantage of log data is that the data are automatically collected without any interference from instructors and staff.In addition, log data present objective information about aspects of a user's behavior that are not easily measured in other ways (Macfadyen & Dawson, 2010;Zanjani, 2015).In fact, intelligent and effective analysis of LMS log data-that is, learning analytics-not only assists in promoting student's success and retention rate and supporting at-risk learners (Atherton et al., 2017) but also provides the possibility to implement personalized learning, which is a revival of the learner autonomy concept in Moore's (2018) transactional distance theory.The result of such analysis displays a student's level of engagement and learning pattern, even at the initial stages of a course, and provides evidence for timely interventions to improve a student's performance (You, 2016).
Hence, regarding the benefits of analyzing LMS log data to approximately measure student engagement, this systematized review of relevant literature was conducted to identify LMS indicators that can be used for this purpose.

Methods
The Cochrane Collaboration's Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher et al., 2009) was followed to conduct this systematized review.A systematized review follows the methodology of a systematic review and includes some of its elements; however, it cannot meet all the criteria for a full systematic review (Grant & Booth, 2009).
An advanced search was conducted in the PubMed, Ovid, Google Scholar, Scopus, Web of Science, ProQuest, Emerald, and ERIC databases on February 5, 2021, to retrieve relevant articles up to the end of June 2021.The search operators included Boolean operators (AND, OR, and NOT), parentheses, and truncation; the following keywords were searched as single terms or in combination with others: online learning, online education, distance learning, distance education, virtual learning, virtual education, e-learning, electronic learning, mobile learning, M-learning, distance study, distributed education, distributed learning, open learning, engagement, achievement, performance, progress, students, adult learners, learners, and users.Due to the limited number of articles dealing with the indicators of student engagement in LMSs, the search was not limited by keywords related to LMSs.The following is an example of a Web of Science search query: TS = (("online learning" OR "online education" OR "distance learning" OR "distance education" OR "virtual learning" OR "virtual education" OR "e-learning" OR "electronic learning" OR "mobile learning" OR "M-learning") AND ("student engagement" OR "student achievement" OR "learners engagement" OR "learner's achievement" OR "users engagement" OR "user's achievement" OR "student involvement" OR "learner's involvement")) After extracting articles, duplicate ones were excluded using Endnote X8.2 (Clarivate Plc, London, UK).
The inclusion criteria for this review were as follows: (a) English language, (b) original articles, (c) student engagement in e-learning software being the subject of the study, and (d) availability of articles' full texts.Studies that had used only special software or hardware facilities for monitoring student engagement, such as equipment for eye tracking, face recognition, or monitoring of mouse scrolling or movements, were not included.The reason was the focus of study was on data that were logged within routinely accessible e-learning software, independent of other external equipment or software.
However, the articles including both types of monitoring were considered, and the results relevant to the aims of the review were retrieved.Studies on indicators of student engagement in online synchronous classes were omitted.
After article retrieval, two independent researchers reviewed the articles' titles and abstracts using a standard checklist form to exclude irrelevant articles.Then they separately performed an in-depth assessment on articles' full texts to determine their eligibility.At this stage, any inconsistencies between reviewers were resolved.
In the next step, the bibliographic data from each eligible article were extracted, and key findings and results were summarized and recorded.The PRISMA diagram (Figure 1) was considered for assessing the articles' retrieval, extraction, and removal.Moreover, the Critical Appraisal Skills Programme checklist was used to investigate each article's quality (Critical Appraisal Skills Programme, 2018).
Finally, two independent researchers thoroughly read the included articles in order to extract mentioned indicators of student engagement in an LMS log.As mentioned, any inconsistencies between these two researchers' results were resolved by a third researcher's review.

Articles' Retrieval and Bibliographic Information
After performing article retrieval steps, 32 articles were eligible to enter the study.Figure 1 shows the PRISMA diagram for this review, and Table 1 shows the search results based on databases.

Figure 2
Student Engagement Themes Note: LMS = learning management system.

Frequency of Articles Stating Each Category of Student Engagement Indicators in LMSs
Note: LMS = learning management system.

Discussion and Recommendations
In this systematized review on 32 articles, student engagement indicators based on LMS log data were identified and categorized into three themes of referring to LMS, student performance, and communication, which included six categories.Among these categories, access to course material was the most mentioned (23 articles), followed by forum participation (22 articles), log-in and usage (16 articles), assessments (14 articles), assignments (12 articles), and messaging (8 articles).
Despite the positive relationship between students' activity levels within LMSs and their success in their courses (Grubišić et al., 2020), no study was found that cumulatively addressed LMS-based student engagement indicators.Hence, the results of this review provide insight into the indicators used for assessing student engagement in LMSs and their relative priority.
According to the findings of this study, access to course material in the LMS was the most mentioned indicator of student engagement in the literature.LMSs provide the option of uploading course materials to create a virtual learning environment in both fully online and blended courses (Chaka & Nkhobo, 2019).The opportunity to monitor and control students' access to course material is one of the must-have features of LMSs and proves to be an indicator of student engagement level within the course (Krasodomska & Godawska, 2021).Students prefer having access to course material 24 hours a day, 7 days a week; and use this feature more than other LMS features.This results in increased engagement with the course and provides the possibility for students' cognitive involvement, self-regulation, and self-paced learning (Chen, 2020).These results show the importance of monitoring such data as an indicator of student engagement in LMSs.
Forum participation was found to be the next considerable indicator for monitoring student engagement in LMSs in this study.In fact, LMSs have the feature of providing an environment for faculty and students communication, which helps in building a community of practice for collaborative learning rather than personalized individualized instruction (Moore, 2018).Moreover, students communicate with peers.A frequently used LMS communication tool is the discussion board or forum, which facilitates asynchronous collaboration among faculty members and students (Kew & Tasir, 2021).
Analysis of data recorded in communication tools of LMS is supportive for assessing students' behavioral, emotional, and cognitive engagement within e-learning environments (Henrie et al., 2017;Yassine et al., 2016).Forums allow for provision of feedbacks and comments on students' work in order to promote academic goals (Kim, 2017).In addition, students' active and passive participation in forum dialogues is positively associated with learning outcomes, and instructors and course designers concentrate on ensuring high levels of participation from students.Analysis of data gathered from forum participation helps in understanding its impacts on academic indicators, including the level of student engagement.Even though participation in forums can be obligatory, the level of students' participation in forums may vary and shows their interest and engagement in course activities (Henrie et al., 2017;Yassine et al., 2016).In this regard, while students' behavioral engagement is determined by the general use of the communication tools and platform (Mogus et al., 2012), their emotional engagement is analyzed through their self-expression in forums or discussion environments (Wang, 2017).
Log-in and usage, as a category within the referring to LMS theme, is the third ranked indicator for student engagement.Based on the existing research, LMS usage log data can be effectively used to measure student engagement (Wang, 2017;You, 2016).Student's log-in and log-out data, as indicators of their behavioral engagement, have a strong positive correlation with their final grades (Mogus et al., 2012).In fact, spending more time within the LMS is associated with more engagement with the course's activities and resources (Wang, 2017).
Other indicators extracted in this review are the categories under the themes of student performance, namely, assessments and assignments.Behaviors such as viewing and uploading assignments and participating in and completing quizzes play an important role in students' academic performance and engagement (Franzoni et al., 2020).In fact, data such as the number of submitted assignments and completed online quizzes are used to quantify students' regularity of participation in course activities and can show the level of students' persistence in fulfilling learning expectations and engagement (Rajabalee et al., 2019).In other words, the more students are engaged in a course through revisiting and performing such activities, the more effectively they learn (Krasodomska & Godawska, 2021).
Messaging in the communication theme is the last ranked indicator of student engagement based on this review, with only 8 out of 32 articles stating it.The low number of mentions in these studies may be due to internal messaging not being a must-have feature of LMSs, and sometimes off-system messengers and social networks are preferred for this purpose (Ross, 2019;Zaaruka & Mosha, 2019).Meanwhile, communication activities in LMSs, such as number of messages, indicate students' engagement in virtual environments (Ramesh et al., 2014).For example, in one study, students' participation within an e-learning environment was analyzed by using the total number of students' sent messages and total access; a moderate relationship was found between such participation and final grades (KunhiMohamed, 2012).
Finally, understanding such indicators is important because LMS usage is increasing in the academic sector, and students are spending more time and effort in these e-learning environments than ever before.Therefore, it is important to choose the appropriate LMS for a course, because it controls the way that learners engage with the course activities and interact with the material, their instructors, and their peers (Roach & Attardi, 2021).In this study, the number of citations of the indicators pertaining to each category in related articles showed that only messaging had a significant difference among LMS types, with institutions' local LMSs having the highest number of citations of the indicator.This result may be due to the low number of the articles for each LMS type.However, even if LMS features are basic ones, it is possible to use them without compromising the quality of teaching through appropriate course design (Roach & Attardi, 2021).An effective course design needs gathering information about students' participation and engagement through features offered by most LMSs that is helpful in substituting the insight that teachers gain about students' learning in traditional classes.

Limitations
Despite the researchers' efforts, this study has some limitations that must be considered.Although the literature search was conducted in multiple databases, as recommended for review articles (Cronin et al., 2008), possible bias in selecting databases or formulating search strategies may have resulted in missing relevant publications.Moreover, this review included only English-language articles.Articles in other languages and non-article publications, such as dissertations, may contain other indicators.

Future Research
Since research on student engagement in e-learning environment is still emerging, there are opportunities for future studies.Analyzing student engagement based on identified indicators of this study and comparing the results with the findings of the students' self-reports of their engagement may expand knowledge in this regard.Furthermore, working on predictive models of student performance based on these indicators would provide the chance for early interventions to support students, prevent dropouts, and improve student retention.In spite of the usefulness of such quantitative analysis, in which course activities have the same weights, it is recommended to work on solutions to determine student engagement levels according to the importance and alignment of activities with learning objectives.

Conclusion
The

Figure 1 PRISMA
Figure 1 PRISMA Flow Diagram for Retrieving Articles in forums 4, 29, 31 Length of sent posts (short or long posts) Note.LMS = learning management system.Descriptions of each category are as follows: Log-in and usage: This category and its indicators show not only the amount of time spent in an LMS but also the regularity and intervals of referring to it.Access to course material: This category consists of indicators that demonstrate how much a student has interacted with course content and additional pages.Furthermore, the student's preference to study the content before doing activities and their evaluation of the content are considered indicators of student engagement.Assignments: Assignments are one of the main learning activities in an LMS-based course, so all potential LMS log data related to assignments are considered a category of student engagement indicators.Assessments: LMS-based assessments provide objective data for estimating a student's level of engagement with the course.Hence, in this category, all LMS log data related to assessments are listed.Messaging:By default, students have access to the LMS messaging module to communicate with other LMS users.Sent and read messages show how much a student has interacted with instructors and peers in a course.Forum participation: Indicators listed in this category estimate a student's level of activity in forums or discussion groups in an LMS.

Figure 2
Figure 2 demonstrates the percentage of obtained indicators for each theme out of the total number of indicators, and Figure 3 depicts the frequency of articles stating each category.Moreover, among 32 results of this systematized review enrich the current literature.No previous studies have addressed a cumulative list of certain LMS-based indicators for measuring student engagement in e-learning environments.Hence, identifying such indicators has expanded the literature in this regard.Institutions and academics can use this list of indicators to (a) constantly monitor students' engagement in asynchronous e-learning platforms, (b) determine strength and weaknesses of delivered e-courses, (c) identify and support students with low engagement levels, (d) plan for implementing personalized learning, (e) plan for faculty development programs to familiarize faculty with the activities that bring about higher levels of student engagement, and (f) compare institutions' current LMS features and logs with the list of indicators to determine the ones that can be added to the software, if there are any, to promote the chance of engagement.

Table 1
Search Results

Table 2
includes information on included articles' bibliographic characteristics.The 32 articles were published in 28 journals, among which Computers & Education had the highest number of publications (4 articles).

Table 2
Bibliographic Information of Included Articles in Chronological Order

Table 3
Student Engagement Indicators in LMS and Articles Stating Them

Table 4
Comparison of Indicators' Number of Citations in the Articles Related to Each LMS Type in Total and for Each CategoryNote.a Number of citations of the indicators pertaining to each category in related articles of each LMS type.