International Review of Research in Open and Distributed Learning Participation in Online Courses and Interaction With a Virtual Agent

Online learning environments could be well understood as a multifaceted phenomenon affected by different aspects of learner participation including synchronous/asynchronous interactions. The aim of this study was to investigate learners’ participation in online courses, synchronous interaction with a conversational virtual agent, their relationships with learner performance, and the participation/interaction factor identification. To examine learner participation, we collected learning management system (LMS) log data that included the frequency and length of course access, discussion board postings, and final grades. To examine synchronous learner interaction, we collected learners’ conversation logs from the conversational agent. We calculated the quantity and quality of discussion postings and conversations with the agent. The results showed that the frequency and length of course access, the quantity and quality of discussion postings, and the quality of conversation with the agent were significantly associated with the learner achievement. This study also identified two factors that comprise online learning participation and interaction: interaction quality and LMS-oriented interaction.


Introduction
Enrollment in online courses has sharply increased, specifically in higher education (Seaman, Allen, & Seaman, 2018), boosting educational researchers' interest in online learning.Researchers in this field are redefining our understanding of presence in light of the ability of individuals to interact extensively online with learning content, instructors, peers, and the learning environment.At the same time, they acknowledge major challenges, such as low levels of learner performance, passive participation, and higher attrition rates (Levy, 2007;Stoessel, Ihme, Barbarino, Fisseler, & Stürmer, 2015).In a comparison study investigating the difference between face-to-face and online courses in higher education, on average 10% of online learners failed in courses whereas only 4% of face-to-face learners did (Ni, 2013).Thus, the expansion of distance education has both benefits and detriments.
Low learner participation is one of the most significant issues in online education.This could be caused by poorly designed interaction opportunities for learners.Research has shown that online learning can be as effective as face-to-face courses, but only if learners are provided well-designed interaction activities (Hawkins, Graham, Sudweeks, & Barbour, 2013;Joksimović, Gašević, Kovanović, Riecke, & Hatala, 2015;Picciano, 2002).Croxton (2014) found that purposefully designed and engaging interaction tasks played a significant role in learner persistence in online courses.Therefore, it is imperative that we design online learning environments to foster meaningful interactions for learners (Bettinger, Liu, & Loeb, 2016;Goggins & Xing, 2016;Hrastinski, 2008).
One of the challenges of encouraging learner participation through purposeful and engaging interactions is that current online learning activities are mostly designed in an asynchronous manner.It is difficult in a typical online course for an instructor to promote positive experiences of interaction for learners because these require immediate and quality feedback from the instructor.In asynchronous environments, it is also demanding to implement the kinds of seamless and continuous learning activities that would assist learners in carrying out real-world projects (Boling, Hough, Krinsky, Saleem, & Stevens, 2012).Although online courses can use synchronous activities such as online conferencing, there are concerns that synchronous meetings could diminish one of the major benefits of open and distributed learning: that learners can learn at any time (Song & Lee, 2014).Accordingly, much of the research in online learning has focused on learners' participation issues in asynchronous interaction activities, while there is a lack of studies on learners' synchronous interaction behaviors in the context of distributed learning environments.

Literature Review
A number of researchers have reported that learners' active participation in online courses is associated with high levels of learner performance and higher retention rates (Bettinger et al., 2016;Goggins & Xing, 2016;Hrastinski, 2008;Stoessel et al., 2015).Michinov, Brunot, Le Bohec, Juhel, and Delaval (2011) found that learner participation levels, measured by the number of messages learners posted to discussion forums, mediated the relationship between learners' procrastination and academic achievement.The researchers suggest that encouraging learner participation leads to the increased performance of online learners, especially those who have a tendency to procrastinate.Bettinger et al. (2016) also examined the effects of learners' participation on their online course performance and persistence.More active participation in discussion forums was associated with higher performance in the courses and lower dropout rates in the following academic term.Learners' active participation can therefore be considered a key factor in learning success in online courses.

Learner Interaction
With respect to the issue of learner participation, note that participation is intertwined with interaction.
Generally, learner participation refers to "a process of taking part and also to the relations with others that reflect this process" (Wenger, 1998, p. 55).This broad definition includes two types of learner behavior (i.e., "a process of taking part," such as submitting an assignment or reading an assigned article) and interaction (i.e., "the relations with others," such as chatting with peers or having a discussion with peers).In this research area, the concept of learner behavior as a "process of taking part" has been employed as a narrower definition of learner participation rather than the concept of learner participation includes interaction.For example, the following studies showed that encouraging learners' active participation by providing more interaction opportunities is one effective approach that promotes success in online courses (Croxton, 2014;Hawkins et al., 2013;Joksimović et al., 2015;Picciano, 2002;Wu, Yen, & Marek, 2011).It seems that interaction is a factor that contributes to participation.
It should be noted that not all interaction activities promote participation and performance.Sabry and Baldwin (2003) explored relationships between learners' preferences and online interactions with information, with the instructor, and with other learners.One hundred eighty-nine undergraduate and graduate students completed a questionnaire that asked about their online learning interaction experiences.
Learners were found to have different perspectives and preferences towards online interactions.
Specifically, regarding frequency of use and perceived usefulness, learners in their study more often interacted with information than with their instructor and with other learners.Similarly, the effects of interaction on online learners' performance might depend on the content of interactions.Kang and Im (2013) investigated learner interactions and perceived learning in an online environment.Six hundred fiftyfour undergraduate students responded to a survey that asked about learners' interactions with their instructor and their perceived performance.Kang and Im's exploratory factor analysis showed that instruction-related interaction factors had more predictive power for perceived performance than noninstructional interaction factors (e.g., social intimacy, social exchange of personal information).Thus, it seems that instructional content-related interaction has a more significant effect on learner performance than the other types of interactions.Still, further research is required to identify different types of interaction and their different roles in online learning.
In the previous studies (Goggins & Xing, 2016;Swan, 2001), the quantity of interaction has been considered a significant factor that predicts learning outcomes.Goggins and Xing (2016) examined learners' asynchronous interaction activities in a discussion forum in an online graduate course.The quantity of learner participation in the discussion (which was measured by the number of posts both written and read) was significantly correlated with achievement.In their investigation of 1,406 undergraduate students' perceptions of their online learning experience, Swan (2001) also found that students who were more active in online courses, higher levels of personal activity including higher perceived levels of interaction with the instructor and peers, reported higher levels of satisfaction and perceived learning and earned higher course grades.Swan found that frequent interaction with course materials was one of the most important aspects of online learning.Therefore, the magnitude of interaction might promote achievement of learning outcomes.
Asynchronous and synchronous interaction.Along with the importance of interaction quantity and content-related aspects, the synchronicity of interaction has been discussed in the previous studies (Baker, 2010;Johnson, 2006;Swan, 2001).Asynchronous interactions are beneficial for learners because they offer the time to find more learning resources, speculate about the topic, reflect on their learning, and elaborate their own knowledge (Johnson, 2006).Online discussion forums provide learners opportunities to reflect on peers' and instructors' contributions and consider their own arguments before sharing them.Conversely, asynchronous interactions often lack timeliness or immediacy.In learners' interactions with instructors, timely and immediate responses can reduce the psychological distance between them and promote learner achievement in online learning environments (Swan, 2001).Baker (2010) investigated undergraduate and graduate students' perception of affect, cognition, motivation, and their instructors' immediacy and presence in online courses.Immediacy of interaction was strongly associated with students' positive affective and cognitive status.Although asynchronous interaction activities such as discussion forums may partially support timely interaction, this requires an instructor's prompt facilitation and learners' immediate responses, which are not common in online courses.
If immediacy is key to positive interaction effects, synchronous interactions could be more effective than asynchronous interactions in certain contexts.However, synchronous interaction activities have not frequently been implemented in online courses because of the challenges they present.The instructor needs to ask all students to be online at a certain time and moderate large-scale online conversations (Yamagata-Lynch, 2014).This type of practical difficulty is one reason that few studies have examined learners' synchronous interaction activities in online courses (Giesbers, Rienties, Tempelaar, & Gijselaers, 2014).
Accordingly, the roles, characteristics, and effectiveness of synchronous interactions in online courses have not been fully explored.Thus, because of the challenges, we need to get creative in our implementation of synchronous interaction for both research and practice areas.

Synchronous Interaction with Conversational Agents
Conversational virtual agents, or chatbots, are computer programs that communicate with users in natural language and so have been used for user-system interaction in many online spaces (Shawar & Atwell, 2007).
By simulating human dialog patterns, they can conduct an interaction task through conversation with a user.One of the first conversational agent systems, ELIZA, was developed in 1966 by Joseph Weizenbaum.
Users interacted with ELIZA in a synchronous manner.ELIZA simulated a therapist role in clinical treatment situations, analyzed what users typed, and created its responses based on predefined decision rules.
There are cases in which conversational agent systems have been used for educational purposes (Fryer, Ainley, Thompson, Gibson, & Sherlock, 2017;Heller, Proctor, Mah, Jewell, & Cheung, 2005;Jia, 2009).Abbasi and Kazi (2014) investigated the use of a conversational agent as an answer retrieval tool to solve programming questions.Seventy-two undergraduate students were randomly assigned to either a Google group that used the Google search engine to retrieve information, or to a conversational agent group that asked questions of the agent to retrieve information.Examining the pre-and post-test memory retention measurement, they found that the conversational agent group significantly outperformed the Google group on learning outcomes.This type of technology appears to facilitate interaction opportunities for online learners.Further, the use of conversational agents in a synchronous manner might prompt more in-depth investigation of the roles and effectiveness of synchronous interaction in online learning environments.
Still, the use of conversational agents in online courses as a synchronous interaction activity tool for both research and practice is in its infancy.Because few studies have examined conversational agents as a synchronous interaction medium in educational settings, there is little knowledge of what roles this type of interaction might play in online courses.

Research Questions
The purpose of this study is to examine learners' participation in online courses, synchronous interaction with a conversational agent, their relationships with learner performance, and identification of the factors that comprise participation/interaction. Our research questions are: 1. What are the relationships between the learner participation in an LMS for online courses, learner interaction with a conversational agent, and learner performance?2. What are the factors that comprise learner participation/interaction in online courses?

Methods
For this study, we adopted a quantitative single-case research design, and used correlation and factor analyses of learners' participation in an LMS, interaction with a conversational agent, and learner performance.The data for this study were collected from online courses at a mid-sized university located in the southern United States.Typically, a completely asynchronous model has been used for delivering the online courses via LMS.The courses were organized into 15 weekly themes and topic modules.The LMS content for the courses included a syllabus, announcements, reading assignments, supplementary reading materials, weekly discussion topics and questions, and related links.

Participants and Task
Fifty-six participants were recruited from four graduate courses in an instructional technology program.
The courses lasted 15 weeks, with an introduction module in the first week and a review and final paper 48 submission in the last week.Each of the 13 regular weekly modules included an asynchronous discussion based on each topic reading (e.g., journal paper, book chapter), conducted on an online forum in the LMS.
Students were required to complete the 13 weeks of asynchronous discussion activity throughout the course term.In addition, students in all courses were assigned interaction with a conversational virtual agent that was designed to encourage the acquisition of content knowledge and logical argumentation skills.Students were asked to interact with the agent about instructional topics using reflection prompts such as "Why do we need to use educational multimedia?", "Are mobile Apps good for teaching?","Peer review would be helpful?","In your project, was the ISD (Instructional Systems Design) Step useful or effective?",and so on.
Students were required to complete seven interaction sessions throughout the semester.

Conversational Agent
The conversational virtual agent system we used in this study was designed and developed to demonstrate the feasibility of the agent for better support of synchronous interaction in online courses (Song, Oh, & Rice, 2017).The conversational agent was an independent online application that was not embedded in the LMS; students could access the agent directly using a web browser, without logging into the LMS.Learners interacted with the agent through text-based chat.The virtual agent analyzed their input and replied to their questions and responses.As shown in Figure 1, the agent asks a question, and the learner answers the question.The agent initiates the question-answer interaction, for example, "Can you teach me the course, Educational Multimedia?"The agent also responds to the learner's answer; when the answer is short, the agent may ask "Would you please explain more about it?"

Data Collection
To examine learner participation, we collected participants' LMS log data, including the frequency of course access, length of course access, discussion board messages, and the final grade.We calculated the length of discussion board messages and rated their quality using procedures we will describe shortly.To examine synchronous learner interactions, we collected participants' conversation logs from the agent system, and also calculated their length and rated their quality.In total, we collected data on seven variables: System Access, Time Spent, Discussion Length, Discussion Quality, Conversation Length, Conversation Quality, and Final Grade.
Discussion and conversation quality measurement.We rated participants' discussion board messages and conversations with the agent using a scoring rubric.We used Bradley, Thom, Hayes, and Hay's (2008) adaptation of a coding scheme developed by Gilbert and Dabbagh (2005) to measure the quality of online discussion, and then modified it for our own context.We scored discussion board messages and conversations with the agent as follows:  Making Inferences (5 points): the participant demonstrates analysis, synthesis, or evaluation, and/or makes broader connections to society or culture.
An instructor from the participating courses and a research assistant separately coded participants' discussion board postings and conversations with the agent.Interrater reliability using the intraclass coefficient for the initial rating was .91.The raters resolved disagreements by consensus reached throughout five different discussion meetings.In the event the raters did not agree, the third author of this study was asked to consider each rater's justifications and make a final decision.

Results
We first calculated means and standard deviations of the seven variables (see Table 1).System Access and Time Spent had relatively large standard deviations, which means that there were large individual differences in learners' access to the LMS.

RQ2. What are the factors that comprise learner participation/interaction in online courses?
To identify participation/interaction factors underlying participants' behavior, we used a Principal Components Analysis (PCA).For the factor analysis, the sample-to-item ratio is important for determining the appropriate sample size.Because the final grade is an outcome of participation, we did not include it in the factor analysis, so our sample-to-item ratio is 9.3 (56 samples divided by 6 items), well within the recommended range of 5-20 samples per item (Pituch & Stevens, 2016).The correlation was investigated in research question 1 (see Table 2).The determinant is .275,which meets the requirement (i.e., greater than .000001)for the assumption of a factor analytic solution (Beavers et al., 2013).The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is .60,which is considered suitable (i.e., greater than .50)for factor analysis (Tabachnick & Fidell, 2013).Bartlett's test of sphericity is significant (p < .0001),which means that the variables are correlated highly enough to provide a reasonable basis for factor analysis (Hair, Black, Babin, Anderson, & Tatham, 2010).All the initial communalities are greater than .40,which means that the sample size is not likely to distort results (Tabachnick & Fidell, 2013).
The initial eigenvalues showed that the first factor explained 35.9% of the variance, and the second factor 22.6% of the variance.Eigenvalues of the third, fourth, fifth, and sixth factors were smaller than 1.No items were eliminated because there was no item that failed to meet the minimum criteria of having a primary factor loading of .4 or above and no cross-loading of .3 or above.Because the first component explained 35.9% of the total variance, which was less than 50%, we conducted a PCA of all items using Varimax with Kaiser normalization rotations to assess how six variables clustered.All items had primary loadings over .5.
Factors 1 and 2 were rotated, based on the eigenvalues-over-1 criterion and the scree plot.Table 3 displays the items and component loadings for the rotated components.After the rotation, the first component accounted for 31.0% of the variance, and the second component 27.5% of the variance.The two factors explain a total of 58.5% of the variance for the entire set of variables.

Discussion
In this study, the frequency and length of course access, the quantity and quality of asynchronous discussion, and the quality of synchronous conversation with a virtual agent were significantly associated with the learner achievement.Overall, the results of this study support previous research findings that established a relationship between learners' participation/interaction in online courses and their learning performance (Croxton, 2014;Stoessel et al., 2015;Wu et al., 2011).In our study, the quantity of conversation with the virtual agent was not significantly tied to achievement.One possible explanation, drawing upon Garrison and Cleveland-Innes's (2005) findings, is that learners' performance depended more on the quality of interaction than on the quantity.
It was not surprising that the frequency of course access was significantly correlated with the length of course access and the quantity of asynchronous discussion.Participants had to access the LMS in order to participate in the discussion forums, but not conversations with the virtual agent.In addition, the quality of discussion board posts and the quality of conversations with the agent were strongly related.It seems that the learners' depth of learning is fairly independent of interaction synchronicity.
Learner participation is a significant factor affecting success in online courses (Bettinger et al., 2016;Goggins & Xing, 2016;Hrastinski, 2008), and learner interaction is seen as central to online learning participation.The concept of interaction is multifaceted, and different types of interaction have different effects on learners' participation, satisfaction, and performance (Jung, Choi, Lim, & Leem, 2002).
Therefore, when aiming to promote learners' quality interaction for their active participation and enhanced learning performance, the confluence of different types and characteristics of interactional aspects should be taken into account.Our results indicate two factors that comprise learner participation/interaction in online courses.We called the first factor Interaction Quality, which includes the quality of discussion forum posts and conversation with the agent.We called the second factor LMS-oriented Interaction, which includes the frequency and length of course access, and the quantity of discussion forum posts.In our factor analysis, there was no clear discrepancy between synchronous and asynchronous interactions.Rather, interaction quality and LMS-oriented interaction emerged as the primary distinction among learner participation behaviors in online courses.Note that the quality of discussion board posts is included in the Interaction Quality factor, and the quantity thereof is included in LMS-oriented Interaction.This suggests that the same mode of interaction, in this case asynchronous communication, can have different roles in online learning.
Our findings also suggest that course topic-related communication between the learner and a virtual agent is practically applicable to online courses in a synchronous manner.Our participants successfully had a conversation with the virtual agent about course topics and materials.The instructional content-related communication with the conversational virtual agent might have a positive effect on learner performance and satisfaction, as previous studies showed that computer agent-based experiences offer the learner meaningful learning experience (e.g., Delialioglu & Yildirim, 2007;Garrison & Cleveland-Innes, 2005).We suggest this is because such interaction motivates the learner to express their opinions and encourages them to complete tasks.

Limitations
There are notable limitations in this study.First, our sample size was admittedly small.We found our sample size appropriate based on the context and the sample-to-item ratio; however, some would suggest that 50-100 is not a sufficient sample size (Comrey & Lee, 1992).Second, we were unable to perform courselevel analysis due to the small sample size.Thus, more in-depth factor analyses and course-level analyses with a larger sample size are required for future research.Third, learners' final grades might not represent learner performance precisely, specifically in graduate courses.Graduate courses encompass diverse aspects including legitimate argumentation, logical discussion, reasoning, academic writing, and critical 56 thinking that could not possibly be reflected in one specific score.We recommend future studies with the inclusion of different types of learner assessment methods and data.Fourth, a text-based chat might not fully serve as a synchronous interaction method.Just as people use language for human interaction, learners want to use spoken language to communicate even in online learning environments (Shawar & Atwell, 2007).Therefore, text-to-speech and voice recognition technologies might provide additional benefits, and verbal interaction with the agent should be investigated.Last, although our motivation in conducting the current study was seeking ways to remediate the higher attrition rate of online courses, we did not directly address retention.Outcomes such as course completion and attrition rates should be examined in future studies.

Directions for Future Research
The results of this study suggest that conversational virtual agents have potential for increasing meaningful interactions for learners in online courses.With the rapid expansion of computing power and artificial intelligence techniques, we can expect to see extensive application of these technologies in our daily lives, including in the field of education.It would be beneficial to the distance education field if we determined how best to incorporate intelligent agent systems into current online courses.Clearly, pedagogy using conversation agents in online learning environments deserves further investigation.
In this study, online learners experienced synchronous interactions with the conversational agent.
Nonetheless, it is uncertain that the conversational agent provides any types of social interaction or social presence for the online learner.Social presence is an important concept in online learning process that encompasses online communication and interaction (Tu, 2002).In online learning environments, it seems that learners' academic performance correlates with the perceptual level of social presence (Richardson & Swan, 2003).Because the learner can share their opinions and exchange critical ideas through social interaction, social presence might be associated with the level of learner interaction, which would establish a meaningful learning experience (Garrison & Cleveland-Innes, 2005).In addition, learner satisfaction could be tied more closely to learners' perceptions of their social and interpersonal interaction than to knowledge demonstration (Dennen, Aubteen Darabi, & Smith, 2007).What we have not examined in this study is the virtual agent system's ability to provide social and interpersonal interaction opportunities for the learner.The social and interpersonal aspects of conversational agents need to be further examined.
Learners need immediate feedback from the instructor and timely support from peers and subject matter experts, which could best be supported through synchronous interactions.Most learners would expect the instructor to be available at all times and to respond to their questions and requests (Drange, Sutherland, & Irons, 2015).Still, asynchronous discussion methods have pedagogical benefits such as supporting learners' writing processes and providing reflection time (Andresen, 2009).For these reasons, a combination of synchronous and asynchronous communication has been suggested to promote online learners' participation and engagement (Giesbers et al., 2014).As researchers have suggested (Beldarrain, 2006;Ohlund, Yu, Jannssch-Pennell, & Digangi, 2000), synchronous and asynchronous interaction might be connected in a complementary mutual relationship.The reflective and collaborative properties of asynchronous interaction might be well supplemented by the immediate and timely attributes of synchronous interaction.In addition, active interaction in synchronous tasks lead to positive interaction in asynchronous communication (Giesbers et al., 2014).This mutual relationship requires further investigation.Since this research area requires the analysis of a large amount of data, learning analytics and data mining techniques should be employed (Song, 2018).

Conclusion
We conducted this study to better grasp learner participation and interaction in online courses.Learner participation is not simply represented as a quantity of interaction or the access to the learning space.Our results offer a representation of how learner behavior indicators in online courses are associated with each other, but we need to further investigate different types of synchronous and asynchronous interaction in different types of online learning environments, specifically the use of conversational agent systems.

Figure 1 .
Figure 1.Screenshot of a learner interacting with the conversational agent system.

50
No Score (1 point): the participant's interaction is off-topic or incorrect;  Reading Citation or Content Clarification (2 points): the participant directly quotes from or paraphrases the reading;  Prior Knowledge (3 points): the participant uses prior knowledge from class or outside resources to support a statement or an understanding;  Real World/Abstract Example (4 points): the participant uses examples that demonstrate the application of knowledge to a real-world context and/or analogies/metaphors to support a statement or an understanding; and ) = .357,p = .01),Time Spent (r(54) = .297,p = .03),Discussion Length (r(54) = .423,p = .001),Discussion Quality (r(54) = .514,p < .001),and Conversation Quality (r(54) = .462,p < .001),but not with Conversation Length (r(54) = .133,p =.33).System Access has a correlation with Time Spent (r(54) = .286,p = .03)and Discussion Length (r(54) = .295,p = .03).In addition, there is a strong correlation between Discussion Quality and Conversation Quality (r(54) = .776,p < .001).The sample size and the correlation analysis are discussed in terms of the factor analysis in the limitation section.

Figure 2 Figure 2 .
Figure 2 shows how closely related the items are to each other and to the two components.Factor 1 was labeled Interaction Quality due to the high loadings by Discussion Quality and Conversation Quality.Factor 2 was labeled LMS-oriented Interaction due to the high loadings by System Access, Time Spent, and Discussion Length.Overall, our analyses indicated that two distinct factors were underlying learners' participation behaviors in online courses including synchronous and asynchronous interaction activities.

Table 2
Correlation of Learner Participation in OnlineCourses and Conversational Activity (N = 56)

Table 3
Component Loadings for the Rotated Components(N = 56)