A Comparison of Learner Intent and Behaviour in Live and Archived MOOCs

The advent of massive open online courses (MOOCs) has created opportunities for learning that are clearly in high demand, but the direction in which MOOCs should evolve to best meet the interests and needs of learners is less apparent. Motivated by our interest in whether there are potential and purpose for archived MOOCs to be used as learning resources beyond and between instructor-led live-sessions, we examined the use of a statistics MOOC and a computer science MOOC, both of which were made available as archived-courses after a live-session and for which enrolment continued to grow while archived. Using data collected from surveys of learner demographics and intent, the course database of major learner activity, and the detailed clickstream of all learner actions, we compared the demographics, intent, and behaviour of liveand archived-learners. We found that archived-learners were interested in the live-MOOC and that their patterns of use of course materials, such as the number and sequence of videos they watched, the number of assessments they completed, their demonstration of self-regulatory behaviour, and their rate of participation in the discussion forums, were similar to the live-learners. In addition, we found evidence of learners drawing on an archived-MOOC for use as reference material. Anticipated areas of impact of this work include implications for the future development of MOOCs as resources for self-study and professional development, and in support of learner success in other courses.


Article abstract
The advent of massive open online courses (MOOCs) has created opportunities for learning that are clearly in high demand, but the direction in which MOOCs should evolve to best meet the interests and needs of learners is less apparent. Motivated by our interest in whether there are potential and purpose for archived MOOCs to be used as learning resources beyond and between instructor-led live-sessions, we examined the use of a statistics MOOC and a computer science MOOC, both of which were made available as archived-courses after a live-session and for which enrolment continued to grow while archived. Using data collected from surveys of learner demographics and intent, the course database of major learner activity, and the detailed clickstream of all learner actions, we compared the demographics, intent, and behaviour of live-and archived-learners. We found that archived-learners were interested in the live-MOOC and that their patterns of use of course materials, such as the number and sequence of videos they watched, the number of assessments they completed, their demonstration of self-regulatory behaviour, and their rate of participation in the discussion forums, were similar to the live-learners. In addition, we found evidence of learners drawing on an archived-MOOC for use as reference material. Anticipated areas of impact of this work include implications for the future development of MOOCs as resources for self-study and professional development, and in support of learner success in other courses.

Introduction
Massive open online courses (MOOCs) can provide flexible enrolment options for learners beyond their registered cohort as these courses are often left accessible as selfstudy after the end of a given session. In this archived-mode, learners continue to be able to access the course materials, although deadlines and the opportunity to earn a credential have passed. In Coursera, for example, about 78% of courses that have finished at least one session on the platform have a session in archived mode (C. Gao, personal communication, November 26, 2013). In this study, we compare the use of two MOOCs as in-session, instructor-led ("live") courses with their use as post-session, selfdirected ("archived") learning resources. The two MOOCs studied here are "Statistics: Making Sense of Data" (STATS) and "Learn to Program: The Fundamentals" (LTP1), offered on Coursera with live sessions in April-May 2013 and September-November 2012, respectively, and available afterwards as archived-MOOCs. When each of these MOOCs ended, many learners persisted and new registrants continued to join the archived courses. Figure 1 shows the number of learners who registered before, during, and after the live sessions of the two courses. We also look at a follow-up course to LTP1, "Learn to Program: Crafting Quality Code" (LTP2), to understand how students transition among related live-and archived-MOOCs.

Purpose of the Study
The purpose of this study is to compare the characteristics of learners and their interactions with two MOOCs during live-sessions and afterwards when the MOOCs are archived. Live-and archived-MOOCs are distinguished by the presence or absence of instructional support, cohort presence, deadlines, and the potential for formal acknowledgement of completion. To this end, we investigated live-and archivedlearners' demographics, intent, and the relationship between intended and actual behaviour, including the amount and nature of interaction with the course materials.
Our goal in this observational study was to examine learner characteristics and behaviour given the MOOC formats that were available to the learners, allowing us to better understand both who chooses to use the course materials in each format and how they engage with the materials.
Research on MOOC learners has recognized the effect of learner intent on the amount and pattern of MOOC component use (Kizilcec, Piece, & Schneider, 2013). Studying patterns of behaviour of learners who have no chance of earning a Statement of Accomplishment (SoA) or some other formal acknowledgement of completion extends the current state of MOOC research. It also distinguishes this study from the dominant discourse of MOOC research, which is focused on low completion rates and the behaviour and characteristics of learners who do and do not earn an SoA (e.g., Seaton, Bergner, Chuang, Mitros, & Pritchard, 2014).
The purpose of this work is not to critique MOOCs, nor address their known shortcomings such as high attrition rates (Adamopoulos, 2013;Catropa, 2013). We also acknowledge the challenges involved in assessing learning achievement of informal learners (Levenberg & Caspi, 2010), let alone archived-learners. Rather, we aim to understand the similarities and differences between the learning paths of live-learners and archived-learners. are guided through the course at the pace at which the materials are released. An instructional team and cohort provide learning support through the discussion forums.

Definition of Terms
Coursework has deadlines and some coursework includes peer assessment.
Archived-MOOC: Post live-session, self-directed course with minimal or no instructional support or cohort presence, no deadlines, no peer-assessment, and no opportunity to earn an SoA. All materials are available on registration, giving learners more flexibility relative to live-MOOCs in the pace and order in which they access the course materials.

Live-learners:
Learners who register in a live-MOOC in time to earn an SoA.
Archived-learners: Learners who are active in an archived-MOOC.
For the purposes of this study, learners are characterized by whether they are accessing the particular live-and archived-MOOCs under consideration, in the form in which these particular MOOCs were available at the time, without consideration to the status of the course when the learners enrolled and how they might be using other MOOCs in which they may be enrolled. In addition, in this study we do not consider other MOOC formats with flexible start dates and pacing such as MOOCs designed for self-study.

Theoretical Framework
Increasing chances for on-demand learning, in the light of intensive interest in life-long learning, is a significant promise and potential of MOOCs (Garrison, 2011). Learners are able to actively choose among available MOOCs to address their professional and learning needs or to pursue personal interests, notwithstanding temporal, geographical, or institutional barriers (Adamopoulos, 2013). Such on-demand learning (Dobrovolny, 2006;Rhode, 2009) is often informal, self-paced, and self-directed online learning. We first examine characteristics of MOOC learning in the light of the aforementioned learning modalities. Then, we examine factors known to impact the quality of learning in this context.
Alongside institution-affiliated online learning, there exist other modalities of online learning that provide to learners greater freedom of choice in duration, content, and modes of assessment (Levenberg & Caspi, 2010). For example, in a self-paced learning mode, students can choose the start date of their course and complete it according to their own time schedule (Anderson, Poellhuber, & McKerlich, 2010;Horton, 2006).
Self-paced courses, however, are usually bound by a deadline to finish all course activities and involve some level of real-time or asynchronous instructional support (Gerlich, Mills, & Sollosy, 2009 Sheu, Lee, Bonk, & Kou, 2013). Live-MOOCs provide an opportunity for informal online learning that can be instructor-led and to some degree self-paced, as learners may often have to stick to deadlines if they strive to earn an SoA and cannot explore course content that is not released yet. Live-learners can also engage in peer interaction, with a defined cohort of students working through the material at the same pace. Archived-MOOCs, on the other hand, fall into the extreme end of the self-paced learning continuum (Lowenthal, Wilson, & Parrish, 2009), as archived-learners have no deadlines to meet and little chance of interacting with course instructors. While interaction with peers is still possible, there is not the same large cohort studying the same material at the same time. Learners can select the material to cover and take as long as they need.

Self-regulation.
Informal learning, such as learning in MOOCs, demands a high level of self-directedness from the learner as they are in charge of their own progress. Self-regulation (Pintrich, 2000;Zimmerman, 2008) explains how learners manage their learning by actively setting goals, planning to achieve their goals, identifying and using resources, monitoring their progress, and using self-corrective measures. We use self-regulation to explain qualities that learners need to develop in order to engage with and persist in informal, non-credit, yet structured, learning environments of MOOCs. Dobrovolny (2006) studied self-paced corporate learners' use of self-regulatory processes using verbal and visual think-aloud strategies. The participants would refer to course material to resolve misunderstandings and confusions, demonstrating self-assessment and reflective strategies. In MOOCs, following each of thousands of learners' self-regulatory processes is not feasible, and it is impossible to provide scaffolding to meet each learner's unique needs. However, clickstream data might provide evidence of learners' self-corrective behaviours, such as accessing relevant resources or posting in discussion forums between repeated attempts at formative assessments.

Peer collaboration and interaction.
While social presence and its necessity for knowledge co-construction is favoured in online learning community frameworks, such as the community of inquiry (Garrison, Anderson, & Archer, 2000), such interaction among learners in self-paced contexts may be of less importance to their learning outcome. In an exploratory study guided by Anderson's (2003) interaction equivalency theorem, Rhode (2009)

Course content.
Learners' depth of learning is necessarily affected by their persistence with the course.
Using text-and opinion-mining methods, Adamopoulos (2013) analyzed 1,163 reviews submitted by 842 learners who had taken at least one MOOC in various disciplines to investigate factors associated with learner retention. While learners' satisfaction with the course material was positively associated with their completion of the course, courses with higher workload and longer duration had greater risk for learner attrition.
The importance of the quality of course material on perceived learning, specifically over peer interaction, has also been observed (Rhode, 2009;Schwier, Morrison, Daniel, & Koroluk, 2009). And in a journalism MOOC, 50% and 40% of learners rated course readings and videos, respectively, as being the most helpful learning resources, with only 6% of learners identifying discussion forums as a useful learning resource (Liu, Kang, Cao, Lim, Ko, & Weiss, 2013).

Role of instructor.
In addition to content, the presence of instructional support may influence self-paced, self-directed learners' experiences, but instructional support may be deemed less essential than high quality content (Rhode, 2009 instructors were found to be the most important factor in learner retention (Adamopoulos, 2013), but this study does not provide detail on the aspects of the instructors' role that foster retention. The amount of reliance on and interest in instructor interaction may also depend on learners' goals and motivation (del Valle & Duffy, 2009). Considering the large enrollments in MOOCs, high volumes of instructor interaction and feedback may not be feasible or as essential as in formal online courses (Hosler & Arend, 2012;Skramstad, Schlosser, & Orellana, 2012;Sheridan & Kelly, 2010). Archived-MOOCs offering high quality content can be of value to self-paced and self-directed learners, since these learners are less reliant on instructors.

Context of the Study
The learners studied in this research were enrolled in courses that teach practical, skillsbased subjects, and for both courses the prerequisite was only high school level mathematics. Moreover, the concepts and skills covered are useful for a variety of professions and fields of study.
We briefly describe the two courses, offered on the Coursera platform, that were used as cases for this study. We study learners in the live-and archived-sessions of STATS. Because of the timing of the conception of this work, we study learners in two sessions of LTP1: archivedlearners in the first-session and live-learners in the second-session. We also investigate the activity of LTP2 live-learners who accessed the archive of LTP1, to further understand how learners make use of archived-MOOCs.

Data Sources
Data corpus was collected through the Coursera platform and included the following.

Live-survey:
A pre-course survey of live-learners included close-ended questions on demographics, reasons for enrolment, intended time investment, amount of videos and assessments they intended to complete, previous knowledge, and MOOC experience.

Archived-survey:
Archived-learners were asked to complete a survey similar to the live-survey with additional questions including why they took the course in its archivedmode and whether they would retake the course in live-mode.

Coursera database:
The database contains records of videos accessed, assessments submitted, and posts to the discussion forums.

Clickstream:
The clickstream includes a log of all user activity.
All data were anonymized at the institutional level before distribution to the researchers.

Analysis
We used descriptive statistics to better understand the characteristics of live-and archived-learners using the survey data. Data from the clickstream were used to identify learners' patterns of use of the course components. The database of each course provided additional evidence to corroborate and complement this analysis.
Anonymized user identifiers provided a map across all data sources, allowing us to connect user behaviour characterized in the clickstream and database with measures of intent captured in the surveys.

Results
We report on learner demographics, intent, and behaviour. Unless otherwise stated, the data presented include only those live-and archived-learners who completed the live- Since the experience of archived-learners is of central focus to this research, we examined the reasons why learners chose to enroll in the archived-courses. Figure 2 shows the percentage of survey respondents who selected each possible response as a

Time learners planned to spend.
For STATS, live-learners planned to devote more hours per week to the course than archived-learners (median of 5 hours for the live-learners and 2 hours for the archivedlearners). Live-and archived-learners in LTP1 intended to devote a similar number of hours per week to the course (median of 5 hours for both the live-and archivedlearners).

Work learners planned to complete.
Learners were asked how much work they planned to do for the course. We have classified their response choices into the following categories.
All required: The learner indicated that he or she planned to complete all requirements, including watching all videos, and completing all assessments.

Most:
The learner indicated that he or she planned to watch most videos and complete some assessments.
Not sure: The learner indicated that he or she was unsure.

Some:
The learner indicated that he or she planned to watch some videos, perhaps on targeted topics, but was unlikely to complete assessments. all requirements may be a reflection of the fact that the peer-assessments could not be completed in archived-mode.

Learner Behaviour
Overview. Table 2 provides summary statistics about the behaviour of learners in the live-and archived-MOOCs, as characterized by how many videos they accessed, assessments they attempted, and the time between first and last video access.
The average number of required videos accessed is similar for live-and archivedlearners in STATS, but LTP1 live-learners watched, on average, about two more videos than archived-learners. While live-learners complete slightly more assessments than archived-learners, the difference is small. The optional videos in STATS are tutorials in the R statistical software and programming language. Archived-learners watch, on average, approximately one more of these videos than live-learners.
We defined access time as the number of days between the first and last access of any video. In this analysis we are only considering archived-learners who had been enrolled for at least the length of the live-MOOC. For both STATS and LTP1, over 50% of learners accessed videos for at most 10 days in both the live-and archived-courses.
However, for the learners in the top quartile of video access times, more archivedlearners than live-learners had long access times. Archived-learners in the 90 th percentile accessed videos over a period of 150 days or more, while for live-learners the 90 th percentile was approximately 50 days. Thus even the live-learners who had the longest access times tended to not access the MOOCs after their formal conclusion, while some archived-learners access the learning resources over long periods of time.
Note that, because the last day of access was necessarily constrained by the date on which the clickstream was extracted, some larger access times are censored, possibly underestimating the 90 th percentile for archived-learners.

Sequencing of videos watched.
Archived-learners who enroll after the live-MOOC has ended have immediate access to all lecture videos. As a result, they have much greater opportunity than live-learners to explore the course in a non-sequential fashion, perhaps picking and choosing topics that are of interest to them.
To investigate this behaviour, we model a learner's transition from video to video using a first-order Markov Chain. The statistics of interest are captured in a video-by-video matrix; rows are the last video watched, columns are the next video watched, and the entry is the estimated probability that a learner will make this transition. Since we are interested in transitions between videos rather than rewatches of the same video, we exclude self-transitions. In STATS, it was very common for both live-and archived-learners to follow the intended sequence, as indicated in the strong pattern from upper-left to lower-right, one to the right of the diagonal. Figures 3 and 4 illustrate the transition matrices for learners whose intent was categorised as all required and most. The strong sequential pattern in video transitions exists regardless of intent.
For LTP1 learners who indicated they intended to do all of the required work for the course, both live-and archived-learners also tend to watch the videos in sequence ( Figure 5). However, for learners who intended to do less than all of the work, video transitions were more sequential in the live-course than in the archived-course. In Figure 6 we illustrate the matrix for the learners who responded in the category most for intent; a similar pattern was observed for learners who intended to do some or who indicated that they were not sure.

Discussion forum interactions.
We also investigated discussion forum activity for live-and archived-learners. More of the live-learners (44.6% for STATS; 41.8% for LTP1) view threads than archivedlearners (31.3% for STATS; 37.3% for LTP1). Of those who do view threads, the mean number of views for STATS live-learners was 10.9 and archived-learners was 14.5, whereas the mean for both live-and archived-learners in LTP1 was 19.6.
As is typical of MOOCs, the number of learners who post or comment on the discussion forums is low for both live-and archived-learners (DeBoer, Ho, Stump, & Breslow, 2014). In both STATS and LTP1, more live-learners (12.1% for STATS and 14.5% for LTP1) posted than archived-learners (9.3% for STATS and 13.2% for LTP1). It is interesting to note that archived-learners did post to the forums, even though the courses were not active. The mean number of posts by those archived-learners who do post was 5.1 for STATS and 3.9 for LTP1.

Activity patterns: activity between reattempts of an assessment.
In both STATS and LTP1, learners were allowed multiple attempts at the quizzes and the maximum grade achieved counted towards their course assessment. As an investigation of the relative evidence for self-regulatory activity in the live-versus archived-courses, we examined the frequency of use of MOOC materials, in particular lecture videos and forums, between repeated attempts at machine-graded quizzes.
In Figure 10, we see that a greater percentage of archived-learners in both courses accessed lecture videos between quiz reattempts, although this behaviour is less evident for the later quizzes. As can be seen in Figure 11, archived-learners use the forums as a resource for self-regulated learning at least as much as learners in the live-course.
Although not shown here, no distinguishing patterns were observed among learners' varying levels of intent.   Figure 11. Percentage of learners who reattempted quizzes who accessed discussion forums between reattempts.

Activity patterns: LTP2 learners active in LTP1.
LTP2 was a sequel to LTP1, with the content from LTP1 a presumed prerequisite.
During the period when LTP2 was live, LTP1 was available as an archived course, and thus was accessible reference material for learners enrolled in the live offering of LTP2. Here we report on the activity in the archived-LTP1 of those 2,192 LTP2 live-learners.
We are only reporting on activity of these learners while LTP2 was live, so the access time available is less than that for the general LTP1 archived-learner population. It appears that watching videos was the primary reason for using the LTP1 archived course, with less interest among these learners in accessing assignments and discussion forums.
Almost all of these learners visited LTP1 to view videos, with 93.1% accessing at least one LTP1 video, and, on average, 9.4 videos accessed. Only 3.7% accessed all of the LTP1 videos. Relative to the general population of LTP1 archived-learners (see Table 2), the LTP2 live-learners access an average of 4.1 fewer videos.
Fewer LTP2 live-learners who concurrently accessed LTP1 completed LTP1 assessments. 73.7% did not submit any quizzes and 92.5% did not submit any assignments. Only 0.8% completed both all seven quizzes and all three exercises.
The LTP1 forums were not a popular resource for the LTP2 live-learners, with over 75% of them viewing either no threads or a single thread once.

Discussion
MOOCs commonly have defined start and end dates for a cohort but remain open after the end date with learners continuing to enroll. Can these archived courses meet learners' needs? Our goal in this research was to examine learner characteristics and behaviour in live-and archived-MOOCs. We found more similarities than differences, with indications that archived-learners interact with the course in much the same way as live-learners. These similarities are consistent with the top reasons why learners used the archived course materials, which were because they arrived too late in the livecourse or they were unable to complete the course during the live-session. Since this is an observational study, we cannot attribute differences between learners and their behaviours to the differences between live-and archived-MOOCs.
Previous research on self-directed learning in MOOCs (Kizilcec, Piece, & Schneider, 2013) and other settings (de Valle, & Duffy, 2009) has stressed the connection of learners' intent to their level of engagement with learning and assessment resources.
Here we took a step towards understanding this relationship by including in our analysis learners who would not be acknowledged externally for their learning effort such as by earning an SoA. Although not as many as for the live-courses, significant proportions of the archived-learners indicated that they planned to complete all required work. Even though fewer archived-learners indicated that they intended to A common topic in the MOOC literature is retention (Breslow et al., 2009;Chen et al., 2012). The existence of instructional support and a peer cohort created a social structure that we thought might positively impact retention in the live-MOOCs. However, archived-learners achieved similar progress, watching a comparable number of videos and displaying a similar pattern in the particular assessments that were attempted. In the archived-MOOC, the flexible pace may have been a contributing factor to this retention, as also seen by Gooding et al. (2013). Yet there is tension between providing the flexibility of archived-MOOCs and the strong social support structure of live-MOOCs, illustrating the potential for continued improvement of MOOC formats.
An important component of live-MOOCs is the online discussion forum. In the archived-MOOCs, despite the lack of instructional and reduced cohort presence on the forums, there was still extensive discussion forum use. Bruff et al. (2013) found that an on-campus cohort of learners using an archived-MOOC viewed forums posts, but few reported posting to the forums themselves. Instead, they chose to ask questions locally.
Our archived-learners, in the absence of a recognizable cohort, both viewed discussion forums posts and posted to the forums.
Although the archived questions and answers were not recently posted, they remained a valuable resource for archived-learners. These forums became another medium for content delivery, rather than an opportunity for social interaction. Existing research (Anderson, Poellhuber, & McKerlich, 2010;Rhode, 2009;Schwier, et al., 2009) has shown that learners value course content over peer and instructor interaction. For our archived-learners, less peer interaction and a minimal chance of instructor interaction did not generally deter them from covering their intended content. Their reasons for and the extent of their desired interaction with instructor and peers, however, remain open questions that are beyond the scope of this study. An investigation into these questions would inform the potential development of a new modality for self-directed, on-demand learning that combines the self-paced structure of an archived-MOOC with the desired instructional support structure.
Live-learners had the opportunity to earn an SoA and all learners had the opportunity to re-take quizzes to demonstrate mastery. In both groups of learners, indications of selfregulatory behaviour were observed in remedial action taken between repeated attempts at quizzes. Archived-learners used the lecture videos and the discussion forums as resources for self-regulated learning at least as much as learners in the live-course, even though there was no external reward for improved results. With all course materials immediately available on registration, archived-learners have the opportunity to view the videos in the order of their choosing, rather than being limited by the release of materials at regular intervals. We had hypothesized that archived-learners may be more likely to explore the course in a non-sequential fashion, picking and choosing topics that were of most interest to them. However, for archivedlearners who intended to complete all required work, the sequence of videos accessed closely matched the sequence that the instructor intended. Thus learners are treating the archived-MOOC as a traditional course, rather than as a learning resource they might access as needed. However, the use of archived-LTP1 by LTP2 live-learners illustrates that there is potential for archived-MOOCs to be used as reference material as well.
Archived-learners have more flexibility, not only in terms of access to content but in the pace at which they complete the course. Additional exploration of the pace at which archived-learners access videos and complete assessments may have valuable implications for course design.
In this study, we investigated two MOOCs, both of which had live-sessions followed by a period of time during which the MOOCs remained available as archives of the livesessions. Archived-learners are interacting with the courses in much the same way as live-learners. They succeed at the same rate as live-learners, with minimal guidance and no obvious cohort. There is potential for MOOCs to be beneficial as self-study courses, and for the development of new modalities that combine the most valued aspects of liveand archived-MOOCs to best meet learner needs and interests.