MOOCs: a systematic study of the published literature 2008-2012

Massive open online courses (MOOCs) are a recent addition to the range of online learning options. Since 2008, MOOCs have been run by a variety of public and elite universities, especially in North America. Many academics have taken interest in MOOCs recognising the potential to deliver education around the globe on an unprecedented scale; some of these academics are taking a research-oriented perspective and academic papers describing their research are starting to appear in the traditional media of peer reviewed publications. This paper presents a systematic review of the published MOOC literature (2008-2012): Forty-five peer reviewed papers are identified through journals, database searches, searching the Web, and chaining from known sources to form the base for this review. We believe this is the first effort to systematically review literature relating to MOOCs, a fairly recent but massively popular phenomenon with a global reach. The review categorises the literature into eight different areas of interest, introductory, concept, case studies, educational theory, technology, participant focussed, provider focussed, and other, while also providing quantitative analysis of publications according to publication type, year of publication, and contributors. Future research directions guided by gaps in the literature are explored.


Introduction
Distance education has a long history, with correspondence courses making use of reasonable cost universal postal services for the delivery of study material to learners and for submission/return of assignments by/to students (Casey, 2008).Further developments of distance education have appeared with each new communication technology: radio, television, video recorders, home computing.The latest development, that of the Internet (including very recently the mobile Internet), has similarly been adopted by many existing higher education providers but has also supported the emergence of a new model dubbed a massive open online course (MOOC(s)), the term coined in 2008 to describe an open online course to be offered by the University of Manitoba in Canada.A range of both topics and platforms have since emerged and the term was described as "the educational buzzword of 2012" by Daniel (2012) reflecting widespread interest in the concept.MOOCs are widely discussed across a range of media, including blogs and the specialist and popular press, however this includes "thinly disguised promotional material by commercial interests … and articles by practitioners whose perspective is their own MOOC courses" according to Daniel (2012).
This paper seeks to classify academic research relating to MOOCs, based on a systematic review of the existing peer reviewed MOOC literature.Search techniques for papers related to MOOCs are considered and a corpus of papers identified, then a grounded research approach is presented from which a classification of the works emerges.

Background
Since the early days of computing, academics have shared digital content (Lane & McAndrew, 2010) and recently there has been much interest in the sharing of open educational resources (OER), particularly relating to higher education, which has also become an important resource base for teachers and learners (Adams, Liyanagunawardena, Rassool, & Williams, 2013).In 2001 the Massachusetts Institute of Technology (MIT) launched its pioneering OpenCourseWare (OCW), with the aim of publishing materials from all its courses permanently on the open Web, with licenses allowing its use, modification, and redistribution.Since then many other established universities have joined the movement such as the Open University of UK through the OpenLearn project and the Open Learning Initiative by Carnegie Mellon University.OER were made available for two purposes: Learners could access the material directly and, hopefully, learn from it; educators could use the material as part of their own teaching (as produced or by amending it themselves).A significant proportion of these OER, however, were of limited use since they were usually produced in order to be a specific part of a larger educational experience within a specific educational framework.This limitation was particularly frustrating for many aspiring learners attempting to use them directly, but could also cause problems when used naively by educators (Liyanagunawardena, 2012;Weller, 2007).The concept of open access to learning was taken in a different direction with the introduction of the massive open online courses or MOOCs (Fini, 2009).
A MOOC brings together people interested in learning (or "students") and an expert or experts who seek to facilitate the learning.Connectivity is usually provided through social networking, and a set of freely accessible online resources provides the content or the study material.Furthermore, they generally have no prerequisites, fees, formal accreditation, or predefined required level of participation (McAuley, Stewart, Siemens, & Cormier, 2010).Participation in a MOOC is completely voluntary and is dependent on the interested individual.The collaborative space of a MOOC can span across many different platforms and technologies.For example, MOOC participants may create their own blog posts discussing aspects of the MOOC in different spaces and/or may use microblogs such as Twitter to express themselves."Connectivism and Connective Knowledge", an online course offered through the Learning Technologies Centre and Extended Education at the University of Manitoba and facilitated by George Siemens and Stephen Downes (Downes, 2008), is considered the first MOOC.This online course had 25 paid enrolments (for credit) with around 2,200 non-credit, non-fee paying students.It used the principles of connectivism (Siemens, 2005) and unlike the traditional form of online learning was not primarily relying on resources posted through a learning management system (LMS) 1 .
With the increasing uptake and interest in MOOCs, it has become a popular topic in the educational press such as the Time Higher Education Magazine article (Corbyn, 2012), which presents an account of running a MOOC.Likewise there are many blog posts relating to MOOCs, posted from a variety of viewpoints including course leaders, participants, and outsiders (for example http://mooctalk.orgchronicles the experience of a mathematician leading a MOOC in late 2012).
Academic papers on MOOCs began to appear in the peer reviewed literature (such as journals, conference proceedings, and professional magazines) in 2008 (Downes, 2008), with an increasing number of papers appearing each year since.This phenomenon is similar to many other technologies that created interest in academic communities.For example, Twitter appeared as a micro-blogging tool in 2006 and the literature on Twitter has grown exponentially from three papers in 2007 to hundreds of papers in 2011 (Williams, Terras, & Warwick, 2013).

Method Data Collection
Researchers use different methods to identify papers to consider for a literature review (Ellis 1989;Ellis & Haugan, 1997): Methods include searching in databases or search engines and chaining from known research papers.For a systematic review it is important that the methods of identifying papers are described and justified, and that the approach can be re-applied by others (Fink, 2010).The aim of this study was to locate and analyse MOOC related academic literature to provide an understanding of developing research areas, methods applied in research, and topics lacking published research.
Relevant papers were identified through a series of search efforts, using an approach based on the methods used in other systematic reviews including two studies of literature related to the microblogging system Twitter (Gao, Luo, & Zhang, 2012;Williams, Terras, & Warwick, 2013).Papers were classed as relevant if their primary focus was to explore the concept of a MOOC or the implications for higher education, report on experiments with MOOCs, or compare MOOCs with other educational approaches.There was insufficient statistical data across the papers found to undertake a meta-analysis (Fink 2010).
Firstly, the search terms and boundaries to be used were established.Initially two search terms (and their plurals) were selected: MOOC Massively Open Online Course However it was identified that some authors used "Massive" instead of "Massively" (for example Kop, Fournier, & Mak 2011) and so the third term (and its plural) was added:

Massive Open Online Course
The search period was limited to the period from the year in which the first MOOC was run ( 2008) to the year this study started (2012).Where possible the search was limited to titles and abstracts to reduce the workload of manual filtering of irrelevant papers.
Secondly, following the approach of Gao, et al. ( 2012), a number of academic journals in the disciplines of educational technology and distance education were selected for a preliminary search (Table 1).These journals were The British Journal of Educational Technology, Distance Education, American Journal of Distance Education, and Journal of Online Learning and Teaching.The search returned a small number of papers in one of these journals, but on detailed examination none of these were relevant for this systematic review.For example Roderick (2008) was predominantly about connected learning in digital spaces, and massively multiplayer online role-playing games were discussed as an example of connectivity.We believe that even though the search terms were used within inverted commas as phrases, the search algorithm may have returned articles such as this by comparing single search terms.
Thirdly, the same search terms were then used to search various academic databases: ISI Web of Knowledge, ProQuest (ERIC, British Education Index and Australian Education Index), JSTOR (education titles), IEEEXplorer, and Scopus (Table 2).A number of the articles returned were unrelated to this study, for example papers returned included some on "Multiple Optical Orthogonal Code Sequences" and some on "Management of Organizational Change" 2 , both of which were abbreviated to MOOC.
Fourthly, the same search was conducted using the Google Scholar search engine with 94 results (November 21, 2012).From these results only 33 were found to be relevant.However, the same search repeated on Google Scholar later (on January 08, 2013) only resulted in 25 results suggesting that some articles previously found were not online and/or Google had changed its search algorithms.For example, the first result returned by the search done on the 21st November 2012 was the article by Fini (2009); but the recent search that returned only 25 results did not contain this article.Presentations and papers that did not discuss MOOCs as the primary concern of the paper were discounted after a careful reading of the abstract.
In order to improve the coverage of relevant publications to be included in this review, the chaining technique of Gao, et al. (2012) was used: consulting the reference lists of papers that were already in the corpus to locate other relevant work.This resulted in the addition of one more article to the corpus.Google Scholar searches were also performed for each of the articles in the corpus to identify other articles that had referenced them (forward referencing).This resulted in the addition of one more conference paper.During December 2012 two additional papers were identified: one in the Communications of the ACM and another in the Journal of Interactive Media in Education.
The data collection process resulted in the identification of 45 distinct articles -17 from journals, 13 conference publications (including one poster conference presentation), 10 academic magazine articles, 3 reports, and 2 workshop presentations.

Data Classification/Analysis
Articles were classified both quantitatively and qualitatively.The quantitative analysis was used to classify the papers according to the publication year and the type of publication in which the article appeared.Papers were qualitatively classified using open coded content analysis, a technique used by the two studies of literature relating to Twitter (Gao, et al., 2012;Williams, Terras, & Warwick 2013).Initially the first author read each of the papers to identify themes, types of inquiry employed, and the future research directions indicated.These classifications were refined in an iterative manner by all authors reading and re-reading articles, and considering alternative classifications and stratifications.

Quantitative Details
The first MOOC related paper was published in 2008, with again just one paper identified in 2009, seven papers in 2010, 10 in 2011, and 26 in 2012.Clearly an increasing pattern in the number of articles published on MOOCs can be seen from Figure 1.
The majority of identified articles published to date are in journals (17 papers), with a smaller number of articles appearing in conference proceedings and magazines (13 and 10 respectively).Figure 2 diagrammatically presents the percentage of papers in each type of publication.A list of authors with more than one article being reviewed here are presented in Table 3. Kop has the highest number of publications (five) in the review while deWaard, Hogue, Koutropoulos, and Rodriguez have four each.It can be seen that the authors with more than three publications have worked in teams and several such teams can be identified from co-authorship, for example Kop and Fournier; de Waard, Abajian, Gallagher, Hogue, Keskin, Koutropoulos and Rodriguez; Mackness, Mak, and Williams.

Publication Topics/Themes
The articles embraced a wide-range of themes relating to MOOCs; an initial list included 3 : agency connectivism actor network theory dangers learner experience pedagogies technology trends Most papers included some introduction to the term MOOC and a history of the concept.The majority of articles were primarily concerned with the concept of MOOCs, discussing challenges and trends, while other themes generally appeared within only one paper except for the concept of connectivism and its implications.The themes were re-stratified into the following categories.
1. Introductory: explaining aspects of MOOCs.2. Concept: encompassing discussion papers on topics such as the threats and opportunities of MOOCs for Higher Education and its existing institutions.3. Case studies: examining one or more MOOCs (including papers studying the same course running in different years and papers studying different courses).4. Educational theory: considering the pedagogic approaches used. 5. Technology: presenting details or consideration of the software and hardware used.6. Participant focussed: considering aspects related to the learners participating in MOOCs.
7. Provider focussed: considering aspects related to the provider of the MOOC, including the course creators and leaders.8. Other: this category was introduced to cover the two articles that did not come under any other category.These were Esposito (2012), which discussed ethical issues in using data generated through a MOOC, and Frank (2012), which presented the author's views on alternative ways to run MOOCs.
The articles were then individually re-considered and assigned to the categories which reflected the contents; many articles were assigned to more than one category.Table 4 shows the spread of these categories.

Case Studies
Across the articles classified, 21 had a case study element, and in total 13 different MOOCs were studied.5 shows the number of articles studying each MOOC and the number of articles studying each as the single central case study.The related MOOCs (PLENK2010 and CCK08) were the most studied.MobiMOOC is the only other course to be considered in more than one paper.

CCK08 -Connectivism and Connective
The majority of case studies have used multiple methods for data collection in line with the general practice of case study research (Yin, 2003).In most case studies online surveys were used to collect data from participants in the MOOC(s).SurveyMonkey (www.surveymonkey.com) and LimeSurvey (www.limesurvey.org)were popular tools used in survey data collection.Four studies considering PLENK2010 as a case used multiple surveys (active participant survey, lurker survey, and end course survey).Researchers also reported collecting data via email interviews, focus groups, Moodle log data, discussion forum data, blogs, and observations.Three articles reported the use of techniques based on virtual ethnography and another used narrative inquiry as their research approach.
Seven of the case study articles defined the researcher's role in relation to the study.In two of these instances the researcher was described as a learner in the MOOC, in another two as a participant, and in the remaining three as an observer.The demarcation (if one really exists) between a learner and a participant in a MOOC is not clear, and may well simply be a matter of differing nomenclature.

Participants.
The majority of the participants in the case study research were people who had participated in the MOOC under consideration.Two articles also included data collected from instructors.

Limitations
The aim of this study is to develop a classification of academic research relating to MOOCs, based on a systematic consideration of the existing peer reviewed MOOC literature.Other materials such as blog posts and unpublished reports are not included in this review.Blog posts always present difficulties for authors of reviews in determining the credibility of the posts: There are few studies on how researchers use blogs (Shema, Bar-Ilan, & Thelwall, 2012); the transient nature of blogs means that they are difficult to suitably include in a systematic consideration.It should be noted that the same search terms applied above used in Google Web search returned over 50,000 items and used in a Google Blog search resulted in 570 results (December 04, 2012), however these items are not analysed here.Daniel (2012) includes references to such resources.Inclusion of such resources in a systematic review is contentious as they are often highly subjective and have rarely been critically reviewed by peers.The question of whether such resources should be included in a systematic review needs further methodological consideration in the literature, but at present it is unusual due to issues of credibility.
Articles published in languages other than English were not considered for this review.When searching forward references using Google Scholar, the authors came across publications that referenced articles in the review but were published in other languages such as Chinese.Because of this, the authors believe that there could be articles that would have been included in the review were they published in English.

Discussion
This analysis seems to reflect the growing interest in MOOCs.However, most research has investigated the learner perspective, with a significant minor focus on the institutional threats and opportunities.The lack of published research on MOOC facilitators' experience and practices leaves a significant gap in the literature.Mak, et al. (2010) suggest that there has been unacceptable behaviour (for example, forceful intellectual debates, feelings of participation being demanded, and rude behaviour) from some MOOC participants, which has led other participants to cease posting on forums.The possible cultural differences of participants in MOOCs and their MOOC experience would be an interesting avenue of research in relation to cultural tension in MOOCs.Many studies that used data generated from MOOCs have limited their qualitative analysis to posting on a formal LMS and a sample of other sources (such as blog posts) due to the volume of data (Kop, 2011;Kop, & Fournier, 2010).Inevitably this leaves out potentially useful data on Twitter, blogs, and other social media spaces.For example, Liyanagunawardena (2012) shows that there is a significant use of external communications (other than the official LMS discussions) to support learner groups in traditional, blended, and online distance education settings.
Despite many research studies using publicly available data from a MOOC for research purposes, only a few papers have considered the ethical aspects of such use.This may well be due to a lack of good guidance and best practice examples on the general question of the use of data from online social media and similar sources for such research (Zimmer, 2010).This opens up a new avenue of research leading to MOOC participants' (both students' and facilitators') views on the ethical issues of using the data generated by them on a MOOC for research and a researcher's role in the MOOC.However this topic is discussed by MOOC participants and observers in blogs such as Robbins (2013).
Many studies have presented participant demographics and it was observed that a large majority of participants were from North America and Europe.There were very few participants, if any at all, from Asia and Africa with a few from South East Asia (deWaard, Abajian, Gallagher, et al., 2011;Kop, 2011;Koutropoulos, et al., 2012).This is hardly surprising due to the possible barriers of access to online learning that can be presumed in those parts of the world, both technological and linguistic (see Liyanagunawardena, 2012 for a discussion of the similar problems in the introduction of online learning in Sri Lanka).However, a number of Asian and African learners participated in MobiMOOC (Koutropoulos, et al. 2012).There are two plausible reasons: It could be that the course content of the MobiMOOC (mLearning) is of more interest to learners from those regions or it could be that mLearning is significantly more accessible in those places.deWaard, Koutropoulos et al. (2011) report that 77.5% of the participants in their survey had used mobile devices to access the course even though it was not required, indicating that the flexibility offered by mLearning is attractive for those MOOC participants.This shows the potential for exploring the possibility of mLearning MOOC provision in the developing world, both for increasing and widening participation.
As discussed already, some of the researchers who used data generated through MOOCs have limited their qualitative analysis due to the large volumes.Regarding MOOC participant approaches, Fini (2009) similarly argues that people seem to be torn between the time-saving advantage offered by the 'Daily' solution [an email message sent daily by the facilitators with a summary of the topics of the existing conversation] and the multi-faceted, time-consuming alternative represented by direct access to unfiltered information.
Due to the large volumes of data generated by the 'massive' number of students engaged in a MOOC, being up-to-date with the ongoing discussions can be challenging or even overwhelming.
Exploring the strategies used by students who continue to be active participants in a MOOC could provide some insight into possible solutions to the information overload in a MOOC environment for both other MOOC learners and for MOOC researchers.In fact, Milligan, Margaryan, and Littlejohn (2013) show that all but one active participants of Change11 MOOC have participated in previous MOOCs and question whether a learner has to learn how to learn in a MOOC.While many MOOC participants struggle to keep going "there's some people who are everywhere you turn in the Change11 MOOC: there's this group of people who are inspirational, just phenomenal the way they just keep going and they know their way around it" (Milligan, Margaryan, & Littlejohn 2013).
Confidence gained after successful participation in one MOOC together with network effects where people start building communities and networks in one MOOC then moving together onto another MOOC may result in more previous MOOC participants taking up other MOOCs.
It is acknowledged that MOOCs have high withdraw/dropout rates (Koutropoulos, et al., 2012); however, data on completion rates of MOOCs are not readily available.According to Jordan's (2013) collated completion rates for 24 MOOCs (as of March 11th, 2013), the highest completion rate achieved was 19.2% on Functional Programming Principles in Scala, a MOOC offered by Coursera in 2012.The majority of MOOCs had completion rates of less than 10%.There is very little known about the experiences of non-completing MOOC participants (Koutropoulos, et al., 2012).This would be an interesting and useful avenue to explore further in future research, though engaging those who have started but not finished an educational course in research into the reasons for their noncompletion is likely to be difficult.
Motivation is identified as an important contributor to student engagement in a MOOC by Milligan, Margaryan, and Littlejohn (2013).One can speculate about an individual's motivation to participate in a MOOC: the desire to achieve an academic credential at a reduced cost, personal enrichment, and/or self satisfaction.However, why individuals participate in MOOCs has yet to be explored.It would be valuable to learn about the actual motivations in place, the percentage of participants taking up MOOCs for those reasons, and to know how those motivations might vary from one course or discipline or even provider to another.
Recognition for MOOCs and their accreditation is another area of debate.MOOCs run by educational technology companies such as Coursera and EdX provide the option to pay for certification.For example, Coursera offers proctored exams for a fee, which will earn (if successful) certification.On the other hand, most MOOCs offer badges for completion of either the full course or each unit (or week of work, for example OLDSMOOC by the Open University).Some MOOC-offering institutions and/or instructors provide a Statement of Accomplishment for successful students.However, these generally do not carry college credits.Recently, the American Council on Education recommended five Coursera courses for college credit (Coursera, 2013) while a California bill is seeking credit for students taking faculty approved courses online (Levin, 2013).However, given that existing inperson "closed book" examinations are subject to cheating (Shimbun, 2011) and that at least one enterprising IT worker in the US successfully outsourced his own job to China (Kim, 2013), the challenges of validating the assessments of MOOC participants are clear.

Conclusion
MOOCs have created wide interest as a change agent in higher education, and the peer-reviewed research literature on them is growing but still limited.Many articles published to date have discussed empirical evidence from case studies, the influence on higher education structure, or educational theory relating to MOOCs.While there is research into the learner perspective, neither the creator/facilitator perspective nor the technological aspects are being widely researched (or at least such research has not yet been published).MOOCs generate a plethora of data in digital form for interested researchers.However, this volume has so far limited researchers to analysing only a tiny portion of the available data, restricting our understanding of MOOCs.There are further interesting research avenues such as cultural tensions within courses and the ethical aspects of using data generated by MOOC participants still to be explored.publish course content, institutions used television broadcasts.For example, many people who were not engaged in the Open University courses watched various Open University programs telecasted, particularly when video recorders became cheaply available.This unregistered informal accessing of educational materials reduced when the Open University main broadcasts stopped and course materials were delivered primarily through the post (on CD and DVD) and later via online delivery.Similar closure happened when early adopters of online provision of course content through the open web closed off access by placing it behind authentication restrictions in LMS (such as Moodle and Blackboard) (Goldberg, 2001).

Figure 3
Figure 3 illustrates the separation of articles by publication type and year.From 2009 to 2012 there is a gradual increase of the number of journal articles and conference papers.It is noticeable that the first magazine article about MOOCs appeared in 2011 and the following year there is a fourfold increase, reflecting the recent general interest in MOOCs.
Knowledge: the first MOOC offered from the University of Manitoba (Canada) in 2008.CCK09 -Connectivism and Connective Knowledge: a later version of CCK08 offered in 2009.CCK11 -Connectivism and Connective Knowledge: a later version of CCK08 offered in 2011.CritLit -Critical Literacies: a 2010 course blending 'critical thinking' and '21st century literacies' offered by the National Research Council of Canada.CS1 -An introductory programming course from the University of Helsinki (Finland) based on an apprenticeship model.CS101 -An introductory computer science course, from Udacity.CS221 -A course on artificial intelligence offered by Stanford (US).EduMOOC -Online Learning Today... and Tomorrow: offered by the University of Illinois (US) in 2011.FSLT11 -First Steps in Learning and Teaching, a 2011 course offered by Oxford Brookes University (UK) .Future of Learning -an education course: offered by the University of Frankfurt (Germany), in 2011.MobiMOOC -Mobile learning: offered by the Institute of Tropical Medicine Antwerpen (Belgium) in 2011.PLENK2010 -Personal Learning Environments, Networks, and Knowledge: a 2010 course offered by Athabasca University (Canada).Qual MOOC -Qualitative Research Methodology: Kibbutzim College of Education, Technology and the Arts, Israel proposed to be run in 2012.The MOOCs CCK08, CCK11, CritLit, and PLENK2010 have designers and deliverers in common.Of the case study papers, one (Rodriguez, 2012) compared different MOOCs (including CCK08, PLENK2010, MobiMOOC, EduMOOC, Stanford CS221, and CS101 from Udacity); two (Kop 2011; Kop, Fournier, & Mak, 2011) considered PLENK2010 and one other MOOC (CritLit and CCK11, respectively); and another (Bell 2010a) considered CCK08 and CCK09; the remainder focussed on single MOOCs.Table led by Stanford University, and edX (www.edx.org) which includes the Massachusetts Institute of Technology, École Polytechnique Fédérale de Lausanne, The Hong Kong University of Science and Technology.Udacity (www.udacity.com),P2P University, and Futurelearn (the UK Open University's MOOC platform) are other related platforms.The numbers registering for MOOCs have reached 160,000 in the case of a 2011 Artificial Intelligence online course offered by Stanford University MOOCs but using the two different labels for the same thing: "AI-Stanford like courses" and "xMOOCs".However, our systematic review has shown that there is a gap in the scholarly literature in defining these different types of MOOCs.
Daniel (2012)12)classifies MOOCs into two categories: connectivist MOOCs (c-MOOCs) and AI-Stanford like courses.He associates courses similar to AI-Stanford predominantly with cognitivebehaviourist approaches and c-MOOCs with connectivist approaches.By comparing multiple case studiesRodriguez (2012)claims that courses similar to AI-Stanford have a more individualist learning approach while c-MOOCs have a more social approach to learning.He also shows the different roles played by facilitators in each type of MOOC.Daniel (2012)on the other hand discusses cMOOCs and xMOOCs, which he claims are a bifurcation of MOOCs.He refers to "xMOOCs now being developed by elite US institutions that follow a more behaviourist approach" (Daniel 2012) but does not provide a definition.It is probable that bothRodriguez (2012)andDaniel (2012)are similarly classifying