International Review of Research in Open and Distributed Learning Case Study as a Research Method for Analyzing MOOCs Presence and Characteristics of Those Case Studies in the Main Scientific Databases

Method


Introduction and Literature Review
The study of MOOCs is very diverse and encompasses many disciplines, fields of study, and ways of understanding research, both epistemologically and methodologically (Bates, 2014).Case study is an adequate, necessary method for certain research tasks in the field of social sciences (Flyvbjerg, 2006), and provided there is a large number of thoroughly executed case studies available, this approach contributes to better and more effective disciplines (Kuhn, 1987).The prevalence of this method in research on MOOCs (Kennedy, 2014;Raffaghelli, Cucchiara, & Persico, 2015) has varied over the past few years.Despite the amount of literature on case study research designs over the past 40 years (e.g., Bennett & Elman, 2008;Gomm, Hammersley, & Foster, 2000;Merriam, 2007;Mitchell, 1983;Simons, 2009;Stake, 1995;Yin, 2009), and attempts at defining a typology to assist researchers in structuring and analyzing such studies (Thomas, 2011), there is more than one way of understanding case studies.
Historically, case studies in the literature were more associated with the interpretive paradigm.
However, there are now some very diverse ways of understanding case study as a research methodology.
Case studies can be conducted using quantitative/qualitative paradigms or mixed methods.Indeed, the vast amount of techniques and methods for data collection and analysis that can be used for such studies has led some authors to explicitly state that "the case study survives in a curious methodological limbo" (Gerring, 2004, p. 341).
A large number of reviews of the literature on MOOC research have been published, some of which focused on analyzing thematic aspects, while others dealt more with the methodological aspects.This study presents a brief review of these papers, with special emphasis on research using the case study as a methodology, and describes the data collection and data analysis methods used.
In the first review to be conducted in a systematic manner (Liyanagunawardena, Adams, & Williams, 2013), the selected literature was categorized into different areas of interest, and the authors proposed directions to guide future research.They also identified a number of thematic and methodological gaps in the scientific literature at that time, describing four challenges facing MOOC researchers and designers: (a) the need for all perspectives of MOOCs (e.g., learners, creators, teachers, institutions) to be explored; (b) cultural tensions among pedagogies, resources, and learning environments; (c) ethical aspects of using data generated by MOOCs; and (d) the implementation of effective learning strategies in order to achieve a successful balance between information overload and self-regulated learning within MOOCs.The authors classified 21 documents that included case study, most of which had used multiple methods, with surveys being the most common data collection method.Ebben and Murphy (2014) postulated that MOOC research at the time of their writing could be divided into two phases, the first being more related to cMOOCs, engagement, and creativity, and the second focusing on xMOOCs, learning analytics, evaluation, and critical discourse on MOOCs.These authors presented the dominant theories, the directions followed by research up to that point, and the most prevalent topics dealt with in the literature.Methodological aspects and case studies were mentioned in their study but were not the main focus of their work.In the same year, Hew and Cheung (2014) conducted another review of the literature, focusing on the motivations and challenges relating to MOOC courses, namely diversity of topics, the perspective from which they were addressed (students or instructors), and main findings to date.Although the authors spoke briefly about the techniques used (p.47), this was not either the main objective of their analysis.Jacoby (2014) presented a review of the literature on the theory of disruptive innovation, reporting that prior research had been predominantly qualitative, particularly comprising case studies and narrative research (p.74).This author also postulated the need for a broader methodological range to enhance data triangulation.In a similar review, Kennedy (2014) analyzed the limitations and gaps identified in previous research on MOOCs and put forward a number of recommendations for future research.Her review included only a short paragraph on methodological aspects (p.7), noting the wide range of methodologies such as mixed methods, case studies, narrative inquiry, and comparative studies.
In a review of research proposals submitted to the MOOC Research Initiative (MRI), Gasevic, Kovanovix, Joksimoviv, and Siemens (2014) identified a number of topics that might be used as a framework for future research.They also analyzed the methodologies used in these proposals, reporting that 42.3% had used mixed methods, 33.3% quantitative methodology, and 24.4% a qualitative approach.However, subsequent analysis showed that this research does not distinguish between paradigm, and data collection and analysis methods (Veletsianos & Shepherdson, 2016).Raffaghelli et al. (2015) was the first published review of MOOC literature with the sole objective of analyzing methodological approaches.The authors identified trends, gaps, and criticalities derived from methodological decisions taken by MOOC researchers in the period 2008 to 2014.They noted that this field of research was still in its infancy at the time of writing, and that much of the research they reviewed relied on theoretical-conceptual research and case studies, which they considered a preliminary step toward identifying methods to deal with large cohorts or large amounts of data.These authors postulated that research on MOOCs was still in the early stages of the full cycle of educational research (Gorard & Cook, 2007).Veletsianos and Shepherdson (2015) carried out a further review on the concept of interdisciplinarity, as well as the ways in which research published in the years 2013 to 2015, which was more aligned with the concept of xMOOC, was more interdisciplinary in nature than research conducted during the first phase (Ebben & Murphy, 2014).Most of the work was carried out by researchers from the field of education or from computer science disciplines, whose contributions to the research on MOOCs are increasingly frequent.A subsequent review published the following year provided an overview of geographic distribution, publication outlets, methodologies used, and research strands followed in studies on MOOCs (Veletsianos & Shepherdson, 2016).The findings of this review showed that researchers had used quantitative rather than qualitative insight in their works, particularly surveys and automated methods.Further, the authors reported that qualitative research on MOOCs in the period analyzed was often basic, and very few studies had actually used methods traditionally associated with qualitative research.
Two reviews have been published in the past two years that focused wholly on methodological issues (Deng & Benckendorff, 2017;Zhu, Sari, & Lee, 2018).The first reported surveys as being the most widely used method for data collection, followed by interviews and log files, and that most of the articles examined had focused on the learner-student perspective (90.6%).The second review was based on 146 articles, of which 45.9% were quantitative, 35.6% had used mixed methods, and 18.5% were qualitative in nature; this review also reported surveys as being the most common method of data collection.
Descriptive statistics, inferential analysis, and content analysis were the most usual data analysis methods found in this review.The above-mentioned review of the literature (summarized in Figure 1) provided an overview of previous MOOC research.It showed that this field of study is expanding and constantly changing (Veletsianos & Shepherdson, 2016), and although in the early years it was mainly comprised of conceptual studies and case studies (Jacoby, 2014), its evolving nature brought forth a large number of macro-type empirical studies (e.g., big data, learning analytics) that were facilitated by the current availability of large datasets.However, no studies were found on the actual concept of case studies, or on the characteristics of case studies that are used to gain a deeper understanding of MOOC platforms.
Generally, case studies in the field of social sciences have been understood as a methodology associated with qualitative, interpretive, or hermeneutic paradigms.Therefore, it is essential-and would enhance this field of study, and advance our understanding of the methodological aspects-that we examine how the scientific community is using case studies to explore online environments, and ascertain what paradigms/approaches and methods are used in these case studies, what is their research focus, and their prevalence in the main scientific databases.

Statement of Research Problem and Purpose
The main aim of this review was to gain a better understanding of the characteristics of case studies relating to MOOCs in the WOS and Scopus databases (i.e., the methodological approaches used, how they were implemented, and their focus).An in-depth review of extant literature in the major academic databases can provide information on the characteristics of the publications available in these peer- In order to explore the presence of case study as a methodology for research on MOOCs, we posed the following research questions: 1. What methodological approaches did case studies on MOOCs follow?
2. What were the usual publication outlets for case studies on MOOCs: journals or conference proceedings?Which journals published the highest number of case studies on MOOCs?
3. Which case studies had the highest citation count, and how were the publications distributed over the years?
4. What data collection and data analysis methods were used in these case studies?
5. What was the main focus of these case studies?

Research Methodology
A systematic literature review was conducted based on an analysis of 92 documents of peer-reviewed literature indexed in the WOS and Scopus databases during the period January 2012 to June 2018.In order to answer the research questions, several methods were systematically followed to collect the extant literature in WOS and Scopus, and analyze the corpus of selected papers.

Data Collection
Searches of WOS and Scopus were performed in July 2018 using the keywords MOOC, MOOCs, massive open online courses, xMOOC, cMOOC, and these were interrelated with the keyword case study in the fields article, keywords, and abstract in the two databases (in WOS, the field is shown as topic).In order to analyze the returned results in greater detail, the following selection criteria were applied in order to selectively eliminate: (a) duplicated search results returned from separate databases; (b) documents that appeared to be case studies but were not, or had not used a case study as a part of their methodology; (c) documents in languages other than English or Spanish; (d) documents that were not proceedings or journal articles (in SCOPUS only journal articles were selected); and (e) any documents with a zero citation count.Application of these selection criteria brought forth a corpus of 92 papers.
The first step was to define the structure of a database to store the most relevant information from the documents analyzed.Records in the database were structured as shown in Table 1.Each of the records included objective information (e.g., title, author(s), keywords, year of publication, DOI/URL) as well as several other fields that were completed after reading and analyzing the full text of each document, including methodological approach, data collection methods, data analysis methods, and focus of the study.

Data Classification and Analysis
The corpus was analyzed quantitatively and qualitatively.Quantitative analysis was used to classify the documents by year of publication, type of publication, country of author affiliation, citation count, and mean citation count per year.The documents were also analyzed qualitatively using open-coded content analysis, a technique that has been used previously in other literature review studies on MOOCs (Liyanagunawardena et al., 2013).First, one of the authors read each of the documents in order to identify the methodological approach, data collection methods, data analysis methods, and focus of the paper.Subsequently, the other authors carried out a similar review, sharing their results with the first researcher.In case of discrepancies, researchers would re-read and examine the pertinent text together, and reach a joint decision.
The basic unit of analysis was a single selected paper; a constant comparison method was used for its classification and analysis.The first text was analyzed and coded by a researcher using emergent coding, and the topic category and methodological approaches were defined.The second text was analyzed in the same way and checked to determine whether it could be classified into the same category as the previous document; otherwise, a new category was created.The process was subsequently repeated until all documents had been read and analyzed.To eliminate the possibility of a document being classifiable in more than one category, the categories were thoroughly examined and verified by all authors to ensure that each category was exclusive and was not repeated in any way.The categories were shared and agreed to by all authors throughout the analysis process.

Results
The following tables show the results of the review of the corpus of collected papers for each of the research questions posed at the outset of this study.

Question One: What Methodological Approaches did Case Studies on MOOCs Follow?
On analyzing the individual documents, based on the previously described method, it was found that 30 documents had used a quantitative approach, and 26 a qualitative approach; 25 had used mixed methods, and 11 an unclear/not explicit approach.Table 4 shows the highest citation counts in the Web of Science database (more than 10 citations) for the publications included in the corpus of this review; three of these were published in the International Review of Research in Open and Distributed Learning, three in Computers & Education, and three in other journals.

Question Four: What Data Collection and Analysis Methods Were Used in These Case Studies?
An analysis of the main data collection methods showed that most of the studies reviewed had used one (41.4%)or two (35.7%)methods for collecting data.Three or more data collection methods were used in 22.9% of the documents.6 shows the data analysis methods used in the corpus documents, statistical analysis (various types) being the most frequent method used (47), followed by qualitative content analysis (17), grounded theory (11), and experimental or quasi-experimental analysis.Other methods used are shown in the table above: thematic analysis (7), software-assisted automated analysis ( 6), crosschecking between different analyses (5), discourse analysis (5), big data analysis (4), and other methods mentioned in only one of the corpus documents.

Question Five: What Was the Main Focus of These Case Studies?
Six categories were established to describe the research reported in this corpus (see Table 7), using a categorization similar to that of Veletsianos and Shepherdson (2016).Most of the 92 MOOC case studies included in the sample were focused on learners/students (39) or the actual platform (28).Just 12 documents focused on pedagogical design, eight on instructors/teachers, three on the community, two on methodological design, and four on other elements.A small number of papers were included in two categories (4).Platform-focused.This category included papers focusing on the platform as the main research element.Papers covered topics such as platform usability (e.g., Yousef, Chatti, Schroeder, & Wosnitza, 2015), innovative tools (e.g., Fu, Zhao, Cui, & Qu, 2017), intelligent and adaptive systems (e.g., García-Peñalvo, Fidalgo-Blanco, & Sein-Echaluce, 2018), mediation and control (e.g., Nyoni, 2013), or the relationship between the pedagogical design and the platform design (e.g., Drake, O'Hara, & Seeman, 2015).
Community-focused.This category included a number of studies that focused on understanding the set of stakeholders in a MOOC, as a community (e.g., Grünewald et al., 2013, Jones, Stephens, Branch-Mueller, & De Groot, 2016).
Other.This category included papers that could not be classified in any of the other categories because they concerned topics such as ethics and privacy (Jones & Regner, 2016), or plagiarism (Tsoni & Lionarakis, 2014).

Discussion
MOOC literature reviews published prior to this paper were examined.For the purpose of the present review, 92 publications that had used case study methodology for MOOC research were collected and analyzed.A dataset of MOOC-related case studies was created to facilitate identification of internal methodological approaches, publication outlet, prevalence in the databases, methods used for data collection and analysis, and the papers' research focus.Findings showed that despite being a methodology generally linked to more interpretive paradigms (Simons, 2009;Stake, 1995), almost 60% of the studies analyzed had used a quantitative approach (32.6%) or a mixed method (27.2%).The papers in the corpus analyzed in this study had been published in a wide range of journals and conference proceedings; some journals had published a greater number of studies than others.Findings also showed that the source of more than 80% of research using the case study method was Europe and North America.Further, it was observed that unless the keyword case study was included in the search query, the search results returned for the 25 most cited papers in the literature on MOOCs did not include any case studies.With regard to the data collection methods that appeared in these documents, the most notable methods were linked to quantitative paradigms such as surveys or the platform dataset; statistical analysis was the main data analysis method used.The focus of the case studies analyzed was essentially the students or the actual platform.Accordingly, these findings have a number of implications for future research on MOOCs using case study methodology, and for the state of the field as a whole.

Our Choices May be Limiting a Deeper Understanding of MOOCs
Coinciding with previous reviews of MOOC literature, this analysis suggests that researchers tended to choose a quantitative rather than a qualitative approach, even for case studies.This might suggest that more interpretive, hermeneutical, or qualitative research is needed.For example, as has also been noted in previous reviews (Veletsianos & Shepherdson, 2016), an in-depth analysis of the role of instructors in MOOC courses could serve to illustrate certain topics that do not seem to be much in evidence in the current literature.Qualitative case study would be a very useful methodological approach for such a purpose, thus providing some excellent possibilities for future research.

Certain Regions are Setting the Pace of Case Study Research on MOOCs
Author affiliation in more than 80% of the documents analyzed was either Europe or North America.This is in keeping with the findings from previous reviews researching geographical distribution, except for a higher number of author affiliations from Spain, given that Spanish was included as a language in this review.Nonetheless, the results show that these databases analyzed may be favoring literature from certain regions, or otherwise limiting and failing to give visibility to the literature from other countries, which would possibly indicate a direction for future research, to analyze this geographical bias.
Countries such as India or China, with a large presence in the MOOC community by the number of people enrolled in these courses, have little or no presence in the selected corpus.
Case Studies Tend to be Less Prevalent in the WOS and SCOPUS Databases Than Are Other Types of Empirical Studies It was noted from the findings that the citation count for the case studies analyzed was somewhat lower than for other types of empirical studies.More than half of the documents returned in the search results had a zero-citation count.Further, the most frequently cited case study in the corpus ranked 26th in the listing.On analysis, only two of the ten most cited case studies had used a qualitative methodology, which raises the question of whether the research reported in the most highly ranked papers in these databases is perhaps quantitative rather than qualitative, even for case studies.Although a number of previous reviews in MOOC literature referred to the existence of case studies and their abundance (Raffaghelli et al., 2015), they were not always listed in the search results returned by the main databases.This highlights the higher presence of case studies in other databases (e.g., Google Scholar, ERIC, EBSCOhost) and a secondary role in WOS and SCOPUS.

Higher Prevalence of Quantitative Methods and Greater Diversity of Qualitative Methods
Surveys, questionnaires, and datasets were the most predominantly used methods, even in a large number of papers reporting so-called qualitative research.However, qualitative data collection methods, which were generally less prevalent, were found to be more diverse.Similar findings were obtained in regard to data analysis methods-a large number of the documents analyzed had used different types of statistical analysis (usually descriptive), and papers that had used methods more closely linked to the qualitative paradigm frequently used multiple techniques to analyze the data (e.g., content analysis, thematic analysis, discourse analysis, grounded theory).The database used for this study also showed a sharp increase in recent years in the use of automated analysis methods, and of research using big data or data mining techniques for data analysis in case studies, most of which were based on a quantitative approach.

Scarcity of Instructor-Focused Research
As noted in other literature reviews, the number of studies highlighting the role of instructors is very limited.Case study would be an appropriate methodology for such research, given that it focuses on the characteristics of unique, specific cases.Case study could help illustrate how instructors experience their involvement in MOOC courses, how they perceive their relationships with colleagues or students, how these differ from relationships in an offline environment, or what motivates them to become instructors.These are research strands that are currently little explored, and case study is a highly flexible methodology that could help to clarify certain aspects regarding the people who perform teaching tasks in these courses.

Limitations
This study has several limitations.First, only the Scopus and WOS databases were used to find case studies, and although these two databases are the most recognized in the academic field, there might be other case studies, in other journals, that were not indexed in these databases.Second, there could be different documents indexed in WOS or Scopus that used case study as a methodology but had failed to explicitly indicate so, and therefore, were not included in this review.Third, by not incorporating articles that have not yet been cited, the study may not reflect the very latest trends in the field (e.g., monetization and business models, big data and learning flexibility, sustainability).Fourth, English, as the language for publications, is over-represented in the WOS and Scopus databases, in the same way that certain scientific fields are more prevalent than others; social sciences, as well as arts and humanities, for example, may be underrepresented (Mongeon & Paul-Hus, 2016).The results of this study are also affected by the rapid evolution of scientific literature on MOOCs in recent years.In this case, it was decided to review solely journal articles and proceedings in order to define the characteristics of these papers in a peer-reviewed process.The authors note that this review did not include a large number of case studies that are available in other formats (e.g., books, blogs, reports, non-indexed journals), or in languages other than English or Spanish, which undoubtedly also form part of the debate in this field of knowledge.

Conclusion
A large number of research papers have been published on MOOCs since their inception, some of which used the case study as a methodology.The data from this review reveals that their prevalence and citation count in databases was limited compared to other empirical works.According to the findings from this analysis of 92 case studies published between 2012 and 2018, (a) more than half of these papers used quantitative methods for data collection and analysis; (b) much of the research focused essentially on learners; and (c) author affiliation was predominantly North American or European, showing a clear geographic bias.Although this study focused on a specific methodology, its findings nonetheless replicated the results of previous studies in which more than one method was examined (e.g., Veletsianos & Shepherdson, 2016;Zhu et al., 2018), and which reported a scarcity of research focusing on instructors, or other studies that found a higher prevalence of quantitative methods (Deng & Benckendorff, 2017;Gasevic et al., 2014).The authors of this paper hope that these findings will encourage future studies on those aspects of MOOCs that have not yet been explored in depth.There are still many possible topics in this area for further research using case study methodology, and the authors recommend that similar reviews of MOOC literature be conducted with regard to other methodological approaches or to different databases containing more research from other regions written in other languages.
reviewed outlets, as well as what has already been done, what remains to be done, and what might be the possible direction for future research based on this methodology.The authors of this paper wished to contribute to one of the ideas for future research proposed by Veletsianos & Shepherdson (2016), namely that "future research endeavors in this area may focus on examining how particular methodologies have shaped the field" (p.215).Their research identified potential research gaps in the study of MOOCs and this paper confirms and extends that research to provide possible directions for future research using case studies.

Figure 3 .
Figure 3. Authors' affiliation and number of documents.

Figure 4 .
Figure 4. Documents by publication year.

Question Two: What Were the Usual Publication Outlets for Case Studies on MOOCs: Journals or Conference Proceedings? Which Journals Published the Highest Number of Case Studies on MOOCs?
Figure 2. Type of document collected.Of the 92 documents collected, 70 articles were from peer-reviewed journals and 22 were from conference proceedings.With regard to journal publications, 44 journals had each published one item; the remainder had published 2 or more documents, as shown in Table 3.The journal International Review of Research in Open and Distributed Learning (IRRODL) published 9 articles, the highest number of documents.

Table 3
Main Publication OutletsFigure3illustrates the geographic distribution of those who authored the 92 articles.Overall, 30 documents had one or more authors from the United States of America (32.6% of the corpus), there were 17 documents with at least one author from Spain (18.5% of the corpus), 11 with at least one author from the United Kingdom (11.9%), six with at least one author from China, five with at least one author from Germany, and so forth.There are other 21 different author affiliations present in just one document, each accounting for 1.1% of the corpus.The most highly represented region in the corpus

Table 5
Data Collection Methods and Their Prevalence The most frequently used methods and techniques as shown in Table5are, in this order: surveys and questionnaires, user datasets or logs, and forum participation.Other collection methods not mentioned in the table were found in solely one document of the corpus (e.g., rubrics or email messages).

Table 7
Main Focus of the Case Studies Zhang & Yuan, 20162014))ments in which the key element was the figure of the learner were included in this category, regardless of the topic covered in the research.Other topics included learner expectations and perceptions (e.g.,Cross & Whitelock, 2017), participation level (e.g.,Veletsianos, 2017), roles and identities (e.g.,Baxter & Haycock, 2014), user behaviors (e.g.,Zhang & Yuan, 2016), or   engagement/retention level (e.g., De Freitas et al., 2015).