Who Studies MOOCs? Interdisciplinarity in MOOC Research and its Changes over Time

The complexity of digital and online education is becoming increasingly evident in the context of research into networked learning/participation. Interdisciplinary research is often proposed as a way to address complex scientific problems and enable researchers to bring novel perspectives into a field other than their own. The degree to which research on Massive Open Online Courses (MOOCs) is interdisciplinary is unknown. We apply descriptive and inferential statistics to bibliometric data to investigate interdisciplinarity in MOOC research. Results show that MOOC research published in 2013-2015 was (a) mostly conducted by researchers affiliated with Education and Computer Science disciplines, (b) far from monolithic, (c) had a greater representation of authors from Computer Science than in the past, and (d) showed a trend toward being more interdisciplinary than MOOC research published in 2008-2012. Our results also suggest that empirical research on xMOOCs may be more interdisciplinary than research on cMOOCs. Greater interdisciplinarity in xMOOC research could reflect the burgeoning interest in the field, the general familiarity with the xMOOC pedagogical model, and the hype experienced by xMOOCs. Greater interdisciplinarity in the field may also provide researchers with rich opportunities to improve our understanding and practice of digital and online learning.


Article abstract
The complexity of digital and online education is becoming increasingly evident in the context of research into networked learning/participation. Interdisciplinary research is often proposed as a way to address complex scientific problems and enable researchers to bring novel perspectives into a field other than their own. The degree to which research on Massive Open Online Courses (MOOCs) is interdisciplinary is unknown. We apply descriptive and inferential statistics to bibliometric data to investigate interdisciplinarity in MOOC research. Results show that MOOC research published in 2013-2015 was (a) mostly conducted by researchers affiliated with Education and Computer Science disciplines, (b) far from monolithic, (c) had a greater representation of authors from Computer Science than in the past, and (d) showed a trend toward being more interdisciplinary than MOOC research published in 2008-2012. Our results also suggest that empirical research on xMOOCs may be more interdisciplinary than research on cMOOCs. Greater interdisciplinarity in xMOOC research could reflect the burgeoning interest in the field, the general familiarity with the xMOOC pedagogical model, and the hype experienced by xMOOCs. Greater interdisciplinarity in the field may also provide researchers with rich opportunities to improve our understanding and practice of digital and online learning.

2
The term Massive Open Online Course (MOOC) describes an evolving ecosystem of open online learning environments, encompassing a spectrum of course designs ranging from networks of distributed online resources (cMOOCs) to structured learning pathways centralized on digital platforms (xMOOCs) (Rodriguez, 2012). As MOOCs have drawn large numbers of participants, researchers have argued that "the ability of MOOCs to generate a tremendous amount of data opens up considerable opportunities for educational research" (Breslow et al., 2013, p. 13). Partly as a result of the breadth and diversity of available data and partly as an outcome of the diversity and motivations of online learners, investigating digital learning has quickly become a complex socioscientific endeavor. Inquiries into digital learning may therefore have a lot to gain from multidisciplinary, interdisciplinary, and crossdisciplinary research, as such research approaches are promising for solving complex problems (Pellmar & Eisenberg, 2000).
Nevertheless, in their assessment of proposals submitted for funding under the MOOC research initiative (henceforth MRI), Gašević, Kovanović, Joksimović, and Siemens (2014) show that more than 50% of the authors in all phases of the MRI grants were from the field of education, even though a common perception in the field is that the MOOC phenomenon is "driven by computer scientists" (p. 166). Research into emerging forms of digital learning is likely to suffer if driven by education researchers alone or computer scientists alone. As Gašević and colleagues note, disparate involvement in MOOC research "could be a worrying sign of the fragmentation in the research community", and as such there is a need to "[increase] efforts towards enhancing interdisciplinarity" (p. 134).
We have recently completed a systematic review of the empirical MOOC literature published between 2013 and 2015 (Veletsianos & Shepherdson, 2015), and as a result of that study, we can use bibliometric data to investigate the extent to which interdisciplinarity is present in the published literature on MOOCs. Thus, in this paper, we combine our data with data from Gašević et al. (2014), and data used in a past systematic review of the literature (Liyanagunawardena, Adams, & Williams, 2013), to examine interdisciplinarity in the MOOC literature, and whether and how this has changed over time.
To examine these issues, we review relevant literature and explain why this study is significant; describe the methods used to conduct this investigation; present the results; and conclude by discussing the implications and limitations of the findings.

Literature Review
Interdisciplinary research is "any study or group of studies undertaken by scholars from two or more distinct scientific disciplines" (Aboelela et al., 2007, p. 341). The value of interdisciplinary research and collaboration rests in both the complexity of the problems societies are faced with (Pellmar & Eisenberg, 2000), and in the novel perspectives that academic "immigrants" can bring to a field other than their own (Nissani, 1997). While specialists working within their own disciplines do so with a depth not achievable by outsiders, such a focus generally necessitates some degree of neglect of potentially relevant concepts and ideas from other fields. For sufficiently complex research questions, this can lead to explanations that lack important details.
Nissani illustrates this problem regarding subjects as diverse as the Cold War, political liberties, and nuclear science. Ideally, interdisciplinary research resolves such issues by using insights from disparate bodies of knowledge to provide more holistic perspectives.
Research investigating interdisciplinarity has explored its utility in academic, educational, and practical contexts (Choi & Pak, 2006), factors mitigating against its occurrence (Bauer, 1990), and whether indications of its presence actually reflect cross-pollination between different disciplines (Schummer, 2004). There has also been research examining how the level of interdisciplinarity in a field changes over time. For instance, Huang and Chang (2011) demonstrated that research in Information Science has become increasingly interdisciplinary over the course of three decades. If interdisciplinarity does allow for a more rounded perspective on research problems, such changes may indicate a field's increasing sophistication and complexity -particularly if this field is relatively nascent.
While the practice of distance and online education has traditionally involved collaboration across disciplines (e.g., instructional designers, content experts, and learning technologists collaborating to design and develop educational programs), the scientific complexity facing research into digital and online education is increasingly expanding. This is evident in the context of research into MOOCs and networked learning/participation. For instance, conducting informed, relevant, rigorous, and theoretically-sound research on the relationships between learning, participation, and learner characteristics using large scale data sets (e.g., those from MOOCs or from social media such as Twitter) may necessitate an understanding of a variety of fields and disciplines ranging from education (e.g., theorizing online learning), to computer science (e.g., data mining), and statistics (e.g., data analysis). Zuckerman, Azari, and Doane, (2013, p. 17) for instance argue that the learning sciences and learning technology communities are at the cusp of catalyzing potentially transformative change in post-secondary education...At the same time, there are many unknowns regarding the fundamental science of learning, the translation of those fundamental ideas into curriculum design 4 and technology development, and the best use of technology to enhance learning. Eisenhart and DeHaan (2005) proposed that doctoral preparation programs for education researchers include interdisciplinary collaborations as one of their core components, and the scientific community has responded to the scientific complexity of digital education in part by developing the field of learning analytics, which Gašević, Dawson, and Siemens (2015, p. 64) describe as a "bricolage field drawing on research, methods, and techniques from numerous disciplines such as learning sciences, data mining, information visualization, and psychology." While we have been able to identify some papers encouraging researchers to cross disciplinary lines to investigate digital education (e.g., Alavi & Leidner, 2001), studies exploring the degree to which digital education research in general, and MOOC research in particular, is interdisciplinary, are scant. The one study that we have identified that examined multidisciplinarity in the MOOC context was the analysis of funding submissions to the MOOC research initiative (MRI) by Gašević et al. (2014). Part of this analysis involved presenting statistics describing the disciplines to which researchers making submissions belonged. Finding a preponderance of researchers from the field of Education (who submitted MRI proposals with a frequency more than four times that of any other discipline), Gašević and colleagues suggested that there might be a need to foster more interdisciplinarity in MOOC research. However, given that their analysis focused solely on submissions seeking funding from a specific source, the extent to which this is reflective of MOOC research more broadly is unclear.
While not directly bearing upon the topic of interdisciplinarity in MOOC research, a study by Kirby, Hoadley, and Carr-Chellman (2005) is also informative. These authors were interested in understanding the relationships between the Instructional Systems Design and Learning Sciences fields -two fields of study interested in the use of technology for learning -and conducted a citation analysis to identify these relationships. They found low rates, but an increasing trend, of cross-field citation. For the purposes of this particular study, their research demonstrates that bibliometric approaches to understanding issues pertaining to digital education may be worthwhile exploring.
Consequently, we decided to use a bibliometric approach to investigate interdisciplinarity in MOOC research. Though not providing as rich an assessment of interdisciplinarity as, for example, identifying and classifying specific research methods and techniques used by individuals with backgrounds in different disciplines, a bibliometric approach does allow us to describe the interdisciplinary nature of MOOC research as regards Aboelela et al.'s (2007) definition of the term: studies undertaken by scholars from different disciplines. Specifically, using the papers collected for a systematic review of the literature reported in Veletsianos and Shepherdson (2015) provides an opportunity to assess the disciplinary distribution of authors conducting recent research into MOOCs. Importantly, for the purposes of the systematic review of the literature that 5 we report elsewhere and from which we draw the corpus for this study, we attempted to ensure that the literature reviewed was exhaustive, which also means that the corpus within which we study interdisciplinarity covers a broader range of research than would have been possible for Gašević et al. (2014).

Research Questions
We sought to examine the disciplinary distribution, composition, and change over time of MOOC research, by posing the following research questions:

Methods
In the sections that follow, we describe the methods used to conduct this study. Although an exhaustive description of the methodology used to systematically gather empirical literature on MOOCs is presented by Veletsianos and Shepherdson (2015), for transparency purposes, we summarize our approach here as well.
To discover empirical MOOC literature we conducted searches using the key words "MOOC" or "Massive Open Online Course" and limited our results to 2013-2015. These searches occurred between January 7, 2015 and February 1, 2015. Our inclusion criteria were the following: To be included, the identified document ought to focus on MOOCs and ought to have been (1) empirical, (2) published in a peer-reviewed journal, in conference proceedings, or in Educause Review, (3) published or was available online as in press between January 2013 and January 2015, and (4) written in English. Three researchers were involved in this process and they read each abstract to identify papers for inclusion. If no decision could be made by examining the abstract, the complete paper was downloaded and examined. The databases, search engines and journals used in this search were the following: Scopus, Journal of Online Learning and Teaching, Summon, Google Scholar, EdITLib Digital Library, and Educause Library. The Journal of Online 6 Learning and Teaching was searched because it was not indexed by Scopus, and because it was also examined by Liyanagunawardena et al. (2013) in their own systematic review of the MOOC literature. 120 empirical papers that fit the inclusion criteria were identified via these search strategies. At this point, a forward referencing search strategy and a reference list search strategy were also used to discover more literature. These strategies were also used by Liyanagunawardena and colleagues. The former strategy involved visiting Google Scholar, identifying the papers citing each one of the 120 papers, and examining all of them to deduce whether they fit the inclusion criteria. The latter strategy involved examining the reference lists of all papers published in 2015 (or that were in press at the time) to identify papers that we may have missed.

Data Collection
For the purposes of this paper, we collected the following data: •Names and affiliations of each author (n=462) who had contributed to at least one paper in the corpus consisting of empirical MOOC research published in 2013-2015; •Affiliation data reported by Gašević et al., (2014) (n=429); and •Names and affiliations of each author (n=56) who had contributed to at least one paper in the corpus identified by Liyanagunawardena et al. (2013).
The process we used to collect the names and disciplinary affiliations from our corpus and from the Liyanagunawardena et al. (2013) corpus was as follows: We identified each author who had contributed to at least one entry in our corpus and identified his/her affiliation(s). Where possible, the affiliation information was obtained from the corpus entries themselves (e.g., by way of the affiliations self-reported in journal articles). Where sufficient information was not included in the entry, we used web searches to obtain this information from university websites, social media profiles (e.g., LinkedIn), personal websites, or other available resources. We collected selfreported affiliations via publications, and Gašević et al. (2014) collected self-reported affiliations via grant proposals. When affiliation information was not available, we performed web searches to complete our data set.

Data Analysis
We used descriptive and inferential methods to answer the research questions. The methods used for each question are described below. After collation of author affiliation information was complete, two independent raters allocated each affiliation to one of a list of 12 disciplinary categories we devised during the collation process. We started with one discipline (i.e. the discipline reported by the first author in the first paper we examined) and added disciplines based on disciplines reported by the authors that were not already encountered. Coding resulted in a 78.4% inter-rater agreement (i.e., the raters initially independently coded 78.4% of authors to the same discipline). All disagreements were subsequently resolved through discussion between raters, and (where necessary) further investigation of author affiliations. The disciplinary backgrounds of the authors were then summarized using descriptive statistics.  Liyanagunawardena et al. (2013) as we had undertaken with our corpus. We then used two statistical measures to assess differences between the corpora.
First, we used a chi-square test to compare the relative frequency of each discipline in the two corpora. Second, we used Brillouin's index (Brillouin, 1956) to assess the overall interdisciplinarity of each corpus. This index is used as a measure of population diversity in the field of ecology (e.g., Peet, 1975), and has been used to assess interdisciplinarity in other research fields (e.g., Huang & Chang, 2011;Steele & Stier, 2000). We chose to use Brillouin's index rather than a measure of sample diversity because of the exhaustive nature of our literature search, suggesting that our corpus is more accurately characterized as a population (i.e., all empirical MOOC research published in English in the time period described) than a sample (i.e., a portion of the research fulfilling those criteria). 1 Larger values of the index indicate greater diversity/interdisciplinarity in a population.
8 Because Brillouin's index is affected by the size of the population being assessed, and our corpus and that of Liyanagunawardena et al (2013) were of different sizes, we needed to ensure that any differences we might find were not a result of different population sizes. We achieved this by using a Bootstrapping-style technique, similar to that suggested by Solow (1993) for use with the Shannon index. This involved combining the affiliations from the two corpora, randomly drawing two samples (one with the size of our corpus, and one with the size of Liyanagunawardena et al.'s corpus), and comparing the Brillouin indices of the two randomly-drawn samples. This procedure was repeated 10,000 times, allowing us to construct a distribution of difference scores comparing the two randomly-sampled Brillouin's index values. We could then use this distribution to determine the probability that the observed differences between the two corpora would result from differences in population size, absent any systematic differences in composition (i.e., we obtained a p-value from this process). Frequencies for each discipline in our corpus are displayed in Table 1

. Education and Computer
Science were the two most frequent affiliations, and between them accounted for over half of the authors. Industry, Social Sciences, Engineering, Medicine, and Administration each occurred in greater than 5% of the affiliations, with all remaining disciplines occurring with relatively less frequency. While Education and Computer Science were by far the most common affiliations for researchers in our corpus to possess, the field appears to be far from monolithic. 9 Table 1 Disciplines Represented in the Three Corpora

Discussion
We examined the extent to which interdisciplinarity is present in recent MOOC literature and whether interdisciplinarity in the MOOC literature has changed over time. We found three noteworthy results. First, if we sort the total number of times each discipline appears in descending order, we find that the field of education ranks first in both our corpus and the Gašević et al. (2014) corpus. In this case, our results are similar to those of Gašević and colleagues. Second, our statistical test showed that these two corpora differ significantly. In particular, our corpus had a greater representation of authors from Computer Science and the Gašević et al. corpus had a greater representation of authors from Education and Industry. In this case, our results differ from those of Gašević et al. Third, by comparing the interdisciplinary composition of our corpus with the interdisciplinary composition of the corpus used by Liyanagunawardena et al. (2013) we find that interdisciplinarity in MOOC research shows an increasing trend over the years. Gašević et al. (2014) noted that even though a common perception in the field is that the MOOC phenomenon is "driven by computer scientists" their results showed that more than 50% of the authors on all phases of the MRI grants were from the field of education. They suggested that this A positive trend uncovered by our investigation is that research on MOOCs appears to be more interdisciplinary than in the past, suggesting that the scientific complexity of the field is being tackled by a greater diversity of researchers. This suggests that even though xMOOCs are often disparaged for their teacher-centric and cognitivist-behaviorist approach (Hew & Cheung, 2014;Kennedy, 2014), empirical research on xMOOCs may be more interdisciplinary than research on cMOOCs, which are deemed to embrace more progressive learning designs (Rodriguez, 2012). We make this statement on the basis of the fact that two distinct "phases" of the MOOC phenomenon appear to exist (cMOOC and xMOOC, as identified by Ebben and Murphy, 2014)  Finally, although our corpus had a greater representation of authors from Computer Science than was apparent in earlier MOOC research, our results may have been influenced by disciplinary norms in publishing. Computer Scientists in general publish their research in conference proceedings which tend to be published more frequently than the journal publications that are generally sought by education researchers. As such, the disciplinary makeup of our corpus may reflect differences in publication cycles between the publication outlets preferred by different disciplines. Unfortunately, the data we have gathered do not allow us to examine this insight empirically; nonetheless, this may be a worthwhile topic for future research.

Limitations
Even though we used authors' disciplinary affiliations as a proxy of their home discipline and we believe this to be a strong proxy, one limitation facing this study is the fact that authors may draw from a variety of disciplines in their research. Because our data do not reflect the content of the articles, it is possible that more interdisciplinary work is present at the article level. It is important to note however, that there is no consensus in the literature on how to evaluate interdisciplinarity (Huutoniemi, Klein, Bruun, & Hukkinen, 2010;Wagner et al., 2011). Direct citation and co-authorship analysis for example, were two other approaches that we could have used to gauge interdisciplinary writing and changes over time (e.g., Huang & Chang, 2011 Finally, it is possible that narrower or coarser definitions of disciplines would have led to different outcomes of our statistical analyses. In coding the author affiliations, we selected disciplinary categories based on author self-reporting and guided in the first instance by the granularity of the disciplines described in Gašević and colleagues' (2014) paper on MRI submissions. While we applied this coding method consistently across the two main corpora studied (ours, and that of Liyanagunawardena et al., 2013) to ensure that the comparison was meaningful, we cannot preclude the possibility that the differences and similarities we found resulted from this chosen approach. Our methods provided sufficient power to obtain marginally significant results, but future research into this topic using alternative methods will contribute to an improved understanding of the topic.

Conclusion
Multidisciplinary, interdisciplinary, and crossdisciplinary research represent promising venues for studying digital learning. Prior research however, discovered that efforts at understanding digital learning via MOOCs were dominated by individuals affiliated with education (Gašević et al., 2014). Using a larger and broader corpus of data than prior research, we discovered that author disciplinary affiliations differed significantly: even though education researchers are still heavily involved in efforts to understand MOOCs, computer science researchers are more greatly 14 represented in the literature than in the past. In addition, we found that compared to research efforts to understand MOOCs conducted during 2008-2012 (Liyanagunawardena et al., 2013), research efforts during 2013-2015 were more interdisciplinary.
The burgeoning interest in digital learning, learning at scale, online learning, and other associated innovations presents researchers with the exceptional opportunity to convene scholars from a variety of disciplines to improve the scholarly understanding and practice of digital learning broadly understood. To do so however, researchers need to engage in collaborations that value their respective expertise and recognize the lessons learned from past efforts at technology- open to the perspectives that academic "immigrants" can bring to this field (cf. Nissani, 1997). At the same time, researchers from fields other than education need to recognize that education research and educational technology research have a long and rich history and an empirical basis that can inform design and research decisions. Working together, multidisciplinary, interdisciplinary, and crossdisciplinary teams of researchers can improve our scholarly understanding of teaching and learning.