In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs)

Practical Application Citation Patrick R. Lowenthal, Ph.D. Boise State University, College of Education Charles B. Hodges, Ph.D. Georgia Southern University, College of Education In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs) The concept of the massive, open, online course (MOOC) is not new, but high-profile initiatives have moved MOOCs into the forefront of higher education news over the past few years. Members of institutions of higher education have mixed feelings about MOOCs, ranging from those who want to offer college credit for the successful completion of MOOCs to those who fear MOOCs are the end of the university as we know it. We set forth to investigate the quality of MOOCs by using the Quality Matters quality control framework. In this article, we present the results of our inquiry, with a specific focus on the implications the results have on day-to-day practice of designing online courses. MOOCs have been in the higher education news often in the last few years. Many university faculty members have viewed them as a threat to traditional course offerings or simply another online learning fad. The authors posit, however, that MOOCs are an opportunity to rethink how we design and teach online courses. In this analysis of six MOOCs, the authors highlight the key differences between the type of learning experiences that some MOOCs offer to learners. They also explain how MOOCs are not drastically different in design than many on-campus courses taught at universities, and provide suggestions for how some MOOCs could be revised to improve their design with respect to the popular Quality Matters online course design framework. Lowenthal, P.R., & Hodges, C.B. (2015). In search of quality: Using Quality Matters to analyze that quality of massive, open, online courses (MOOCs). International Review of Research in Open and Distributed Learning, 16(5), 83-101. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2348/3435. For more information, contact Charles B. Hodges chodges@georgiasouthern.edu February 2016


Introduction
During the past few years, massive open online courses (MOOCs) have become one of the most talked about trends in higher education (e.g., EDUCAUSE, 2012;Markoff, 2011;Rushkoff, 2013).
In fact, 2012 was described as the year of the MOOC (Pappano, 2012). A MOOC is basically a large, open, online course. However, in practice, "MOOCs" differ in many ways. For instance, MOOCs vary in size and in their degree of openness. MOOCs can range from a few hundred students (e.g., Downes, 2011) to thousands (e.g., Markoff, 2011). Some MOOCs are completely open, whereas others limit enrollments (Fain, 2013). Most MOOCs are not offered for college credit; however, there are recent trends to offer college credit for the successful completion of 84 certain MOOCs (Coursera, 2013;Kolowich, 2013). Finally, while some MOOCs intentionally place often famous instructors at the center of the learning experience (sometimes called cMOOCs), other MOOCs focus more on the learners and the connections they can make with others, sometimes called xMOOCs (see Kilgore & Lowenthal, 2014). Despite this diversity, one constant across nearly every MOOC appears to be low completion rates; in other words, while hundreds or thousands might sign up for a MOOC, less than 10% tend to complete them (Jordan, 2014).
Despite reports of low completion rates (Pretz, 2014;Rosen, 2012;Yang, Sinha, Adamson, & Rose, 2013)-which some argue is not necessarily a bad thing and others point out is a misleading metric to begin with (Carey, 2013;DeBoer, Ho, Stump, & Breslow, 2014;Reich, 2014)-the fascination with MOOCs persists. The alluring promise of MOOCs and what keeps people interested in them is their ability to offer free or low-cost education to anyone, anytime, anywhere, and on a massive scale (Delbanco, 2013;Jordan, 2014;Yang et al., 2013). Thus, companies like Coursera, edX, and Udacity are striving to make MOOCs a household name and, ultimately, to challenge how and where people learn.
Many people, however, remain skeptical of MOOCs (Carr, 2012;Delbanco, 2013;Mazoue, 2013;Sharma, 2013). Among other things, they question whether meaningful and effective educational experiences are even possible in large enrollment courses (see Glass & Smith, 1979;Lederman & Barwick, 2007;Mulryan-Kyne, 2010), let alone completely online courses with massive enrollments (see Leddy, 2013;Mazoue, 2013;Rees, 2013). Therefore, if MOOCs are going to persist and become more than a passing fad, they must be shown to meet some of the same quality standards that traditional online courses are expected to meet. The problem, though, is that very little research has been done to investigate the instructional quality of MOOCs. Due to this problem, we decided to investigate the design of MOOCs as determined by certain, accepted online course quality frameworks. We began this study with an assumption that MOOCs, just like formal online courses, are not inherently good or bad. Further, there are likely some things that members of the academy can learn (both good and bad) from analyzing MOOCs. In the following article, we report the results of using the Quality Matters framework, a widely adopted approach to assessing the quality of online course design (Quality Matters, 2011;Shattuck, 2007), to evaluate the quality of six randomly selected MOOCs. We conclude this article with a discussion of the implications this research has on practice.

Theoretical Framework and Related Review of Literature
Online courses differ from face-to-face courses in certain ways (Inglis, 2005). For instance, the instructional content and instructional strategies in online courses can be designed and developed before a course is ever offered. Thus, when people talk about the quality of online courses, they often differentiate between how a course is designed and how it is taught. While a bad instructor can arguably find a way to ruin a well-designed online course (e.g., by being non-responsive), a well-designed online course is generally recognized as a hallmark of online course quality. With this in mind, in the following section, we briefly summarize literature about online course quality and MOOCs to establish a way to discuss the quality of MOOCs.

Online Course Quality
Despite the increase in enrollments in online courses, many people remain skeptical of online learning (Allen, Seaman, Lederman, & Jaschik, 2012;Bidwell, 2013;Jaschik & Lederman, 2014;Parker, Lenhart, & Moore, 2011;Public Agenda, 2013;Samuels, 2013). They question whether online learning is as good as face-to-face instruction (Allen et al., 2012;McDonald, 2002)specifically, whether students learn as much in online courses as in face-to-face courses. This skepticism has fueled hundreds of "comparison studies" seeking to equate learning outcomes between face-to-face and online learning (Bernard et al., 2004;Meyer, 2002Meyer, , 2004Phipps & Merisotis, 1999). The majority of these studies have resulted in no significant difference (Bernard et al., 2004;Russell, 1999). The no significant difference finding is often interpreted to mean that online courses are no worse, or no better than, more traditional face-to-face courses. Despite volumes of similar research, many people-even those who previously have taught online-still question whether students learn as much online as they do face-to-face (Jaschik & Lederman, 2014).
Many researchers have come to the conclusion that comparison studies like these are a waste of time because researchers cannot control for extraneous variables that may impact student achievement (Bernard et al., 2004;Lockee, Moore, & Burton, 2001;Meyer, 2004;Phipps & Merisotis, 1999;), instructors teach differently online (Palloff & Pratt, 1999;Salmon, 2000;Wiley, 2002), and this line of inquiry typically places face-to-face instruction inappropriately as the gold standard (Duffy & Kirkley, 2004;McDonald, 2002).
Others remained skeptical of online learning simply because they believe that education is inherently a face-to-face process (Bejerano, 2008;Edmundson, 2012;Kroll, 2013). They questioned the ability of people to establish presence online and the quality of teacher-to-student and student-to-student interaction. There also is a related, deep fear that teachers will eventually be replaced by computers (Lytle, 2012). Still others fear that cheating and plagiarism are rampant in online courses (Gabriel, 2011;Wilson & Christopher, 2008).

Measuring Online Course Quality
This skepticism of online learning led to a number of quality assurance programs for online courses. The following are a few of the more popular programs in the United States: • California State University Chico developed the Rubric for Online Instruction (see: http://www.csuchico.edu/roi/the_rubric.shtml); • iNACOL developed multiple standards and rubrics to measure quality course design, instruction, and programs (see: http://www.inacol.org/resources/resourcesearch/?resource_topics=16);

86
• Online Learning Consortium (previously, Sloan-C) developed the five pillars of quality framework for quality online course design (see: http://onlinelearningconsortium.org/about/quality-framework-five-pillars/); and, • Quality Matters developed a faculty-centered, peer review process focused on eight standards to ensure quality course design (see: https://www.qualitymatters.org).
iNACOL is the only quality assurance/standards framework listed above that focuses on both online teaching and online course design. While other rubrics focus on measuring quality teaching online, the most popular quality assurance frameworks focus on online course design but not online teaching.
Quality Matters is one of the most popular and widely used quality assurance frameworks in the United States. Quality Matters began under a Department of Education Fund for Improvement of

Post-Secondary Education (FIPSE) grant. Quality Matters (QM) is now an international
organization focused on improving the quality of online courses at the K-12, Higher Education, and Professional Education levels. There are currently more than 800 QM subscribers (Shattuck, Zimmerman, & Adair, 2014). QM is a peer review and faculty development process that is centered on the following eight general standards: Each of these general standards has a number of related and more specific sub-standards. While each subscriber arguably could use QM differently, the formal QM process involves taking a course that has been taught before and having it reviewed by three peer reviewers (which must include one master review, one subject matter expert, and one external reviewer) to see if each standard has been met and then revising the course to meet any standards that were not met. The process is not perfect, but it is a widely accepted model for designing quality online courses (Shattuck, 2012).

MOOCs and Quality
While MOOCs date back to 2008, their widespread popularity is relatively new. Thus, not surprisingly, the majority of articles written about MOOCs come from the popular press. Except for a couple of special issues of journals focused on MOOCs (e.g., Journal of Online Teaching and Learning and Distance Education), relatively little formal research has been conducted on MOOCs, compared to all of the hype surrounding them. Most of the research that has been conducted has focused on enrollments, or some element of the learner experience in MOOCs.
Very little research has focused specifically on the curriculum or instructional design of MOOCs (Margaryan, Bianco, & Littlejohn, 2015), which is exactly the focus of the present study.
Our experience indicates that there are two basic types of MOOC critics. The first type consists of those who are critical of any type of online learning. These people generally focus on the problems with online learning addressed earlier (e.g., lack of presence, fear of cheating, fear of computers replacing teachers). The second type of critic is the people who generally support online education, but remain critical, or at least skeptical, of MOOCs. Generally speaking, these critics tend to question the pedagogical approach of MOOCs and, therefore, in turn question using an institution's limited resources on offering MOOCs, as well as any attempts to offer college credit for the completion of what they perceive as inferior (pedagogically speaking) online courses. The second type of critics see MOOCs as a step back in time (pedagogically speaking) by simply focusing on the transmission of content (Anderson, 2013;Conole, 2013). Anderson (2013) pointed out, though, that this type of pedagogy is still the norm in many classrooms (online or not) at all levels of education. Further, there are many different types of MOOCs (Adair, Alman, Budzick, Grisham, Mancini, & Thackaberry, 2014). For instance, Anderson (2013) argued that despite attempts to act like all MOOCs are the same, "different MOOCs [as in any form of educational delivery or organization] employ different pedagogies" (p. 4).
Faced with the hype of MOOCs, trends to offer college credit for MOOC completion, and the general skepticism of online learning in general-and MOOCs in particular-we decided to actually investigate the quality of MOOCs. It seemed logical to apply a commonly adopted quality control framework like QM to investigate the quality (at least in terms of course design) of MOOCs. If colleges and universities are going to offer credit for the completion of some of these courses, then, generally speaking, MOOCs ideally should score well on QM reviews.

Method
Many of the high-profile MOOC initiatives involve some of the leading MOOC providers like Coursera, edX, and Udacity. Therefore, we began our inquiry interested in the quality of so-called xMOOCs from these three companies. In an effort to narrow the scope and compare similar MOOCs across providers, we decided to look at only science-, technology-, engineering-, and mathematic-(STEM) focused MOOCs. We then went to Coursera, edX, and Udacity's websites and randomly selected six STEM-focused MOOCs-two from each of these three providers-for analysis.

88
The six identified MOOCs were analyzed using the 2011-2013 edition of the Quality Matters Rubric Standards with Assigned Point Values (Quality Matters, 2011), which basically involves a type of content analysis by three different reviewers using a standard coding scheme. QM has a rubric for Continuing and Professional Development that would be appropriate to use on MOOCs . However, we intentionally chose to use QM's higher education rubric rather than the continuing and professional development focused rubric because of the increased initiatives about offering college credit for MOOC completion. In other words, a MOOC should score as well as a traditional online course if it is going to be worth college credit.
Three trained Quality Matters peer reviewers (the first author of this article and two contracted QM master reviewers) analyzed each of the MOOCs using the Quality Matters 2011-2013 rubric.
Following a standard QM review process, a standard was met if two of the three reviewers marked it as met; a course passed a review if all of the essential standards were met and an overall score of 85% or higher was achieved. The reviewers shared their evaluations and arrived at consensus.
Notes generated during the reviews served as a secondary data source.

Results and Discussion
A course must meet all of the essential standards and get a score of 85% to pass a review. None of the six MOOCs passed the initial Quality Matters review (see Table 1). However, one Coursera MOOC and one edX MOOC scored very well overall in terms of total points (see Table 1). The Udacity courses, on the other hand, (which were both self-paced courses offered year round) performed the worst.

The self-introduction by the instructor is appropriate and available online.
Standard 3: Assessment and Measurement (includes 5 review standards) 3.1 The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources.
3.5 Students have multiple opportunities to measure their own learning progress.
Standard 4: Instructional Materials (includes 6 review standards) 4.3 All resources and materials used in the course are appropriately cited.

4.4
The instructional materials are current.

The instructional materials present a variety of perspectives on the course content.
Standard 5: Learner Interaction and Engagement (includes 4 review standards) 5.2 Learning activities provide opportunities for interaction that support active learning.
Standard 6: Course Technology (includes 5 review standards) 6.2 Course tools and media support student engagement and guide the student to become an active learner.
6.3 Navigation throughout the online components of the course is logical, consistent, and efficient.
6.4 Students can readily access the technologies required in the course.
6.5 The course technologies are current. are designed as professional development experiences. As such, these "courses" were never designed with supporting a learner over time the same way that college courses are designed. We contend, though, that many MOOCs easily could be updated to address Standard 7 and, therefore, (at least in terms of this sample) increase their overall score and, in turn, come much closer to passing a QM review. For example, two of the six MOOCs (i.e., 30% of MOOCs) in our sample would have an overall score over 85% (see Table 2) and simply have a handful of essential standards to address to pass a QM review (see Figure 1).

91
While the QM standards and review process is useful and widely adopted, one must remember that: A point system is associated with the specific standards, with some standards assigned more value than others. Those assigned with the greatest number of points are considered essential standards that must all be satisfied for a course to meet overall Quality Matters standards. (Quality Matters, 2014, p. 2) Regarding the MOOCs in this sample, none of the MOOCs scored well on Standard 2, which focuses on learning objectives. While learning objectives are important and serve a purpose, some might argue that explicitly sharing learning objectives with learners (i.e., making them visible) is not the hallmark of a quality course. In fact, depending on a designer's position on learning, instructional objectives may have different purposes, or may not be appropriate at all (Ertmer & Newby, 1993). Some of the small amount of research conducted on QM suggests that students and faculty view the importance of learning objectives differently-with students valuing clearly articulated objectives compared to faculty who focus on measurable objectives that are aligned with assessments (see Ralston-Berg, 2014;You, Hochberg, Ballard, Xiao, & Walters, 2014). The failure of meeting Standard 2 is more about transparency than whether or not the course was designed around clear learning objectives. For instance, the courses could have been designed to meet clear and measurable learning objectives, but the course itself does not clearly communicate these objectives (in a traditional format) to the learner. Part of the issue with QM's heavy focus on clearly stating learning objectives is that it ignores another school of thought in the field of instructional design that suggests learning objectives are meant more for instructional designers to aid in designing instruction and should not be shared with learners-at least not in the traditional format that includes a measurable verb, condition, and criteria (see Allen, 2003;Batchelder, 2009;Bean, 2009;Moore, 2007). Further, but related, more research needs to be conducted on the difference between standards that are worth 3 points vs. those worth 2 points or 1 point. For instance all of the learning objectives standards (Standard 2) are seen as essential and therefore worth 3 points, but other important standards like Learner Activities and Learner Interaction (Standard 5) have fewer overall substandards and overall points. Yet quality learner activities and learner interactions are arguably even more important components of a highquality course than clearly stated learning objectives. For instance, one could easily write clear and measurable learning objectives that align with watching recorded lectures and taking multiple-choice tests. In the end, however, MOOCs are not known for high-quality learning activities and learner interactions so the greater point is simply that a course can be designed in a way to meet QM standards but still be a relatively boring course. Thus, perhaps more differentiation needs to be made between a quality course and a truly high quality or exceptional course.

Implications
Given all of the hype surrounding MOOCs, some of the recent efforts to offer college credit for the completion of MOOCs (see Adair et al., 2014), and many people's skepticism of MOOCs, we decided to investigate the quality of massive open online courses to identify whether or not they can meet the same quality standards as traditional online courses. We began our inquiry assuming that MOOCs cannot be as bad as many of our colleagues feared, but we were unsure how they would compare with traditional for-credit online courses. We decided to use the popular Quality Matters (QM) framework to assess the quality of MOOCs. While none of the MOOCs passed an informal QM review, to our surprise some of the MOOCs analyzed scored very well and, with some minor revisions (which is part of the QM process), two of the MOOCs could pass a QM review and, therefore, be considered high quality online courses. This suggests that MOOCs have the potential to be high quality online courses, at least in terms of course design. But high quality (designed) online courses, in our experience, do not simply happen on their own. Instead, they are the result of the intentional application of a systematic process of design and evaluation aimed at improvement over time. It is unclear at this point how much effort the major MOOC providers will spend improving their online courses over time. It is important to note that our findings suggest something different than the work of Margaryan et al. (2015). Margaryan et al.'s analysis of the instructional quality of MOOCs found that while MOOCs were well organized, the instructional design quality was low. Therefore, additional research, using multiple quality assurance frameworks is needed to further investigate the quality of MOOCs.
Keeping our results in perspective is important. We only investigated six MOOCs. As such, our results should not be generalized to apply to all MOOCs. A larger sample might reveal that the MOOCs that scored well in our sample were an anomaly or that at the same time, a larger sample might have revealed that some MOOCs may score even better on a QM review. Further, we intentionally did not review any cMOOCs. We were more interested in analyzing the MOOCs being developed by the large MOOC providers-the same types of MOOCs for which many are contemplating offering credit for completion. Our inquiry was exploratory and descriptive in nature. We never intended to conduct an exhaustive review of all possible MOOCs. Instead, we were simply interested in addressing a gap in the literature about the overall quality of MOOCs and seeing how a random selection of MOOCs performed on a process used by colleges and universities throughout the United States. With this in mind, our results suggest a few important 93 considerations, especially in light of continued efforts to offer college credit for completing different MOOCs.
First, despite people's biases or the little research done on MOOCs and course quality (e.g., Margaryan et al., 2015; or see Adair et al., 2014 for references to other unpublished research), MOOCs are not all inherently bad or poorly designed. Some MOOCs can be "quality" courses with just a few minor course design changes based on the QM standards. Perhaps even more importantly, MOOCs are often described as poor learning experiences because of the little-to-no teacher-student interaction, the large class sizes, and the greater emphasis on lecture and testing that is common in these types of courses. But as mentioned earlier, this type of pedagogy is still the norm in many classrooms (online or not) at all levels of education (Anderson, 2013). Further, as heretical as it may seem, some literature suggested that teacher-student interaction is not always needed. For instance, Anderson's Equivalency Theorem posits that "deep and meaningful learning is supported as long as one of the three forms of interaction (student-teacher; studentstudent; student-content) is at a high level" (Miyazoe & Anderson, 2013, Definitions and Concepts section). Miyazoe and Anderson argued that based on this theorem, high levels of quality learning can take place in MOOCs even when certain types of interaction are limited. We are not arguing for the elimination of teacher-student interaction or arguing for large online courses, but simply pointing out that discounting the quality of courses when they mirror common practices elsewhere in colleges and universities might not be fair.
Second, not all MOOCs are designed to offer the same type of learning experience. For instance, some MOOCs are intentionally designed to be self-paced (e.g., Udacity MOOCs) where a learner can start at any time; other MOOCs adhere to a strict schedule and are facilitated in some manner. Further, even describing MOOCs as either xMOOCs or cMOOCs is too limiting (Conole, 2013) because there are even differences across so-called xMOOCs and cMOOCs. These differences are rarely talked about in the larger discussion about MOOCs.
Third, MOOCs (especially those by Coursea, edX, and Udacity) often have a similar instructional approach that relies heavily on professionally produced videos, readings, and quizzes, all with minimal instructor contact. Instructional designers and faculty alike can learn a lot from highly produced MOOCs in terms of effective design and production. At the same time, this approach suggests that there might be limitations to the Quality Matters rubric if such simply designed courses can pass a review and be deemed "quality" courses. The Quality Matters rubric might focus too much on the basics (e.g., clear learning objectives) and not enough on instructional approaches for active engagement, communication, and collaboration.
Our investigation suggests, among other things, that MOOCs are an opportunity to rethink how we design/teach online courses and should be seen for what they can add to the online learning landscape rather than as a complete threat or gimmick. MOOCs also have the ability to spark technological innovation; for example, some MOOC providers are pushing some boundaries with how threaded discussions and videos are used in learning management systems (e.g., with their up-voting discussions or video players).

94
While the results from this study cannot be generalized to all MOOCs, they do provide some needed data about the nature of MOOCs. This data can then serve to advance our discussions about MOOCs beyond mere hyperbole. For instance, can MOOCs meet the same design requirements as other for credit online courses? If not, where do they fall short? Questions such as these are important as institutions struggle with offering credit for completing MOOCs.
Finally, this investigation also has the possibility of informing and further evolving online quality assurance systems like Quality Matters by analyzing MOOCs-a non-traditional format of online learning. As Anderson (2013)