Astronomy for Astronomical Numbers: A Worldwide Massive Open Online Class

Astronomy: State of the Art is a massive, open, online class (MOOC) offered through Udemy by an instructional team at the University of Arizona. With nearly 24,000 enrolled as of early 2015, it is the largest astronomy MOOC available. The astronomical numbers enrolled do not translate into a similar level of engagement. The content consists of 14 hours of video lecture, nearly 1,000 Powerpoint slides, 250 pages of background readings, and 20 podcast interviews with leading researchers. Perhaps in part because of the large amount of course content, the overall completion rate is low, about 3%. However, this number was four times higher for an early cohort of learners who were selected to have a prior interest in astronomy and who took the class in synchronous mode, with new content being added every week. Completion correlates with engagement as measured by posts to the online discussion board. For a subset of learners, social media like Facebook and Twitter provide an additional, important mode of engagement. For the asynchronous learners who have continuously enrolled for the past 15 months, those who complete the course do so quickly, with few persisting longer than two months. The availability of a free completion certificate had no impact on completion rates when it was added midway through the period of data analyzed in this paper. This experiment informs a new offering of an enhanced version of this MOOC via Coursera, along with a co-convened “flipped” introductory astronomy class at the University of Arizona, where the video lectures will be online and class time will be used exclusively for small group labs and hands-on activities. Despite their typically low completion rates, MOOCs have the potential to add significantly to public engagement with science, and they attract a worldwide audience. Astronomy for Astronomical Numbers: A Worldwide Massive Open Online Class Impey, Wenger, and Austin This work is licensed under a Creative Commons Attribution 4.0 International License. 58


Introduction
Higher education in the United States is in a state of flux and stress. Two and four year higher education institutions are still seen as a gateway to a better future and financial security, but the cost of a college education has increased by a factor of two (inflation-adjusted) over the past 30 years, and outstanding student loans of nearly a trillion dollars exceed credit card and automobile loan debt (Frye 2012). STEM (Science, Technology, Engineering, and Mathematics) education faces its own particular set of challenges. The science literacy of the average citizen is modest, leaving them poorly prepared to understand or judge technical societal issues like energy production, climate change, and genetic engineering (NSB 2012). Meanwhile, there is growing demand for graduates with technical skills and majors but the supply of trained graduates is inadequate. Completion rates for STEM majors are 40%, much lower than in non-technical subjects. The administration of President Obama has set a challenging goal of training 100,000 new STEM teachers and producing a million more STEM graduates (NSTC 2013).
The rapidly increasing capability of the Internet and the availability of online content are rapidly eroding the "sense of place" in a college education. Online learning is not new but it has started to become a major component of the educational landscape. The number of students taking one or more online class grew from 2 million to 7 million from 2002 to 2011 and over that period the percentage of total college enrollment online grew from 10% to 32% (Figure 1, Meeker and Wu 2013). Reticence over the quality of online instruction is steadily diminishing; in a 2012 survey, 2012 to 9 million in April, 2013. By lowering the bar for access to instructional materials and expert teachers, private companies like Coursera, edX, and Udacity are helping to democratize higher education. Companies like Udemy are making substantial contributions to professional development. For example, 75% of iTunes U downloads and 70% of Coursera users are located outside the United States (Masters 2011;Koller and Ng 2012). Coursera pioneered the delivery of online courses for transferable college credit. This opens up a new and powerful marketplace in higher education, where students can combine elements of a degree using local, face-to-face classes, and the best of the online offerings. MOOCs are often aimed at adult "recreational" or lifelong learners but a majority of the undergraduate population is now composed of students older than twenty-two, and these are often people with families and jobs. They will increasingly demand flexible delivery of the courses they are taking.
Along with spectacular growth and obvious promise, MOOCs have been subject to skepticism and pushback. Administrators are fearful of the threat to the current, unsustainable cost model for higher education but they also hope it may be a way to spread their "brand" to new markets, while educators are excited by the potential of the Internet as a learning medium but are suspicious that the quality of learning will be poorer when it is online. Research on MOOCs is in its early stages so there is a need for analysis informed by data gathered on classes that have firm pedagogical foundations. Some research questions are difficult to address due to privacy issues in the use of learner demographics an online behavior, and due to limited data analytics provided by some MOOC hosting companies. The factors that make for a successful MOOC are starting to emerge and increasing student engagement in the disembodied, online environment will be essential if MOOCs are to spread and flourish (Ramesh et al. 2014). A second, more profound issue is the distinction between engagement and learning. If the goal of a science class is to increase content knowledge and facilitate a high level conceptual understanding of science principles and process, which is much harder to facilitate and measure than mere engagement, particularly in an online environment (Reich 2015).

Astronomy: State of the Art
Astronomy: State of the Art is a MOOC specifically developed to study student participation in an online learning environment. The project aims to serve multiple audiences of learners. One is those self-motivated adults who take a free course out of interest in the subject and not to get a degree or for university credit; this is the traditional audience for a MOOC. Another is a set of college students who use the online environment for lectures and quizzes but whose classroom time is devoted to hands-on activities and group work; this is the "flipped classroom" model. A third we plan to serve in the future is composed of students remote from the university offering the MOOC, who are paying to take a completely online class for transferrable university credit; this is the part of the landscape with the potential to disrupt the traditional economic model of higher education. Our group is developing tools to work with all of these audiences, as far as possible using methods that are interactive, centered on learners, and grounded in research methods. Astronomy is an attractive vehicle for science literacy and outreach because it is a dynamic subject with new discoveries announced almost weekly. The web site for the class is http://www.udemy.com/astronomy-state-of-the-art/. Enrollment is free.
The first implementation of Astronomy: State of the Art was intended as an outreach vehicle for a broad public audience. All aspects of astronomy from planets to cosmology were included, but the emphasis was on exciting topics where the research progress has been rapid: large telescope construction, adaptive optics, Solar System exploration, exoplanets, black holes, dark energy and dark matter, and big bang cosmology. Udemy was selected as the provider of the free course in part because a course can be offered by a single instructor without an institutional commitment.
The privately-held company is smaller than Coursera and edX but has also been growing very rapidly, reaching 5 million students and 22,000 courses in early 2015 (www.udemy.com). Most Udemy courses are on technical subjects and for professional development and the courses are asynchronous in the sense that a student can sign up for the course at any time and follow the material at their own pace. The company doesn't offer the infrastructure for any testing beyond multiple choice questions. But the architecture and user tools for video content are robust and scalable and they were entirely adequate for this experimental course. The instructor for the course was a senior professor in the Department of Astronomy at the University of Arizona, who has taught many face-to-face astronomy courses at the introductory and advanced undergraduate levels (CI). He was assisted by a half-time staff educational technologist (MW), and a half-time undergraduate teaching assistant (CA), who were responsible for creating and editing video lectures and weekly assignments, as well as communicating with the students on the course web site and using social media.
Our goals in this project included creating the first MOOC that presented astronomy in terms of the cutting edge topics that define the research frontier. A MOOC can have a broader goal than an introductory college class, since it accesses a demographic that is typically more mature, with more people who take the class out of general interest than because of a university requirement.
Adult learners do not need or expect to master the formal apparatus of physics and mathematics needed to comprehend the universe. But they can be given a conceptually high level treatment that lets them share in the excitement of research on black holes and exoplanets and cosmology.
Another goal was to investigate the behavior of two cohorts: the pre-registered group who had a prior interest in astronomy and the continuously enrolling (and much larger) group encountering the class in the Udemy marketplace. A third motivation was to investigate correlations between various modes of engagement, such as accessing course materials and usage of social media.

Course Content
This course has gone through two modes: one "active" and a subsequent "passive" mode. The course was initially designed as a seven-week class to be offered beginning in March, 2013. In this synchronous early phase, a series of lecture videos was released each week, along with self-guided assignments and additional material for students to explore on their own. The additional content consisted of reading material and podcast interviews with scientists at the University of Arizona engaged in research related to the weekly topic. The core content is a series of video lectures that total about 14 hours. Video was primarily filmed as first person "talking head" of the instructor shot against a green screen so that backgrounds could be added later, interspersed with relevant animations, images, and video clips. The footage was edited and rendered in Final Cut Pro 10.
There were between five and seven videos for each week's topic; each video lasted an average of ten minutes with the shortest being around five minutes and the longest less than 20 minutes. All videos were subsequently posted on YouTube so they would be available for people not enrolled in the Udemy course (https://www.youtube.com/user/AstronomySOTA).
In addition to the videos, each of the seven major sections of content had a brief overview video, lasting 2 to 3 minutes. Additional material for each section included a downloadable pdf file of a Powerpoint slide presentation consisting of 80-130 slides, a reading 20-30 pages long taken from one of the instructor's popular science books, and links to several 40-minutes podcast interviews with researchers and faculty at Steward Observatory and the Department of Astronomy. A quiz was created for the first section, and the rest of the assignments consisted of participation in the citizen science of the Zooniverse project (http://www.zooniverse.com). Participants could also communicate with the teaching team through question and answer and discussion threads on the course website, and through social media on Twitter (https://twitter.com/AstronomySOTA) and Facebook (https://www.facebook.com/AstronomySOTA). In addition, there have been live video discussions where students can have questions answered in real-time by the instructor. These sessions occurred weekly during the first seven weeks of the class, decreasing to once every few weeks subsequently.
An astronomy MOOC does not have the same instructional goals or technological underpinnings as a conventional class where students are evaluated on both content and physical principles. As a result, the video lectures and ancillary material did not go into detail on some core topics in a college-level astronomy class: positional astronomy, radiation physics and the inverse square law, atomic structure, comparative planetology, the Hertzsprung-Russell diagram that describes stellar evolution, and the classification and structure of galaxies. Table 1 provides an outline of the course content. Inspecting Other Galaxies Evidence for dark matter in all galaxies, evidence for black holes in the centers of galaxies. Active galactic nuclei, structure formation, and types of galaxies. 7 Examining the Cosmos Hubble expansion, dark energy, evidence for the big bang, inflation and the early universe, and the multiverse scenario. 8 Uncovering Evidence for Life The origin and unity of terrestrial biology, extremophiles, the search for biology and biomarkers beyond Earth, the Drake equation, and the search for extraterrestrial intelligence.

Participants and Participation
Astronomy: State of the Art went "live" on March 26, 2013. To "prime the pump" for the new class, the course was announced on the Teach Astronomy web site, which was created by the instructor in 2011 to support General Education astronomy classes at the University of Arizona (http://www.teachastronomy.com). That site was at the time getting 300 unique visitors a day, the majority of which were not college students but were members of the general public with an interest in astronomy. The instructor also publicized the course in emails to over 600 astronomy instructors in the United States, and with flyers mailed to over 800 amateur astronomy clubs in the United States. These initiatives, plus use of social media to reach astronomy fans, probably reached as many as ten thousand people in the run up to the launching of the class, and by the time the class started over 700 had pre-registered on the Teach Astronomy web site. Although Udemy offers limited demographic data to instructors, a snapshot of the enrollment habits of the class was obtained by viewing the home pages of 500 participants. This showed that, at the time they first enrolled for Astronomy: State of the Art, 45% of the participants had only enrolled for this astronomy class, so it was their first exposure to a MOOC. But there was a long tail of people who were enrolled in many classes on Udemy. About 10% were signed up for 50 or more classes and a handful had signed up for several hundred classes. Since no financial commitment is required for the many free classes on Udemy, this illustrates the phenomenon where some people engage in "stamp collecting," or signing up for many classes that they have no intention or time to complete. Again, the more committed nature of the early cohort stands out.
Among the 400 students who had enrolled before the class formally started, 85% were only signed up for Astronomy: State of the Art and fewer than 10% had signed up for more than one other Udemy class.

Completion and Engagement
The classic MOOC sets the bar low for learners since there is no cost and no grades, with the corollary that the level of engagement can also be very low. Astronomy: State of the Art has a large amount of content, amounting to about 14 hours of video, 800 lecture slides, 500 pages of readings, and 15 hours of podcasts. The most sobering statistics come from completion metrics ( Figure 5). Udemy measures completion as the fraction of the content viewed or accessed. For video lectures this is the percentage of the total video played, bearing in mind that the video being streamed does not prove that it was actually watched. For lecture slides and readings, an algorithm converts the content into equivalent minutes of engagement. In this course, podcasts were accessed by links outside of Udemy so they don't count as content in terms of completion.
Just over half of the people who enroll never access the materials, consistent with statistics for many MOOCs. Of those accessing the class materials, 3.2% complete all of them, 4.3% complete half of them, and 7.3% complete a quarter of them. The rate of completion has remained fairly consistent over time with a completion rate of 25 students per month, or just a few percent of the number of students enrolling in the course each month. The astronomical number enrolled is largely misleading; participation statistics bring it down to Earth.
The emerging research literature on MOOCs is beginning to identify the factors that affect completion and retention (Adamopoulos 2013). Factors causing high attrition include lack of time, lack of motivation, feelings of isolation and lack of interactivity online, poor preparation and skills, and perceived hidden costs to MOOCs (Khalil and Ebner 2014). Completion rates should only be evaluated in the light of students' self-reported intentions, which also affects strategies for personalizing a MOOC (Reich 2014). One study of dropouts from a MOOC found that the time commitment required and a lack of challenging activities contributed to the poor completion rate (Gutl et al. 2014), but in common with other studies also determined that most students never had any intention of completing the MOOC when they enrolled.
Course completion rates were noticeably different between the initial cohort who signed up during the "active" or synchronous portion of the course, and those who signed up during the "passive" phase. Of the initial cohort of 1,991 students who joined the class during the "active" mode, 7.5% completed the course. The cohort of students who joined the course during the "passive mode" has lower completion rate of 3.3%. In addition, the time to completion is noticeably different between cohorts ( Figure 6). Students in the first "active" mode cohort took around seven weeks (median=51 days, mode=45-49 days) to complete the course content. This should come as no surprise, since the course content was released on a weekly basis for seven weeks. Students in this first cohort who joined slightly after the start date took a little less time to finish the course since the content was already posted when they arrived. By contrast, students in the second "passive" mode cohort have generally completed the course quickly (median=33 days, with the largest group of students in this second cohort completing the course content in less than five days (mode=0-4 days). What we find is that completion times for the first cohort are clustered more closely together, compared with students in the second cohort who were able to pace themselves, and completion times are more varied. Even though there were 10 days in between the day the final "active" content was posted, and the cutoff date for the first cohort, no one in the first cohort finished in less than 25 days, whereas with the self-paced students in the second cohort, nearly 50% of the students completed the course in that amount of time.
Most people access the content sequentially, so a much higher fraction views the first and early modules of the class than views the later modules (Figure 7). The last two graphs can be used to loosely categorize students of this MOOC into three groups: "phantoms," who enroll but never access the materials, "dabblers," who view a handful of video or completion a small fraction of the course, and "persisters," who complete the course or view most of the course materials. In fact, research on one biology MOOC with Coursera showed that first week performance and social interaction is a strong predictor of final performance (Jiang et al. 2014). Even though the participation and completion rates are low, the high aggregate engagement can make a MOOC an effective vehicle for outreach (Figure 8). Early adopters were watching 90 minutes or more of content, and even the less engaged later cohort is on average watching 40-50 minutes of content.
The aggregate after just over a year is nearly 400,000 minutes or 6,700 hours, equal to 10,000 people watching a typical public lecture on astronomy, involving reaching audiences who might not otherwise be able to enjoy cutting edge science. MOOCs will undoubtedly play a role in combatting low levels of science literacy in the United States (Impey et al. 2011(Impey et al. , 2012.
While there is a natural explanation for the high degree of participation of the early students who where people act initially out of curiosity and with no intention of a major commitment.
Completion rate in this MOOC is correlated with a high prior interest in astronomy, which is unsurprising. But it is also correlated with engagement through both synchronous (live sessions) and asynchronous (Q&A threads, social media) channels, which suggests that the video lectures and online quizzes that form the pedagogy of most MOOCs are not sufficient. Learners will do better when they have multi-modal engagement with the material and the instructor.
We plan a second experiment in 2015 with Coursera, with which the University of Arizona has recently become a partner. Coursera has superior infrastructure than Udemy for online evaluation and testing, and they have facilitated a growing base of instructional research (Koller et al. 2013).
The Udemy MOOC will continue to be open to anyone who is interested. For Coursera, several additional audiences with professional or personal motivations for deeper engagement will be recruited. High school astronomy teachers will be recruited from a national pool of about 1,500 to enhance their content knowledge and for professional development credit (Berendsen 2005).
Also, there are about 800 astronomy instructors at community colleges and two-year colleges, who teach a quarter of a million students per year (Fraknoi 2001). Most do not have a degree in astronomy; they will value the content rigor and research currency of this MOOC. The most dedicated members of these cohorts will be recruited as citizen scientists for projects in the course, and as learning assistants in subsequent offerings of the MOOC. There is very little control over the motivations of students who participate in a MOOC as a free-choice learning experience, just as museums have little control over the motivations of their customers (Bell et al. 2009;Donovan and Woods 2001). But it is possible to control the degree to which participants' expectations are met. One eventual goal of this project is to understand the motivations of MOOC participants, to shape their expectations for how the MOOC will serve their education needs.
In the future, the classic MOOC, which primarily has an outreach function, will be co-convened with "The Physical Universe," a Natural Sciences course taught at the University of Arizona that satisfies a General Education requirement for non-science majors. Using the same core material as Astronomy: State of the Art (with additional modules on the physics of radiation, atomic structure, energy, and gravity that are not necessary for the informal learners), the local course will employ a "flipped" model where the students access lectures and podcasts online but are in a face-to-face classroom two times a week for labs and hands-on activities, lecture tutorials, group discussions, and other research-validated tools for enhancing learning. A key goal in developing a "next generation online science class" is to migrate the most pedagogically effective methods of the face-to-face classroom to a completely online environment (Prather et al. 2011). A flipped or hybrid model gives students flexibility, uses the online medium for the aspects of instruction where interaction with an instructor isn't required, and optimizes the scarce resource of time in a large classroom (Figure 10). The eventual goal is to recruit undergraduates from anywhere in the United States and award them transferrable credit for completing the class. The audience will be students who need a science class as a graduation requirement or who are piecing together a degree with a mixture of online classes and face-to-face classes from their local institution.    Astronomy -State of the Art. Nearly 7,000 out of the nearly 13,000 enrolled never access the materials so they are "phantom" students. A large number access less than 20% of the materials; enough to get a flavor of the subject. A small percentage complete more than half of the materials, and the last bin shows an increase from the dedicated cohort who persist to the end of the course.   The early cohort consisting primary of amateur astronomers and educators had a higher level of engagement that all the cohorts that followed. The reason for the decline in the past six months is not clear. (b) Total content consumed, showing the high engagement of the early cohort, and then a continuing rise as the enrollment grows but with a lower typical level of engagement. Figure 9. Total number of page "likes" per day. Rate of "likes" closely follows the rate of student completion. Figure 10. A schematic view of the combination of informal and formal learning that is possible when online content is used to serve both audiences. The MOOC is a purely online experience, is offered free, and is not for college credit. Meanwhile, undergraduate students needing a General Education science requirement at the University of Arizona and elsewhere view online lectures, and use the face-to-face time for active engagement, with learning facilitated by peer assistants from within the class.