MOOC Integration into Secondary School Courses

We investigated how high school students taking a university preparatory economics course would engage with the learning and assessment components of a Behavioural Economics MOOC that was integrated into their school-based course. Students were divided into two groups, MOOC-only, with no teacher support, and blended-mode, with weekly tutorials. MOOC only students scored slightly lower on a teacher designed knowledge test but scored slightly higher in a MOOC test. Although the MOOC-only students watched more unique videos, the blended-mode students stayed more on-track with the MOOC. The blended-mode students showed more persistence in retaking quizzes, yet they scored lower than the MOOC-only students. MOOC Integration into Secondary School Courses Najafi, Evans, and Federico Vol 15 | No 5 Nov/14 307 Introduction Massive online open courses (MOOCs) have been used in various ways often in the form of a flipped classroom, to complement traditional classroom teaching through integration of a whole course or specific parts of a course both in K-12 and in higher education. High school students, for example, could use MOOCs for university preparation in the absence of available face to face or online courses or in addition to them (Vihavainen, Luukkainen, & Kurhila, 2013). Furthermore, contents of a live or an archived MOOC could be integrated into an existing course in a hybrid format (Bruff, Fisher, McEwen, & Smith, 2013). In this research we investigated how high school students taking a university preparatory economics course would engage with learning and assessment components of a Behavioural Economics MOOC that was integrated into their school-based course. Of specific importance to this research was probing any potential difference in students’ use of various components of the integrated MOOC depending on receiving supplementary instructional support or studying on their own. For this purpose, three weeks of the "Behavioural Economics in Action" (BE101x) MOOC were chosen for integration into the school-based course "Analyzing Current Economic Issues" (ACEI). Twenty nine students from a Canadian independent university preparatory school who had enrolled in ACEI were randomly assigned to two groups for the BE101x integration period with one group studying on their own and the other receiving weekly tutorial sessions from their classroom teacher. Here, we report on how students in the two groups engaged with the learning and assessment components of BE101x and persisted throughout the three weeks of integration. Also, we compare students’ outcomes based on pre-post BE101x-integration content knowledge surveys and BE101x assessments. Review of Related Literature Research on online university preparatory courses, MOOC-like university preparatory courses, and MOOC integration in credit courses informed our study. Online University Preparatory Courses Online advanced placement (AP) courses could increase students' access when face to face options are not viable due to restricted budgets or geographical location. In Newfoundland and Labrador, for example, the percentage of students in rural schools who enrolled in advanced-level courses through the Centre for Distance Learning and Innovation was higher than the provincial average (Barbour, & Mulcahy, 2013). Similarly, students in Florida who enrolled in online AP courses available to all students, regardless of income level and district budget, scored higher than the state average in their AP exams (Johnston, & Barbour, 2013). Yet students who participated in a follow up interview reported a preference for face to face classes where they could readily communicate with a teacher and improve their performance based on teacher MOOC Integration into Secondary School Courses Najafi, Evans, and Federico Vol 15 | No 5 Nov/14 308 feedback. Teacher presence in online secondary school courses positively affected student outcomes in introductory mathematics courses (Liu & Cavanaugh, 2012). For more advanced courses, on the other hand, the amount of teacher feedback was negatively related to course outcomes which may indicate that to be successful in more advanced online courses, high school students need to be self-directed and less reliant on teacher feedback. Time spent in the online course environment was another factor that would have a positive effect on students’ assessment outcome. MOOCs for Curriculum Enrichment and Reach-Ahead Some MOOCs have imposed minimal prerequisites for registration, and high school students have already established themselves as a demographic group in MOOCs of various difficulty level (Breslow et al., 2013). 2.39% of survey respondents of an electronics MOOC, with suggested academic prerequisite of having advanced knowledge of electricity and magnetism, reported their highest education level to be below high school diploma (DeBoer, Stump, Seaton, & Breslow, 2013). In terms of course achievement, however, total points gained by those students with junior high school education level were comparable with students holding a bachelors degree. Aside from MOOCs offered to the general public, specialized MOOCs have been developed for middle and high school students, either as university preparatory courses or as high school level courses. To address the paucity of pedagogically sound high school level computer science curricula, a mini course titled “Foundations for Advancing Computational Thinking” (FACT) was developed on the Stanford OpenEdX platform and piloted in a public middle school (Grover, Pear, & Cooper 2014). The FACT mini course has demonstrated promising results regarding students’ learning and it is being offered as a MOOC for either self-study or for teachers to use in their computer science classrooms. A similar project, MyCS, geared towards middle school students, was also considered for further development as a teacher-mediated computer science MOOC (Schofield, Erlinger, & Dodds, 2014). To reinforce students’ knowledge of science and mathematics and to facilitate their entrance to universities, an Italian school board implemented high school specific online open courses by recording classroom lectures and making them available online (Cannesa & Pisani, 2013). Test results showed that students who watched online lecture videos in addition to their face to face classroom scored higher than those who only relied on face to face classroom learning. Although not a typical MOOC in terms of design, this project supports the notion that self-paced open access material, geared towards high school students may increase students' depth of learning and academic success. In another study, a university preparatory mathematics MOOC was developed to increase mathematical proficiency of incoming students (Daza, Makriyannis, & Rovira Riera, 2013). First year university students were encouraged to participate in the MOOC voluntarily while the MOOC was also open to the public. Initial findings, showed students level of satisfaction with the contents of the MOOC, although the impact of MOOC learning on students’ success was not measured due to the scope of the study. MOOC Integration into Secondary School Courses Najafi, Evans, and Federico Vol 15 | No 5 Nov/14 309 While several projects such as the University of Wisconsin's College Readiness Math MOOC and the MIT physics MOOCs offer preparatory courses to secondary school students, research on the impact and implications of such initiatives is still in its infancy. MOOC Integration With respect to integrating existing MOOCs into regular classroom instruction, flipped classrooms could offer one model. In flipped classrooms, the online component, for example the MOOC, allows students to prepare for in-class discussion by reviewing relevant material beforehand and at their own pace. In other models of integration, even MOOC assessments could count towards students’ evaluation of the on-campus course. The instructor of a machine learning graduate course, for example, integrated an archived MOOC into his face to face course where students were required to take MOOC quizzes and write MOOC assignments, submitted to their instructor, in addition to covering learning components of the MOOC. Students responded positively to the experience and noted that the face to face classes would help them keep track of the selfpaced MOOC component (Bruff et al., 2013). Research Questions In this study, we examined potential impacts of the presence or absence of teacher support on secondary school students’ behavior and outcome during a MOOCintegration initiative. Three research questions guided our study: • How do students in MOOC-only and blended-mode groups differ in their learning outcomes? • How do students in MOOC-only and blended-mode groups differ in the level of engagement with the learning and assessment components of the integrated MOOC? • How do students in MOOC-only and blended-mode groups differ in their persistence to complete the activities of the integrated MOOC? Methods A case study encapsulates our research design in examining the impact of different levels of instructional support on students’ engagement with learning and assessment components of BE101x and their outcome as evident in MOOC quizzes and knowledgebased tests. To this end, BE101x integration in the ACEI course could be an instrumental case (Stake, 1995) that allows us to gain insight into the issue of using MOOCs in the context of a secondary school course. MOOC Integration into Secondary School Courses Najafi, Evans, and Federico Vol 15 | No 5 Nov/14 310 Research Context This study was conducted in the context of a university preparatory economics course at a competitive independent Canadian school from October 14 2013 to November 21 2013. The two participating class sections were taught by an experienced teacher, also a co-investigator of the study. The age range of students enrolled in this course was between 15 to 17 years. Twenty nine stu


Résumé de l'article
We investigated how high school students taking a university preparatory economics course would engage with the learning and assessment components of a Behavioural Economics MOOC that was integrated into their school-based course. Students were divided into two groups, MOOC-only, with no teacher support, and blended-mode, with weekly tutorials. MOOC only students scored slightly lower on a teacher designed knowledge test but scored slightly higher in a MOOC test. Although the MOOC-only students watched more unique videos, the blended-mode students stayed more on-track with the MOOC. The blended-mode students showed more persistence in retaking quizzes, yet they scored lower than the MOOC-only students.

Introduction
Massive online open courses (MOOCs) have been used in various ways often in the form of a flipped classroom, to complement traditional classroom teaching through integration of a whole course or specific parts of a course both in K-12 and in higher education. High school students, for example, could use MOOCs for university preparation in the absence of available face to face or online courses or in addition to them (Vihavainen, Luukkainen, & Kurhila, 2013). Furthermore, contents of a live or an archived MOOC could be integrated into an existing course in a hybrid format (Bruff, Fisher, McEwen, & Smith, 2013).
In this research we investigated how high school students taking a university preparatory economics course would engage with learning and assessment components of a Behavioural Economics MOOC that was integrated into their school-based course.
Of specific importance to this research was probing any potential difference in students' use of various components of the integrated MOOC depending on receiving supplementary instructional support or studying on their own. For this purpose, three weeks of the "Behavioural Economics in Action" (BE101x) MOOC were chosen for integration into the school-based course "Analyzing Current Economic Issues" (ACEI).
Twenty nine students from a Canadian independent university preparatory school who had enrolled in ACEI were randomly assigned to two groups for the BE101x integration period with one group studying on their own and the other receiving weekly tutorial sessions from their classroom teacher. Here, we report on how students in the two groups engaged with the learning and assessment components of BE101x and persisted throughout the three weeks of integration. Also, we compare students' outcomes based on pre-post BE101x-integration content knowledge surveys and BE101x assessments.

Review of Related Literature
Research on online university preparatory courses, MOOC-like university preparatory courses, and MOOC integration in credit courses informed our study.

Online University Preparatory Courses
Online advanced placement (AP) courses could increase students' access when face to face options are not viable due to restricted budgets or geographical location. In Newfoundland and Labrador, for example, the percentage of students in rural schools who enrolled in advanced-level courses through the Centre for Distance Learning and Innovation was higher than the provincial average (Barbour, & Mulcahy, 2013).
Similarly, students in Florida who enrolled in online AP courses available to all students, regardless of income level and district budget, scored higher than the state average in their AP exams (Johnston, & Barbour, 2013). Yet students who participated in a follow up interview reported a preference for face to face classes where they could readily communicate with a teacher and improve their performance based on teacher

MOOC Integration
With respect to integrating existing MOOCs into regular classroom instruction, flipped classrooms could offer one model. In flipped classrooms, the online component, for example the MOOC, allows students to prepare for in-class discussion by reviewing relevant material beforehand and at their own pace. In other models of integration, even MOOC assessments could count towards students' evaluation of the on-campus course.
The instructor of a machine learning graduate course, for example, integrated an archived MOOC into his face to face course where students were required to take MOOC quizzes and write MOOC assignments, submitted to their instructor, in addition to covering learning components of the MOOC. Students responded positively to the experience and noted that the face to face classes would help them keep track of the selfpaced MOOC component (Bruff et al., 2013).

Research Questions
In this study, we examined potential impacts of the presence or absence of teacher support on secondary school students' behavior and outcome during a MOOCintegration initiative. Three research questions guided our study: • How do students in MOOC-only and blended-mode groups differ in their learning outcomes?
• How do students in MOOC-only and blended-mode groups differ in the level of engagement with the learning and assessment components of the integrated MOOC?
• Prior to the start of the study, students were randomly divided into two groups for research purposes: a MOOC-only group with 14 students and a blended-mode group with 15 students.
In the three weeks outlined above, regular classes were not held. Instead, students in the MOOC-only group would study BE101x independently with no support from their teacher. On the other hand the blended-mode group students met their teacher once a week in an hour-long tutorial session. Materials from these sessions were given to MOOC-only students after week 5.
Tutorials were structured to include three common sections: Questions and answers, Students would attend regular classes in weeks 3 and 4, between October 29 and November 10, 2013. Upon the culmination of week 5 of BE101x, regular classes were resumed for the rest of the school term.
Students received an email prior to the start of their course that explained the study and its goal. This email was followed by another orientation email sent from the teacher on the first day of BE101x asking students to sign up for the MOOC on edX and also recommending that they refrain from starting the course until October 16th where a face to face orientation class meeting would be held.
Time and date for knowledge-based pre and post tests were also shared with the students. On October 16, 2013 the teacher met all of the students where the students received further explanation about BE101x and the study and took a knowledge-base pre-test. The teacher explained that the students would not be evaluated on their participation in BE101x and that participation and the extent of engagement were absolutely the responsibility and choice of the students.

Data Sources
To maintain student anonymity, we refrained from using any data that could lead to revealing their identity including their contributions to BE101x discussion forums and weekly debates. We collected data from the following sources.
We selected specific events in the clickstream that could reveal differences in use of and engagement with the integrated BE101x resources between the MOOC-only and the blended-model groups. For videos, we specifically considered "play_video" and "pause_video" events to determine if a video was watched. "Load_video" event were disregarded as it is generated when a page containing a video is loaded and this does not necessarily mean that the video • BE101x quizzes in weeks 1, 2, and 5 and BE101x test in week 3. The test in week 3 covered material from weeks 1 and 2. Data related to quizzes taken, number of attempts, and final score was extracted from the database for students in both groups. Another test was written in week 6 of the course but students' performance in this test could not be extracted from the raw data.

Data Analysis
Three dimensions of student engagement, student persistence, and student outcomes during the duration of their MOOC participation were of importance to this research.
Student engagement related to students' use of and access to BE101x learning components, specifically lecture videos that preceded each quiz, debate videos, and debate debrief videos. These learning components were located within the clickstream data by their unique identifiers. Four quizzes were preceded directly by either articles or slides. We could not track them in the clickstream data and thus did not include them in our analyses. For student engagement dimension, we compared the two groups on unique videos accessed, total number of videos accessed, and videos accessed on-track with BE101x pace.
Student persistence referred to the number of quizzes they took in weeks 1, 2, and 5 of the integration and if quizzes were retaken until a perfect score was achieved. The last dimension, student outcome, attended to differences between MOOC-only and blendedmode groups on their performance in BE101X test, overall quiz score, and in content knowledge pre-post tests. We would like to note that technical problems in extracting clickstream data largely affected the depth and scope of our analysis.

Students' Learning Gains and Outcome
Students' scores in content knowledge pretest, written at the start of BE101x integration, and posttest, written after the integration was over, showed an increase in their knowledge of Behavioural Economics. As illustrated in Figure 1, MOOC-only We investigated if the observed overall difference between the MOOC-only and the blended-mode groups in terms of their pretest and posttest scores also reflected in their level of persistence and engagement with learning and the assessment components of the integrated MOOC.  ACEI, we included these two weeks in our analysis as we assumed that students may have reviewed week 1 and week 2 material or have caught up with activities that they had missed.
The number of students who logged into the integrated BE101x in weeks 1, 2 and 5, in addition to weeks 3 and 4 in between, is shown in Figure 2. We corroborated high and low levels of activity with important dates during the integration period. Note that any activity before October 16 is negligible as students were not yet divided into groups.
The highest level of activity occurred on the two days when BE101X week1&2 and week5 tests were administered, October 31, 2013 and November 21, 2013 respectively. The day of the first tutorial showed another peak of activity with 7 MOOC-only and 9 blendedmode students online. A high activity level is observed on November 18 for the blendedmode group, which can be attributed to the third tutorial session. As expected, the level of activity was low for both groups after the October 31 test and remained relatively low throughout week 4. Both groups were comparable in the level of activity until November 16, 2013 when a peak of activity is evident for the blended-mode students that was sustained until November 18 and then dropped quickly in the next two days. In that three-day period, students in the MOOC-only group maintained a lower level of activity.
However, their activity level steadily increased and eventually equated with the blendedmode students in the two days leading to test 2 on November 21, 2013.

Use of Resources: Unique Number of Videos Accessed
We examined students' access in both groups to the lecture videos that would precede a MOOC-only students on average watched, that is, played, more unique videos than students in blended-mode group. Overall engagement level with lecture videos based on the average number of lecture videos watched was close to a maximum of 20 videos in both groups.
To better understand any noticeable differences in how students in the two groups accessed unique lecture videos of the integrated MOOC, we graphed the percentage of students in each group who watched lecture videos of week 1, week 2, and week 5 at least once any time during the integration and in the two weeks in between. As depicted in Figure 3, 80% of all lecture videos, were watched by more students from the MOOConly group. For the rest of the lecture videos, the difference between the number of blended group students and MOOC-only group students who watched those videos was a maximum of two students. Throughout the three weeks and between the two groups, the percentage of students watching lecture videos fell slightly with only one of 7 videos being watched by more than 90% of the students. This number was 6 out of 6 for the week 1 video and 3 out of 7 for the week 2 lecture videos. Still, the overall lowest percentage of students watching a lecture video, which is a week 5 lecture video, was nearly 67% by students in the blended-mode group.   Table 2 summarizes the average percentage of lecture videos in weeks 1, 2, and 5 that were watched within those specific weeks. As Table 2   We, then, considered if there were students in both groups who consistently remained on-track during the three weeks of integration or consistently lagged behind. In weeks 1, 2, and 5, four MOOC-only students, 28.57%, completely stayed on track whereas six others, 48.85%, completely lagged behind and watched none of the lecture videos belonging to those weeks. These numbers for the blended-mode group were three students, 20%, and four, 26.66% respectively. Figure 4 and figure 5 show the difference between the two groups regarding following the pace of the integrated MOOC by watching lecture videos in Weeks 1, 2, and 5 ontrack, in more detail. More students from blended-mode group followed the pace of the course for more than 80% of the lecture videos. Consequently, although a higher percentage of MOOC-only students watched all videos following the pace of the course, blended-mode students as a group remained on-track more consistently.

Engagement With and Persistence In Quizzes
Students could take a total of 29 quizzes with a maximum total quiz points of 29. A quiz would only be graded if students chose to submit their answer. As a result, a zero quiz point would either mean that the student submitted an incorrect answer, or else, the student did not take the quiz at all. To examine students' performance in quizzes in the MOOC-only and blended-mode groups we considered total points received in all of the 29 quizzes regardless of whether a quiz was actually submitted for grading. We factored such differences into our analyses of how students attempted to retake quizzes to achieve maximum points possible. Finally, we investigated students' persistence while taking quizzes in terms of achieving maximum quiz points possible for the number of quizzes taken. Blended-mode group students outperformed MOOC-only students in persistence as 40% of the former versus 21.42% of the latter correctly answered the quizzes that they submitted for grading.

Discussion
Of specific importance to this research was probing any potential differences in students' use of learning and assessment components of the integrated MOOC depending on receiving supplementary instructional support or studying on their own.
Students' scores in the pretest, written before starting the integrated BE101x, and posttest, written after the integrated BE101x was over, showed an increase in their knowledge of behavioural economics. Thus, the integration had no adverse effect on students' learning.
With the opportunity to expand on BE101x content in the three tutorial sessions, we would expect blended-mode students to watch more video lectures and be more engaged with these learning components. Surprisingly, MOOC-only students on average watched more unique videos than students in blended mode group. Regarding students' performance in MOOC quizzes, MOOC-only students were at no disadvantage as they received more quiz points compared to blended-mode students. However, we noticed that students from the blended-mode group were more likely to retake quizzes to achieve complete quiz points.
MOOC quizzes, considered as data sources for this study, may only reveal students' ability to recall information. The specific MOOC discussed in this study also contained knowledge transfer and application assignments and weekly debates that were not included as data sources. We also inferred from clickstream data that students participating in this study did not participate in the MOOC's discussion forum.
However, with the evidence that students would access MOOC resources and take quizzes with no direct contact with their classroom teacher, we posit that MOOCs could provide another means for reaching ahead and preparing for university for students who rely on online advance course for reasons such as geographical location (Barbour & Mulcahy, 2013).
In this study, we questioned if students would have followed the course week by week or would have lagged behind and compensated for it before the tests in week 3 and week 6 to cover lecture videos. Fewer students in the blended-mode group lagged behind, which could be attributed to the weekly face to face tutorials. Previous research confirms this finding (Bruff et al., 2013) that instructional presence may positively affect students following the expected pace of integrated content.
Although a higher percentage of MOOC-only students watched all videos following the pace of the course, blended-mode students as a group remained on-track more consistently. Knowing that they would discuss lecture videos of a given week during the tutorial session, these students would have preferred to attend the class prepared rather than postponing watching lecture videos to a few days before test 1 in week 3 and test 2 in week 6.
Empirical literature to investigate potential learning benefits of developing MOOCs or MOOC-like initiatives for specific age or grade level group is still in its infancy. Existing studies either report on preliminary findings in the form of overall satisfaction (Daza et al., 2013) or improved test results (Cannesa & Pisani, 2013) or propose the development of a MOOC based on pilot projects (Grover et al., 2014;Schofield et al., 2014). We contributed to this growing line of research, by comparing how high school students engage with a MOOC integrated into their school-based course, in a self-directed manner or with teacher support.

Conclusion
The results of our study are promising regarding integrating MOOCs in school-based courses in a self-study manner. Students engaged with learning and assessment components of an integrated MOOC in a self-study manner. It is important to note however, that participants of this study were high achieving and intrinsically motivated students which may have affected their level of engagement. Integrating MOOCs into school-based courses entails curriculum design challenges as classroom teachers need to find relevant content that enriches their existing curriculum. One implication would be for teachers to carefully examine the added cognitive value of MOOC integration.
Another challenge is the persistence of an integrated MOOC over time. In case an integrated MOOC is removed from its provider platform, the teacher may have to look Vol 15 | No 5 Nov/14 320 for a replacement or abandon the integration. MOOCs specifically developed for school courses may alleviate such problems.
Understanding students' experience through interviews and detailed reflection notes would shed more light on their perception of the usefulness of the integration and the challenges they may have faced. This study only included quantitative measures but our next round of research will also consider in-depth qualitative data related to the MOOC integration design process, student interviews and reflection, and a knowledge integration project as data sources so that we are able to compare the intended and the implemented MOOC integration effort in more depth.