The Impact of Enrollment in an OER Course on Student Learning Outcomes

Open Educational Resources (OER) are gaining acceptance as legitimate and effective teaching materials in higher education, particularly in 2-year institutions. Despite the steady growth in the availability and use of OER, there have been relatively few studies on the efficacy of OER and student achievement. This study analyzed the effect enrollment in an OER course had on student academic achievement when controlling for prior academic achievement in an introductory online history course at a large community college. Correlation analysis and simple linear regression were conducted. The results of this research indicate a significant positive moderate correlation between OER and student achievement. The study provided a process by which future, more rigorous efficacy studies can be conducted.


Article abstract
Open Educational Resources (OER) are gaining acceptance as legitimate and effective teaching materials in higher education, particularly in 2-year institutions. Despite the steady growth in the availability and use of OER, there have been relatively few studies on the efficacy of OER and student achievement. This study analyzed the effect enrollment in an OER course had on student academic achievement when controlling for prior academic achievement in an introductory online history course at a large community college. Correlation analysis and simple linear regression were conducted. The results of this research indicate a significant positive moderate correlation between OER and student achievement. The study provided a process by which future, more rigorous efficacy studies can be conducted.

Introduction
For the past decade, a more open approach to learning has emerged. The Open Educational Resources, or OER, movement is changing the way students learn and institutions operate. The most widely used definition of OER comes from the William and Flora Hewlett Foundation: OER are teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and re-purposing by others. Open educational resources include full courses, course materials, modules, textbooks, 232 streaming videos, tests, software, and any other tools materials or techniques used to support access to knowledge. (2013, para. 2) The Open Movement began in a wider context of a culture of open knowledge and peer collaboration.
More recently, the OER movement has gained traction, particularly in American community colleges with their open admissions policies and inclusive philosophies. OER have proven to be a key component to student access and success, providing free or low-cost textbook alternatives to financially strapped students (Bradley, 2013). As more community colleges are adopting OER (Ashford, 2017), it is important to examine the impact of OER beyond cost saving to students.

Literature Review
To date, there have been relatively few studies on the efficacy of OER and student achievement. Robinson, Fischer, Wiley, and Hilton III (2014) used a quantitative quasi-experimental design to determine whether the choice of open textbooks had a significant impact on student science learning in public high school.
After controlling for teacher efficacy and student characteristics, the researchers found that the use of OER did have a statistically significant effect on student learning in chemistry but not in biology or earth systems as demonstrated by student scores on end-of-year state standardized test scores. This study demonstrated that the adoption of OER can both dramatically reduce cost for students while at the same time improve student learning or, at the minimum, not negatively impact student learning. This study took place in a secondary setting, differentiating it from other studies reviewed here. Even so, the study does provide a framework for further efficacy studies, whether in secondary or post-secondary school settings. Robinson (2015) examined Open Educational Resource (OER) adoption as a correlate of postsecondary school students' academic success. Using an ex-post facto quasi-experimental research design, the study analyzed the academic performances of students using the open textbooks and those that were using the hardcopy traditional textbooks in seven Project Kaleidoscope post-secondary educational institutions. Having controlled for selection bias using propensity score matching (PSM), multiple regression and logistic regression models were used to analyze the effect of the OER on the students' academic performance. The findings showed that students using the traditional textbooks performed relatively better than those using the open textbooks in business and psychology courses. At the same time, students in this study who used open textbooks enrolled in more credits than students who used textbooks.
Fischer, Hilton III, Robinson, and Wiley (2015) analyzed the 2 nd year of OER adoption in the same schools as did Robinson (2015) (who examined the first year of adoption). They sought to know whether the adoption of no-cost digital textbooks had an impact on post-secondary students' learning outcomes.
The study examined course completion, class achievement, and intensity of enrollment during and after semesters in which students used OER and compared these to the same outcomes for students enrolled in the same courses that did not use OER. The sample was large, including 16,727 students, 4909 who were in the treatment group and 11,818 who were in the control group enrolled in 15 different undergraduate classes across 10 different institutions spread out across the United States. The size of this study makes the results more generalizable to a larger population than any other study of its kind conducted to date.
There were statistically significant differences in enrollment intensity between students enrolled in courses which used OER and those that did not, with students who used OER taking more credits than their counterparts. There were also moderate differences in completion rates and final grades between the two groups, with those in OER courses performing as well as or better than their counterparts in courses where faculty used commercial textbooks.
Hilton III, Gaudet, Clark, Robinson and Wiley (2013)  Learning" study in the Spring of 2007. The study employed a quasi-experimental study design in which students were assigned to different groups: those who used OLI-Statistics to supplement classroom instruction or those who received the traditional classroom instruction by itself. Twenty-two students were randomly selected to use the OLI-Statistics course in hybrid mode, while 42 students were assigned to the control group. The control group received traditional classroom instruction. Participants for both groups were selected from a pool of volunteers. The learning outcomes measured by the in-class exams scores of the OLI-Statistics group were compared with those of the instructor-led groups. Findings revealed that the OLI group completed the course in about half the time of the control group and achieved similar learning outcomes. Bowen, Chingos, Lack, and Nygren (2014) examined the learning outcomes of a hybrid machine-guided online interactive learning termed Interactive Learning Online (ILO). The study employed a randomized trial study design with two independent groups of public university students in six campuses in the United States. In essence, this study was a replication and extension of the Lovett et al. (2008) study. The first group was assigned to the traditional manual face-to-face teaching and learning while the other group textbooks as an alternative to the traditional hardcopy textbooks. The study employed an experimental design with an experimental group using the ChemWiki Textook in one chemistry class and a control group using a traditional chemistry text in the other chemistry class. The researchers analyzed the performance of the two groups, controlling for possible instructor bias by using the same set of instructors to teach both classes. The researchers examined grades obtained during in-class assessments, pre/post exam scores, and student attitude and self-reported study habits as measures of the students' learning outcomes. Although there was a difference in the students' attitudes and beliefs between the two groups, the difference was not statistically significant. Likewise, no significant difference was found in the students' learning outcomes or study habits between the two groups.
The efficacy studies performed to date indicate that students who use OER do about as well or better as students who enroll in courses which use traditional materials. However, there are still limitations to the work that has been done to date. A particular weakness of more than half of the OER efficacy studies performed to date is that they make no effort to control differences in student characteristics or success metrics.
The present study seeks to address this gap in the literature by examining the academic achievement (grades) of students enrolled in an introductory online history course which utilized OER compared to students enrolled in the same introductory online history course which used traditional textbooks. Specifically, the study attempted to answer this research question: What effect does enrollment in an OER course have on student achievement in that course when controlling for prior academic achievement?

Method
The data for this study was collected from the student information system at Northern Virginia Community College (NOVA), a large, multi-campus urban community college located near Washington, D.C. The sample includes 146 students. These students were enrolled in an online 8-week section of an introductory history class in the Fall of 2013 or in the Spring of 2014. In Fall 2013, there were two sections of the course which utilized OER and two sections which did not. In Spring 2014, there were three sections of the course which utilized OER and three which did not. Of the students enrolled across all sections of the courses, 64% were female and 36% were male. Since the study utilized student information after the fact, there was no way to control for selection bias. The student information system allowed identification of the study's predictor variables: student cumulative previous GPA prior to taking the online introductory history course (prior academic achievement) and identification of the course as either an OER version or a non-OER version. A correlation analysis was computed to assess the relationship between prior academic achievement and student grade in the course (student achievement). The correlation analysis also assessed the relationship between enrollment in an OER version of the introductory history course and student achievement.
In addition, a multiple linear regression was calculated to predict student grade in the introductory history course (student achievement) based on OER and prior academic achievement. Dummy variables were created for course section identification with OER being the baseline group for comparison to non-OER sections of the introductory history course. Also, students' final course grades were converted from letters to numbers, i.e. A=4, B=3, C=2, D=1, and F=0. For this study, W grades were converted to 0.

Results
There was a moderate positive correlation between prior academic achievement and student achievement (r = .30, p < .001). There was an even stronger positive moderate correlation between OER versus Non-OER and student achievement (r = .41, p < .001). These results show that there is a moderately positive relationship between taking an OER course and academic achievement.
A regression of student grade in the introductory history course on student GPA prior to taking the online introductory history course and identification of the course as either an OER version or a non-OER version explained 22% of the variance in student Number Grade in the introductory history course F (2, 143) = 20.30, MSE = 1.67, p < .001. Student cumulative previous GPA was a significant predictor of student number grade in the introductory history course (β = .34, p < .01), which indicates that a .34 increase in previous student GPA led to a one standard deviation increase in student grade in the introductory history course after controlling for OER vs. non-OER. Enrollment in an OER course was also a significant predictor of student grade achievement (b = .37, p <.001).

Discussion
The results indicate that there is a positive relationship between the use of OER and student academic achievement in an online history course. Because of the lack of demographic variables, we cannot definitively conclude that OER leads to increased student achievement. However, by controlling for prior college achievement, we show there may be evidence to support this claim. Students enrolled in the OER sections of the introductory history course performed as well or better than students enrolled in sections that used commercial textbooks. Usually, comparative studies examine the mode of delivery or instructional design as possible factors influencing student learning outcomes (Russell, 2015). One factor that this study was able to control for was the instructional design of the course. This study examined final course grades of students enrolled in 8-week sections of online courses with an identical instructional design.
Other factors were not controlled in this study. First, the study did not control for teacher effect. Even in online courses created from a master template, teacher effect can impact student learning. How often an instructor posts announcements, responds to class discussion forums, answers student email inquiries, and grades student work can have an impact on student success in a course. The study also did not control for student characteristics besides previous GPA, which may also have a significant impact on academic achievement. It would be useful to examine the interaction effect of these variables in future studies that examine the impact of OER on student academic achievement.

Conclusions and Future Research
While the results of the study may not have implications for OER efficacy per se, the study is important for showcasing the ways that researchers in community colleges can begin to use data available at their institutions to study the effect of OER on student learning. While a more robust study, which includes a greater sample size and controls for confounding variables is needed, certainly this study provides the first step in developing an experimental research design that does have implications for future OER research.
Further research is needed within other contexts to determine which elements of OER seem to have the greatest impact on student learning outcomes. Also, since this study was limited to one online course over the duration of one academic year, similar experiments, which control for various potential confounding variables must be undertaken in larger contexts to substantiate or repudiate the findings of this study.
Finally, future studies may also examine the ways in which OER interact with particular subject areas and what text characteristics make some OER better than others.
This study may be helpful for community college leaders who want to begin to use data more effectively or who seek to take on more of researcher/practitioner role at their institutions. By presenting a simple research model on which to build, this study can motivate community college leaders to conduct research of their own using data available from their own institutions to make decisions about OER use, adoption, and policy.