The Effects and Implications of Using Open Educational Resources in Secondary Schools

Open educational resources (OER) constitute a curriculum innovation that is considered revolutionary and has the potential to change the landscape of curriculum at all levels and content areas. OER have gained attention and widespread acceptance by educators and policy makers since 2002. The promise of OER is that they provide cost savings, promote collaboration, and are adaptable to the needs of teachers and students while providing a legitimate alternative to commercially produced print textbooks. Determining the relevance and viability of the movement to embrace OER requires an examination of theoretical foundations and empirical research to illuminate the effect of using OER as core curricula. While advocates promote the use of OER as a financially liberating model of curriculum and as a source of constructivist learning materials, more research is needed. The purpose of this study was to examine the relationship between OER and student learning. The study critically analyzed previous studies on OER and applied empirical analyses to the use of OER by a sample of middle schools. Twenty-eight middle schools from Washington State served as the subjects for the study. The study followed an ex post facto causal comparative model. Three research questions provided the focus for the study to investigate the effects of OER curriculum, duration of curriculum use, and other factors on student achievement in middle school mathematics. The results of the study found non-significant effects for OER use in relationship to school performance in mathematics, and significant effects on math scores for the variables of student poverty, curriculum duration, and cohort size.


Introduction
Considered to be an educational innovation with the potential to fundamentally change the nature of curricula from kindergarten through graduate school, open educational resources (OER) have gained widespread attention and acceptance by educators and policy makers over the past two decades (McKerlich et al., 2013;Smith & Casserly, 2006; United Nations Educational, Scientific and Cultural Organization [UNESCO], 2016).As educators, students, and policy makers become familiar with and adopt OER as an acceptable form of curricula, the need to assess the effects of OER on student learning has become increasingly important (Fisher et., 2015).Districts and teachers contemplating the use of OER as core curricula have reason to be concerned about the risks associated with abandoning familiar, mass-produced, and market-tested learning materials for resources that have open access and allow for liberal use, re-use, and repurposing.While advocates promote the use of OER as a financially liberating model of curriculum and as a source of constructivist learning materials, more research is needed.
This study was designed to examine the effects of using OER curriculum in secondary schools.It also examined the effect of other variables on student performance including cohort size, socioeconomic status, and duration of curriculum implementation.The study critically analyzed previous studies on OER implementation in order to provide insight and recommendations for future use of OER.

Development of OER
In 2002, UNESCO held a global forum that introduced the world to the concept of open educational resources.UNESCO developed an online community intended to provide a platform where educators and learners could access, copy, and change learning material without restrictions of copyright laws and economic resources.UNESCO stated that the OER movement had the potential to improve the quality of education and policy dialogue, as well as make it easier to share knowledge and build educational capacity.
A product of the UNESCO summit held in 2002 was a working description of OER, which is now the generally accepted definition used by educational organizations.Open educational resources "are teaching and learning materials that reside in the public domain or have been released under an open license.These resources may be used free of charge, distributed without restriction, and modified without permission" (Office of Superintendent of Public Instruction [OSPI], 2015).Weiland (2015) provided additional qualifications for OER which included independent learning objects or content, and tools such as open software, collections, and licensing.
Current trends show increasing interest in and implementation of OER across the United States and across the world (McKerlich et al., 2013;Smith & Casserly, 2006;UNESCO, 2016).However popular the movement may be, there is a need to determine the value of OER to education as a whole.Determining the relevance and viability of the movement requires connecting policies and practices to theoretical foundations and empirical research in order to show the effectiveness of using OER as a main source curriculum.
In the area of learning reading, for example, an Internet search for OER lessons for reading elementary school yielded over 17 million articles, books, or reports.Narrowing the search to fourth grade reduced that bulk to approximately 1.3 million items.It is unrealistic to expect individual teachers to research and assess each item for quality.Even at a pace of reviewing OER daily, teachers would likely find the materials they

109
deemed most useful modified or replaced by more relevant and useful OER.Such an exercise would be a time-consuming and frustrating endeavor.If OER are to be viable and credible as tools for teaching, then there is a need for demonstrations of positive impacts on learning and also an identification of barriers to their use.

Theoretical Foundations
The use of OER within curriculum and instruction can be connected to constructivist and progressivist ideals regarding learning.Dewey (1938) promoted experience, adaptation, and expansion of knowledge as key elements to authentic and meaningful learning.The perpetual re-creation and expansion of knowledge and meaning is at the core of what the OER movement has to offer.As Dewey described, the progressivist tradition emphasizes free activity that includes thinking and questioning and interacting, as well as acceptance and interaction with a changing world.Piaget (1971) promoted the ideal that prior knowledge is key to the development of new knowledge and understanding.OER constitute a curricular form of this process: prior knowledge is adapted and repurposed to form new knowledge in the form of new learning objects.Adaptation and repurposing can be done at the individual teacher or student level, or on a level involving whole school systems.Knox (2013) affirmed that the OER are "the building blocks of a constructivist-informed 'learning 2.0,' comprised of social learning, legitimate peripheral participation and learning through communities of practice" (p.825).
The OER movement shows strong ties to Dewey's philosophy of experiential learning, as it meets his description of ideal curricula."Scientific study leads to and enlarges experience, but this experience is educative only to the degree that it rests upon a continuity of significant knowledge and to the degree that knowledge modifies or 'modulates' the learner's outlook and attitude" (Dewey, 1938, p. xii).Furthermore, OER provide the continuity Dewey discussed, as they are borne by educational choices and they produce artifacts for future learning."From this point of view, the principle of continuity of experience means that every experience both takes up something form those which have gone before and modifies in some way the quality of those which come after" (Dewey, 1938, p. 36).resulted in better scores on the CRT for students in chemistry.However, this was not the case for students in earth systems and biology.Though there was not sufficient effect size to promote wholesale adoption of OER in science courses, the data supported the notion that OER use had, at worst, a neutral impact on student achievement.Therefore, it would be appropriate to promote OER adoption for the sake of cost savings."These findings conformed with our belief that teacher efficacy and prior ability would play a much more important role in educational achievement than textbook selection" (Robinson et al.,p. 346).Wiley and Hilton (2012) examined both the hypotheses of cost savings and impact on student learning promised by the use of OER.Their study was conducted over two years and included 20 middle school and high school science teachers.Data supported assumptions of cost savings through the use of OER but did not validate assumptions of learning improvement.While much OER are accessed and used electronically, the study prompted further inquiry into the effect of OER format (e.g., paper vs. electronic) and its effect on student learning.Conclusions drawn from the data showed no support for the hypothesis that OER improved student learning."Simply substituting open textbooks for traditional textbooks did not appear to have an effect on student test scores" (Wiley & Hilton,p. 212).

Effect on Teaching Practice
In addition to examining the effect of changing the licensing format of the curriculum, it is important to consider any change in pedagogy employed by teachers in the context of using OER.Dotson and Foley (2017) emphasized that the variable of curriculum, in itself, does not account for student learning outcomes.
Rather, OER elevated the effect that teachers and their pedagogies had on student learning as important factors to be considered.Pearcy (2014) reaffirmed the important role of the teacher for providing feedback and guidance in the context of innovative curriculum and instruction.

Demographics as a Variable in Student Learning
This study tested the hypothesis that other factors, including socioeconomic status, affect student achievement within the context of curriculum licensing format.Dotson and Foley (2017) stated that student achievement has a negative correlation with the students' degree of poverty-the poorer a student, the less likely they will score at standard in standardized tests.They emphasized that this statistical relationship is consistent with the available research on student poverty and is pervasive across regions in the US."Schools with high levels of poverty score very low on current measures of effectiveness which are primarily based on standardized tests" (Dotson & Foley, 2017, p. 299).

Barriers to Use
Kelly (2014) referenced the social learning theory regarding user efficacy and the context of trying something new, specifically OER. Kelly referenced the theory to address perceptions, intentions, and actual technology use.Kelly's research indicated that attitudes and perceptions about OER were strongly correlated to the attitudes regarding new technology and moderately correlated to actual use of OER."This indicates that OER must be considered easy to use or the perceived utility of the resource will be negatively impacted" (Kelly,p. 37).A potential barrier to the use of OER by instructors could be the perception of difficulty in finding and using OER, or actual negative experiences by teachers and students who have used OER without success or with great difficulty.

The Effects and Implications of Using Open Educational Resources in Secondary Schools
Harvey and Bond

111
Time and access to training are also factors related to acceptance of OER, as teachers who wish to explore and develop their own skills around the use of instructional innovations find little opportunity within their contracted work to do so.The fast rate at which change occurs, particularly in regard to technology, confounds these obstacles.

Method
This study was guided by the following three research questions: 1. Is there a significant difference in school-level achievement scores in mathematics between eighth grade students who use OER compared to those who use commercially published print curriculum?
2. Does the length of time a given curriculum is implemented have a significant effect on school-level achievement scores in mathematics among eighth grade students?
3. Are there additional effects on school-level achievement scores in mathematics of eighth grade students besides the variables of cohort size and socioeconomic status?Is there a difference in the effect of those variables between students who use OER compared to those who use commercially published non-OER curriculum?
A causal comparative study was selected as the research design for this study.While Gall et al. (2007) cautioned that such studies do not permit strong cause-and-effect conclusions, they are appropriate in exploratory investigations in which manipulating the independent variable is a challenge.In this study, this researcher was not in a position to direct school districts as to the type of curriculum they adopted, therefore, a causal comparative design was appropriate.
One motivating factor for this study was to provide empirical evidence that would help future researchers decide whether the variables had a strong enough relationship to warrant the expense and time required to conduct experimental research.Our purpose was to help them focus on more specific variables on which to base controlled experimental studies.Future studies could compare the effects on student learning using independent variables such as curriculum type, specific source or title of OER, students' grade level, or other student or school demographics.

Variables
The main independent variable was the type of curriculum format-traditional, commercially produced print curriculum or OER curriculum.Other independent variables included (a) cohort size of each school's eighth grade class, (b) duration of curriculum adoption, and (c) socioeconomic status as indicated by each school's percentage of students receiving free and reduced-price lunches.The dependent variable was the school-level scores on the mandatory state assessments in mathematics.

Participants
The subjects of the study were 28 public middle schools in Washington State, representing 6,984 students.
The cohorts consisted exclusively of eighth grade students who completed the Smarter Balanced Assessment (SBA) for mathematics in the spring of 2017.Each school and its related demographic data was considered to be one case in the statistical analysis.The schools were divided into two equal groups to compare the percentage of students proficient on the math assessment based on their use of OER.The original pool of all schools in the study consisted of 32 schools that used OER for mathematics curriculum and 14 schools that used non-OER curriculum.Because it is preferred to have equal group sizes for comparison in the administration of t tests, a computerized random number generator program was used to select schools into the OER comparison group which resulted in the formation of two comparison groups with 14 schools in each group.
The 28 schools included in the sample represented 11 of the 295 school districts in the State of Washington.
Many of the schools were within a common geographic area only 600 square miles large, and 5 of the 11 school districts bordered at least one other sampled school district.One of the selected districts was located over 500 miles away from the nearest school district included in the sample.The original intent of the study was to include schools in districts that were contiguous within a given geographical region.However, the number of schools in Washington was limited, and so was a constraint on the available districts from which to draw data.

Data Collection
Demographic data was collected on each school in both the OER and non-OER groups, including (a) the number eighth grade students who were assigned to complete SBA math examination, (b) the number of years which the school used its particular eighth grade math curriculum, and (c) the percentage of students who participated in the National School Lunch Program.
Ex post facto data were drawn from a convenience sample of eighth grade mathematics scores from the 2017 SBA results, obtained from OSPI and used to compare means of school scores between schools using OER and schools using non-OER curricula; this served as the criterion variable representing student achievement.The SBA is an assessment used by several states including Washington as a requirement to show schools and school district students' proficiency in mathematics and English language arts.Passing the SBA in mathematics is also an official pathway to graduation from high school in Washington State.
This assessment provided valid, reliable, and fair assessments of the deep disciplinary understanding and higher-order thinking skills increasingly demanded by a knowledge-based global economy (Smarter Balanced Assessment Consortium, 2016, p. vi).
There were seven different curricula used by all schools in the sample.The OER group used four different open math curricula, and the non-OER group used three different commercially published textbooks.
Duration of curriculum use was a variable constructed to measure the number of years a given curriculum was in use prior to the 2017 SBA eighth grade mathematics assessment.The outcome variable of this study was the school-level scores on the Smarter Balanced Assessment (SBA).

The Effects and Implications of Using Open Educational Resources in Secondary Schools
Harvey and Bond

113
An anonymous, informal survey of nine questions was provided to teachers in schools which used OER.
Five of the questions asked for responses regarding perceptions of the use of OER from the vantage point of their role as teachers and their interaction with the curriculum, particularly regarding the effects of OER on student learning and their own teaching practices.The remaining four questions were more general about the respondent's experience as a teacher, such as years of service.

Data Analysis
The dependent variable in this study was the school-level score on the SBA mathematics examination for eighth grade.The independent variables included the (a) licensing format of curriculum, (b) size of the student cohort, (c) duration that a curriculum had been used prior to testing, and (c) percentage of students enrolled in the free and reduced-price lunch program.
This study aimed to compare the effect of using OER as curriculum on student achievement to the effect of using commercially published print curriculum.Grade-level mathematics scores were examined regarding the variable of OER condition and demographic variables to complete the data set.The data were screened for outliers, missing values, and normality.Descriptive and inferential statistics were calculated.The reported data included means, standard deviations, and statistics of skewness and kurtosis.The t test analysis (Gall et al., 2007) was used to compare means between groups to determine whether any differences of a common variable were statistically significant.An independent-samples t test compared means between groups for the main effect of OER use on math scores.
Since other factors affect learning outcomes beside the licensing format of the curriculum, a multiple regression analysis was performed to determine the relationship of variables to school test scores, beyond the use of OER.For all inferential statistics tests, a value of .05 was set as the threshold for significance.

Results
From the quantitative and qualitative comparisons, a few trends emerged and are worth notice.The non-OER schools outperformed the OER schools by nearly 5% on the SBA mathematics test.The mean duration of curriculum use prior to the 2017 test was greater for the non-OER group by three years.Finally, the correlation for student poverty and test performance had an effect size of 77% (β = .77).

Results of Quantitative Analysis
All variables were normally distributed for both comparison groups.Table 1 shows the mean percentage of students who were proficient on the math assessment, by school, for both curriculum groups.The schools in the OER curriculum group had a lower percentage of students who passed the SBA mathematics test (M = 39.66,SD = 10.24),than did the schools that utilized the non-OER curriculum had (M = 42.82,SD = 12.69), though the differences were not statistically significant.
For all schools in the total sample (N = 28), each variable had a range of values.

114
of years of use of the particular curriculum by each school ranged from 1 to 8 years.Finally, the range of students enrolled in the free and reduced-price lunch program was between 24.7% and 87.9%.Duration of curriculum was the second predictor variable studied.The non-OER schools had a greater mean number of years using their respective curricula (M = 6.14, SD = 1.74) than did the schools using OER (M = 2.50, SD = .65).In an effort to further understand the relationship between duration of curriculum and curriculum format, a t test was run to determine if there was a significant difference between the groups on the variable of duration of curriculum.The result of that analysis showed that the non-OER group had on average significantly more time using their curricula than did the OER schools t(1,16) = 7.309, p < .001.
The OER schools had a larger mean cohort size as measured by the number of eighth grade students who took the SBA for mathematics in 2017 (M = 276.85,SD = 62.83) than did the non-OER schools (M = 222.00,SD = 39.87).As a demographic statistic for measuring the sample schools' level of poverty, the non-OER schools (M = 51.62,SD = 17.61) had a higher mean percentage of students enrolled in the free and reducedprice lunch program than did OER schools (M = 49.22,SD = 13.30).
The main effect for the first hypothesis regarding OER use by schools was found by conducting a t test which compared the mean scores of OER schools against the non-OER schools on the measure of school-level math scores on the SBA.The results of the t test showed that the non-OER schools (M = 42.8,SD = 12.69) had a higher percentage of students meeting proficiency on the SBA mathematics test than did the OER schools (M = 39.66,SD = 10.24),however the difference was not statistically significant t(26) = .726,p = .474.
A simple regression analysis was run for the effect of duration of all schools' curriculum implementation.
The effect for curriculum duration by itself was found to be not significant, R 2 = .050,F(1, 26) = 1.36, p = The Effects and Implications of Using Open Educational Resources in Secondary Schools Harvey and Bond

115
.254, indicating that the variable of how many years a curriculum is used by a school, when isolated from other variables, does not have a significant effect on student achievement.
A multiple regression analysis was conducted to examine the correlation between the variables of duration of time curriculum was used, student cohort size, and the percentage of students in the free and reducedprice lunch program as these related to the percentage of students who were proficient on the SBA mathematics test.The multiple regression analysis was run in two models.The first model isolated the predictor variable of curriculum format, namely status of OER use.Statistics within the first model were used to infer that use or non-use of OER did not significantly correlate to a school's math score, F(1,26) = .527, p = .474.In the second model, the predictor variables of free and reduced-price lunch, cohort size, and curriculum duration were added in to determine if there was a relationship between those variables and math scores.The second model included data which allowed for rejection of the null hypothesis, namely that variables other than OER use do not correlate to a school's mathematics scores.The results of the second model in the multiple regression analysis, R 2 = .633,F(4,23) = 9.93, p < .001,provided detail as to each predictor variable's relationship with the schools' math scores (Table 2).
In order of largest to smallest effect size, each predictor variable was found to have a relationship to the criterion variable.The free and reduced-price lunch program's percentage of a school's population had a significant negative correlation (β = -.611,p < .001); the correlation for curriculum duration was positive and significant (β = .604,p = .014);and the correlation for cohort size was positive and significant (β = .344, p < .001).These results supported the notion that the three variables in combination have a significant effect on a school's SBA math scores.The results also supported the claim that there are variables other than curriculum licensing status which affect the outcome of school-level scores in the math assessment, and that 63% of a school's math scores can be explained by the three variables included in multiple regression analysis.

Qualitative Analysis Results
Results from the informal survey showed mostly favorable views of using OER.While a change in teacher practice was noted by more than half of the respondents, this was not the experience for all teachers using OER.Commentary from teachers who believed OER changed their teaching practices included: "I look The Effects and Implications of Using Open Educational Resources in Secondary Schools Harvey and Bond 116 more closely at how math is applied to everyday life"; "The curriculum is designed around group/partner talk"; "The basic format of most chapters is inquiry (i.e., a problem that is approachable for students to teach them the concept, as opposed to the traditional methods of lecture and teach)." Some teacher commentary indicated that OER was not effective in improving student learning.For example: (a) the OER curricula over-emphasized examples and practice problems; (b) having just consumables rather than a textbook has been an adjustment to get used to, and that adjustment was cause for dissatisfaction with OER; (c) grading the workbooks was a challenge; (d) the definitions of terms were found through the material, not just in one central location, which was perceived to be a challenge for students; (e) some of the content did not cover the standards required for the grade level; and (f) the OER curriculum did not provide enough basic practice for some of the concepts.
In general, the teachers' responses to the use of OER indicated that the curriculum provided more opportunities for inquiry, deeper understanding of math concepts, and more student-to-student interaction.These positive attributes of OER are relevant and timely, particularly in the context of the growing movement to include those attributes in pedagogy improvement efforts.This is particularly so within the frameworks of universal design for learning, with its emphasis on choice and having students find relevancy in the curriculum, and the habits of mind as well as of mathematical practice.The positive traits of OER described by the teachers in the study align well to the elements currently being promoted in discussions around student-centered educational reform.Future study of the use of OER versus non-OER curricula in combination with traditional and emerging pedagogical frameworks may reveal more useful data about the combined variable effects on teacher confidence in the curriculum and student learning outcomes.

Discussion
The purpose of this study was to determine if the use of OER math curricula had a statistically significant effect on student achievement as measured by the annual required state assessment in mathematics.The analysis of the data supported the hypothesis that the licensing format of curriculum as a single variable does not have a statistically significant effect on a school's SBA mathematics scores.Schools that used OER curriculum performed slightly lower in eighth grade mathematics than schools that used traditional commercially published print curriculum, but not at a significant level.This may be interpreted as a positive sign for schools that are considering adopting OER math curricula, as the data in this study showed there was no significant difference between student achievement at OER and non-OER schools.However, because schools are officially and casually rated by their tests scores, any difference in scores may be interpreted by the public as an indication of one curriculum format being better or worse than another.
Lacking any evidence of harm to learning, a switch from commercially published curricula could save school districts tens of thousands of dollars in curriculum purchases.Such savings could be redistributed to implement intervention programs, hire teachers and other support staff, and purchase learning materials, all of which can successfully address achievement and opportunity gaps.In other words, the cost savings that is implied with the adoption of OER can be used to accelerate student achievement.(2012) discovered, teacher effect and previous performance on examinations had larger positive effect sizes than the use of OER curriculum in relation to student learning.The finding that curriculum licensing format did not significantly affect student learning outcomes, combined with the finding that poverty had a significant and negative correlation to student achievement, prompts a call to action on the latter.Findings from this current study supported the notion that putting effort into finding effective measures that reduce or eliminate the negative effects of poverty will have greater value over experimenting with different curriculum formats.
The discussion of poverty and its effect on student achievement is germane to this study for two reasons.
First, it causes the researcher to consider student socioeconomic status along with other factors when studying student achievement.And, second, it is a cause for inquiry regarding whether a particular curriculum or level of curriculum accessibility, that is, purchased or open-access, is more or less effective in helping students in poverty perform at standard on standardized tests.

Recommendations for Further Research
Experimental studies are lacking in the available research regarding OER.Two areas that could be better addressed by more experimental studies are the effect of curriculum type on student learning and also the student experience, both of which were outside the scope of this study.There would be widespread interest in a large-scale study comparing the different sources or titles of OER.

Conclusion
The results of this study indicate that the use of OER curriculum by itself does not significantly affect student learning in mathematics.The findings can be used to promote OER as a viable source of curriculum, as its implementation will not significantly benefit nor harm student achievement scores in math.Other variables are statistically significant contributors to student achievement in mathematics and include size of student cohort, duration of curriculum use, and rate of poverty.These factors should continue to be included in future studies.
Reasonable caution should be employed when making any shift in curriculum.Selecting an OER curriculum simply because it is more accessible through ubiquitous technology will not yield significant changes in student achievement.Even with the exponential growth in access to OER, its lifespan to date has been relatively short in the history of whole curricula.As Pearcy (2014) explained "teachers, like all individuals, are more likely to adopt an innovation if it proves to be a more effective means to accomplish something, and has observable effects" (p.180).In the face of new and widespread exposure to OER by the hundreds of districts and dozens of states affected by the global COVID-19 pandemic and massive school closures, there is still a need for more evidence that OER improves teaching and learning before recommending a wholesale shift away from traditional forms of curricula to an exclusively OER structure.The current condition of forced remote or distance learning and teacher collaboration may be fertile ground for the proliferation of OER development, use, and study.

The Effects and Implications of Using Open Educational Resources in Secondary Schools Harvey and Bond
The range of percentage of students showing proficiency on the SBA mathematics examination was 23.3% to 67.9%.The range for cohort size of each school's eighth graders who took the examination was 154 to 385 students.The number

Table 1
Descriptive Statistics for Percentage of Students Proficient on Eighth Grade SBA(Math)

Table 2
Multiple Regression Analysis Variable Statistics Related to Student Proficiency Rates on the Eighth Grade

The Effects and Implications of Using Open Educational Resources in Secondary Schools Harvey and Bond 117
Other factors affect learning outcomes beside open-access status of the curriculum.As Wiley and Hilton