Measuring the Impact of an Open Educational Resource and Library e-Resource Adoption Program Using the COUP Framework

Grant programs that foster the use of open educational resources (OER) significantly reduce undergraduate student spending on textbooks per semester. The Zero-Cost Course Materials (ZCCM) grant program at the University of California, Merced (UC Merced), eliminated text costs and ensured access to course materials by replacing required commercial materials with OER and library licensed e-resources. The present study applies the COUP framework (cost, outcomes, usage, and perceptions) to evaluate the ZCCM program. The ZCCM program resulted in student cost savings and did not negatively impact student outcomes. Students in ZCCM courses demonstrated higher rates of course completion than students enrolled in previous sections. For the outcomes of final course grade, passing with a C− or better, and number of credit hours enrolled in, findings were comparable between the cohorts. Student usage and perception of course materials were gathered using a survey. Though students reported favorable views of zero-cost materials, they reported using them less frequently than commercial texts. This research contributes to a growing body of literature that confirms beneficial cost savings for students using zero-cost materials without jeopardizing students’ success.


Introduction
The formal designation of open educational resources (OER) at the UNESCO Forum on the Impact of Open Courseware for Higher Education in Developing Countries (2002) launched OER as a compelling alternative to commercial educational materials.The definition of OER as "learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been  (UNESCO, n.d.).While the 2002 forum brought participants together to examine worldwide applications of MIT OpenCourseWare, participants left with a collaborative plan to evaluate, use, and develop OER "for the whole of humanity" (UNESCO, 2002).
Within the United States, the development, curation, promotion, and use of OER in higher education curricula has been bolstered by funding at the federal, state, and institutional levels.The U.S. Department of Education allocated $47 million in grant funding starting in 2018 through the Open Textbooks Pilot program (Allen, 2022), and according to the SPARC (Scholarly Publishing and Academic Resources 1.What are the student cost savings of using zero-cost materials in place of commercial materials?2. What is the impact of using zero-cost materials on student success?3. How does the use of zero-cost materials influence unit enrollment patterns?4. How do students perceive the quality and value of zero-cost materials?

Effects of Textbook Costs
Between 2006 and 2018, the average price of a college textbook increased by 190% (Bureau of Labor Statistics, U.S. Department of Labor, 2023;Smith Jaggars et al., 2019).While textbook prices have plateaued over recent years, students continue to take action to minimize textbook costs.The most recent National Association of College Stores (NACS) Student Watch Report finds that despite course material spending averaging $339 per year, the lowest since NACS started tracking this data in 1998, one in four students still skipped purchasing at least one course text, and 11% of students have pirated course materials (National Association of College Stores, 2022).Over 87% of respondents taking the Florida Virtual Campus (FLVC) Student Textbook and Instructional Materials Survey reported that they pursued measures to reduce textbook costs, such as renting a digital copy (50.5%) or sharing books with classmates (17%).Over half reported not purchasing the text at all (Florida Virtual Campus, 2022).
Students have expressed concerns about their ability to succeed academically when they do not purchase textbooks required for their classes.Sixty-five percent of respondents to a Student Public Interest Research Groups (PIRGs) survey indicated that they chose not to buy course materials due to costs, and of those respondents, 94% expressed concerns that this decision would negatively impact their course grade (Senack, 2014).Over a third (32.4%) of FLVC survey participants believed that high textbook costs resulted in poorer grades, and 24.2% reported dropping a course due to high textbook costs (Florida Virtual Campus, 2022).Additional literature suggests that these actions are not uncommon.In Jenkins et al.'s study (2020), 56% of surveyed students reported that they did not buy a textbook for a course due to costs and later felt like it had hurt their class performance; 12% dropped a class due to textbook costs, and 9% attributed failing a class for the same reason.Textbook costs can also influence students' course selection and enrollment intensity (credit hour) decisions.In the FLVC survey, 38.5% of students reported not registering for a specific course, and 43.7% reported taking fewer courses due to textbook costs (Florida Virtual Campus, 2022).

Cost Savings
Instructors frequently point to student cost savings as the primary reason for adopting OER (Hilton III, 2020).Switching to an OER can quickly yield significant savings, especially in large enrollment courses requiring high-priced texts.Fialkowski et al.'s assessment of an OER program at the University of Hawaiʻi at Mãnoa calculated savings of $39,000 for 311 students in an introductory nutrition course within a single semester (2020).
Researchers have employed various methods to calculate the cost savings of replacing a commercial text with an OER.One widely used method multiplies the sticker price of the previously assigned commercial text by the number of students enrolled in the OER course.Though Cozart et al. (2021) used the sticker price method, they note that cost savings can be difficult to determine precisely and may be inflated since not all students will purchase a new textbook at the list price.When determining cost savings of OER initiatives from 600 courses across 120 institutions, SPARC researchers (along with David Wiley) developed a formula to account for student purchasing behavior.Their calculation averaged the most expensive version of a text with the least expensive version, representing the text's available format and delivery options, from which the cost of the OER, if any, is subtracted.In applying this formula, they determined $116.94 in cost savings per student when a commercial text was replaced with an OER (Nyamweya, 2018).
Another method uses a standardized amount to represent textbook costs; the Open Education Network and Open Oregon Educational Resources both apply a rate of $100 per student to determine cost savings (Hofer, 2017).

Student Outcomes
Many studies assessing the impact of OER examine one or more of the following undergraduate student outcomes: course grades; passing with a C− or better; withdrawal, drop, failure, and course completion rates; and enrollment rates.They vary in terms of their methodologies, sample sizes, and ability to control for variables.

Final Course Grades
Researchers regularly, but not exclusively, use course grades as an indicator of student learning and success.
In an analysis of final grades for eight courses at the University of Georgia from 2010 to 2016, students in OER courses had significantly improved course grades in contrast to students in non-OER courses (Colvard et al., 2018), while Fischer et al.'s examination of 15 undergraduate courses from 10 institutions found no significant differences in course grades between OER and non-OER courses when results were reported in aggregate.Robinson reported the opposite; students using OER had lower grades than those using commercial textbooks (2015).For data at the course level, results from Fischer et al. (2015) and Robinson (2015) were mixed.
Additional findings differ by student population.For instance, Winitzky-Stephens and Pickavance (2017) did not see a significant correlation between course grades and OER until they analyzed new first-year student and continuing student populations separately.When disaggregated, they found course grades positively impacted new students in OER courses.Colvard et al.'s (2018) research found that OER courses had even greater positive impacts for Pell-eligible students, part-time students, and historically underserved students.

Passing With a C− or Better
When examining three measures of student success, including the likelihood of passing with a C− or better, Winitzky-Stephens and Pickavance (2017) found that there were no adverse effects for students using OER compared with students using commercial texts.Fischer et al.'s (2015) work also investigated if students in OER and non-OER groups were more or less likely to earn a C− or better.Results were mixed, though for most courses (9 of 15), there were no significant differences between the two groups.Five OER courses and one commercial course yielded higher rates of students passing with a C− or better.While Robinson (2015) found no statistically significant differences between OER and non-OER groups earning a C− or better, analysis at the course level surfaced two of seven courses in which the OER groups had lower rates of earning a C− or better.While findings are not entirely uniform, the literature predominantly reports positive impacts or no negative impacts of OER usage on students' ability to pass with a C− or better.

Withdrawal, Drop Rates, Failure Rates, and Completion Rates
Researchers have also examined students' withdrawal rates, drop rates, failure rates, and rates of completion in OER versus non-OER courses as additional measures of student success.Most findings point to OER as having positive impacts on these measures, while others report findings with no statistical significance or findings with mixed results.Numerous researchers report lower withdrawal rates of students in OER courses (Bol et al., 2022;Clinton, 2018;Clinton & Khan, 2019;Feldstein et al., 2012;Hilton III & Laman, 2012;Hilton III, 2016;Pawlyshyn et al., 2013;Suriano, 2023), lower drop rates (Hilton III, 2016;Wiley et al., 2016), lower failure rates (Colvard et al., 2018;Feldstein et al., 2012), and higher completion rates (Ross et al., 2018).
Additional studies record a lack of statistically significant differences between the rates of OER and non-OER groups.Winitzky-Stephens and Pickavance (2017) found no statistically significant differences in withdrawal or pass/fail rates between students in OER and non-OER classes.Croteau (2017) concluded that there were no differences in either DFW (drop/fail/withdraw) or completion rates between students in courses pre-and post-OER adoption.Fischer et al. (2015) also compared completion rates of students in OER and non-OER sections in 15 courses and found few significant differences.

Enrollment Rates and Enrollment Intensity
There is a limited amount of literature on OER's potential impact on student enrollment patterns.Some studies highlight students' avoidance of classes due to textbook prices, and these costs may also have a greater impact on certain student populations.In their survey of students enrolled in a general psychology course, Hardin et al. (2019) found that non-White students and first-generation college students (FGCS) "reported that the cost of the book had a significantly higher impact on decisions to enroll in the course" in comparison to their non-FGCS or White peers; yet, the study's larger conclusion pointed to textbook costs having a minimal effect on students' decisions to enroll in courses of interest (p.54).Similarly, Jenkins et al. (2020) reported that 27% of their student respondents avoided enrolling in a class due to textbook costs, and this rate of avoidance increased for Latinx and transfer students.Other studies have gone beyond student survey data to examine enrollment rates and enrollment intensity (the number of credits in which a student is enrolled each semester).In both cases, Fischer et al. (2015) and Robinson (2015) found that students enrolled in OER courses had a higher credit load on average.

Student Use and Perception of OER
Students have reported using OER at similar rates to commercial texts (Bliss et al., 2013;Hendricks et al., 2017;Jhangiani et al., 2018;Sack Illowsky et al., 2016).Gurung's results (2017) differ, finding that students using OER read less than those using a commercial textbook.
In regards to OER quality, students are usually positive.When asked to compare the quality of an OER to that of commercial texts they had used previously, the majority of students in multiple studies responded that the OER was of better/higher or equal/similar quality to commercial texts (Bliss et al., 2013;California Open Educational Resources Council, 2016;Cooney, 2017;Hendricks et al., 2017;Hunsicker-Walburn et al., 2018;Ikahihifo et al., 2017;Jhangiani et al., 2018;Ross et al., 2018;Sack Illowsky et al., 2016;Watson et al., 2017).This finding is highlighted in Hilton's synthesis of user perceptions of OER from 2015 to 2018 (Hilton III, 2020) and in Mullens and Hoffman's 2023 systematic review of OER publications from 2002 through 2023, which included 48 studies that investigated student perceptions of OER quality.Additional studies indicate no significant differences in students' perceptions of OER and commercial texts (Clinton, 2018;Nusbaum et al., 2020), while a limited number of studies report that students rated their standard commercial texts as higher quality than OER (Gurung, 2017;Lawrence & Lester, 2018).Some studies have identified what students value about OER.Desirable characteristics include ease of use (Hilton III & Laman, 2012), ease of access (Brandle et al., 2019), greater readability (Jhangiani et al., 2018), better writing (Clinton, 2018), targeted content, organizational flexibility, and digital capabilities (Cooney, 2017;Ikahihifo et al., 2017).

Methodology Participants
During the grant program, 1,870 students were enrolled in one or more ZCCM-grant-supported courses, which included large enrollment introductory courses, upper division courses, and one graduate level course.At each semester's end, students in ZCCM courses were invited to complete a survey and consent to the release of specific student data to be linked to their survey responses.Of those enrolled in a ZCCM course, 426 provided their student ID number and completed the survey.The students who correctly recorded their ID number on the survey and were in a ZCCM course that could be matched to a comparison section comprised a treatment group of 198 students (ZCCM cohort; see Figure 1).Their survey responses were linked with demographic data and student success indicators.The same set of demographic data and student success indicators were obtained for a control group of 2,732 students (comparison cohort) who were enrolled in a prior offering of the course from the same instructor when commercial materials were required.

Data Collection
We worked with UC Merced's Institutional Research & Decision Support (IRDS) and campus registrar to obtain final grade data, drop and withdrawal rates, and credit unit enrollment intensity.We obtained demographic and background data for students in both treatment and control groups, including gender, race/ethnicity, field of study (student major), year of study, grade point average, on-campus housing/commuter status, and first-generation college student status.For analysis, final grade data were recoded from a letter grade into a four-point scale, in which an F grade was valued at 0 and A and A+ grades valued at 4. Students' course drop, withdrawal, and failure rates were used to calculate course completion.
Student self-identified race and ethnicity data were categorized using Integrated Postsecondary Education Data System (IPEDS) classifications and described at the highest specificity that maintains student privacy.
These variables were collected to control for non-treatment effects and differences between the cohorts, and we hoped to learn if any student populations had positive or negative outcomes based on zero-cost material or OER usage (Colvard et al., 2018;Nusbaum et al., 2020).
The survey asked ZCCM students about their use and perceptions of course materials (including zero-cost materials), their textbook purchasing behavior, and textbook costs.The survey was developed by reviewing surveys from the literature (Bliss et al., 2013;Dastur et al., 2015;Open Education Group, n.d.-b).
Researchers obtained an exempt approval for this project from the UC Merced Institutional Review Board.

Data Analysis
Demographic data describing the treatment and control groups were compared using a chi-squared test or t-test to identify significant differences between the groups.To examine the impact of zero-cost materials on final grade, final grade of C− or higher, enrollment intensity, and course completion, we applied a doubly robust (DR) treatment effects model with an inverse-probability-weighted-regression-adjustment (IPWRA) estimator, a statistical technique applied to estimate the effect of an intervention while effectively minimizing the selection bias via modeling the selection process and outcome mechanism simultaneously (Dudik et al., 2011;Funk et al., 2011;Hall et al., 2015;Wooldridge, 2010).Prior research found that such an approach consistently estimates the unbiased average treatment effects if the propensity score model or the outcome model is correctly specified (Bang & Robins, 2005;Sant'Anna & Zhao, 2020;Tan et al., 2022).
All the DR models matched and controlled for students' gender, race/ethnicity, on-campus housing/commuter status, first-generation college student status, major of degree, and semester.We analyzed survey data to examine students' experiences and perceptions of zero-cost materials as compared with commercial texts using descriptive statistics, chi-squared tests, or t-tests.STATA 17 was used for all of the analyses.

Cost Savings
Between 2019 and 2021, the ZCCM grant resulted in student textbook savings of $202,899.This estimate uses the sticker price method to calculate savings for the total number of students enrolled in the 15 grantsupported courses.The highest cost savings were generated by two introductory courses: Biology 001 and Economics 001.Biology 001 enrolled 515 students and previously used a textbook that cost $81.Though Economics 001 enrolled about half that number of students, its prior textbook was $248.Cost savings are reported for the first semester in which zero-cost materials were implemented and do not represent continued savings over subsequent semesters.
Survey data point to the importance of cost savings for students.Ninety-one percent of student survey respondents indicated that potential cost savings of course materials are moderately to extremely important.During course registration, grant-supported courses were listed with a "no textbook costs associated with this section" label.Of the 24% of students who indicated seeing this label, 64% reported that this statement influenced their decision to register for the section.Students were later asked to imagine a future course with two identical sections, one using zero-cost materials and the other section using commercial texts.Eighty-four percent of students indicated that they would enroll in the section using zerocost materials, with 15% indicating no preference and 1% preferring the section using commercial texts.

Cohort Comparison
Before analyzing impacts on student outcomes, we ran descriptive statistics for both groups, which are displayed in Table 1.The ZCCM and comparison cohorts were found to be largely comparable, with no significant difference by gender, race/ethnicity, on-campus housing/commuter status, school year, or number of credit hours.However, students in ZCCM classes were less likely to be first-generation college students (p = .027).There were some differences in the declared major between the two cohorts; for example, the comparison cohort comprised a larger proportion of students majoring in biological sciences.

Impact on Student Outcomes
When matched and controlled for student demographics and characteristics to isolate the treatment effect, students in the ZCCM group had outcomes that were better and no different than those in the comparison group, as summarized in Table 2.

Final Course Grades
The control and treatment groups did not differ in final grade (p = .091).These comparable or similar final grades align with the literature (Clinton, 2018;Clinton & Khan, 2019;Grewe & Davis, 2017).Since the same instructor taught the courses compared, one might expect little fluctuation in final grades despite different course materials being used.This finding may suggest that instructors' selections of course materials were not significantly inferior or superior to their prior choices.

Passing With a C− or Better
No differences were found between ZCCM and control groups in the attainment of a final grade of C− or above (p = .076),which is similar to what studies with aggregated findings revealed (Robinson, 2015;Winitzky-Stephens & Pickavance, 2017).As with final grades, instructor consistency across comparison groups may have contributed to this finding.Grimaldi et al.'s 2019 work would expect this outcome.Their access hypothesis suggests that OER primarily benefits a subset of students who would not have purchased the course text due to the expense; therefore, any positive learning outcomes from OER adoption impact a smaller group of students and are unlikely to yield a measurable effect.

Course Completion Rates
In our statistical analysis, there was evidence that students enrolled in ZCCM courses had higher course completion rates (p = .002)than their peers in the control group.This finding was significant, though the effect size was small.This result aligns with the majority, though not all, of the literature on OER and course retention.However, this finding is not strongly supported by our survey data.Almost 92% of respondents reported that they had never (80.11%) or rarely (11.83%) dropped or withdrawn from a course due to the cost of the course texts.Still, this finding suggests that textbook costs may compromise students' ability to complete a course.

Credit Hours Enrolled
The two groups (p = .263)did not differ significantly in the number of credit hours enrolled, suggesting that students in ZCCM courses were not compelled to take on more credit hours due to the reduced cost of course materials, even though 20% of student survey respondents indicated that they had previously enrolled in fewer units due to the cost of the course text, and studies have found that students taking an OER course have, on average, enrolled in a higher number of units in contrast to a control group of peers (Fischer et al., 2015;Robinson, 2015).Other factors may be more influential in students' enrollment decisions.A campus survey asked first-year students to report why they had enrolled in fewer than 15 units, and the most

Student Use and Perception
ZCCM students were surveyed about their usage of required texts in their courses overall as compared with their usage of required zero-cost materials in their grant-supported course (see Figure 2).Students reported using the required texts in their other courses more frequently than they reported using zero-cost materials (3.7 [SD:1.2] vs. 3.4 [SD:1.4],p < .001).Fifty-one percent of respondents indicated that, on average, they used their required texts once a week or more, as opposed to 46% of students who reported using required zero-cost materials at the same frequency.Gurung's 2017 analysis highlights a similar finding, in which students reported reading and studying less when using OER versus commercial textbooks.Gurung suggests this difference can be attributed more to student engagement patterns with print versus online materials than to the quality of the content.There may be several reasons students reported using zero-cost materials less frequently than the texts in other courses, such as zero-cost materials being adapted or created specifically to support in-class lecture and activities, or the implementation of zero-cost materials coinciding with fewer assessments that relied on course readings.
When asked about the percentage of materials used, 28% of students reported using a larger amount of commercial texts than zero-cost materials, while 25% reported using a larger amount of zero-cost materials than commercial texts.Forty-seven percent selected "Unsure" in response to this question.See Figure 3 for a summary of student responses on material usage.Students were also asked to rate the quality of the text(s) used in the zero-cost materials course in contrast to texts in other courses (see Figure 4).Over half of the students (52%) thought the zero-cost materials were of similar quality to other texts with costs while 33.8% reported that they were of better or much better quality, and under 8% of students reported the quality of the zero-cost materials as being worse or much worse.(2018) propose that the free nature of OER can provoke a halo effect on quality ratings, these positive perceptions may also be driven by desirable features found in freely available resources (Brandle et al., 2019;Hilton III & Laman, 2012;Jhangiani et al., 2018).When asked about features of zero-cost materials, students indicated that the ability to access materials immediately (91%) and the convenience and portability of the digital format (87%) was moderately to extremely important.
In addition, students were asked to compare their experiences in the ZCCM course to those in previous courses.A small percentage of students reported negative impacts associated with the ZCCM course, while most students noticed a positive or neutral effect, as summarized in Table 3.In their systematic review, Mullens and Hoffman (2023) note six studies that found an increase in student engagement.The authors caution that engagement is not well defined through self-reported measurements but stipulate that access to materials and interactive features of OER could be driving engagement.Another driver could be changes in instructors' pedagogical approach, associated with their adoption of OER.

Limitations and Future Directions
Though we used the DR estimator to address potential selection bias in this observational study, some bias may be present due to recruitment challenges, which became more difficult when classes pivoted to online instruction in March 2020 due to COVID-19.This resulted in uneven levels of student participation across grant-supported courses.For example, biological science majors comprised 6.6% of our ZCCM cohort, while they made up 31.2% of the comparison group.
Many students who consented to participating in this research project incorrectly entered their student ID numbers on the survey, which prevented us from matching many survey responses to student data.This limited our ability to perform more granular analyses of the impact of zero-cost materials on specific

95
student groupings, such as historically underrepresented students.Furthermore, this study only compared courses taught by the same instructor and did not account for variations in learning experiences (e.g., inperson vs. online).The current study did not attempt to assess the potential effects of the zero-cost material type (e.g., open textbooks, course-specific materials authored by the instructor, library e-resources) or how the instructors used the materials with their students.For future assessments, we plan to obtain students' Pell Grant eligibility to examine how this group of students who qualify for low-income aid may be impacted by textbook costs and the use of zero-cost materials.

Conclusion
Our research using the COUP framework contributes to a growing body of literature confirming beneficial cost savings for students using zero-cost materials without jeopardizing students' success.Students in our ZCCM cohort had higher course completion rates than their comparison group of peers, and no differences were found between these cohorts in final grades or passing with a C− or better.In addition, enrollment intensity did not vary.These results can alleviate any local concerns about the efficacy of using OER and/or library e-resources in course curricula.Though students' perceptions of zero-cost materials were positive, more investigation is warranted into why students may have used these materials less frequently than other texts.Usage may have been impacted by the challenges related to unexpected distance education, a primary mode of instructional delivery for some of the classes in this study.While this less frequent engagement with the materials did not seem to negatively impact students' academic success, it reminds us of the multiple dimensions that contribute to student learning and the need for additional research to uncover what elements of zero-cost materials-whether they are OER or library e-resources-may facilitate student learning.Also of concern, and another area for further study, is the potential impact of textbook costs on historically underserved student populations.These populations may be the ones most likely to lack access to course materials.Feedback from students confirms that they pay attention to textbook costs and seek to avoid or minimize them.Using zero-cost materials relieves some financial burden from our students while still supporting their academic success.These positive outcomes should encourage institutions, librarians, and faculty to strongly advocate for the use of OER and/or library e-resources in course curricula.Ongoing work should also consider how to promote further integration and engagement of course materials in curricula.
released under an open license that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others" encapsulates the growing open movements at the time, including open courseware, open learning solutions, open access publishing, and open licensing common reason (47%) was a lack of open seats in the section(s) needed (UC Merced, Institutional Research & Decision Support, 2022b).Other factors influencing enrollment choices may revolve around times courses are offered, perceived course difficulty, or commitments to activities such as employment or student research.

Table 1
Demographic Characteristics by ZCCM Treatment Group Versus Comparison Group

Measuring the Impact of an Open Educational Resource and Library e-Resource Adoption Program Using the COUP Framework Davidson Squibb, Salmon, and Yan
The sum of the percentages may not round up precisely to 100 due to the precision of one-

Table 2
Student Outcomes by ZCCM Treatment Group Versus Comparison Group

Table 3
Experiences in Zero-Cost Courses as Compared With Previous Courses