International Review of Research in Open and Distributed Learning

Volume 17, Number 6

December - 2016

Maintaining Momentum Toward Graduation: OER and the Course Throughput Rate

John Hilton III¹, Lane Fischer², David Wiley³, and Linda Williams^4
1,2,3Brigham Young University, ³Lumen Learning, ⁴Tidewater Community College

Abstract

Open Educational Resources (OER) have the potential to replace traditional textbooks in higher education. Previous studies indicate that use of OER results in high student and faculty satisfaction, lower costs, and similar or better educational outcomes. In this case study, we compared students using traditional textbooks with those using OER at Tidewater Community College to compare their performance on what we call course throughput rates, which is an aggregate of three variables - drop rates, withdrawal rates, and C or better rates. Two self-selecting cohorts were compared over four semesters, with statistically significant results. The study found that, subject to the limitations discussed, students who use OER perform significantly better on the course throughput rate than their peers who use traditional textbooks, in both face-to-face and online courses that use OER. This suggests that OER are a promising avenue for reducing the costs of higher education while increasing academic success.

Keywords: open educational resources, computers in education, textbooks, financing education

Introduction

The high cost of textbooks represents a significant challenge in higher education in the United States. A survey of 22,129 post-secondary students in Florida found that 64% of students reported not purchasing a required textbook because of its high cost (Florida Virtual Campus, 2012). In the same study, nearly half of students reported that the cost of textbooks caused them to take fewer courses, and one-third stated that they had earned a poor grade in a subject because they could not afford to buy the textbook.

One solution to the high cost of textbooks is found in a replacement for textbooks called Open Educational Resources (OER). The term Open Educational Resources comes from the 2002 UNESCO Forum on the Impact of Open Courseware for Higher Education in Developing Countries, which defined OER as "The open provision of educational resources, enabled by information and communication technologies, for consultation, use and adaptation by a community of users for non-commercial purposes" (UNESCO, 2002, p. 24). In essence, OER are free educational materials that are licensed in such a way so as to grant legal permissions to reuse, remix, and redistribute them. Wiley, Bliss, and McEwen (2014) chronicle many aspects of the growth and development of OER since its inception.

To date, a wide variety of high-quality OER have been created, although there are varying quality of OER and they are not all a natural substitute for conventional paid material. Nevertheless, many OER are sufficiently robust so as to be useful in replacing traditional textbooks. These OER typically have Creative Commons licenses that provide the legal permissions necessary to free share, modify, and reuse them (Bissell, 2009; D'Antoni, 2009; Hewlett, 2013). The Minnesota Open Textbook Library (open.umn.edu/opentextbooks/) provides a repository of open textbooks and also hosts faculty reviews of these materials. Despite the widespread intuition that freely available educational materials must be less effective or of lower quality than expensive, published materials, emerging research demonstrates otherwise.

Although available OER has grown dramatically in the past decade, Allen and Seaman (2014) found that across a nationally representative survey of 2,144 college faculty members in the United States, only 34% of respondents were aware of OER. However, several studies have shown faculty who are familiar with OER tend to have positive perceptions of it. Pitt (2015) demonstrated that faculty who utilize OER viewed it positively. She surveyed 127 teachers who adopted an open textbook and found that approximately 75% of faculty members said that after using open textbooks as their primary teaching tool, they would continue to use open textbooks in the future. Similarly, Bliss, Robinson, Hilton, and Wiley (2013) examined OER adoption at several colleges and surveyed approximately sixty faculty members regarding their experiences using OER. They found that 55% of teachers who had used OER said that the OER were of the same quality as the materials that were previously used, and 35% felt that they were better, with the remaining 15% feeling that they were worse. A recent study of faculty perceptions of OER at British Columbia post-secondary institutions produced similar results (Jhangiani, Pitt, Hendricks, Key, & Lalonde, 2016). Similar positive findings have also been demonstrated at the secondary level (Kimmons, 2015).

Students also tend to be very positive about their use of OER. Feldstein, Martin, Hudson, Warren, Hilton, and Wiley (2012) surveyed 1,393 students at Virginia State University who utilized OER as a substitute for traditional textbooks. Out of the 315 students who responded, 95% strongly agreed or agreed that OER were "easy to use," and 78% felt OER "provided access to more up-to-date material than is available in my print textbooks" (para. 29). Two-thirds of students either agreed or strongly agreed that the digital OER were more useful than traditional textbooks and stated they preferred OER to traditional textbooks. One reason students support OER relates to the cost savings, significant in many instances. Huggins and Smith (2015) wrote that the launch of an OER initiative at Kaplan University had led to cost-savings of more than one million dollars over a one-year period of time.

While cost-savings are important to some educators, the more vital issue relates to student learning. Pawlyshyn, Braddlee, Casper, and Miller (2013) reported on the adoption of OER at Mercy College. The researchers compared the pass rates of students who used OER in the Fall of 2012 with those who had used traditional materials in the Fall of 2011. The pass rates of math courses increased from 63.6% in Fall 2011 to 68.9% in Fall 2012. The contrast between the Spring of 2011 (no OER, pass rate of 48.4%) and the Spring of 2013 (OER, pass rate of 60.2%) is even more significant. However, it must be pointed out that the change in textbooks coincided with a change in pedagogy, which may have been the decisive factor in the growth in student learning.

Other studies have not shown such dramatic results when OER were adopted. Allen et al. (2015) examined the use of a collection of OER called ChemWiki in a chemistry class taught at the University of California-Davis. Students in back-to-back hours were taught by the same instructor with the same teaching assistants. In one section, 478 students used ChemWiki as the primary learning resource, while the other class (of 448 students) used a traditional textbook. Pretests indicated no significance differences between the two groups. All students took the same midterm and final exam, and there were no significant differences between the overall results of the two groups. A review of several additional studies by Hilton (2016) indicates that generally, students who use OER tend to do as well or better than their peers using traditional textbooks in terms of course completion and passing rates.

Thus, the literature seems to indicate that open textbooks are connected with high student and faculty satisfaction, lower costs, and similar or better educational outcomes. However, it should be noted that the literature to date is relatively sparse. One aspect that has yet to be studied is the relationship between the students dropping courses and courses that utilize open textbooks. One theory put forth by Wiley, Williams, DeMarte and Hilton (2016) is that if students who use OER drop classes at lower rates than their peers who use traditional textbooks, then institutions of higher education stand to gain financially through OER adoption, as they will retain tuition money that they would otherwise need to refund. These researchers studied drop rates at Tidewater Community College in the Fall 2013 and Spring 2014 semesters. They found that there was a small but statistically significant difference in drop rates between courses using OER versus those that utilized traditional textbooks. This study was limited by the small amount of data available in the initial pilot semesters, as well as in that the data for both online and face-to-face sections were aggregated and not reported separately.

Another issue connected to drop rates is the overall rate of student success, which we refer to in this paper as "course throughput rate." There are multiple filter points at which a student might not successfully "survive" a class. A student could drop the class before the add/drop deadline, or might withdraw from the class, or could persist in the class but not pass it. The purpose of the present study is to expand on the research done by Wiley et al. (2016) and examine course throughput rates across the four semesters of the pilot program at Tidewater Community College. Specifically, the research questions that guide the present study are as follows:

1. What was the difference (if any) in the drop rates between students taking courses using OER versus those using traditional textbooks for both online and face-to-face classes?
    
2. What was the difference (if any) in the withdrawal rates between students taking courses using OER versus those using traditional textbooks for both online and face-to-face classes?
    
3. What was the difference (if any) in the proportion of students getting a C grade or better in the courses between students taking courses using OER versus those using traditional textbooks for both online and face-to-face classes?
    
4. What is the cumulative impact of these three effects (if any), which we call the course throughput rate?

Context

Founded in 1968 as a part of the Virginia Community College System, Tidewater Community College (TCC) has multiple campuses in Virginia. TCC is the largest provider of higher education and workforce services in Hampton Roads, Virginia, enrolling almost 40,000 students annually - the second largest undergraduate student body in the Commonwealth of Virginia. TCC is 14th largest public two-year community college in the U.S., and the second largest provider of undergraduate public education in Virginia. It also has the largest undergraduate African American enrollment in Virginia higher education, and is the seventh largest associate degree producer among two-year colleges for African American students in the nation. TCC is the 16th largest associate degree producer in the U.S. among two-year institutions and offers 12 nationally accredited degree programs. The student body is diverse and is comprised of 45% White, 34% African American, and 11% other minorities. Of the students attending TCC, either full or part-time, 56% receive financial aid (TCC Fact Book, 2015).

In January 2013, Tidewater Community College began the process of becoming the first college in the U.S. to create an Associate of Science degree based entirely on openly licensed content. This program is referred to as a "Z Degree," referring to a degree with "Zero" textbook costs. The Z Degree is made up of a series of "Z Courses," which are courses with "Zero" textbook costs. The goals of the Z Degree are threefold: 1) to improve student success, 2) to increase instructor effectiveness, and 3) to save students money. Courses were stripped down to the learning outcomes and rebuilt using openly licensed content, reviewed and selected by faculty based on its ability to facilitate student achievement of the objectives.

Acknowledging the tendency for course design to be based upon anecdotal feedback or "gut feel," the courses that make up the Z Degree were designed in a systematic data-driven process that closely examined existing curriculum frameworks and sought ways to increase their efficiency and effectiveness. Essential to this process was the implementation of an outcomes based backwards design process. The argument for backwards curriculum design and laser- focused alignment of course materials has been made after realizing that this process gives educators the ability to control curriculum, use current information, and most importantly, foster student success. The combined efforts of a 13-member faculty team, college staff, and administration culminated on August 22, 2013 when more than 420 students enrolled in the first 16 "Z Courses." Additional courses were launched in Spring 2014, completing the path to the degree for TCC's business administration students. The courses that comprise the Z Degree are designated as "Z courses" within the registration system that students use to enroll in classes. These Z courses are delivered online, face-to-face, and in hybrid/blended formats and are taught by both full-time and adjunct faculty. Enrollment in Z courses is open to all TCC students, regardless of their field of study. They use the same process for enrollment as the other courses at TCC and are filled on a first-come, first-served basis.

Method

Data for this case study were drawn from the Tidewater Community College institutional research database in Fall 2013, Spring 2014, Fall 2014, and Spring 2015. Data included the drop rates, withdrawal rates, and final grades in courses with non-Z and Z sections in the same semester. Most Tidewater classes are 16-week courses; however, 8 and 12-week versions of some classes are available. We obtained drop rate data for only the 16-week courses, and withdrawal and grade data for all courses, leading to a difference in the total n population between these figures.

Outcomes were analyzed separately according to their modality: face-to-face or online/hybrid. Data were aggregated across four semesters: Fall 2013, Spring 2014, Fall 2014, and Spring 2105. Non-Z sections (sections requiring commercial publisher materials) were labeled control, while Z-sections (OER) were considered treatment. There were 67 courses taught across the four semesters that had both treatment and control sections. Because of the much greater availability of traditional (Non-Z) courses, there were notably more students enrolled in control sections than treatment sections. Because of the discrepancy in sample sizes, unequal variances were assumed in each statistical test. For the course throughput rate we computed Cohen's d using a logit method.

Results

Table 1 summarizes the results of our analyses:

Table 1

Differences Between Non-Z and Z Courses

Note: Calculations: Control group proportions minus treatment group proportions

The first research question we asked was whether there was a significant difference between the control and treatment groups in terms of the proportion of students that dropped the course before the deadline to receive a tuition refund. In face-to-face courses, 2.3% of the control subjects dropped, while 1.8% of treatment subjects dropped their courses. A two sample Z-test of differences in proportions rendered a value of 1.29, which was not statistically significant (p = 0.19). In online/hybrid courses, 4.0% of the control subjects dropped, while 1.4% of treatment subjects dropped their courses. A two sample Z-test of differences in proportions rendered a value of 4.66, which was significant (p < 0.001).

The second research question asked whether there were significant differences between the control and treatment groups in terms of the proportion of students that withdrew after the drop deadline and did not receive a tuition refund. In face-to-face courses, 9.9% of the control subjects withdrew, while 8.1% of treatment subjects withdrew. A two sample Z-test of differences in proportions rendered a value of 2.07, which was significant (p = 0.04). In online/hybrid courses, 13.7% of control subjects withdrew, while 13.1% of the treatment subjects withdrew. A two sample Z-test of differences in proportions rendered a value of 0.52, which was not significant (p = 0.60).

The third research question asked whether there was a significant difference between the control and treatment groups in the proportion of students that achieved a C grade or better. In face-to-face courses, 68.0% of control subjects received a C grade or better, while 73.7% of the treatment subjects received a C grade or better. A two sample Z-test of differences in proportions rendered a value of -4.29, which was significant (p <.001). In online/hybrid courses, 65.5% of control subjects received a C grade or better, while 69.8% of the treatment subjects received a C grade or better. A two sample Z-test of differences in proportions rendered a value of -2.58, which was significant (p = 0.009).

The fourth research question combined the drop, withdrawal, and C or better rates into a single metric we call the "course throughput rate" to estimate the differences between the groups in their overall success rate from registration to final grade. We used the following calculations to determine the course throughput rate: (total students registered on day 1) x (1-drop rate) x (1-withdrawal rate) x (percent passing with a C or better) / (total students registered on day 1) = course throughput rate. In the face-to-face courses, 59.8% of students in non-Z courses made it through the successive hurdles of drop, withdrawal, and passing the class, compared with 66.4% of students in the Z courses, for a difference of 6.6%. A two sample Z-test of differences in proportions rendered a value of -4.66, which was significant (p <.001). Cohen's d = 0.15, a positive but small effect. In the hybrid/online courses, 54.2% of students who started in non-z courses successfully made it through the course with a C or better, compared with 59.8% of students in the Z courses, for a difference of 5.6%. A two sample Z-test of differences in proportions rendered a value of -3.13, which was significant (p =.002). Cohen's d = 0.12, which, similar to the face-to-face courses, represents a positive but small effect.

Discussion

The results of this study indicate that students do no worse academically when they enroll in course sections that do not require them to purchase commercial textbooks. In fact, students in face-to-face Z courses were significantly less likely to withdraw from a course and more likely to receive a C or higher in the course than their peers who took non-Z courses. Overall, these students enrolled in face-to-face Z courses were almost 7% more likely to succeed than those who took non-Z courses. Similarly, students in hybrid/online Z courses were significantly less likely to drop out of a course and more likely to receive a C or higher in the course than their peers who took non-Z distance courses. The success rate of students in the distance Z courses was nearly 6% higher than those in the distance non-Z courses. Whether in face-to-face or online courses, these improved student success rates translate into more students moving forward toward graduation without repeating courses.

The design of this study cannot establish causation, and one must be careful in making statements about results, particularly given the limitations described in the following section. At the same time, the results of this study are roughly in line with similar previous studies that have shown that students enrolled in courses that use OER perform as well or better than their peers who use traditional textbooks (Hilton, 2016). Thus, this study adds to the body of research suggesting that OER are a promising avenue for reducing the costs of higher education without compromising academic success. It may be that increasing student access to learning materials is connected with their increased academic success.

A unique contribution of the present study is that it examines the changes to the course throughput rate of students whose faculty assign OER. While individually the differences in drop, withdrawal, and passing rates are important, it is also useful to examine their collective influence. In the present study, students in both face-to-face and distance Z courses performed higher in two of these three categories, which led to a larger cumulative effect in student success. Not only did students in the face-to-face courses pass the class at a higher rate than those enrolled in the non-Z courses, but there was also a higher proportion of students who did not withdraw. While we might hypothesize that students who withdrew from a course would have done worse than those who do not, in this case, even though the face-to-face Z classes had a higher proportion of students who did not withdraw, they still performed better than their peers. Similarly, we might suspect that students who drop a course are less likely to pass the class than students who remained. Thus, we would hypothesize that in the hybrid/online Z sections, the students who remained would perform worse since their peers because fewer of them dropped out initially. However, students passed the class at higher rates, even though the proportion of those who had dropped the class was smaller.

Limitations

While the results of this study do suggest significant value stemming from courses offered with zero textbook costs, there are a number of limitations which must be addressed, particularly since we are only presenting a case study of a pilot program. One of these limitations is the large disparity in numbers between the enrollment numbers in the non-Z and Z sections of courses. The Z-degree program is still in its infancy, and the total number of Z courses offered is small relative to the non-Z courses. A related weakness is that OER adoption or non-adoption is perfectly confounded with faculty in this analysis. It is possible that the observed differences in course throughput rates are more attributable to differences in faculty than differences in the price and licensing of the required instructional materials they assigned. Given that the teachers of Z-degree courses in these four pilot semesters were largely those who were involved in the creation of the courses (and may have been selected for this role because of their teaching abilities), there is a possibility that selection bias plays an important factor in the present study.

An additional limitation of this study is that we do not have data indicating that the student groups in the two sets of courses are equivalent. It is possible that certain types of students specifically seek out Z courses, and that these students tend to do better academically. At the same time, we do not have any evidence suggesting this is the case; indeed, it is equally possible that the students who seek out Z courses would generally tend to do worse academically, thus understating the results of this study. This limitation could be addressed in future studies by focusing on a few specific courses where pre-tests could help assess the initial equivalency of the two groups.

Conclusion

We believe that the course throughput rate, which provides the combined effect of drops, withdrawals, and final grades, is a critical student success metric that merits additional study and perhaps expansion. Improving any of its component measures is only potentially helpful - students who do not drop a course may still withdraw and students who do not withdraw from a course may still fail. Consequently, the most powerful educational interventions may be those that lead to increases in the overall course throughput rate, which could in turn lead to higher graduation rates. While additional methodological rigor will be required in future studies before OER can be said to cause improvements in course throughput rates, the current study demonstrates that OER adoption by faculty is associated with improvements in course throughput rates. This finding merits further exploration.

Conflict of Interest

David Wiley is the Chief Academic Officer for Lumen Learning, which provides support to Tidewater Community College through their ongoing process of adopting and improving Z courses.

Funding

This work was supported in part by the William and Flora Hewlett Foundation.

References

Allen, I. E., & Seaman, J. (2014). Opening the curriculum: Open educational resources in U.S. higher education. Retrieved from www.onlinelearningsurvey.com/oer.html

Allen, G., Guzman-Alvarez, A., Smith, A., Gamage, A., Molinaro, M., & Larsen, D. S. (2015). Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for traditional general chemistry textbooks. Chemistry Education Research and Practice, 16(4), 939-948.

Bliss, T., Robinson, T. J., Hilton, J., & Wiley, D. (2013). An OER COUP: College teacher and student perceptions of open educational resources. Journal of Interactive Media in Education, 17(1), 1-25.

Bissell, A. (2009). Permission granted: Open licensing for educational resources. Open Learning, The Journal of Open and Distance Learning, 24, 97-106.

D'Antoni, S. (2009). Open educational resources: Reviewing initiatives and issues. Open Learning, The Journal of Open and Distance Learning, 24, 3-10.

Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open, Distance and E-Learning, 15, 2.

Florida Virtual Campus. (2012). 2012 Florida student textbook survey. Tallahassee, FL. Retrieved from http://www.openaccesstextbooks.org/pdf/2012_Florida_Student_Textbook_Survey.pdf

Hewlett. (2013). Open educational resources. Retrieved from http://www.hewlett.org/programs/education-program/open-educational-resources

Hilton, J. (2016). Open educational resources and college textbook choices: a review of research on efficacy and perceptions. Educational Technology Research and Development, 64(4), pp. 573-590.

Huggins, S., & Smith, P. (2015). Using an ‘open approach ‘to create a new, innovative higher education model. Open Praxis, 7(2), 153-159. Retrieved from http://www.openpraxis.org/index.php/OpenPraxis/article/view/193/159

Jhangiani, R. S., Pitt, R., Hendricks, C., Key, J., & Lalonde, C. (2016). Exploring faculty use of open educational resources at British Columbia post-secondary institutions. BCcampus Research Report. Victoria, BC: BCcampus. Retrieved from https://bccampus.ca/files/2016/01/BCFacultyUseOfOER_final.pdf

Kimmons, R. (2015). OER quality and adaptation in K-12: Comparing teacher evaluations of copyright-restricted, open, and open/adapted textbooks. The International Review of Research in Open and Distributed Learning, 16(5).

Pawlyshyn, N., Braddlee, Casper, L., & Miller, H. (2013, November 4). Adopting OER: A case study of cross-institutional collaboration and innovation. Educause Review. Retrieved from http://www.educause.edu/ero/article/adopting-oer-case-study-cross-institutional-collaboration-and-innovation

Pitt, R. (2015). Mainstreaming open textbooks: Educator perspectives on the impact of openstax college open textbooks. International Review of Research on Open and Distributed Learning, 16(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2381/3497

TCC Fact Book. (2015). Office of institutional effectiveness (34^th Ed.). Tidewater Community College. Retrieved from http://web.tcc.edu/welcome/collegeadmin/OIE/IR/factbook/

UNESCO. (2002). Forum on the impact of open courseware for higher education in developing countries: Final report. Retrieved from http://unesdoc.unesco.org/images/0012/001285/128515e.pdf

Wiley, D, Bliss, T.J., & McEwen, M. (2014). Open educational resources: A review of the literature. In J.M. Spector, M.D. Merrill, J. Elen, & M.J. Bishop (Eds.), Handbook of research on educational communications and technology (781-789). New York: Springer.

Wiley, D., Williams, L., DeMarte, D., & Hilton, J. (2016). The tidewater z-degree and the INTRO model for sustaining OER adoption. Education Policy Analysis Archives, 24(41), pp. 1-12. Retrieved from http://epaa.asu.edu/ojs/article/view/1828

Maintaining Momentum Toward Graduation: OER and the Course Throughput Rate by John Hilton III, Lane Fischer, David Wiley, Linda Williams is licensed under a Creative Commons Attribution 4.0 International License.