The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions

Assigning open textbooks in college and university courses can help students save money on increasingly expensive commercial textbooks, and recent research shows that this savings can often be achieved with little to no sacrifice in textbook quality or student learning outcomes. We add to this body of research by examining the use of an open textbook in an introductory physics course at a large research university in Canada that enrols approximately 800-900 students per year. In this course, the instructors revised an open textbook and combined it with other learning resources onto a single website, whereas more than one source of learning materials was used previously. We used the COUP framework to structure our analysis, focusing on cost, outcomes, use, and perceptions in relation to the open textbook assigned in the course. Through the use of a survey of students and data about student learning outcomes in the form of final exam and course grades, and shifts on the pre-/postColorado Learning Attitudes about Science Survey, we show that student savings by moving to an open textbook were accompanied by little change in learning outcomes. We also show that the vast majority of survey respondents perceived the open textbook to be of the same or better quality than commercial textbooks used in their other courses. Further, many of them appreciated the fact that the textbook was customized to this particular course— which is made possible by the use of a textbook with an open license.


Résumé de l'article
Assigning open textbooks in college and university courses can help students save money on increasingly expensive commercial textbooks, and recent research shows that this savings can often be achieved with little to no sacrifice in textbook quality or student learning outcomes. We add to this body of research by examining the use of an open textbook in an introductory physics course at a large research university in Canada that enrols approximately 800-900 students per year. In this course, the instructors revised an open textbook and combined it with other learning resources onto a single website, whereas more than one source of learning materials was used previously. We used the COUP framework to structure our analysis, focusing on cost, outcomes, use, and perceptions in relation to the open textbook assigned in the course. Through the use of a survey of students and data about student learning outcomes in the form of final exam and course grades, and shifts on the pre-/post-Colorado Learning Attitudes about Science Survey, we show that student savings by moving to an open textbook were accompanied by little change in learning outcomes. We also show that the vast majority of survey respondents perceived the open textbook to be of the same or better quality than commercial textbooks used in their other courses. Further, many of them appreciated the fact that the textbook was customized to this particular course-which is made possible by the use of a textbook with an open license.

Introduction
The cost of textbooks at North American colleges and universities is a source of concern for many students, faculty, and staff. In August 2016 the U.S. Bureau of Labor Statistics reported that the price of 79 college textbooks has increased 88% since 2006, and is increasing at a faster rate than college and university tuition and fees (U.S. Bureau of Labor Statistics, 2016). Still, students are spending less on textbooks over time, not more: according to Kestenbaum (2014), student spending on textbooks went down from 2002-2013, and the average number of textbooks purchased per year by post-secondary students went down from about 13 to just over 8. Other research shows a significant percentage of students simply going without textbooks for some of their courses due to rising costs, as discussed below.
Thus, as textbook prices go up, students seem to be responding by trying to get by in at least some of their courses without the required textbooks, which could negatively impact their learning. In a survey of over 22,000 post-secondary students in Florida, 38% said they believed they did worse in a course because they could not afford the textbook, and 20% said they had failed a course for that reason (Florida Virtual Campus, 2016). The high cost of textbooks is not just a financial concern, but a pedagogical one as well.
One way to address the problem of students' failure to access the required materials for their courses due Research focusing on "cost" includes studies of how much students typically spend on textbooks, how much they save in particular courses by using open textbooks, and how the cost of textbooks affects students' academic choices. The literature so far suggests that many students are paying at least $600 (in U.S. or Canadian currency) per year on textbooks. In a survey of over 22,000 students from 40 universities and colleges in Florida, 53% of students reported that they spent over $300 on books in the Spring 2016 semester, and 33% spent over $400 (Florida Virtual Campus, 2016). In the Canadian context, Jhangiani & Jhangiani (2017) surveyed 320 undergraduates in various postsecondary institutions in British Columbia in 2015 and asked how much they had spent on textbooks in the last 12 months; the mean was about C$700 (median was $500). In a 2016 survey of about 1500 students from the University of British Columbia, nearly half of respondents (49%) said they spend at least C$500 on textbooks each year, and the mean reported cost per year for textbooks was $620 (So & Doering, 2016).
The high cost of textbooks is, not surprisingly, affecting students' academic behaviours, such as whether or not they purchase the required books for their courses and which courses they choose to take. Florida Virtual Campus (2016) asked over 22,000 post-secondary students whether the cost of textbooks had led them to do the following actions (among others) during their post-secondary studies (numbers below refer to the percentage of students who said they had done each action seldom, frequently, or often):  Didn't buy the required book(s) for a course because of cost: 67%.  Dropped a course because of the cost of the book(s): 26%.
A report by Senack and the Student PIRGs (2014), using data from a survey of over 2000 students in 150 colleges and universities in the US, provided similar numbers: 65% of student respondents reported that they had decided not to buy a textbook for one or more of their courses because of cost, and 48% said that the cost of textbooks had somewhat or significantly affected which or how many courses they were able to take each semester. Jhangiani & Jhangiani (2017) report that of over 300 students in British Columbia, 54% said they had not bought a textbook for a course at least once due to cost, 27% said they had taken fewer courses, 26% said they did not register for a particular course because of the cost of the book(s), and 17% said they had dropped or withdrawn from a course. So & Doering (2016) reports that 75% of 1500 students at the University of British Columbia said they had gone without purchasing a textbook for at least one course. The research on cost reveals that the price of commercial textbooks is affecting students not just financially, but also in terms of their learning. courses, and students in the control group (N=9264) were asked to purchase traditional textbooks for different sections of the same courses. The authors found that the students using OER did just as well or better in most of the courses studied in terms of course grades and completion rates. Allen et al. (2015) studied students in a chemistry course at the University of California, Davis who were assigned a traditional textbook (448 students) and students in another section of the same course taught by the same instructor (478 students) that used Chemwiki (https://chem.libretexts.org/)as a primary course resource (an openly licensed wiki site that can be used as a textbook). They found no significant differences in outcomes between these students, as measured through pre-and post-tests on content covered in the course, the same midterm and final exams, and the administration of the Colorado Learning Attitudes about Science chemistry survey (Barbera, Perkins, Adams, & Wieman, 2008), which asks about students' "beliefs on learning chemistry, content of chemistry, structure of chemistry, and connection of chemistry to the outside world" (p. 942).
Hilton ( textbooks, and in the 2015 survey, 93% of student respondents did so. In a survey of over 300 students in British Columbia, 63% of respondents said the quality of the open textbook used in their course was "above average" or "excellent," with another 33% rating the quality as "average" (Jhangiani & Jhangiani, 2017).
The literature on open textbooks so far suggests that: (1) students are spending a significant amount of money on commercial textbooks; (2) it is likely they could achieve the same or better learning outcomes if  (4) students overwhelmingly rate the quality of open textbooks to be just as high or higher than that of traditional textbooks. Overall, then, the picture emerging from the research suggests that assigning open textbooks in post-secondary courses is likely to provide the same benefits as commercial textbooks at no cost (or a fraction of the cost, for print versions).
Our study adds an interesting and unique case to this literature for several reasons.

Context
This study reports on the use of an open textbook in Physics 100: an introductory, algebra-based physics course at a large, Canadian research university. homework and as a test bank for frequent testing. There is also a space for in-class materials, such as the worksheets used in lectures and tutorials. These materials were previously created by instructors and did not change after the revision. Furthermore, the edX web site allows the seamless integration of a discussion forum (piazza.com) that was already used before the revision. Table 1 provides a list of all course elements and materials. From a pedagogical perspective, the instructors were aiming at creating a more coherent appearance of the course resources that helps students make connections between the in-class activities and the homework. The instructors found that in previous iterations of the course, the pre-class reading assignments did not prepare the students well enough for class. This is of particular importance because the course is delivered in a "flipped" format in which students read the textbook (or watch a video) to learn basic terminology, definitions, and simple facts before coming to class. Class time is spent on sense- were posted on the online homework system, (b) reading the textbook, and (c) answering quiz questions that were also posted on the online homework system. A study by Heiner, Banet, and Wieman (2014) has shown that such a supported format is successful in getting most students to read the textbook regularly (typically 80%). However the study does not address the quality of the reading: for example, how much students understand. Anecdotally, we find that reading levels may have declined in recent years as more courses adopt a "flipped" format. This is supported by the rates in which simple reading questions are correctly answered in class (using clickers). The new reading assignments in Physics 100 have the quiz questions embedded into a customized version of the text that is now broken up and organized by weeks and only has the textbook sections that are relevant to the classroom activities for each week. Reading instructions are therefore no longer needed and students are more likely to read the text due to the proximity of quiz questions and relevant text.
As mentioned above, the end-of-chapter questions of the open textbook now replace the commercial online homework system that was previously used. There are two main reasons for this switch: cost and administration. Online homework systems are widely used in first-year science courses and major publishers offer packages that include the textbook and an online homework system customized for the textbook. The cost difference between such a package and the textbook alone or the online homework system alone is so small that adopting just the open textbook and keeping the commercial online homework system would not have saved the students much money. The instructors therefore decided to create an online homework system directly in the course web page by importing the end-of-chapter questions of the open text. The other reason for this decision is in avoiding administrative overhead for students and instructors. Using a commercial online homework system means adding another online tool that students have to learn how to use, register, and pay for. At our institution, first-year courses in chemistry, physics, and math all use different online homework systems in addition to the learning management system. This can lead to confusion, technical difficulties, and delays in getting the students up and running on these systems. By embedding the question database in the web page we avoid these issues.

Research Questions and Method
We used the COUP framework to design research questions for this study.

Cost
 How much do students in this course report usually spending on traditional textbooks, and how much did they save with the open textbook?
 What is the impact of the cost of traditional textbooks on the academic behaviours of students who responded to our survey? Use  Did the students in this course report using the open textbook more, less, or about the same as they report usually using textbooks for their courses?
Perceptions  How did students perceive the quality of this open textbook, as compared to traditional textbooks in other courses?
We addressed these questions using data from three sources: (1)    Of the respondents, 85% were first-year students (compare to an average of 80% in the course since 2013), 71% identified as female, and 27% identified as male (compared to an average of 65% female and 35% male from years 2013-2015). The majority (66%) said that their primary area of study was in the sciences (e.g., Chemistry, Biology, Physics), while 13% said it was in the health sciences, and the rest were spread between disciplines in the social and applied sciences (5% chose "I haven't decided yet"). From The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions Hendricks, Reinsberg, and Rieger 86 2013-2015, 65% of students in the course were in the Faculty of Science, and 10% in the Faculty of Arts.
Despite the fairly low response rate we can say that the demographics of the survey respondents correspond roughly to those of the course itself except that few students on the survey were from the Faculty of Arts.
We used descriptive statistics in our analyses of the multiple-choice and Likert-style questions on the survey. For analysis of the answers to open-ended questions, we used an open coding process: we generated codes from patterns found in the data itself. Two of the authors of this study separately coded the qualitative data using the same set of codes and either agreed with the other upon seeing both sets of codes or discussed disagreements to come to a final conclusion. We report below on statistical methods used to evaluate whether the changes to the course resulted in changes to student learning outcomes.

Cost
As some previous studies in the literature review above have done, we asked students how much they typically spend on textbooks for their courses (see Table 3). The last column on the right lists the average number of courses taken per year by students who reported each dollar range. We also asked about the impact of textbook costs on students' academic choices (see Figure 1).

Figure 1. How often respondents have taken certain actions as a result of textbook costs. N=150. See
Appendix A for percentages for each category.
Adding up the percentages for "rarely," "sometimes," "often," or "very often," 57% of students stated that they did not purchase a textbook for at least one course because of the cost, 25% said they had registered for a different section of a course, 19% said they had taken fewer courses than they would have liked, and 16% said they had dropped or withdrawn from at least one course.

Outcomes
To address the question of whether the changes in the course and adoption of the open textbook affected students' learning outcomes, we examined student final exam and course grades and student attitudes measured with the CLASS diagnostic survey that is well established in the physics education community. For student course grades, we considered both the distribution of grades into As, Bs, Cs, and Ds, and also the percentage of students who failed the course ("fail" is a grade of F).  The results from 2015-2016 are overall similar to previous years, but slightly more negative (see Table 4).
Performing statistical tests using one-way ANOVA shows that the changes are statistically significant in

Results of CLASS Survey
Year Negative shifts are typically observed in first-year physics classes and very few courses have been reported with positive shifts (Redish & Hammer, 2009). Nevertheless, we have been able to obtain consistent positive shifts in the problem solving categories in previous years. That these shifts are now slightly negative is currently actively discussed by the instructors, and possible reasons for this are given in the Discussion section below.

CLASS-PS_Shift
The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions Hendricks, Reinsberg, and Rieger

Use
In two separate questions, we asked students how often they tend to use textbooks in a typical course, as well as how often they used the open textbook in Physics 100 (see Table 5). 6 The patterns of usage for both questions are very similar, with over 55% of students saying they typically use their textbooks, and used the Physics 100 textbook, 2-3 times per week.
We also asked students to explain why they did or did not typically use textbooks very often for their courses, as well as for Physics 100. The most commonly reported reasons in these open-ended questions were similar for both. Common reasons for why students do typically use textbooks often in their courses, and for Physics 100, were: to do assigned pre-readings, to help understand the material better than they could just from going to class alone, and to prepare for quizzes and exams. The most common reasons for not using textbooks in a typical course were: one can get all the information one needs from class, and much of the material in textbooks is not relevant to exams. But a smaller percentage of students gave these reasons for not using the Physics 100 textbook: 23 respondents (18% of 130 who answered this question) said that in typical courses, they can get what they need better from class than from the book, whereas only five respondents (5% of 122) said this for the Physics 100 textbook. In addition, 10 students (8% of 130) said that in typical courses, much of the material in textbooks is not relevant to exams, but only four respondents (3% of 122) said that was the case for the Physics 100 textbook. These results could be due to the fact that because the Physics 100 textbook had an open license, the instructors were able to customize it by only including those parts of the text that were directly relevant to the course.

Perceptions
When asked how they would rate the quality of the textbook used in Physics 100 as compared to textbooks they have used in other courses, 93% of 143 respondents said it was about the same or better than textbooks in other courses (72% said it was about the same, and 21% said it was better). Students were invited to explain their answers in open-ended responses, though only 33 of them did. Of those, seven (21% of those who explained their answers) said that they appreciated the fact that the text was The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions Hendricks, Reinsberg, and Rieger 91 customized to the course. Three respondents (9%) said they found the book easy to understand and that it explained topics well, but another three said that they thought the text did not explain the material well enough. Those three students were in the distance education course, and said that the text was not enough to explain what they needed to know without there being in-person class meetings. Finally, five students who said the quality was worse than that of their other textbooks said this was because of the format of the online book, such as that it was hard to navigate or that scrolling through PDFs was bothersome.
We also asked students how important various aspects of the Physics 100 textbook were to them, and three answers stood out as most important: the book was customized to the course, it was free, and it could be accessed anywhere with an internet connection (see Figure 3). The vast majority of the student respondents (94%) appreciated that the textbook was customized to their particular course. Unsurprisingly, then, when asked on a different survey question whether or not they agreed that the textbook was relevant to the course, many respondents said it was: 78% of 142 respondents stated that they either somewhat or strongly agreed that the readings were linked directly to lectures, labs or other class activities, and 66% at least somewhat agreed that the readings were useful in preparing for exams.
As another way to gauge their perceptions of the open textbook for Physics 100, we asked students to rate their agreement with the following statement: "I would have preferred to purchase a traditional textbook for this course rather than using the free online textbook." The majority of students disagreed, saying they would rather use the free, open textbook: 64% of 143 respondents disagreed somewhat or strongly with that statement, 18% said they neither agreed nor disagreed, and 18% said they somewhat or strongly agreed that they would rather have purchased a traditional textbook for the course.
Among the 91 students who explained their answers, the most commonly-reported reasons given for not wanting to purchase a traditional textbook were that the online textbook was free (65%), that traditional textbooks are heavy to carry around (12%), and that the textbook was customized to fit this particular course (8%). Among the 25 out of 143 respondents who said they would have preferred to purchase a 92 traditional textbook instead of using the free, online one for Physics 100, only 15 explained their answers.
Among those, 14 gave reasons related to preferring reading print versus online books (e.g., because they do not like reading on a screen, or they like to highlight text in print books), not because of the quality of the textbook. Only one gave a negative comment about the quality of the textbook.

Discussion and Limitations
The results of this study strongly suggest that the students in Physics 100 were able to save money by using an open textbook that allowed for customization to the course and embedding in a consolidated website, while getting similar quality as they would from a commercial textbook. Compared to the one other survey so far of Canadian students that asked a similar question, or results on whether students would have preferred to purchase a traditional textbook rather than using the open textbook, are slightly more positive. 64% of respondents in our survey said they would not have preferred to purchase a traditional textbook instead of using the OpenStax book; 56% of respondents said this in a survey of 320 students in British Columbia (Jhangiani & Jhangiani, 2017).
Students in our study reported spending less on textbooks per academic year than those in previously published studies, especially considering that the Canadian dollar has been worth less than the U.S. dollar for the past five years (XE Currency Charts (CAD/USD), 2016). Florida Virtual Campus (2016) reports that between 45-50% of students say they spend over US$300 per term ($600 per year), whereas 48% of our student respondents reported spending between C$200 and C$400 per year, with only 22% saying they typically spend over $600 a year on books. In a survey of post-secondary students in British Columbia, respondents reported spending a mean of about $700 and a median of $500 in the last 12 months on textbooks (Jhangiani & Jhangiani, 2017). Our numbers are somewhat closer to a survey of 1500 students at the University of British Columbia (So & Doering, 2016): 49% reported spending at least C$500 per year on textbooks whereas 50% of students on our survey reported spending at least $400 per year (our closest comparable number). One possible reason the reported numbers on our survey are lower than some other results is because 85% of our survey respondents were in their first year, so they did not have a great deal of experience to draw from in giving their answers (they did, however, take the survey at the end of their first year, after purchasing textbooks for two terms). The data from earlier literature does not split out reported spending by first-year students versus data from other students.
Our data about how textbook costs affect students' academic choices are lower than numbers from studies of U.S. students. In our survey, 57% of respondents reported not buying a required textbook for at least one course because of cost, 25% reported registering for a different section of at least one course, and 19% said they had taken fewer courses. In comparison, data from earlier studies (Florida Virtual Campus, 2016;Senack & The Student PIRGs, 2014) shows over 60% reporting not purchasing a textbook for at least one course because of cost, and over 40% saying that the cost of textbooks affected their choice of courses as well as the number of courses they could take. Our data is closer to that from students in British Columbia, however: Jhangiani & Jhangiani (2017) report that 54% of students said they had not purchased a textbook because of cost, 26% said they had chosen not to register for a specific course because of the textbook cost, and 27% had taken fewer courses. Still, our numbers are lower than those from a larger survey of students at our institution (So and Doering, 2016), which reported that 75% of all 1500 respondents, and 70% of those in their first or second year, had gone without a textbook for at least one course because of cost. In comparison to these numbers at our institution, it seems the first-year students in our survey are skewed towards the lower end of being affected by textbook costs in terms of purchasing textbooks or choosing courses.
Looking at student learning outcomes, we find overall that final exam marks and course marks after the changes to the course (including the adoption of the open textbook) are comparable to data from the previous three years in Physics 100. We see no significant change in the overall post-scores and overall shift-scores of the CLASS survey, while a small negative shift in the "problem-solving" category in 2015-2016 occurred. Possible reasons include the homework incentive and feedback as well as a change in the tutorial structure. Previously, students received marks for the homework that was worth 10% of the overall grade, while students received no marks for doing the homework in 2015-2016. In the future, we will again reward doing the homework with a few percent of the final grade. It is also possible that the current feedback on the homework is not sufficient. The previous commercial homework system had tutorial-type questions that provided hints and students could get feedback on important steps using these hints. Our homework system only provides correct/incorrect feedback, but we will enhance our homework system in the near future. For the Fall 2016 term, animated worked examples were inserted in the homework sets, which have been shown to enhance learning. In addition, the instructors have developed tutorial-type questions with step feedback, similar to commercial homework systems used previously. Finally, there may also be an effect due to the textbook. Although the open text features student-centered learning support such as problem-solving strategies and worked examples, it not as detailed as the previously used commercial text. However, we do not think that the textbook has a significant influence on the students' problem solving ability and confidence as compared to the course components that specifically focus on problem solving, in particular the homework and the tutorial sessions.
Regarding use of the open textbook, similar to results from other published studies we did not see a significant difference between how often students reported using textbooks in a "typical course" versus how often they used the textbook in Physics 100. Combining this with the data about their perceptions of the quality of the textbook suggests that they found the textbook useful for their course work while also having relatively few complaints or concerns about it.
This study is limited by its relatively small sample size compared to the number of students enrolled in Physics 100, as well as by the fact that the data is from one course at one university only. However, our results are largely consistent with previous research on open textbooks, with the exception of how much our students reported typically spending on textbooks per academic year and how costs affect their academic behaviours such as choosing which section of a course to take based on textbook cost. Our study is also focused on a course in which several changes were made, in addition to adopting an open textbook, including that the book was customized to fit this course and all course materials were consolidated on a

Conclusion
The results of this study add to and mostly support the existing literature on open textbooks in terms of cost, learning outcomes, student use, and student perceptions. We have added to the so far limited data on students in the Canadian context, and our study focuses on a case of not just adopting an open textbook, but also significantly adapting it to fit a particular course.
However, a project like ours is not without cost: a significant amount of effort went into the project, in particular into the creation of the question database and the test bank. Four graduate physics students and two instructors worked over the summer months to get the resources ready; approximately C$20,000 from a teaching and learning grant from our university have been spent to date to pay the graduate  Note: The Fall version of Physics 100 is a face-to-face course, while the Spring version is a distance course.