International Review of Research in Open and Distributed Learning

Volume 18, Number 4

June - 2017

The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions

Christina Hendricks, Stefan A. Reinsberg, and Georg Rieger
University of British Columbia

Abstract

Assigning open textbooks in college and university courses can help students save money on increasingly expensive commercial textbooks, and recent research shows that this savings can often be achieved with little to no sacrifice in textbook quality or student learning outcomes. We add to this body of research by examining the use of an open textbook in an introductory physics course at a large research university in Canada that enrols approximately 800-900 students per year. In this course, the instructors revised an open textbook and combined it with other learning resources onto a single website, whereas more than one source of learning materials was used previously. We used the COUP framework to structure our analysis, focusing on cost, outcomes, use, and perceptions in relation to the open textbook assigned in the course. Through the use of a survey of students and data about student learning outcomes in the form of final exam and course grades, and shifts on the pre-/post- Colorado Learning Attitudes about Science Survey, we show that student savings by moving to an open textbook were accompanied by little change in learning outcomes. We also show that the vast majority of survey respondents perceived the open textbook to be of the same or better quality than commercial textbooks used in their other courses. Further, many of them appreciated the fact that the textbook was customized to this particular course—which is made possible by the use of a textbook with an open license.

Keywords: open educational resources, open textbooks, online learning

Introduction

The cost of textbooks at North American colleges and universities is a source of concern for many students, faculty, and staff. In August 2016 the U.S. Bureau of Labor Statistics reported that the price of college textbooks has increased 88% since 2006, and is increasing at a faster rate than college and university tuition and fees (U.S. Bureau of Labor Statistics, 2016). Still, students are spending less on textbooks over time, not more: according to Kestenbaum (2014), student spending on textbooks went down from 2002-2013, and the average number of textbooks purchased per year by post-secondary students went down from about 13 to just over 8. Other research shows a significant percentage of students simply going without textbooks for some of their courses due to rising costs, as discussed below. Thus, as textbook prices go up, students seem to be responding by trying to get by in at least some of their courses without the required textbooks, which could negatively impact their learning. In a survey of over 22,000 post-secondary students in Florida, 38% said they believed they did worse in a course because they could not afford the textbook, and 20% said they had failed a course for that reason (Florida Virtual Campus, 2016). The high cost of textbooks is not just a financial concern, but a pedagogical one as well.

One way to address the problem of students' failure to access the required materials for their courses due to high costs is through the use of open textbooks and other open educational resources (OER). According to the William and Flora Hewlett Foundation (n.d.), OER are "teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and re-purposing by others" (para. 2). OER can be texts (including open textbooks), videos, assessments, lecture notes, slides, or other learning materials. There is a growing body of research literature on open textbooks that strongly suggests students can save a significant amount of money if instructors assign open textbooks for their courses, while getting the same or better quality learning materials. Much of this research can be analyzed under what Bliss, Robinson, Hilton, & Wiley (2013b) call the COUP framework: measuring factors related to cost, outcomes, use, and perceptions of open textbooks.

Research focusing on "cost" includes studies of how much students typically spend on textbooks, how much they save in particular courses by using open textbooks, and how the cost of textbooks affects students' academic choices. The literature so far suggests that many students are paying at least $600 (in U.S. or Canadian currency) per year on textbooks. In a survey of over 22,000 students from 40 universities and colleges in Florida, 53% of students reported that they spent over $300 on books in the Spring 2016 semester, and 33% spent over $400 (Florida Virtual Campus, 2016). In the Canadian context, Jhangiani & Jhangiani (2017) surveyed 320 undergraduates in various postsecondary institutions in British Columbia in 2015 and asked how much they had spent on textbooks in the last 12 months; the mean was about C$700 (median was $500). In a 2016 survey of about 1500 students from the University of British Columbia, nearly half of respondents (49%) said they spend at least C$500 on textbooks each year, and the mean reported cost per year for textbooks was $620 (So & Doering, 2016).

The high cost of textbooks is, not surprisingly, affecting students' academic behaviours, such as whether or not they purchase the required books for their courses and which courses they choose to take. Florida Virtual Campus (2016) asked over 22,000 post-secondary students whether the cost of textbooks had led them to do the following actions (among others) during their post-secondary studies (numbers below refer to the percentage of students who said they had done each action seldom, frequently, or often):

• Didn't buy the required book(s) for a course because of cost: 67%.
  
  
• Took fewer courses because of the cost of the book(s): 48%.
  
  
• Didn't register for a course because of the cost of the book(s): 46%.
  
  
• Dropped a course because of the cost of the book(s): 26%.

A report by Senack and the Student PIRGs (2014), using data from a survey of over 2000 students in 150 colleges and universities in the US, provided similar numbers: 65% of student respondents reported that they had decided not to buy a textbook for one or more of their courses because of cost, and 48% said that the cost of textbooks had somewhat or significantly affected which or how many courses they were able to take each semester. Jhangiani & Jhangiani (2017) report that of over 300 students in British Columbia, 54% said they had not bought a textbook for a course at least once due to cost, 27% said they had taken fewer courses, 26% said they did not register for a particular course because of the cost of the book(s), and 17% said they had dropped or withdrawn from a course. So & Doering (2016) reports that 75% of 1500 students at the University of British Columbia said they had gone without purchasing a textbook for at least one course. The research on cost reveals that the price of commercial textbooks is affecting students not just financially, but also in terms of their learning.

Research on "outcomes" considers how the use of open textbooks and other OER affects students' learning outcomes, including grades on exams or in courses, and retention rates. The literature suggests that in the majority of cases, students using open textbooks or other OER do just as well as or better than students who are required to purchase commercial textbooks or other course materials. Fischer, Hilton, Robinson, & Wiley (2015) studied a large sample of students from six colleges and four community colleges in the US. Students in the treatment group (N=1087) used an open textbook or other OER in their courses, and students in the control group (N=9264) were asked to purchase traditional textbooks for different sections of the same courses. The authors found that the students using OER did just as well or better in most of the courses studied in terms of course grades and completion rates. Allen et al. (2015) studied students in a chemistry course at the University of California, Davis who were assigned a traditional textbook (448 students) and students in another section of the same course taught by the same instructor (478 students) that used Chemwiki (https://chem.libretexts.org/) as a primary course resource (an openly licensed wiki site that can be used as a textbook). They found no significant differences in outcomes between these students, as measured through pre-and post-tests on content covered in the course, the same midterm and final exams, and the administration of the Colorado Learning Attitudes about Science chemistry survey (Barbera, Perkins, Adams, & Wieman, 2008), which asks about students' "beliefs on learning chemistry, content of chemistry, structure of chemistry, and connection of chemistry to the outside world" (p. 942).

Hilton (2016) provides a thorough analysis of these and other studies focusing on outcomes in the OER literature. In his synthesis of the data he concludes that "an emerging finding is that utilizing OER does not appear to decrease student learning" (p. 586). However, he emphasizes that there are significant limitations to such a synthesis of the research to date, because the OER may have been used differently in different courses and the design of some studies was too weak to claim any causal connection between open textbooks and learning outcomes. Still, we can at least say that in the majority of courses studied, using an open textbook is not correlated with a reduction in grade averages or retention rates.

Research on "use" includes data about how students use OER and how often, as well as in what format students access the materials. In addition, it can include empirical study of how the open licenses on learning materials allow for them to be used in new ways, such as instructors or students deleting material from, adding material to, or otherwise revising OER. Much of the research on use of open textbooks to date has looked at how often students use the books. Bliss, Robinson, Hilton, & Wiley (2013a) asked 490 students from eight U.S. community colleges how often they tended to use textbooks in "a typical course," and how often they used the open textbooks in the courses that assigned open textbooks (p. 23). Nearly 75% of the students reported using textbooks 2-3 times per week or daily in "a typical course;" for the courses that used an open textbook, the percentage saying they used the book 2-3 times per week or daily was about the same. The California Open Educational Resources Council (2016) reports similar results from a survey of 351 students: most students said they used the open textbook they were assigned at about the same rate as compared to traditional textbooks in other courses.

There are not many studies yet that discuss how students and faculty are taking advantage of the open licenses on open textbooks to alter them. In a survey of 127 post-secondary educators, Pitt (2015) reports that 25% said using an open textbook had "enabled innovation or changed their pedagogical approach" (p. 146), but only a few explained in open-ended questions that they had revised an open textbook. From interviews with faculty who have adopted, adapted, or created an open textbook or other OER, Delimont, Turtle, Bennett, Adhikari, & Lindshield (2016) report that 12 out of 13 faculty said they preferred to teach with OER rather than commercial textbooks, and out of those, 11 gave customizability of OER as a reason for their preference. Neither study, however, focuses on how faculty are revising OER, and overall there is little research in the extant literature on the degree to which faculty or students are revising OER, how they are doing so, and what impact this may have on student learning outcomes.

Research on "perceptions" considers how students, faculty or teachers, or others perceive open textbooks or other OER, such as their views about the quality of such resources. The literature shows that the vast majority of students report open textbooks to be of the same or better quality than the commercial textbooks they have used. Bliss et al. (2013a) reported that 94% of 490 students said they found open textbooks to be of equal or higher quality than traditional textbooks. Illowsky, Hilton, Whiting, & Ackerman (2016) reported on two surveys of students using an open textbook at a U.S. community college, one in 2013 of 231 students and one in 2015 of 94 students (the students surveyed in 2015 used a significantly-revised version of the textbook the students in 2013 had used). In the 2013 survey, 87% of students rated the quality of the open textbook to be the same or better when compared to traditional textbooks, and in the 2015 survey, 93% of student respondents did so. In a survey of over 300 students in British Columbia, 63% of respondents said the quality of the open textbook used in their course was "above average" or "excellent," with another 33% rating the quality as "average" (Jhangiani & Jhangiani, 2017).

The literature on open textbooks so far suggests that: (1) students are spending a significant amount of money on commercial textbooks; (2) it is likely they could achieve the same or better learning outcomes if their courses assigned open textbooks (keeping in mind the caveats above noted by Hilton, 2016); (3) students appear to be using open textbooks at the same rate as they use traditional textbooks; and (4) students overwhelmingly rate the quality of open textbooks to be just as high or higher than that of traditional textbooks. Overall, then, the picture emerging from the research suggests that assigning open textbooks in post-secondary courses is likely to provide the same benefits as commercial textbooks at no cost (or a fraction of the cost, for print versions).

Our study adds an interesting and unique case to this literature for several reasons. First, the research literature on open textbooks to date has focused largely on U.S. post-secondary institutions, and our study provides survey data from students at a large, research-intensive university in Canada. McKerlich, Ives, & McGreal (2013) and Jhangiani, Pitt, Hendricks, Key, & Lalonde (2016) report on survey data from educators, staff, or administrators about use and creation of OER, but do not focus specifically on open textbooks and do not provide data from students. Our survey shares some similar questions to that used by Jhangiani & Jhangiani (2017) to gather data from students in British Columbia who have used open textbooks in their courses. In terms of providing empirical data from students in Canada on open textbooks, to our knowledge their article and ours are the first ones in the published literature so far.

In addition, very little research on open textbooks has considered the specific ways in which faculty are revising open textbooks and whether this has any impact on learning outcomes. In the course we report on here, an open textbook was adopted primarily because the open license allows for revision. Two of the authors of this article participated in a significant revision of a first-year physics course (Physics 100), involving moving all of the learning resources to a single website, and changing from a traditional publisher's textbook accompanied by a paid online homework system to an open textbook and homework questions embedded in the course website. Unlike some studies that focus on OER or open textbooks generally, such as McKerlich et al. (2013), Jhangiani et al. (2016), Bliss et al. (2013a), Fisher et al. (2015), and Delimont et al. (2016), we provide a very specific description of how the open textbook used in the course we are studying has been adapted to fit into that course.

Using data from the 2015-2016 academic year, we studied students' experiences with and perceptions of the new website and textbook for Physics 100. We also examined student learning outcomes in the form of final exam grades, course grades, and the Colorado Learning Attitudes about Science Survey (CLASS; Adams, Perkins, Podolefsky, Dubson, Finkelstein, & Wieman, 2006), in the years before and after the revision to the course.

Context

This study reports on the use of an open textbook in Physics 100: an introductory, algebra-based physics course at a large, Canadian research university. As of November 1, 2016, over 43,000 undergraduate students and 9600 graduate students (full time and part time) were registered at the University of British Columbia in Vancouver (UBC Planning and Institutional Research, 2017). Physics 100 is taken by 800-900 non-physics majors annually that did not take grade 12 physics in high school, and it is a required course for 99% of the students taking it. The changes to the course described above, including the adoption of an open textbook, were first implemented during the Fall 2015 and Spring 2016 terms. Physics 100 in the Fall term is a face-to-face course; in the Spring term it is a fully online, distance course.

Data from 2013-2016 show that the vast majority of students in Physics 100 are in their first year (average 80%), and an average of 65% are in the Faculty of Science, 10% in the Faculty of Arts, and 25% in other faculties. In addition, on average 66% of students in the course identify as female and 36% as male. In comparison, at the University of British Columbia-Vancouver, 55% of undergraduates identify as female, 31% are in the Faculty of Arts and 19% are in the Faculty of Science (UBC Planning and Institutional Research, 2017).

The course designers chose to use the open textbook College Physics (https://openstax.org/details/college-physics), published by OpenStax. An electronic version of the textbook was customized, re-formatted into weekly reading assignments, and integrated into the course website on edX edge (edge.edx.org).¹ This website also contains all other course resources and replaces the previous commercial online homework system and the Learning Management System. The end-of-chapter textbook questions as well as previous exam questions were imported into edX and coded into a question database to be used for online homework and as a test bank for frequent testing. There is also a space for in-class materials, such as the worksheets used in lectures and tutorials. These materials were previously created by instructors and did not change after the revision. Furthermore, the edX web site allows the seamless integration of a discussion forum (piazza.com) that was already used before the revision. Table 1 provides a list of all course elements and materials.

Table 1

Overview of Physics 100 Course Elements

From a pedagogical perspective, the instructors were aiming at creating a more coherent appearance of the course resources that helps students make connections between the in-class activities and the homework. The instructors found that in previous iterations of the course, the pre-class reading assignments did not prepare the students well enough for class. This is of particular importance because the course is delivered in a "flipped" format in which students read the textbook (or watch a video) to learn basic terminology, definitions, and simple facts before coming to class. Class time is spent on sense-making and applying this knowledge, and it is therefore very important that the students come prepared to class by having done the reading. Before the revision to the course, the reading assignments consisted of (a) reading instructions (what sections to read, which examples and concepts to focus on, etc.) that were posted on the online homework system, (b) reading the textbook, and (c) answering quiz questions that were also posted on the online homework system. A study by Heiner, Banet, and Wieman (2014) has shown that such a supported format is successful in getting most students to read the textbook regularly (typically 80%). However the study does not address the quality of the reading: for example, how much students understand. Anecdotally, we find that reading levels may have declined in recent years as more courses adopt a "flipped" format. This is supported by the rates in which simple reading questions are correctly answered in class (using clickers). The new reading assignments in Physics 100 have the quiz questions embedded into a customized version of the text that is now broken up and organized by weeks and only has the textbook sections that are relevant to the classroom activities for each week. Reading instructions are therefore no longer needed and students are more likely to read the text due to the proximity of quiz questions and relevant text.

As mentioned above, the end-of-chapter questions of the open textbook now replace the commercial online homework system that was previously used. There are two main reasons for this switch: cost and administration. Online homework systems are widely used in first-year science courses and major publishers offer packages that include the textbook and an online homework system customized for the textbook. The cost difference between such a package and the textbook alone or the online homework system alone is so small that adopting just the open textbook and keeping the commercial online homework system would not have saved the students much money. The instructors therefore decided to create an online homework system directly in the course web page by importing the end-of-chapter questions of the open text. The other reason for this decision is in avoiding administrative overhead for students and instructors. Using a commercial online homework system means adding another online tool that students have to learn how to use, register, and pay for. At our institution, first-year courses in chemistry, physics, and math all use different online homework systems in addition to the learning management system. This can lead to confusion, technical difficulties, and delays in getting the students up and running on these systems. By embedding the question database in the web page we avoid these issues.

Research Questions and Method

We used the COUP framework to design research questions for this study.

Cost

• How much do students in this course report usually spending on traditional textbooks, and how much did they save with the open textbook?
  
  
• What is the impact of the cost of traditional textbooks on the academic behaviours of students who responded to our survey?

Outcomes

• Are the changes implemented in this course, including the use of an open textbook, correlated with any changes in student grades on the final exam, grades in the course, or results on the CLASS survey?

Use

• Did the students in this course report using the open textbook more, less, or about the same as they report usually using textbooks for their courses?

Perceptions

• How did students perceive the quality of this open textbook, as compared to traditional textbooks in other courses?

We addressed these questions using data from three sources: (1) a survey of students who had taken Physics 100 in Fall 2015 or Spring 2016, (2) final exams and course grades for students in both terms, and (3) results from the Colorado Learning Attitudes about Science Survey (CLASS) for physics.

We administered the survey to students who had taken Physics 100 in Fall 2015 (face-to-face course) or Spring 2016 (online course); the questions and results of the survey is publicly available here: https://osf.io/928wb/. It consisted of a mix of multiple-choice, many-choice, Likert-type, and open-ended questions. A link to the online survey was sent via email in April 2016 to students who had taken the course in the Fall of 2015, and in May 2016 to those who had taken it in the Spring of 2016 (after final course grades were submitted). Table 2 shows survey response rates.³

Table 2

Response Rate for the Survey, out of the Number of Students who Completed Physics 100

Of the respondents, 85% were first-year students (compare to an average of 80% in the course since 2013), 71% identified as female, and 27% identified as male (compared to an average of 65% female and 35% male from years 2013—2015). The majority (66%) said that their primary area of study was in the sciences (e.g., Chemistry, Biology, Physics), while 13% said it was in the health sciences, and the rest were spread between disciplines in the social and applied sciences (5% chose "I haven't decided yet"). From 2013-2015, 65% of students in the course were in the Faculty of Science, and 10% in the Faculty of Arts. Despite the fairly low response rate we can say that the demographics of the survey respondents correspond roughly to those of the course itself except that few students on the survey were from the Faculty of Arts.

We used descriptive statistics in our analyses of the multiple-choice and Likert-style questions on the survey. For analysis of the answers to open-ended questions, we used an open coding process: we generated codes from patterns found in the data itself. Two of the authors of this study separately coded the qualitative data using the same set of codes and either agreed with the other upon seeing both sets of codes or discussed disagreements to come to a final conclusion. We report below on statistical methods used to evaluate whether the changes to the course resulted in changes to student learning outcomes.

Results

Cost

As some previous studies in the literature review above have done, we asked students how much they typically spend on textbooks for their courses (see Table 3). The last column on the right lists the average number of courses taken per year by students who reported each dollar range.

Table 3

Reported Amount Spent on Textbooks and Average Courses Taken per Academic Year (Fall, Spring, and Summer Terms Combined )

Note: N=151; amounts are in Canadian dollars.

Most respondents reported typically spending between $200 and $400 per academic year on textbooks (48%), though if we combine those who said they spend between $400 and $800 per year that's another 46%. These numbers are somewhat surprisingly low, since, for example, if students who spend $200-$400 per year are taking an average of nine courses a year, that means they could be spending between $22 and $44 per course on textbooks. We consider possible reasons for our low numbers in the Discussion section of the article, below.

For Physics 100, the learning materials used in previous years cost students about C$100. The instructors were using a customized version of a textbook that included only the sections covered in the course, which already saved students money over the standard version. The $100 cost also included the price of students' access to the online homework system. For the 856⁴ students who were enrolled in Physics 100 during the 2015-2016 academic year, that's over $85,000 saved compared to what they would have spent if they had to purchase the $100 course materials used in previous years.

We also asked about the impact of textbook costs on students' academic choices (see Figure 1).

Figure 1. How often respondents have taken certain actions as a result of textbook costs. N=150. See Appendix A for percentages for each category.

Adding up the percentages for "rarely," "sometimes," "often," or "very often," 57% of students stated that they did not purchase a textbook for at least one course because of the cost, 25% said they had registered for a different section of a course, 19% said they had taken fewer courses than they would have liked, and 16% said they had dropped or withdrawn from at least one course.

Outcomes

To address the question of whether the changes in the course and adoption of the open textbook affected students' learning outcomes, we examined student final exam and course grades and student attitudes measured with the CLASS diagnostic survey that is well established in the physics education community.

Measuring student performance with final grades is usually not a reliable method for longitudinal comparisons due to changes in the weighting of different course components. Similarly, final exam results are influenced by variability in question difficulty and it is fairly difficult to establish an overall level of difficulty on an absolute scale. For instance, over the period of 2008-2015, we observed final exam averages between 54 to 64%, or when excluding two outliers, 60-63%. In this case however, the exams are very similar and are designed by the same instructors. The exam score average for Fall 2015 was 63.1% and was slightly higher than the one in Fall 2013 (61.4%, p=0.04) and the one from Fall 2014 (61.2%, p=0.02). The distributions were compared with a Wilcoxon rank sum test with continuity correction—p values quoted above have not been corrected for multiple comparisons. Given the marginal absolute change (2% grade points) in final exams scores, we can say with some confidence that student performance on the final exam is at least as good as in previous years.

For student course grades, we considered both the distribution of grades into As, Bs, Cs, and Ds, and also the percentage of students who failed the course ("fail" is a grade of F). Figure 2 shows these numbers for Fall 2012 to Spring 2016. The changes to the course, including the adoption of the open textbook, were made in the Fall of 2015.

Figure 2. Percentages of students earning Grades A-F in Physics 100, from Fall 2012 to Spring 2016. Fall terms are face-to-face versions of the course, and Spring terms are online, distance versions. See Appendix B for percentages in each category.

Grade distributions in Fall 2015 were very similar to previous Fall semesters, before the changes to the course were made. In Spring 2016 grades of "A" (51%) went up quite a bit compared to Spring 2015 (36%). However, non-parametric testing of differences in the final grade distributions showed no significant changes between the terms in which students used an open textbook and the previous iterations of the course. As discussed above, final grades are influenced by many factors, and the fact that there are no significant changes in grade distributions allows us only to conclude that the introduction of the open textbook caused no large changes in the final grade.

Finally, we used the CLASS attitude survey to compare the student attitudes and beliefs about physics and about learning physics to previous years. Students choose how much they agree or disagree with 42 statements on a five-point Likert scale. The results are then evaluated and compared to the answers of discipline experts. The CLASS is administered as a pre-/post- survey during the first and last week of the course and shifts in responses are linked to the course instruction. Average positive shifts in student responses are interpreted as students' views becoming more expert-like. Negative shifts indicate a more novice-like attitude after taking the course. The survey questions fall into eight empirically determined categories (Adams et al 2006): real world connections, personal interest, sense making/effort, conceptual connections, applied conceptual connections, problem solving general, problem solving confidence, and problem solving sophistication.

The results from 2015-2016 are overall similar to previous years, but slightly more negative (see Table 4). Performing statistical tests using one-way ANOVA shows that the changes are statistically significant in the problem-solving category (F = 4.16 > F_crit. = 2.61; p < 0.05) while there are no significant changes in the shifts for all questions combined (F = 1.36 < F_crit. = 2.61) and in the overall post course results (F = 0.82 < F_crit. = 2.61).

Table 4

Results of CLASS Survey

Note. Pre-post shifts for the general problem-solving category are listed in column 2 and shifts for all survey questions are presented in column 3. The fourth column shows the post course averages for all survey questions categories.⁵ The "year" in column 1 includes Fall and Spring terms (e.g., 2015 includes Fall 2015 and Spring 2016 terms).

Negative shifts are typically observed in first-year physics classes and very few courses have been reported with positive shifts (Redish & Hammer, 2009). Nevertheless, we have been able to obtain consistent positive shifts in the problem solving categories in previous years. That these shifts are now slightly negative is currently actively discussed by the instructors, and possible reasons for this are given in the Discussion section below.

Use

In two separate questions, we asked students how often they tend to use textbooks in a typical course, as well as how often they used the open textbook in Physics 100 (see Table 5).⁶

Table 5

Reported Usage Rates of Textbooks

The patterns of usage for both questions are very similar, with over 55% of students saying they typically use their textbooks, and used the Physics 100 textbook, 2-3 times per week.

We also asked students to explain why they did or did not typically use textbooks very often for their courses, as well as for Physics 100. The most commonly reported reasons in these open-ended questions were similar for both. Common reasons for why students do typically use textbooks often in their courses, and for Physics 100, were: to do assigned pre-readings, to help understand the material better than they could just from going to class alone, and to prepare for quizzes and exams. The most common reasons for not using textbooks in a typical course were: one can get all the information one needs from class, and much of the material in textbooks is not relevant to exams. But a smaller percentage of students gave these reasons for not using the Physics 100 textbook: 23 respondents (18% of 130 who answered this question) said that in typical courses, they can get what they need better from class than from the book, whereas only five respondents (5% of 122) said this for the Physics 100 textbook. In addition, 10 students (8% of 130) said that in typical courses, much of the material in textbooks is not relevant to exams, but only four respondents (3% of 122) said that was the case for the Physics 100 textbook. These results could be due to the fact that because the Physics 100 textbook had an open license, the instructors were able to customize it by only including those parts of the text that were directly relevant to the course.

Perceptions

When asked how they would rate the quality of the textbook used in Physics 100 as compared to textbooks they have used in other courses, 93% of 143 respondents said it was about the same or better than textbooks in other courses (72% said it was about the same, and 21% said it was better). Students were invited to explain their answers in open-ended responses, though only 33 of them did. Of those, seven (21% of those who explained their answers) said that they appreciated the fact that the text was customized to the course. Three respondents (9%) said they found the book easy to understand and that it explained topics well, but another three said that they thought the text did not explain the material well enough. Those three students were in the distance education course, and said that the text was not enough to explain what they needed to know without there being in-person class meetings. Finally, five students who said the quality was worse than that of their other textbooks said this was because of the format of the online book, such as that it was hard to navigate or that scrolling through PDFs was bothersome.

We also asked students how important various aspects of the Physics 100 textbook were to them, and three answers stood out as most important: the book was customized to the course, it was free, and it could be accessed anywhere with an internet connection (see Figure 3).

Figure 3. How respondents rated the importance of various features of the Physics 100 open textbook.⁷ N=143. See Appendix C for percentages for each category.

The vast majority of the student respondents (94%) appreciated that the textbook was customized to their particular course. Unsurprisingly, then, when asked on a different survey question whether or not they agreed that the textbook was relevant to the course, many respondents said it was: 78% of 142 respondents stated that they either somewhat or strongly agreed that the readings were linked directly to lectures, labs or other class activities, and 66% at least somewhat agreed that the readings were useful in preparing for exams.

As another way to gauge their perceptions of the open textbook for Physics 100, we asked students to rate their agreement with the following statement: "I would have preferred to purchase a traditional textbook for this course rather than using the free online textbook." The majority of students disagreed, saying they would rather use the free, open textbook: 64% of 143 respondents disagreed somewhat or strongly with that statement, 18% said they neither agreed nor disagreed, and 18% said they somewhat or strongly agreed that they would rather have purchased a traditional textbook for the course.

Among the 91 students who explained their answers, the most commonly-reported reasons given for not wanting to purchase a traditional textbook were that the online textbook was free (65%), that traditional textbooks are heavy to carry around (12%), and that the textbook was customized to fit this particular course (8%). Among the 25 out of 143 respondents who said they would have preferred to purchase a traditional textbook instead of using the free, online one for Physics 100, only 15 explained their answers. Among those, 14 gave reasons related to preferring reading print versus online books (e.g., because they do not like reading on a screen, or they like to highlight text in print books), not because of the quality of the textbook. Only one gave a negative comment about the quality of the textbook.

Discussion and Limitations

The results of this study strongly suggest that the students in Physics 100 were able to save money by using an open textbook that allowed for customization to the course and embedding in a consolidated website, while getting similar quality as they would from a commercial textbook. The vast majority of the survey respondents judged the open textbook to be the same or better quality than textbooks used in their other courses (93%). This result is similar to those in other published research on open textbooks. Compared to the one other survey so far of Canadian students that asked a similar question, or results on whether students would have preferred to purchase a traditional textbook rather than using the open textbook, are slightly more positive. 64% of respondents in our survey said they would not have preferred to purchase a traditional textbook instead of using the OpenStax book; 56% of respondents said this in a survey of 320 students in British Columbia (Jhangiani & Jhangiani, 2017).

Students in our study reported spending less on textbooks per academic year than those in previously published studies, especially considering that the Canadian dollar has been worth less than the U.S. dollar for the past five years (XE Currency Charts (CAD/USD), 2016). Florida Virtual Campus (2016) reports that between 45-50% of students say they spend over US$300 per term ($600 per year), whereas 48% of our student respondents reported spending between C$200 and C$400 per year, with only 22% saying they typically spend over $600 a year on books. In a survey of post-secondary students in British Columbia, respondents reported spending a mean of about $700 and a median of $500 in the last 12 months on textbooks (Jhangiani & Jhangiani, 2017). Our numbers are somewhat closer to a survey of 1500 students at the University of British Columbia (So & Doering, 2016): 49% reported spending at least C$500 per year on textbooks whereas 50% of students on our survey reported spending at least $400 per year (our closest comparable number). One possible reason the reported numbers on our survey are lower than some other results is because 85% of our survey respondents were in their first year, so they did not have a great deal of experience to draw from in giving their answers (they did, however, take the survey at the end of their first year, after purchasing textbooks for two terms). The data from earlier literature does not split out reported spending by first-year students versus data from other students.

Our data about how textbook costs affect students' academic choices are lower than numbers from studies of U.S. students. In our survey, 57% of respondents reported not buying a required textbook for at least one course because of cost, 25% reported registering for a different section of at least one course, and 19% said they had taken fewer courses. In comparison, data from earlier studies (Florida Virtual Campus, 2016; Senack & The Student PIRGs, 2014) shows over 60% reporting not purchasing a textbook for at least one course because of cost, and over 40% saying that the cost of textbooks affected their choice of courses as well as the number of courses they could take. Our data is closer to that from students in British Columbia, however: Jhangiani & Jhangiani (2017) report that 54% of students said they had not purchased a textbook because of cost, 26% said they had chosen not to register for a specific course because of the textbook cost, and 27% had taken fewer courses. Still, our numbers are lower than those from a larger survey of students at our institution (So and Doering, 2016), which reported that 75% of all 1500 respondents, and 70% of those in their first or second year, had gone without a textbook for at least one course because of cost. In comparison to these numbers at our institution, it seems the first-year students in our survey are skewed towards the lower end of being affected by textbook costs in terms of purchasing textbooks or choosing courses.

Looking at student learning outcomes, we find overall that final exam marks and course marks after the changes to the course (including the adoption of the open textbook) are comparable to data from the previous three years in Physics 100. We see no significant change in the overall post-scores and overall shift-scores of the CLASS survey, while a small negative shift in the "problem-solving" category in 2015-2016 occurred. Possible reasons include the homework incentive and feedback as well as a change in the tutorial structure. Previously, students received marks for the homework that was worth 10% of the overall grade, while students received no marks for doing the homework in 2015-2016. In the future, we will again reward doing the homework with a few percent of the final grade. It is also possible that the current feedback on the homework is not sufficient. The previous commercial homework system had tutorial-type questions that provided hints and students could get feedback on important steps using these hints. Our homework system only provides correct/incorrect feedback, but we will enhance our homework system in the near future. For the Fall 2016 term, animated worked examples were inserted in the homework sets, which have been shown to enhance learning. In addition, the instructors have developed tutorial-type questions with step feedback, similar to commercial homework systems used previously. Finally, there may also be an effect due to the textbook. Although the open text features student-centered learning support such as problem-solving strategies and worked examples, it not as detailed as the previously used commercial text. However, we do not think that the textbook has a significant influence on the students' problem solving ability and confidence as compared to the course components that specifically focus on problem solving, in particular the homework and the tutorial sessions.

Regarding use of the open textbook, similar to results from other published studies we did not see a significant difference between how often students reported using textbooks in a "typical course" versus how often they used the textbook in Physics 100. Combining this with the data about their perceptions of the quality of the textbook suggests that they found the textbook useful for their course work while also having relatively few complaints or concerns about it.

This study is limited by its relatively small sample size compared to the number of students enrolled in Physics 100, as well as by the fact that the data is from one course at one university only. However, our results are largely consistent with previous research on open textbooks, with the exception of how much our students reported typically spending on textbooks per academic year and how costs affect their academic behaviours such as choosing which section of a course to take based on textbook cost. Our study is also focused on a course in which several changes were made, in addition to adopting an open textbook, including that the book was customized to fit this course and all course materials were consolidated on a single website. While this makes the results of our study less applicable to courses that adopt an open textbook only, it also provides evidence of the value of customizing an open textbook and embedding it into a website with homework and quiz questions.

Conclusion

The results of this study add to and mostly support the existing literature on open textbooks in terms of cost, learning outcomes, student use, and student perceptions. We have added to the so far limited data on students in the Canadian context, and our study focuses on a case of not just adopting an open textbook, but also significantly adapting it to fit a particular course.

However, a project like ours is not without cost: a significant amount of effort went into the project, in particular into the creation of the question database and the test bank. Four graduate physics students and two instructors worked over the summer months to get the resources ready; approximately C$20,000 from a teaching and learning grant from our university have been spent to date to pay the graduate students. The instructors contributed a significant amount of time and effort to secure funding, guide the project, select materials, get familiar with edX, and post materials, troubleshoot, etc. Additionally, one of the graduate students and the university's information technology staff provided technical help. Although the open text is free, integrating it into a coherent course package comes with significant development costs.

The literature on open textbooks related to cost focuses on cost savings to students, but it's important to keep in mind the possible costs for faculty and institutions in terms of time and support when using open textbooks. Even adopting an open textbook "as is" comes with a cost insofar as any change in a textbook for a course means at least some extra time is needed to connect the new text to the course. Our study suggests that taking advantage of the open license of open textbooks to make significant revisions can be pedagogically useful and appreciated by students, but it may require a good deal of time and resources.

Notes

1. EdX is a platform that has a number of Massive, Open, Online Courses (MOOCs), but there is also a platform for courses that are accessible only with a registration called edge.edX. The Physics 100 course, though the course materials are on edX, is not a MOOC. EdX is used as a Learning Management System for students and faculty in the course.
   
   
2. Another change to the course is that the labs are now completed at home, rather than in person. However, the labs mostly build experimental data skills and we do not believe this change is relevant to the study of the open textbook presented in this article.
   
   
3. This survey was incentivized: students in both the Fall 2015 and Spring 2016 courses who were invited to participate in the survey were eligible to enter a draw for one of four C$25 gift cards to either the bookstore or a coffee shop on campus (their choice).
   
   
4. 885 students registered for and completed at least some of the course; 74 students withdrew or otherwise did not complete the course during the normal term, leaving 811 students who completed the entire course (as reported in Table 1, above). Those who withdrew or otherwise did not complete the course would have left after the official add/drop period, and thus may not have been able to return their textbooks for a full refund.
   
   
5. For experts: we just show here the shifts for the "favorable" category. The "unfavorable" shifts are highly correlated with these. We just show here the general problem solving category. The shifts for problem solving confidence and sophistication are similar.
   
   
6. We did not define "typical course" on the survey; the question was: "For a typical course, how often do you use the required texts?" However, since most of the students who responded to the survey reported majoring in the sciences, then a "typical course" for them is probably a science course.
   
   
7. There could be a close connection between "somewhat important" and "of average importance." The latter is meant to signify something like "neutral." The answer range for this question was: "very unimportant," "somewhat unimportant," "of average importance," "somewhat important," "very important." We took this wording from another survey of post-secondary students in Canada, and in future will change the middle term to distinguish it further from "somewhat important."

References

Adams, W. K., Perkins, K. K., Podolefsky, N., Dubson, M., Finkelstein, N. D., & Wieman, C. E. (2006). A new instrument for measuring student beliefs about physics and learning physics: The Colorado Learning Attitudes about Science Survey. Physical Review Special Topics—Physics Education Research, 2(010101), 1–14.

Allen, G., Guzman-Alvarez, A., Smith, A., Gamage, A., Molinaro, M., & S. Larsen, D. (2015). Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for traditional general chemistry textbooks. Chemistry Education Research and Practice, 16(4), 939–948. https://doi.org/10.1039/C5RP00084J

Barbera, J., Perkins, K. K., Adams, W. K., & Wieman, C. E. (2008). Modifying and Validating the Colorado Learning Attitudes about Science Survey for Use in Chemistry. Journal of Chemical Education, 85, 1435-1439.

Bliss, T. J., Robinson, T. J., Hilton, J., III, & Wiley, D. A. (2013a). An OER COUP: College teacher and student perceptions of Open Educational Resources. Journal of Interactive Media in Education, 0(0). Retrieved from http://jime.open.ac.uk/article/view/252

Bliss, T.J., Robinson, T.J., Hilton, J.L., III, & Wiley, D.A. (2013b). The COUP framework. Retrieved from http://openedgroup.org/coup

California Open Educational Resources Council. (2016). White Paper: OER Adoption Study (April 1 2016). Retrieved from http://tinyurl.com/WPOERAdoption040116

Delimont, N., Turtle, E. C., Bennett, A., Adhikari, K., & Lindshield, B. L. (2016). University students and faculty have positive perceptions of open/ alternative resources and their utilization in a textbook replacement initiative. Research in Learning Technology, 24. http://dx.doi.org/10.3402/rlt.v24.29920

Fischer, L., Hilton, J.L., III, Robinson, J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159–172. http://doi.org/10.1007/s12528-015-9101-x

Florida Virtual Campus. (2016). 2016 Florida Student Textbook and Course Materials Survey [Draft]. Retrieved from https://florida.theorangegrove.org/og/items/3a65c507-2510-42d7-814c-ffdefd394b6c/1/

Heiner, C. E., Banet, A. I., & Wieman, C. (2014). American Journal of Physics, 82(10), 989-996. http://dx.doi.org/10.1119/1.4895008

Hilton, J., III (2016). Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development, 64(4), 573–590. http://doi.org/10.1007/s11423-016-9434-9

Illowsky, B. S., Iii, J. H., Whiting, J., & Ackerman, J. D. (2016). Examining Student Perception of an Open Statistics Book. Open Praxis, 8(3), 265–276.

Jhangiani, R. & Jhangiani, S. (2017). Investigating the perceptions, use, and impact of open textbooks: A survey of post-secondary students in British Columbia. The International Review of Research in Open and Distributed Learning, 18(4).

Jhangiani, R., Pitt, R., Hendricks, C., Key, J., & Lalonde, C. (2016). Exploring faculty use of Open Educational Resources in B.C. post-secondary institutions. BCcampus. Retrieved from https://bccampus.ca/2016/01/27/exploring-faculty-use-of-open-educational-resources-in-b-c-post-secondary-institutions/

Kestenbaum, D. (2014, October 9). How college students battled textbook publishers to a draw, in 3 graphs. Retrieved from http://www.npr.org/sections/money/2014/10/09/354647112/how-college-students-battled-textbook-publishers-to-a-draw-in-3-graphs

McKerlich, R. C., Ives, C., & McGreal, R. (2013). Measuring use and creation of Open Educational Resources in higher education. The International Review of Research in Open and Distributed Learning, 14(4). https://doi.org/10.19173/irrodl.v14i4.1573

Pitt, R. (2015). Mainstreaming open textbooks: Educator perspectives on the impact of OpenStax College open textbooks. The International Review of Research in Open and Distributed Learning, 16(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2381

Redish, E.F., & Hammer, D. (2009). Reinventing college physics for biologists: Explicating an epistemological curriculum. American Journal of Physics, 77(7), 629-642.

Senack, E., & The Student PIRGs. (2014). Fixing the broken textbook market. Retrieved from http://www.uspirg.org/reports/usp/fixing-broken-textbook-market

So, S., & Doering, K. (2016). AMS academic experience survey report. Retrieved from http://www.ams.ubc.ca/wp-content/uploads/2014/03/2016-ACADEMIC-EXPERIENCE-SURVEY-REPORT-FINAL.pdf

UBC Planning and Institutional Research Office. (2017). Enrolment and demographic data. Retrieved November 1, 2016, from http://pair.ubc.ca/data-access/off-campus-access/

U.S. Bureau of Labor Statistics. (2016, August 30). College tuition and fees increase 63 percent since January 2006. The Economics Daily: U.S. Bureau of Labor Statistics. Retrieved from http://www.bls.gov/opub/ted/2016/college-tuition-and-fees-increase-63-percent-since-january-2006.htm

William and Flora Hewlett Foundation. (n.d.). Open Educational Resources. Retrieved from http://www.hewlett.org/programs/education/open-educational-resources

XE Currency Charts (CAD/USD). (2016). Retrieved from http://www.xe.com/currencycharts/?from=CAD&to=USD&view=5Y

Appendix A

Data for Figure 1: How Often Respondents Have Taken Certain Actions as a Result of Textbook Costs

Note : N=150

Appendix B

Data for Figure 2: Grades Distribution in Physics 100, Fall 2012-Spring 2016

Note: The Fall version of Physics 100 is a face-to-face course, while the Spring version is a distance course.

Appendix C

Data for Figure 3: How Important Students Found Various Features of the Physics 100 Open Textbook

The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions by Christina Hendricks, Stefan A. Reinsberg, and Georg Rieger is licensed under a Creative Commons Attribution 4.0 International License.