Rating the Quality of Open Textbooks: How Reviewer and Text Characteristics Predict Ratings

Using data collected from peer reviews for Open Textbook Library titles, this paper explores questions about rating the quality of open textbooks. The five research questions addressed the relationship between textbook and reviewer characteristics and ratings. Although reviewers gave textbooks high ratings generally, reviewers identified differences in quality according to criteria and discipline. Physics and chemistry textbooks earned significantly lower ratings than other textbook types. Ratings were not significantly associated with reviewers’ status and experience, but were associated with reviewers’ place of residence. We discuss the implications for OER efficacy studies and other research.


Résumé de l'article
Using data collected from peer reviews for Open Textbook Library titles, this paper explores questions about rating the quality of open textbooks. The five research questions addressed the relationship between textbook and reviewer characteristics and ratings. Although reviewers gave textbooks high ratings generally, reviewers identified differences in quality according to criteria and discipline. Physics and chemistry textbooks earned significantly lower ratings than other textbook types. Ratings were not significantly associated with reviewers' status and experience, but were associated with reviewers' place of residence. We discuss the implications for OER efficacy studies and other research.

Introduction
For several years, the cost of higher education has been increasing at an alarming rate. The U.S. Government Accountability Office (2013) reported that from 2002 to 2012, tuition and fees increased by 89% and new textbook prices increased 82%, while overall consumer prices increased 28%. Each year over 1 million college students graduate with student loan debt averaging about $30,000 per student, which has very serious implications (Cochrane & Reed, 2015). Part of that debt can be attributed to textbook costs. Faced with prohibitive costs, some students are enrolling in fewer courses than they otherwise would, taking on more debt, or choosing not to buy required texts (Hilton, Gaudet, Clark, Robinson, and Wiley, 2013;Senack, 2014).
One promising development helping to address burgeoning college costs is the growing availability of open textbooks. Open textbooks are a type of Open Educational Resource (OER), with an open license that may allow copying, distribution, modification, and in some cases, sale (Bissell, 2009). Most open textbooks are 143 available free online, often with a low-cost print version. Potential benefits of using open textbooks include significant cost savings to students and increased flexibility for instructors Senack, 2015).
Although open textbooks offer some substantial cost-advantages, they represent only a small fraction of the textbooks adopted by faculty members. One reason for this low adoption is that most professors do not know that open textbooks exist, much less where to find them. In a nationally representative survey of college faculty, Allen and Seaman (2014) found that two-thirds of the 2,144 respondents were not aware of OER. Even instructors who are aware may avoid open textbooks because they question the quality of freely available resources, especially when such sources aren't connected to a reputable publishing house, institution, or author (Pitt, 2015). College instructors who have their students' best interest in mind are not willing to replace a high-quality, high-cost textbook with a low-quality, low-cost alternative, and determining the quality of a non-traditional text may take more time and effort than instructors prefer to spend.
To help instructors determine the quality of open textbooks, the Open Textbook Library, housed in the Center for Open Education at the University of Minnesota, has been aggregating peer reviews of these textbooks. Reviews are for faculty, by faculty; "The project pays faculty members to review open textbooks, as does a similar effort at the University of British Columbia (Canada), and the two initiatives share those peer reviews" (Center for Education Attainment and Innovation, 2015, p.2). To be included in the library, textbooks "[m]ust be currently in use at multiple higher education institutions, or affiliated with a higher education institution, scholarly society, or professional organization" (Center for Open Education, n.d., para. 1). In this paper, we examine data collected from peer reviews for Open Textbook Library titles to discuss questions about rating the quality of OER. Human Nutrition courses. While online students used the open textbook more frequently and rated it more highly than campus students, both campus and online students reported high levels of satisfaction and preferred using the online text to buying a traditional textbook.

Review of Literature
Unlike the previous two studies, most research articles on OER involve multiple open resources. In one such study, Bliss, Hilton, Wiley, and Thanos (2013) reported results from a survey of 11 instructors and 132 students using OER at eight community colleges. A large majority of respondents reported positive 144 perceptions of OER. Ten of the eleven instructors stated their students were as or more prepared than students using traditional texts, and 97% of students said that the open textbooks were as good as or better than traditional textbooks. Building on the previous study, Bliss, Robinson, Hilton, and Wiley (2013), surveyed an additional 58 teachers and 490 students about the cost, outcomes, usage and quality of OER used in their college courses. As with the previous study, a large majority of respondents rated OER as equal or higher in quality compared to traditional textbooks. The respondents who rated OER quality lower than that of traditional textbooks (11% of instructors and 6% of students) cited poor text quality and technology problems as the reasons for their ratings. Hilton, Gaudet, Clark, Robinson, and Wiley (2013) examined the impact, in terms of cost savings, student retention and success, and user perceptions of adopting open resources in mathematics classes at Scottsdale Community College (SCC). Using data from 2,043 students taking five different courses, researchers determined that by using OER, students saved a significant amount of money without suffering adverse effects in terms of educational outcomes. In surveys, students and faculty reported mostly favorable perceptions of the OER materials used in the courses. Feldstein et al. (2012) reported findings from a year-long pilot study in which 991 students taking courses in the Virginia State University School of Business used free online textbooks and other resources provided by Flat World Knowledge. Of the 315 students who completed the semester-end survey, 95% agreed that the OER course materials were easy to use, and most respondents preferred the digital content to traditional textbooks. Students in the courses that used open textbooks earned higher grades than students in similar courses that used traditional textbooks.
All of the above studies include student perceptions; several studies focus only on instructor perceptions.
In their 2014 article, Allen and Seaman (2014) reported that among respondents aware of OER, 74% rated the quality of OER materials as similar or superior to that of traditional resources. About 85% rated the efficacy of OER materials as similar or superior to the efficacy of traditional materials. Kelly's 2014 study examined educator perceptions of OER and the impact of perception on adoption and use. Survey participants included 128 educators from higher education, K-12 schools, and workplace training. Kelly found that individuals with high self-efficacy were more likely to perceive OER as easy to use, and that ease of use correlated with perceived usefulness.
As part of a collaborative project between the BCcampus-led Open Textbook Project and OER Hub, researchers surveyed post-secondary educators interested in OER to examine awareness, usage, barriers, outcomes, and perceptions (Jhangiani, Pitt, Hendricks, Key, & Lalonde, 2016). Of the 78 respondents, 77% had used OER. Most respondents rated OER quality as comparable or superior to that of traditional materials; educators who had adopted OER rated the quality significantly higher than educators who had not. Respondents reported that the top two barriers to using OER were finding relevant and high quality OER. Faculty at research-intensive universities reported significantly lower barriers to finding high-quality OER than did faculty at teaching-intensive universities or colleges/institutes. Pitt (2015) reported results of two surveys of 126 educators who had used open textbooks that were published by OpenStax. In relation to the issue of quality, almost 75% of the 2013 respondents and 95.7% of 2014/2015 respondents reported that they would be more likely to choose OER produced by a trusted To date, concerning OER quality have typically been surveys of students and/or faculty members that have used OER in their classroom. One weakness with this approach is that students may be more likely to report based on convenience, rather than true efficacy (Kvavik, 2005). In addition, the faculty members may have been biased in terms of reviewing materials that in many instances they helped to create or select. In the present study, we will address this limitation by looking at a different aspects of reviews of open textbook quality. Rather than use survey results, we will us data from the peer reviews facilitated by the Open Textbook Library. These faculty reviewers had not adopted the resource at the time of their review.
Therefore, their reviews may be somewhat less biased. Moreover, beyond reporting descriptive values of quality, we considered evaluations in terms of reviewers' characteristics (tenure status, job track, location), and textbook content areas. Our specific research questions will be discussed below.

Method
The Question 5: Do the nine textbook types differ across the quality criteria or overall quality?
Because the data were ordinal Likert scales and the distributions were significantly skewed (as seen in Table   1), all inferential tests were completed using non-parametric Kruskal-Wallis tests of differences. Although the evaluations were nested with repeated measures, the data were not robust enough to use multi-level modeling. The results should be interpreted cautiously, with an eye toward general trends more than solid inferential tests of null hypotheses.

Results
The distributions of the rating across the 10 quality criteria and overall quality are displayed in Table 1. In general, the median ratings were high (4 & 5). The means were high with reasonable standard deviations and all the distributions were significantly negatively skewed.  Table 2 illustrates the results associated with all inferential tests. Other tables will present the descriptive statistics across all criteria. As seen in Table 3, all reviewers generally rated the textbooks favorably across the 10 criteria. There were no significant differences between the Tenure Track and Non-tenure Track reviewers on any of the 10 quality criteria or overall quality score ( Table 2). The only significant difference among the four subgroups of reviewers on quality criteria or overall quality score was observed between the Other reviewers and the Adjunct Faculty reviewers in terms of their assessment of modularity ( Table 2).
The Other reviewers judged the modularity of the books to be lower than Adjunct Faculty reviewers.  As seen in Table 4, US and Canadian reviewers generally rated the textbooks favorably. There were some significant differences between US and Canadian reviewers (Table 2). These occurred on

149
Comprehensiveness, Accuracy, Clarity, and Overall Quality. In all these cases, the Canadian reviewers tended to rate the textbooks lower than the US reviewers. There were some significant differences in ratings among the textbook types (Table 5). These occurred on Accuracy, Clarity, Organization, and Interface. In the post-hoc pairwise comparisons the 12 Physics/Chemistry textbooks tended to be the consistently outlying (lower-rated) textbooks. 151 for overall quality. The highest-rated criteria were grammar (4.67) and consistency (4.52). The lowest-rated criteria were interface (3.95) and comprehensiveness (4.13). This finding indicates that reviewers are identifying strengths and weaknesses of the textbooks they review across the different categories. The fact that the reviews were favorable across all nine criteria indicates that the reviewers judged the textbooks housed on the Open Textbook Library to be of relatively high quality.
Our second and third research questions asked whether reviewers' professional track and status affect their ratings. We found no significant difference in the quality of ratings by tenure-track and non-tenure track reviewers. Dividing reviewers into four groups-Adjunct, Tenure Track Assistant Professors, Tenured Associate or Full Professors, and Other (such as student life advisors that teach courses)-again yielded no significant difference in the quality of reviews. This finding has important implications for research studies of OER use at community colleges, where many instructors are adjuncts. The data suggests that quality can be identified without particular bias according to status; more experienced teachers (such as full professors) were not more discriminating than less-experienced teachers in terms of the quality of textbooks. In other words, in studies involving faculty ratings, it would not be accurate to argue that reviews should be discounted because they are by faculty of different ranks, because faculty in this study tended to agree about the textbook quality, regardless of rank.
Fourth, we asked whether U.S. and Canadian reviewers rate the textbooks differently. Across all categories, typical ratings by Canadian reviewers were lower than typical ratings by U.S. raters. Our data indicates that raters' country of residence predicts some differences in ratings; however, with the data we have we cannot determine why this occurred. We might speculate that the reviewers from Canada seem to have higher standards than U.S. reviewers. There are a few important implications of this finding. First, although OER are typically available online facilitating international use, they may not be perceived as being equally useful in different educational systems. In some contexts, instructors may need to significantly revise open textbooks produced elsewhere, which revising is allowed under most open licenses. Second, when consulting online reviews of open textbooks, readers should be aware of reviewers' contexts. Reviews may be most useful when the reviewer and reader teach similar courses in the same country. Third, in studies such as this one involving raters from multiple countries, average scores may hide differences among various groups of raters.
Finally, we asked whether quality varies among textbook types. According to the data, perceived quality varies by discipline or textbook type. Among the nine disciplinary categories, physics and chemistry books tended to be rated significantly lower in some criteria than textbooks in the other disciplines. In our data, there is a correlation between low scores for science textbooks and low ratings by Canadian reviewers; Canadian reviewers rated a higher proportion of physics and chemistry books than any other disciplinary category.
Taken together, the data indicates that overall quality of textbooks in the Open Text Library is high, but textbooks do vary in quality, and those variances can be identified by reviewers.

Limitations
As noted previously, this study was limited by the available data. This study included 416 reviews of 121 textbooks, but some textbooks received many more reviews than others. Reviewers were given very little guidance in the rating process. Of the 416 reviewers, only 44 were Canadian, and the Canadians reviewed only 5 of the 10 text types. Based on these limitations, the results suggest patterns, but may not be generalizable. Further studies are needed to understand how textbook type and reviewer characteristics affect textbook ratings.
Another limitation stems from the nature of the reviews. In the Open Textbook Library, reviewers attach their names to their ratings. The rationale for publishing reviewers' names was to increase rater accountability and consumer trust. However, it is possible that named reviewers inflate scores compared with anonymous reviewers. Based on this limitation, comparisons between Open Textbook Library ratings and anonymous ratings may not be valid. For this study, all comparisons are among Open Textbook Library ratings.

Conclusion
In online textbook rating systems. We hope that others will pursue similar research to help explain OER ratings and quality.
In this study, we examined faculty ratings of textbooks in the Open Textbook Library, and found that the ratings tend to be high, with some variance. Based on the ratings, instructors wishing to find high quality open textbooks should be able to trust the quality of texts in the Open Textbook Library. As Pitt (2015) suggested, recommendations by faculty are highly influential. However, the ratings reflect only perceptions, not outcomes for students who use the textbooks, nor comparisons of those outcomes with outcomes for students using proprietary textbooks. Perhaps the Open Textbook Network and other open textbook repositories could add measures of effectiveness to their online rating systems. Combining ratings with efficacy measures would provide a more complete picture than ratings alone, greatly benefiting potential users of open textbooks.