Evaluation of Open Educational Resources for an Introductory Exercise Science Course

While open educational resources (OER) have gained popularity, nearly three quarters of faculty are not aware they are available for use. However, when used, they are well received and do not negatively impact quality of learning. OER can be used within a variety of platforms, including software that aims to be more interactive and engage students in active learning and assessment. One such platform is Top Hat, which was used by the authors of this study to develop a textbook for an introductory exercise science course. We assessed student’s perceptions of Top Hat and barriers to use for reading their textbook and for class assessments over the course of two years. A total of 486 students were registered for this course. Although two thirds of students had previous experience with Top Hat and half of those used the textbook feature, students (n = 39, 38%) were apprehensive about reading their textbook online via Top Hat. However, these feelings resolved as students became comfortable with the platform’s features. Nearly 80% of students have sometimes or never acquired their textbooks before the start of the semester, despite 96% who expressed the importance of having their materials accessible online and available on or before the first day of the course. This indicated that students understood the importance of having their materials for the start of the semester, however they perceived the barriers of purchasing books to be greater. Therefore, using OER and Top Hat removed student learning barriers and had potential to increase course participation and success.


Introduction
Open educational resources (OER) are defined as "teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and re-purposing by others" (Hewlett Foundation, 2013, p. 1). Such OER include things like "full courses, course materials, modules, textbooks, streaming videos, tests, software, and any other tools, materials, or techniques used to support access to knowledge" (Hewlett Foundation, 2013, p. 1). Unfortunately, as many as 75% of higher education faculty were not aware of OER and although they indicated they would try them, they often cited barriers to adopting OER, such as difficulty finding appropriate or enough resources for their needs, and time and effort needed to evaluate the resources (I. E. Allen & Seaman, 2014). Locating resources and the time to evaluate them might be the most critical, as faculty indicated that quality and proven efficacy of materials were the most important criteria for adopting resources for a course.
Interestingly, things like cost, currency, and being rated highly by other faculty were very low in importance (I. E. Allen & Seaman, 2014). Although these barriers are real, OER have been well accepted by both faculty and students.  found that 55% of faculty believed the quality of the OER they were using was equivalent to the material they had used previously, while 33% felt the quality was better. Greater still, 90% of students perceived OER to be as good as or better than traditional textbooks (Bliss, Robinson, et al., 2013). Similarly, Feldstein et al., (2012) reported 72% of students perceived the OER to be better than a traditional text. Perhaps most importantly, quality of learning did not appear to be impacted negatively by using OER (G. Feldstein et al., 2012; J. L. Hilton et al., 2013;Lovett et al., 2008;Nusbaum et al., 2020) or may have been improved (J. Hilton & Laman, 2012;Pawlyshyn et al., 2013).
OER materials can take several forms, including images, videos, e-books, existing course materials like quizzes or assignments, or entire courses. Additionally, there are many platforms available for hosting OER content, such as OER Commons or OpenStax. OER can also be accessed from within many learning management systems (LMS). Another option is Top Hat, educational software designed to blend in-person and virtual learning. The software allows instructors to use it for (a) in-class presentation and clicker-type functionality; (b) hosting a course textbook and interactive material; and (c) providing assessments including assignments, quizzes, and exams. The advantages of the textbook feature in Top Hat are that it (a) can host any OER content, (b) is completely customizable by the instructor, and (c) follows the strategic goal for OER that "educators have the legal rights to retain, reuse, revise, remix, and redistribute educational resources as they determine-without having to ask permission" (N. Allen et al., 2015, p. 2).
Top Hat also allows authors to update content and material easily and quickly both during and between semesters. For example, if an instructor would like to add additional study materials or practice quizzes, they can either create new material or look for an available OER to upload to the platform. This also makes it easier to determine the effectiveness of materials-questions and real-time grades, showing how well material is being understood and by whom, can be viewed student-by-student or as a whole class. However, while students may be familiar with using an LMS for coursework, many are not familiar with online educational software hosting sites, and there are barriers to students' use of e-books (Pierard et al., 2020).
It is imperative that we understand how students perceive these sites and if they have a role to play in the future of OER adoption.
In 2015 our institution established the Textbook Initiative Taskforce (Ohio University, 2016) to find ways to lower or eliminate the high cost of course materials. One of the main foci of the taskforce was the Alt-Textbook program, to promote the use of OER in place of traditional textbooks. In 2017, the University became a member of the Open Text Network and implemented incentive programs to encourage faculty members to learn about and implement such resources. In fall 2017, the University formed a partnership with Top Hat, and since then increasing numbers of faculty have joined the program. Savings to students in the first academic year (2017-2018) exceeded $1 million (Business Wire, 2018). The instructors in this study were new participants in the Alt-Textbook program and selected Top Hat as their delivery platform.
In addition to saving money for students, this approach also ensured that they had the material on the first day of the course, and made the textbook and assessments more interactive. Therefore, the purpose of this study was to investigate student perceptions pre-and post-semester regarding the use of Top Hat for reading their textbook and for class assessments.

Methods
This study was conducted at a large, research institution located in Appalachian Ohio where approximately 26% of the students were considered low income. Prior to the fall semester 2018, three instructors wrote a textbook for Introduction to Exercise Science and made it available via Top Hat. The course was a major requirement, typically completed by freshmen and sophomore students studying exercise science. It was also a required course for one minor in a different college, and an elective course for two other minors, one in the same school and the other in a different college. The instructors had been teaching the course for one to six years, having previously adopted a traditional textbook. However, following the partnership between Ohio University and Top Hat, aimed at fostering OER adoption, the instructors opted to author their own textbook to better fit the course objectives and to make it free for enrolled students (approximately 240 per academic year). The book consisted of 12 chapters, six of which were adapted from OER materials and six written by the instructors. Each chapter contained text, videos, and images to support learning. The resource also contained a highlighting feature, note-taking feature, in-chapter review questions, and endof-chapter quizzes.

Data Collection Procedure
Two anonymous online-based surveys (available at doi.org/10.7303/syn24195429) were e-mailed to all students enrolled in Introduction to Exercise Science during fall and spring semesters from 2018 to 2020, for a total of four semesters. The course was offered with two face-to-face and one online section in the fall and one face-to-face and one online section in the spring. A pre-semester survey was e-mailed to students once during the first two weeks of the semester, with a post-semester version e-mailed during the final two weeks of the semester.
In the pre-semester survey, the first set of questions pertained to basic demographics (e.g., age, gender) and academic history (e.g., college level, course delivery format, previous experience with this course, previous online course experience). The next set of questions asked students about their comfort level with using technology (e.g., cellphones, laptops, e-books, Websites, word-processing software) both in their personal and their academic life. Students were then asked about their familiarity with Top Hat including previous experience (i.e., other courses using it) and were also asked open-ended questions regarding opinions or feelings about using the software. Next, students were asked about their textbook buying habits and barriers to purchasing books for courses. Finally, students were asked to respond to questions regarding the importance of course materials, including quality and reliability, availability (e.g., online, for free, available first day of class), and the credibility of their instructors as authors.
In the post-semester survey, students were asked the same demographic questions as during the pre-survey followed by questions related to their experience. First the students were asked to look back at their precourse opinions on using e-books to see if they had changed, and if they did or did not like using Top Hat to read their textbook. They were also asked if they used Top Hat's optional highlighting and note-taking features and if so, how often and if not, why not. Next, they were asked if they felt Top Hat was user friendly, and if in future they would drop or avoid taking a class that used it. Students were again asked to respond to questions regarding the importance of course materials, including quality and reliability, availability (e.g., online, for free, available first day of class), and the credibility of their instructors as authors. Finally, students were asked how well they felt the Top Hat textbook and their instructor helped them achieve the course objectives, which were explicitly listed.

Data Analysis
Multiple-choice and multiple-select survey items were analyzed quantitatively. Where it was not possible to qualitatively compare data, a descriptive analysis of results was conducted instead. Open-ended items were analyzed using thematic analysis (Braun & Clarke, 2006).

Results
Over the four semesters examined for this study, a total of 486 students were registered for the course; 323 students were registered for in-person sections, while 163 were registered for online sections. The presemester surveys had a 49% completion rate (n = 240), while the post-semester completion rate was 33% (n = 150; based on course completion numbers). Demographics for participants are presented in Table 1.
Approximately 50% of the students had previous experience with an online course; 44 (36%) stated that they preferred online courses, while 78 (64%) preferred in-person courses. Responses to questions about their comfort using technology in their personal versus academic lives can be seen in Table 2. A Wilcoxon rank test revealed that there was a significant difference in comfort between personal and academic use of cellphones (W = 350, p < 0.001), but there were no statistically significant differences in use of any other technologies between personal and academic use. When asked about their textbook buying habits, 77% of students said they always bought the book or bought if it was required. Of the remainder, 21% said they sometimes bought the book if it was required, recommended, or their decision depended on the cost of the book. Only 1% said they hardly ever purchased their books (Figure 1).

Students' Textbook Buying Habits
When asked about the barriers to purchasing a textbook, 50% stated the biggest barrier was cost. Other barriers included not wanting the book once the course was finished (19%), the hassle of buying (16%), and the hassle of returning (8%). Other reasons (6%) given were that "professors rarely used them" and "I don't need it" (Figure 2).

Barriers to Purchasing Books for Courses
When asked how often they had their textbooks for the first day of class, 37% said never, 25% said sometimes, 16% said half the time, and 22% said always or almost always. When asked about the maximum amount they would pay for their textbook, 32% said they would pay no more than $75, while 37% said they would pay no more than $100. Interestingly, 27% said they would purchase the book regardless of the cost ( Figure 3).

Theoretical Cost Students Were Willing to Pay for Textbooks
Finally, students were asked how important it was to know their instructors were the authors versus someone else in the field.; 89% felt it was extremely to moderately important that instructors authored the textbook, while 11% felt it was slightly or not at all important. Students were also asked the importance that class materials be free, available on or before the first day of the course, and accessible online. Nearly all (99%) believed it was moderately to extremely important that the materials were free, 97% felt it was moderately to extremely important that the materials were available on or before the first day of the course, and 96% felt it was moderately to extremely important that the materials were accessible online. Finally, when asked about the importance of the reliability and quality of the materials, all students reported that it was moderately to extremely important that the materials were reliable and of good quality.

Post-Semester Survey
When asked to reflect on whether their use of Top Hat changed their pre-course feelings on using e-books, 75 (50%) said yes and 75 (50%) said no. Of those who did change their mind, a common theme was that of user-friendliness, including "my mind changed because I did not realize the text would be so user friendly," "it's easy to use," "it turned out better than expected," and "I really enjoyed using the tools Top Hat offered that were unique." Additional comments included those on convenience such as "at the beginning I hated it, but it was nice not to have to lug around a textbook" and "it changed a bunch, I didn't realize how convenient Top Hat was." Further comments suggested it had impacted their learning including "Top Hat was as key part of my learning," "it has helped me improve my study skills and I noticed a huge change in my grades from beginning to now," and "it was much easier to retain information." The vast majority of individuals who did not change their mind simply stated this, but others opined "I still feel that I'd prefer to use pencil and paper since I learn better that way than reading online," "the textbook was sometimes hard to navigate," and "I did not like the Website as a whole." Interestingly, one student noted they are "less likely to do in-depth reading with e-books" and approximately 20% of students would have liked to have a physical book because they do not like reading e-books.
Students were asked if they used Top Hat's special features of highlighting or note-taking within the online textbook; only 44% reported they used highlighting and 14% used note-taking. Of those who used the highlighting feature, 51% used it for most or all chapters, while 49% used it for a few chapters. Of those who used the note-taking feature, 40% used it for most or all chapters, while 60% used it for a few chapters.
When prompted as to why they did not use these features, the most common response was "didn't know it existed" or they did not see it as useful and preferred to take their own notes.
When asked if they felt Top Hat was user-friendly, 96% of students said yes, 92% said they would not avoid taking a class that used it in the future, and 99% said they would not drop a course that used it. The reasons for avoiding or dropping a course that used Top Hat was "didn't like it" and "didn't understand the requirements within." When reflecting on the questions regarding the authoring and availability of the materials that they were asked pre-semester, students still reported that it was extremely to moderately important that the materials were reliable (99%) and of good quality (98%); 97% believed it was moderately to extremely important that the materials were free, and 96% felt it was moderately to extremely important that the materials were accessible online and that the materials were available on or before the first day of the course. In the postsemester survey, the importance of knowing that the instructors were the authors dropped to 81% who said this was either extremely to moderately important.
Finally, students were asked how well they felt the textbook and instructor helped them achieve the learning objectives for this course, which were explicitly listed. Figure 4 reports results of a Wilcoxon rank test that found scores for the instructor were significantly higher than for the textbook in terms of all objectives except one, namely "define key physiological and anatomical systems," likely because there was a chapter in the book titled "Anatomical Key Terms."

Discussion
Some faculty, like some parts of society, tend to think of modern students as being technology savvy. This may be the result of perceiving that students, born into an environment with increasing exposure to technology, are no doubt highly skilled technology users (Howe & Strauss, 2009;Oblinger & Oblinger, 2005;Tapscott, 2008). Each subsequent generation has the potential for exposure to more technologies and more opportunities to use them. However, the claims that each generation increases their technical skill and expertise, especially educational application of technology, have been heavily critiqued (Bennett et al., 2008;Helsper & Eynon, 2010;Kirschner & De Bruyckere, 2017). In our study, 92% of the students reported being extremely comfortable using cellphones for their personal lives versus only 58% in their academic lives, which is statistically significant. Even so, students are frequently asked to use their phones for academic purposes, and faculty may assume students feel comfortable doing so. Indeed, nearly the same number (60%) reported feeling that smartphones are important for their learning needs (Galanek et al., 2018). This appears to be an important disparity and may exist because many instructors ban the use of personal devices in the classroom, though students have reported that they want their instructors to use more technology in the classroom and would like to see more use of OER (Brooks & Pomerantz, 2017).
It may be important to explain expectations for technology use in the classroom, as students have reported feeling more comfortable with technology the more it is used (Margaryan et al., 2011). This was clearly evident in our study, as students frequently reported being apprehensive or not liking the online textbook at the start of the semester, though their feelings had changed by the end. There were some common themes in the comments from students when they were asked to reflect on the use of the textbook in Top Hat, including interaction, navigation and ease of use, and convenience. Most students appeared to enjoy using the textbook, in particular the fact that it was interactive and included quizzes, stating that "I liked how you had to engage with the textbook on Top Hat" and "I like seeing what I miss, and what questions I may get correct, as this helps me learn what I need to study more." While some noted that initial use and learning was a challenge, they "felt it was easy to navigate and helpful." Additionally, some remarked that "I really enjoyed that it was broken down by section," and "it was easy to use and find information." Many students remarked that the textbook in this format was convenient, that it was online and free, and they appreciated not having to have a physical book; even so, there were many who also remarked that they would still have preferred a hard copy version.
Students recognized the ease of use and convenience in the online textbook, including highlighting and note-taking features, as well as being able to have the book open while in lecture but not having to carry a textbook with them. They felt that it helped them improve their learning, though we did not assess how this compared to a traditional textbook. However, nearly half of those surveyed did not use some of the interactive features within the textbook, such as highlighting (yes: n = 66, 44%; no: n = 84, 56%) and notetaking (yes: n = 21, 14%; no: n = 128, 86%). This is despite instructors providing in-class demonstrations of the features at the start of the semester. Many of the students noted that they preferred to take handwritten notes, which should be encouraged as it is well documented that this leads to better learning (Mueller & Oppenheimer, 2014).
A major advantage of using an online textbook is the availability of the materials, not only in terms of cost but to ensure students are ready for the start of their courses. When surveyed pre-semester, 97% of students believed it was extremely or very important that their textbooks were free, however interestingly, only 3% of respondents reported they did not purchase or hardly ever purchased their textbooks. About a quarter of students stated they would purchase their books regardless of cost while a third would pay up to $75. The upper limit for cost for a textbook appeared to be $100 and 37% said they would buy a book at up to this price; however, many science-based courses require books that are double or triple this cost (Vitez, 2018).
Adopting OER materials can save students hundreds of dollars per course, which would undoubtedly add up to thousands over the span of their education. Indeed, the partnership between Ohio University and Top Hat has saved students an estimated $1 million in textbook costs, based on publisher prices (Business Wire, 2018).
It is important to ensure that students have materials at the start of the semester. When surveyed presemester, 37% of students said they never have their textbook before the first day of class, 41% sometimes do, while only 22% always or almost always have their materials. This is despite federal regulations requiring textbook information be posted in time for registration (Higher Education Opportunity Act, 2008), some five months in advance of the start of the semester. When asked how important it was to them that the course materials were accessible online and available on or before the first day of the course, 96% felt it was moderately to extremely important. This indicates that students understand the importance of having their materials for the start of the semester, however they perceive the barriers of purchasing books (such as cost and effort) to be greater. Additionally, retention rates are higher when students have their materials earlier in the semester (Bliss, Robinson, et al., 2013;Ozdemir & Hendricks, 2017). Therefore, if we remove the barriers of having to purchase a textbook and ensure its arrival before the start of term, students are more likely to be prepared. This is a main advantage that an OER can provide, whether it is through an LMS or a stand-alone course option.
We assessed student opinion on how successful the textbook and instructors were in helping students achieve the course objectives. Instructor ratings were significantly higher versus the textbook in all but one objective, however, it is not clear whether this result is meaningful.  stated that using OER should be done if it does not disrupt learning outcomes. Clearly the use of OER in our courses has not disrupted the achievement of the learning outcomes, and it should be noted that the students felt the instructors were more important, but that the textbook was complementary to their learning.
This study is not without limitations. For example, sample sizes were dictated by student response rates, which were relatively high pre-semester (49%) but lower post-semester (33%). Additionally, there was likely sample bias due a non-random sample (i.e., sampling a population of students in a course) and potential for response-bias based on how the students felt about Top Hat (i.e., responses might be primarily from students who either really liked or really disliked Top Hat). Even so, we saw low ratings combined with positive comments, so this issue may not be major concern.
Conclusions Most importantly, the use of Top Hat was well received by students. It removed the barriers of cost and access to course materials before courses began, ensuring students were better prepared, which may improve outcomes. This may also provide a benefit to the university as a whole by improving student satisfaction and retention rates.
In addition, as OER gain increased acceptance and systems like Top Hat are used by more faculty across institutions like our own, there may be a greater need to orient students to the use of these resources and systems and make each instructors' expectations as clear as possible. Many of our participants noted they had used Top Hat and some OER in a variety of ways in past classes. During this transition it will be important that we support students through this change and remind instructors that there is no one-sizefits-all approach with Top Hat or use of OER.