Volume 18, Number 4
David Wiley1, Ashley Webb2, Sarah Weston3, and DeLaina Tonks4
1Lumen Learning, 2,3,4Mountain Heights Academy
This article explores the relationship between open educational resources (OER) created by students for use by other students, the long-term sustainability of the movement toward OER, and the success of students who use OER created by other students as part of their core curricular materials. We begin by providing definitions and a broader context for thinking about the possibility of student-created OER. We then describe a course context in which student-created OER have been slowly integrated into an online class over several years and examine the impact on student learning associated with their introduction.
Keywords: open educational resources, student-created content, sustainability
As defined by the William and Flora Hewlett Foundation, open educational resources or OER are:
teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and re-purposing by others. Open educational resources include full courses, course materials, modules, textbooks, streaming videos, tests, software, and any other tools, materials, or techniques used to support access to knowledge. (Hewlett, 2016, para. 7)
While a simple library or Google Scholar search demonstrates that OER have been the topic of hundreds of research articles over the past decade, recent research suggests that most postsecondary faculty are unaware of the existence of OER. For example, survey work by Allen and Seaman (2016) shows that only 25.6% of US faculty are either "Aware" or "Very Aware" of OER. Consequently, most US faculty have yet to be introduced to the idea of open educational resources.
When faculty initially hear about OER, their first impulse is often to ask questions like how do they get created? Even with the involvement of major philanthropy, how can the creation of OER be sustained over the long term? The topic of development models that can sustainability support the creation of OER has been discussed at length for over 10 years. OECD commissioned a series of articles on the topic, including papers by Downes (2006), Dholakia, King, and Baraniuk (2006) and Wiley (2006). These articles list and describe several potential revenue models for the ongoing acquisition of funding to support the creation and upkeep of open educational resources. Many of the projects described in the OECD papers were constantly searching for funding to continue operations, and not all of them succeeded in that task (Parry, 2009).
As they maintain a largely financial focus, the commissioned papers overlook the possible role of students in the creation and maintenance of OER. In describing their primary user groups, Dholakia et al. (2006) list authors, instructors, and "students, who consume the educational materials, and learn" (p. 8). This quote is representative of a broader view of students as novices without the expertise necessary to meaningfully contribute to the creation and maintenance of OER. However, when framed in this manner, the issue of the potential role of students as creators of OER closely resembles the history of the Nupedia and Wikipedia projects.
Jimmy Wales and Larry Sanger launched Nupedia in 2000. The goal of the Nupedia project was to publish free, openly licensed encyclopedia articles that were written and peer reviewed by experts. This proved to be a slow and laborious effort, and fewer than 30 articles were produced in the project's first year. Wales and Sanger then created Wikipedia in 2001 as a sandbox in which anyone could contribute to the creation of articles that would be subjected to Nupedia's peer review process before publication. As Sanger (2005) later wrote:
My initial idea was that the wiki would be set up as part of Nupedia; it was to be a way for the public to develop a stream of content that could be fed into the Nupedia process. I think I got some of the basic pages written--how wikis work, what our general plan was, and so forth--over the next few days. I wrote a general proposal for the Nupedia community, and the Nupedia wiki went live January 10. The first encyclopedia articles for what was to become Wikipedia were written then. It turned out, however, that a clear majority of the Nupedia Advisory Board wanted to have nothing to do with a wiki. Again, their commitment was to rigor and reliability, a concern I shared with them and continue to have. Still, perhaps some of those people are kicking themselves now. They (some of them) evidently thought that a wiki could not resemble an encyclopedia at all, that it would be too informal and unstructured, as the original WikiWikiWeb was (and is), to be associated with Nupedia. They of course were perfectly reasonable to doubt that it would turn into the fantastic source of content that it did. Who could reasonably guess that it would work? But it did work, and now the world knows better. (para. 33)
The success of Wikipedia as a "fantastic source of content" was completely unexpected, even by its creators. Having witnessed an existence proof of the creation of excellent novice-produced content in the Wikipedia context, there is reason to believe it may be possible that novices (students) might be able to produce excellent open content in educational settings as well.
Shirky (2010) argues that the success of projects like Wikipedia can be attributed (in part) to a combination of what he calls a "cognitive surplus" and the manner in which technology allows us to put this previously unproductive free time to better use. Where the potential of a quiet evening at home was previously limited to playing cards or listening to the radio, that same evening can now be spent contributing to Wikipedia or participating in a broad range of citizen science projects online. While students may not have a large quantity of free time at their disposal to volunteer to participate in projects that make the world a better place, they do spend an incredible amount of time doing homework. We pause here to provide the reader with a sense of how much time students spend doing homework.
According to the US Department of Education's Digest of Statistics (Department of Education, 2016), there were over 20 million students enrolled in a degree-granting postsecondary institution in the US during the fall of 2014. Let us assume that each of these students enrolled in one three credit course that term. For each of the three hours those students spend in class each week they can expect to do two hours of homework outside of class, for a total of six hours of homework per week over a typical 15-week course. Multiplying 15 total weeks times six hours of homework per week times 20 million students equals 1,800,000,000 hours spent collectively on homework per semester. Because many students take more than a single course each semester, this estimate of approximately 2 billion hours per semester spent doing homework is conservative but adequate for our purposes.
Following Wiley (2013), we define disposable assignments as those assignments that both faculty and students understand will ultimately be thrown away. Essays and practice problems are examples of assignments that frequently fit into this category – students spend significant time and energy doing this work, faculty spend significant time and energy grading this work, and then it gets returned to students who throw it away. While there is likely a learning benefit to students who complete these disposable assignments, literally throwing away 2 billion hours of work each semester seems like a missed opportunity.
To contrast disposable assignments, we define renewable assignments as those which both provide a learning benefit to the student and result in OER that provide a lasting benefit to the broader community. Assigning students to create, revise, or remix OER results in an artifact that can be used both to evaluate student learning and increase the diversity of voices and perspectives available for study by later students. While not all disposable assignments can be converted easily into renewable assignments, if even a tiny fraction of the 2 billion hours (or more) of work done on disposable assignments could be put to more productive use, this
Murray (2008) provides an excellent example of a renewable assignment. Instead of writing essays that would be graded by their instructor and then thrown away, students enrolled in SPAN312: Murder, Madness, and Mayhem: Latin American Literature in Translation at the University of British Columbia did their writing on Wikipedia where others could read and benefit from their work:
This was not a course about Wikipedia but rather, as with my other courses, its focus would continue to be on Latin America and on the reading of a set number of Latin American literary texts.... These books are neither short nor simple; most of the students' time would be spent reading these hefty tomes, and most of the class time spent explaining and discussing them. So there was little direct discussion of Wikipedia in the classroom. Rather, the assignment was that, in groups, the students should edit (and in a couple of cases create) Wikipedia articles on the texts and authors that we were covering, and that over the course of the semester they should bring these articles up to what in Wikipedia parlance is called "featured article" status. (para. 5-6)
Of the 12 articles written by students, three achieved Featured Article status and eight achieved Good Article status.
Randall, Johnson, West, and Wiley (2013) provide another example of a renewable assignment in their discussion of an openly licensed textbook on the topic of project management written for use in a business school context. A faculty member teaching project management in a graduate program on educational technology adopted the textbook understanding that much of the context and examples provided in the book would be inappropriate for his students. Over several semesters, he assigned students to work with him to collaboratively rewrite the book, adding examples relevant to educational technology, integrating new video case studies they produced, and making other changes that further improved the book for educational technology students.
Azzam et al. (2017) describe a renewable assignment in which fourth year medical students improve the quality of health-related content in Wikipedia. They describe 43 students engaging in research on specific medical topics and making 1528 edits to 43 Wikipedia articles about high interest health topics like diabetes, dementia, and stroke. Collectively, these articles have been viewed over 22 million times since they were improved by the students. Assam et al. demonstrate how students can be assigned work which both demonstrates their mastery and provides lasting benefit to the community.
In addition to questions about sustainability, faculty often worry about the quality of OER. According to Allen and Seaman (2016), 28% of faculty list "Not high-quality" as one of the three most important barriers to their use of OER in courses they teach (though "quality" is never defined). The concern is greater (32%) among faculty who describe themselves as being aware of OER than among those who are not aware (19%). This is somewhat surprising given the empirical work on the impacts of OER adoption on student success. For sake of clarity, for the remainder of this article we limit the meaning of the "quality" of educational resources to their effectiveness in supporting learning.
Hilton (2016) reviews nine empirical studies published in peer reviewed journals that focus on the effectiveness of OER assigned in place of commercial textbooks as the primary resource for courses. After acknowledging that assigning OER in place of commercial materials saves students a significant amount of money, he writes:
In total 46,149 students have participated in studies relating to the influence of OER on learning outcomes. Only one of the nine studies on OER efficacy showed that the use of OER was connected with lower learning outcomes in more instances than it was with positive outcomes, and even this study showed that the majority of the classes were non-significant differences. Three had results that significantly favored OER, three showed no significant difference and two did not discuss the statistical significance of their results. In synthesizing these nine OER efficacy studies, an emerging finding is that utilizing OER does not appear to decrease student learning. (para. 55)
While remaining silent on the issue of whether OER use is associated with better learning outcomes, Hilton concludes that OER use is not associated with decreases in student learning. This is an incredibly important finding as it means that faculty who adopt OER in place of traditional commercial materials can expect to save their students a significant amount of money without harming student learning.
But what about student-created OER? It appears from the above examples that it is possible for students to participate in the sustainable creation and maintenance of OER, but none investigate the impact on student learning when students use OER created by other students. Hilton demonstrates that using OER in place of commercial materials does not harm student learning, but none of the OER used in the studies he reviews were created by students. Consequently, we do not know what happens when these two strands are brought together. How effective are student-created OER in supporting learning by later students? If student-created OER are unable to effectively support learning by other students, they are of little value even if they can be sustainably created and maintained.
While the examples of renewable assignments above are powerful, they describe the work of upper-level undergraduate and graduate students who may not be accurately described as novices. To better understand the question of the learning impacts of using OER produced by other students we identified a setting in which the student creators would be more likely to be true novices – OER created by high school students for other high school students.
Mountain Heights Academy opened in 2009 under the name Open High School of Utah as a virtual charter school with a mission to serve students in 9-12 grades from across the state of Utah (Open High School of Utah, 2009). In 2013, the school added 7th and 8th grades and changed its name to Mountain Heights Academy. The school currently serves 550 full-time students and 200 part-time students in grades 7-12. Each student receives a laptop from the school and access to online courses hosted in the Moodle LMS. Full-time teachers are available both asynchronously and synchronously for a total of four hours per day during their office hours to work with students individually or in small groups (Mountain Heights Academy, 2014). The entire curriculum is built using open educational resources, and the resulting courses are publicly released for universal access worldwide. Students are invited to contribute their own OER to the content of their courses if they choose to do so. Mountain Heights Academy thus provides a rich setting in which to study the impact of student-created OER on student learning.
The Digital Photography course was created and released for the 2010-2011 school year and has been taught each subsequent year. This course is a basic introduction for students in the field of digital photography. The course includes units on camera functions, lenses, image editing and output, photography genres and art history, and composition. After completing the course, students should be able to use artistic images to communicate ideas and themes make educated decisions in purchasing photography equipment with the functionality they desire.
The course is a semester course organized into two quarters. In the State of Utah, students are able to take the course for either Fine Arts or Career Technology Education credit. At the beginning of each term (quarter), there is a section titled "Navigating the Course" that contains tutorials and guides for students, highlighting key skills instructors have identified as necessary for student success in the class. Each quarter of the course is structured into nine weekly modules. Lesson and assessment pieces for each module are due on the Friday of each assigned week. Students are given multiple summative and formative assessment opportunities in the course to demonstrate mastery of concepts, including portfolio creation, essays, multiple choice quizzes, self-evaluation, journaling, and teacher observation. The weekly modules cover at least one major concept and for each concept, there are specific learning objectives and linked assessments. On average, a weekly module contains seven pages, one assignment, and one quiz.
Three opportunities are available for students to add OER to the course each year. First, students are encouraged with extra credit to release their own photos with a Creative Commons license. This involves creating their own account on a photo sharing site like Flickr, uploading their photos, and applying the Creative Commons license. These photos are then evaluated and the strongest photos of a particular concept are selected to be added to the course as examples. Many students have taken advantage of this opportunity since the course was first offered in 2011-2012. Second, at certain points in the course where students have struggled in the past, all students are given the opportunity to raise their grade if they create a tutorial video for a particular assignment. These tutorial videos are evaluated and a few of these are selected to be placed into the course. Approximately five students each year take advantage of this additional opportunity.
Finally, after demonstrating high levels of mastery, strong students are offered the opportunity to be a teaching assistant for the upcoming semester. These students create notes for each unit, study guides for exams, tutorial videos, and review presentations and games that are all added to the course and released as OER. For all materials created by teaching assistants the instructor thoroughly vets the material, edits for formatting and quality, and then implements the OER in the course. As future teaching assistants work with the instructor, they revise and remix material that was created by prior teaching assistants.
When Digital Photography was first taught in 2011-2012 it contained no student-created OER. During the 2012-2013 school year the first collection of student-created OER was added, and additional student-created OER were added again in 2013-2014 and 2014-2015. By 2014-2015, between 5% and 10% of all course content was comprised of student-created OER.
During the 2011-2015 period the instructor, assignments, and grading criteria all remained stable. The only significant changes made to the course were the additions of student-created OER. This setting provides a compelling opportunity to ask questions and draw preliminary conclusions about the impact of student-created OER.
We examined student performance data from a cross section of class assignments including understanding basic photography concepts, applying photography processes to students' own compositions, synthesizing prior knowledge to create new work via tutorials and study of art history, and analyzing and evaluating students' own and others' work to give constructive feedback. We then ran a t-test for each assignment to determine if the average grade for the assignment had changed significantly over the period. The average grades and for each assignment are presented in Table 1.
Table 1
Average Assignment Grades and T Test Values
| 2011-2012 | 2014-2015 | t | p | ||
| Filters | 0.591 | 0.766 | -2.617 | 0.010 | * |
| Popping | 0.517 | 0.686 | -2.387 | 0.018 | * |
| Lenses | 0.554 | 0.706 | -2.511 | 0.013 | * |
| Shutter Speed and ISO | 0.592 | 0.731 | -2.166 | 0.032 | * |
| Pixlr | 0.587 | 0.718 | -1.850 | 0.066 | |
| Cropping | 0.728 | 0.831 | -1.695 | 0.091 | |
| Light and Exposure | 0.587 | 0.586 | 0.004 | 0.997 | |
| Camera Modes | 0.715 | 0.714 | 0.015 | 0.988 | |
| Black and White | 0.746 | 0.733 | 0.209 | 0.834 | |
| Rules of Composition | 0.824 | 0.708 | 1.756 | 0.081 | |
| All Assignments | 0.638 | 0.719 | -3.829 | 0.000 | * |
As shown in Table 1, average grades rose a statistically significant amount for 4 of the 10 assignments. The average grade for all assignments increased by eight points from 2011-2012 to 2014-2015. We calculated a t-test to determine if the difference between the average course grades in 2011-2012 and 2014-2015 were statistically different from each other. The results indicate that the 2014-2015 scores are statistically significantly higher than the earlier scores (t = -3.829, p < 0.001).
The Digital Photography course was neither designed nor taught with the goals of providing a context for research on the sustainability of student-created OER or the impact of learning from student-created OER. Consequently, all of the conclusions described below must be considered preliminary and in need of additional research.
Students in the Digital Photography course were invited to create OER as an optional, extra credit activity. Very few students took advantage of this opportunity. This suggests that offers of extra credit alone may not be sufficient to incentivize a sustainable level of OER creation and maintenance. If the opportunity to create OER were integrated into required assignments as an option for students, as we saw in the Murray (2008), Randall et al. (2013), and Azzam et al. (2017) examples, we suspect a greater amount of OER would be created. The degree to which students will create and contribute OER in extra credit settings versus required assignment setting merits additional study.
While the overall trend of student grades increasing in a statistically significantly manner as additional student-created OER were added to the course and other course elements remained unchanged is encouraging, additional research is needed before broad claims regarding the degree to which student-created OER supports better student learning can be made. Given that student-created OER made up only five to ten percent of the course content by the end of the fourth year, our result is particularly surprising.
Future studies addressing similar questions may benefit by framing their research as dosage studies that look for relationships between the amount of student-created OER present in the course and degrees of difference in student performance across dosages. It seems unlikely that student performance will continue to increase as more student-created OER are added to the course. For example, while we found that when 5% to 10% of course content was comprised of OER, students experienced an 8-point improvement in their overall assignment score, we do not necessarily believe that if 15% to 20% of the course had been comprised of student-created OER we would have seen a 16-point improvement in student grades. The relationship between the amount of student-created OER added to a course and the resulting changes in student performance needs much more investigation.
Allen, E., & Seaman, J. (2016). Opening the textbook: Educational resources in U.S. higher education, 2015-16. Babson Survey Research Group. Retrieved from http://www.onlinelearningsurvey.com/reports/openingthetextbook2016.pdf
Azzam, A., Bresler, D., Leon, A., Maggio, L., Whitaker, E., Heilman, J....McCue, J. D. (2017). Why medical schools should embrace Wikipedia: Final-year medical student contributions to Wikipedia articles for academic credit at one school. Academic Medicine, 92(2). Pp. 194–200. doi: 10.1097/ACM.0000000000001381
Department of Education. (2016). Table 304.10: Total fall enrollment in degree-granting postsecondary institutions, by state or jurisdiction: Selected years, 1970 through 2014. Retrieved from http://nces.ed.gov/programs/digest/d15/tables/dt15_304.10.asp?current=yes
Dholakia, U., King, J., & Baraniuk, R. (2006). What makes an open education program sustainable? The case of connextion. Organisation for Economic Cooperation and Development. Retrieved from https://www.oecd.org/edu/ceri/36781781.pdf
Downes, S. (2006). Models for sustainable open educational resources. Organisation for Economic Cooperation and Development. Retrieved from https://www.oecd.org/edu/ceri/36781698.pdf
Hewlett. (2016). Open educational resources [Web blog]. Retrieved from http://www.hewlett.org/programs/education/open-educational-resources
Hilton, J., III. (2016). Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development, 64(4). pp 573–590. doi: 10.1007/s11423-016-9434-9
Open High School of Utah. (2009). Mountain heights academy: Charter application. Retrieved from http://www.mountainheightsacademy.org/wp-content/uploads/2014/08/2013.02-Charter-Application-MHA.pdf
Mountain Heights Academy. (2014). Mountain Heights Academy: Annual report – 2013-2014. West Jordan: UT: 2014. Retrieved from http://www.mountainheightsacademy.org/wp-content/uploads/2015/01/Annual-Report- 2013-2014-FINAL.pdf
Murray, J. B. (2008). Was introducing Wikipedia to the classroom an act of madness leading only to mayhem if not murder? [Web blog]. Retrieved 1 May, 2017 from https://en.wikipedia.org/wiki/User:Jbmurray/Madness
Parry, M. (2009, September 3). Utah State U.'s OpenCourseWare closes because of budget woes [Web blog]. The Chronicle of Higher Education. Retrieved from http://www.chronicle.com/blogs/wiredcampus/utah-state-us-opencourseware-closes-because-of-budget-woes/7913
Randall, D. L., Johnson, J. C., West, R.E., & Wiley, D. (2013). Teaching, doing and sharing project management: The development of an instructional design project management textbook. Educational Technology, 53(6), 24-28.
Sanger, L. (2005). The early history of Nupedia and Wikipedia: A memoir [Web blog]. Retrieved from https://features.slashdot.org/story/05/04/18/164213/the-early-history-of-nupedia-and-wikipedia-a-memoir
Shirky, C. (2010). Cognitive surplus: How technology makes consumers into collaborators. New York, NY. Penguin.
Wiley, D. (2006). On the sustainability of open educational resource initiatives in higher education. Organization for Economic Cooperation and Development. Retrieved from http://www.oecd.org/dataoecd/33/9/38645447.pdf
Wiley, D. (2013). What is open pedagogy? [Web blog]. Retrieved from https://opencontent.org/blog/archives/2975
A Preliminary Exploration of the Relationships Between Student-Created OER, Sustainability, and Students Success by David Wiley, Ashley Webb, Sarah Weston, and DeLaina Tonks is licensed under a Creative Commons Attribution 4.0 International License.