Identifying Student Perceptions of Different Instantiations of Open Pedagogy

As the adoption of open educational resources (OER) continues to increase, instructors have started using these resources for more than simply delivering content. Open pedagogy is a term used to describe a range of instructional practices that often incorporate OER into the learning process. This study examined student perceptions of two approaches to open pedagogy—student creation of multiple-choice questions and student creation of the syllabus and corresponding course assignments. The sample included responses from 84 students at two colleges in the United States. Results showed that students who created the syllabus and assignments had a more positive experience and were more likely to enroll in a future course that implements this strategy. Those in the multiple-choice course felt that the approach was less conducive to learning than traditional learning activities. The significant differences in student feedback on two different approaches, both of which could be termed open pedagogy, indicate that more research is needed to examine the efficacy of the wide variety of approaches to open pedagogy. Moreover, the perceived efficacy of one instantiation of open pedagogy does not equal the effectiveness of open pedagogy, broadly defined.


Introduction
Open pedagogy is an increasingly popular choice for instructors seeking to motivate students toward deeper levels of learning, synthesis, networking, and collaboration. Instructors employing open pedagogy often use Open Educational Resources (OER) to facilitate student-directed education. Different instantiations of open pedagogy occur when instructors use OER for different elements of course material (Wiley & Hilton, 2018). For example, Jhangiani (2017) uses OER by assigning students the task of creating multiple-choice questions. DeRosa and Robison (2017) use OER by assigning students the task of editing Wikipedia articles.

Review of Literature
Open education research highlights "the tendency for 'open' to encompass many different interpretations and the capacity for the field to evolve accordingly" (Cronin & MacLaren, 2018, p. 135) (Cronin & MacLaren, 2018). We note that the term open pedagogy itself has shifted over time.
Earlier definitions of open pedagogy describe a classroom setting where the teacher facilitates informal discussions, thereby assisting students with co-creating the context of the class (Elliott, 1973). Mai (1978) viewed open pedagogy as an "informal classroom where children might be trusted to learn by exploring according to their own interests, instead of being bored, demeaned, and alienated" (p. 231).
In more recent years, the focus of open pedagogy has shifted towards student-centered technological approaches that emphasize collaboration outside the classroom (Hegarty, 2015;Hodgkinson-Williams & Gray, 2009;Mackintosh, McGreal, & Taylor, 2011). Weller (2013) (Cronin & MacLaren, 2018;Bali, 2017 (2017) as a platform for "learning, collaboration and engagement with the world outside the classroom" (p. 118). We note that some would likely disagree with this definition as being too expansive, while others might find it too narrow. This highlights one of the challenges of studying open pedagogy. We investigated two specific instantiations of open pedagogy by examining the perceptions of students who were either (a) in a class where they created multiple-choice questions, or (b) in a class in which they co-created the syllabus and assignments. Specifically, our research questions were as follows: 1. How do students perceive the educational value of creating multiple-choice questions relative to traditional teaching approaches? 2. How do students perceive the educational value of creating the syllabus and assignments relative to traditional teaching approaches?
3. What differences (if any) are found in student perceptions of these two approaches to open pedagogy?

Method
Our dataset includes two instructors teaching in colleges in the United States who used open pedagogy during the spring of 2018. These instructors had been part of a seminar in open pedagogy and agreed to share their experiences in using it. Although their approaches differed significantly, both faculty members believed that they were implementing a form of open pedagogy. At the end of the semester, students in both classes were given a Qualtrics survey regarding their experiences with the open pedagogy activities in the course (Appendix). Quantitative data were reported through descriptive statistics, and the data analysis software SPSS was used to conduct chi-square tests to make cross-group comparisons on student perceptions of open pedagogy. Instructor data were collected through one-on-one interviews and e-mail correspondence. All research activities were cleared through an institutional review board.

Context
One of the instructors adopted an open textbook and used supplemental materials that she had created for the course. Her purpose in using OER was not only to save money, but more specifically to start conversations with class members about how their work might extend outside their classroom. In connection with her OER adoption, she had students create their own multiple-choice questions in lieu of doing the more traditional assignments she had used in the past. Her purpose was to have students do a renewable assignment, a renewable assignment being one that could be used in future courses to increase student learning (Hendricks, 2015;Wiley, 2013). In this case, the theory was that openly-licensed questions created by current students could benefit future students. This instructor stated that implementing her version of open pedagogy was an opportunity for her to learn new pedagogical strategies and improve her instruction. Throughout this paper, this course will be referred to as the Create MC Course (N=43).
The second instructor adopted an existing open textbook that she and other faculty contributors edited. Her implementation of open pedagogy was centered around students in the class collectively creating a course syllabus for the class (within certain parameters). Students then chose their own projects for the course that had to extend outside of the classroom in some way. This was considered open pedagogy in that it put significantly more control than normal into the hands of the students who were, in a sense, co-creators of the course. Throughout this paper, this course will be referred to as the Create Assignments Course (N=41).

Results
In this section, we first examine the student perceptions of the pedagogy in the Create MC Course.
Subsequently, we do the same for students in the Create Assignments Course. Finally, we compare the results of the two courses.

Student Perceptions of Creating Multiple-Choice Questions
Students in the Create MC Course were asked, "Was the educational value of creating multiple-choice questions better, worse, or the same as that of traditional learning activities (e.g., writing papers, taking quizzes, etc.)?" Seven students (16%) said creating multiple-choice questions was better than traditional learning activities, with eighteen (42%) saying it was the same, and eighteen (42%) stating it was worse.

Students provided responses explaining their beliefs about the educational value of creating multiple-choice
questions.
An analysis of factors prompting students to mark better centered on themes of increased understanding and depth of thought. For example, student comments included: "I had to understand what the topic is," "[I] made application of concepts more in depth," and " [It] made me think about the problem more." Students who felt creating multiple-choice questions had less educational value focused on a feeling that the assignment did not help them effectively learn. For example, one student stated, "We were … required to regurgitate somewhat ultimately arbitrary information which didn't necessarily require critical thinking on the whole." Another student wrote, "I still had to read the entire chapter if I wanted to do good on the multiple-choice questions, therefore I still got the same information out of doing the multiple-choice questions as I would if we were to take quizzes." Students in the Create MC Course were asked the following question: "Imagine a future course you are required to take. If two different sections of this course were offered by the same instructor during equally desirable time slots, but one section had traditional learning activities (such as writing papers and taking tests), and the other used learning activities like creating multiple-choice questions, in which section would you prefer to enroll?" Students were more negative than positive about their desire to enroll in future courses using this type of open pedagogy activity. Eighteen students (42%) said they would enroll in a course with traditional learning activities, twelve (28%) said they would enroll in a course with activities like creating multiple-choice questions, and thirteen (30%) expressed no preference.
Students selecting "I would enroll in the section with traditional learning activities" focused on themes of consistency, reliability, learning better, and earning better grades. For example, one student stated, "I learn better in those types of classes." Another student wrote, "It's what I'm used to and [excel] in." It is interesting to note that students who said they would enroll in the sections with activities like creating multiple-choice questions also included themes of learning better and earning better grades, as well as more in-depth learning. One student stated, "It keeps you reading and keeps you understanding the content," and another wrote, "You are able to understand a concept more when you have to create questions, you have to really wrap your head around the ideas." Students were asked to compare five different aspects of the educational efficacy of creating multiple-choice questions with traditional learning activities. Table 1 summarizes their responses. Note. N = 43.
Ten students provided a free response comment on how this approach to open pedagogy helped them master core academic content more than traditional learning activities. One student wrote, "I learned the information better by having to actually think of a question to ask," and another said, "Until I could understand the topic well, I couldn't create multiple questions." In contrast, eleven students wrote about why they had lower concept mastery, focusing on a lack of learning. Two representative quotations are as follows: "I felt I was too worried about the formation of a question that I never really gave the material much thought," and "I was unable to grasp all aspects of the readings, focusing on the graded aspect of my question formatting." When asked to explain why the activities helped them become more or less collaborative learners than in traditional learning activities, the most consistent response was that they collaborated less. Five students bluntly made statements such as, "There was nothing collaborative about [this assignment]." Students who felt writing multiple-choice questions helped them learn to think critically or solve complex problems more than traditional learning activities wrote about depth of learning and application of knowledge. For example, one student wrote, "It made me have to apply knowledge and think more in depth about the topic." Two students wrote about why this pedagogy was less effective for critical thinking. One wrote, "I didn't think critically or solve any complex problems." When asked to explain why creating multiple-choice questions helped them learn to communicate more effectively than traditional learning activities, one student wrote, "Framing thoughts, organizing material." Two students provided statements about why the activities were less useful in learning to communicate effectively. One wrote, "I couldn't really communicate anything, I was just writing questions." With respect to learning how to learn, three students provided statements about why their outcome was more positive, centering on a theme of understanding. For example, one student wrote, "I had to understand the questions/material itself before actually creating the questions." Two students provided statements about why their outcome was less positive. The students' responses centered on a lack of focus.
For example, one student wrote, "I wasn't able to focus on understanding the material as a whole. Instead I had to focus on the section I was assigned and try to make a question, often out of the most trivial aspects of its subject material, leaving me with a very loose understanding of the material."

Student Perceptions of Creating the Syllabus and Assignments
As with the students in the course with multiple-choice questions, students in the class that created the syllabus and assignments were asked, "Was the educational value of creating the syllabus and assignments better, worse, or the same as that of traditional learning activities (e.g., writing papers, taking quizzes, etc.)?" Twenty-seven students (64%) said creating the syllabus and assignments was better, with eleven (26%) saying it was the same, and four (10%) stating it was worse.
The students who felt that creating a syllabus and assignments was better than traditional learning activities mentioned several factors including ownership over learning, self-pacing the course, increased engagement, development of skills such as communication and independence, the applicability of concepts, and the feeling that students were doing "less busy work" than they would have in a traditional course. The general sentiment was summarized by one student who said that the course "allowed [the students] to really take control of how [they] wanted the class to be run, and set [them] up for great success in the class." The students who felt that creating a syllabus and assignment was the same as traditional learning activities felt that the information itself didn't change in a significant way. For example, one student wrote, "We ended up getting the same information, we were just given it in a different way." Students who felt creating a syllabus and assignment was worse than traditional learning activities focused on factors such as lack of structure and priorities. For example, one student wrote, "Nothing is set in stone and I am unaware of what to prioritize and what the goals of the class are." This indicates that at least some students felt that a drawback of this form of open pedagogy was a lack of desired structure.
In response to the question regarding student preference in taking a future course using activities similar to creating a syllabus or assignments versus traditional activities, the majority of students in the Create Assignments Course were willing to enroll in the section using this style of open pedagogy activity. Twentyfour students (60%) said they would enroll in a course with activities like creating a syllabus and assignments, seven (18%) said they would enroll in a course with traditional learning activities, and nine (23%) expressed no preference (total does not equal 100% due to rounding).
When asked why they would choose to enroll in a class with this type of open pedagogy activity, sixteen students responded. Their statements centered on themes of engagement, freedom, control, and ease of learning. Student comments included: "[This pedagogy] allows more freedom for the student which I like a lot more than the traditional style," "It gives me an option to structure the class how I would want to learn and be able to understand the information better," and "It's an easier way to learn and a way to choose what I want to work on." When asked why they chose traditional activities, seven students provided responses centered on themes of preferring current norms and habits of learning. A representative comment is as follows: "I am used to [a structured class] and I feel in a course that connects to my major I can succeed more if I study and find the initiative to do good on a test rather than figure everything out on my own and be independent in my research." Table 2 summarizes student responses to five different aspects of the educational efficacy of creating a syllabus and assignments when compared with traditional learning activities. something." One student noted, "In a typical classroom, most students learn the information just so that they can do well on the test," whereas in this course, the student felt that the new knowledge was adding to a larger body of knowledge and skills.
Nineteen students explained how creating a syllabus and their own assignments made them a more collaborative learner; no students provided a response stating that it did not. Students felt that working with others on their course and assignments encouraged them to develop skills related to collaborative work and that doing so increased accountability to themselves and those around them. They also felt that it not only increased collaboration with students but between the professor and students as well. One student stated that they "work [ed] in groups a lot more than in a traditional class." Eighteen students wrote about how the open pedagogy helped them learn to think critically or solve complex problems more than traditional learning activities. These responses discussed engagement, collaboration, and imagination. For example, one student wrote, "Engaging with students made me think more about my opinion and other students." Another student stated, "I had to come up with ways to solve real problems." When asked to explain why creating a syllabus and learning activities helped them communicate more effectively, thirteen students wrote about how communication was vital to succeeding on the assignment.
For example, one student wrote, "I had to communicate with group members, and without effective communication our product would have been crap." With respect to how students perceived the open pedagogy helping them learn how to learn more effectively, twelve students focused on themes such as involvement, responsibility, independence, and a new style of thinking. One student wrote, "Creating a syllabus makes you responsible and forces you to be an independent learner." Another said, "It helped me think from a different angle and to look outside the box when trying to find an answer." Three students provided statements about why this approach to open pedagogy helped them learn less than traditional teaching activities. These students were concerned about the effectiveness of group participation. As one representative comment stated, "We chose how we learned which meant people could slack off in their work."

Comparing Student Perceptions of Two Instantiations of Open Pedagogy
There were strong and significant differences in how students perceived these two approaches to open pedagogy. Likewise, there were significant differences in each of the five questions asking students to compare aspects of the learning with open pedagogy versus traditional activities, as described in Table 3.

Discussion
The results indicate that a strong minority of students (42%) felt that creating multiple-choice questions had less educational value than traditional learning activities. Only 16% of those who created multiplechoice questions felt this approach had more educational value than traditional learning activities. In contrast, students who engaged in creating their own syllabus and learning activities were generally (although not universally) positive, with nearly two-thirds reporting that it was better than traditional learning activities and an additional one-fourth stating that they had the same value.
There were statistically significant differences in how students viewed these two open pedagogy activities.
Students consistently rated creating multiple-choice questions as being less conducive to learning, and stated that they would be less likely to choose a class with this approach to open pedagogy than the one focused on students creating their own syllabus and activities.
These data cannot be used to conclusively declare that creating multiple-choice questions is a poor form of open pedagogy or that students creating their own syllabus and/or learning activities is a successful one.
There are several factors that we were not able to measure as part of this study that could have had significant bearing on how students perceived the specific learning activities. However, the significant

Limitations
There are several limitations in this exploratory study, and our results must be understood in that light.
First, there was no attempt to control for instructor variables. The instructor who assigned students to create multiple-choice questions was attempting this approach for the first time and would likely approach it differently were she to use this technique again in the future. In contrast, the instructor who assigned students to create their own syllabus and assignments had used this pedagogy frequently in the past. Thus, what was measured in the present study may not be the efficacy of the strategy itself but, rather, the difficulty of implementing a new pedagogical approach. Similarly, it is possible that the instructor who had students create their own syllabus and assignments may have done so in a masterful way that could only be easily accomplished by a minority of faculty. Replication by a variety of instructors in different settings, as well as studying individual teachers over successive semesters would be necessary before generalizing the results of the present study.

12
A second weakness is that no attempt was made to control for student differences. Nearly all the students