Writing to Learn and Learning to Write across the Disciplines: Peer-to-Peer Writing in Introductory-Level MOOCs

This study aimed to evaluate how peer-to-peer interactions through writing impact student learning in introductory-level massive open online courses (MOOCs) across disciplines. This article presents the results of a qualitative coding analysis of peer-topeer interactions in two introductory level MOOCs: English Composition I: Achieving Expertise and Introduction to Chemistry. Results indicate that peer-to-peer interactions in writing through the forums and through peer assessment enhance learner understanding, link to course learning objectives, and generally contribute positively to the learning environment. Moreover, because forum interactions and peer review occur in written form, our research contributes to open distance learning (ODL) scholarship by highlighting the importance of writing to learn as a significant pedagogical practice that should be encouraged more in MOOCs across disciplines.


Introduction
Massive open online courses (MOOCs) could be poised to transform access to higher education for millions of people worldwide (Waldrop, 2013). From a pedagogical standpoint, the sheer scale of these courses limits the extent of student-instructor interpersonal contact, and this leads to a central question involving how a reliance on peer interaction and review impacts student learning. Student-student interaction, once called "the neglected variable in education" (Johnson, 1981), is now recognized as a fundamental high-impact practice in education (Salomon & Perkins, 1998;Chi, 2009).
Clearly humans interact via multiple modes, including in person, but also more and more frequently via long-distance digital communication such as by telephone, email, social media websites, online chats and forums, video conferencing, and blogs; all of these modes are emerging in modern pedagogies. To this end, deWaard et al. cite "dialogue as a core feature of learning in any world, whether face-to-face or digital" (deWaard et al., 2014). In fact, in face-to-face and smaller-scale online learning contexts, peer-to-peer dialogues have been shown to be critical to developing deep conceptual understanding (Chi, 2009). In MOOCs, peer-to-peer dialogues occur primarily through writing: in forums and via peer-reviewed assignments.
MOOCs, because of their scale, offer a significant opportunity for peer-to-peer interaction in the form of dialogic, networked learning experiences (Clarà & Barberà, 2013). However, also because of their scale and the diversity of student learners enrolled, MOOCs present substantial challenges in this domain (Kim, 2012). Some scholars have suggested that MOOCs limit or underestimate the importance of interpersonal engagement for learning (Kolowich, 2011;Kim, 2012;Pienta, 2013).
Questions about how or whether to facilitate interpersonal engagement in MOOCs have particular importance since the majority of MOOC learners are adults (Guo & Reinecke, 2014). Research maintains that constructivist approaches to learning are especially effective with adult learners (Huang, 2002;Ruey, 2010). It is within this context that we endeavor to examine one of the key questions concerning the efficacy of MOOCs: How can interactive learning be promoted and assessed in this context? Any exploration of this question, though, also demands an inquiry into writing. The primary mechanisms for student interaction in MOOCs occur through writing in course forums and peer reviewed assignments. 1 The act of writing has been identified as a highimpact learning tool across disciplines (Kuh, 2008), and efficacy in writing has been shown to aid in access to higher education and retention (Crosling, Thomas, & Heagney, 2007). Writing has also been shown to be effective in the promotion of learning and student success in relatively large enrollment face-to-face courses (Cooper, 1993;Rivard, 1994;Reynolds et al., 2012). Research suggests that writing instruction in online settings can provide enhanced learning experiences and opportunities for pedagogical 1 In some cases dialogic interaction occurs verbally through video chats in such platforms as Google Hangouts.
Introduction to Chemistry (January 20, 2014-April 6, 2014, taught by Dorian Canelas through Duke University and Coursera. Although at first glance these courses might seem unrelated, common threads weave them together into a research project: both specifically target introductory students; focus on critical thinking and writing-to-learn to develop expertise; foster key skills for access to fields in higher education; and employ a combination of video lectures and quizzes along with formal writing assignments and informal written communication via forums. We specifically chose to conduct research across disciplines because we wanted to contribute to emerging MOOC literature that examines how disciplinarity impacts MOOC pedagogy and learning outcomes dimensions (Adamopoulos, 2013;Cain, 2014).
The main objective of this study was to evaluate how peer-to-peer interactions through writing impact student learning in introductory-level MOOCs across disciplines.
Specifically, we explored the following research questions: • How do peer-to-peer interactions through writing impact student learning in introductory-level writing and chemistry MOOCs?
• What is the impact of peer-to-peer writing on engaging students in MOOC coursework who identify as less academically-prepared and less self-motivated?
• How can peer-to-peer writing function as a metric to assess student success in MOOC delivered introductory writing and science coursework? 2 English Composition was funded largely through a grant from the Bill & Melinda Gates Foundation. Writing-to-learn is a pedagogy that actively involves students across disciplines in the construction of their own knowledge through writing (Sorcinelli & Elbow, 1997;Carter, 2007). Peer review makes this process not only active but interactive. Student-student and student-faculty dialogues have been shown to be critical to developing a community of scholarship for enhanced learning and deep conceptual understanding among learners (Chi, 2009;Johnson, 1981). The capabilities of MOOCs make it possible to bring this active-constructive-interactive framework (Chi, 2009) to thousands of students at one time. Indeed, emerging research suggests that MOOCs have the capacity to create unique "networked learning experiences" with unprecedented opportunities for collaboration, interaction, and resource exchange in a community of learners (Kop, Fournier, & Mak, 2011;Siemens, 2005). And, also in keeping with these findings, research has found evidence that the most successful MOOC students are typically heavy forum users (Breslow, 2013).
Given that MOOCs promise to increase access to postsecondary education (Yuan & Powell, 2013), we are particularly interested in how peer-to-peer interactions through writing in introductory-level MOOCs impact the learning outcomes for less academically advanced and/or under-resourced learners. Although research has also indicated that MOOCs are not yet reaching less academically prepared students (Emanuel, 2013), we endeavor to learn how less academically prepared students can best learn in these introductory-level MOOCs. Research suggests that less well-prepared students can behave more passively in academic settings, relying on memorization and imitation as strategies for learning (Mammino, 2011). This has been shown to arise at least partly from lack of comfort with the use of language, particularly if trying to communicate in a non-native language (Mammino, 2011). Research in developmental writing suggests that early emphasis on writing in a student's academic career can improve retention and academic performance (Crews & Aragon, 2004).
Learning more about how peer-to-peer interactions through writing impacts retention and academic performance is especially critical in the context of STEM. Research suggests that the greatest loss of student interest in STEM coursework occurs during the first year of college study (Daempfle, 2004). Scholarship has found that peer interactions in introductory-level science courses, especially through writing, have in some contexts doubled student retention rates in STEM disciplines (Watkins & Mazur, 2013). Writing to learn has been used extensively in chemistry and other science disciplines and has been shown to help students confront and resolve several key barriers to and misconceptions about effective science learning (Pelaez, 2002 to improve student performance, communication, and satisfaction even in large enrollment undergraduate chemistry courses (Cooper, 1993

Coding Protocol
Prior to the workshop, we developed a coding protocol, with the assistance of a doctoral student and postdoctoral scholar in developmental psychology. The protocol included nodes for such items as affect, length of post, attitude, learning objectives, student challenges, and elements of writing (for full coding protocol, see Appendix A).

Coding Workshop
During the workshop, coders were first led through processes to become familiar with the structure of MOOCs in general, and our study MOOCs in particular, and with important NVivo components, including data capture, import, and coding to themes.
Second, coders were introduced to the pre-designed project file and node structure, and leaders oversaw use of the protocols by team members. After this step, based on coding of the same data sources by team members, leaders examined inter-rater reliability and made adjustments to the team's work. Third, coders worked individually on various data sources as assigned. Twice a day, time was taken for team discussion, and leaders were present at all times during the coding workshop to answer individual questions.

Coding Reliability
When introducing the node structure, the coding workshop leader walked all coders through each node and its planned use. Subsequently, teams of three coders coded the same two documents to the node structure. This led to assessment of inter-rater reliability and team discussion. At several points node definitions and/or structure were discussed as a whole group.  Maxwell (2005) refers to as "organizational" or "substantive" nodes, which are more topical, descriptive, or straightforward to interpret than "theoretical" coding (which is more abstract) (p. 97). Because most of the nodes have literal definitions, with a few exceptions, to which we paid close attention, we believe little room existed for coders to differ substantially from each other on inferential coding of text.
Reliability was also established through having coders collect and evaluate different types of data from different disciplines throughout the coding workshop (for a summary of items coded, please see Table 4). This form of triangulation, called "investigator triangulation" (Denzin, 2009, p. 306), involves multiple researchers in the data collection and analysis process, and the "difference between researchers can be used as a method for promoting better understanding" (Armstrong et al., 1997, p. 597).
Finally, we spent the second half of Day Five of the coding workshop having coders review nodes in the merged file for coding inconsistencies. Various coders were given a family of nodes to open and review, using NVivo queries to consider the consistency of coding that they found.
We coded data from two different areas of the MOOCs: discussion forums and peer assessments.

Discussion Forum Data
The following data (Tables 1 and 2) provide a sense of the total discussion forum volume for these courses, from which we culled our sample. Please see Appendix A for more definitions and descriptive details.  From this total volume, we coded a sampling of two types of discussion forum data.

Point in time (PIT) and general peer-to-peer (P2P) discussion forum posts.
We coded 35 full discussion forum threads in Weeks One, Four, and Seven of both courses. In addition, we coded 35 full threads from Week 12 for English Composition (the Chemistry course was not 12 weeks long.) We also coded general forum posts for both courses.
Top poster P2P discussion forum posts.
We captured all activities of the top three posters in each of the two courses and coded a sample of these posts. (See Table 3 for top three posters in each course and statistics.) In addition to coding data from discussion forums, we also coded data from peer assessments in each course. Students provided feedback on other students' writing for both courses. We did not code the assignments themselves.

Peer assessment sources in Chemistry.
In Chemistry, this feedback was located in a specially designated open forum for peer review of a writing assignment. Students in the Chemistry class submitted an essay on a chemistry-related topic of their choice to the peer-review tool (see Appendix B).
Coursera then randomly assigned each submission to be read and commented upon by two peers according to a rubric (see Appendix C). After the first student had reviewed a peer's essay by entering their feedback as written comments, Coursera automatically populated a designated forum with the essay and anonymous peer review, whereupon any additional anonymous peer reviews would appear over time and more students could read and comment on each essay. Seven hundred and fourteen students submitted this assignment and received peer feedback. We coded evaluations on 120 submissions (16.8 percent of the total volume of submissions), randomly selected by capturing every 6th submission with correlating feedback on the Chemistry peer assessment forum.

Peer assessment sources in English Composition.
We reviewed three different types of English Composition peer-assessment data.
1. Peer feedback on a brief introductory essay, "I Am A Writer," posted to a specially designated open forum (see Appendix D). This first introductory writing activity, designed to facilitate conversations about writing among the community of learners, was conducted on the forums as opposed to through the formal peer-assessment mechanism. Thus, students could voluntarily respond to as few or as many peers' submissions as they wanted. Approximately 8,000 students posted the "I Am A Writer" assignment. We chose to capture feedback on 80 peer responses, which amounts to feedback on about 1% of the submissions. This was roughly equivalent to taking the first submission from each page of posts on the designated "I am a Writer" forum for a random sample.
2. Peer feedback provided through the formal peer-assessment mechanism. For each of the four major writing projects in English Composition (see Appendix E), students submitted a draft and a revision to the formal peer-assessment mechanism in Coursera.
For each project, Coursera randomly distributed each student's draft submission to three peers in order to receive "formative feedback" according to a rubric. Then, each student's final version was randomly distributed to four other peers in order to receive "evaluative feedback" according to a rubric. Formative and evaluative peer feedback rubrics included a series of specific questions as well as several open-ended questions (see Appendix F). 3 We only coded data from questions that seemed relevant to peer-to-peer interaction, namely, • What did you like best about this essay?
• What did you learn about your writing/your own project based on responding to this writer's essay/project?
• What overall comments do you have for the writer as he or she moves on to project 2/project 3/project 4/beyond the course?
These peer-assessment submissions and feedback were private for students, and so in this case, as required by Duke University's Internal Review Board, the only student submissions evaluated were those from students who approved our use of these data.
Throughout the course, the students provided 14,682 separate project peer assessment feedbacks. Approximately 250 students gave permission for their work to be included in this research process. We coded a random sample of the feedback provided by 50 of these students, which amounted to 342 project peer-assessment feedbacks.
This data enabled us to look at feedback on a student-by-student basis (as opposed to assignment by assignment).

Comments about peer feedback written in final reflective essays. Students in English
Composition compiled portfolios of their writing at the end of the course. These portfolios consisted of drafts and revisions of each of the four major projects as well as a final reflective essay in which students made an argument about their progress toward the learning objectives of the course (see Appendix G). One thousand four hundred and fifteen students completed final reflective essays; approximately 250 students gave permission for their final reflective essays to be included in the research process. We coded comments about their experiences providing and receiving peer feedback in 48 of these final reflective essays. Table 4 shows the total number of items coded for each type of source. 3 These formative and evaluative rubrics were developed largely through a consultation with assessment expert Edward White.  Table 4 Number

Limitations
Our research included several limitations. A primary limitation is that not all enrolled students participated by posting in the forums, so any analysis of forum posts will only include data from those students who felt motivated to post. Additional limitations include the following: • Coders were calibrated through coding common text passages on the first day.
For the rest of the coding session each piece of data was coded by individuals.
• We estimated the number of threads by multiplying 25 times the number of pages. This may be a slight overestimate, because the last page in each forum would by definition have less than 25 threads.
• Within a thread, we did not manually count the number of posts, but took the statistic from Coursera. A very small number of posts were empty or deleted by the forum moderators as spam or containing inappropriate material. Therefore, our post count may be a slight overestimate.
• We did not review student assignment submissions. We only coded the student feedback. Therefore, our coding inferences may be limited by this constraint.
• We captured threads from the website for coding using the NCapture software from NVivo. We used the Explorer browser exclusively. However, we learned that, irregularly and unpredictably, NCapture drops a line of text at a page break in the pdf. We did not try to go back and recover these lost lines of text in our analysis.
• Although we captured the number of views of each thread, we recognize that simply by entering each thread to capture it (and thereby adding a view count), we are increasing the number of views of each of our threads. To minimize this, our procedure was to document the number of views on an individual thread before actually opening the thread. Some researcher thread views are included in the overall view counts for the courses.
• Although unlikely due to the enormous number of posts to the forums, and our limited sampling frame, we may have inadvertently coded the same post twice, because we coded to various points-in-time, and we also sampled posts from the top three posters in each discipline.

Results
After the coding was completed, we ran several queries through NVivo. Below are several of the most significant results.
Word Frequency Queries Figure 1 illustrates the 100 most common words in weekly forums in each course. The larger the word, the more commonly it appeared in the forum.
These results illustrate visually that students were staying on topic by primarily engaging in discussions that paralleled the weekly content of the courses. For example, in Chemistry, the syllabus has the following descriptions for content in weeks six and seven, respectively: Week 6 (Brinton et al., 2013) and these data qualify the conclusion that "small talk is a major source of information overload in the forums" (Brinton et al., 2013).
Week Chemistry English Composition  Table 5 7 shows post length in the forums. In general, Chemistry students' posts were shorter than those of the English Composition students. The Chemistry forum included a much higher percentage of posts that were coded as very short or short; over 90% of the posts fell into these two categories. On the other hand, English Composition forums also included many posts that were coded as very short or short (approximately 60%), but nearly 40% of the posts in this course were coded as medium or long. While Chemistry forums had about 2% of posts coded as long, English Composition had nearly 23% coded as long.

Attitude
Attitude is well established as being critically important to learning: "In order for student-student interaction to have constructive impact on learning, it must be characterized by acceptance, support, and liking" (Johnson, 1981, p. 9). Research indicates that learners' conceptions of and attitudes toward learning have a deep impact on the efficacy of online peer assessment and interactions (Yang & Tsai, 2010).
Every post, or part of a post, if warranted, was coded as either positive, negative, or neutral ( Table 6). Attitude of student writing in the forums was tracked as a function of time in the courses. Considering all coded weeks, the majority of content coded in student posts were neutral in attitude in both courses, and a relatively small percentage was coded as negative in both courses. The attitude expressed in student posts was generally more positive than negative in both courses: 2.8 times more positive than  The tenor of posts across all weekly forums was coded as slightly more positive in English Composition than in Chemistry (27% of all words coded in English weekly forums compared to 17% in Chemistry). Both Chemistry and English were coded as having roughly the same amount of negative comments (6% and 7% respectively). Note that we also endeavored to distinguish attitude from types of writing critique. One could have a positive attitude while providing constructive critique for writing improvements, for example. The greater degree of positivity than negativity in the forums suggests that the forums can provide a meaningful mechanism for establishing a 9 This table includes data from three Chemistry weekly discussion forums (weeks 1, 4, and 7) and four English Composition weekly discussion forums (weeks, 1, 4, 7, and 12). The Chemistry course was not 12 weeks long. 10 The sum of each column is 100%, meaning that each cell refers to the percentage of overall words coded to all weekly forums sampled. learning community that has the potential to enhance students' learning gains and course experience.

Affect and Emotion
Affective and emotional factors are known to play a role in the success of any pedagogical practice (Gannon & Davies, 2007). Research shows that affect impacts students' response to feedback on their writing (Zhang, 1995). Affect and emotions have also been shown to be particularly important in engagement in science-related activities, and this, in turn, has been suggested as a link to improving public science literacy (Lin, 2012;Falk, 2007). Since MOOCs may be considered a pathway to increasing public understanding of scientific information and development of broadbased efficacy in essential skills such as writing, we were interested in how affect and emotion emerged in the discussion forums.  We coded to distinguish between attitude, as an evaluation, and affect, as an expression of feeling or emotion. Of course, students often expressed both an attitude and an affect, and in those cases, we coded to both types of nodes, but text was only coded to affect when appropriate. For example, the first quote below would be coded both to negative attitude and to the affect/emotion of "frustration", whereas the second would be coded only to negative attitude. anyone taking this course is a writing "Einstein" (genius)." "Gratitude" and "encouragement" were in the top three of affects coded in discussion forum posts for both Chemistry and English Composition. Text coded to these affects ranged from simple phrases, such as "Thank you for your insights" or "I do think your efforts are praiseworthy," to lengthier: "Do not give up! It can't always be easy. Believe me or not I do some review before the quizzes and I have not yet reached 100%. Some of the questions are tricky! Try hard. Ask for help on the forums. You'll make it! :)" In the English Composition course, "Belonging to community" was the most frequently coded affect. Text coded to this affect included the following types of posts: "I believe most learners here are also not expert writers, just like you and me. So let's just keep writing and keep improving together, okay?" "Most of the time, I feel like I'm an individual learner, but when I see the discussions, answers, and so on, I feel like there is someone who is doing something with me also, so I feel sometimes a group member." "I'll hope we can interact, learn and share knowledge together." In Chemistry, "frustration," "humor," and "belonging" were frequently coded affects: "I am so confused about how to determine the protons and electrons that move and create different reactions.
So frustrated." "I got strange looks from people who don't think that a sleep-deprived working single mother should be giggling at chemistry at 2am." "I've learnt so much from you all, and I know I can come with any question no matter how trivial." While some writing was coded to "competitiveness" (for example, "I do not want to sound blatant or arrogant but I would expect it to be more challenging"), it was not a

Affect, Top Posters, and the General Discussion Forums
In addition to the weekly forums which were set up by the course instructors, each course also contained other forums including one called "General Discussion." Table 7 compares the affect of top posters in each course to general discussion forum posters.
In Chemistry, the top posters most strongly expressed encouragement (23% of the words coded to affect), a feeling of belonging to the community (21%), and motivation (11%). Comparatively, other posters in the Chemistry general discussion forum heavily expressed motivation (64%), with the next most commonly expressed affect being gratitude (6%). The top posters in Chemistry were also more frequently coded as being receptive to critiques (11%) than the general posters (2%).
Similarly, in English, the top posters in the general discussion forum were much more frequently coded as being receptive to critiques of their work by peers (29%) than general posters (5%). The top posters also much more frequently expressed defensiveness (20%) than general forum posters (6%). Posts in the forums most frequently were coding as expressing encouragement (20%), belonging to the community (18%), and gratitude (15%).

Learning Gains and Forum Posts
One criticism of MOOCs is that assessment of student learning can be difficult when relying on multiple-choice quizzes (Meisenhelder, 2013). Many MOOCs, however, have much more versatile assignment types and answer formats available (Balfour, 2013;Breslow et al., 2013). Writing in MOOCs-whether through formal writing assignments, short-answer quizzes, or discussion-forum dialogue-can offer a strong opportunity for students to gain in learning objectives and for researchers to assess student learning (Comer, 2013). Some prior literature has even suggested that people can be more reflective when their engagement is via online writing than in face-to-face interaction (Hawkes, 2001).
Prior research reveals that through forum writing and peer assignment exchanges, students could be viewed as moving through phases of practical inquiry: triggering 11 The discussion forums may include some of the posts written by top posters. 12 The sum of each column is 100%, meaning that each cell refers to the percentage of overall words coded to top posters. 13 The sum of each column is 100%, meaning that each cell refers to the percentage of overall words coded to forums.  (Garrison, Anderson, & Archer, 2001).
Discussion forums in particular offer a rich opportunity for examining student learning gains. Learning gains can be probed by analyzing student dialogue in the discussion forums to evaluate the nature and quality of the discourse. Through our coding of discussion forum posts, we were able to gain insights into student learning gains. Some students enthusiastically post about their learning experiences. Table 8 shows the percentage of discussion forum posts that demonstrated learning gains.  Figure 3 shows the distribution of these learning gains in Chemistry forums. When learning gains were present in Chemistry discussion-forum posts, they were frequently coded to demonstrating understanding. Figure 4 shows the distribution of these learning gains in English Composition discussion forums. When learning gains were present, they were most often related to demonstrating understanding, but also showed significant gains in evidence of incorporating feedback. Like the learning gains in Chemistry, English Composition students also had a very low incidence of discussions about their grades (3.27% and 1.58%, respectively).   In both courses, very little text was coded to the "improved grades" node. Research shows that a focus on grades can be counterproductive to learning gains (Kohn, 2011).
The coding results here suggest, therefore, that when students were discussing learning gains they were discussing more meaningful measures of learning gains than grades.
Indeed, students posting on the forums in these MOOCs were much more focused on learning than on grade outcomes. As an illustration, one student expressed this sentiment concisely by writing, "I am not hung up on the grade I am too excited about what I learned and how I am putting it into practice and getting results."

Writing Elements in Peer Review
Enrollees' tendency to discuss more meaningful measures of learning gains in the forums also extended to their interactions through peer review. Peer review is more effective when peers focus on higher order writing elements as opposed to lower order concerns (Elbow, 1973;Clifford, 1981;Nystrand, 1984;Keh, 1990). Figure   The greater prevalence of peer notations about plagiarism in Chemistry is likely due to the course instructor specifically asking peers to look for plagiarism: "This is going to come up some small fraction of the time, so here is the procedure: What should you do if you are reviewing an essay that you believe is blatant plagiarism?" Suspected plagiarism was then confirmed by the instructor, who investigated student flagged work.
Editorials have expressed concern that MOOC providers and faculty need to be more rigorous at facilitating academic integrity and discouraging or penalizing plagiarism (Young, 2012). Continued work should indeed be done in this area. This is especially important given that, when writing assignments are used on this scale, observations made in face-to-face settings can be magnified. For example, Wilson noted in the summary of his work about writing assignments in a face-to-face chemistry course that "Not all students submitted original work" (Wilson, 1994(Wilson, , p. 1019 Chronicle of Higher Education faculty forums reveals that plagiarism continues to constitute a challenge in all educational settings rather than being unique to MOOCs.
However, while academic honesty is of the utmost importance, it is also important to continue facilitating peer commentary based on other elements of writing, especially the higher order concerns named above. Some students expressed a negative impact from what they perceived to be too great a focus by their peers on plagiarism in the Chemistry peer review: "This peer review exercise is rapidly turning into a witch hunt. My opinion of this course has, during the past 2 days, gone from wildly positive to slightly negative."

Type of Feedback in Peer Review
Research shows the kind of peer feedback provided impacts peers' perceptions of the helpfulness of that feedback (Cho, Schunn, & Charney, 2006). We categorized peer feedback by type: positive, constructive, or negative. We defined positive as consisting of compliments that were not related to improving the paper; constructive comments included helpful feedback that a writer could use to improve his or her project or take into consideration for future writing occasions; and negative feedback included comments that were not compliments and were also unconstructive/unhelpful. The   "I liked your essay, it is cohesive and concise and its subject is intriguing!" "You did the great research … and your bibliography is impressive. The introduction is brief, but sufficient, the problem you've built your text on is claimed clearly, and 15 Compliments were also least often coded to plagiarism in both courses, but we eliminated that from the compliments because noting plagiarism is implicitly not considered a compliment. The writing element labeled Citations enables learners to comment on citation in the form of a compliment. 16 Total distribution for each course is 100%, meaning that each percentage refers to the percentage of overall words coded to the positive feedback (or compliment) node for a given course. Peers in English Composition were most likely to provide constructive feedback on argument and analysis, English language skills, citations, and format and style. Peers in Chemistry were most likely to provide constructive criticism on additional resources, topic, format and style, and factual accuracy. Coding frequency for both courses is shown in Figure 7. 17 Total distribution for each course is 100%, meaning that each percentage refers to the percentage of overall words coded to the constructive feedback node for a given course. Unconstructive criticism in peer review.

0%
Unconstructive feedback was defined as comments that were negative but not helpful in terms of recommending specific improvements to the student whose work was being reviewed. Figure 8 shows the distribution of unconstructive feedback in the open-ended peer reviews in each course. Peers were most likely to center unconstructive feedback in English Composition on matters of argument and analysis, clarity, and format and style.
For Chemistry, peers were most likely to provide unconstructive feedback on topic, opinion, and additional resources. It is important to note that because we did not code the assignment submissions "Even more important bit I learned was the importance of feedback. Feedback provides an opportunity to rethink the project, and dramatically improve it." Table 9 shows the frequency of when peer feedback explicitly expressed learning gains.
The English Composition peer review rubric specifically asked reviewers to indicate what they had learned from reading and responding to the peer-writing project (see Appendix F). The Introduction to Chemistry peer rubric did not ask this. This probably accounts for why the learning gains were so much more evident in peer review in English Composition than in Chemistry.

Learning Gains and Student Challenges
Because we are interested in the impact of peer-to-peer interactions with less academically prepared students, we specifically looked for challenges faced by students, such as the following: lack of time or energy; 21 less academically prepared; and less or not self-directed. Interestingly, little text was coded to these nodes. For example, only one coding reference was found at the intersection of any of these challenges and learning gains. Therefore, these barriers did not come up in the forum threads that we coded in either class.

Discussion/Conclusions
We have identified several significant themes that show the importance of and impact of the peer-to-peer interaction through writing in MOOCs.

MOOC Discussion Forum Posts are Connected to Course Content
Both courses examined generated substantial student dialogues on the forums. Students in the English Composition: Achieving Expertise course tended to write longer forum posts than students in the Introduction to Chemistry course. Peer-to-peer dialogue on the weekly forums closely mirrored the content of the course described in the syllabus for that week. This shows that students are primarily discussing course content in these 21 Lack of time or energy could be a factor associated with less academically prepared students, or it could be unconnected to that mode of student challenge. forums and suggests that peer-to-peer writing in the forums can provide one measure of student success in a MOOC.

MOOC Discussion Forums Generally Contribute Positively to the Learning Environment in Chemistry and English Composition
Since attitude was generally positive and the top affects in the forums include belonging to a community, gratitude, and encouragement, we conclude that the forums are in general a positive space for learners to interact. This finding operated across disciplinary context, both in a natural science course and in a more humanities oriented course.

MOOC Discussion Forums Contribute to Learning Gains, Especially in Understanding
In terms of observed learning gains, peer-to-peer interaction on the forums seemed to make the most impact on enhancing and facilitating understanding. Students sought, offered, and provided tips or support from one another on the forums as a way of increasing their understanding of course content.

Peer Review Can Facilitate Learning Gains If This Possibility is Made Explicit
The disparity between the coding for learning gains in English peer reviews and in Chemistry peer reviews suggests that the English students were indicating learning gains because they were asked to do so explicitly. This suggests that faculty should encourage students to reflect on their learning gains explicitly as a way of facilitating those very learning gains.

Peer Feedback on Writing can Meaningfully Focus on Higher Order Concerns across Disciplines
Feedback on writing can be differentiated between that which focuses on higher order or lower order concerns. Effective formative feedback generally must include a focus on higher order concerns, and can then be considered an integral part of the learning and assessment environment (Gikandi, 2011). Peers in both courses focused predominately on higher order concerns, even as they were also able to focus on lower order concerns.
This may be due to the peer feedback rubrics. Our data also suggest that peers will follow closely the rubric provided by the instructor. In English Composition, students were asked to focus on argument and analysis. In Chemistry, students were asked to focus on strengths, insights, areas for improvements, and plagiarism. In both cases the students were likely to adhere to the rubric guidelines.

Writing through the Forums Enhances Understanding
Since forum discussions in a MOOC happen through writing, one can extrapolate from our data that writing enhances understanding in MOOC forums. This bolsters evidence for writing-to-learn and suggests that MOOC forums are a key pathway for writing-tolearn and a key pathway for assessing student success in MOOCs across disciplines.

A Limited Group of Learners Posts to the Forums
One of our key areas of inquiry was to understand how peer-to-peer interaction through writing might impact the learning gains of less academically prepared learners. We found, however, that people posting to the forums did not identify themselves explicitly as less academically prepared. This generates questions about how many people post to the forums, and who is or is not likely to post to the forums. The total number of people who posted to the discussion forums in English Composition represents 23% of the total number of people who ever actually accessed the course (51,601); in Chemistry, the total number of people who posted to the discussion forums represents 7% of the total number of people who ever actually accessed the course (22,298). 22 Given the overall positivity of the forums, one wonders if these data indicate that the forums are only positive for certain types of people. Given that the top posters coded higher for "defensiveness" than general posters, one also wonders if there might be drawbacks to certain levels of forum participation. We did not see any significant information about student challenges in the coding data, despite looking for it as one of our coding nodes.
Since interactive learning offers so much promise for these learners, and since MOOCs continue to provide the possibility of increased access to higher education, more research is needed about how to facilitate forums in as inclusive and productive a way as possible for less academically prepared learners.
The development of quality educational opportunities through MOOCs, and learning more about how peer interactions through writing contribute to student retention and learning, has the potential to make a significant global impact and increase postsecondary access and success in unprecedented ways. As we discover more through this research about how peer interactions with writing contribute to student learning outcomes and retention, we will be better positioned to understand and work towards a model of higher education that is more flexible, accessible, and effective for the great many individuals in the world interested in pursuing lifelong learning. The following information helps define our terms: Forum: A forum is the top level discussion holder (Week 1). These are created in Coursera by instructional team staff.
Subforum: A subforum is a discussion holder that fall under the top forum (Week 1 Lectures, Week 1 Assignment). These are also created by instructional team staff.
Thread: A thread is a conversation begun by either instructional team staff or by students. A single forum typically contains many threads covering many different subjects, theoretically related to the forum's overarching topic. Post: A post is an individual's response to a thread. Posts can be made by either instructional team staff or by students, and may be posted with the students identifying name, or may be posted as anonymous. Staff with administrative privileges may "toggle" a setting on each post to reveal the identity of students who have chosen to post anonymously. Although in theory a post is a new "top level" contribution to an existing thread (as opposed to comment (read below), many students don't pay attention to whether they are posting or commenting, and therefore, we didn't feel that we could accurately distinguish between the two. Comment: A comment is a reply to a post. Comments can be made by either instructional team staff or by students. Again, we decided not to distinguish semantically between a post and a comment, because we felt that distinguishing them was not possible in (the very common) complex web of post, response, subsequent post, subsequent response, etc.
The main page of a forum lists all the primary threads begun in that forum or subforum.
If threads are begun in a subforum, they are only listed in the "all threads" area of the subforum, not in the "all threads" list of any parent forum. Subforums may themselves have subforums (which are also called subforums). At the bottom of a list of "all threads," you can read the total number of pages of threads that exist; each page contains 25 thread headings. Therefore, an estimate of the number of threads can be obtained by multiplying the number of pages by 25.
Coursera provides a number of views received by each thread, as well as the number of times a thread has been opened and read. Of course, by opening a thread to capture it for analysis, we are increasing the number of views of that thread, so we used the "views" number in our analysis with this limitation in mind.
Coursera also allows the viewer to sort by "  1) to encourage you to learn more about the chemistry related to a specific topic that interests you through research and writing.
2) to allow you to learn more about diverse topics of interest to other students by reading, responding to, and reviewing their essays.
Assignment: Pick any topic related to chemistry that interests you (some global topics are listed below to give you ideas, but you do not have to restrict yourself to that list.) Since most of the global topics are much too broad for the length limit allowed, narrow your interest until the topic is unique and can be covered (with examples) in less than a couple of pages of writing.
Once you have a topic, write an essay in which you address the following questions: • What are the chemicals and/or chemical reactions involved with this topic?
• How does the chemistry involved with this topic relate to the material in the course?
• Are there economic or societal impacts of this chemistry? If so, then briefly describe aspects of ongoing debate, costs, etc.
• What some some questions for future research papers if you or someone else wanted to learn more about how chemistry intersects with this topic?
• Did this research lead you to formulate any new questions about the chemistry itself?
Individual Research Paper Guidelines and Requirements: • Because other students will need to be able to read what you have written, the assignment must be submitted in English. If you are worried about grammar because English is not your native language, then please just note that right at the top of the essay and your peers will take into account the extra effort it requires to write in a foreign language.
• Think about what terms your classmates already know based upon what has been covered to date and what terms might need additional explanation.
• The final paper should be 400-600 words, not including references or • Be careful not to use the "cut and paste" method for your writing. Each sentence should be written in your own words, with appropriate references to the works of others if you are getting your ideas or information for that sentence from a source.
• Online references should be used, and these should be free and available to everyone with internet access (open source, no subscription required.) At least three distinct references must be included.
Wikipedia and other encyclopedias should not be cited, but these can be a starting point for finding primary sources. Please be sure to cite your source websites. Please provide the references at the end of the essay as a numbered list, and insert the citation at the appropriate spot in the essay body (usually right after a sentence) using square brackets around the number that corresponds to the correct reference on the list.
• The paper can include up to 3 tables/figures . Tables and figures are optional, but might be helpful in conveying your ideas and analysis. • The paper should include data and/or chemistry related to the topic and might also include an analysis of the impacts of the issue upon society (yourself and the community.) Political, economic, and/or historical analysis may also be appropriate depending upon the topic.
Every paper MUST contain some chemistry.
• Submission will be electronic, and submitted papers will be copied to the course forum as soon as the first peer feedback is received so that others may learn and continue the discussion. Author names will be posted with their writing on the forum as well. Including your name promotes accountability in your work and closer collaboration among peers.
• Sample Writing and Sample Peer Feedback: Prof. Canelas has secured permission from a few real students from former courses to post their essays and sample peer feedback to help guide your work. These will be posted in a separate section under the "Reference Information" section on the course main page no later than the beginning of the third In this project, I will ask you to write a 600-800 word critical review of Coyle's article, summarizing the project in his terms, quoting and analyzing key words and passages from the text, and assessing the limits and uses of his argument and approach.

Project Two: Analyzing a Visual Image
In your second project, I will ask you to develop a 600-800 word analysis of a visual representation of your chosen field of expertise. I will ask you to apply Coyle's and

Project Three: Case Study
In this project, I will ask you to extend your work with Projects One and Two by researching additional scholarship about expertise in your chosen area, reading more texts about expertise theory through a crowd-sourced annotated bibliography (a collection of resources, with summaries, posted by all students), and applying those to a particular case study (example) of an expert or expertise in your field. I will ask you to extend these scholarly conversations through a 1000-1250 word case study in which you can articulate a position about expertise or an expert in the area of inquiry you have chosen.

Project Four: Op-Ed
Since the academic ideas are often made public (and arguably should be), I will ask

Project Components and Key dates
Project 3 will be completed in sequenced stages so you can move through the writing process and have adequate time to draft and revise by integrating reader feedback.

Overview
Case studies offer academic writers the chance to research a particular example in a deep, sustained way, and then consider the ways in which that case study might offer generalizable conclusions. For Project 3, extend your work with Projects One and Two by researching additional scholarship about expertise in your chosen area, read more texts about expertise theory through a crowd-sourced annotated bibliography (a collection of resources, with summaries, posted by all students), and apply that research to a particular case study (example) of an expert or expertise in your field. Specifically, we will continue to work with the elements we learned in Units 1 and 2, as well as build on them by focusing on how to: • conduct research; • write an extended argument; • develop an intertextual conversation; • understand different limits of and uses for popular sources and scholarly sources; • create effective introductions; and • write strong conclusions. Your steps for this project include the following: • Identify a potential case study you would like to use for Project 3. Remember that the process of research is sometimes recursive, and you might find through your research that you would like to change or modify your original idea for a case study. This is a natural part of the research process. • Find and read texts about this case study.

Readers
Your readers will be interested in questions about expertise, but are perhaps unfamiliar with the texts you have read or the area of expertise you have chosen.

Questions to Help You
Consider the following questions as you develop your argument: • What can we learn about expertise by researching a particular case study?
• What does it take to succeed based on this case study?
• What are the defining features of expertise based on this case study?
• What can you learn about expertise based on this case study?
• Based on this case study, how is expertise being defined?
• How might this case study reinforce, challenge, or otherwise modify our prior thinking about expertise, such as the ideas of Coyle, Colvin, or others? • How might this case study raise new questions about expertise?
• What questions does the case study raise for you?

Integrating Evidence and Citing the Evidence
Integrate evidence into your essay by including quotes and/or paraphrases from the research.
Strategies for effectively incorporating quotes and paraphrases are described in the video, "Integrating Evidence." Refer to OWL for specifics on the school of citation you are choosing to write within. You should choose a school of citation with which you would like to gain more familiarity and/or that seems most relevant for your future pursuits.
Include a "Works Cited" or "References" page at the end of your work listing all texts you have referred to.

Submission Guidelines
Post all documents to the appropriate Assignments section no later than the due date so your responders can read and send you comments for your final version.

Providing Evaluative Feedback For Project 3, Final Version
Using the grading criteria above, the writer's "Note to Readers," and our overall learning objectives/criteria for this unit, you will be scoring your colleague's projects on a 6-point scale in order to help them improve as writers for subsequent writing occasions.
Think of the 6-point scale as two halves: a top half of 4, 5, or 6 representing different levels of successful projects and a lower half of 1, 2, and 3 representing different levels of unsuccessful projects.
You can think of a paper scoring 5 as the center of success and one scoring 2 as the center score for lack of success, with the other scores as a minus or plus. Thus a score of 4 is successful, but marginally so, a kind of 5-. A score of 6 is exceptionally successful, a kind of 5+. Only one whole number, without pluses or minuses, can be entered on the SCORE line. Your score will be combined with three other peer scores to obtain a grade for the writer's project.
Score of 6: This project will meet all criteria and goals for unit 3 and be very clear and well written. It need not be perfect but it will be well reasoned, show a deep understanding of the case study, evaluates and discusses relevant research, and shows a compelling discussion of how the case study reflects, contrasts, or modifies our thinking about expertise. The project uses the case study to raise new questions about expertise.
The introduction and conclusion are strong. Evidence is integrated effectively, and the title is strong. Citations are mostly correct.
Short description: Exceptionally successful Score of 5: This project not only presents the case study, but also uses it to make an argument about expertise. It is clear and well written. The project includes relevant research. Paragraphs are unified and the paper is organized clearly. The introduction and conclusion are strong. Evidence is integrated effectively, and the title is strong.
Citations are mostly correct.

Fill in the following boxes:
What overall comments do you have for the writer as he or she moves on to Project 4?
What did you learn about you your own writing based on reading and evaluating this writer's project?  Reflective Quizzes) and you are preparing to share it with others. These potential readers might be administrators at a school you are applying to, current or potential employers, friends, or other acquaintances. Your task is to write a cover letter that introduces your work and makes an argument about your understanding and achievement of the course learning objectives.
In the space provided here, discuss what you have learned in this course and choose 2-4 of our course learning objectives, describing each objective and referring specifically to particular passages from your coursework that demonstrate your progress towards and/or struggles with that objective. Indicate why you have chosen those objectives as the most important for you. Cut and paste specific portions of your coursework, and use them as evidence for your argument. In this way, by having an introduction, argument, evidence, and conclusion, your "portfolio cover letter" will both discuss and