Are Online Learners Frustrated with Collaborative Learning Experiences?

Online education increasingly puts emphasis on collaborative learning methods. Despite the pedagogical advantages of collaborative learning, online learners can perceive collaborative learning activities as frustrating experiences. The purpose of this study was to characterize the feelings of frustration as a negative emotion among online learners engaged in online computer-supported collaborative learning (CSCL) experiences and, moreover, to identify the sources to which the learners attribute their frustration. With this aim, a questionnaire was designed to obtain data from a sample of online learners participating in the Master of ICT and Education program of the Universitat Oberta de Catalunya (UOC). Results revealed that frustration is a common feeling among students involved in online collaborative learning experiences. The perception of an asymmetric collaboration among the teammates was identified by the students as the most important source of frustration. Online learners also identified difficulties related to group organization, the lack of shared goals among the team members, the imbalance in the level of commitment and quality of the individual contributions, the excess time spent on the online CSCL tasks, the imbalance between the individual and collective grades, and difficulties in communication, among other factors leading to frustration. The analysis of the students’ sources of frustration in online CSCL is followed by a list of recommendations to the distance education stakeholders, aiming to reduce students’ frustration and improve the quality of their experiences in online CSCL contexts such as the UOC.


Introduction
Online learning programs are growing at exponential rates (Allen & Seaman, 2007;Bishop & Spake, 2003;Kariya, 2003), and most of their participants are adult learners (Kuenzi, 2005). The design of learning experiences within these online programs is also evolving.
Quality requirements of higher education and the need for the assurance of learning outcomes are increasing the challenges with which online universities are faced. In order to meet these challenges, virtual campuses are promoting learning methodologies that prioritize learning "through interactions among students" (Stahl, Koschmann, & Suthers, 2006, p. 2) and contribute to the development and practice of teamwork competencies. While early online programs focused mainly on the transmission and mastery of bodies of information, more emphasis has been placed on collaborative methods in recent years (Bruffee, 1999;Dirkx & Smith, 2004), examples of which are case studies, problem-based learning, and the development of learning communities in online contexts. According to Dirkx and Smith (2004), in theory, online collaborative activities could be considered as the key to this new learning paradigm. Despite this, they stress that learners' perceptions and experiences could generate a profound sense of ambivalence when learning and working in a group. While many participants laud the opportunity to interact and work with fellow group members, they clearly find consensus decision making and production of a product much less satisfying (Baltes, Dickson, Sherman, Bauer, & LaGanke, 2002;Dirkx & Smith, 2004). Difficulties with interpersonal issues and trying to ensure that all group members do their share seem to cloud their initial enthusiasm. In addition, communication and interaction limitations caused by technology (Ragoonaden & Bordeleau, 2000;Ng, 2001) only serve to exacerbate these concerns, which then become a channel for frustrations.
In the next section, we introduce the concept of frustration in online computer-supported collaborative learning (CSCL). Then we provide an explanation of the methodology designed to analyze the frustration among a group of online learners in the Universitat Oberta de Catalunya's (UOC) virtual campus.
Frustration with Online CSCL also cause negative effects (Artino & Stephens, 2007;Artino, 2008;Hickey, Moore, & Pellegrino, 2001;Järvenoja & Järvelä, 2005;Sierpinska, Bobos, & Knipping, 2008). In particular, some computer-supported collaborative learning environments may interfere with students' willingness to engage in the project. They may also experience stress and frustration in collaborating with people they do not know well (Curtis & Lawson, 2001). In CSCL, the success and efficiency of the collaboration cannot be taken for granted (Dirkx & Smith, 2004;Järvenoja & Järvelä, 2009;Kreijns, Kirschner, & Jochems, 2002). In some cases, CSCL experiences may also evoke negative emotions and create new challenges for motivation when people experience conflict with their own characteristics, objectives, and requirements. Student frustration can be caused by internal or external factors (Bessiere, Newhagen, Robinson, & Shneiderman, 2006), and it can adversely affect a student's learning experiences. The consequences of student frustration (Borges, 2005) can generate a load that has to be borne by all the agents involved in the learning experiences: students, teachers, and institutions. Finally, disillusionment and frustration can lead students to abandon their studies and leave the institution (Conrad, 2002).
In online CSCL, students' frustration could be considered as the intersection of the frustration involved with both collaborative learning and online education. The specific sources of frustration in online CSCL are related to the delay of the interactions and feedback (Hara & Kling, 1999;Vonderwell, 2003), to time pressure (Goold, Craig, & Coldwell, 2008), to time zone differences (Grinter, Herbsleb, & Perry, 1999;Romero, 2006Romero, , 2010 and to the reduced level of cues within the social activity and context (Rettie, 2003;Sallnäs, 2004).

Objectives
Despite the identification of factors leading to online CSCL frustration, as previously referred to, only a few studies have addressed students' negative emotions such as frustration. Moreover, we have not identified studies focused on specifying the sources of student frustration in the online CSCL context. Considering the relevance of the well-being of students during their online learning experience, this research aims to estimate the magnitude of the phenomenon of frustration among students involved in online CSCL, with the sources of frustration being identified by the students themselves.

Context and Methodology
To study online CSCL frustration factors, we considered the analysis of a situated task at the Universitat Oberta de Catalunya's (UOC) virtual campus. The consideration of an authentic context aims to preserve the conditions that could lead to sources of frustration.
First, we will introduce the online CSCL context and its participants as well as the characteristics of the collaborative learning activities at the UOC's virtual campus. Then we will introduce the methodology for the analysis of the students' frustration perception in this real context.

Participants
The study was conducted on a group of students (N = 40) enrolled in the university's master's degree program on e-learning at the UOC's virtual campus during the second semester of the 2010-2011 academic year. The students were adults whose average age was 37 (SD = 8.91). There was a higher representation of females (n = 30) than males (n = 10). None of the participants were freshman, and most of them had completed at least three semesters at the UOC's virtual campus (SD = 1.03).

UOC's Pedagogical Model and the Online CSCL Tasks
The master's degree program follows the UOC's educational model (Sangrà, 2002), which is oriented toward collective participation and knowledge building from an interdisciplinary plan and is open to student-oriented learning and social and working experience. The UOC is committed to collaborative learning through methodologies that require resolving problems, project development participation, combined product creation, discussion, and enquiry.
During the master's program, the students were engaged in various collaborative learning activities in different courses simultaneously. The online CSCL tasks had a general duration of several weeks (between two and four weeks), and the groups were composed of four to six students for the development of the task. The online CSCL activities were to be developed in the UOC's virtual campus, and the interactions among the students were carried out in text-based tools, primarily through a discussion forum and email. The task concerned the collaborative writing of papers, where the students should develop a topic or solve a case study.
Online Collaborative Learning Experiences Frustration Questionnaire (OCLEFQ) Educational research on subjective measures, such as emotions, uses both qualitative and quantitative approaches (Schutz & Pekrun, 2007), with a predominance of self-reported measures (Larsen & Fredrickson, 1999), learner log files, and online messages as data sources for analysis (Shih, Feng, & Tsai, 2008). Considering the lack of previous studies in the field of frustration characterization in online collaborative learning, we based the data collection on the students' declarations by creating an ad hoc survey that we named Online Collaborative Learning Experiences Frustration Questionnaire (OCLEFQ). The OCLEFQ was designed with the objective of identifying the sources of frustration of online learners in CSCL. The literature review has provided the background for studying the dimensions, categories, and conceptual elements needed to develop a set of variables and guidelines for the construction of the items.
After taking all of this into account, we designed the OCLEFQ and divided it into six sec-

tions.
Section 1 is composed of questions related to the learning and the learners' situation. It is used to obtain demographic information regarding the learner's gender, age, number of se-30 mesters completed, number of courses per semester, and the time dedicated to the master's program.

Sections 2 and 3 aimed to collect information about the students' prior experiences in on-
line collaborative learning and their attitudes and conceptions about teamwork.
In section 4, subjects were requested to report their own frustrating experiences through an open question wherein the students could describe their online collaborative learning experience in their master's program. The data collected from this section helped to develop an understanding about the personal experiences of the participants and their conceptions about online collaborative learning.
Section 5 was designed to focus on sources of frustration in order to determine if there were differences between results reported by students' own experiences and those that were observed by others. To address this concern, our assessment strategy used rating scales. The four themes that structured and organized this section were the four dimensions identified by Dillenbourg (1999) that must be considered in an experience expected to be collaborative: the learning situation, the interactions, the processes, and the effects.
Section 6 consisted of six questions that assessed how frustrated the individual was (level and frequency of variables) and how this frustration affected his perception of participating in appropriate training that would satisfy his personal and professional ambitions (affect variable).
To measure accurately beliefs and feelings of participants a total of 37 closed-ended and one open-ended question were considered in the six sections. The closed-ended questions in the questionnaire were multiple choice (used to gather demographic information) and interval scale questions (when a feeling, an attitude, or an agreement level had to be measured ). An open-ended question was used to gain insight into how the respondent felt. To further enhance validity, the presentation of discrepant information was used to provide a full account of the responses (Creswell, 2003) and was implemented in multiple items.
Reliability was tackled by placing emphasis on the target audience and taking ethics into account (Dillman & Bowker, 2001) and included questions designed to be response-friendly (Dillman, Tortora, & Bowker, 2001) to maintain certain standards of quality, accessibility, and usability.
Students attending the master's program were invited to participate in the study via email.
Each invitation provided an overview of the research and an explanation of how to access the online survey.

Results
First, the results section introduces the characterization of the feeling of frustration among the students involved in online CSCL at the UOC's virtual campus. Afterwards, the potential sources of frustration are introduced.

Characterization of Frustration in Online CSCL
In this section, we estimated the magnitude of the phenomenon of frustration among students when involved in online collaborative learning experiences. The mean score for the level of frustration fell on a moderate level (M = 3.15, SD = 1.14). The largest number of scores fell also on a moderate level of frustration at 30% (n = 12), followed by a high level at 27.5% (n = 11). A low level received 22.5% (n = 9), and a very high range was 12.5% (n = 5).
A very low level accounted for 7.5% (n = 3) of all the scores. The results of multiple regressions between levels of frustration as the criterion variable and each of the demographic variables found correlations of significance for attitude.
Focusing on the frustration perception of the online CSCL learners, the mean score (M = 2.58, SD = 1.2) indicated that, overall, respondents occasionally felt frustrated. The largest percentage of responses (40%, n = 16) rarely feel frustrated, and 17.5% (n = 7) of all responses, very rarely. However, 7.5% (n = 3) of the responses and 17.5% (n = 7) of the responses were very frequently and frequently frustrated.
The results of multiple regressions to find out if a relationship existed between frequency of frustration as the criterion variable and each of the demographic variables as predictors found no correlations of significance.
Scores for frustration with regard to the perception of participating in an appropriate and satisfying training program showed that the majority of respondents (45%, n = 18) felt little affect. The mean score was also at this rate (M = 2.10, SD = 1.01). Participants who had perceived no affect were 30% (n = 12) of scores. Scores for those who felt quite affected accounted for 15% (n = 10) of the scores, while the neutral position accounted for 10% (n = 4) of scores. None of the participants responded that they felt very affected. The results of multiple regressions between affect of frustration as the criterion variable and each of the demographic variables as predictors found a correlation of significance for attitude.
The bivariate correlation was used to explore the relationships among the three frustration it is to be obeyed." Conflict and problems in reaching consensus is one of the sources of frustration, as participant number 32 manifests: "There was not a good feeling between us because our points of view were extremely different and nobody was able to give in to reach consensus." Participant 30 describes frustration related to a focus only on results, specifically on the marks obtained: "Assessment focuses only on the final product, and individual participation was not taken into account. There are always people that take advantage of this." Finally, a new source of frustration was found: the instructor role. In this case, participant 31 thinks that the instructors never interact with the group, and participant number 26 feels frustrated because of a lack of any type of orientation or guidance.
The descriptive analysis of nine quantitative variables were used to confirm the findings of the qualitative analysis: unshared goals/difficulties in organization; imbalance in the level of commitment, responsibility, and effort; imbalance in quality of individual contributions; excess of time spent and workload; problems with conversation skills; difficulties in communication/dialogue in terms of frequency; misunderstandings; conflict and problems in reaching consensus; and imbalance between individual expected mark and group mark.

Discussion
The purpose of this study was to explore the phenomenon of frustration among students involved in online collaborative learning experiences. We aimed to advance knowledge about the characterization of the frustration experienced by e-learners in online CSCL and identify the sources of frustration that may affect the quality of these experiences from a learner-centered point of view. First, we will discuss the characterization of frustration in this study then we will analyze the factors related to this frustration. We will conclude this section by introducing the main research prospects of this exploratory study on online CSCL frustration.

Characterization of Frustration in Online CSCL
Frustration is a common phenomenon among students involved in online collaborative learning experiences, according to the literature review previously introduced. In the context of the UOC's master's degree program, data indicated that the students deal with frus-trating experiences sporadically (40%), and when this occurs it adversely affects students' emotions and learning experiences at a level between moderate (30%) and high (27%).
This illuminates the fact that students deal with frustrating experiences sporadically, and a significant portion associates this with a high level of frustration towards the online CSCL experience.
Despite the initial consideration, the analysis of the students' self-reported declarations does not reveal any significant impact on student satisfaction related to the master's program. The majority of respondents (45%) felt little affect and had a positive perception with regard to participating in an appropriate and satisfying training program. This could be explained by data results that presented medium correlations through level, frequency, and affect frustration variables. Affect would increase in accordance with frequency and level of frustrating online collaborative learning experiences, but these experiences are only part of the whole learning experience in which students participate.
Attitude was the only demographic variable found to be a significant predictor of frustration. This finding is consistent with previous studies (Dirkx & Smith, 2004;Roberts, 2005) that suggest that prior experiences and attitudes would affect an individual level of frustration. Some students coming to online collaborative learning for the first time do not care for the idea of group work and can be apathetic or even on occasion actively hostile to the whole idea (Roberts & McInnerney, 2007). Students struggle with the development of a sense of interdependence and intersubjectivity with their online groups (Lushyn & Kennedy, 2000) but end up holding fast to subjective, individualistic conceptions of learning.

Sources of Frustration in Online CSCL
Participants in the study provided data about their own frustrating experiences and reported that when individuals were unwilling to participate or only minimally contributed to the activity, frustration appeared. A common frustrating experience was associated with a poor work ethic and with some members who did not fulfill their obligations. Creating more work for the other members in the group is the direct consequence. Mentions of 57.5% (n = 23) of participants have proved the imbalance in the level of commitment, responsibility, and effort to be the major source of frustration related to the task category and the CSCL experiences in general. This finding was confirmed by the main score for the related quantitative variable that shows agreement. It is also consistent with other studies that found that one or more students often end up taking responsibility for completing the work, whether they want to or not (Burdett & Hastie, 2009;Mills, 2003). In some cases, these leaders may be encouraged to do more by other members in the group, and greater responsibility and workload is the result (Payne & Monk-Turner, 2006). Social loafing behavior creates an imbalance of effort and participation (Goold, Craig, & Coldwell, 2008), such that free riders (Kerr & Bruun, 1983) are able to take advantage of the contributions of others.
Having unshared goals among the teammates of the group is the second most important factor, considered by 22.5% of the respondents. This finding is confirmed by the mean score from the quantitative analysis, which was in agreement, as well as by related literature. As the teacher role is de-authorized, group members are encouraged to assume responsibility for their own structure and direction and demonstrate their development by creating their own goals, guidelines, and rules (Wheelan, 1994). Establishing common goals is part of the construction of common grounds since actions cannot be interpreted without referring to (shared) goals, and, reciprocally, goal discrepancies are often revealed through disagreement on action (Dillenbourg, 1999).
Qualitative analysis revealed difficulties with regard to communication as another source of frustration. Communication was reported to be essential in keeping group members focused on their responsibilities in relation to the common goal; further, a lack of communication prevented groups from clarifying goals, roles, and other group functions. A total of 45% of participants agreed that conversations are often characterized by multiple and somewhat schizophrenic patterns of interaction. This is a confirmation of existing research (Dirkx & Smith, 2004), which finds that students' perspectives and schema of group acquired from face-to-face groups are not appropriate for technologically mediated environments.
There are often time lapses between contributions in asynchronous discussions (McConnell, 2000). Spatial and time distances are potential barriers that reduce the probability of spontaneous communication (Grinter, Herbsleb, & Perry, 1999;Romero, 2010;Saunders, Van Slyke, & Vogel, 2004), and the lack of connection, contact and sense of reality, and immediate social presence are strong influences as well as the sense of community (Melrose & Bergeron, 2006;Rettie, 2003, Sallnäs, 2004. Thus, a lack of nonverbal communication cues and a lack of spontaneity serve only to add to participants' ambivalence and uncertainty about the value of their learning (Ragoonaden & Bordeleau, 2000).
The fourth position in sources of frustration, registering with 17.5% of the participants, is related to problems with negotiation skills. Reponses mentioned a lack of member attributes that foster relationship building, such as amiability, openness, and respect for others.
Frustration generated by the quality of communication in the group discussions confirms reports (Kreijns, Kirschner, Jochems, & Buuren, 2004) that identified a connection between quality negotiation skills and a sense of the development and maintenance of a mutual commitment to a common goal. Skill in learning collaboratively means knowing when and how to question, inform, and motivate one's teammates, knowing how to mediate and facilitate conversation, and knowing how to deal with conflicting opinions (McManus & Aiken, 1995).
A total of 15% of participants agreed that the imbalance in quality of individual contributions leads to frustration. There is no situation of pure knowledge or skills or development symmetry: There are no two individuals in the world with the same knowledge (Dillenbourg, 1999). Distributing the knowledge needed to accomplish a task may have the effect of distributing task roles, creating a local expert effect in which each student independently applies his or her knowledge (Stasser, 1999). When this happens, it may inhibit the group's ability to collaboratively construct new knowledge. The participants in the study reported feeling frustrated by the presence of an expert and dominant member who impeded the development of shared understanding and effort.
Student frustration and conflict often seem linked to uneven sharing of workload in group groups seem to always be meeting (McConnell, 2000). The 15% of participatants mentioning frustration related to workload and excess of time needed to work collaboratively confirmed these studies.
A total of 10% of the students reported, like Boud, Cohen, and Sampson (1999), the assessment of individuals within the group to be one of the major sources of complaint. Individual contributions, or lack thereof, of group members may not be acknowledged in the group mark awarded (Sharp, 2006). Assigning group grades without attempting to distinguish between individual members of the group is both unfair and deleterious to learning and may in some circumstances even be illegal (Kagan, 1997;Millis & Cottell, 1998).
Consensus is critical to the collaborative process (Crook, 1994) because it is only through consensus that members of the group are required to listen, understand, and finally accept the points of view of fellow group members. Mentions of frustration of 15% of participants are related to conflicts and problems in reaching consensus, but they were not confirmed by the quantitative analysis; as a result, this study will not consider it as a critical factor.
Finally, 10% of students considered the instructor role to be a source of frustration, which was not revealed in the pilot study. Instructor inaction was a frustrating factor that was reported as undermining the collaborative process, especially when an instructor is made aware of a problem but does not take any corrective actions. Participants expected the instructors to be actively engaged with learners, providing them with clear guidance, expectations, and requirements. This finding confirms that students expected consistent and timely feedback from the instructor (Vonderwell, 2003) and identified the lack of immediate interaction as a major source of frustration because they were concerned about their performance (Hara & Kling, 1999).

Educational Implications of Students' Frustration in Online CSCL
Findings from this study provided more understanding of the phenomenon of frustration, as well as the needs and requirements of students in online collaborative experiences. Based on these findings, in this section we will provide recommendations for distance education stakeholders to consider at the institutional level, the instructional design level, and the students' level.
First, there is a need at the institutional level to offer the students information about the learning models in general, and the online CSCL activities specifically, in order to allow them to adjust their expectations, preferences, and decision making concerning their enrolment in distance learning programs involving online CSCL activities. Moreover, online universities should explain the interest and objectives of online CSCL activities to the incoming students and provide an introduction to online CSCL and teamwork competencies.
In our study, we observed that the students' main source of self-declared frustration is the teammates' commitment imbalance. Preparing the learner for collaboration through instruction and development of the social and group skills necessary to work effectively in a group will have a positive effect upon the collaborative experience (Chapman & van Auken, 2001;Tideswell, 2004). The main implication for students is the need to have realistic expectations and exercise responsibility in course enrollment by ascertaining beforehand the time, effort, prior knowledge, volume, and quality of work required to carry out online CSCL. The students may struggle with the development of a sense of interdependence and intersubjectivity (Lushyn & Kennedy, 2000) and must abandon subjective, individualistic conceptions of learning (Dirkx & Smith, 2004). From the instructional perspective, we think that the online CSCL activities should be designed with the aim of guaranteeing a certain level of positive interdependence (Johnson & Johnson, 1998) and individual accountability (Slavin, 1989) to the students.
The analysis of student frustration in our study also shows that assessment inequities are important sources of frustration; the implication for institutions is that they must conduct a coherent assessment. The use of individual, self, peer, and group assessment techniques can be extremely beneficial for both students and instructors in all forms of online collaborative learning (Roberts, 2005).
Institutions may supply learning environments that facilitate social interaction and collaboration and assure effective support to students with technological difficulties. Technological difficulties can cause student frustration as well as communication problems, which in return hamper collaborative processes such as explanations, sharing answers, and negotiation (Ragoonaden & Bordeleau, 2000).
The implication for instructors is that it is improtant to to know when intervention is needed in online CSCL and to what degree. Teachers with instructional and student experience in online CSCL (having completed at least one course) will be aware of sources of frustration and will take corrective actions. The instructor should play an active role in the collaborative process. He or she should be proactive in monitoring and intervening in collaborative activities (Chapman & van Auken, 2001;Hansen, 2006) and should ensure that the group works effectively (Tideswell, 2004;Brindley, Walti, & Blaschke, 2009) through mechanisms for assistance, feedback, and evaluation.

Future Research
The study introduced numerous relations among sources of frustration variables from a student-centered perspective. The relations of these factors to the learning situation, the interactions, the processes, and the effects are the key to understanding collaborative learning (Dillenbourg, 1999) as well as student frustration. In this study, we advanced knowledge about the factors related to frustration in the authentic learning context of the professional master's student in e-learning at the UOC. Despite strong evidence on the perception of some factors, such as the imbalance in the level of commitment and the lack of common learning goals among the students, which are in agreement with previous studies, it is rec-