International Review of Research in Open and Distributed Learning

Volume 16, Number 6

November – 2015

iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience

José Coelho1,5, António Teixeira1,4,6, Paula Bacelar Nicolau1,2, Sandra Caeiro1,3, Vitor Rocio1,4

1Universidade Aberta, Portugal, 2Centro de Ecologia Funcional, FC/UC, Portugal, 3CENSE - Center for Environmental and Sustainability Research, UNL/FCT, Portugal, 4LE@D - Laboratory of Distance Education and Elearning, UAb, Portugal, 5INESC TEC, Portugal, 6Centre for Philosophy of the University of Lisbon, Portugal

Abstract

MOOCs are a recent phenomenon, although given their impact, they have been subject to a large debate. Several questions have been raised by researchers and educators alike regarding their sustainability, both economically and as an efficient mode of education provision. In this paper we contribute to this discussion by presenting a case study of the MOOC on Lived Experiences of Climate Change, which piloted the iMOOC pedagogical model developed at Universidade Aberta (UAb), the Portugese Distance Learning University. The iMOOC is a hybrid model which incorporates elements from existing MOOCs but adds other features drawn from UAb's experience with online learning and aims at better integrating in the larger context of the institutional pedagogical culture. The iMOOC implied also an integration of platforms - Moodle and Elgg. The pilot course had more than one thousand registrations, and it was the largest MOOC course on Portuguese language delivered so far. We discuss the effort required to design and deliver the course, the technological solution developed, and the results obtained. We registered a moderate effort to create and run the course, ensured by internal staff from the University. The technological solution was a success: an integrated architecture combining well-established, well-tested open software. The completion rate was 3.3%, but the high success of this innovative learning experience was demonstrated by the active involvement of about 50% of the registered participants, that followed the course until the end. Lessons learned from this experience and future research on the field are also discussed.

Keywords: Massive Open Online Course, iMOOC, Pedagogical Model, Learning Effectiveness, Completion Rate, Cost Analysis.

Introduction

Massive Open Online Courses (MOOCs) were originated from the success of the now historical “An Introduction to AI” course designed by Sebastian Thrun and Peter Norvig at Stanford University in 2011 (Rodriguez, 2012). This event set in motion what would become the educational phenomenon of 2012 (Daniel, 2012). MOOCs can be best defined as courses designed for large numbers of participants that can be accessed by anyone anywhere as long as they have an internet connection, are open to everyone without entry qualifications, and offer a full/complete course experience online for free (Brouns et al., 2014). Although most MOOCs consist of relatively short video lectures and related content with feedback being managed either with peer-review and group collaboration or by automation (Alraimi, Zo & Ciganek, 2015), others provide a study guide or syllabus, include limited interaction with academic staff, have authentic activities and tests, and offer recognition options [formal and non-formal] (Brouns et al., 2014).

Literature recognizes two main types of MOOCs: Connectivist Massive Open Online Courses (cMOOCs) and eXtended Massive Open Online Courses (xMOOCs). cMOOCs are focused on the learning community and connections between members of the community across the web, rather than on course content or the instructor, and have been carried out with great success (Rodriguez, 2012). On the other hand, xMOOCs are based on scalability of provision, i.e., they are focused on giving many students access to an online course within the same platform. The style of learning is also different; xMOOCs tend to privilege individual studying, while cMOOCs focus on networked learning across several web tools and services (Conole, 2013, Rodriguez, 2012, Hew & Cheung, 2014). As George Siemens summarizes: cMOOCs focus on knowledge creation and generation whereas xMOOCs focus on knowledge duplication (Siemens, 2012). However, recently there's a trend, especially in Europe, towards the emergence of hybrid approaches which try to mix the potential of open socially-driven learning with structured learning paths. In fact, the disruptive nature of the connectivist approach to learning is quite difficult to articulate with typical institutional environments (Hernández, Morales, Mota &Teixeira, 2014).

MOOC providers are usually organizations that accept, select, and endorse contributions for open courses (e.g. from Universities) and make them available for student enrolment and participation. Often, providers develop their own platforms, emphasizing their features as a means to attract authors and participants. In addition to traditional learning management systems, MOOC platforms provide novel features, such as karma points, badges, and communication tools. Most MOOCs offer additional, usually paid, online and offline services, such as tutoring, course materials, and formal certification (Hew & Cheung, 2014).

MOOC courses have greatly grown in number and impact, and there is a large supply involving several of the most prestigious research universities in the USA. This movement generated European responses, either isolated, or in coordinated initiatives, but increasingly supported by National Governments and by the European Commission. One of the first European initiatives was Openup Ed (http://www.openuped.eu/; supported by the European Commission), which consisted in the release of a regional MOOC portal with a high number of course providers in different languages. Universidade Aberta (UAb), the Portuguese Distance Learning University, recently contributed to this movement by developing a specific MOOC platform as well as an innovative pedagogical model for MOOC, the iMOOC (Teixeira & Mota, 2013).

Increasing access to quality education, raising institutional visibility or building brand, increasing student recruitment, and improving or innovating pedagogy are the main goals of MOOCs for Higher Education institutions (Hollands & Tirthall, 2014). The desire to learn about a new topic or to extend current knowledge, the curiosity about MOOCs, for personal challenge, and the desire to collect as many completion certificates as possible, are the main reasons why students sign up for MOOCs (Hew & Cheung, 2014).

Typically, MOOCs not only integrate the connectivity of social networking, the facilitation of an acknowledged expert in a field of study, and a collection of freely accessible online resources, but most importantly, they build on the active engagement of “students” who self-organize their participation according to learning goals, prior knowledge and skills, and common interests (Hmoudova , 2014).

Nevertheless, Spector (2014) argues that MOOCs are overstating the likelihood of having a significant and sustained impact as an educational technology. MOOCs can be used effectively to support existing learning and instruction, but as stand-alone environments they fall short, unless they have a learning guidance, formative feedback, and overall assessment. MOOCs have sparked an intense new debate about the role of technology in shaping higher education and they pose inherent challenges in trying to automate instruction for an unlimited audience. If MOOCs are to achieve the revolutionary potential often claimed, the critics argue that MOOC advocates may want to reflect upon the origins of the movement and focus on developing and incorporating more creative and empowering forms of open online learning (Toven-Lindsey, Rhoads & Lozano, 2015).

MOOCs inspire many research questions. What are their business models? What educational and technological challenges are posed by this type of courses? Who bears the cost of the infrastructure, and the development and implementation of courses? Which platforms to use? What is the capacity of an installation for massive numbers of students? (Hollands & Tirthall, 2014). Should participant accomplishment in non-formal educational provision as massive online education be assessed using the same indicators as in formal education? In fact, completion rates have been extremely low in MOOCs (Alraimi et al., 2015), averaging 7 - 10%, which points to limitations in the design and execution of these courses (Toven-Lindsey et al., 2015; MaHew & Cheung, 2014; Hmoudova, 2014). But, independently of the design shortcomings of such MOOC provision, is the previous statement an appropriate assumption to make?

If decision makers are to decide rationally about engaging in MOOC production, it is critical to know whether MOOCs are both learning and cost effective in delivering quality education (Hollands & Tirthall, 2014). Thus, some of these MOOC developments seem to be operating in a policy vacuum regarding open learning in general. At some point, institutions will need to develop a clearer, more consistent strategy for open learning, in terms of how it can best be provided, how it calibrates with formal learning, and how open learning can be accommodated within the fiscal constraints of the institution, and then where MOOCs, other OER, and conventional for-credit online courses might fit with the strategy (Bates, 2015). For example dropout rate and behaviour of lurkers (silent and non-active participants) in such large groups of non-formal students represents one of the most puzzling issues for most educators in online courses (Rodriguez, 2012).

The aim of this paper is to assess the participants’ activity, benefits and impacts of the novel approach used on the pilot course of the iMOOC developed by UAb. This particular MOOC model includes an integration of platforms and an adaptation and development of pedagogical resources, which combined features of both c and x MOOC approaches. Within this work, the following research questions were addressed: i) is the chosen integrated technology together with the pedagogical iMOOC model efficient (in terms of connectivity and individual instructor design)?; ii) does the institution benefit with the iMOOC development? Lessons learned from this pilot are discussed to help decision-makers in higher education to rationally decide on effective strategies to deliver MOOCs.

Massive Open Online Courses at UAb (The iMOOC Model)

In 2007, UAb became a reference European institution in the area of advanced elearning and online learning as a result of the implementation of its Virtual Teaching Model, unprecedented in Portugal and developed by this institution.

As such, the pedagogical model for MOOCs at UAb (iMOOC) was built upon the four main pillars of UAb’s virtual pedagogical model: learner-centredness, flexibility, interaction, and digital inclusion (Pereira, Mendes, Morgado, Amante & Bidarra, 2008). The iMOOC pedagogical model can therefore be integrated as a variant of UAb’s institutional formal elearning pedagogical model, a specifically designed variant for MOOCs combining autonomous and self-directed learning with a strong social dimension. It articulates flexibility with the pacing necessary to help students “getting things done” in face of their pressing everyday commitments (Teixeira & Mota, 2013). UAb wanted to make learning available to as many as possible, but also to bring these people into the digital online environment, where a crucial part of modern life happens, thus helping curb the digital divide (Teixeira & Mota, 2014a). Most importantly, the iMOOC pedagogical model was designed in such a way as to facilitate the transition of students from non-formal education to formal education (Teixeira & Mota, 2014b).

One of the main characteristics of the iMOOC approach is the high level of transparency of the learning process. In fact, registration is only required for publishing in the institutional spaces but all content is open access, i.e. anyone can read it. Another innovative element is the introduction of a first module, usually a week long, for participants to get acquainted with the spaces, tools, and services, as well as with the processes of work and communication that will be used in the course. This “familiarization” process, already present in Universidade Aberta’s Pedagogical Model as a standalone module that all students have to take when they start their learning at UAb, has proved to be an essential component in student success and satisfaction regarding their online learning experience (Teixeira & Mota, 2014a).

Learning is learner-centered and based on the realization of activities. It should be evidenced through the creation of online artifacts (texts, videos, presentations, slidecasts, mind maps, mash-ups, etc.) that demonstrate the learner’s knowledge and competencies regarding the material studied. The learning process combines autonomous self-study and reflection with interaction with other participants in an open social context. Participants are expected to take an active role in and be responsible for their own learning, but also to actively engage in helping build a supporting learning community (Teixeira & Mota, 2013).

This same integration strategy was followed in the design of the virtual learning environment. Instead of building a custom platform, or adapting one of the recently created (and rather unstable) special-purpose platforms for MOOCs (like OpenMOOC, http://openmooc.org/, Gil & Martin, 2013, or EdX, http://code.edx.org), an integrated architecture combining well-established, well-tested software, with large bases of users was designed. Moodle (https://moodle.org/) and Elgg (http://elgg.org/) were chosen as components, fulfilling all pedagogic and technical requirements, and having the additional advantage of sharing the same underlying technologies (PHP language, databases, and operating systems).

Formal and formatted learning aspects (content access, course organization and sequencing, quizzes, assessment) are typically provided by learning management systems (LMS), like Moodle. Most elearning and online distance learning initiatives use a LMS for the development of closed, instructor-controlled courses. A MOOC is, by definition, an open course, and to provide an open environment for students to freely interact and share their opinions, thoughts, contents, web bookmarks, etc., a social networking platform was added, based on Elgg, integrated in a global environment, as seamless as possible, with Moodle. The integration was done using the IMS consortium specification for LTI (learning tools interoperability), in its basic version (Severance, 2010), providing user single sign-on and profile information sharing.

Moodle is an open-source learning management system, with over 71 million users globally, and used in more than 68,000 organizations all over the world, including major distance teaching universities. Moodle is built with “social constructionist pedagogy” in mind although it doesn’t force users to use that style of learning. It is organized around courses, each one with a list of enrolled students that typically are allowed access to that course (Moodle, 2015). In this way, Moodle works as a set of separate virtual classrooms, where resources are delivered and teaching/learning activities occur according to a plan and within the control of the instructor. For organizing contents and activities within an iMOOC, Moodle was chosen, as its strength lies precisely in providing a learning space with standard tools for displaying resources, performing activities, and assessing students. There are clearly distinct teacher and student roles, which determine what each user can do in a virtual classroom. Thus, Moodle functions both as an authoring tool for creating courses, and as a back office for tutor access and their interaction with students.

Elgg is an open-source, social networking platform also widely used where web-based communities can be created and managed. The central concepts are the user and her/his relations with other users. Contrary to Moodle, Elgg focuses on open interaction and promotes resource sharing. As MOOCs are much more student-centered than typical online learning (Hew & Cheung, 2014), a platform like Elgg was a natural choice for managing informal contact among learners, sharing resources and learning outcomes, and also for conducting formal learning activities in a more open way.

In iMOOC, Elgg functions as a front office, where students register and access their courses, with Moodle acting as a back office (Figure 1). The BLTI connection is configured to treat Moodle as a learning tool provider, and Elgg as a consumer, so that students can access the platform through Elgg and conduct all their activities by navigating seamlessly between both systems.

With this platform architecture, the iMOOC pedagogical model follows a mixed approach between the xMOOCs and the cMOOCs, aiming to gather the advantages of both MOOC types. On one hand it follows the xMOOC principles by using a designed LMS software (Moodle) that allows for a teaching model focused on the transmission of information, with high quality content delivery, computer-marked assessment (mainly for student feedback purposes), and automation of all key transactions between participants and the learning platform (Bates, 2014). On the other hand, by integrating in the Elgg social networking platform, the iMOOC environment allows for a strong emphasis on networking and in particular on strong content contributions from the participants themselves, according to the cMOOC principles. Indeed the principles of cMOOCs, like autonomy, diversity, interactivity, and openness, to allow the sharing of knowledge between participants (Bates, 2014) are put into practice in the iMOOC model.

Figure 1. iMOOC platform architecture.

The Pilot iMOOC on “Climate Change: The Context of Life Experiences”

The pilot course “Climate Change: the context of life experiences” was designed according to the iMOOC model. This course was chosen as the first MOOC experience at UAb, due to the broad interest of climate change issues, and the importance of education and engagement of the general  public as way forward to minimize and adapt to this worldwide environmental problem.

The aim of the course was to introduce the concept of climate change in the context of sustainable development, relating it to each individual’s life experience, such as in adaptation and mitigation of climate change effects, related to floods, water shortages, increase of sea level, coastal erosion, and changes of biodiversity. The course also aimed to help comprehending and critically analysing the information on climate change produced by a variety of organisations, under a diversity of perspectives, including its scientific, political, economic, and social perspective. The pilot course had a duration of 8 weeks, from the May 6th until July 1st, 2013, and was preceded by a five-day “boot camp” module. This component is quite a distinctive and unique feature of the iMOOC model as well as UAb's virtual pedagogical approach. The “bootcamp” module is meant for participants to get acquainted with the spaces, tools, and services, as well as with the processes of work and communication that will be used in the course (Teixeira & Mota, 2014b). Participants effort was 4 ECTS (European Credit Transfer System), but the course remained open after learning activities had been completed.

Two main learning tools were used: (1) the "Learning Guide" containing all the information concerning the course, which included course objectives, syllabus and developed skills, pedagogical model, learning resources, timetable of learning activities; and (2) the forum "News and Headlines" for publication of messages relating to course which established the main weekly study guidelines and feedbacks. The course was structured in 5 sequential topics: 1. What science tell us about climate change; 2. Economy and climate changes: 3. Politics of climate changes; 4. The sociological perspective of climate changes and 5. Perspectives integration within sustainable development. To ensure instructional quality, according to Margaryan, Bianco & Littlejohn (2014), several strategies were applied: each topic had an introductory short video presented by the course teachers, a summary of suggested activities and a document with detailed guidelines for their development, as well as learning materials (OER, in the iMOOC model) and other support materials. Each topic also had a discussion forum where participants could interact with each other. Weekly feedback by the teachers was drawn upon the tutors’ synthesis of subjects and debates on the Moodle forum and on the Elgg tweets and blog messages of the participants.  A video chat conference using Google Hangout (https://plus.google.com/hangouts) was conducted with an invited expert on climate change and the course teachers, moderated by the tutors, as another way to engage participants.

The participants were expected to study independently, exploring resources, searching and sharing for other relevant learning material, conducting activities and reflecting on their learning experience, also contributing themselves for the production of learning materials (texts, videos, presentations, mind maps, etc.), i.e. demonstrating their understanding of the topics and skills by producing artifacts as evidences of their successful learning. Participants were also expected to become actively involved in the interaction with other participants and to assume an active role in dialogue on the issues under study and contribute significantly to the knowledge being built, creating in this way a dynamic and collaborative community.

Upon completion of the course, the participants who wished to obtain a Certificate of Completion, had to be positively evaluated in the two course assignments, as defined in the course Learning Guide. Assignments were evaluated by a process of peer review (by three participant reviewers), based on a criteria matrix provided by the teachers responsible for the course, and through the use of the Workshop tool, provided by Moodle. Participants wishing to obtain a certificate of completion had to submit their assignment on the platform and to participate in the peer review process. Participants who wished to obtain formal accreditation of the course (i.e. 4 ECTS) had to register and to be positively evaluated on a written face-to-face exam (50% of final classification), as well as to be positively evaluated on a portfolio of activities developed during the course, assessed by the university staff (remaining 50%). As the pilot course clearly showed, the certification options embedded in the courses allow participants to manage their own learning experience, determining the kind of final output that is best suited for their learning needs. This includes the possibility of their non-formal learning experience being subject to formal accreditation (Teixeira & Mota, 2014b).

The team involved in the planning, developing, and teaching of the course had two experts in computer science, for platform development; two education experts, for developing the iMOOC pedagogical model; two environmental sciences experts, for scientific contents and course organisation; three tutors for students attendance, and four multimedia technical staff, for video development and production.

Methods

In order to assess the efficiency of the chosen integrated technology for the iMOOC, all events registered on both Elgg and Moodle platforms were analysed, to learn about the profile of the pilot iMOOC participants and to assess their active participation. Indicators for ”pageviews”, ”users” and ”posted messages” were collected in three timescales - day, month, and year, from January 1st 2013 until January 1st 2014, allowing an analysis of activity before, during, and after the course (since it remained open for visitors). A ”pageview” occurs each time a user clicks on a link. The “users” indicator measures the number of users that access the course in a certain time period. The “posted messages”  (both in Elgg and Moodle) count public messages, and is an indicator of participants’ involvement, as active users. Daily data were gathered for specific activities, to determine degree of participation, and message types were separated to find evolving patterns of behavior throughout the course. Since some users are more active than others, we also cross number of events and messages with the number of activity days, to distinguish sporadic participants from the ones that regularly follow the course (allowing us to assess the participants profile). Analysis of the number of visualisations of resources and activities throughout the course was also conducted and charted.

To evaluate the benefits to the University with the development of iMOOC, as well as its effectiveness as an educational tool, participants success was assessed by several indicators, the most traditional and formal one being completion rate. This was calculated on the basis of the number of participants with positive evaluation in the assignments, and face-to-face exam and portfolio. However, as stressed earlier, this is not a reliable indicator for non-formal modes of education delivery as MOOCs. The benefits for UAb were also assessed by analysing if iMOOC contributes to increase student recruitment, one of the aims that literature identifies as a driver for Higher Education Institutions involvement on MOOC provision. For that purpose, we tracked participants in the iMOOC that also enrolled in UAb formal courses, during the 18 months after completion of the iMOOC on Climate Change. Cost analyses of delivery of the pilot iMOOC were conducted using the ingredients method (Levin & McEwan, 2001) to estimate the costs of production and delivery of this pilot iMOOC. Personnel costs were assumed to represent 75% of total costs of the educational interventions (Levin & McEwan, 2001) and were calculated based on the salaries for each job category of the team personnel. Cost of re-running the iMOOC was also estimated.

Results

Analysis of the Platforms’ Events and Participant Activity

The course was advertised for two weeks, resulting in the enrolment of 1,024 participants. Further enrolments were blocked at this point, to ensure platform stability, and considering that this was a pilot experience.

The analysis of participant-generated events during a day in both platforms shows that most course activities occur between 9 AM and 11 PM, i.e. during daytime (Figure 2).

Figure 2. User events in both platforms by hour of day (PV - pageviews; USR - users; MSG - posted messages).

In addition, the number of users in Moodle and Elgg were similar throughout the day, indicating that most users that accessed Elgg, also entered Moodle through the integrated environment (see Figure 2). However, differences exist in pageviews: during the high activity period (from 9 AM until 11 PM) there are more page views in Elgg than in Moodle, while the opposite occurred in the low activity period. According to the number of posted messages, posting activity is always higher in Elgg than in Moodle, even though the number of users logged in is similar. In fact, messages are more easily posted in Elgg than in Moodle, since Elgg is a well-known social/collaborative platform.

Figure 3 shows the activity on the two platforms on a monthly basis. The course ran during months 5 and 6 (May and June), so the amount of actions before (course preparation and two week online registration), and after the course (visiting users) is much lower, compared to the period of the course proper.

Figure 3. Monthly user events in Moodle and Elgg before, during (months 5 and 6) and after the course (PV - pageviews; USR - users; MSG - posted messages).

During the course, the number of pageviews in Elgg was higher than in Moodle. After the course ended the opposite situation was observed: the number of pageviews in Moodle was consistently higher than in Elgg. This appears to confirm that during the course, interaction through social networks is key to drive the learning process and is also at the center of visitors’ attention, but after the course finishes, visitors focus on course materials rather than on content produced on the social media platform.

The daily analysis of events in both platforms during the course (Figure 4) shows, as before, that almost all users that entered Elgg also entered Moodle, but viewed more pages in Elgg. The activity in both platforms increased in the first days (from May 6th) of and then progressively lowered until the end of the course (July 1st). Nevertheless, several peaks of activity, mainly as Elgg pageviews (and less as Moodle pageviews) were observed during quizzes and assignments weeks (quizzes: May 21st to 27th and June 11th to 17th; assignments: June 4th to 10th and 18th to 24th), and the expert video conference (June 20th). Instructional messages of motivation conducted by teachers were also posted at the beginning of the week, when those peaks were observed.

Figure 4. Daily user events in Moodle and Elgg during the course (PV - pageviews; USR - users; MSG - posted messages).

Profiling the iMOOC participants in terms of number of pageviews and posted messages, accounted for both Moodle and Elgg (Table 1), we observed that all 246 users with one pageview didn’t post any messages, hence dropping out very soon. Similarly, most of the participants with 2-10 pageviews didn’t post any messages. Participants with the most number of posted messages were also the ones with most pageviews (although there were 40 participants in the group of 101-1000 page views with no posts). Globally, most of the pilot iMOOC participants (867, i.e. 85%) didn’t post any messages, regardless of pageviews.

Table 1

iMOOC participants in terms of numbers of pageviews and posted messages (both platforms).

Analysis of the participant profile of the course, in terms of activity days with pageviews and posted messages (Table 2), shows that participants with less than two days of pageviews didn’t post any messages. Participants who were more active during the course fall into the groups shown in bold in Table 2, i.e. the groups with greater global number of users (respectively 39 and 469). Still, a considerable number of participants (425) were active in terms of page viewing (11-1000) but  did not post messages.

Table 2

iMOOC participants in terms of the number of days of pageviews and posted messages (both platforms combined)

Considering the activity of participants in Moodle alone, activity decreased globally from the beginning of the course through topic 1 and 2 (Figure 5), but stabilised thereafter until the end of the course, as shown by the number of visualisations of the introductory videos, assignments and Moodle fora. Quizzes and complementary materials were the least visualised resources, as was the e-textbook, that covered all topics, and whose accesses probably were in the form of downloads.

>

Figure 5. Number of visualisations of activities/resources in Moodle, throughout the 5 topics of the pilot course. ‘Course’ refers to the common resources: “Learning guide”, “News and Headlines” forum, e-textbook, and introductory video.

Benefits and Impact of the iMOOC Pilot

The participants learning success on the iMOOC’s pilot on Climate Change may be assessed as the success rate in terms of user’s participation and of motivation to complete the training (Table 3). This was measured by the participants’ positive evaluation of the two assignments in the peer review process, as 3.1%. Even though there was a decrease in the delivery of assignments by the participants (and in peer review), the data appears to indicate that the four main assessment phases were important to retain participants throughout the course. Only 0.3% registered for the exam and from these, 67% were accredited for their training.

Table 3

iMOOC participants evaluation success

Many participants (50.1%) were already UAb students, prior to their registration in the pilot iMOOC. However, 1.5 % of these were newly-enrolled in formal courses at UAb. Still, 48.4% were unidentified and may represent a potential group for future recruitment.

If we broaden the definition of learner achievement in MOOCs to cover other objectives of participants who may be interested in just following the course but not participating in the assignments, we can calculate another success rate. In Table 2 we registered 497 (469+28) participants with a period of activity covering the course, that represent 48.5% of the participants.

Benefits to the institution may also be accounted for from the income from participants registering and accreditation processes (0.3% of participants), and from new enrollment in UAb non-formal and formal programs (1.6% of participants - 9 long life learning courses, 4 first degree and 3 post-graduates), which amounts to a minimum of 2,640 Euros in the pilot run, based on the iMOOC exam fee and UAb program’s tuition. Nevertheless, we can’t prove that these new enrollments were strictly and only related with iMOOC participation.

A clearer indicator of success is the high level of satisfaction shown in the user survey. Answers to the questionnaire revealed that 83% of the respondents rated the course quality as excellent or good (Teixeira & Mota, 2014b).

Costs of the Pilot iMOOC Production and Delivery

Based on the analysis of costs that were conducted for the production and delivery of this pilot iMOOC at UAb, personnel costs were estimated as 11,124 Euros (Table 4). As assumed methodologically, facilities, equipment and overhead were not calculated, but should represent 25% of the total costs (Levin & McEwan, 2001), hence the total cost of the pilot iMOOC was estimated as 14,830 Euros. The costs of development of the iMOOC pedagogical model and platform were also not included.

Table 4

Personnel Costs Estimated for the Pilot iMOOC.

Re-runs of the course require less design and development time. The personnel cost of the course re-run, was estimated on the basis of the personnel costs of the three instructors (32h each), two environmental experts, and two computer science experts (respectively 20, 10 and 20 hours each), as 1,900 Euros. Global cost is hence estimated as 2,530 Euros.

Discussion and Conclusions

The predictable university’s benefits from the iMOOC production and delivery included (based on Hew & Cheung, 2014, Hollands & Tirthall, 2014, Hmoudova, 2014): (i) increasing participants knowledge and awareness on the topic of Climate Change, (ii) raising of the institutions’ profile and visibility, (iii) increasing student recruitment, and most importantly, (v) improving or innovating pedagogy.

The success rate obtained in the pilot iMOOC developed at UAb, accounted for as number of participants who were issued a completion certificate, was within the usual low values reported for MOOCs (3.1%). Nevertheless 45.8% of the participants were active during the course, suggesting that MOOCs can be useful learning opportunities, though not necessarily for getting the final approval and certificate. On this case study the social network platform Elgg was used, where a strong and active participation was observed and where the periodic (weekly) feedback and instructional messages by the teachers and continuous monitoring by tutors was conducted. Also iMOOC participants frequented a five-day “boot camp” module prior to the course for familiarization with the online environment and for initiating social bonding, following the best practices specified on Pereira et al. (2008). Furthermore, on the pilot iMOOC, the use of introductory videos in each topic and a web conference were also important tools as participant motivators. Margaryan et al. (2015) randomly selected a sample of MOOCs and concluded that although most MOOCs are well-packaged, their instructional design quality was low which is an important variable in the overall quality of a course. These authors advised for the application of instructional design principles (such as integration and activation) in particular, since MOOCs are still in the experimental phase.  So, the use of instructional measures and activities could justify the high active participation rates found in the pilot iMOOC. Also, it is well known that in online distance learning, the establishment of a strong sense of community among learners encourages higher levels of satisfaction, persistence, engagement, and cognition (Toven-Lindsey et al., 2015, Garrison, 2011, Morris, Finnegan & Wu, 2005). The iMOOC appears to be able to articulate this with a massive and open environment, thus managing to scale up online collaboration between learners. In addition the common design features of cMOOC (Bates, 2014) were guaranteed in the iMOOC model, based on learning tools, instructions, assessment, and materials.

The peer assessment process was used for students’ assignments evaluation, using the workshop tool in Moodle. Peer assessment potentially enables large classes to offer assignments that are impractical to grade automatically. Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints (Kulkarni, Wei, Le, Chia, Papadopoulos, Cheng, Koller & Klemmer, 2013). However, peer assessment imposes a particular schedule on class and limits student flexibility, which could explain the low number of participants that deliver and peer-assessed the assignments. Nevertheless, this assessment process is under several research improvements as described in Kulkarni et al. (2013).

The analysis of this pilot iMOOC experiment suggest than iMOOC courses would have higher completion rate if their duration was shorter. This is hinted by the decreased activity of participants along the course. If only one peer review activity existed, more participants might find the time needed to make it. In accordance the iMOOC model in its current version has decreased the suggested duration of the courses to six weeks. However, as pointed out, and as the experience of UAb and other MOOC institutional providers proves, learner satisfaction and, most importantly, the success of the learning experience, cannot be measured by completion rates. This cannot be a quality indicator for non-formal open forms of education delivery because participants in MOOCs are typically non-homogeneous groups of learners with quite different backgrounds, expectations and ultimately aiming at different learning outcomes (Hernández et al., 2014).

About 1.6% of new students enrolled in UAb, both on formal and non-formal programs, had been registered participants for the first time at UAb with the pilot iMOOC. Furthermore, the pilot iMOOC served for raising institutional visibility. Perceived reputation and perceived openness are the two strongest predictors to explain MOOCs continuance intention to use (Alraimi & Ciganek, 2014). Nevertheless, individual instructors designing of MOOCs need to make sure that the design is consistent in terms of educational philosophy, and be clear as to why they are opting for a MOOC rather than a conventional online course (Bates, 2015).

Overall the cost of developing and delivering this pilot iMOOC at UAb vary between 14,830 Euros (for the pilot run) and 2,530 Euros (for a course re-run). This global costs can be estimated lower if we account for registering and accreditation processes of the participants and for student recruitment, which was estimated as a minimum of 2,640 Euros. Also those costs are internal, since no team member received any financial compensation and the course was developed in the framework of a research project. In fact, staff costs have been calculated as a part of salaries of the university staff, researchers and students work, university facilities and equipment, and no external funding was used for course production and delivery. Nevertheless, Hollands and Tirthali (2014) argue that MOOC production is often more costly than the development of regular online courses, the ability to scale MOOCs and the absence of associated student supports results in a dramatically lower cost per completer. Considering that MOOCs can help achieve other objectives not generally addressed by regular online courses, including branding, global reach, and large scale research, MOOCs would appear to be a wise use of resources, if only the costs could be recovered through tuition or other fees. However, as stated previously, it is arguable that course completion per se is not a satisfactory measure of effectiveness and that MOOCs should be judged on the quality and quantity of learning that takes place.

The lack of a clearly defined business model associated with MOOCs, or at least those run for profit, could lead to their demise in the near future. This could be significant for universities which view joining the MOOC party as a way of dealing with shrinking budgets and rising costs (Hmoudova, 2014).  Bernhard, Bittle, Van Der Vlies, Bettoni & Roth, (2013) proposes a new business model where the learning process of all the participants can be enriched, and the services provided by teachers can be funded through a specific payment system.       

Other assessments to evaluate the real effectiveness of this pilot and possible re-runs should be conducted. Other learning analytics can also be used to track better online learners more likely to drop out. Tabaa and Medouri (2013) proposes a specific Learning Analytics System for MOOCs that  analyses, with low latency, students’ behaviour with the intent of increasing the impact of analytics on teaching and learning in such environments. In addition, longitudinal studies tracking post-MOOC outcomes such as sequences of courses taken, professional certifications obtained, or job opportunities received would help assess the longer term economic value of participating in these courses and allow for cost-benefit analyses to estimate the overall returns to society of investing in MOOC creation (Hollands & Tirthali, 2014).

Nevertheless, several key challenges of teaching MOOCs are still on the table, namely: difficulty in evaluating students’ work; having a sense of speaking into a vacuum due to the absence of student immediate feedback; being burdened by the heavy demands of time and money; and encountering a lack of student participation in online forums. Issues that have yet to be fully resolved are the quality of MOOC education, and the assessment of student work (Hew & Cheung, 2014). Research needs to come up with theoretical underpinnings that will explain factors related to social aspects in MOOCs that have a completely new context and offer practical guidance of course design and instruction (Gašević, Kovanović, Joksimovićm & Siemens, 2014). The design of MOOCs continues to evolve, with all kinds of variations and challenges to address, e.g. Yousef, Chatti, Schroeder, Wosnitza & Jakobs (2014) discuss the use of other MOOCs, such as sMOOCs that represent small open online courses and bMOOCs that represent MOOCs that are blended with on-campus teaching.

In this paper we presented the largest Portuguese MOOC to date, on the topic of climate change, with more than one thousand enrolments. This course piloted what appears to be a successful innovative pedagogical model - the iMOOC, where the advantages of c and x MOOC were successfully joined, such as collaborative group work, paced instruction, and a creditable online course. It was clear from the technological point of view, the advantage of using open source platforms, reducing the cost of development, and having at the same time, a large set of stable and well tested features. The technology aspects were never a constraint factor.

The further development of the iMOOC experience has been assured by its influence in other new hybrid approaches emerging in Europe, as the sMOOC model developed in the framework of the ECO project (Brouns et al., 2014). Inspired by the iMOOC, the pedagogical approach used in the sMOOC not only is learner-centred and promotes independent learning, as also supports adaptive learning strategies and ubiquitous, pervasive, and contextualized learning. As a result of this, the courses have the potential to adapt to the changing intentions of participants during its duration.

Moreover, UAb's participation in a number of leading EU-funded projects, namely the above mentioned ECO (http://ecolearning.eu) and EMMA (http://www.europeanmoocs.eu), has secured resources and funding for medium-term improvement of the iMOOC experience in terms of technology, pedagogies, and content development, and also in the definition of appropriate business models. Future long term research is indeed needed to answer the various questions highlighted in this work.

Acknowledgments

The authors we would like to express their gratitude to José Mota, co-author of the iMOOC pedagogical model, in supporting the course pedagogical organization, Rui Nicolau, for statistics and students information, Helder Matta for multimedia support and Maria João Spilker, Paulo Simões and Antje Disterheft, for course tutorial support.

References

Alraimi, K. M., Zo, H., & Ciganek, A. P. (2014).  Understanding the MOOCs continuance: The role of openness and reputation. Computers & Education, 80, 28 - 38.

Bates, A. (2015). Variations in MOOC designs. BC Open Texbooks. Available at http://opentextbc.ca/teachinginadigitalage/chapter/section-7-4-design-models-for-moocs/

Bernhard, W., Bittle, N., Van Der Vlies, S., Bettoni, M. & Roth, N. (2013). A MOOC business Model. Procedia - Social and Behavioral Sciences, 106, 2931 - 2937.

Brouns, F., Mota, J., Morgado. L., Jansen, D., Fano, S., Silva, A., & Teixeira, A. (2014). A networked learning framework for effective MOOC design: The ECO project approach. Proceedings Eighth EDEN Research Workshop - EDEN RW8, 161-171. Oxford, UK.

Conole, G. (2013). MOOCs as disruptive technologies: Strategies for enhancing the learner experience and quality of MOOCs. Revista de Educación a Distancia, 39, 1 – 17. Retrieved from http://www.um.es/ead/red/39/conole.pdf        

Garrison, D. R. (2011). E-learning in the 21st century a framework for research and practice. New York: Routledge.

Gašević, D., Kovanović, V., Joksimovićm S. & Siemens, G. (2014).  Where is research on massive open online courses headed? A data analysis of the MOOC research initiative. The International Review of Research in Open and Distance Learning, 15(5), 134 - 176.

Gil, L. & Martin, S. (2013). The OpenMOOC project. Platform based on free software for an open education. TERENA Networking Conference, Maastricht. Retrieved from https://tnc2013.terena.org/getfile/229

Hernández, R., Morales, M., Mota, J. &Teixeira, A. (2014).Promoting engagement in MOOCs through social collaboration: Common lessons from the pedagogical models of Universidad Galileo and Universidade Aberta. Proceedings Eight EDEN Research Workshop - EDEN RW8, 131-142. Oxford, UK.

Hew, K. F. & Cheung, W. S. (2014). Students' and instructors' use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58.

Hollands, F.M. & Tirthali, D. (2014). Resource requirements and costs of developing and delivering MOOCs, The International Review of Research in Open and Distance Learning, 15(5), 113 - 133.

Hmoudova, D. (2014). MOOCs motivation and communication in the cyber learning environment. Procedia - Social and Behavioral Sciences, 131. 29 – 34.

 Levin, H.M. & McEwan, P.J. (2001). Cost-effectiveness analysis: Methods and Applications (2nd ed.) Thousand Oaks, CA: Sage Publications.

Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D. & Klemmer, S. R. (2013). Peer and self-assessment in massive online classes. ACM Transactions on Computer-Human Interaction, 20(6), 33, 1 – 31.

Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77-83.

Moodle (2015). About Moodle - Retrieved from https://docs.moodle.org/28/en/About_Moodle.

Morris, L. V., Finnegan, C. & Wu, S. S. (2005). Tracking student behavior, persistence, and achievement in online courses. The Internet and Higher Education, 8(3), 221–231.

Pereira, A., Mendes, A.Q., Morgado, L., Amante, L., &  Bidarra, J. (2008). Universidade Aberta´s pedagogical model for distance education: A university for the future. Lisboa: Universidade Aberta, p. 109.   

Rodriguez, C. O. (2012). MOOCs and the AI‐Stanford like courses: Two successful and distinct course formats for Massive Open Online Courses, European Journal of Open, Distance and eLearning. Retrieved from  http://www.eric.ed.gov/PDFS/EJ982976.pdf

Severance, C. (2010). IMS global learning tools interoperability basic LTI implementation guide v1.0. IMS Global Learning Consortium. Retrieved from http://www.imsglobal.org/lti/index.html.

Siemens, George (2012). MOOCs are really a platform. elearnsapce. Retrieved from http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform.

Spector, M. (2014). Emerging educational technologies: Tensions and synergy. Journal of King Saud University – Computer and Information Sciences, 26, 5–10.

Tabaa, Y. & Medouri, A. (2013). ASyM: A Learning Analytics System for MOOCs. (IJACSA) International Journal of Advanced Computer Science and Applications, 4(5), 113 - 119.

Teixeira, A. M. & Mota, J. (2014a). A proposal for the methodological design of collaborative language MOOCs. In Elena Martín-Monje, Elena Bárcena, Language MOOCs: Providing Learning, Transcending Boundaries, pp. 33-47. De Gruyter Open.

Teixeira, A. M. & Mota, J. M. ( 2014b). The iMOOC Pedagogical Model: Bridging the gap between non-formal and formal education. Actas del V Congreso Internacional sobre Calidad y Accesibilidad de la Formación Virtual - CAFVIR 2014, 512-517. Antigua Guatemala, Guatemala.

Teixeira, A., & Mota, J. (2013). Innovation and openness through MOOCs: Universidade Aberta's pedagogical model for non-formal online courses. In Proceedings EDEN Conference 2013, 479-488. Oslo, Norway.

Toven-Lindsey, B., A., Rhoads, R. A. & Lozano, J. B. (2015). Virtually unlimited classrooms: Pedagogical practices in massive open online courses. Internet and Higher Education, 24, 1–12.

Yousef, A M. F., Chatti, M. A., Schroeder, U., Wosnitza, M. & Jakobs, H. (2014). MOOCs: A review of the state-of-the-art proceedings of 6th international conference on computer supported education – CSEDU 2014, Barcelona, Spain.

iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience by José Coelho1, António Teixeira1, Paula Bacelar Nicolau1, Sandra Caeiro1, Vitor Rocio1 is licensed under a Creative Commons Attribution 4.0 International License.