International Review of Research in Open and Distributed Learning iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience

MOOCs are a recent phenomenon, although given their impact, they have been subject to a large debate. Several questions have been raised by researchers and educators alike regarding their sustainability, both economically and as an efficient mode of education provision. In this paper we contribute to this discussion by presenting a case study of the MOOC on Lived Experiences of Climate Change, which piloted the iMOOC pedagogical model developed at Universidade Aberta (UAb), the Portugese Distance Learning University. The iMOOC is a hybrid model which


Résumé de l'article
MOOCs are a recent phenomenon, although given their impact, they have been subject to a large debate.Several questions have been raised by researchers and educators alike regarding their sustainability, both economically and as an efficient mode of education provision.In this paper we contribute to this discussion by presenting a case study of the MOOC on Lived Experiences of Climate Change, which piloted the iMOOC pedagogical model developed at Universidade Aberta (UAb), the Portugese Distance Learning University.The iMOOC is a hybrid model which incorporates elements from existing MOOCs but adds other features drawn from UAb's experience with online learning and aims at better integrating in the larger context of the institutional pedagogical culture.The iMOOC implied also an integration of platforms -Moodle and Elgg.The pilot course had more than one thousand registrations, and it was the largest MOOC course on Portuguese language delivered so far.We discuss the effort required to design and deliver the course, the technological solution developed, and the results obtained.We registered a moderate effort to create and run the course, ensured by internal staff from the University.The technological solution was a success: an integrated architecture combining well-established, well-tested open software.The completion rate was 3.3%, but the high success of this innovative learning experience was demonstrated by the active involvement of about 50% of the registered participants, that followed the course until the end.Lessons learned from this experience and future research on the field are also discussed.

Introduction
Massive Open Online Courses (MOOCs) were originated from the success of the now historical "An Introduction to AI" course designed by Sebastian Thrun and Peter Norvig at Stanford University in 2011 (Rodriguez, 2012).This event set in motion what would become the educational phenomenon of 2012 (Daniel, 2012).MOOCs can be best defined as courses designed for large numbers of participants that can be accessed by anyone anywhere as long as they have an internet connection, are open to everyone without entry qualifications, and offer a full/complete course experience online for free (Brouns et al., 2014).Although most MOOCs consist of relatively short video lectures and related content with feedback being managed either with peer-review and group collaboration or by automation (Alraimi, Zo & Ciganek, 2015), others provide a study guide or syllabus, include limited interaction with academic staff, have authentic activities and tests, and offer recognition options [formal and non-formal] (Brouns et al., 2014).
Literature recognizes two main types of MOOCs: Connectivist Massive Open Online Courses (cMOOCs) and eXtended Massive Open Online Courses (xMOOCs).cMOOCs are focused on the learning community and connections between members of the community across the web, rather than on course content or the instructor, and have been carried out with great success (Rodriguez, 2012).On the other hand, xMOOCs are based on scalability of provision, i.e., they are focused on giving many students access to an online course within the same platform.The style of learning is also different; xMOOCs tend to privilege individual studying, while cMOOCs focus on networked learning across several web tools and services (Conole, 2013, Rodriguez, 2012, Hew & Cheung, 2014).As George Siemens summarizes: cMOOCs focus on knowledge creation and generation whereas xMOOCs focus on knowledge duplication (Siemens, 2012).However, recently there's a trend, especially in Europe, towards the emergence of hybrid approaches which try to mix the potential of open socially-driven learning with structured learning paths.In fact, the disruptive 155 want to reflect upon the origins of the movement and focus on developing and incorporating more creative and empowering forms of open online learning (Toven-Lindsey, Rhoads & Lozano, 2015).
MOOCs inspire many research questions.What are their business models?What educational and technological challenges are posed by this type of courses?Who bears the cost of the infrastructure, and the development and implementation of courses?Which platforms to use?What is the capacity of an installation for massive numbers of students?(Hollands & Tirthall, 2014).Should participant accomplishment in non-formal educational provision as massive online education be assessed using the same indicators as in formal education?In fact, completion rates have been extremely low in MOOCs (Alraimi et al., 2015), averaging 7 -10%, which points to limitations in the design and execution of these courses (Toven-Lindsey et al., 2015;MaHew & Cheung, 2014;Hmoudova, 2014).But, independently of the design shortcomings of such MOOC provision, is the previous statement an appropriate assumption to make?
If decision makers are to decide rationally about engaging in MOOC production, it is critical to know whether MOOCs are both learning and cost effective in delivering quality education (Hollands & Tirthall, 2014).Thus, some of these MOOC developments seem to be operating in a policy vacuum regarding open learning in general.At some point, institutions will need to develop a clearer, more consistent strategy for open learning, in terms of how it can best be provided, how it calibrates with formal learning, and how open learning can be accommodated within the fiscal constraints of the institution, and then where MOOCs, other OER, and conventional for-credit online courses might fit with the strategy (Bates, 2015).For example dropout rate and behaviour of lurkers (silent and non-active participants) in such large groups of non-formal students represents one of the most puzzling issues for most educators in online courses (Rodriguez, 2012).
The aim of this paper is to assess the participants' activity, benefits and impacts of the novel approach used on the pilot course of the iMOOC developed by UAb.This particular MOOC model includes an integration of platforms and an adaptation and development of pedagogical resources, which combined features of both c and x MOOC approaches.Within this work, the following research questions were addressed: i) is the chosen integrated technology together with the pedagogical iMOOC model efficient (in terms of connectivity and individual instructor design)?; ii) does the institution benefit with the iMOOC development?Lessons learned from this pilot are discussed to help decision-makers in higher education to rationally decide on effective strategies to deliver MOOCs.

Massive Open Online Courses at UAb (The iMOOC Model)
As such, the pedagogical model for MOOCs at UAb (iMOOC) was built upon the four main pillars of UAb's virtual pedagogical model: learner-centredness, flexibility, interaction, and digital inclusion (Pereira, Mendes, Morgado, Amante & Bidarra, 2008).The iMOOC pedagogical model can therefore be integrated as a variant of UAb's institutional formal elearning pedagogical model, a specifically designed variant for MOOCs combining autonomous and self-directed learning with a strong social dimension.It articulates flexibility with the pacing necessary to help students "getting things done" in face of their pressing everyday commitments (Teixeira & Mota, 2013).UAb wanted to make learning available to as many as possible, but also to bring these people into the digital online environment, where a crucial part of modern life happens, thus helping curb the digital divide (Teixeira & Mota, 2014a).Most importantly, the iMOOC pedagogical model was designed in such a way as to facilitate the transition of students from nonformal education to formal education (Teixeira & Mota, 2014b).
One of the main characteristics of the iMOOC approach is the high level of transparency of the learning process.In fact, registration is only required for publishing in the institutional spaces but all content is open access, i.e. anyone can read it.Another innovative element is the introduction of a first module, usually a week long, for participants to get acquainted with the spaces, tools, and services, as well as with the processes of work and communication that will be used in the course.This "familiarization" process, already present in Universidade Aberta's Pedagogical Model as a standalone module that all students have to take when they start their learning at UAb, has proved to be an essential component in student success and satisfaction regarding their online learning experience (Teixeira & Mota, 2014a).
Learning is learner-centered and based on the realization of activities.It should be evidenced through the creation of online artifacts (texts, videos, presentations, slidecasts, mind maps, mashups, etc.) that demonstrate the learner's knowledge and competencies regarding the material studied.The learning process combines autonomous self-study and reflection with interaction with other participants in an open social context.Participants are expected to take an active role in and be responsible for their own learning, but also to actively engage in helping build a supporting learning community (Teixeira & Mota, 2013).This same integration strategy was followed in the design of the virtual learning environment.
Instead of building a custom platform, or adapting one of the recently created (and rather unstable) special-purpose platforms for MOOCs (like OpenMOOC, http://openmooc.org/,Gil & Martin, 2013, or EdX, http://code.edx.org), an integrated architecture combining wellestablished, well-tested software, with large bases of users was designed.Moodle (https://moodle.org/)and Elgg (http://elgg.org/)were chosen as components, fulfilling all 157 pedagogic and technical requirements, and having the additional advantage of sharing the same underlying technologies (PHP language, databases, and operating systems).
Formal and formatted learning aspects (content access, course organization and sequencing, quizzes, assessment) are typically provided by learning management systems (LMS), like Moodle.
Most elearning and online distance learning initiatives use a LMS for the development of closed, instructor-controlled courses.A MOOC is, by definition, an open course, and to provide an open environment for students to freely interact and share their opinions, thoughts, contents, web bookmarks, etc., a social networking platform was added, based on Elgg, integrated in a global environment, as seamless as possible, with Moodle.The integration was done using the IMS consortium specification for LTI (learning tools interoperability), in its basic version (Severance, 2010), providing user single sign-on and profile information sharing.
Moodle is an open-source learning management system, with over 71 million users globally, and used in more than 68,000 organizations all over the world, including major distance teaching universities.Moodle is built with "social constructionist pedagogy" in mind although it doesn't force users to use that style of learning.It is organized around courses, each one with a list of enrolled students that typically are allowed access to that course (Moodle, 2015).In this way, Moodle works as a set of separate virtual classrooms, where resources are delivered and teaching/learning activities occur according to a plan and within the control of the instructor.For organizing contents and activities within an iMOOC, Moodle was chosen, as its strength lies precisely in providing a learning space with standard tools for displaying resources, performing activities, and assessing students.There are clearly distinct teacher and student roles, which determine what each user can do in a virtual classroom.Thus, Moodle functions both as an authoring tool for creating courses, and as a back office for tutor access and their interaction with students.
Elgg is an open-source, social networking platform also widely used where web-based communities can be created and managed.The central concepts are the user and her/his relations with other users.Contrary to Moodle, Elgg focuses on open interaction and promotes resource sharing.As MOOCs are much more student-centered than typical online learning (Hew & Cheung, 2014), a platform like Elgg was a natural choice for managing informal contact among learners, sharing resources and learning outcomes, and also for conducting formal learning activities in a more open way.
In iMOOC, Elgg functions as a front office, where students register and access their courses, with Moodle acting as a back office (Figure 1).The BLTI connection is configured to treat Moodle as a learning tool provider, and Elgg as a consumer, so that students can access the platform through Elgg and conduct all their activities by navigating seamlessly between both systems.
With this platform architecture, the iMOOC pedagogical model follows a mixed approach between the xMOOCs and the cMOOCs, aiming to gather the advantages of both MOOC types.On one 158 hand it follows the xMOOC principles by using a designed LMS software (Moodle) that allows for a teaching model focused on the transmission of information, with high quality content delivery, computer-marked assessment (mainly for student feedback purposes), and automation of all key transactions between participants and the learning platform (Bates, 2014).On the other hand, by integrating in the Elgg social networking platform, the iMOOC environment allows for a strong emphasis on networking and in particular on strong content contributions from the participants themselves, according to the cMOOC principles.Indeed the principles of cMOOCs, like autonomy, diversity, interactivity, and openness, to allow the sharing of knowledge between participants (Bates, 2014) are put into practice in the iMOOC model.The participants were expected to study independently, exploring resources, searching and sharing for other relevant learning material, conducting activities and reflecting on their learning experience, also contributing themselves for the production of learning materials (texts, videos, presentations, mind maps, etc.), i.e. demonstrating their understanding of the topics and skills by producing artifacts as evidences of their successful learning.Participants were also expected to become actively involved in the interaction with other participants and to assume an active role in dialogue on the issues under study and contribute significantly to the knowledge being built, creating in this way a dynamic and collaborative community.
Upon completion of the course, the participants who wished to obtain a Certificate of Completion, had to be positively evaluated in the two course assignments, as defined in the course Learning Guide.Assignments were evaluated by a process of peer review (by three participant reviewers), 160 based on a criteria matrix provided by the teachers responsible for the course, and through the use of the Workshop tool, provided by Moodle.Participants wishing to obtain a certificate of completion had to submit their assignment on the platform and to participate in the peer review process.Participants who wished to obtain formal accreditation of the course (i.e. 4 ECTS) had to register and to be positively evaluated on a written face-to-face exam (50% of final classification), as well as to be positively evaluated on a portfolio of activities developed during the course, assessed by the university staff (remaining 50%).As the pilot course clearly showed, the certification options embedded in the courses allow participants to manage their own learning experience, determining the kind of final output that is best suited for their learning needs.This includes the possibility of their non-formal learning experience being subject to formal accreditation (Teixeira & Mota, 2014b).
The team involved in the planning, developing, and teaching of the course had two experts in computer science, for platform development; two education experts, for developing the iMOOC pedagogical model; two environmental sciences experts, for scientific contents and course organisation; three tutors for students attendance, and four multimedia technical staff, for video development and production.

Methods
In order to assess the efficiency of the chosen integrated technology for the iMOOC, all events registered on both Elgg and Moodle platforms were analysed, to learn about the profile of the pilot iMOOC participants and to assess their active participation.Indicators for "pageviews", "users" and "posted messages" were collected in three timescales -day, month, and year, from January 1 st 2013 until January 1 st 2014, allowing an analysis of activity before, during, and after the course (since it remained open for visitors).A "pageview" occurs each time a user clicks on a link.The "users" indicator measures the number of users that access the course in a certain time period.The "posted messages" (both in Elgg and Moodle) count public messages, and is an indicator of participants' involvement, as active users.Daily data were gathered for specific activities, to determine degree of participation, and message types were separated to find evolving patterns of behavior throughout the course.Since some users are more active than others, we also cross number of events and messages with the number of activity days, to distinguish sporadic participants from the ones that regularly follow the course (allowing us to assess the participants' profile).Analysis of the number of visualisations of resources and activities throughout the course was also conducted and charted.
To evaluate the benefits to the University with the development of iMOOC, as well as its effectiveness as an educational tool, participants success was assessed by several indicators, the most traditional and formal one being completion rate.This was calculated on the basis of the number of participants with positive evaluation in the assignments, and face-to-face exam and 161 portfolio.However, as stressed earlier, this is not a reliable indicator for non-formal modes of education delivery as MOOCs.The benefits for UAb were also assessed by analysing if iMOOC contributes to increase student recruitment, one of the aims that literature identifies as a driver for Higher Education Institutions involvement on MOOC provision.For that purpose, we tracked participants in the iMOOC that also enrolled in UAb formal courses, during the 18 months after completion of the iMOOC on Climate Change.Cost analyses of delivery of the pilot iMOOC were conducted using the ingredients method (Levin & McEwan, 2001) to estimate the costs of production and delivery of this pilot iMOOC.Personnel costs were assumed to represent 75% of total costs of the educational interventions (Levin & McEwan, 2001) and were calculated based on the salaries for each job category of the team personnel.Cost of re-running the iMOOC was also estimated.

Analysis of the Platforms' Events and Participant Activity
The course was advertised for two weeks, resulting in the enrolment of 1,024 participants.Further enrolments were blocked at this point, to ensure platform stability, and considering that this was a pilot experience.
The analysis of participant-generated events during a day in both platforms shows that most course activities occur between 9 AM and 11 PM, i.e. during daytime (Figure 2).
162 In addition, the number of users in Moodle and Elgg were similar throughout the day, indicating that most users that accessed Elgg, also entered Moodle through the integrated environment (see Figure 2).However, differences exist in pageviews: during the high activity period (from 9 AM until 11 PM) there are more page views in Elgg than in Moodle, while the opposite occurred in the low activity period.According to the number of posted messages, posting activity is always higher in Elgg than in Moodle, even though the number of users logged in is similar.In fact, messages are more easily posted in Elgg than in Moodle, since Elgg is a well-known social/collaborative platform.
Figure 3 shows the activity on the two platforms on a monthly basis.The course ran during months 5 and 6 (May and June), so the amount of actions before (course preparation and two week online registration), and after the course (visiting users) is much lower, compared to the period of the course proper.During the course, the number of pageviews in Elgg was higher than in Moodle.After the course ended the opposite situation was observed: the number of pageviews in Moodle was consistently higher than in Elgg.This appears to confirm that during the course, interaction through social networks is key to drive the learning process and is also at the center of visitors' attention, but after the course finishes, visitors focus on course materials rather than on content produced on the social media platform.

163
The daily analysis of events in both platforms during the course (Figure 4) shows, as before, that almost all users that entered Elgg also entered Moodle, but viewed more pages in Elgg.The activity in both platforms increased in the first days (from May 6 th ) of and then progressively lowered until the end of the course (July 1 st ).Nevertheless, several peaks of activity, mainly as Elgg pageviews (and less as Moodle pageviews) were observed during quizzes and assignments weeks (quizzes: May 21 st to 27 th and June 11 th to 17 th ; assignments: June 4 th to 10 th and 18 th to 24 th ), and the expert video conference (June 20 th ).Instructional messages of motivation conducted by teachers were also posted at the beginning of the week, when those peaks were observed.Profiling the iMOOC participants in terms of number of pageviews and posted messages, accounted for both Moodle and Elgg (Table 1), we observed that all 246 users with one pageview didn't post any messages, hence dropping out very soon.Similarly, most of the participants with 2-10 pageviews didn't post any messages.Participants with the most number of posted messages were also the ones with most pageviews (although there were 40 participants in the group of 101-1000 page views with no posts).Globally, most of the pilot iMOOC participants (867, i.e. 85%) didn't post any messages, regardless of pageviews.165 Considering the activity of participants in Moodle alone, activity decreased globally from the beginning of the course through topic 1 and 2 (Figure 5), but stabilised thereafter until the end of the course, as shown by the number of visualisations of the introductory videos, assignments and Moodle fora.Quizzes and complementary materials were the least visualised resources, as was the e-textbook, that covered all topics, and whose accesses probably were in the form of downloads.

Benefits and Impact of the iMOOC Pilot
The participants learning success on the iMOOC's pilot on Climate Change may be assessed as the success rate in terms of user's participation and of motivation to complete the training (Table 3).
This was measured by the participants' positive evaluation of the two assignments in the peer review process, as 3.1%.Even though there was a decrease in the delivery of assignments by the participants (and in peer review), the data appears to indicate that the four main assessment phases were important to retain participants throughout the course.Only 0.3% registered for the exam and from these, 67% were accredited for their training.Benefits to the institution may also be accounted for from the income from participants registering and accreditation processes (0.3% of participants), and from new enrollment in UAb non-formal and formal programs (1.6% of participants -9 long life learning courses, 4 first degree and 3 post-graduates), which amounts to a minimum of 2,640 Euros in the pilot run, based on the iMOOC exam fee and UAb program's tuition.Nevertheless, we can't prove that these new enrollments were strictly and only related with iMOOC participation.
A clearer indicator of success is the high level of satisfaction shown in the user survey.Answers to the questionnaire revealed that 83% of the respondents rated the course quality as excellent or good (Teixeira & Mota, 2014b).

Costs of the Pilot iMOOC Production and Delivery
Based on the analysis of costs that were conducted for the production and delivery of this pilot iMOOC at UAb, personnel costs were estimated as 11,124 Euros (Table 4).As assumed methodologically, facilities, equipment and overhead were not calculated, but should represent 25% of the total costs (Levin & McEwan, 2001), hence the total cost of the pilot iMOOC was estimated as 14,830 Euros.The costs of development of the iMOOC pedagogical model and platform were also not included.

167
Table 4 Personnel Costs Estimated for the Pilot iMOOC.encountering a lack of student participation in online forums.Issues that have yet to be fully resolved are the quality of MOOC education, and the assessment of student work (Hew & Cheung, 2014).Research needs to come up with theoretical underpinnings that will explain factors related to social aspects in MOOCs that have a completely new context and offer practical guidance of course design and instruction (Gašević, Kovanović, Joksimovićm & Siemens, 2014).The further development of the iMOOC experience has been assured by its influence in other new hybrid approaches emerging in Europe, as the sMOOC model developed in the framework of the ECO project (Brouns et al., 2014).Inspired by the iMOOC, the pedagogical approach used in the sMOOC not only is learner-centred and promotes independent learning, as also supports adaptive learning strategies and ubiquitous, pervasive, and contextualized learning.As a result of this, the courses have the potential to adapt to the changing intentions of participants during its duration.
Moreover, UAb's participation in a number of leading EU-funded projects, namely the above mentioned ECO (http://ecolearning.eu) and EMMA (http://www.europeanmoocs.eu),has secured resources and funding for medium-term improvement of the iMOOC experience in terms of technology, pedagogies, and content development, and also in the definition of appropriate business models.Future long term research is indeed needed to answer the various questions highlighted in this work.

Figure 2 .
Figure 2. User events in both platforms by hour of day (PV -pageviews; USR -users; MSGposted messages).

Figure 3 .
Figure 3. Monthly user events in Moodle and Elgg before, during (months 5 and 6) and after the course (PV -pageviews; USR -users; MSG -posted messages).

Figure 4 .
Figure 4. Daily user events in Moodle and Elgg during the course (PV -pageviews; USR -users; MSG -posted messages).

Figure 5 .
Figure 5. Number of visualisations of activities/resources in Moodle, throughout the 5 topics of the pilot course.'Course' refers to the common resources: "Learning guide", "News and Headlines" forum, e-textbook, and introductory video.
The design of MOOCs continues to evolve, with all kinds of variations and challenges to address, e.g.Yousef, Chatti, Schroeder, Wosnitza & Jakobs (2014) discuss the use of other MOOCs, such as sMOOCs that represent small open online courses and bMOOCs that represent MOOCs that are blended with on-campus teaching.In this paper we presented the largest Portuguese MOOC to date, on the topic of climate change, with more than one thousand enrolments.This course piloted what appears to be a successful innovative pedagogical model -the iMOOC, where the advantages of c and x MOOC were successfully joined, such as collaborative group work, paced instruction, and a creditable online course.It was clear from the technological point of view, the advantage of using open source platforms, reducing the cost of development, and having at the same time, a large set of stable and well tested features.The technology aspects were never a constraint factor.

Table 1
iMOOC participants in terms of numbers of pageviews and posted messages (both platforms).Analysis of the participant profile of the course, in terms of activity days with pageviews and posted messages (Table2), shows that participants with less than two days of pageviews didn't post any messages.Participants who were more active during the course fall into the groups shown in bold in Table2, i.e. the groups with greater global number of users (respectively 39 and 469).Still, a considerable number of participants (425) were active in terms of page viewing (11-1000) but did not post messages.
Many participants (50.1%) were already UAb students, prior to their registration in the pilot iMOOC.However, 1.5 % of these were newly-enrolled in formal courses at UAb.Still, 48.4% were unidentified and may represent a potential group for future recruitment.If we broaden the definition of learner achievement in MOOCs to cover other objectives of participants who may be interested in just following the course but not participating in the assignments, we can calculate another success rate.In Table2we registered 497 (469+28) participants with a period of activity covering the course, that represent 48.5% of the participants.
of the course require less design and development time.The personnel cost of the course re-run, was estimated on the basis of the personnel costs of the three instructors (32h each), two environmental experts, and two computer science experts (respectively 20, 10 and 20 hours each), as 1,900 Euros.Global cost is hence estimated as 2,530 Euros.Nevertheless, several key challenges of teaching MOOCs are still on the table, namely: difficulty in evaluating students' work; having a sense of speaking into a vacuum due to the absence of student immediate feedback; being burdened by the heavy demands of time and money; and