Evaluation of the UNED MOOCs Implementation: Demographics, Learners' Opinions and Completion Rates

The paper is a study about the MOOC experience at the Spanish National University of Distance Education (UNED), where we have collected initial and final information about learners' profiles and opinions, as well as enrolment, completion and certification rates. It is a survey-based study covering 17 MOOCs offered in UNED's own platform, and collects information from a sample of more than 24000 learners (initial survey) and 2003 learners (final survey). The paper first presents an overview of the MOOC experience at UNED, introducing the main features of these courses. Afterwards, it focuses on the methodology used in the study and in the information gathered in the second edition of UNED MOOCs, which took place from November 2013 until March 2014. Learners' average profile is a Spanish female, approximately 37 years old, with a University degree, and generally employed. For many of the participants, UNED MOOCs were their first experiences with these sorts of courses, and the main reasons for enrolment were the course topic and the perceived usefulness for professional development. The expectations regarding completion and certification where initially very high, but completion rates remain below 14%. In the final survey, the overall experience in the MOOCs and the different tools used in the courses were highly valued by learners, except the support figures (curator, facilitator, peers), which received lower ratings. These findings are of interest for the institution and further research, refining learning analytics, is encouraged.


Introduction
The so-called Massive Open Online Courses (MOOCs) have become the most dominant emerging phenomenon in recent years in the field of higher education. Although the objectives have been different in different cases -institutional, promotional, advertising, etc.,-the most prestigious universities in the world have created a growing and, above all, varied range of courses, both in terms of content and quality. Attentive to the development of this phenomenon, and assuming responsibility within the Spanish educational system for spearheading innovative initiatives, especially in the field of higher education distance learning, the Spanish National University of Distance Education (UNED, http://www.uned.es) has, from October 2012 142 onwards, launched a number of courses using this teaching modality and with significant success, in terms of the large response achieved and the number of people enrolled in the various courses that have been offered.
At the end of the second series of these courses (November 2013to March 2014 and bearing in mind the commitment of the UNED to the quality of the courses it offers, we have made a first assessment of the experience, focusing specifically on: the analysis of the socio-demographic profile of the students of MOOCs offered by the UNED; the assessment of the MOOCs from the learners' perspective; and the analysis of the various activities of the students related to the courses (registration, completion, and certification rates). Veletsianos and Shepherdson (2016) identify demographics of MOOC users as one of the themes that emerge across systematic analyses of the MOOC literature. This paper tries to fill one of the gaps in MOOCs literature, which is the geographic distribution -most studies about MOOCs proceed from North-American authors (Gašević, Joksimović, Kovanović, & Siemens, 2014) and refer to North-American or European (mainly UK) participants (Liyanagunawardena, Adams, & Williams, 2013), thus native English speakers-. This is the first empirical analysis of one of the main MOOCs' offering university in Spain, i.e., UNED, through UNED MOOCs.

UNED MOOCs Overview
UNED's mission is the public service of higher education through the modality of distance education, characterised for utilising a specific didactical methodology by a conjoint use of printed, audiovisual, and technological media, along with face-to-face support from tutors in regional centres (more than 60 all over Spain) and communication means with professors (virtual fora, email, phone). It has the largest student population in Spain: more than 150000 registered students in Graduate programmes, almost 8000 in Master's programmes and more than 1000 in Doctorate studies, almost 13000 in the access to university course and more than 18000 in language courses in 2014- 15 (UNED, 2016). since 2015, UNED uses OpenEdx in its own server (https://iedra.uned.es/). The 17 courses on which this paper focuses were run in openmooc.
UNED MOOCs can be considered x-MOOCs, as they fall into the cognitive-behaviourist pedagogy and rely primarily on information transmission, computer marked assignments, and peer assessment (Rodriguez, 2013). Instructional design is based on providing content -mainly using videos-and proposing activities, mainly multiple choice question tests (computer marked) and other activities like peer-assessed essays, open questions, etc. According to Bates (2015), UNED MOOCs have the following design features supported by a specially designed platform software (openmooc in this case): video lectures, computer-marked assignments, peer assessment, supporting materials, a shared comment/discussion space, badges or certificates, and learning analytics. Regarding a remaining feature that this author identifies (Bates, 2015, p. 154), i.e., "no, or very light, discussion moderation," UNED MOOCs use different approaches to this aspect and two agents are in charge of supporting learners, as will be explained later, thus approaching to network-based MOOCs (Lane, 2012).
Regarding this mixture, there has been an opportunity to explore and experiment diverse pedagogical models through UNED MOOCs. Professors could use different approaches to tutoring and teacher-presence. Some professors have preferred to be actively involved in the delivery of the course, by encouraging debates in the fora, solving learners' doubts about content. Others have been actively involved in preparing material for self-learning and not so present in the delivery of the course. Regarding support to learners, besides professors-curators (authors of the contents, designers of the course, and in some MOOCs, actively involved in the implementation of the courses), there have been two different figures with active roles during the delivery of the courses: assistant tutors-facilitators (in charge of motivating and supervising debates along with professors, reordering messages, compiling FAQs, solving doubts about how to get access to the course content, about how to apply for completion certificates, etc.) and technical support staff (in charge of solving technical problems: registration, duplicated users, downloading badges and certificates, etc.) (Marauri, 2014;Gil-Jaurena, 2015).
Learners who complete the course (at least 80% of progress) have the following certification options:  Digital badge: based on Mozilla Open Badges Infrastructure (OBI), it is free and it doesn't have academic effect.
 Digital credential: digital certificate with no academic effect, it costs 15€.
 Certificate: digital certificate for learners who, besides completing the course, take an exam in any of our regional centres. It costs 50€ and can have academic effect (recognition in UNED degrees).

Methodological Strategy and Data Collection
Evaluation of the UNED MOOCs Implementation: Demographics, Learners' Opinions and Completion Rates Gil-Jaurena, Callejo, and Agudo

144
In order to carry out a comprehensive evaluation of the experiences during the second series of the UNED MOOCs, the following objectives, dimensions of analysis and information sources (in parentheses) have been established: 1. Methodological evaluation, at a meta-level, especially designed to reflect on the empirical material and information which is currently available and which would be necessary for future evaluations. c) Number of students who successfully finish the course.
i. Those who complete 85% of the course.
ii. Those who obtain a credential or certificate.
3. Socio-demographic profile of the enrolled students (gender, age, education level, employment status, country of residence), previous experience of MOOCs, motivations, expectations, and assessments about the use and usefulness of the elements of the course was gathered by two online surveys. Initial survey was completed by registered students, given at the beginning to the students enrolled in each course, and it was composed by nine questions, three of which were multi-choice questions. The average time to complete this questionnaire was between two and three minutes. On the other hand, final survey was given at the end of the course and mainly replied by those students who had completed at least 85% of the course (it includes watching the videos, passing the tests and undertaking peer activities if it is the case) and voluntarily agreed to participate. This end survey was composed by socio-demographic variables and 26, 10-point (1-5: insufficient, poor, 5-7: enough, sufficient, 7-9: notable, good, 9-10: outstanding, very good) scale items (6 in Q7, 8 in Q8, 8 in Q9, and 4 in Q10). Learners were informed that the surveys are anonymous. Internal consistency reliability can be considered high: 0.938 Cronbach's coefficient alpha. Validity is obtained from the significance and strong correlation (0.701 Pearson's correlation coefficient, P<0.01 two tails) between additive score from the responses to Q7, Q8, and Q9 items and responses to recommendation of the course to other people interested in the subject (Q10 item 3). The average time to complete this second questionnaire was between six to eight minutes.
Both online surveys were conducted by Lime Survey software and analysis was carried out using SPSS v.24 software for Windows. The level of statistical significance was established in P<0.01. i. Overall level of satisfaction.
iii. Satisfaction with the organisation of the course (Q7.5).
iv. Satisfaction with the length of the course (Q7.4).
c) Assessment of the use and usefulness of the elements of the course (final survey).
The methodological strategy followed is based on two main principles:  Given that the questionnaires have been customised to suit the needs of each course, our analysis only considers questions common to all courses;  The focus is on the overall experience of the project, without going into a detailed analysis of each course.
The 17 courses analysed were the following (ordered by size: number of registered learners): 17. Basic analytical chemistry.
These 17 courses were selected among the 25 offered considering that they were 100% run by UNED with their professors. The other eight courses were developed in partnership with other institutions and had specific characteristics, this they are not part of this study.
The main characteristics of both surveys are summarised in Table 1.

148
English courses were the most popular (Basic and Professional English courses). Financial accounting and ICT were also popular. More specialised courses, such as chemistry or logic, had fewer learners, but still more than 500 in the second edition.

Socio-Demographic Profiles
The gender distribution is the same in both the initial and the final surveys, being 36.4% men and 63.6% women ( Figure 1). The equal distribution suggests that gender is not a relevant factor in determining whether or not the course is finished, or at least almost entirely completed. But the predominance of female learners challenges other studies that report that a majority (Ebben & Murphy, 2014)   The average age of those completing the initial questionnaire is 36.17 years; while for those completing the final survey the average age rises slightly to 37.89 years, almost two years more. Worth mentioning the presence, even if the percentages are very low, of people under 18 and also of elderly people carrying out the courses, although the bulk of the students are found in the middle age ranges, especially between the ages of 31 and 45 ( Figure 2). However, Table 2 shows that the age group that is most likely to complete a substantial part of the course -given indirectly by the completion of the final questionnaire of the course-is the group of students between 46 and 60 years of age.  By nationality, almost four out of five of those responding to the initial questionnaire reside in Spain (79.5%).
When referred to the general MOOC offer in the world, Oliver, Hernández-Leo, Daza, Martín, and Albó (2014), explain how "from the demand side, Spain also stands within the top-five countries with more participants, right after USA, UK, Canada and Brazil" (p. 5).  Venezuelans, and 0.7% from other Spanish-speaking countries. If these figures are added to the previous figure, it can be seen that residents of Spanish-speaking countries make up almost all of the students.
In the sample of students who completed the final questionnaire, the vast majority (89.3%) are resident in Spain, while the 1.9% residing in Mexico and 1.6% in Colombia are also worth highlighting.
The majority of people who have attended the UNED MOOCs are either currently studying or have completed higher, university-level education ( Table 3). This result is in accordance with findings reported by other authors (Emanuel, 2013;Ho et al. 2014;Jordan, 2014;Bates, 2015). The persistence in MOOCs only achievable for privileged learners (those who have previously attained higher education qualifications) has been identified by Schuwer et al. (2015) as an example of inequality in access to education, and as one of the threats of MOOCs on the macro level. In view of these distributions, and if we assume that those who have completed the final questionnaire have more or less satisfactorily completed their respective courses, we can see there is greater likelihood of this completion occurring for those who are either currently studying or have completed higher education than for those who have achieved a lower level of education ( Figure 3). Students make up the third largest group with 19%. Self-employed represent just 7.7% of the total; but their high presence in some of the courses, such as the Mini-Videos and Social Responsibility courses, should be highlighted (Table 4).

Previous Experience of MOOCs
Of the enrolled students who participated in the initial questionnaire, 70.5% stated that this was their first MOOC experience with the UNED open courses, while 8.3% had had only one previous experience. As little as 3% come to these courses with prior experience involving three or more courses of this sort. Therefore, it is fair to say that for the majority this opportunity represents their only point of reference regarding the MOOCs experience.
In terms of previous experience in COMA/MOOC courses, 9% have had this in different UNED platforms.
However, 7% of the total sample have already tried the UNED-COMA courses themselves. This is followed by 4% for MiriadaX and just under 4% for Coursera.

Motivation and Reasons for Enrolment
Bearing in mind that an average of 2.5 reasons for embarking on the UNED MOOC courses analysed was identified by respondents, the most common of these are interest in course content, subject, and expected usefulness of this subject for professional development. The first of these reasons is given by almost 72% of participants in the initial survey; while the second is given by 60.6%. It should be borne in mind that a third (33.8%) give the two reasons together. In fact, between the two they account for 51% of the total collected answers. This is shown in Table 5. Looking beyond the subject of each specific course, it should be noted that 30% of participants in the questionnaire have selected the fact that the course methodology suited them as a reason for taking part in this experience. This is relatively independent of what is meant by the concept of methodology in this case: Distance learning? Free? Short courses? Participation and collaboration among peers? That is to say, there is something specific in this approach to teaching that attracts a significant number of those who have enrolled.
Given the importance of this specific reason, we will concentrate on it now. Thus, an analysis using binary logistic regression, inasmuch as this demonstrates the direct relationship with the dependent variable, putting the other variables in the model to one side and taking reason as the dependent variable, shows us the statistical significance of variables such as gender. Focusing on the latter, the probability that women indicate this option is 1.2 times more likely than it is for men. Age, however, although significant in the sense that the probability of selecting this reason increases with increasing age, is a factor of little importance. With regard to activity, it is the self-employed students who, compared with those involved in other activities, appear to be more motivated by the methodology of these courses. And, looking at the level of education, the probability of selecting this reason is significantly higher among those who have completed higher education, as would be expected. Thus, those with a university degree or equivalent, or who are studying for this, are more than twice as likely to select this option as respondents who have completed basic or primary education. Further detail and clarification is provided by Table 6, which shows the percentage of those who choose this as a reason within each of the different levels of education, independently of the regression analysis. 4

Expectations
What do the people enrolled in the UNED MOOCs expect from them? What are their expectations? The majority of the answers focus on obtaining the credential. In a credentialist society it is the certificate that counts, to the extent that four out of five people enrolled in the courses point to the credential as the objective for taking the course (Table 7). This pursuit of the credential accounts for as many as 92% of those enrolled in the MOOC on "School tutoring," 88% of those in the course "ICT for teaching and learning" and 87% of those in "Entrepreneurship and Social innovation." Where this expectation has least relative weight is in the "Corporate Social Responsibility" course, where it has not been selected by anyone, although in this case the sample base is Evaluation of the UNED MOOCs Implementation: Demographics, Learners' Opinions and Completion Rates Gil-Jaurena, Callejo, and Agudo 156 very low. The next course where this aspect has relatively low importance is "Logic and uses," with 67% selecting it.
However, in contrast with these high expectations regarding completion of courses and requesting the corresponding certificate, the average course completion rate stands at 13.71% of those who enrol, and the application for certificates at 19.31% of those who complete the course, as reflected in Table 8. This difference between expectations and completion rates in MOOCs has been previously reported; as Reich (2014) stated, "the majority of students who intended to complete a course -in some cases the overwhelming majority of students-were not successful in doing so." (para. 22)

Learners' Behaviours and Rates
With regard to the behaviour of students, the most relevant data regarding enrolment, commencement, completion, and requesting of the certificate for each course are given in Table 8. About dropout rates, the average of students who start the course but don't finish it is higher than 80%.
Completion rate is 13.71% (considering whole enrolment) and 17.79% if we consider only those we started the course. Despite completion rates are low, they are higher than those reported in other studies, such as a rate between 2% and 10% (Reich, 2014), 12.6% (Jordan & Weller, 2013, as cited in Weller, 2014 or average as low as 5% (Jordan, 2014). The high dropout rate and low completion rate have been identified by Schuwer et al. (2015) as a threat of the MOOC movement at the micro level.

Learners' Evaluation and Satisfaction
In this final section on the final evaluations of the MOOCs we must begin by stating that 94% of those who responded have completed the course. Therefore, an overwhelming majority of the satisfaction ratings set out below are from people who have completed the courses.
The rating for what might be considered general aspects of the course is quite similar, being around eight.
The most highly rated aspects are the operation of the virtual platform, the organisation and structure of the courses, and their length. The lowest rated aspect is course contents (Table 9). The course contents were adequate and sufficient (n=1707) 7.69 1.989 The variability in the evaluation of the dimensions of the course is higher when the specific elements that have been used are addressed, as shown in Table 10. Videos, tests, and self-assessments are the most highly used dimensions or materials, the fact that the first of these is close to an average rating of nine (8.94) being Evaluation of the UNED MOOCs Implementation: Demographics, Learners' Opinions and Completion Rates Gil-Jaurena, Callejo, and Agudo 158 noteworthy. However, the different forms of personal support were the lowest used elements, neither the support of the curator (4.87), the lowest rated of all, nor that of the facilitator (5.23), nor that of peers (5.01) having been convincing. Or at least it seems that more was expected of them. This is probably logical in the case of unfamiliarity with this kind of course and its comparison with "normal" courses. Students were asked about their perception of the degree of usefulness of each of these tools for the implementation of the course. The result of the responses are shown in Table 11. Thus we see that tools that have been used and appear to be highly rated (Table 10) are also considered very useful for learning and course development (Table 11). This is the case for the videos, supplementary materials, tests, and evaluations. Meanwhile, aspects that obtained a lower rating have been considered useful as support tools. This is the case for the support of the facilitator and curator. On a third level, the evaluation of activities by peers and peer support itself ( Figure 5) received a relatively low rating, and its usefulness was less highly valued than the previously mentioned aspects. These results are in agreement with those obtained in the analysis of 10 courses in another platform that offers MOOCs in Spanish, Miriadax, as reported by González and Carabantes (2017). The use and satisfaction with various MOOC tools are aligned if we consider these two studies. In any event, the general experience seems to have been quite satisfactory, bearing in mind that, for example, the likelihood of recommending the course taken scored an average of 8.34, which can be interpreted as a high probability. Perhaps most important of all is the fact that the phrase "I would like to take more courses of this type at the UNED" was rated at 9.05. This indicates a high level of satisfaction (Table 12). Do these courses serve to promote the institution and the educational opportunities it offers, considering that this is one of the functions that might be assigned to the MOOCs? In the light of what is shown by the last row of Table 12, the table provides a very positive answer to this question, since 8.14 is the rating for the degree of agreement with the statement: "After completing this course, I would like to take another type of course at the UNED." Its advertising function appears to have been successful. That is to say, it serves as a point of entry -a sort of trial-for the UNED, as is also underlined by the average rating of 8.13 of the sentence: "The completion of this course has allowed me to find out about the training on offer at the UNED." A minimal analysis of the behaviour of this dimension tells us that it has very little relation with age, with a low correlation coefficient of almost zero, with a greater weight among males (8.29) than among women (8.09), and, perhaps most important, with the highest value being reached among those with average levels of education, as shown in Table 13. Moreover, controlling the other socio-demographic variables available (sex and age), it can be said that there is a slight tendency for the rating of this aspect -the probability of taking other courses at the UNED-to decrease as level of education rises. This is something that Table 13 itself suggests. Performing a factor analysis of the principal components (Appendices 2 and 3), focusing the analysis on the first two components, tells us that the assessment of these MOOC courses is structured around two axes. The first axis relates primarily to the general aspects that have been assessed: "I would recommend this course to others interested in the subject," "The course is well organised/structured," "I have found the methodology of the course motivating and appropriate," "The course matched my expectations," or "I think that the course contents have been suitable and sufficient." The second axis involves more specific aspects, especially related to the perception of the support received, highlighting this as a relatively key dimension when it comes to assessment, almost like a demand. Thus, in this axis the assessment of the role that the support of peers, facilitator, and curator played in the course has special weight.
Still bearing in mind these two axes, what is their relationship with fostering an interest in taking other courses at the UNED? Through the correlations in the matrix of components (Appendix 2), this relationship can be seen. That is to say, the correlation with the first axis, or general aspects, has a fairly high correlation (0.659, being P<0.01); while in the second axis it has much less weight. Meanwhile, the correlation of the variable that denotes interest in taking other courses at the UNED and variables that reflect the assessment of the support (facilitator, curator, and peers) received during the course is low, not reaching 0.3 under any circumstances. Also, if we take into account, not so much the evaluation of these tools in the course taken but the evaluation of their usefulness in this type of course, we can see that the correlation with the variable representing interest in taking other courses in the UNED is still low, reaching a peak of 0.321 for the support of the curator.

Conclusions
The main results have been presented in the previous section. Below is a summary of those aspects that are of particular interest for finalising the details of the next series of MOOCs and for their evaluation.
At the institutional level, the results of the analysis demonstrate the existence of a high level of interest in this type of course, as well as high expectations and the demand for more similar courses. As a way of attracting potential students, we must recognise and take advantage of MOOCs as a point of entry to the UNED, bearing in mind that it is the first contact with the institution for most of those who enrol in these courses.
Linking these courses with other UNED academic courses (lifelong learning, regulated studies, etc.) could meet the demand of the students and might result in the sustainability of the project. In this sense, UNED regulated the approval of new MOOCs according to some criteria, among them the relation with other educational offer in the institution (Gil-Jaurena, 2015). It would be useful to continue the evaluation, analysis, and research on MOOCs offered by the UNED. In particular, aspects such as the difference between the expectations and the actual rates of completion of courses or the predominantly well-educated profile of the users are striking and require specific studies.
At a pedagogical level, although in general the methodology and materials in the MOOCs have been positively valued, we must rethink the personal support roles (curator, facilitator), which have been the least highly used and rated by the learners. Independent studies about each MOOC considering their pedagogical specificities (more video-based, more social interaction-based, etc.) would lead to refining the analysis and providing valuable recommendations for teachers, instructional designers, and the institution itself.
Another important issue relates to the identification and better knowledge of learners in different MOOCs or knowledge areas: who they are, which motivations and expectations they present when they approach MOOCs, which use they make of the resources, what other needs they show, etc. Besides the profile, information about the learning paths that students follow is also of great interest in order to improve the learning experience and reduce attrition rates. A more complex and accurate learning analytics system would be recommended. Clustering (Khalil, Kastl, & Ebner, 2016) is an example of analysis we would use to reveal patterns and study the correlation between learners' behaviours, engagement, and completion rates. In this regard, a reconsideration of the automated measures from a pedagogical perspective is needed, in line with the position held by DeBoer, Ho, Stump, and Breslow (2014). Finally, a deeper analysis of the dimensions that influence learners' evaluation and satisfaction with MOOCs would be of interest in order to design and offer effective MOOCs and other online courses in the near future.