How MOOC-Takers Estimate Learning Success: Retrospective Reflection of Perceived Benefits

Massive open online courses (MOOCs) have attracted a great deal of interest in recent years as a new learning technology. Since MOOCs inception, only limited research has been carried out to address how learners perceive success in MOOCs after course completion. The aim of this study was to investigate the perceived benefits as the measurement of learning success. Narrative interviews were conducted with 30 Russian-speaking learners who completed at least one MOOC in full. By employing text analysis of interview transcripts, we revealed the authentic voices of participants and gained deeper understanding of learners' perceived benefits based on retrospective reflection. The findings of the study indicate that after finishing MOOCs, learners have received tangible and intangible benefits that in general justified their expectations. University-affiliated students, as well as working professionals, recognized the complementarity of MOOCs, but their assessments were limited to educational tracks. We discovered that taking MOOCs often coincided with the time when an individual was planning to change career, education, or life tracks. The results of the study and their implications are further discussed, together with practical suggestions for MOOC providers.


Résumé de l'article
Massive open online courses (MOOCs) have attracted a great deal of interest in recent years as a new learning technology. Since MOOCs inception, only limited research has been carried out to address how learners perceive success in MOOCs after course completion. The aim of this study was to investigate the perceived benefits as the measurement of learning success. Narrative interviews were conducted with 30 Russian-speaking learners who completed at least one MOOC in full. By employing text analysis of interview transcripts, we revealed the authentic voices of participants and gained deeper understanding of learners' perceived benefits based on retrospective reflection. The findings of the study indicate that after finishing MOOCs, learners have received tangible and intangible benefits that in general justified their expectations. University-affiliated students, as well as working professionals, recognized the complementarity of MOOCs, but their assessments were limited to educational tracks. We discovered that taking MOOCs often coincided with the time when an individual was planning to change career, education, or life tracks. The results of the study and their implications are further discussed, together with practical suggestions for MOOC providers.

Introduction
The educational landscape in the early 21 st century is constantly changing due to global challenges.
Competence building in education should prepare graduates for new digital economies and jobs that are emerging but not yet fully developed. For over 20 years, e-learning has been part of the process of redefining the purpose of education and addressing the needs of the digital generation. Massive open online courses (MOOCs) present a rapidly growing and potentially disruptive innovation in the field of online learning, as they are empowered with modern technology capabilities and the potential to impact the lives of millions (Karnouskos, 2017).
As MOOCs serve as a rich source of quantifiable information, most studies have concentrated on learning behavior, and other activities that can be explained with numbers (Littlejohn, Hood, Milligan, & Mustain, 2016;Rai & Chunrao, 2016). While enormously valuable for our understanding of the impact of the new technology, these numbers tell us only half of the story, as they provide little insight into learners' perceptions and do not necessarily explain what happens in learners' minds before, during, or after the completion of a MOOC. There are also no longitudinal studies investigating MOOCs' impact on graduates' skills and their potential for up-skilling (Calonge & Shah, 2016). This study aims to investigate through retrospective reflection how MOOC-completers themselves measure learning success, and what they see as benefits from completing a MOOC. It adds to existing research by focusing on unique experiences of individual learners who now have a chance to have their voices heard.
MOOCs are gaining popularity in Russia and post-Soviet countries, and the number of MOOCs offered through various platforms by Russian universities is growing (Kulik & Kidimova, 2017). However, their impact on Russian-speaking learners has not been sufficiently explored. Even though the findings might be more relevant within the former-soviet context, the use of qualitative approach is well needed to understand the perceived learner success.

Exploring Relationship Between Individual and Society in Post-Modern World
This research uses post-modernity theories to examine people's perceptions of personal success and benefits from completing a MOOC. The relationship between the individual and society becomes more complex in the postmodern world, which is characterized by the increasing uncertainty and experiencing forces and trends beyond the individual's control (Bauman, 2001). Societal institutes can no longer guarantee financial and professional security (Beck, 1992), and personal success is now linked to an individual's ability to discard irrational or ineffective life strategies and develop new ones.
In post-modern society, the role of educational institutions as the main source of acquiring knowledge is declining, and the initiative in learning is shifting to the individual. Learning becomes especially valuable as it forms the ability to adapt to the changing social reality and helps connect fragmentary elements of knowledge and experience into meaningful patterns (Bauman, 2001). Educational institutions are slow to respond to the needs of the post-modern society (Chandler, 2013), as they largely remain teacher-centered, distributing knowledge only in one direction on the conditions defined by educational institutions themselves. At the same time, distance learning formats, such as online courses and self-study materials, are more learner-centered and can flexibly adapt to the changing needs of the society, thus making it an attractive option for self-development and becoming the central element of irreplaceable life-long "equipment" (Bauman, 2001). Universities are still invaluable in producing formalized proofs of education, as they continue to serve as an entry ticket into the labor market, but graduation certificates, diplomas, or certificates of any level are no longer sufficient to receive or maintain a job (Beck, 1992).

MOOCs in the Context of Post-Soviet Countries
The traditional model of higher education that is still widely used in post-Soviet countries involves students declaring their major as they enter the university, with most of the coursework determined by the department or program. Combined with the tendency to place the most emphasis on theoretical training, this lack of curriculum flexibility results in post-Soviet universities falling behind in equipping their graduates with the practical skills that are in demand on the constantly changing labor market. This skills gap phenomenon is not restricted to the post-Soviet space: a number of studies have revealed the discontinuity between the skills of recent graduates and the needs of the employers (Calonge & Shah, 2016). In these conditions, MOOCs present a viable alternative, as they have proved their significance for the development of professional competencies, including digital and management skills (Radford et al., 2014;Calonge & Shah, 2016).
At the same time, most employers in post-Soviet countries have limited knowledge of MOOC platforms and the specifics of MOOCs, as they have been a relatively recent development in the field of online education (Kulik & Kidimova, 2017). Employers still rely heavily on traditional diplomas and certificates to evaluate job candidates and give promotions. This local tendency is confirmed by the review of the current trends in the world labor market, which shows that employers worldwide are reluctant to consider certificates received through MOOCs, as they are either not familiar with the concept or unable to assess the quality of these courses (Radford et al., 2014;Thompson, 2016).
Considering the gap between MOOC providers' orientation towards attracting potential learners with the prospects of receiving a certificate and current demand for such certificates in the labor market, questions need to be asked regarding associating learning success in MOOCs with tangible benefits alone.

Measuring Learning Success in MOOCs
As MOOCs are a relatively new technology, limited research has been carried out on some of their aspects, such as learning success and perceived benefits (Alraimi, Zo, & Ciganek, 2015;Karnouskos, 2017). Most studies employed primarily quantitative or mixed methods, such as analysis of course statistics, student survey data, and learning analytics. Learning success is traditionally measured by the total number of registered users; their engagement and intention to continue using MOOCs; the percentage of content viewed; the dropout, progression, retention, and completion rates; the scores on assignments; the number of certificates of achievement; or percentage of students who received a certificate of completion (Breslow et al., 2013;Littlejohn et al., 2016;Rai & Chunrao, 2016). According to MOOC providers, learning success may be measured by receiving a score and a certificate for completing a course.
This approach to success assessment of MOOCs resulted in very low reported success rates. Statistics of online platforms and research findings have revealed that MOOC completion rates hover between 5% and 12% and dropout rates have been recorded as high as 90% (Stich & Reeves, 2017). Considering the reported low completion rates for MOOCs (Hew & Cheung, 2014;Jordan, 2014;Watted & Barak, 2018), measuring success with conventional methods can be problematic (Stich & Reeves, 2017).
However, there are other potential benefits from MOOCs that could be considered "intangible" (Zhenghao et al., 2015). First, open and accessible character of MOOCs allows individuals to follow their own personal learning paths, using strategies of their choice, which are adapted to their abilities (Milligan & Littlejohn, 2014). Secondly, MOOCs allow learners to become part of an international community with unique identity and dynamics that is strongly oriented towards self-development (Waard et al., 2011). Thirdly, taking a MOOC can positively affect an individual's self-efficacy: people start feeling more confident about their abilities and feel better positioned to make important life, education, and career decisions (Longstaff, 2017).
The potential diversity of learners, each with different prior experience, educational backgrounds and skill levels, and varied expectations and motivations for taking the MOOC (Kizilcec, Piech, & Schneider, 2013;Hood, Littlejohn, & Milligan, 2015), is followed by redefining the meaning of success which may not be focused on completion (Breslow et al., 2013)

Recruiting Participants
Research participants were recruited using convenience sampling among Russian-speaking MOOCtakers through announcements on discussion forums in the social network "VK.com" (https://vk.com/courserarussia, https://vk.com/coursera, https://vk.com/perevedemcoursera). Furthermore, personal messages were sent to participants of discussions with a high level of activity and to posts commentators on the main page of the online community. Thirty people replied and were willing to participate in an interview. All of them have completed one or more MOOC since 2012 on one of the platforms, such as Coursera, edX, Udacity, FutureLearn, Open Education, Lektorium, Stepik, etc.
Successful completion of at least one MOOC, with or without a certificate and fluency in Russian, were the main selection criteria for participation in the study. At the time of survey, the informants lived in different cities in Russia (Moscow, Saint Petersburg, Novosibirsk, Volgograd, Samara, Magnitogorsk, Stavropol, and Kaliningrad) as well as in Ukraine, Kazakhstan, Moldova, Germany, and Vietnam.
First interviews were conducted face-to-face in Novosibirsk due to the location of interviewers and participants. Then, Russian-speaking MOOC-takers were recruited from other locations in Russia and abroad.
Most interviews were conducted via Skype with all ethical procedures overseen by the first author, who also managed the project. Data was collected between the end of 2016 and the beginning of 2017 resulting in 17 male and 13 female participants between the ages of 18 to 44-years-old. Among the Russian-speaking MOOC-takers were specialists in the area of information technologies, managers, engineers, teachers who use MOOCs to develop their own courses, and university students.
Undoubtedly, based on the research of the international audience of online courses (Kizilcec et al., 2013), the interviewees did not represent all types of audiences.

Data Collection
Narrative interview was used as a data collection instrument in this study since "a person lives his or her life as a story to be told" (Bauman, 2001). This technique allows the researcher to focus on the stories narrated by the individuals who have experienced events and reproduced them (Schwandt, 2001, p. 171).
In order to establish contact and "warm up" the relationship, at the beginning of the conversation the interviewers shared their own story of acquaintance with online courses and encouraged interviewees to do the same. The interviewer's narrative helped to stimulate the informant's narrative, increasing the likelihood of receiving a frank answer in exchange for a true and sincere story of the interviewers.
In addition, it served as a model narrative, which retrospectively -introduced some "inner logic" in the "narrated lives" (Bauman, 2001). To avoid a potential bias, the interviewers were sharing the stories of failing to complete a MOOC, which was the opposite of what was requested from the participants, and therefore their narratives could not have influenced the participants' answers to any significant extent. In addition, the technique was not used in the cases when MOOC-takers began to give details about their life and online learning experience without any stimulation.
The main part of the interview contained the informants' narrative about life as a sequence of events from the moment they first heard about MOOCs to the present time. During the conversations, the interviewers asked some follow-up questions to elicit commentary on the perceived benefits. These questions helped obtain explanation of the meaning intended by particular comments of informants.

Data Analysis
The interviews were recorded and transcribed verbatim to preserve the validity of the data. The names of the participants were removed to protect anonymity and a number was assigned to each interview (e.g., P1 stands for "Participant 1").
The transcribed interviews were analysed using thematic analysis, in accordance with the guidelines of Braun and Clarke (2006). The first stage of the analysis identified particular situations, events, and sequences of events within each interview that were meaningful to the informant. This allowed us to see how learners evaluated the impact of completing a MOOC at different stages of their life. The set of initial codes was determined. Then we looked for consistent and repeated patterns of meaning searching across interviews. As a result, potential themes were identified, verified, and cross-referenced with quotations from the interviews (Braun & Clarke, 2006). This multi-staged analysis allowed us to explore MOOC-takers' explanations of perceived benefits as they retrospectively reflected on their completion of an online course. The possibility to match the periods of taking an online course with the participants' life events enabled the researchers to unravel the latent surface of perceived benefits. All researchers were involved in data analysis to ensure consistency, transparency and triangulation of findings (Johnson & Christensen, 2017). Combining interpretations of results attained by co-authors helped to avoid researcher bias.

Limitations of the Study
The present study has several limitations. One important issue to recognize is that the number of participants was very small. Another limitation of this research is focusing on the data from Russianspeaking participants, so the research findings are context-specific. However, MOOCs reception in the local labor market is consistent with the global trends, which should make the results of the study relevant for wider audiences.
The third limitation stems from the sampling method and selection criteria. The participants were recruited through online platforms, which suggest higher levels of digital literacy among those who were selected. In addition, it is possible that the individuals who responded to a call for participation in the study are naturally more active and motivated to discuss their experiences with MOOCs. In that regard, less active and less digitally literate MOOC-takers were not represented in the study.
Lastly, this study was conducted with learners who completed MOOCs in full. Nevertheless, users who only download materials or participate in other learning activities may benefit from a MOOC even if they do not complete it.

Research Findings
Despite  Causal relationships between different events in the life of informants were complex and ambiguous.
One participant summarized it as follows: "plans somehow change by themselves, which is caused by

MOOCs Impact: University-Affiliated Students
Working professionals described MOOCs impact in the context of their professional tracks as "routes" of life, whereas university-affiliated students focused more on educational trajectories. First, MOOCs broadened the horizons and knowledge in different areas, often adjacent and not directly related to university majors. Interviewees commented that "[MOOCs were] just for general knowledge, for general understanding of some things" (P4) and "an online course is always a great way to understand something at least in general terms or to start learning something new" (P17).
Secondly, online courses helped decide on further plans related to continuing studies or changing the major at the next level of training. As one student said, "it is possible to choose another major for my Master's program... I did not really think about this direction before the online courses" (P3). Thirdly, for people with special educational needs, this format of education is particularly beneficial and allows overcoming certain barriers: I have poor eyesight, so I cannot see the board even sitting in the first row; and in online courses you can stop the video, zoom it in: the screen can be either larger or smaller. It's very convenient for people with disabilities. (P2) Fourthly, some participants' reported that taking MOOCs affected their academic achievement. As one student related, "[now] it is easier for me to solve integration problems, because I passed an [online] integration course" (P16).
Some of the participants noted that online platforms were providing high-quality learning experiences on as needed basis. In their view, MOOCs have a positive reputation and should be integrated into individual educational trajectories of students, since "knowledge that online courses give has similar quality to traditional classroom teaching at the university" (P19). In this sense, traditional degrees and online learning could be best understood as a continuum that involves interplay and overlap between different activities.

Perceived Benefits: Patterns in Discourse
The retrospective discourse revealed a common pattern of perceived benefits that included three elements: initial expectations, invested efforts to acquire new knowledge and practical skills, and expected benefits or rewards. If a person has received expected benefits after completing a MOOC, then it justified his or her efforts. The universal condition for success was that reality had to match initial expectations: "My expectations were met, there were a lot of courses, for the vast majority of them I got what I wanted: some new knowledge, or new skills, or contacts, which later were followed by offline communication" (P28).
Although it followed a common pattern, the combination of expectations and rewards was unique for each person due to several reasons. Firstly, despite the fact that among the interviewees there were only some categories of MOOC-takers, almost the entire palette of the goals and motivations described in previous studies (Barak, Watted, & Haick, 2016;Littlejohn et al., 2016)

Crystallisation of Perceived Benefits in Participants' Minds
As a result, the current study reveals how perceived benefits crystallize in the minds of informants as they reflect upon their MOOC experience. 2. Free or paid certificate for completing a course. Informants who have experience working abroad or in international companies saw the value of certificates on the global labor market as an indicator of self-organization, responsibility, independence, and time management skills: "In the eyes of other people, this certificate gives you weight as a professional, so you can sell yourself in the labor market more expensively" (P15). Sometimes a certificate is important for improving selfesteem: "My goals are changing, I develop as a person. What is more, I develop as a professional.
When you have a certificate, your importance grows in your own eyes: I completed a course" (P19).
In some cases, respondents reported how other people benefited from receiving a certificate: If you have certificates, from the standpoint of selling yourself as a professional, it will help you. I have a friend who presents himself as an internet marketing and brand specialist. However, the majority of respondents treated learning very pragmatically and expected to obtain practical skills, necessary for their current or future line of work. To achieve this goal, a certificate is not required; the most important outcome is gaining new knowledge that can be applied in real life.
Course participants saw receiving a certificate only as an additional motivational factor.
3. New knowledge and practical skills. Usefulness of MOOC, from the participants' viewpoint, was often associated with additional practical skills for professional development and improving career opportunities in the future: Those things that I learned could be applied almost instantly in my work, even the same day. There was a failure at work: there were no results, so I wanted to give up everything, stop working, and find another profession. But at that moment I started to take these courses, which was a pure coincidence, but they somehow pushed me: I became more optimistic about everything. (P9) 5. Accumulation of social capital. Depending on their interest, MOOC participants may establish interpersonal relationships and develop learners' networks in MOOCs and beyond. One of the research participants summarized this as follows: When I take courses, I try to find like-minded people and common ground between me, the author, and those who are also taking this course. While taking the most recent course, I was offered a job from its author. (P24) The same person who completed several courses can describe different perceived benefits depending on life situation. In one case, the participant focused on "obtaining a mark of distinction [certificate], which will be waiting in the wings" (P23), but in another situation, the same individual aimed to gain practical knowledge and skills for changing a current job and preparing for a new one.

Indicators of Success for Online Platforms and for Learners
Since all of the informants had completed at least one MOOC in full, the majority of them viewed obtaining a certificate as one of the perceived benefits. However, most interviewees questioned the usefulness of certificates on the job market in Russia with the predominance of pessimistic evaluation: "Very few Russian employers would need this" (P8). The respondents cited lack of awareness about MOOCs and online learning among the employers in post-Soviet countries: "They don't know what those MOOCs are. That is true in most cases" (P14).
According to the informants, in the future there would be greater interest to certificates on the labor market in the post-Soviet countries. Participants predicted the relevance of MOOCs on CVs, proposing that they may be economically beneficial in terms of helping to secure future employment; something "extra" to distinguish job applicants from other candidates. Some interviewees felt optimistic about the usefulness of online courses in their future lives, although their potential has not been revealed in the past or in the present: "I hope that online courses will still give me some benefits in the next year or two, even though they have not yet brought any tangible result" (P28).
The usefulness of MOOCs for most working professionals was often associated not only with the certificate, but also with additional practical skills, future career opportunities, and with professional

Discussion
According to Beck (1992), in "risk societies," each person is expected to use intelligence, resources, and diligence to achieve more satisfactory living conditions. Through the narrative interviews, our informants demonstrated that they were responding to the challenge of keeping up with the changing society by taking MOOCs. For working professionals, online learning was seen as a "cementing" factor, forming the clusters of competencies for their professional life and career development. The necessity to stay competitive in the workforce served as an impetus for searching for new knowledge and practical skills.
While university students also recognized the ability of MOOCs to complement formal learning, their descriptions were limited to educational tracks due to a lack of experience in the labor market. These results are consistent with the previous studies (Brahimi & Sarirete, 2015;Schmid, Manturuk, Simpkins, Goldwasser, & Whitfield, 2015;Milligan & Littlejohn, 2017;Watted & Barak, 2018), and suggest that students use MOOCs to supplement or complement their formal learning with an additional source and improve their knowledge. In this sense, online platforms are becoming an important tool for the educational market.
The results revealed a continuum of perceived benefits in participants' minds: from tangible to intangible. As far as tangible benefits, MOOC completion, with or without the certificate, is viewed as a significant achievement that can "strengthen" one's CV and improve career prospects. However, some of the perceived impact is indirect: one can use MOOC completion to demonstrate commitment to life-long learning or to increase self-esteem. This finding supports the previous studies, which claimed that the MOOCs' impact should not be evaluated solely based on certification rates (Ho et al., 2014;Kahan, Soffer, & Nachmias, 2017).
Acquired knowledge and practical skills are seen as very important intangible benefits of completing a MOOC. In post-modern society, competent labor is the deciding factor, especially in the fast-paced knowledge-based economies (Karnouskos, 2017). MOOCs provide Russian-speaking learners with a flexible and accessible platform for acquiring new knowledge and immediately applicable practical skills.
When it comes to even less tangible benefits of completing a MOOC, one of the unexpected findings was the notion that taking online courses may change a person's outlook on life. A possible explanation could be that completing a MOOC not only opens new career opportunities, but also teaches self-organization and perseverance, which eventually become invaluable personal strategies that help in coping with difficult life situations (Hammond, 2007). As suggested by Longstaff (2017) At the same time, there seems to be a discrepancy between how MOOC users themselves perceived learning success and how these perceptions corresponded to the ideas of MOOC providers. Since all of the research participants were MOOC-completers, it was not surprising to hear that they viewed obtaining a certificate as one of the perceived benefits. In this case, a certificate of completion was an indicator of success both for the online platforms and for learners. However, as noted by the interviewees, in post-Soviet countries specifically, this certificate is considered more important on the educational market rather than anywhere beyond it, including the job market. This observation aligns with the research by Karnouskos (2017) stating that participants complete courses mostly for personal satisfaction rather than for receiving widely accredited certificates. Employees doubt their value, whereas "employers do not seem to honor the credentials and certificates earned through MOOCs too much, as they are not linked closely enough to job-related learning" (Egloffstein & Ifenthaler, 2017, p. 69). The findings demonstrate that merely looking at course completion as a measure for success, especially for working professionals, does not suffice in the context of a MOOC, which is consistent with the research by Henderikx et al. (2017) and Littlejohn et al. (2016), which calls for a redefinition of "success" in a MOOC.

Conclusion
The significant gap in previous literature regarding the perceived benefits for learners completing MOOCs motivated the research team to conduct this study. The aim was to investigate the perceived benefits as the measurement of success in MOOCs. The perceived benefits in this case were relying on retrospective reflections from the participants of the narrative interview.
Research findings lead to the conclusion that learners' perceived benefits may be inconsistent with the Educational practices in a post-modern world are being deregulated and transferred to the area of individual initiative. In addition to new knowledge and skills, online learning can help resolve problems with self-determination, self-management, and self-assertion, which become the responsibility of every individual (Beck, 1992;Bauman, 2001). Due to its flexibility and easily renewable content, learning through MOOCs can have adaptive value and become an important element of life "equipment."