International Review of Research in Open and Distributed Learning Motivation and Knowledge: Pre-Assessment and Post-Assessment of MOOC Participants From an Energy and Sustainability Project

open online course (MOOC) is an important research topic, as attrition rates remain high for this environment. Motivation and digital skills have been identified as aspects promoting student engagement in a MOOC, and they are considered necessary for success. However, evaluation of these factors has often relied on tools for which the psychometric properties have not been explored; this suggests that researchers may be working with potentially inaccurate information for judging participants’ profiles. Through a set of analyses (t-test, exploratory factor analysis, correlation), this study explores the relationship between information collected by administering valid and reliable pre and post instruments to measure traits of MOOC attendees. The findings from this study support previously reported outcomes concerning the strong relationships among motivation, previous knowledge, and perceived satisfaction factors for MOOC completers. Moreover, this study provides evidence of the feasibility of developing valid assessments for evaluation purposes.


Introduction
Since their emergence in 2008, massive open online courses (MOOCs) have ignited the academic community due to their potential concerning a variety of interests beyond presenting a flexible educational alternative (Gaebel, 2013).Ranging from college instructional purposes (e.g., blended learning; Rayyan et al., 2016) to international workforce training (Garrido et al., 2016), educators constantly evolve MOOC scopes, moving forward educational content design and technological platforms (Zhang & Nunamaker, 2003).As a result of the variety of applications that involve fewer resources and financial costs when compared to a traditional options, MOOC projects have become a feasible response to contemporary massive educational challenges (Pegler, 2012).
However, beyond a merely educational response perspective, MOOCs have the potential to become a massive research laboratory (Diver & Martinez, 2015).By individualizing learning, this environment challenges well examined dynamics under traditional educative settings (Mazoue, 2013).For instance, unlike traditional courses, MOOCs' characteristics not only determine the ways in which content is delivered (e.g., asynchronically, massively, etc.; Kilgore, Bartoletti, & Freih, 2015), but they also challenge what is known about students' learning characteristics (e.g., learning and habit styles, interest in learning, etc.; Barcena, Martin-Monje, & Read, 2015).Moreover, given that MOOCs are courses designed to reach needs sought by huge audiences (Kennedy, 2014), a continuous research approach is required to understand better teaching and learning characteristics present in this format.Therefore, researchers are contributing constantly to the literature by examining MOOCs' technology, design, delivery conditions, and learning and assessment, among other aspects (Daradoumis, Bassi, Xhafa, & Caballe, 2013).However, despite an increasing amount of research promoting learning aspects in MOOC participants, MOOC completion rates remain low (0.7%-52.1%, with a median value of 12.6%; Jordan, 2015).This makes it necessary to examine what prevents or promotes an attendee's completion of a MOOC, as completion rates challenge efforts to ensure a MOOC meets quality features for its educational content (Kilgore, Bartoletti, & Freih, 2015) or design (Kerr, Houston, Marks, & Richford, 2015).
In this regard, educational and psychological aspects have been reexamined to compare outcomes between traditional and MOOC learning settings (e.g., students' characteristics, course design, etc.;Durksen, Chu, Ahmad, Radil, & Daniels, 2016).However, given that within a traditional setting, learner´s expectations are more standardized and course completion rates can be a sign of student success (Littlejohn, Hood, Milligan, & Mustain, 2016), researchers must evaluate outcomes from this environment when working with MOOC attendees.
Given that motivation is strongly related to student engagement (Shapiro et al., 2017), MOOC researchers have included this factor into their agenda.Now, educators deem motivation as an important ingredient for participants´ self-regulated learning (Magen-Nagar & Cohen, 2017) and as a requirement to succeed when acquiring content from a MOOC (Barak, Watted, & Haick, 2016).
Although social motivation is an important aspect for traditional learners, inner factors are required to learn from MOOC (e.g., intrinsic and extrinsic motivations; Xiong et al., 2015).Because the scope of MOOCs enables the delivery of education asynchronously and massively (Chen, 2013), continuing to examine motivation remains a fruitful direction for research (de Barba, Kennedy, & Ainley, 2016) as MOOCs reach enormous and diverse audiences (Admiraal, Huisman, & Pilli, 2015).
On the other hand, digital skills are essential features to address in MOOC research, as technology is part of the MOOC environment by definition (Rivera & Ramírez, 2015).Moreover, these courses evolve continuously thanks to educational technology (Yuan & Powell, 2013).It has been found that people with high levels of digital skills choose to participate in MOOCs whereas people with lower levels opt for traditional training (Castaño-Muñoz, Kreijns, Kalz, & Punie, 2017).Thus, limited technology skills hamper participants' opportunities to finish a MOOC as this format involves a high level of self-management of educational content (Onah, Sinclair, & Boyatt, 2014).
Among the required skills to attend a MOOC, searching and processing information and digital communication are central (Aesaert, Nijlen, Vanderlinde, & Braak, 2014).Thus, it is not surprising that researchers are interested in continuing to evaluate motivation and digital skills given their importance for MOOC education.
Although traditional assessments (e.g., scoring, providing feedback, etc.) are considered to examine a learner's motivation and digital skills, these kinds of assessment cannot be used in a MOOC design because a course offered under this format reaches a massive audience regularly (Admiraal, Huisman, & Pilli, 2015).
Even though validity aspects of traditional tools used to assess readiness toward e-learning remains uncertain (Farid, 2014), criterion-referenced (Dray, Lowenthal, Miszkiewicz, Ruiz-Primo, & Marczynski, 2011); and theoretical or empirical data can be used to develop valid and reliable tools to explore factors contributing to or impeding students' participation in MOOCs (Xiong et al., 2015).
Given the existing need to reinforce tools used to evaluate motivation and digital skills traits, along with the data-enriched environment of a MOOC (Thille, Scheneider, Piech, Halawa, & Greene, 2014), information collected from MOOC participants is a suitable opportunity to research motivation and digital skill assessments.
Because it is imperative to understand learning specific to the MOOC context (Littlejohn et al., 2016), and to continuously gather information about factors encouraging MOOC completion (Blackmore, 2014), this study examines participants' motivation and digital knowledge characteristics via data collection using a new set of pre assessments and post assessments.
The objective of this study is to examine relationships between motivation and digital aspects influencing participants to attend a MOOC.In addition, using information obtained from MOOC completers, this examination is extended to evaluate pre-reports and post-reports.To accomplish this objective, procedures were executed (a) identify information among MOOC completers and non-completers, (b) evaluate psychometric properties of the post-measurement tool, and (c) correlate initial and ending information from MOOC completers.
For the second set of analyses, we used the "Encuesta final sobre intereses, motivaciones y conocimientos previos en MOOC" ("Ending assessment for interests, motivations, and previous knowledge"; EFMC-MOOC; Valenzuela, Mena, & Ramírez-Montoya, 2017b).The EFMC-MOOC is a mixed-format, 17-item tool designed to evaluate the changes in motivation and knowledge that participants experience after attending a MOOC related to the topic of energy.Given that the EFMC-MOOC was conceived to post-evaluate participants' motivation and knowledge, its second section emulates the EIIMC-MOOC tool in content and format.Examples of the items include "Este curso satisfizo las necesidades de formación que me llevaron a inscribirme en él" ("This course satisfied the training needs that motivated me to enroll in it"; motivation and interests) and "Creo que este curso me permitió adquirir los conocimientos básicos de los contenidos estudiados" ("I believe this course allowed me to acquire basic knowledge from the content explored"; acquired knowledge).Experts in education and methodology have evaluated the EFMC-MOOC for content validity, and its format and content have been piloted to evaluate examinees' comprehension (Valdivia, Valenzuela, & Ramirez-Montoya, 2017).For this study, the second section of the EFMC-MOOC was examined for its psychometric properties (the first section collects demographics).

Exploratory Factor Analysis
The descriptive statistics showed that the normality assumptions were met; the set of 13 items presented an absolute value smaller than 2.3 for skewness (mean of −1.39; range from −2.34 to −0.76), and kurtosis had a mean of 3.29 (range from -0.009 to 9.13).In the presence of large samples, absolute values greater than 3.0 and 10.0 indicate problematic skew and kurtosis indices, respectively (Kline, 2005).
The Kaiser Meyer Olkin (KMO) measure verified the sampling adequacy for the analysis; the result was .97,which is well above the acceptable limit of .5 (Kaiser, 1974).Bartlett's test of sphericity, χ² (78) = 2123.559,p < .00,indicated that correlations between items were sufficiently large for executing an exploratory factor analysis procedure. 122 An initial analysis was run to obtain eigenvalues for each factor in the data.Two factors had eigenvalues over Kaiser's criterion of 1.In combination, they explained 52.18% of the variance (see Table 3).This criterion is a good indicator for the number of factors that are tenable to retain when considering a combination of sample size (>250), and the average retained communality is .51or higher (Field, 2009).
Table 4 shows the item communalities extracted for this solution.The scree plot showed a clear inflexion that would justify retaining two factors (Figure 1).Thus, given the large sample size, convergence of the scree plot, and Kaiser criterion found on this solution, two factors were retained in the final analysis.

124
A clear pattern matrix was obtained for this two-factor solution (see Table 5).The items that clustered higher than 0.40 on the same components suggested that Factor 1 represents a motivation and interest dimension (6 items), whereas Factor 2 represents gained knowledge (4 items).Notes.Extraction method: principal axis factoring.Rotation method: oblimin with Kaiser; rotation method converged in 10 iterations.

Correlations
In terms of the correlation results obtained when preinformation and postinformation was obtained from participants who finished the MOOC, there were significant (p < .01level) outcomes across all factors examined.Motivation presented a higher correlation when examined with factors taken from the final tool (r = .606and r = .506for Factors 1 and 2, respectively).As for the other initial factors, previous general and specific knowledge correlated moderately significantly with final Factors 1 and 2, although previous specific knowledge presented a weaker relationship (see Table 6).

Discussion
MOOC environments are becoming an important setting for exploring learners' characteristics.
Accordingly, the results from this study support efforts to continue investigating such characteristics, especially to understand the participants' motivation, knowledge (previous and acquired), and levels of satisfaction.This research line is important because after a completer"s profile is identified, MOOCs can be personalized to engage attendance more effectively as a strategy to reduce dropout rates (Alario-Hoyos, Pérez-Sanagustín, Delgado-Kloos, Parada, & Muñoz-Organero, 2014).
When examining the initial information, the scores for motivation and previous general and specific content knowledge factors were higher for the completers group compared with the non-completers group.These outcomes also showed low effect sizes, suggesting that the results need to be interpreted cautiously; however, they are in agreement with the literature reporting that completers obtain significantly higher ratings because they have confidence in their ability to complete MOOCs successfully (Barak et al., 2016).
Moreover, it is notable that the scores were consistently significant across all factors, although the categories for grouping attendees (completers vs. non-completers) did not account for heterogeneous background profiles.Thus, future analysis to differentiate attendees' profiles should also consider reviewing other types of information (e.g., educational levels, work training, etc.) about participants to evaluate differences by subcategories as well.
As for the structure of the EFMC-MOOC, the results support the claim that this tool meets the initial validity and reliability standards.Item loadings for each factor suggest this tool measures participants' levels of satisfaction about the gains obtained after attending the MOOC.This satisfaction level can be evaluated by a two-factor structure involving (a) professional development gains and (b) technology skills growth.These factors correlate highly, but they are well differentiated (r = .737),and, together, they explain 52% of the variance, which is consistent with the findings reported in the literature when exploratory analyses are executed.In terms of reliability, the EFMC-MOOC shows internal consistency for the overall structure and across factors.An advantage of examining the psychometric structure of an instrument relates to the viability of interpreting students' scores properly, for instance, to identify students at risk (Farid, 2014).In traditional education, administering pre-assessment and post-assessment tools with similar content is a regular activity to evaluate learning; however, for MOOC environments, this activity is still developing (Chudzicki, Chen, Zhou, Alexandron, & Pritchard, 2015).Accordingly, the present results align with such efforts.Moreover, developing reliable measures provides opportunities for current efforts to track and understand participants' changes in behavior and performance occurring across MOOC attendance (Aiken et al., 2014;Perna et al., 2014).Future research projects could include using valid tools as formative assessments to track such changes, as it is desirable to have immediate measures rather than a delayed measure of situational interest (de Barba et al., 2016).
In terms of the pre-information and post-information derived from attendees finishing the MOOC, all scores from factors measured initially correlated significantly to the final scores.The results showed a consistent moderate association across variables.As in a previous report about the role motivation plays in perceived learning (Horzum, Kaymak, & Gungoren, 2015), the motivation factor measured in this study appeared to be the stronger variable associated with perceived satisfaction levels for attending a course.In contrast, the previous specific knowledge variable correlated less with the final information, and although prudence recommended when to interpret previous knowledge self-evaluation scores (Lui & Li, 2017), this finding agrees with reports asserting that this factor not only relates to engagement, but is also a strong predictor for success in a MOOC (Kennedy, Coffrin, & de Barba, 2015).
Overall, the findings from this study are consistent with the previous literature focussing on the need to understand attrition factors and motivational transition across MOOCs (Xu & Yang, 2016).Accordingly, it has also been suggested that pedagogical models should consider the technology practices involved (e.g., digital skills) to engage participants continuously to increase retention (Petronzi & Hadi, 2016).The combination of factors evaluated in this study (motivation, knowledge, satisfaction level) follows suggestions about not relying on behavioral aspects exclusively, but instead, including cognitive elements, as both aspects are related to MOOC engagement, and both increase the probability of completing a course (Li & Baker, 2016).

Finally
, examining potential relationships among information collected before and after attending a course and comparing initial profiles of completers and non-completers may have benefits in terms of orienting MOOCs to the work market because a solely academic-oriented objective can detract from participants' learning, as transfer of knowledge is not guaranteed (Sanchez-Acosta, Escribano-Otero, & Valderrama, 2014).Thus, future research should consider how motivation and previous knowledge result when MOOCs target different objectives, as in the applied project supporting participants from this MOOC.Moreover, data emerging for such research should also consider an open-access perspective because by nature, MOOCs comprise open-access learning materials.Thus, the results and tools derived from MOOCs should align to this perspective to ensure that they are innovative (McGreal, Mackintosh, & Taylor, 2013).

Table 1
Means by Group Significant at the p<.01 level.b) Lower and upper levels at 95% of confidence intervals of the difference.

Table 3
Total Variance Explained for the EFMC-MOOC Sample

Table 5
Pattern Matrix for the EFMC-MOOC Sample

Table 6
Correlations Between Initial and Final Information