Applying the Rasch Model to Evaluate the Self-Directed Online Learning Scale (SDOLS) for Graduate Students

With the rapid growth of online learning and the increased attention paid to student attrition in online programs, much research has been aimed at studying the effectiveness of online education to improve students’ online learning experience and student retention. Utilizing the online learning literature as a multi-faceted theoretical framework, the study developed and employed a new survey instrument. The SelfDirected Online Learning Scale (SDOLS) was used to examine graduate student perceptions of effectiveness of online learning environments as demonstrated by their ability to take charge of their own learning, and to identify key factors in instructional design for effective improvements. The study applied the Rasch rating scale model to evaluate and validate SDOLS through a psychometric lens to establish the reliability and validity of SDOLS. Results from Rasch analysis addressed two research questions. First, evidence was found to generally support the new instrument as being psychometrically sound but three problematic items were also identified as grounds for future improvement of SDOLS. Second, the study assessed the importance of various factors as measured by the SDOLS items in contributing to students’ ability to self-manage their own online learning. Finally, the new instrument is expected to contribute to the work of various stakeholders in online education and can serve to improve students’ online learning experience and effectiveness, increase online retention rates, and reduce online dropouts.


Conceptual Framework
SDL has been a core theoretical construct in adult education and its research has evolved over time (Garrison, 1997). The existing literature on SDL has established an understanding of SDL as both a process and a personal attribute (Song & Hill, 2007, p. 38). Knowles (1975) defined SDL as adult students' ability to self-manage their own learning, and his work served as a how-to book for adult students planning to develop competency as self-directed learners (Long, 1977). Next, Caffarella (1993, pp. 25-26) described three principal ideas underlying the SDL process: (a) a self-initiated process of learning, (b) more learner autonomy, and (c) greater control by the learner. Under SDL, learners take primary responsibility for their own learning to meet their unique needs and achieve personal goals. Hiemstra (1994) interpreted self-directed learning as indicating individual adults had the capacity to plan, navigate, and evaluate their own learning on the path to their personal learning goals. By contrast, Garrison (1997) presented a more comprehensive theoretical model of self-directed learning, focused on the learning process itself containing both motivational and cognitive aspects of learning. This model integrated three overlapping dimensions related to learning in an educational setting: (a) external management, (b) internal monitoring, and (c) motivation. Finally, noting that SDL may function differently in different learning situations, Song and Hill (2007) examined various learning contexts (the online context, in particular) where self-direction in learning takes place. They argued that a better understanding of trans-contextual SDL attributes unique to the online setting contributes to better online teaching and learning experiences.

Online Teaching and Learning
The past decades have witnessed a rapid development of technology contributing to the rise of online teaching and learning, which has led to increasing interest in SDL (Chou & Chen, 2008). Known for its flexibility allowing learning to accommodate adult learners' busy schedules, online education has been the right place for them to take charge of their own learning. On one hand, online learning supports the selfmanagement dimension of Garrison's SDL model (1997). Online learning platforms lend themselves to greater learner control and autonomy, and ultimately, intrinsic motivation to learn. Due to being able to self-direct their own learning, learners more willingly turn what they have learned into professional practice (Beach, 2017). On the other hand, SDL is a critical characteristic a learner should possess for better adjustment and success in online learning, and for improving learning outcomes (Bonk, Lee, Kou, Xu, & Sheu, 2015;Heo & Han, 2018, p. 62;Hyland & Kranzow, 2011;Kim et al., 2014;Loizzo, Ertmer, Watson, & Watson, 2017). With interest, curiosity, and desire for self-improvement being among the most important motivating factors, learners are independent and autonomous in their use of various devices and places to learn, and for meeting their self-directed learning needs at their own pace (Bonk et al., 2015;Heo & Han, 2018, p. 62). Therefore, given the increasing opportunities for online learning, an area of particular interest to online learning researchers addresses the learner's ability to guide and direct his or her own learning (Beach, 2017;Hyland & Kranzow, 2011;Song & Hill, 2007, p. 27).

Existing SDL Measures
The measurement of self-direction in learning has been operationalized in studies that develop and validate instruments measuring various aspects of SDL, and, many times, revalidate these instruments again in culturally relevant settings, in different student populations, and so on.
Many SDL instruments are based on Knowles's andragogic theory (Cadorin, Bressan, & Palese, 2017;Knowles, 1975). First, in Guglielmino (1977), the Self-Directed Learning Readiness Scale (SDLRS) was developed based on Knowles's original concept of self-directed learning. Here, SDL readiness refers to the extent to which the individual possesses the abilities, attitudes, and personality characteristics necessary for self-directed learning (Wiley, 1983, p. 182). The SDLRS purported to measure the complex of attitudes, skills, and characteristics comprising an individual's current level of readiness to manage his or her own learning. Next, also adding to the SDL literature is the Self-Rating Scale of Self-Directed Learning (Williamson, 2007) measuring self-directed learning abilities in five dimensions. The instrument was subsequently revalidated in the Italian context to have a reduced number of items measuring SDL in eight dimensions (Cadorin, Bortoluzzi, & Palese, 2013;Cadorin, Suter, Saiani, Williamson, & Palese, 2010).
Besides SDL instruments designed for the general student population, SDL assessment tools have also been developed for students in specific domains. For example, in nursing education, multiple SDL instruments have been constructed measuring students' SDL skills to enhance the quality of their professional practice, including: (a) Self-Directed Learning Instrument (Cheng, Kuo, Lin, & Lee-Hsieh, 2010); (b) Self-Directed Learning Readiness Scale for Nursing Education (Fisher, King, & Tague, 2001); and (c) Autonomous Learner Index (Abu-Moghli, Khalaf, Halabi & Wardam, 2005).
Finally, many more SDL scales have been developed to serve various purposes and student populations, including: (a) Self-Directed Learning Scale (Lounsbury & Gibson, 2006); (b) Self-Directed Learning Inventory, for elementary school and college students (Jung, Lim, Jung, Kim, & Yoon, 2012;Suh, Wang, & Arterberry, 2015); and (c) Oddi Continuing Learning Inventory (Oddi, 1986). For a comprehensive listing of SDL measures, readers should refer to systematic reviews of SDL scale development studies, such as Cadorin et al. (2017) and Sawatsky (2017).

New SDL Instrument
Despite the existence of multiple SDL instruments, the literature review in this study has not identified any such instrument which is designed specifically for the online learning environment and dedicated to students who have had prior online learning experience. First, there are indeed a few SDL items written for the online environment buried somewhere in large scale surveys that measure multiple aspects of online education, such as items measuring student autonomy in the lengthy, 62-item Online Learning Environment Survey (Trinidad, Aldridge, & Fraser, 2005). A long, complicated survey tends to be associated with a low response rate, and when administered, may not collect any responses to the items specific to SDL. Second, among the existing SDL items for online education, many are formulated as prospective, instead of retrospective. Questions surveying students' future opinions when taking an online course may not always, when administered, be answered by students with prior online education experience.
Students with no prior online education experience can also respond to those questions by imagining what their experiences would be like if they were to take an online course, and responses from such students are likely to lack validity.
Taking into account the two issues outlined above, as part of a larger study, a new, concise SDL instrument, Self-Directed Online Learning Scale (SDOLS; Su, 2016) has been developed to use retrospective, instead of prospective, questions aimed to collect responses from only students with prior online learning experience.
The instrument measures students' SDL ability after he or she has taken an online course; it helps instructional designers determine if an online course meets the needs of students and identifies grounds for improvement. The construction of the SDOLS items was based on brainstorming, referring to existing SDL measures and adapting items from available SDL instruments (Abu-Moghli et al., 2005;Cheng et al., 2010;Fisher et al., 2001;Garrison, 1997, Guglielmino, 1977Jung et al., 2012;Lounsbury & Gibson, 2006;Oddi, 1986;Suh et al., 2015;Trinidad et al., 2005;Watkins, Leigh, & Triner, 2004;Williamson, 2007).
University faculty members with expertise in scale development and instructional design were also consulted to enhance the content validity of the instrument. Although SDOLS was developed based on the responses of graduate students in one research university in the Southeast US, the items are universal enough as an inquiry into the online learning experiences of students in other universities as well.

Research Questions
This study utilized a non-experimental survey research design, based in a post-positivist worldview (Creswell, 2013;Devlin, 2006) to explore graduate students' self-directed online learning ability. The study aimed to assess the psychometric properties of SDOLS, and examine issues related to graduate student perceptions of their SDL ability. Post-positivism holds "a deterministic philosophy in which causes probably determine effects or outcomes" (Creswell, 2013, p. 7). This study was passive in design, as there was no intent to manipulate any variables. The study was also exploratory, as it provided only preliminary psychometric evidence of the instrument and its use in investigating SDL and served as the foundation for examining future application of the instrument to broader contexts.
A Rasch measurement approach was taken using the rating scale model (RSM) to evaluate the psychometric properties of SDOLS (Bond & Fox, 2015). Rasch modeling and its variants have been used in similar research in online education (Choi, Walters, & Hoge, 2017;Wilson, Gochyyev, & Scalise, 2016). Besides scale validation, the study also examined students' perceptions of their SDL ability. Specifically, the study addressed two research questions: 1. What are the psychometric properties of SDOLS as measured by the Rasch rating scale model? 2. How important do students think various factors, as documented by SDOLS items, are in terms of contributing to their self-directed online learning ability?

Methodology Research Context
The draft SDOLS instrument was pilot-tested in the fall semester of 2014. A group of 10 graduate students taking an online course in that semester participated in the pilot testing. They were surveyed through Qualtrics after the conclusion of the semester and provided the feedback which was later incorporated into the final survey instrument. Their feedback revolved around identifying any aspects of the draft instrument that could lend themselves to misunderstanding or logical flow problems in the survey delivery and revising such aspects. After factoring in the feedback, the final instrument had 17 items and was administered to another, larger group of students. Table 1 presents the final SDOLS instrument; each item is a question related to how students take charge of their learning on a 1 to 5 Likert scale: 1 for strongly disagree (SD), 2 for disagree (D), 3 for neutral (N), 4 for agree (A), and 5 for strongly agree (SA). The 17 items make up two subscales-autonomous learning (AUL; eight items) and asynchronous online learning (AOL; nine items). Finally, all SDOLS items were worded positively; a higher score indicates a higher level of SDL ability. I was able to remain motivated even though the instructor was not online at all times. AUL Q09 I was able to access the discussion forum at places convenient to me. ASL Q10 I was able to read posted messages at times that were convenient to me. ASL Q11 I was able to take time to think about my messages before I posted them. ASL

Q12
The process of writing and posting messages helped me articulate my thoughts.

ASL Q13
My writing skills have improved through posting messages. ASL Q14 I was able to ask questions and make comments in online writing. ASL Q15 I was able to relate the content of online course materials to the information I have read in books.

Q16
I was able to understand course-related information when it was presented in video formats. ASL Q17 I was able to take notes while watching a video on the computer. ASL

Research Participants
After securing required Institutional Review Board approval, the study proceeded to obtain a nonprobability convenience sample. The sample consisted of all 909 graduate students in the aforementioned university who were taking online courses during the fall semester of 2014. In January 2015, these 909 graduate students were contacted by e-mail through Qualtrics, inviting them to participate in the study.
To address the possible low response rate issue common in online surveys, the study first sent a mass prenotification e-mail to all 909 students, informing them of an upcoming solicitation to participate in a study about their online learning experiences during the fall semester of 2014. After the data collection started, several follow-up e-mails were sent to remind the students to complete the survey. This continued until the data collection came to an end in April 2015. As an incentive to participate in the survey, all potential participants were entered into a draw to win one of five gift cards valued at $50 each. In the end, 238 participants provided complete responses to all 17 items, which, despite a low response rate of 26.2%, still led to a high student-item ratio of about 14:1, satisfying the criterion that the sample size should be at least six times the number of items for stable results (Mundfrom, Shaw, & Ke, 2005). Hispanic/Latino students, 188 White students, and 8 students who identified as being of more than one race. Finally, regarding marital status, the proportion of students who were married was moderately higher than that of students who were not (58.0% for married vs. 42.0% for not married).

Data Analysis
First, the responses of the 238 students were used to compute two sets of descriptive statistics: (a) cumulative response category percentages from all 17 items, and (b) response category percentages for each individual item. In Figure 1 (consisting of subfigures 1a through 1r) the statistics are presented graphically using bar charts (from left to right: SD, D, N, A and SA). Subfigure 1a represents the cumulative percentages of response categories from all 17 items put together. As is seen, as high as 77.7% of the responses were in the agree and strongly agree categories, indicating the participants tended to endorse item statements.
From subfigures 1b through 1r for individual items, the highest bar is always associated with either the SA or the A category, whereas the SD category is always selected least frequently. Therefore, all 17 items elicited similar response patterns and the participants tended to hold a favorable view of the statement for each item. Next, a unidimensional RSM-based Rasch analysis was conducted within Winsteps 4.1.0 to assess the degree to which students agree with item statements covering various SDL factors (Linacre, 2018). A unidimensional Rasch model assumes the survey items measure only a single underlying construct (e.g., ability to self-manage one's own learning) and establishes the relative difficulty (or relative endorsability) of each item statement with regard to that latent construct (Bond & Fox, 2015).
In the Rasch analysis, several aspects of SDOLS were investigated.
• Unidimensionality: The study investigated the fundamental Rasch analysis assumption of a single underlying construct. The assumption was assessed using a principal component analysis of the correlation matrix of the Rasch standardized residuals.
• Reliability and validity: To identify reliability and validity evidence for SDOLS, person/item reliability and separation statistics were examined. First, person separation classifies people whereas item separation verifies item hierarchy. Next, person and item reliability statistics refer to reproducibility of relative measures or score location (e.g., high reliability of persons (or items) means the probability is high that persons (or items) estimated with high scores actually do have higher scores than persons (or items) estimated with low scores).
• Rating scale effectiveness: The quality of the rating scale was critiqued to see if the response categories functioned as intended, and if students were able to consistently and correctly understand and interpret the response categories.
• Item and person measure quality: Infit and outfit mean square (MNSQ) statistics were used to assess the fit of the 17 items to the Rasch model. According to Linacre (2018, pp. 582-588), the ideal value of an item infit/outfit mean square statistic should fall between 0.50 to 1.50, and values exceeding 2.00 may suggest a noisy problematic item.
• Construct hierarchy: The Wright map was assessed to investigate the construct hierarchy of SDOLS (Wilson, 2005). The map visually presents relative difficulty of items and students' ability to selfmanage their learning. From top to bottom, items (to the right of the vertical line) are ranked from the least favorite item (hardest to endorse) to the most favorite item (easiest to endorse); students (to the left of the vertical line) are ranked from those who are most able to self-manage their learning to those who are least able to.

Results
The results support SDOLS as having excellent psychometric properties with the exception of three items.
The results also rank-order various factors of SDL regarding their relative importance to students' selfdirected online learning ability.

Dimensionality Analysis
Results indicate as high as 51.1% of the raw variance is explained by the Rasch dimension, with 30.9% attributed to persons and 20.2% to items. The largest secondary dimension, indicated by the first contrast under Winsteps, explains only 8.1% of the raw variance with an eigenvalue of 2.8, the strength of at most three items. Therefore, the ratio of the variance explained by items (20.2%) to that explained by the second largest dimension (8.1%) is about 2.50. Despite a possible secondary dimension made up of at best three items, it is also true that virtually all survey datasets consist of multiple dimensions (hardly any dataset is perfectly unidimensional), albeit to varying degrees (Royal & Gonzalez, 2016). Given evidence to support a single, primary underlying construct being measured by the Rasch dimension, the study concludes the unidimensionality assumption is reasonably satisfied for a unidimensional Rasch analysis (Linacre, 2018, pp. 557-558;Royal, Gilliland, & Kernick, 2014).

Reliability and Validity
Person and item separation statistics are, respectively, as high as 2.71 and 4.64. The high person separation statistic indicates SDOLS is sufficiently sensitive to distinguish between individual students with higher and lower levels of SDL ability, and the high item separation statistic suggests the student sample is large enough to confirm item difficulty hierarchy. Overall, these observations support the construct validity of the instrument.
Person reliability is 0.88 (i.e., SDOLS discriminates the sample into enough levels), and item reliability is even higher at 0.96 (i.e., the sample is large enough to precisely locate the items on the latent difficulty continuum). Person reliability being high could be due to ability variance being large. By contrast, item reliability being high could be attributed to large variability in item difficulty and a relatively large number of students.

Rating Scale Effectiveness
First, based on the response category probability curves in Figure 2, each category has a distinctive peak suggesting it is a meaningful endorsement choice for students at a specific ability level. Stated differently, students are able to sufficiently separate one response option from another, thus providing additional evidence of validity. Next, based on the shape of the response category count distribution in Table 3, it is evident that, although students do make full use of all five response categories, they still prefer to select those on the side of agreement (the agree category, in particular). Notably, almost all infit and outfit MNSQ statistics fall into the recommended range of 0.50 to 1.50 (Linacre, 2018, pp. 582-588) with only the outfit MNSQ for the SD category being only 0.04 points higher than 1.50. Besides, the category measures and Andrich threshold measures each advance in a stepwise manner, as expected. Third, the study examined the construct key map for the five response categories (see Figure 3). In the map, items are ordered from the least endorsable item Q13 (top) to most endorsable item Q10 (bottom). Evidently, the ordering of categories remains consistent as 1, 2, 3, 4, and 5 across all 17 items. Such consistency indicates none of the 17 items might cause misunderstanding or unexpected answers, thus supporting the validity of SDOLS (Ren, Bradley, & Lumpp, 2008). In summary, the results here support that the rating scale structure of SDOLS functioned in the intended way, and the response options were consistently and correctly interpreted by research participants.

Item Measure Quality
In Table 4, Q13 has an unusually large outfit MNSQ statistic (2.35). Because 2.35 > 2.00, it indicates that, with this item, off-variable noise is greater than useful information. Because this item degrades measurement, it should be revised to remedy the misfit. Besides Q13, Q01 and Q02 have relatively serious misfit issues with inflated infit and outfit MNSQ statistics for each item being greater than 1.50 (ranging from 1.52 to 1.84). These two items may be problematic and thus require further scrutiny to reduce their off-variable noise and improve their fit to the model. Table 4  and that the items have excellent discriminatory abilities (Linacre, 2018, pp. 526-532).

Item/Construct Hierarchy
According to the Wright map in Figure 4, students most easily endorse items Q10 and Q11, suggesting students highly value the ability to read posted messages at convenient times and to take time to think about their own messages before posting them. Next, students equally easily endorse items Q04, Q06, and Q09.
This indicates students believe discussion forum access at convenient places is a very important factor in online learning. Plus, students believe they take control of their own online learning and are confident of completing their work despite online distractions. Next, students easily endorse Q02, indicating they tend to work online during convenient times. Then, at the average item difficulty level is a group of six items:

Discussion
The study assessed the psychometric properties of SDOLS for measuring students' perceptions of their selfdirected online learning ability. Overall, the study supports SDOLS as having decent psychometric properties. Next, the study rank-ordered items regarding students' level of endorsement to offer insights into how important the attributes are for facilitating students' online, self-directed learning.

Psychometric Properties of SDOLS
Regarding psychometric properties of SDOLS, the study was based on the validity framework by Messick (1989) which has been implemented in the Rasch literature (Long, Wendt, & Dunne, 2011, pp. 388-389;Royal & Elahi, 2011, p. 369;Royal et al., 2014, pp. 458-459). According to Messick, validity is the integration of any evidence that impacts the interpretation or meaning of a score. Messick's framework is made up of six unique aspects of validity: (a) substantive, (b) content, (c) generalizability, (d) structural, (e) external, and (f) consequential. The Rasch analysis findings are discussed next within four of these six aspects of validity.
First, the fundamental assumption of unidimensionality is satisfied from a practical perspective due to the existence of a single, primary Rasch dimension which explains over 50% of the raw variance. This finding supports the substantive aspect of validity. Next, measures of reliability are extremely close to or above .90, which serves as evidence of the generalizability aspect of validity. Also, a diagnostic of the rating scale effectiveness indicates the response categories of SDOLS functioned as intended, and the participants were able to correctly and consistently interpret response options, which supports the structural aspect of validity.
An assessment of the item fit measures indicated the vast majority of the 17 items provided an adequate fit to the Rasch model; this finding speaks to the content aspect of validity. In summary, multiple pieces of evidence under Messick's validity framework supported SDOLS as being psychometrically sound, thus indicating the instrument is able to produce high-quality data.
Next, the analysis of item misfit reveals three items (Q13, Q01, and Q02, presented in order of misfit per Table 4) that did not provide adequate fit to the Rasch model. These items should be either removed or revised in future iterations of SDOLS.

Implications for Online Teaching and Learning
The SDOLS instrument addresses many issues associated with students' ability to self-manage their learning in online education. Because SDOLS offers insights into online students' feelings regarding various aspects of their SDL ability, the instrument is likely to be relevant to various stakeholders in online education including students, instructors, administrators, instructional designers, researchers, and so on.
For example, instructional designers may use the SDOLS data to identify grounds for improvements to an online learning environment, and as a guide in their work to improve their designs. In another instance, they may use the instrument as a diagnostic tool to measure online learners' readiness, and screen for learners whose self-directed learning ability is likely to be weak, before tailoring course designs in a way that improves online learners' success. On the other hand, data collected through the SDOLS instrument will enable instructors, administrators, and researchers to better understand how students' self-directed learning characteristics may relate to their success in online courses and completion of online programs, thus effectively contributing to improving online course and program designs. In summary, the study recommends the SDOLS instrument should be used for improving student online learning experience and effectiveness, increasing online retention rates, and reducing online dropouts.

Limitations and Future Research
The study is not without limitations, but limitations could be directions of future research. First, the research data could have been subject to self-selection bias, due to the self-selected sample being nonprobabilistic and therefore lacking in randomization, and to nonresponse bias exacerbated by a low response rate of 26.2%. Graduate students who chose to complete the online survey could be different demographically and behaviorally than those who chose not to. Second, the study has not assessed SDOLS on two other aspects of validity under Messick's framework. On one hand, because the study is the first introducing and validating SDOLS, there is no way to investigate the consequential aspect of validity, since the instrument was not previously used. On the other hand, findings from the study have not been correlated with those from others, so the external aspect of validity has not been evaluated. Third, the study is yet to examine the extent to which items remain invariant across various subgroups (e.g., by gender). In future research, a differential item functioning analysis could further assess whether SDOLS items function differently across these subpopulations. Finally, given the limitations described above, although the findings here support the scale as having potential, they are still preliminary regarding the issues in the two research questions. Fortunately, the research design and the analytic methodology are straightforward to implement, which makes it easier for future researchers to replicate the study under broader research contexts.

Conclusion
The study develops and validates SDOLS measuring students' ability to self-manage their online learning with a secondary goal of understanding their perceptions of various SDL factors. First, the study finds validity evidence for SDOLS from multiple perspectives under the Messick framework as well as evidence supporting SDOLS as a reliable instrument. The study also finds three problematic items (Q13, Q01, and Q02) based on criteria from the Rasch literature and suggests they should be revised or removed. Second, the study provides insights into students' perceptions of various SDL factors regarding contributions to their SDL ability.
As a final reflection, SDOLS is designed to survey students with prior online learning experience regarding their perceptions of SDL ability under the unique nature and features of the online education environment.
The preliminary results here indicate SDOLS can be administered with confidence to students for a reliable and valid measurement of their SDL ability. Because these characteristics of students ultimately determine whether self-directed learning will take place, the instrument is expected to help researchers better understand students' self-directedness in learning within the online environment, which in turn will contribute to the call for adequate social and academic support to enhance students' online learning experience and to reduce the rate of attrition. Besides, despite limited coverage in this study, SDOLS can be used for diagnostic purposes by analyzing the Wright map to identify, characterize, and rank-order learners regarding their level of self-directedness in learning (i.e., distinguishing students who are more independent learners good at determining their learning needs, planning, and implementing their own learning from students who feel more comfortable with more structured learning options such as traditional classroom environments). This diagnostic use of SDOLS is valuable because, until very recently, there have been few validated tools for identifying the self-directed learners (Sahoo, 2016, p. 167).