Critical Issues in Open and Distance Education Research

Despite its mainstreaming into the broader educational ecology, open and distance education (ODE) still leaves much to be desired in terms of both practice and research. Inspired and informed by the author’s 35 years of experience as an ODE practitioner, researcher, reviewer, and editor, this article concentrates on 10 critical issues of ODE research that have long existed but may have a consequential impact on its healthy growth. The issues discussed cover scarcity of longitudinal research, paucity of scaling-up and generalization research, preference for success over failure presented in research, the need for a systems approach, lack of sociocultural sensitivity, technologization of research, scant attention to ODE for the underprivileged and disadvantaged, insufficient research on ODE policy, negligence of historical research, and disinterest in revisiting ODE theories. The causes of these problems are critically interpreted and their possible negative impacts on the field of ODE are explored in a concise manner. The purpose of this article is to encourage further discussion and debate on ODE research to sustain its presence and acceptance as a legitimate mode of education in the wider educational community.

Despite its increasing popularity, especially since the emergence of extended massive open online courses a decade ago, it was the COVID-19 pandemic that put ODE in the spotlight, almost overnight, turning it into the only option for the entire education sector, including institutions that previously cast doubt on it. ODE continues to gain momentum and is sure to remain post-COVID-19. No longer can dedicated ODE institutions claim ODE to be their prerogative. Never before has it been so imperative, against a backdrop featuring diversity in all senses, to inform ODE practice with meaningful, rigorous, and trustworthy research.
Nevertheless, despite abundance in quantity and a substantial body of solid research, ODE research is often criticized for, among other things, lacking methodological rigor (Panda, 1992;Simonson et al., 2011), being under-theorized (Perraton, 2000;Saba, 2000;Ukwoma & Ngulube, 2021), and focusing on isolated studies (Bulfin et al., 2013;Fryer et al., 2023). Inspired and informed by my experience as an ODE practitioner, researcher, reviewer, and editor for 35 years, I intend to focus on the following 10 issues deriving from ODE research: longitudinal research, scaling-up and generalization, success and failure, a systems approach, sociocultural sensitivity, technologization of research, ODE for the underprivileged and disadvantaged, ODE policy, historical research, and revisiting of ODE theories.
The purpose of this article is to call on ODE researchers to solve "significant problems" (Reeves & Lin, 2020, p. 1999) and hopefully spark discussions and debates on issues meriting our concern.

Longitudinal Research
ODE research tends to be predominantly characterized by one-off studies of short duration (Fryer et al., 2023). For example, a review of 238 empirical studies published in 2021 in eight ODE journals listed in the Social Sciences Citation Index and the Emerging Sources Citation Index (Peng & Xiao, 2022) shows that about 80% (n = 189) of the studies were cross-sectional in nature, with some studies whose intervention or treatment was only about one hour long (e.g., Arnò et al., 2021;Juarez & Critchfield, 2021) or even as short as eight minutes (Stadler et al., 2021). We are in dire need of research such as that by Dempsey et al. (2021), who conducted a study using three experiments to explore factors affecting interactive television (ITV) course satisfaction. The first experiment engaged 954 students from six instructors over five years, involving 36 face-to-face courses and 34 ITV courses, aiming to collect end-of-course evaluation data. The second experiment was intended to collect in-the-moment feedback right after a class meeting over a single semester. The third experiment, which lasted one week, was intended to investigate whether there was any difference between the students' experience in attending ITV courses with the instructor present in the same classroom and those with an instructor at a different location. Bond et al. (2021) reports a higher percentage (92.9%) of cross-sectional studies on emergency remote teaching in higher education. Isolated studies also make up a long-existing trend in the field of educational technology (Reeves & Lin, 2020) as well as other related areas of scholarship, for example, regarding e-portfolios (Scully et al., 2018), mobile learning (Song & Xiao, 2017), and learning analytics (Viberg et al., 2020). Findings from one-off, short-duration studies may be applicable to a particular event or intervention but are unlikely to be generalized to a wider scale and may also be biased. The effectiveness of an educational intervention needs to be tested over time. Instant effectiveness may be no more than an illusion. What is seen or experienced at the beginning of an intervention may be radically different from the reality if the intervention continues long enough, involving different learners and instructors in different learning environments and with different learning objectives and domains of knowledge (Selwyn, 2012). To ensure the validity and reliability of research outcomes, we need more longitudinal research to turn studies into iterative processes.
The effectiveness of an educational intervention takes time to become robust; only those interventions that survive the tests of both time and practice are worthwhile. A new direction of research suggested by Reeves and Lin (2020) also applies to the field of ODE, whereby we develop robust, multi-year research agendas focused on important problems and innovative solutions, judge our worthiness for promotion and tenure on evidence of impact rather than simple article counts, closely collaborate with practitioners, and establish our field as preeminent in meeting global problems related to education. (p. 1999)

Scaling-Up and Generalization
Echoing the call for longitudinal, iterative research is the issue of scaling-up and generalization. Moore (2008) aptly observes that "too much energy is wasted on research that produces ungeneralizable findings because they focus only on the analysis of specific programs within specific institutions" (p. 67), a finding echoed by Tynan et al. (2023). This is because much of the research in ODE is on a pilot stage or a very small scale in an idealized environment. Many factors of distraction are under control or not taken into (full) account. And because of the short duration, all participants concerned are still on their honeymoon with the experiments conducted, experiencing a sense of euphoria and seeing only the positive side. The seeming effectiveness also may not represent the actual effect and cannot be generalized to a wider variety of contexts. Therefore, scaling up research and generalizing research results to other contexts should be on the research agenda. People often complain about the glaring gap between research and practice or policy making. Whether this gap can be bridged depends to a great extent on whether research results can be generalized. Although no one questions the importance of practice or decision-making being informed by research, misinformation will ensue unless the research results are fit for the purpose or context. Practitioners or policy makers should not be blamed for not applying research results. If research results are conducive to practice, the temptation to draw on research will overcome resistance to changes.
Replication and extension research should also be encouraged (Dennis & Valacich, 2015) in the efforts to scale up research and generalize outcomes. It is astonishing that there was only one replication study that aimed to further test the "blended learners' online component challenges" scale previously developed by the authors and assess its validity and reliability, among other things (Bayyat et al., 2021), and 12 extension studies (building on the findings of an earlier study in an attempt to refine, enrich, and/or expand its findings) among the 238 empirical studies reviewed in Peng and Xiao's (2022) study.

Sociocultural Sensitivity
Education is culturally constrained; technology is culturally loaded. Neither is culturally neutral. Traxler and Crompton (2015) challenge the popular assumption that the practices and principles of mobile learning can be automatically transferred from one culture to another. Therefore, "academics should maintain an interest in, and sensitivity towards, the importance of local contexts, cultures and circumstances" (Selwyn, 2012, p. 216; also see Tynan et al., 2023).

ODE research should be socioculturally specific, an argument advocated by researchers from both
Western and non-Western backgrounds. Dominant ODE theories tend to be derived from the Western world (Gaskell, 2017;Jung, 2019c) and may not always suit other sociocultural contexts in which they are used to inform local practice (Gaskell & Mills, 2017). Jung (2019a) states, "We need to bring educational philosophies and ODE traditions from previously unexplored regions into the refinement and reinterpretation of ODE theories" (p. 120), elaborating and further developing extant ODE theories so that they are equally applicable to new contexts. In the light of this imperative, Jung (2020) proposes the contextualization-generalization-recontextualization cycle for ODE theory building and application.
Certain features of ODE are universal across different sociocultural contexts. However, other aspects of ODE may vary from one socioculture to another. Research into these differences will surely strengthen the explanatory and guiding power of ODE theories so that research results can effectively inform practice in a wider variety of sociocultural contexts. This line of inquiry can be deemed as a kind of generalization effort. Currently, the sociocultural dimension of ODE research remains less well researched than it deserves to be (Kimmons, 2020).

Success and Failure
Longitudinal research, including scaling-up and generalization, can never be completely free from failure. Nonetheless, very few studies, if any, report on the failure of an intervention. In his reflections on the 2017 World Conference on Online Learning held in Toronto, Prinsloo (2018) observes the absence of reports and presentations on failures, asking whether we could talk about failures. According to him, only success is worth reporting and circulating while failure is useless or a shame to admit even among fellow researchers. This is an unwritten rule not only seemingly conformed to by gatekeepers and researchers but also matching other stakeholders' expectations (Kram & Dinsmore, 2014). Pursuing "best practice" or what works is the ultimate goal of research (Bulfin et al., 2013;Reeves & Lin, 2020).
Nevertheless, "best practice" is often built on both success and failure rather than success only. Ignoring a failure will likely lead to repeating the failure or end in more failures rather than contributing to success. In fact, there is as much to learn from failure as from success. This is the rationale behind the instructional design of productive failure (Kapur, 2008). The concept of productive failure can be equally applicable to research. Given that an intervention is usually carefully designed in advance, aiming for good results, the rate of success tends to be much higher than that of failure. However, it is unusual to always have success but never failure. If we favor success over failure, or even choose to ignore failure, we simply refuse to accept that research "is as much about investigating the imperfect 'state of the actual' as it is about exploring the perfected 'state-of-the-art'" (Selwyn, 2012, p. 216). In addition to communicating inaccurate research results, we are also distorting the ecology of research, which will have a lasting harmful impact on the development of ODE.

A Systems Approach
A systems approach to ODE is essential to ensuring a generation of meaningful research results. The systems approach to ODE was first tested in Wedemeyer's Articulated Instructional Media project in the 1960s (Moore & Kearsley, 2012) and further developed and elaborated by Moore in his distance education theory (Moore, 1972). Distance education is "a system consisting of three sub-systems: a learner, a teacher, and a method of communication" (Moore, 1973, p. 663). This is a narrow-sense systems approach emphasizing the interplay between the three micro-level variables in teaching (Xiao, 2023b). Later, Moore expounded a broad-sense systems approach (Moore & Kearsley, 1996), which was further refined into the systems model of distance education (Moore & Kearsley, 2012). The systems approach has been adopted and practiced in the construction of open universities all around the world.
However, it should be borne in mind that this approach is not just instrumental in practice but conducive to research as well (Xiao, 2023b). Unfortunately, it is seldom used in ODE research.
For example, despite the complexity and dynamics of student success due to the interplay between factors at micro-, meso-, and macro-levels, research overwhelmingly tends to study select individual variables, hence often mixing up correlation and causation (Prinsloo et al., 2020). Causation is much more complex than correlation and can only be defined through a systems lens in most cases. Mistaking correlation for causation may lead to false effectiveness of an educational intervention or wrong diagnosis of causes of failure. The socio-critical model of student success proposed by Subotzky and Prinsloo (2011) is a good example of a systems approach to ODE research.
Focus on select individual factors seems to be a common feature of ODE research. For example, technological determinism or Silicon Valley solutionism is basically a reductionist approach to education, naively taking technology or algorithm as the panacea for all educational problems.
However, access does not necessarily translate into success, which is affected by many other factors (Gaskell & Mills, 2017). A systems approach to ODE research may set higher demands on researchers, involve more resources, and take longer to produce outcomes but will definitely generate more robust and consequential findings. As mentioned above, the COVID-19 pandemic has greatly accelerated the mainstreaming of ODE. Nevertheless, given that there has been no systems thinking to speak of in the emergency remote teaching during the pandemic (Bozkurt et al., 2020), a systems approach to ODE research cannot be overemphasized today.

Technologization of Research
Technological determinism is nothing new; a frequent criticism is that education is a very complex process and technology can never be a panacea for all educational problems (Prinsloo et al., 2020).
Similarly, technological determinism has permeated ODE research and seems to be a fashionable trend.
Nowadays, more and more research relies on the use of sophisticated software as research instruments for data mining or collection and analysis. Findings appear dehumanized and, hence, seemingly scientific and unbiased. However, education is an inherently human enterprise (Xiao, 2021). It can never be dehumanized, and neither can its research. Software can help researchers do a better job than manual work by, say, harvesting massive data and identifying patterns much more effectively and perhaps more accurately. However, it needs human input to interpret the meanings behind the statistics. Statistics can tell what happens but are less likely to explain how and why it happens. They may indicate a correlation between two variables but not causation, which is where human interpretation is needed, in particular, through a systems lens.
For example, it is often assumed that the more interactive a learner is, the more messages one posts, the better the learning achievement. Interactivity in this sense can be easily determined with the help of software; however, whether it affects achievement is far more complicated than software can tell.
First, distance learning involves three types of interaction: learner-content, learner-instructor, and learner-learner (Moore, 1989). Learner-learner interaction alone cannot represent the entire interaction process. Second, whether learner-learner interaction impacts on learning outcomes depends not only on the quantity but also on the quality of interaction. Surely, interaction must happen, and considerably, before it is likely to have an effect on learning. However, the content of an interaction matters more. If the interaction is irrelevant to the course objectives or purely phatic communication, how can we establish the causation between interactivity and learning progress? Even if it is relevant to the course objectives, we must also account for the contributions of learner-content interaction and learner-instructor interaction to learning achievement, among other factors. We cannot possibly attribute learning progress to one factor only, in this case, learner-learner interaction.
Commenting on the dehumanization approach to education practice, Bates (2016) criticizes computer scientists for being too proud of themselves and believing they can solve any problem with computer technology without knowing anything about the problem itself. When it comes to ODE research, some researchers take the power of software for granted but have yet to develop insights into ODE practice.
Such insights are essential to aptly interpreting software analysis results. Technology can replace neither human teachers nor human researchers; it can only play the role of a tool, not an agent.

ODE for the Underprivileged and Disadvantaged
Similarly, we should guard against the tendency to favor the technologization of ODE at the sacrifice of less technologized forms of ODE. Given the core values of ODE such as equity in education, social justice, knowledge sharing, democratization of education, and accessibility, Bozkurt (2019) rightly argues that despite all the exciting changes in ODE, we should not forget that "realities such as [the] information gap and the digital divide are still very alive and, therefore, it is important to keep the back door open" for those who are underprivileged (p. 510). These realities have been even more conspicuous during the COVID-19 pandemic (Bozkurt et al., 2020) and in developing countries (Jha & Ghatak, 2023).
Technology-supported ODE may not benefit all in that ODE needs to cater to those whom it is supposed to serve in the first place. The increasing use of cutting-edge technologies in ODE may even remarginalize the underprivileged rather than realize the core values of ODE (Gaskell & Mills, 2017). A typical case in point is massive open online courses (MOOCs). Despite the hype around democratizing higher education, the actual beneficiaries of MOOCs are mostly well-educated individuals with university degrees from relatively better-developed regions of the world rather than those who are in desperate need of higher education opportunities (Christensen et al., 2013;Fernandez-Dıaz et al., 2020). A review of publications in seven peer-reviewed ODE journals from 2009 to 2013 also shows that the underprivileged and disadvantaged population seldom constitutes the research subjects (Bozkurt et al., 2015). More research in ODE for underprivileged cohorts is needed post-COVID-19 (Hao & Xiao, 2021).

ODE Policy
According to Diehl (2018), "any distance education system depends upon policies, management, and an administration to provide the guidelines and leadership that are required for success" (p. 321).
Across the world, ODE is deeply influenced by government and institutional policies, which play a key role in shaping the ODE landscape. A pertinent case in point is the pivot to online learning during . The former consists of four parts, with the third part dedicated to management, policy, and administration but including only one chapter focusing on policy (Davis, 2018). The latter does not have a chapter with an exclusive focus on ODE policy (although Paul's [2014] chapter has a section on educational policy), despite the frequent appearance of the word policy across the volume.
Reviews of publications in relation to technology-supported education also come to the same conclusion: that policy is an under-researched theme (e.g., Bond et al., 2019;Zawacki-Richter & Latchem, 2018). Given the role that policies play in shaping the development of ODE, policy-focused research cannot be overemphasized, especially when ODE has been integrated into campus-based education in various forms.

Historical Research
In the same vein, historical research is seriously understudied in the ODE literature, also not forming a thematic region of its own in the above-mentioned literature reviews. Fifteen years ago, Moore (2008) pointed out that "very few articles of a historical nature have been published in the past twenty years in any of the main journals" (p. 68), warning that there will be a price to pay if we ignore distance education history. Moore explains the paucity of research on ODE history from three aspects. First, historical knowledge is considered "of only marginal importance," hence giving place to the mastery and application of "currently fashionable technologies" (Moore, 2008, p. 68). Second, people are not familiar with the methodology of historical research. Third, the dominance of technological determinism leads to the neglect of "the amazingly rich trove of unanswered questions as well as the resources that wait for anyone who gets involved in historical research" (Moore, 2008, p. 69). Moore's first and third arguments are echoed by Selwyn (2012), who asserts that "the mere thought of digital technology compels many people to look forward rather than back, … anticipating what is about to happen with technology rather than attempting to make sense of what has already happened" (p. 216). Selwyn (2012) thus suggests that research and writing maintain "a sense of history" (p. 214).
There is so much to be learned from ODE policy makers, institutions, researchers, and practitioners about its history (Xiao, 2023a). Unfortunately, historical research has yet to attract due attention from the ODE community. Baggaley (2014Baggaley ( , 2017 laments the fact that we continue to make the same mistakes and waste time researching the same "old" questions raised in previous studies as a result of our ignorance of erstwhile ODE research, a point echoed by Mishra et al. (2021). For example, MOOCs have renewed interest in researching educational videos. Nevertheless, researchers often ask questions that were studied in depth by education television researchers in the 1960s through the 1980s (Baggaley, 2017). Another typical case in point is research regarding flipped learning, which is actually the norm of dedicated ODE institutions around the world but is heralded as an innovative pedagogy, hence a hot research topic. Fewer studies, however, are appealing to researchers and practitioners from ODE institutions. Similar issues exist in online and blended learning in general. This situation needs rectifying now. At no time in the history of education has lack of knowledge about the history of ODE been as costly as it is now as a consequence of the increasing use of cutting-edge technologies and also of the mainstreaming of ODE into campus-based universities (Moore, 2014).

Revisiting ODE Theories
The importance of theory to practice is self-evident. Whoever denies this is doomed to act like a blind man feeling an elephant, merely relying on one's own imagination and perception to make decisions on practice. Jung (2019b) makes a case for updating and developing ODE theories in the digital age, saying that "there is a pressing need to revisit the time-honored theories developed in the era of correspondence education and traditional distance education" (p. 4).
With so many changes taking place in so many aspects of ODE over time, we certainly need to examine how well existing ODE theories cater to the changing landscape of ODE and, more importantly, where they need to be updated, revised, and further developed to better inform emerging practice. For example, research specific to a socioculture, as mentioned above, can be deemed as an effort to revisit existing ODE theories. The emergence of newer theories for digital learning spaces (Downes, 2023) can also be taken as the result of revisiting "old" ODE theories (also see Anderson, 2016); some examples of "old" theories include Wedemeyer's (1981) theory of independent study, Peters's (1973) industrialization theory, Moore's (1972Moore's ( , 1983Moore's ( , 1993 theory of transactional distance, and Holmberg's (1983) theory of guided didactic conversation.
ODE research must be underpinned by sound theories but should also aim to improve established theories. Only in this way can we ensure sustainable development of both ODE as a field of practice and of scholarship. As ODE is catering to increasingly diverse learners, finding favor in a wider range of contexts, and involving stakeholders from various sectors, the need to evolve existing theories is unquestionable.

Concluding Remarks
ODE has been mainstreamed into the broader ecology of education (Xiao, 2018). Nevertheless, more research efforts are needed, given that ODE has now entered a broader field of practice, involving a greater diversity and quantity of stakeholders and serving a wider range of needs and demands. In light of this imperative, this article focuses on ten issues that have long existed but may impact on the healthy growth of ODE. This is not an exhaustive list of issues of concern for ODE researchers but a cri de coeur for researching ODE's significant problems.
Indeed, now that ODE has entered the wider education sector through the "front door," researchers should seize this opportunity to strengthen its presence as a mainstream mode of education in the educational ecology. To this end, researchers need to resist the temptation of instant benefits brought about by one-off, short-duration studies and adopt a longitudinal approach to ODE research. Pilot studies must be further scaled up and generalized to maximize their relevance beyond the case under investigation and cater to the sociocultural uniqueness of particular ODE contexts. Meanwhile, despite a greater probability of success, educational experiments are never immune to failure; both successful and unsuccessful educational interventions merit research attention and efforts. Education is a fundamentally human transaction, and its complexity requires the adoption of a systems approach to as well as human agency in undertaking educational research; ODE is no exception. Given that underprivileged and disadvantaged populations have always been the concern of the ODE community, it is ODE researchers' moral and social responsibility to include members of this group as subjects of their research so that their needs can be duly nurtured. Compared with other sectors of education, ODE may be more politically driven, and the importance of policy research is self-evident. Equally critical are lessons from ODE history, knowledge of which may shape future development in the field as well as help practitioners avoid repeating the same mistakes again and again. Finally, existing ODE theories need to be revisited, with the aim of better informing emerging practices or practices in new contexts and even developing new theories.
It is worth noting that studies pertaining to the above issues usually do not bring immediate benefits to the researchers, impactful as they are. Patience and commitment are called for: two desirable qualities that all serious researchers, regardless of disciplinary backgrounds, must possess.