Resonance and Current Relevance of IRRODL Highly-Cited Articles: An Integrative Retrospective

This paper provides an integrative-retrospective analysis of highly-cited articles from past issues of the International Review of Research in Open and Distributed Learning (IRRODL). Download metrics of published articles were used to identify the top 100 downloaded IRRODL articles from 2000–2020. Publicly-available citation metrics for the top 100 articles were used to further refine the analysis and identify the top 10 most-cited IRRODL articles. These articles were then categorized into resonant themes to frame a discussion of the key topics in present-day online and distributed learning research and pedagogy. This paper is intended to provide a broad overview and will be of interest to those who are new to the field of online learning.


Introduction
The 2020 global COVID-19 pandemic, and the way it abruptly disrupted the educational landscape, has shed considerable light on a variety of open and distributed education considerations. As a long-time student of distance education and a parent of a 1st-year university student, I have watched with great interest as traditional brick and mortar institutions attempt to move quickly from classroom-based to online learning. It seems somewhat ironic that in this transition many traditional institutions of higher education make, what appears to be, no attempt to understand the evidence and scholarship from years of distance education practice. There is a large and growing body of research that can better inform the transition to online learning.
Terms, such as emergency remote teaching, pandemic pedagogy, hybrid learning, hyflex learning, and online pivot, proliferate in academic discourse on a variety of social media and professional learning network platforms. However, the lexicon is less important than the intent and underlying assumptions, as well the practical specifics of what is being attempted. Many traditional institutions view the transition to fully online or blended learning as burdensome and temporary. They consider, without evidence, online learning as being of lesser quality than classroom-based learning. Although the circumstances of the pandemic have necessitated rapid change, the evidence to guide this shift and, quite possibly, create sustainable and positive change has long been established. This body of evidence has been provided by scholars and practitioners who have had the time and experience to provide reasoned research and reflection. As Shattuck (2020) advises, "distance education literacy begins with the literature" (p. 179).
Coincidently, 2020 marks 20 years of publication of the International Review of Research in Open and Distributed Learning (IRRODL). There is no better time to undertake a retrospective study of the top IRRODL articles and strands of investigation that have left an enduring imprint through their impact on the field. This article provides an integrative content analysis of 20 years of open, online, distance, and distributed learning research, anchored by the top 10 most-cited IRRODL articles, according to Google Scholar on August 8, 2020.

Methods
Since 2000, the IRRODL has published more than 1,400 scholarly articles on issues related to open and distributed learning (ODL). The publication corpus primarily includes refereed research articles, along with notes from the field, book reviews, literature reviews, and special issues. In preparation for this retrospective study, all IRRODL articles from 2000-2020 were sorted by the number of PDF downloads; then, the top 100 downloaded articles were compared based on a variety of citation and utilization metrics, including Google Scholar citations, PlumX citation indexes, and Altmetrics, including social media mentions, and reference manager downloads. The IRRODL offers a number of format types, but the correlation between article downloads and citations has been established in a preceding bibliometric analysis of the journal (Martínez & Anderson, 2015). Though, in the present analysis, no direct article-toarticle correlation was made. Figure 1 combines the number of PDF downloads of the top 10 IRRODL articles with their corresponding Google Scholar citation metrics during the same week in August 2020.

PDF Downloads and Google Scholar Citation Metrics of the Top 10 IRRODL Articles
The goal of this analysis was to generate a list of the top 10 articles in terms of their enduring impact on the field of ODL. Figure 2 outlines the top 10 citation list in ascending order as identified by Google Scholar citation metrics on August 8, 2020 and on October 24, 2020.

Google Scholar Citation Metrics of the Top 10 IRRODL Articles in a 3-Month Period
Research impact is both a quantifier and qualifier, depending on the purpose and type of metric used. There is no formalized agreement on either the definition or utility of citation indices (Penfield et al., 2014).
However, it is known that open access journals, like the IRRODL, tend to have a higher research impact than other journals in general (Antelman, 2004;Björk & Solomon, 2012). This article will not repeat the cogent articulation of the problems with automated-quantitative citation metrics offered by many authors, including Anderson (2009) in the IRRODL editorial "The Ratings Game." Impact and citation indexing are both vague and mutable, especially when contrasting 20-year-old articles to more recently published articles, where a broader variety of access options and impact indicators are available. On the same theme, the longevity of an article's use is an important indicator (Fahy, 2013). In this regard, Figure 2 depicts the citation increase of the top 10 IRRODL articles throughout the duration of this study, indicating the continued and recent growth of citations.
That said, the purpose of this article is to provide a more qualitative, integrative-retrospective description rather than rely purely on the quantitative data. Specifically, this article explores the top-10 cited articles as guideposts to highlight content areas that have, at times, sparked a line of research or garnered continued interest in the field of ODL. This analysis consists of a thematic content analysis of some of the more impactful articles and a discussion on why these articles continue to resonate and, therefore, provide important fodder for consideration in present-day ODL inquiry, framed by the immediacy of the COVID-19 pandemic.
Resonance and Current Relevance of IRRODL Highly-Cited Articles: An Integrative Retrospective Dell

Key Themes in Online and Distance Learning Research and Pedagogy
The sudden uptake in interest in online and distance learning at all educational levels is a direct result of the unplanned need to pivot classroom-based learning environments to online learning environments to maintain educational continuity during the COVID-19 pandemic. But, like a Global Positioning System (GPS), we must set the direction for where we are headed by first taking stock of where we are in terms of knowledge about online and distance learning. This is made possible by the long history of educational technology research, a deep and wide field from which to take direction. It is not surprising that some of the most frequent questions about the pivot to online learning have already been asked, answered, and reasked across 20 years of IRRODL publications.

Building Community Online
Community building in online environments continues to be an area of significant interest. Studies about community started by considering questions of whether or not face-to-face (f2f) learning communities can be replicated online. Today, a more complex understanding has led us to believe that replication may not be the best goal and that some of the benefits of community building online, using asynchronous communications, can be more significant than what can be achieved in time-and space-bound, face-to-face classes (Garrison & Anderson, 2003). Scholarship on community building spans the entire publication life of the IRRODL.
Under the theme of community building, Alfred Rovai contributed two of the top 10 IRRODL articles.
"Building Sense of Community at a Distance" (Rovai, 2002) is the most-cited IRRODL article and has been cited more than 1,751 times since its publication. Citation metrics were sourced on several dates throughout August 2020, and continued citation shows that this article has enduring resonance and relevance. Rovai's (2002) article had already amassed 74 citations in the first 9 months of 2020. Scholars across diverse disciplines continue to rely on his article. Rovai (2002) draws together many important foundational studies, including Garrison et al.'s (2000) seminal article on community of inquiry, to assert that not only is community building possible in online and distance learning, it is a pedagogical imperative directly correlated to seven key community building considerations. Rovai's second article in this theme, together with Hope Jordan, is ranked number three.
In this article, "Blended Learning and Sense of Community: A Comparative Analysis With Traditional and Fully Online Graduate Courses," Rovai and Jordan (2004) investigate perceptions of community in faceto-face versus blended-learning environments and conclude that sense of community was stronger in the blended format.

248
The field and the variety of distance delivery tools, social media, and mobile devices and their affordances have come a long way since the question of online learning communities emerged. There is a significant body of literature that points to the why, how, and learning impact of community building in online and distance learning. Community building remains an important topic in 2020. The 2020 EDUCAUSE Horizon Report centres on the idea of socially-informed learning and calls on educators to consider going beyond community formation to include a deeper understanding of equity, diversity, and mental-health informed pedagogy within a learning community (EDUCAUSE, 2020). As Rovai (2002) suggests, an important part of community building is considering the "four dimensions: spirit, trust, interaction, and commonality of expectation and goals, in this case, learning" (p. 4).
In the years since Rovai's IRRODL articles, many scholars have added an important voice to the practical application of emotional attuning, and the instructor training needed to enact it, to create equitable, transformative, and interactive learning communities (Bali & Caines, 2018;Cleveland-Innes, 2019). Not surprisingly, investigations into designing for interaction form another highly-cited thematic area in the IRRODL.

Learner Interaction
In "Getting the Mix Right Again: An Updated and Theoretical Rationale for Interaction," Terry Anderson (2003) opines that getting the interaction mix right in a learning community is a complex endeavour.
Anderson (2003) could not have predicted how closely his words would resonate in 2020 when he wrote, "the landscape and challenges of 'getting the mix right' have not lessened in the past 25 years, and, in fact, have become even more complicated" (p. 1). In this article, Anderson presents the interaction equivalence theorem, a theory that has guided numerous later studies on online interaction (Figure 3).

Figure 3
Types of Interaction It is impossible to do the breadth of the theorem justice in a short article. The theorem is, essentially, based on the premise that online learning interaction is largely distributed among three main types of interaction (i.e., learner-learner, learner-instructor, learner content) and that instructional design choices can be balanced with what is doable and available without losing educational integrity (Anderson, 2003).
In 2020, as educational institutions struggle to determine how to make the pivot to online learning, "getting the mix right" is a constant challenge. Educators with no experience in distance education routinely replicate existing 2-to-3-hour classroom-based time slots in online synchronous sessions and then erroneously conclude that distance education is not as effective. But the mix matters, and it matters even more now as we experiment with class types, ages, and stages that have yet to experience the benefits of the right mix or of theoretically-grounded online and distance learning. Anderson's study has enduring relevance as interaction affordances and interactive content types continue to evolve.

Theoretical Foundations of Online and Distance Learning
Two more articles in the top 10 citation list focus on the theoretical grounding necessary to deliver quality online and distance learning: "Three Generations of Distance Education Pedagogy" (Anderson & Dron, 2011) and "Theoretical Challenges for Distance Education in the 21st Century: A Shift From Structural to Transactional Issues" (Garrison, 2000). In their article, Anderson and Dron (2011) take an important retrospective look at the roots of three major theoretical families that have influenced distance education: cognitive-behavioural, social constructivist, and connectivist theories. They analyze each of the theoretical families alongside cognitive, teaching, and social presence descriptors. Key messages help readers understand that context matters in designing for quality in distance education and that one generation of distance learning pedagogy does not supersede another in the design of the distance learning ecology. In a poignant and present COVID-related contention, Anderson and Dron state (2011) that "different models of teaching and learning have evolved when the technological affordances and climate were right for them" (p. 90). Definitely food for thought in the pandemic climate. Garrison's (2000) article, which appeared more than 10 years earlier, seeded this line of thinking in its review of a century of distance education considerations and questioned the kinds of distance education theory that will develop into the 21st century. The subject of transactional distance is discussed in most of the top 10 IRRODL articles, showing that this construct is a foundational part of understanding distance education delivery. Garrison's (2000) article appeared in the IRRODL's 1st year of publication; yet it has likely never been more important than it is now, as so many educational organizations are struggling to redesign learning with socially-safe-distance in mind.

Connectivism
Along the lines of theoretical development, Rita Kop contributed two of the top 10 IRRODL articles (Kop, 2011;Kop & Hill, 2008). In "Connectivism: Learning Theory of the Future or Vestige of the Past?" Kop and Hill (2008) agree with Garrison (2000), and Anderson and Dron (2011), about the need for a comprehensive theory of distance education. In their literature synthesis, Kop and Hill (2008) discuss the emergence of connectivism and, in particular, the discourse around its position as a theory. In 2005, Siemens and Downes described connectivism in two separate papers (Downes, 2005;Siemens, 2005). The principles of connectivism certainly fit in well with the three prior themes, including the idea that learning and knowing come from a diversity of opinions that can be accessed through technological interactions both human and non-human (Siemens, 2005). However, Kop and Hill (2008), conclude that connectivism may not be sufficiently developed to be positioned as a comprehensive theory of distance education.
Three years later, Kop (2011) are more questions than answers, but that "the synergy between mobile learning and distance learning…holds enormous potential" (Traxler, 2007, p. 10). Four years later, Park (2011)  Mobile learning in all its forms continues to resonate throughout the world. Traxler and Park could not have known how important the affordances of mobile learning would become during a global pandemic. The pandemic brought to light how many learners all over the world, in both developed and underdeveloped countries, do not have access to reliable internet; and cellular or mobile device access became the only way that educational continuity could be attempted. Opening Access to Knowledge (McGreal, 2013);in 2015, OER andMOOCs (McGreal, 2015); and in 2017, Outcomes of Openness: Empirical Reports on the Implementation of OER (McGreal, 2017). While no particular article on OER appears in the top 10 citation list, OER articles have enjoyed an extensive readership in the 20 years of the IRRODL's publication.
The pandemic pivot to online learning has, in many ways, highlighted the need for both OER and OEP.
Articles about mobile learning and OER, universal education, and K-12 research and practice have proven very popular, with large numbers of PDF downloads. The concept of OER continues to grow in both awareness and complexity. Figure 4 shows the three most downloaded IRRODL articles on the theme of OER, each of which was downloaded more than 4,000 times: "Open Education Resources and Mobile Technology to Narrow the Learning Divide" (Ally & Samaka, 2013), "Open Content and Open Educational Resources: Enabling Universal Education" (Caswell et al., 2008), and "Research and Practice in K-12 Online Learning: A Review of Open Access Literature" (Cavanaugh et al., 2009).

Figure 4
The Top Three OER Articles in the IRRODL by PDF Downloads Attention to the affordances that embracing OER and OEP can provide in terms of access has been amplified due to the pandemic. For several years, the adoption of OER has been highlighted as an important Horizon Report educational trend (Johnson et al., 2015). Pedagogical shifts have also been highlighted in a UNESCO directive that states, the COVID-19 crisis has resulted in a paradigm shift on how learners of all ages, worldwide, can access learning. It is, therefore, more than ever essential that the global community comes together now to foster universal access to information and knowledge through OER (Huang et al., 2020, p. 3).
OER and OEP have many benefits that are in direct alignment with the themes discussed in this paper, including diverse community building, increased interaction, and improved digital pedagogy (Adam et al., 2020;Jhangiani, 2020).

Classroom-Based -Online Comparison
No discussion about the pandemic pivot to online learning would be complete without highlighting historical articles that specifically speak to the proclivity to draw comparisons between classroom-based and online distance learning. "Online learning carries a stigma of being lower quality than face-to-face learning, despite research showing otherwise" (Hodges et al., 2020, para. 4 Russell, 1999). In fact, some of the authors in the top 10 citation list have highlighted ways in which distance learning may be superior in terms of educating for critical thinking, facilitating critical discourse, and prompting reasoned reflection (Garrison & Anderson, 2003). Garrison and Anderson (2003) advise that educational opportunity afforded via the internet "does not represent a reinvention of the educational transaction. But it does call for a refocusing and rededication to traditional higher education ideals…ideals that can be brought back within our grasp by technological developments" (p. 18). This sentiment exemplifies the idea that attentive use of technology can support meaningful educational goals.
One of the top downloaded IRRODL articles in this theme is coincidently titled "Replacing Face-To-Face Tutorials by Synchronous Online Technologies: Challenges and Pedagogical Implications," written by Kwok Chi Ng in 2007. Long before the world was thrust into panic and continuity-driven substitution behaviour, Ng found that in straight replication of classroom-based learning to online learning, students "indicate dissatisfaction with the one-way communication and teacher-control functionalities of the system" (Ng, 2007, p. 1). This finding highlights the importance of many of the themes discussed above, in terms of distribution of knowledge, and reiterates the necessity to bridge the move to online educational offerings with considered reflection from the literature.

Conclusion
Collectively, the IRRODL articles that have been often cited, downloaded, and viewed tell us a cogent story.
The frequency and recency of the citations of some of the older articles point to a continued interest in some of the foundational elements of quality ODL design. Purposeful attention to pedagogical theory, transactional distance, interaction, community building, and equitable and sustainable open access and mobile access options is much more important than replicating classroom-based content structures and lecture-based or transmission-based teaching methodologies that all too often dominate traditional educational structures.
When considered together, the often-cited articles and the content analysis show that negotiating the shift to online learning in a beneficial way is as much an educator mind shift as it is a shift in delivery modality.
"Education is a purposeful activity, and theory provides us with the understanding necessary to take effective action" (Garrison, 2000, p. 3). When we as educators respond to the call for the development of a deeper understanding of equity, diversity, and mental-health informed pedagogy within a learning community (EDUCAUSE, 2020); we realize that there is room to look more deeply at some of the democratic, distributed, and open affordances that well-planned ODL can offer.
In a recent webinar about the future of campus culture, hosted by the University of Toronto Press, the pandemic reality was referred to as "a reckoning." The reckoning was described as a time to review some of the historic educational practices that are based on structures of controlled time, voice, and content (James et al., 2020). These frequent calls may be best addressed by adopting an empathetic lens and spending time in the shoes of distance education students. The mastery of online learning pedagogy is much more than a pivot. It is a worthwhile investment to be a student of good ODL practices and a student of literature, by pursuing the 20 years of scholarship that the IRRODL has to offer. Happy Anniversary IRRODL.