Using a Design-Based Research Study to Identify Principles for Training Instructors to Teach Online

Within the overall framework of design-based research, this paper reports on a study that focused on evaluating an online training course for online instructors. This intervention was designed as a possible solution to the problem facing many higher education institutions of how to provide quality, accessible training for mostly part-time instructors who are making the transition to teaching online. The research project explored whether the training course had any impact on the participants’ later teaching practice. The major outcome of this research study is the identification of design principles that can be used by other researchers and practitioners designing online instructor training.


Introduction
Within the overall framework of design-based research (Barab & Squire, 2004;Design-Based Research Collective, 2003;Sandoval & Bell, 2004), this paper reports on a research project that focused on evaluating a training course for online instructors: MarylandOnline's Certificate for Online Adjunct Teaching (COAT) course. COAT was designed as a possible solution to the problem facing some higher education institutions of how to provide quality, accessible training for part-time instructors who are making the transition to teaching online. COAT is a nine-week, online, instructor-led course that was designed to prepare adjunct faculty to teach their first online course. Although designed specifically for adjunct faculty with no online teaching experience, COAT participants were found to be more diverse than originally planned for with full-time faculty, administrators, and instructors with extensive prior online teaching experience enrolling in the course (Shattuck, 2013). This article reports the findings from a study that explored the impact of the COAT course on the participants' later teaching practice.

Design-Based Research
Design-based research (DBR), also called design experiments (Brown, 1992), design research (Collins, Joseph, & Bielaczyc, 2004), and educational design research (McKenney & Reeves, 2012), has generated increasing interest among educational researchers in the last decade (Anderson & Shattuck, 2012). Wang and Hannafin (2005) defined DBR as "A systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings and leading to contextually-sensitive design principles and theories" (pp. 6-7). DBR positions researchers with practitioners as part of a team that works together, usually over an extended period of time, to provide a solution(s) to a practical problem that faces a specific educational context. DBR studies use the term intervention to denote the object, activity, or process that is designed as a possible solution to address the identified problem. McKenney and Reeves (2012) identified intervention as a broad term used "to encompass the different kinds of solutions that are designed" (p. 14); these solutions include educational products, processes, programs, and policies. This study identified the COAT course as the intervention that was developed as a potential solution to the perceived need for better high quality training for online adjunct faculty.
DBR projects can span many years with multiple research cycles that focus on the iterative stages of the project analysis, design, development, implementation, and evaluation phases. In order to clearly explain how the research reported in this paper was positioned within a collaborative, ongoing DBR project, it is helpful to use McKenney and Reeves' (2012) generic model for design research (GMDR) to provide an outline of the COAT project phases. The GMDR consists of three main phases, analysis and exploration, design and construction, and evaluation and reflection, that lead to the two eventual outputs of increased theoretical understanding and effective intervention maturation. The three phases of analysis/exploration, design/construction, and Vol 14 | No 5 Dec/13 188 evaluation/reflection interact with ongoing practice as the intervention is adopted, enacted, and sustained (implementation) in a particular educational setting while information about the intervention is disseminated and diffused to a wider audience.
Using the GMDR to frame the COAT project, this study is situated in phase three: evaluation/reflection. Table 1 briefly outlines how the COAT project aligned with the initial iteration of the three phases of the GMDR with implementation added to the second phase in order to include how the COAT project implemented the first iteration of the training course after a successful pilot. and discussion of the design/construction phase which included a pilot run of the course are published elsewhere (Shattuck, Dubins, & Zilberman, 2011). This paper focuses on the initial evaluation/reflection phase of the COAT project.
DBR has proved to be an effective approach for other research projects focused on the design and evaluation processes of instructor training programs and initiatives (Dede, Ketelhut, Whitehouse, Breit, & McCloskey, 2009 (Collins et al., 2004, p. 20); ethnographic research describes in detail what and why relationships and events occur, but it does not try to change practice; and large-scale studies "do not provide the kind of detailed picture needed to guide the refinement of a design" (p. 21).
Action research has similarities with DBR in terms of collaboration, researchers having multiple roles, and reflection on practice. Typical action research, as opposed to critical action research, positions the teacher-as-researcher conducting "a form of disciplined inquiry, in which a personal attempt is made to understand, improve, and reform practice" (Cohen, Manion, & Morrison, 2007, p. 297). Action research can be Developing design principles is part of an ongoing DBR process that may eventually lead to theoretical understanding: The outcomes of design-based research are a set of design principles or guidelines derived empirically and richly described, which can be implemented by others interested in studying similar settings and concerns.
While the ultimate objective is the development of Vol 14 | No 5 Dec/13 190 theory, this might only occur after long-term engagement and multiple design investigations. (Amiel & Reeves, 2008, p. 35) One of the goals of the study reported in this paper was to use the detailed data that were collected and analyzed from the evaluation of the first iteration of the COAT course to articulate design principles that are relevant to other distance learning professionals and that are transferable to similar contexts.

Purpose and Research Questions
The purpose of this research study was to evaluate whether the content, structure, and instructional approaches of the COAT course effectively helped prepare higher education instructors to teach online, and, through reflection, to extract design principles that could prove useful for other researchers and practitioners working in the field of online instructor training. The research model used to evaluate the COAT course was Guskey's (2000) model of five critical levels of professional development evaluation. This study collected and analyzed data focused on evaluating Level 4: participants' use of new knowledge and skills. Guskey (2000) argued that most evaluations of professional development occur at Level 1 (initial participant reactions) as it is the easiest level to assess. However, data gathered from the lower levels are not informative for measuring the impact of training on subsequent practice. Guskey highlighted that it is challenging to make a connection between teaching practice and earlier training experiences:

Research Method
This study collected data using online, asynchronous, threaded discussion groups as focus groups to explore the research questions using online discussion boards within a learning management system (LMS). Turney and Pocknee (2005)  An online, asynchronous format was appropriate for the participants of this study for the following reasons. All COAT alumni were accustomed to interacting asynchronously using a discussion board as this was an integral part of the COAT course, and, as such, technical barriers were not expected to be problematic due to the participants' familiarity with these tools. In addition, the lack of visual clues is something that COAT alumni are used to dealing with in their roles as online learners and instructors. The LMS used to host the focus group discussion boards is a secure site that is passwordprotected, and participants were given pseudonyms to protect their anonymity. The convenience of interacting asynchronously meant that geographic and time constraints were less likely to impact the feasibility of setting up the groups. Krueger

Focus Group Participants and Logistics
The purposive sample for the focus groups was derived from the respondents to a questionnaire sent to all participants who had completed one of 11 COAT course sections that ran fall 2010 to spring 2012. All 126 (out of a possible 179) respondents to the online questionnaire were sent an invitation to participate in the focus groups if they had taught online after completing COAT which led to 24 COAT alumni participating in the focus groups.
Participants were offered a choice of five dates for the focus groups. Two dates were not popular which resulted in three separate groups that ran in July, August, and Each focus group was opened a few days early for participants to preview. The focus groups were left open for a week after day three finished, so that participants could make any changes or additions to their postings before the data collection period closed and data analysis began. No changes were made, but three participants (one in each focus group) did add a post the day after the third day. The researcher provided summaries for each day's discussion and a final summary of the whole focus group.
Participants were invited to make changes and corrections to these summaries. Only one clarification was suggested, and three participants verified that the summaries had captured what was important from their perspectives.
Focus group participants were not asked to give detailed demographic information about themselves, as this may have compromised their anonymity. However, some participants did disclose personal details in their introductions which included the information that participants held a number of professional roles within education with ten people saying they had worked or were currently working as adjunct faculty, five as administrators, three as full-time faculty, two as instructional technologists, and seven as Kindergarten-12 th grade (K-12) instructors. These roles were often held simultaneously with the K-12 instructors, administrators, and technologists working as higher education adjuncts too. Of the nine participants who gave information about how many institutions they were currently employed in, seven people worked in only one institution with two others identifying that they worked at more than one institution simultaneously. The institutions people worked at were varied with ten community college, two university, and one K-12 institutional type identified. Six people had not taught online prior to taking COAT, and 11 people had prior online teaching experience ranging from one course to over ten years.
Additional information about participants included prior experience as online students with 15 (63%) people having taken online courses before participating in COAT and three saying they had no online student experience prior to COAT. Participants also talked about the subjects they taught online which included accounting, art, astronomy,

Analysis of Research Results
This study drew on grounded theory techniques to inform data analysis decisions and followed Saldaña's (2009) recommendation to approach coding method choices with "pragmatic eclecticism" (p. 47) by letting initial data collection and review occur before deciding on which coding method(s) to use. Data analysis for the three focus groups was ongoing with preliminary analysis beginning after the first focus group ended and further analysis continuing through iterative cycles of initial and focused coding which informed data collection decisions for subsequent focus groups. Constant comparison (Strauss & Corbin, 1998) of new data against previously collected data and existing codes against new codes and emerging categories occurred throughout the process.
Detailed notes were kept in reflection blogs and analytic memos during the focus group data collection and analysis period.

Important Elements
All 24 focus group participants identified at least one element of COAT that had positively influenced their subsequent online teaching practice, and their comments were grouped into the category Taking COAT influenced subsequent online teaching practice. This category encompassed 138 quotations which were organized into a code family (see Figure 1). This code family included one category (placed top center in Figure 1); five codes that stated a particular element of taking COAT which influenced later practice or a specified impact on later practice that was attributed to taking COAT (shown with the relationship is a in Figure 1); one code that was seen as contributing to later teaching practice (shown with the relationship contributes to), and 11 subcodes (shown with the relationship is part of). Some of the codes are associated with more than one code and these relationships are depicted with is associated with relationship arrows. The numbers after each code in Figure 1 show the groundedness and density of the code. "Groundedness counts the number of links to quotations; density counts the links to other codes and memos" (Friese, 2012, p. 140 Experiencing being an online student. As shown in Figure 1, this code has four subcodes with a combined total of 28 linked quotations that were made by 16 (out of 24) focus group participants. To be included in this code, quotations had to explicitly highlight that the experience of being positioned as a student in the COAT course influenced participants' subsequent online teaching practice. These experiences could be positive or negative. The 11 quotations directly linked to the code were all positive comments about experiencing life as an online student affecting later online teaching practice. For some participants COAT was their first experience as an online student: I think participating as a student in the COAT training was of utmost importance. As a novice to online teaching, I really needed to be a student. Plus participating as a student opened my eyes to so many things I would have never even considered if I was just reading about these topics.
Others had taken online courses: "I have been an online student before, but this course helped me to focus on the student experience a little better because I was taking the course as an instructor wanting to provide a better experience for my students." This concept of benefiting from looking at their own teaching practice through the lens of an Experiencing the COAT course design as a student also led to changes in participants' later practice with eight quotations related to this idea. For example, one participant highlighted copying design features from COAT that she liked into her own courses: As a result of taking the COAT course I modified my own courses to incorporate some of the instructional design features from the COAT course… for example, a separate button for "Weekly Course Work," and separate folders for each week.
Three participants mentioned that COAT introduced them to features of the LMS they had been unaware of which led to them using these features in their own practice.
COAT was designed to give participants the experience of being an online student in a paced, facilitator-led, cohort-based course while learning about teaching in a similarly configured online learning environment. The code Experiencing being an online student and its four subcodes had 28 quotations that specifically highlighted that the way the course was purposefully structured to position participants as online students had an impact on later practice. This code is also associated with three other codes which were populated with quotations that can be seen to relate in part to participants' experience in the course as students, but primarily highlight different elements of the COAT course, some of which were expected results that aligned with the COAT project's planned outcomes, and others which were unexpected outcomes. An expected outcome was for participants to reflect on the role of an online instructor.

Reflecting on personal teaching role as online instructors.
This code and its two subcodes encompassed 24 quotations that focused on how taking the COAT course prompted participants to reflect on their role as an instructor and to make changes to both the types of activities they included in their subsequent online courses and their presence in their courses as a result of this reflection. For five participants, a key takeaway from COAT was their role shifting to being a facilitator of the learning process. Other participants discussed how COAT had made them think about their readiness for teaching in the online environment, the importance of their response time to students, how to deal with disruptive students online, and the need for This previous quote also demonstrates how this subcode/code can be viewed as being associated with the code Experiencing being an online student, as the experience of doing group work in COAT led to the participant incorporating group work into her subsequent courses which resulted in continued reflection on her teaching practice.
Another subcode that is also associated with the experience of being an online student in COAT is Modeling online teaching. This subcode contained three quotations that referred to participants learning from the COAT facilitator modeling good online teaching practice.

Being part of a community of learners.
Just over half of the focus group participants (13 out of 24) identified that a positive benefit of taking COAT was that it provided them with the opportunity to interact with other instructors who were participating in the course, and these interactions played a part in influencing later practice (shown as the relationship contributes to in Figure 1).
For example, one participant who had not taught online prior to COAT stated: "During the COAT class I appreciated learning from other students who either had taught online already or who were teaching in real time while taking the COAT class. Their stories and examples were invaluable." Participants who had already taught online before COAT also found being part of a community of instructors/learners beneficial: "I agree, the COAT course gave me a place to talk to other teachers, to troubleshoot issues with likefolks. We do tend to teach in isolation." Participants also expressed the wish that the community of learners had survived the end of the course. This desire to interact with other instructors separate from the assigned course curriculum or discussion prompts

Discussion
In order to reflect on the categories and codes that emerged from the data analysis phase, metaphors were used to abstract the findings to a more conceptual level.
Metaphors of immersion in a foreign/alien culture and of COAT being a pebble making ripples in pools of practice led to a further review of relevant literature.

Immersion in an Online Learning Environment
Two-thirds (16 out of 24) of focus group participants identified that their experience of being an online student in the COAT course had influenced their subsequent online teaching practice. Their comments about seeing an online course from a student's perspective, experiencing frustration with the course navigation, and feeling overwhelmed resonated with a metaphor of how making the transition to online learning and teaching can be compared to living in a foreign country. Being immersed in a new environment and faced with a different culture can first cause frustration, confusion, self-doubt, and fear that can then lead to rethinking what is taken for granted as normal or commonplace behaviors. In a similar way, moving from a campus-based to an online learning environment can be a discombobulating experience that can make instructors question what they feel they know as truths about teaching and learning. Brookfield (1993) argued that "experiencing what it feels like to learn something unfamiliar and difficult is the best way to help teachers empathise with the emotions and feelings of their own learners as they begin to traverse new intellectual terrains" (p. To talk about academic disciplines, professions, or even manual trades as communities or cultures will perhaps seem strange. Yet communities of practitioners are connected by more than their ostensible tasks. They are bound by intricate, socially constructed webs of belief, which are essential to understanding what they do (Geertz, 1983). The activities of many communities are unfathomable, unless they are viewed from within the culture…. In a significant way, learning is, we believe, a process of enculturation. (Brown, Collins, & Duguid, 1989, p. 33) COAT participants, as adult learners, found that the sometimes frustrating experience of being situated as a student within an unfamiliar, authentic online learning environment contributed to them rethinking their teaching practice.
The instructional approach of learning from the COAT facilitator modeling online teaching practice was informed by Bandura's (1977) social learning theory. This approach aligns with cognitive apprenticeship (Brown, Collins, & Duguid, 1989). The idea of apprenticeship also appears in research focused on the teaching beliefs and practices of higher education instructors with Kane, Sandretto, and Heath (2002) arguing that As preparation for teaching, university academics can be like. An instructor who has not taken an online class is at a real disadvantage.
Being situated as a student in an online course, that had content focused on online teaching theories and practice, allowed participants to observe the COAT facilitator model online teaching skills and strategies while she also participated in discussions about what she was doing and why she was doing certain actions.
In addition to learning from the COAT facilitator, alumni also learned from the other COAT participants, especially those who had prior online teaching experience. Over half of the focus group participants (13 out of 24) identified that taking COAT provided them with the opportunity to interact with other instructors and these interactions played a part in influencing later practice. The code Being part of a community of peers that emerged from the analysis of the data in this study is not a new concept and is embedded in the literature on communities of practice (CoP). Hildreth and Kimble (2008) argued that "Teaching is a very personal and 'individual' activity, yet teachers benefit greatly from links with other teachers, both with colleagues in their own establishment and with colleagues in the wider teaching community" (p. x).
Reflection on the learning that occurred in the community of COAT peers while they were immersed in an authentic online learning environment as students led to a review of the literature on situated cognition (Brown, Collins, & Duguid, 1989); CoP (Lave & Wenger, 1991); practice fields (Barab & Duffy, 2000; and collectives, networks, and groups (Dron & Anderson, 2007). Lave and Wenger (1991), in their book on situated learning, introduced CoP as nonacademic learning environments where novices learn from more experienced practitioners through legitimate peripheral participation, a form of apprenticeship. CoP as a concept has been applied to many fields since its inception, and Hildreth and Kimble (2008) considered it to have evolved into an "umbrella term" (p. xi) that now covers a range of group types with similar characteristics of being informal learning environments with voluntary membership of people interested in discussing practice and learning from each other while sharing resources and knowledge in a specific area. This knowledge is often tacit in nature. Andriessen (2005), in his research into the classification of knowledge community archetypes, concluded that "the same term of 'community of practice' has been applied to different types of communities, that is, to strategic communities, to informal communities and to informal networks" (p. 209). According to Dron and Anderson (2007)  In a similar manner, using Barab and Duffy's (2012) definition of educational practice fields, "Contexts in which learners, as opposed to legitimate participants, can practice the kinds of activities that they will encounter outside of school" (p. 34), COAT can be seen to be a type of practice field where participants practice online teaching and learning activities and skills separate from their real-life teaching situations. However, what is missing from this picture of COAT as a practice field is the reality that COAT participants were also already legitimate participants in communities of educators, and some had extensive prior online teaching experience which was demonstrated in the learning afforded by the community of peers. Figure   The first principle for designing training for online instructors emerged from the concept of being immersed in an unfamiliar online learning environment: Training for online instructors should be designed using a situated learning perspective that positions instructors as students in an authentic learning environment that is similar to the targeted teaching environment.
The recommendations that participants made for COAT to consider training for online teaching in nonLMS learning environments resonate with this design principle as the authentic learning environment could include any number of new and emerging learning technologies and social media. It is important to stress that all the design principles that emerged from this study are situational principles for designing faculty training. "Situational principles are ones that are not universal -they only apply in some situations. They exist on a continuum from situations that are very common (close to universal) to ones that are highly local (apply very rarely)" (Reigeluth & Carr-Chellman, 2009, p. 57). By including the voices of the diverse professionals who participated in COAT, it is hoped that practitioners working in similar situations to the instructors who participated in this study will find these design principles useful and possibly transferable to their teaching and learning environments.

Pools of Practice
All 24 focus group participants identified at least one element of COAT that influenced their subsequent online teaching practice. A primary influence was on course (re)design with 18 focus group participants making comments on how taking COAT had influenced their subsequent online course development, design, and redesign. Research participants also identified other influential elements of COAT such as learning about LMS features, pedagogy, online instructor role, and so on. In addition, other influences on online teaching practice were identified such as prior teaching experiences, other professional development opportunities, and institutional input. Using the metaphor of the COAT course being one of many pebbles making ripples in individual pools of practice helped to conceptualize some of the ideas captured in the data analysis. In the same way that a pebble is a concrete object with defined edges, COAT is a specific course with defined learning outcomes. The number and type of ripples from a pebble being thrown into a pool may be expected or unexpected depending on the situational circumstances. The impact of COAT on an individual's practice may align with the defined learning outcomes of the training, but may be unexpected depending on the instructor and the teaching context. Figure 3 portrays that a training course such as COAT is one of many possible influences on the professional practice of individual instructors and that the ripples from a training course are diverse, specific to a particular teaching context, and may be unanticipated. Figure 3 shows some examples of other possible influences on practice, but more pebbles are possible depending on the instructor's prior and current learning and work-related experiences.  (Re)designing online courses.
COAT was designed to prepare instructors to teach predesigned courses, and, although basic instructional design principles were introduced in the COAT course, the emphasis of the COAT content was on the delivery, not on the design of online courses. Some participants in this research study were disappointed with the lack of emphasis on designing courses. The data collected in this study highlighted that for many instructors course design issues were one of their major concerns, as they are asked to design and then teach online courses with limited, and sometimes nonexistent, prior online teaching and instructional design experience or institutional support. Second, instructors have an almost implicit responsibility to customize the course to meet the divergent and particular needs of students, thus many both want to and feel a responsibility to edit and improve existing courses that they are hired to teach. The assumption of the COAT project that participants would teach courses predesigned by teams proved to be incorrect. The findings demonstrated that COAT's impact was broader than planned with an unexpected outcome being that a key takeaway from the COAT course was its impact on participants' (re)designing online courses which highlighted the need for attention within the COAT project for offering optional skill development in course design. Participants in this study made suggestions that COAT could consider on how course design could be further explored either in the current course or in potential advanced courses. Other professional practice.
Another unexpected outcome of participating in the COAT course was the impact the training had on professional practices other than online teaching practice. Four focus group participants commented that COAT had affected their campus-based teaching practice. In addition to COAT's impact on campus-based teaching, six focus group participants said COAT had impacted their nonteaching practice in the areas of instructional design, managing online programs, and training faculty.
The second principle for designing training for online instructors emerged from the findings on unexpected outcomes from taking COAT: Training for online instructors should prepare participants for diverse teaching situations which might include requirements to (re)design online courses and opportunities to teach in emerging learning environments.

Conclusion
The COAT project originated in the desire of a group of instructional designers, online faculty, and administrators from various institutions to collaboratively tackle the growing problem of how to best provide quality, accessible training for instructors who are making the transition to online teaching. Using a DBR methodological approach within an overall interpretivist research paradigm, this study evaluated whether the content, structure, and instructional approaches of the COAT course effectively helped instructors teach their subsequent online courses. Research participants identified that the experience of being situated as students in an authentic online course focused on online teaching and learning positively influenced their later online teaching, campusbased teaching, and nonteaching professional practice. This study provided detailed feedback for the COAT project, and the design principles that emerged from this study may be of interest to researchers and professionals who are involved in developing training for instructors who teach online. The findings from this study expand knowledge and contribute to the research literature on training for both experienced and inexperienced online instructors. Other recent studies focused on training for online instructors have resulted in similar findings to this study (see, for example, Eliason & Holmes, 2010;Kukulska-Hulme, 2012;MacDonald, 2010;McQuiggan, 2011;Storandt, Dossin, & Piacentini Lacher, 2012;Terantino & Agbehonou, 2012). By combining the findings from these research studies that were conducted in different contexts, the resulting design principles become more grounded in diverse situations and learning environments thus adding to the likelihood of transferability as effective design principles and practices to additional contexts.