Improving the K-12 Online Course Design Review Process: Experts Weigh in on iNACOL National Standards for Quality Online Courses

Within the K-12 online learning environment there are a variety of standards that designers can utilize when creating online courses. To date, the only research-based standards available are proprietary in manner. As such, many jurisdictions have begun adopting online course design standards from the leading advocacy organization, which that have yet to be validated from a research perspective. This article reports on the second phase of a three-stage study designed to examine the validity and reliability of the iNACOL National Standards for Quality Online Courses. Phase two utilizes two groups of expert reviewers to examine and provide feedback with goal of further refining these standards (after the standards had been scrutinized through the lens of the available K-12 online learning literature).


Methodology
Upon completion of the construct validity phase of this research initiative (see Adelstein & Barbour, 2016), the next stage was the content validity of the revised rubric. The purpose was to test the design of the new rubric through expert review. It was recommended to involve content-area experts, as content validity is a result of their verification that the rubric meets the standards as outlined in phase one (Roblyer & Wiencke, 2003;Taggart, Phifer, Nixon, & Wood, 2001). Roblyer and Wiencke (2003) denoted that a properly designed rubric used in educational technology is a meaningful way to both assess and guide practitioners. It should not be surprising to see a leader in the field, such as QM, used a rubric for their proprietary design standards during the creation process (Hixon, Barczyk, Buckenmeyer, & Feldman, 2011).
Eight experts, who were divided into two groups, reviewed the standards over the course of three rounds, examining each standard from a course design perspective. The experts were selected based on their background and experience in K-12 online education (see Table 1). Ron (all names are pseudonyms)  Researcher with approximately 20 years' experience in K-12 online learning.
Jason  Educator with experience in K-12 online curriculum and assessment design .

Louise
 Administrator with over 20 years' experience in K-12 online learning.
Amanda  Administrative responsibilities in online education for 8 years, 15 years overall in education. Joanne  Educator with over 20 years in both online and traditional K-12 and higher education.
Kim  Educator for 16 years, half of which in K-12 online learning. Connor  Educator, administrator, and designer with 12 years of experience in online education.
Kelly  Educator with five years' experience in K-12 online educational research.
Specifically, each group consisted of a researcher, administrator, designer, and teacher; all of whom had been directly involved with K-12 online learning.
During round one, each of the experts received a document containing the 52 iNACOL elements listed under the five main standards based on the results of the first phase of this research initiative. The document was color coded to indicate the nature of research supported for each standard (i.e., green for significant K-12 online learning research support, yellow for limited K-12 online learning research support, or orange for supported only by non-K-12 literature). There were also two additional sections added to the end of the document. The first section offered four new standards that were found to be present in the K-12 online learning research, while the second suggested combining elements that were seen as similar in scope. In round one, the experts were asked to rate the importance of each standard as it related to course design using a basic Likert scale (i.e., 1 for low relevancy, 2 for some relevancy, and 3 for significant relevancy). An area for comments was also included for each section.
After compiling the ratings from round one, a second document was created that listed the average rating for each of the standards and the comments that experts made. Based upon both the raw rating, as well as expert suggestions, the researcher made suggestions about revising or removing certain standards.
Experts were asked to select one of four options (i.e., keep the standard as is, revise the standard, combine with another standard, or delete the standard) and to provide a written rationale for that decision.
The responses from round two were again compiled in a new document that consisted of three sections: 1. standards where there were general agreement that should be kept as written, 2. standards where the expert feedback from the previous two rounds that had a clear consensus for either revision or deletion, and 3. standards that did not have a clear consensus from the experts and would require further discussion.
The experts' feedback from the previous rounds was listed under each standard. Round three consisted of 60-minute discussion with each expert group using Google Hangout that focused on the second and third sections of the round three document. 1 During the Google Hangout, the researcher facilitated discussion around the standards recommended for revision or deletion until a consensus was achieved on whether to revise or delete a particular standard, as well as the specific wording for any revised standards.

Results
In this section, we organize the data using the complete iNACOL National Standards for Quality Online Courses, broken down by section (see Tables 2-7). The revised iNACOL standards based on the expert review are provided in Appendix. there was overwhelming agreement to keep the majority of the elements in some form, with the exceptions of A5 (i.e., Multiple learning resources and materials to increase student success are available to students before the course begins) and A12 (i.e., Online instructor resources and notes are included).
Both elements were further discussed in round two, with experts still divided on how to move forward. All experts shared in round three that the phrasing of A5 was problematic, questioning how realistic it was to have all materials present before the course begins. Ron mentioned that due to the logistics of certain courses, having all material available: is technically not possible in some settings. Because you do an enrollment and that's when the students are there and they can't get access to the course until they are enrolled and they are enrolled at date of start. So it's not physically possible.
It was suggested, and agreed upon by Group A, to change the wording to "All course materials are available to students at the course start." Group B, on the other hand, did not think the element was appropriate. Amanda noted: you really don't know what's going to work until the students start the course and actually, you know, get their feedback as to what's working and what's not working. The other thing is, is that your course materials could be specific to that student as well, like some students may better at, um, a virtual lab or something else, and another student might learn better by watching a video or doing something else.
With other elements in the rubric discussing additional materials, Group B moved to delete A5 (i.e., Multiple learning resources and materials to increase student success are available to students before the course begins).
The round two discussions of A12 (i.e., Online instructor resources and notes are included) lead to a suggestion of combining it with A13 (i.e., Assessment and assignment answers and explanations are included) or keeping it as is. Both groups were quick to lean towards combining the elements. Ron suggested a further revision to include the pedagogy behind the material, as this understanding would help teachers "to grade [the assignment] appropriately, but they would also be given the grading rubrics which they would then communicate clearly in an easy to understand manner to the students and parents." The rest of the Group A members agreed.
Round one did include two suggestions from experts that were put forth in round two.  first week of the course) that confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address non-responsive students.
Combine with another: 0 Delete standard: 1 (Revise/Delete:1, Keep/Revise: 1) Element B10: The course provides opportunities for appropriate instructor-student and student-student interaction to foster mastery and application of the material.

Round two responses
Element B11: Students have access to resources that enrich the course content.

2.375
Keep standard as is: 3 Revise standard: 3 Combine with another: 0 Delete standard: 2 Much like Section A, Section B only had two elements, B9 (i.e., The course design includes explicit communication/activities (both before and during the first week of the course) that confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address non-responsive students.) and B11 (i.e., Students have access to resources that enrich the course content), which required further discussion in round two. The wording of B9 was a point of contention for a few experts, with the element only mentioning the importance of checking engagement before and during the first week. There was a strong overall push to revise the element to include practices throughout the course, which lead to the proposed rewording, "The course design includes explicit communication/activities at multiple intervals throughout the course that confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address non-responsive students." Group B was in agreement with suggestion, while Group A continued the discussion. Joanne mentioned that courses also come with tools to assess engagement, and these tools should be mentioned and used. She was also concerned with the length of the first sentence, so it was split into two sentences for final consideration.
B11 was debated at length in both expert groups. There were numerous suggestions from round two, such as revising to include examples or combining with either A5 (i.e., Multiple learning resources and materials to increase student success are available to students before the course begins) or B2 (i.e., The course is organized by units and lessons that fall into a logical sequence. Each unit and lesson includes an overview describing objectives, activities, assignments, and resources to provide multiple learning opportunities for students to master the content.). To start the conversation, the researcher offered the suggested revision, "Course design provides students with resources (e.g., alternate assignments, multimedia, simulations) that enrich course content". Group A agreed with the suggestion, with one edit recommended from Ron to include mention of Universal Design for Learning (UDL). It was suggested to include it both in this standard and at the beginning of the new rubric.
Group B had already eliminated A5, knowing that B2 and B11 covered much of the same territory. A B2 revision was previously agreed upon, but the group was quick to see similarities. Jason summed up the group's thoughts when he commented B2 seems to be talking about the overview, and B11 is what is actually there, I guess. Or at least describing the opportunities, then. To go along with the overview….It just seems like they need to be focused together, to make them one.
Further, Kelly suggested, "Course design provides students with resources (e.g., alternate assignments, multimedia, simulations) that enrich course content. Each unit and lesson includes an overview of the key objectives that incorporate a variety of activities, assignments, and resources to provide multiple learning opportunities for students to master the content." The experts in Group A agreed.
There were two expert suggestions from round one that impacted B2 and B10 (i.e., The course provides opportunities for appropriate instructor-student and student-student interaction to foster mastery and application of the material). The concern over B2 stemmed from the use of the word logical, which appeared to lock the element into a traditional mode of design. Group B was quick to agree upon the revision, which simply eliminated "that fall into a logical sequence" from the end of the first sentence.
Group A shifted their conversation to the use of units and lessons, with Joanne offering up modules. Ron agreed, adding When we design courses, we design them around weeks. Not units not lessons, but around weeks.
And I don't know if units and lessons precludes weeks, but I'm also not sure that it encourages that. And units and modules is better. But I would go around, I think organized by modules and take out the units.
Louise was unsure of eliminating units, but came to an understanding that the delivery depends on the instructor and mechanisms used. Therefore, modules could stand alone.
B10 was questioned by an expert for the use of foster, which implied that mastery only comes from the suggestions listed in the element. A revision, "The course provides opportunities (e.g., instructor-student and student-student interaction, assessments, access to resources) for mastery and application of the material," was suggested to the experts. Group A unanimously agreed, while Kelly had a further revision for Group B. Her thought was to keep the examples listed in the element similar to one another by relating each interaction to the student. The list was changed to "student-instructor interaction, student-student interaction, student-course content, student-LMS," and experts were content to move on.
After collecting the expert suggestions, a final decision was made on how to phrase B2, B9, B10, and B11.
The most complex of the revisions involved B2 and B11. The similarities brought up by Group B were logical, and the reasoning from Jason was enough to move forward with a combination. Group A's suggestion of changing units and lessons to modules was taken under consideration and added to the final wording. Group A's addition of tools and punctuation were accepted for B9, and Group B's wording was used for B10. Taken as a whole, Section C was positively viewed by the experts, with only C3 (i.e., Ongoing, varied, and frequent assessments are conducted throughout the course to inform instruction) averaging below a cumulative 2.5 score. The use of the word "frequent" was an issue for the majority of experts, and it was suggested to replace it with "quality." Both groups were quick to agree with the new wording.
Three expert suggestions were taken from round one and shared with the group as a whole. The first was C2 (i.e., The course structure includes adequate and appropriate methods and procedures to assess students' mastery of content), with the wording "adequate and appropriate" seeming too vague, leaving some experts to wonder who determines this. The initial comments from round two were fairly split between keeping the original wording and revising the element. Group A promptly decided that the original, while a bit vague, gave enough direction for design. Group B, on the other hand, moved to eliminate and not replace "adequate and appropriate." C6 (i.e., Grading rubrics are provided to the instructor and may be shared with students), according to one expert, suggested that the word "may" implies the rubric does not need to be shared with students.
Another expert was concerned that a rubric will be forced upon a qualified teacher. A rubric must be supplied in the course, but a qualified instructor should have final say over which rubric to use. While there was unanimous agreement amongst the experts that the instructor will share the rubric with students, there was some discussion as to the phrasing of the final revision. The suggested wording supplied used, "Suggested grading rubrics are provided to the instructor. The instructor will share a final grading rubric with students." Group B accepted the revision, but Group A was concerned over misinterpretations about the word "final," as some might view it in the context of a final exam. Ultimately, "final" was replaced by "chosen" in the element.
Experts were also concerned over language in C7 (i.e., The grading policy and practices are easy to understand), and looked to replace "easy to understand" with "clearly communicated." Group B unanimously agreed, while Ron from Group A suggested both phrases should be used. Louise and Ron offered continued revisions by adding "to students and parents" at the end, as they are the stakeholders who will interpret the policies.
C2, C6, and C7, had minor revision details that had to be accounted for. C2 was kept as is, as the wording, even though vague in nature, gives some direction to the designer. Group A's version of C6 was kept to avoid misinterpretation, and C7 was also finalized by group A. Much like C2, the wording gives appropriate direction to the designer.

Round two responses
Element D10: Course materials and activities are designed to provide appropriate access to all students. The course, developed with universal design principles in mind, conforms to the U.S. Section 504 and Section 508 provisions for electronic and information technology as well as the W3C's Web Content Accessibility Guidelines (WCAG 2.0).

Round two responses
Element D11: Student information remains confidential, as required by the Family Educational Rights and Privacy Act (FERPA).

N/A
Section D proved to be one of the most contentious for the experts throughout the process. D1 (i.e., The course architecture permits the online instructor to add content, activities and assessments to extend learning opportunities), D2 (i.e., The course accommodates multiple school calendars, e.g., block, 4x4 and traditional schedules), D6 (i.e., Prerequisite skills in the use of technology are identified), D7 (i.e., The course uses content-specific tools and software appropriately), D8 (i.e., The course is designed to meet internationally recognized interoperability standards), and D9 (i.e., Copyright and licensing status, including permission to share where applicable, is clearly stated and easily found) were all flagged for further discussion coming out of round one. There was concern that D1 was not appropriate for all instructors, so adding "where applicable" at the end of the element was suggested by an expert. Both groups unanimously agreed with the revision. D9 was quickly agreed upon as well, with both groups acknowledging the importance of copyright laws.

D2 was a lengthier discussion for both groups. All experts agreed that giving calendar examples limited
what an online course could fit into. Group B suggested that the element was not needed at, and voted to delete the element. Group A, on the other hand, simplified the wording and related it to the module design previously mentioned in Section B.
In round two, the experts were split on how to handle both D6 and D7. For both groups, the conversation began with a possible combination the elements, using the suggestion "Prerequisite skills, course tools, and course software are identified and appropriate in relation to the students and course." Louise mentioned to group A that the notion of prerequisite skills should be part of communication, but was not a function of course design. The other experts agreed, and removed "prerequisite skills" from the revision.
Group B was fairly adamant that D6 had to remain in some form or another. As Amanda put it I'm looking at this from trying to explain to a parent, you know, why their student shouldn't take this specific course because maybe they don't meet those prereqs.…prerequisite skills in the use of technology are identified. That is, that is something that they need to know how to do. How to navigate, you know, different parts of the course. And it might be course specific, meaning different courses will have different prereqs, but I don't think you can delete this.
After a bit more discussion, Group B approved the combination of D6 and D7 as suggested.
From the round one and two comments, it appeared that some experts were not familiar with what D8 (i.e., The course is designed to meet internationally recognized interoperability standards) was referring to. Even after further explanation, Group A was quick to delete the element, not viewing it as a necessary part of design. Group B took a different stance, viewing D8 as something that will be important in the future of design. Jason brought up that as instructors and students move from one proprietary software to another, it is important they have the ability to keep communicating and creating. The other experts agreed, and opted to keep the element.
As with the previous elements, a final version of the suggestions had to be obtained for D2, D6, D7, and D8. D2 (i.e., The course accommodates multiple school calendars, e.g., block, 4x4, and traditional schedules) appeared to have middling support from both research and the experts, so the decision was made to eliminate the element. The additional thought was that the modules in the course could be manipulated to fit any calendar, so there was not an overwhelming need to mention this as a design requirement. It would instead fall to the instructor and institution to make the course work for them.
There was a strong argument for keeping D6 (i.e., Prerequisite skills in the use of technology are identified), and the suggested combination of D6 and D7 (i.e., The course uses content-specific tools and software appropriately) was used. Finally, Group B's suggestion that D8 (i.e., The course is designed to meet internationally recognized interoperability standards) would be relevant in the future of design was enough to keep the element intact. Much like the previous section, E brought about much discussion as to how the elements pertained to course design, or if they did at all. E4 (i.e., Course instructors, whether face-to-face or virtual, are certificated and "highly qualified." The online course teacher possesses a teaching credential from a statelicensing agency and is "highly-qualified" as defined under ESEA), E6 (i.e., The course provider offers technical support and course management assistance to students, the course instructor, and the school coordinator), E7 (i.e., Course instructors, whether face-to-face or virtual, have been provided professional development in the behavior, social, and when necessary, emotional aspects of the learning environment), E8 (i.e., Course instructors, whether face-to-face or virtual, receive instructor professional development, which includes the support and use of a variety of communication modes to stimulate student engagement online), and E10 (i.e., Students are offered an orientation to taking an online course before starting the coursework) were all forced into the discussion for round two. E2 (i.e., The course is evaluated using a continuous improvement cycle for effectiveness and the findings used as a basis for improvement) and E3 (i.e., The course is updated periodically to ensure that the content is current) were suggested to be combined by an expert in round one, and E4 through E10 were all put up for deletion in various round one suggestions. The conversation in round three began with combining E2 and E3. Group A believed the standards did not fit into design and should therefore be eliminated. Group B saw it differently, believing that the findings from the evaluation should be used to improve and update the course. However, there was concern over the use of periodically and what that actually meant. Kim suggested adding "as needed" to the end of the combined revision, and the rest of Group B agreed.
During round one, it was suggested that elements E4 through E10 could be completely eliminated, as they do not relate to course design. Group A quickly moved to eliminate all remaining elements, with the exception of E6 (i.e., The course provider offers technical support and course management assistance to students, the course instructor, and the school coordinator). Louise was adamant that support should be built directly into the course, and not remain at the institution level It's the program that is delivering, designing, and then delivering this online course that makes the determination how the technical assistance is going to be provided. But the given is that within that course design, is the tool for technical assistance. But it's a programmatic decision. We don't care…who provides the assistance. As long as it can be found.
The rest of Group A agreed, and a revised version of E6 remained. Group B, however, came to the conclusion that E6, as well as the rest of the elements in the suggestion, could be eliminated. Kelly summed up the collective thought I think too there's a lot of them that are…higher level program. The course provider in terms of technical support, they're going to provide that. I see a lot of program level, like orientation for students, I think that's program related piece, too. That should be for all students taking any online course within the program or whatever it might be.
Group B concurred, and elements E4 through E10 were deleted.
Reviewing data and all reviewer comments, the suggested combining of E2 (i.e., The course is evaluated using a continuous improvement cycle for effectiveness and the findings used as a basis for improvement) and E3 (i.e., The course is updated periodically to ensure that the content is current) was accepted into the final rubric. Course design can be continuous and ongoing, meaning there should be an evaluation and improvement process in place. The revised version of E6 (i.e., The course provider offers technical support and course management assistance to students, the course instructor, and the school coordinator) was also added. Group A made a strong case for the need of technical support to be located in each course.
While it does not matter who eventually supplies the support, there should be access to help for every instructor and student directly within the course. In round one, the experts were presented with four additional elements and two revisions. Elements F1 (i.e., The syllabus promotes a student plan of work with attainable expectations), F2 (i.e., Technology is used to help increase self-efficacy of students), and F3 (i.e., Activities are designed to encourage students' individual interests and goals) were all readily accepted. In round two, there was a strong consensus to delete F4 (i.e., The instructor understands student goals and personalizes support). When mentioned in round three, there was no call for discussion from either group, and F4 was eliminated. F1 and F3 were placed in Section B under the Instructional Strategies and Activities subsection. F2 was located in Section D under the User Interface subsection.
The revisions were split with the experts. There was strong support in round one to combine B4 (i.e., The course and course instructor provide students with multiple learning paths, based on student needs that engage students in a variety of ways) and B6 (i.e., The course provides options for the instructor to adapt learning activities to accommodate students' needs). The suggested revised wording was not brought up by experts for further discussion and was added to the final rubric. However, most experts did not believe E6 and E7 were closely related, and the combined suggestion was dropped. In the end, both elements were ultimately recommended for deletion by both expert groups.

Discussion
The first section of the iNACOL National Standards for Quality Online Courses (i.e., Section A: Content) received a relatively high level of support from the experts during all three rounds of review. This result was not surprising, given the fact that these standards were primarily centered on structural and preparatory aspects of the online course. For example, in his study of course developers at a provincewide supplemental virtual school, Barbour (2005Barbour ( , 2007 reported several principles of effective course design that were focused on items like the consistency of navigation and structured course content. Similarly, students have also stressed the importance of structural and preparatory material in an online course. Gallini andBarron (2001-2002) reported that students preferred "a course structure with clear guidelines along with opportunities in the course to suggest alternative approaches to meeting course objectives" (p. 149), all aspects of structural and preparatory material found in an online course. Even most of the QM general standard areas (i.e., course overview and introduction, learning objectives/competencies, assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility) were focused on what online course designers would describe as structural and preparatory items (MarylandOnline, 2013).
Considering the significant tie between instructional and course design, expert support for the majority of the Section B (i.e., Instructional Design) elements was not unexpected. There was agreement that opportunity for higher order thinking, differentiating, and active learning be taken into consideration when designing the course. This was also supported by Mastropieri et al. (2006), who discussed how differentiating helped middle school science students achieve higher score on both in-class unit and state exams. The largest obstacle in Section B was actually related to the wording of certain elements. Experts agreed that resource materials could help with mastery, as have been seen in the K-12 online learning literature with algebra students who used virtual manipulatives (Cavanaugh, 2013). The wording and redundant nature of certain elements led to combining parts of Section B.
As a whole, the Section C (i.e., Student Assessment) elements were agreed upon and accepted by the experts in the revised rubric. This level of agreement was consistent with DiPietro (2010), who interviewed 16 online educators and found that participants agreed that assessment and feedback helped students engage with the content, along with meeting their individualized needs. In fact, as students become engaged with the learning, they are generally open to hearing feedback on how to improve and reach mastery of the subject material (Naidu, 2013). This feedback can be aided by the use of various resources, including rubrics, and by viewing course rubrics students become aware of expectations (Rice, 2012). As with Section B, the experts were mainly concerned with the wording of various elements, and moved forward with the section largely intact.
Unlike the previous three areas, Section D (i.e., Technology) garnered more discussion with regards to both wording and how the elements pertained to course design. The experts agreed that flexibility was important to scheduling online courses, a notion that Wicks (2010) also supported. However, the experts thought an element specifically about different calendar types was unnecessary, and that element was subsequently deleted. Further, there was also open debate over interoperability of the course, with some experts not seeing the necessity of integration. However, Watson and Watson (2007) noted that LMSs needed to "truly become systemic, integrating systems seamlessly to allow for improved collaboration across systems among stakeholders" (p. 32). While many of the remaining elements were eventually reworded or combined, the experts were generally agreeable with the general sentiment found in Section D (i.e., that understanding that the technology used played an important part in course design). This is consistent with earlier studies into the design of K-12 online courses. For example, Barbour (2007) interviewed six online educators who found that minimal and simple navigation gave a consistency that was appreciated by the students. The same group of educators, however, didn't feel that a course should shy away from multimedia and interactive elements, which could be used to enhance the curriculum.
Section E (i.e., Course Evaluation and Support Elements) was by far the most retooled area from the experts, but this is not to say that the elements were not important when creating an online course. As a few of the experts pointed out, all the elements were significant, but simply belong to different rubrics, as opposed to one focused on online course design. For example, the VHS required a 26-week class in how to design a course that utilized the LMS (Zucker & Kozma, 2003). Further, the Illinois Virtual High School (IVHS) used a similar practice shell as well, as was pointed out by Barbour, Kinsella, Wicks, & Toker (2009). IVHS also realized that continuous support was needed, and offered monthly professional development using face-to-face, synchronous, or asynchronous methods. However, that did not mean that the need for professional development and support should be included in a rubric designed to measure quality online course design. Yet, not all elements from Section E were deleted. For example, continuous course updates were fully supported by experts, and was a practice utilized by many K-12 online learning programs (Ebert & Powell, 2015).
The final areas were Section F and G (i.e., Suggested Elements and Revisions), which focused on the elements that were suggested for addition or elements that should be revised. The suggested elements looked to include student motivation in the course design structure, which was not a part of the original iNACOL standards. Three of the four suggestions were strongly supported by the experts, and found their way into the revised rubric without revisions. Both Chen and Jang (2010) and Kim, Park, and Cozart (2014) reported that motivation was an essential part of education, particularly in the online learning environment. As such, it was important that components that fostered student satisfaction in autonomy and self-efficacy were planned for within the online course design.

Conclusions and Implications
After examining the iNACOL National Standards for Quality Online Courses (2011) based on current literature and research in phase one of the study (see Adelstein & Barbour, 2016), a revised set of standards were reviewed by eight experts for phase two. The review occurred over three rounds, with the first two happening via e-mail and the third through a video conference. During all three rounds the experts recommended that certain elements be kept, combined, or deleted using their knowledge and understanding of online course design as a guide. The end result was a K-12 online course design rubric based off the original iNACOL quality standards that was further revised and refined.
The iNACOL standards, while praised by the experts, are purposefully broad, covering all aspects of online courses. The results of phase two of this study helped to bring essential online course design standards into focus. This specialized and more focused view may be able to help curtail how overwhelming the standards can appear, especially for those new to the field of K-12 online learning and designing online courses for a K-12 population. The revised rubric will allow stakeholders, including educators, course designers, and administrators, to focus specifically on the aspects of online course design, creating a stronger base upon which to build asynchronous online course content.
Having said that, the researchers would recommend that further expert review be conducted. Due to time and resource constraints, the experts for this study were limited to eight individuals. Also, while the first two rounds were vital to giving the experts some guidance, the majority of the discussion related to and refinement of the individual elements occurred during the video conference; which was limited to approximately one hour. It would be beneficial to provide the experts multiple opportunities to video conference over the course of the refinement of the standards. Finally, the iNACOL standards were chosen due to their open, non-proprietary nature. However, there are also other widely used standards that could be used or supplemented as the basis for this model of expert discussion. As for our own line of inquiry, with the expert review completed, the next phase of this particular study will test the application of the rubric. Three to five teams of two reviewers will apply the rubric against current K-12 online courses.
Using inter-rater reliability, the researchers will examine the reliability and validity of the rubric.  Barbour, M. K. (2005). The design of web-based courses for secondary students. Journal of Distance Learning,9(1). 27-36.

Subsection: Academic Content Standards and Assessments
A1: The course content and assignments are aligned with the state's content standards, common core curriculum, or other accepted content standards set for Advanced Placement courses, technology, computer science, or other courses whose content is not included in the state standards.
The content and assignments for the core courses are explicitly and thoroughly aligned to the credit granting state's academic standards, curriculum frameworks and assessments. Advanced Placement® courses must be approved with the College Board and other elective courses should be aligned to other nationally accepted content standards such as computer science, technology courses, etc.
A1 Rating (1 = not applied, 2 = partially applied, 3 = applied): A2: The course content and assignments are of sufficient rigor, depth and breadth to teach the standards being addressed.
The course components (objectives, assessments, instructional strategies, content, assignments and technology) are sufficiently broad, deep and rigorous such that successful students will have the knowledge and skills required by the standards upon completion of the course.
Before the course begins, students are provided learning resources that are utilized during the online course. These could include textbooks, instructional materials links to browser plugins, and other software, which students must install.

Subsection: Course Overview and Introduction
A4: A complete course overview and syllabus, which clearly states course goals and objectives, are included. Course goals are consistent with course requirements and are measurable in multiple ways.
Within the learning management system the syllabus and overview objectives are present, explicitly stated, and can be easily found by students. The syllabus and overview objectives include: course objectives and student learning outcomes; assignments; student expectations; time requirements; required materials; the grading policy; teacher-student, teacher-parent contact policies; the intended audience; and the content scope and sequence.

Element Further Explanation
A5: Information is provided to students, parents and mentors on how to communicate with the online instructor and course provider.
Instructor information is provided to students with contact, availability, and biographical information. Information on how to contact the instructor via phone, email, and/or online messaging tools is provided within the contact information. If regular contact with the instructor is required as part of the course, clear expectations for meeting this requirement are posted within the course.

Subsection: Legal and Acceptable Use Policies
A6: The course reflects multi-cultural education, and the content is accurate, current and free of bias or advertising.
The course creates equal educational opportunities for students from diverse racial, ethnic, social-class and cultural groups. The content is up to date, accurate and free of any bias. A variety of instructional and assessment methods, materials and assessments are used throughout the course, which allow students to demonstrate their achievement of the goals and objectives of the course.

Subsection: Modules and Resources
B2: The course is organized by modules. Course design provides students with resources (e.g. alternate assignments, multimedia, simulations) that enrich course content. Each module includes an overview of the key objectives that incorporate a variety of activities, assignments, and resources to provide multiple learning opportunities for students to master the content.
The course is organized by modules that fall into a logical sequence. At the start of each module, an overview is posted describing the activities, assignments, assessments, and resources to be used to complete the key objectives. A variety of activities, assignments, assessments, and resources are used to provide students with different paths to master the content. A wide variety of supplemental tools are clearly identified and readily available as well.

Subsection: Instructional Strategies and Activities
B3: The course instruction includes activities that engage students in active learning.
The course provides multiple opportunities for students to be actively engaged in the content that includes meaningful and authentic learning experiences such as collaborative learning groups, student-led review sessions, games, analysis or reactions to videos, discussions, concept mapping, analyzing case studies, etc.
B3 Rating (1 = not applied, 2 = partially applied, 3 = applied): Students are given a variety of activities, assignments, assessments and resources to allow them to successfully master the content. If a student is unsuccessful with mastering a particular concept or is not challenged with the current module, the course content provides the instructor with suggestions they are able to use in order to provide additional remediation activities or alternative assignments. The instructor has access to adapt the course to meet the students' needs by providing additional assignments, resources and activities for remediation or enrichments for the course. The syllabus provides an academic outline for students in the course, which includes academic expectations at specific intervals.
The course provides activities and assignments which are broad enough to allow for student connections. The connections are real world, such as personal interests, goals, or situations.

Subsection: Communication and Interaction
B9: The course design provides opportunities for appropriate instructor-student interaction, including opportunities for timely and frequent feedback about student progress.
Learning activities and other opportunities are created to foster instructor-student interaction. Students receive timely and frequent feedback on their progress that emphasizes the intended learner outcomes. The feedback is highly individualized, detailed, and recommends specific, individualized improvement, and strategies to encourage continued progress toward mastery.
B9 Rating (1 = not applied, 2 = partially applied, 3 = applied): B10: The course design includes explicit communication/activities/tools at multiple intervals throughout the course. The instructor confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address nonresponsive students.
Instructor-student interactions begin early enough in the course to confirm active participation by all students and continue throughout the course.

Element
Further Explanation B11: The course provides opportunities (e.g. student-instructor, student-student interaction, student-course content, student-LMS) for mastery and application of the material.
Learning activities and other learning opportunities are developed to foster student-instructor, student-student, and student-LMS interaction. The technology and course content encourage exchanges amongst the instructor and students through email, discussions, synchronous chats, simulations, lab activities and other group projects. Within the grading policy, guidelines defining student participation and expectations are provided.
Threaded and/or synchronous discussions are available for developing community, asking and finding answers to questions about the course, and around the content. Access is available to groups or individual students based on the purpose of the activity. Rules, roles, and expectations for the discussion are clear and posted within the discussion forum.

Subsection: Evaluation Strategies
C1: Student evaluation strategies are consistent with course goals and objectives, are representative of the scope of the course and are clearly stated.
The strategies used to assess students throughout the course are consistent with and aligned to what is presented in the course goals and objectives document posted within the course.
C1 Rating (1 = not applied, 2 = partially applied, 3 = applied): C2: The course structure includes adequate and appropriate methods and procedures to assess students' mastery of content.
Assessment types are matched to the level of knowledge being tested. Both formative assessments (that inform and support learning) and summative assessments (that demonstrate mastery) are a part of the course structure. Student-selected assessment options, enabling learners to demonstrate mastery in different ways, are available.

Subsection: Feedback
C3: Ongoing and varied quality assessments aligned with course learning outcomes are conducted throughout the course to guide student instruction.
The course provides quality and ongoing formative assessments to check for student understanding and to ensure they are prepared for the next lesson. Initial pre-tests may be provided to assess student readiness.
C3 Rating (1 = not applied, 2 = partially applied, 3 = applied): C4: Assessment strategies and tools make the student continuously aware of his/her progress in class and mastery of the content.
Feedback tools and procedures are built into the course to allow students to periodically selfmonitor their academic progress.

Subsection: Assessment Resources and Materials
C5: Assessment materials provide the instructor with the flexibility to assess students in a variety of ways. D2 Rating (1 = not applied, 2 = partially applied, 3 = applied): D3: Rich media are provided in multiple formats for ease of use and access in order to address diverse student needs.
Course makes maximum use of the robust capabilities of the online medium and makes these resources available by alternative means (video, CDs, podcasts).
Technology used in the course does not hinder the student's ability to accomplish the academic goals set forth by the syllabus.

Subsection: Technology Requirements and Interoperability
D5: All technology requirements (including hardware, browser, software, etc.) are specified.
All technology requirements (including hardware, browser, software, etc.) are identified in the course description or during the student registration process and specified to students before they begin the course.

Element Further Explanation
D6: Prerequisite skills, course tools, and course software are identified and appropriate in relation to the students and course.
All prerequisite technology skills, software, and online tools necessary for the specific class are identified in the course description or during the registration process and are shared with students before they begin the course. Tools should be appropriate, necessary for teaching and/or enriching the lesson, cross-platform and free to the student (or built into the course).
Interoperability technical standards allow sharing content among different learning management systems and ensure sharing of questions, assessments and results with others.
D7 Rating (1 = not applied, 2 = partially applied, 3 = applied): D8: Copyright and licensing status, including permission to share where applicable, is clearly stated and easily found.
Course developers or publishers clearly state the copyright and licensing status of all content, including permission to share where applicable. Copyright and licensing information should be readily available, understandable and standardized in terms of use.

Subsection: Accessibility
D9: Course materials and activities are designed to provide appropriate access to all students. The course, developed with universal design principles in mind, conforms to the U.S. Section 504 and Section 508 provisions for electronic and information technology as well as the W3C's Web Content Accessibility Guidelines (WCAG 2.0).
Through the use of web accessibility evaluation tools, all web pages required for students to engage in online education (e.g., registration, library, course materials, grade retrieval) are validated to conform to accessibility standards. NIMAS is used to ensure textbooks and other instructional materials are accessible to the visually impaired.

Subsection: Accessing Course Effectiveness
E1: The course provider uses multiple ways of assessing course effectiveness.
A combination of student, instructor, content experts, instructional designer and outside reviewers may be used to evaluate the course for effectiveness. A variety of methods may be used including course evaluations, student completion rates, satisfaction surveys, peer review, teacher and student feedback, and student performance on in-course as well as state or national assessments. University researchers have been encouraged to conduct studies on the effectiveness of the course. E1 Rating (1 = not applied, 2 = partially applied, 3 = applied):

Subsection: Course Updates
E2: The course is evaluated using a continuous improvement cycle for effectiveness. The findings are used to improve and update the course content as needed.
The provider indicates the frequency of course evaluations, whether reviews are conducted internally or externally, and how the provider uses evaluation results to improve courses. Courses should be reviewed to keep the content current, engaging, and relevant.

Subsection: Instructor and Student Support
E3: Technical support and course management assistance are provided to students, the course instructor, and the school coordinator.
Online technical help and support should be available any time. If 24/7 support is not available, support hours are clearly posted within the course or on the online program's website and a maximum response time is noted. Assistance may take the form of Frequently Asked Questions, training resources, mentors, or peer support.