Online Course Design: A Review of the Canvas Course Evaluation Checklist

The evaluation of online courses is an important step in providing quality online courses. There are a variety of national and statewide evaluation tools used to help guide instructors and course designers of online courses (e.g., Quality Matters, OSCQR). This paper discusses a newly released course evaluation instrument from Canvas, the second largest learning management system (LMS) used by higher education institutions in the United States. The characteristics and unique features of the Canvas Course Evaluation Checklist (CCEC) are discussed. The CCEC is also compared to established national and statewide evaluation instruments. This review is helpful for those interested in online course design and developments in the field of online education.


Introduction
Online education has become a mainstream component of higher education. Annually, nearly one-third of students enroll in online courses (Seaman, Allen, & Seaman, 2018), with online course offerings representing the fastest growing sector in higher education (Lederman, 2018)

. A recent Inside Higher Ed
Survey of Faculty Attitudes on Technology (2018;N=2,129) found that 44% of faculty have taught an online course and 38% have taught a blended or hybrid course (Jaschik & Lederman, 2018).
To teach a distance education or blended course, faculty members generally rely on some form of learning management system (LMS; Ismail, Mahmood, & Abdelmaboud, 2018). The LMS has a large impact on the way online education is presented and perceived. The LMS influences "pedagogy by presenting default formats designed to guide the instructor toward creating a course in a certain way" (Lane, 2009, para. 2).
In the United States, the current LMS market is dominated by Blackboard, Canvas, Moodle, and Desire2Learn (also known as Brightspace, D2L), which account for 90.3% of institutions, and 92.7% of student enrollment (Edutechnica, 2019). Blackboard, released in 1997, has a 30.9% share of institutions and 33% of student enrollment. Canvas, released in 2011, supplies 30.6% of institutions and 35.47% of student enrollments. In comparison, Moodle has a 17.7% share of institutions and 12.41% share of student enrollment-an indication that smaller schools typically utilize Moodle (Edutechnica, 2019).
Canvas is the fastest growing learning management system in the United States (see Figure 1; Edutechnica, 2019). Nearly 80% of new LMS contracts in U.S. and Canadian higher education result in a move to Canvas (Hill, 2016). According to U.S. News and World Report's (2018)  Educators have a vested interest in offering quality courses. Nine out of 10 faculty members (N=2,129) surveyed by Inside Higher Ed and Gallup (Jaschik & Lederman, 2018) said they were involved in online or hybrid course design. And, only 25% of these faculty members reported using an instructional designer to help design or revise online courses (Jaschik & Lederman, 2018). Evaluation instruments for online course design can be an important tool to provide support and guidance. Providing easy to use evaluation tools and determining the elements that should be assessed can help guide instructors who design online and blended learning, and highlight best practices.
A review of six publicly available national and statewide online course evaluation instruments for the design of higher education online courses in the United States was recently published (see Baldwin, Ching, & Hsu, 2018). This review identified 12 universal criteria found in all six national and statewide evaluation instruments, nine criteria found in five out of six evaluation instruments, and one criterion found in four out of six evaluation instruments. Since then, the Canvas Course Evaluation Checklist (CCEC), a new national course evaluation instrument has been released (Instructure, 2018b). This evaluation tool is similar to the Blackboard Exemplary Course Program Rubric in that it has been published by a learning management system company. Due to the potentially large impact of this new evaluation tool, this paper aims to explore: • What are the characteristics of the CCEC?
• How does the CCEC compare to other national and statewide evaluation instruments?
• What are the unique features of the CCEC?

Method Data Analysis
Both authors independently compared the CCEC to the previously reviewed national and statewide evaluation instruments (Baldwin et al., 2018): • Blackboard Exemplary Course Program Rubric (Blackboard; Blackboard, 2017).
• Open SUNY Course Quality Review Rubric (OSCQR; State University of New York, 2018).
Initially we assessed the CCEC's characteristics to those on the national and statewide evaluation instruments above (e.g., intended usage, audience, ease of adoption, rating scale, cost/availability, training requirements for users; Table 1). Yes, by team of 3 certified peer reviewers, 1 master reviewer. One reviewer must be external to institution, and one reviewer must be content expert. No.

Training
For official review, a peer group of Blackboard clients review. We also compared the CCEC's physical characteristics to the previously reviewed instruments and identified the breakdown of each instrument, including the number of sections, the section names, and sub-sections (Table 2). Then, we coded the CCEC against the 22 common criteria found in previously reviewed evaluation instruments by comparing phrases used in the instruments. Next, we compared our analysis, to reach a consensus of the characteristics and unique features of the CCEC. We used our experience in online instruction, instructional design, and online course evaluation instrument research to guide us.

Characteristics of the Canvas Course Evaluation Checklist
The CCEC focuses on course design within the parameters of the Canvas LMS. The checklist was developed by a team of Instructure employees (Instructure is the developer and publisher of Canvas) and released in 2018. The CCEC is intended for all Canvas users, which conceivably could include instructors and instructional designers. The checklist's stated purpose is to share universal design for learning (UDL) principles, the checklist creators' expertise in Canvas, and their "deep understanding of pedagogical best practices" in an effort to "elevate the quality to Canvas courses" (Instructure, 2018b, para. 2 License (CC-BY-NC-SA 4.0) on the Internet; sharing, as well as remixing, of the tool is encouraged by Canvas, provided attribution is given.
The checklist can be used with new and mature courses, and is primarily useful to Canvas users since many of its features are Canvas-centric. It is comprised of four sections (course information, course content, assessment of student learning, and course accessibility) and 33 criteria (compared to the average national/statewide evaluation instrument of over six sections, and 56 criteria). The CCEC uses the rating scale of expected, best practice, and exemplary to rank the importance of the design components. The CCEC indicates 19 expected and standard design components, seven best practices/added value design components, and seven exemplary/elevated learning design components (Table 3). 275

Comparing the CCEC to Other Evaluation Instruments
In Baldwin et al.'s (2018) article comparing national and statewide evaluation instruments, 12 universal criteria were included in all of the instruments: • Objectives are available.
• Navigation is intuitive.
• Technology is used to promote learner engagement/facilitate learning.

276
• Student-to-student interaction is supported.
• Communication and activities are used to build community.
• Instructor contact information is stated.
• Expectations regarding quality of communication/participation are provided.
• Assessment rubrics for graded assignments are provided.
• Assessments align with objectives.
• Links to institutional services are provided.
• Course has accommodations for disabilities.
• Course policies are stated for behavior expectations. (p. 52-53) The CCEC indicates each of these criteria, with the following exceptions: Technology is used to promote learner engagement/facilitate learning, Links to institutional services are provided, and Course policies are stated for behavior.
Nine criteria were previously identified in five out of six national and statewide evaluation instruments: • Learners are able to give feedback on the course for improvement.
• Course activities promote achievement of objectives.
• Instructor response time is stated.
• Collaborative activities support content and active learning.
• Self-assessment options are provided.
• Assessments occur frequently throughout course.
• Instructions are written clearly.
• Guidelines for multimedia are available.
• Guidelines for technology are available. (Baldwin et al., 2018, p. 54) The CCEC includes three of these criteria (Instructions are clearly written, Guidelines for multimedia are available, and Guidelines for technology are available) but not the other six (Learners are able to leave feedback, Course activities promote achievement of objectives, Instructor response time is stated, Collaborative activities support content and active learning, Self-assessment options are provided, and Assessments occur frequently throughout course). The standard found on four out of six national and 277 statewide evaluation instruments, "Information is chunked" (Baldwin et al., 2018, p. 54), was also found on the CCEC. When comparing the CCEC to other national and statewide evaluation instruments, some criteria are subtly different. For example, the CCEC focuses on having a variety of assessments throughout the course, whereas other instruments focus on the frequency of assessments in the course. The Blackboard Exemplary Course Program Rubric states, "Assessment activities occur frequently throughout the duration of the course" (Blackboard, 2017, p. 7).

Unique Features of the CCEC
The CCEC is visionary in some aspects. The CCEC is an easy to use checklist, complete with checkboxes.
While checklists have been identified as useful screening devices for evaluating online course design (Baldwin & Ching, 2019;Herrington, Herrington, Oliver, Stoney, & Willis, 2001;Hosie, Schibeci, & Backhaus, 2005), the other established course evaluation tools are in the form of rubrics. Also, the CCEC is relatively short. It is four pages (including citations), compared to an average of 10.71 pages for other evaluation instruments. And, it was released through a Canvas Community discussion page, where conversations between the checklist creators and users occur. This is likely to lead to further discussion about the importance of course quality. In addition, the CCEC instructs users on how to design their courses (e.g., "Home Page provides a visual representation of course; a brief course description or introduction…" "Home Page utilizes a course banner with imagery that is relevant to subject/course materials" (Instructure, 2018b, p. 1)). In contrast, other evaluation tools provide more general information such as, "A logical, consistent, and uncluttered layout is established" (SUNY, 2018, p. 2).
The CCEC has a unique focus on UDL, with 25 of the 33 criteria referenced to the UDL guidelines. UDL guidelines provide a set of principles that offer multiple means of representation, action and expression, and engagement to provide all individuals equal opportunities to learn (CAST, 2019). The concept of UDL is to create education that accommodates the widest number of learners, including those with disabilities, without the need for adaptations or special design (Rose & Meyer, 2002). UDL is used to support the variability and diversity of learners (CAST, 2019;Rose, Gravel, & Gordon, 2013) by how information is presented, how learners express what they learn, and how learners engage in learning (Hall, Strangman, & Meyer, 2014).
Four of the CCEC exemplary design components are linked to personalized learning ("Personalized learning is evident", "Differentiation is evident (e.g. utilized different due dates)," "MasteryPaths are included," and "Learning Mastery Gradebook is enabled for visual representation of Outcome mastery" (Instructure, 2018b, p. 2)). Personalized learning is defined as developing learning strategies and regulating learning pace to address individual student's distinct learning needs, goals, interests, or cultural backgrounds (iNACOL, 2016). Personalized learning has become increasingly popular with the advent of more affordable and available software. Technology can be used to develop learner profiles, track progress, and offer individualized feedback. With both personalized learning and UDL, educators are encouraged to understand learner variability, use multiple instructional delivery and assessment methods, and encourage student engagement (Gordon, 2015;McClaskey, 2017).

278
Interaction is emphasized (e.g., as a separate category) in all of the previously reviewed state and national evaluation instruments (Baldwin et al., 2018). In contrast, the CCEC instructs course designers to include at least one of three forms [of interaction]:  Student-Student Interaction (e.g., discussions and/or collaborative projects).
 Student-Content Interaction (e.g., engaging content and resources with which students must interact and not just read or watch. (Instructure, 2018b, p. 2) While these three forms of interaction are individually significant, Moore (1989) instructed educators of the vital importance of including all three forms of interaction in distance education in his perennially cited editorial.

Unlike established instruments like Quality Matters (QM) or the Blackboard Exemplary Course Program
Rubric, the CCEC does not have an official review process or certification outcome. No training is required to use the CCEC, but like the Open SUNY Course Quality Review Rubric (OSCQR), the CCEC supplies the user links to explain some of its criteria (e.g., "Canvas Guide-Add Image to Course Card" [Instructure, 2018b, p. 1]).
The CCEC is based on research that is different than the research cited by other well-known national and statewide evaluation instruments. The CCEC cites the UDL guidelines, a K-12 quality course checklist, and an online course best practices checklist from a California community college. In contrast, OSCQR provides research-based evidence for each of its criteria. Likewise, the QM Rubric is supported by an intensive review of literature involving 21 peer-reviewed journals conducted by an experienced staff (Shattuck, 2013). The CCEC is generally grounded in the UDL framework, while other instruments are grounded in a more comprehensive synthesis of research-based pedagogical practices.

Conclusion
Providing a short, easy-to-use checklist that provides specific guidelines for designing an online course and promotes personalization and accessibility is beneficial to Canvas users. The LMS influences the design and pedagogy of online courses (Vai & Sosulski, 2016). However, while the CCEC is innovative in some aspects, this review shows it falls short on evaluating other critical aspects of online design (e.g., interaction).
Introducing an easy-to-use evaluation checklist that neglects previously identified research-based practices (established by six other national and statewide evaluation instruments) to potentially a third of higher education online course instructors/designers is disconcerting. It is suggested that the CCEC be revised to include the identified practices to better support Canvas LMS users.
It is critical to support instructors and course designers with guidelines based on best practices to encourage quality course design. Currently, the CCEC may serve as a good starting point for online course designers who wish to use an evaluation instrument to improve course quality. However, we highly recommend that 279 instructors and course designers consult other national and statewide online course evaluation instruments that offer guidance based on more comprehensive research-based practices to supplement the CCEC in their quest to design quality online courses.