October - 2002

Quality Standards in e-Learning: A matrix of analysis

Jia Frydenberg
Irvine Distance Learning Center
University of California


Most institutions of postsecondary and higher education are creating or adopting quality statements, standards, and criteria regarding their niche of the “e-Learning enterprise.” In doing so, they have a tendency to reinvent the wheel. This article summarizes current published quality standards in the US, and analyzes and organizes them into a nine-cell matrix. It concludes with discussion of emerging issues with respect to the nine standards-areas.

Keywords: e-Learning; online learning; Web-based learning; quality standards; development delivery and support of online learning

Introduction: Who Are The Standard-Setters?

The defining feature of most forms distance education is that learners and teachers are not within eyesight of each other. They may also be separated in time (asynchronous learning). In electronic learning, or e-Learning, instructors, at times, may be a machine, as in computer-based training or computer-based tutorials (CBT). e-Learning or Web-based learning, a relatively new form of distance education, is only now being integrated into distance learning research literature. It is, however, rapidly becoming the dominant form of distance learning delivery in developed countries.

This article focuses on the e-Learning that institutions of higher education make available to an ever-growing student body. However, this article does not consider e-Learning as created by, and provided for, for-profit non-educational organizations. Nor does it consider e-Learning in the context of the K-12 system. Instead, the purpose here is to present a matrix within which to examine, compare, contrast, and synthesize the standards of e-Learning quality that have been proposed in the literature.

Higher education institutions in the US have a long history of independence from governmental control. The US constitution, unlike the constitutions of many other countries, gives individual states the right to determine how the education of its citizenry should be organized and monitored. All 50 states have chosen to allow tertiary education to be largely self-governing and self-policing, similar to those organizational models used by the mediaeval guilds that reserved the right to professional training, certification, and censure of its members.

While providers of higher education, colleges, and universities, are long lasting and frequently appear to be permanent fixtures in the teaching and learning landscape, the seekers of learning tend to be temporary members of these “guilds.” As a result, the seekers (students) are not organized into lasting associations or other groupings that can effectively influence the education they seek. Naturally, there are exceptions to this; one need only look back in time to the sit-ins and demonstrations of the 1960s to find a time when student pressure rocked the academic establishment. But, in general, such upheavals have been limited in time and level of influence. It remains that the primary influences in tertiary education in the US are: 1) professional faculty associations; 2) regional accrediting agencies that have the charge of guiding and evaluating education providers; and 3) university faculty and administrators. These three groups are the main players in the arena of presenting and debating standards of quality for electronically mediated teaching and learning in US higher education.

Distance Education and the World Wide Web

Distance education in the US comes out of the tradition of “independent learning,” in which learners who did not have ready geographical access to a physical campus, studied on their own using materials (e.g., texts, assignments, exams) mailed to them by universities. The norm in independent learning has been that learners do not follow the standard university terms (semesters, trimesters, or quarters) nor do they have a known cohort group with whom to interact. A considerable amount of print-based independent learning still occurs across the US. Although e-Learning may be offered in this non-term, non-cohort fashion, the quality standards considered in this article apply primarily to group-based, time-limited activities. In fact, most providers of standards still operate within a paradigm that mimics the “group of learners coming together to study with an expert” tradition. However, other standards of quality will be needed for the emerging e-Learning forms of individual tutoring and non instructor-led, on-demand Web-based learning.

Certain attributes of the World Wide Web that have fueled the explosion of learning opportunities include: capacity to enable sharing of rich media files (pictures, complex diagrams, video, audio); and interactivity of electronic communication in user-friendly modalities such as email, bulletin boards, and simultaneous chat rooms, as well as more bandwidth intensive forms of Web-enabled video and audio teleconferencing. A third attribute, the non-linearity of the platform-independent standards of hypertext markup language (HTML) and its successors, has so far been much less employed in designing WWW education. In fact, most of the “best designed” e-Learning still follow the hierarchical, linear, sequential mode of presenting learning material, in that learners are “forced” to proceed through their learning experience in a chronological fashion. Many such examples of e-Learning garner the derogatory epithet “HTML page-turners.”

Why has e-Learning gathered such momentum? With the increasing pace of change, most of us need to continue to acquire university-level understanding and knowledge throughout our working lives. This learning, combined with our involvement with family, community, and work, encompasses an ever-larger slice of our 24-hour pie. Secondly, because we continue to locate our places of residence further away from our places of employment and education, we also find ourselves increasingly caught in gridlock as we attempt to move from one location to the other. The question becomes: When can we access the thinking and new ideas we need to be successful in our profession, or the art we need to excite our soul? The emerging answer seems that we seek such learning as we commute in the car, train, or ferry, or after the evening news, during lunch hour, or on company time.

The brave new pioneers who choose to experiment in e-Learning are often those who are challenged by constraints of time and space. In addition, various authors have drawn conclusions that those who will be most successful in online learning have certain personality traits or characteristics, such as independence, assertiveness, persistence, and a reflective attitude (See Wahlstrom, Williams, and Shea, 2003; Gibson, 1998; and Gilbert, 2001).

However, learners’ criteria for quality in their e-Learning experiences are generally not well understood. There are not yet any Consumer Reports from the learner’s perspective. e-Learning is most frequently offered by institutions at prices representing “full cost recovery” (which, in the US, means programs that are not supported by state or government tax-based expenditures, are instead required to be fully self financing, including all overhead costs). Nevertheless, learners certainly apply market tests for these expenditures – specifically quality and service value. Although institutions pay attention to the evaluations learners offer of their experiences, we are not always sure we are asking the right questions on the often-lengthy end-of-course evaluations. More research remains to be done to fully explore consumer standards for e-Learning and to integrate those standards with traditional academic institutional concerns. In the face of the lack of consumer information, this article presents and assesses the points of view of the academic institutions, rather than that of the “education consumer.”

Standards Domains

What, then, constitutes the quality of e-Learning from the viewpoint of the educator? The following nine domains are repeatedly described in the literature:

1. Institutional Commitment

One frequently mentioned domain is that of institutional or executive commitment to the education and the provision of learning. Yet there is considerable variation in the understanding of what constitutes such a commitment. The Western Interstate Commission for Higher Education (WICHE) presents nine aspects of the domain of “commitment” (WICHE, 2001). These encompass such aspects as financial commitment, the physical plant, articulation and other policies, technical support, legal compliance, etc. The Institute for Higher Education Policy in the report Quality on the Line: Benchmarks for Success in Internet-Based Distance Education emphasizes technological aspects of institutional support in terms of having a technology plan, security, redundancy in the delivery structure, and systems for maintenance of the technological infrastructure (IHE, 2000). Although the American Council on Education (ACE, 1996) adds faculty and staff development and a commitment to research to their list activities relating to “organizational commitment,” it does not specify that the research needs to be directly related to distance education.

2. Technology

The technological infrastructure necessary for the delivery of a quality e-Learning program is often described separately as a second domain. While none of the assessors venture to set absolutes such as: “There must be technological opportunity for synchronous interaction between learners and teachers,” they generally include such aspects as security and privacy of data and communication as well as the need for interactivity. None of the reports and position papers this author has reviewed, indicate availability of materials (text, video, audio) as essential as a highly rated component of a quality education program. This is hardly surprising given that the standards-writers arrive at their proposals from the context of traditional educational institutions in which the role of faculty is paramount to their mission. But failure to stipulate standards for instructional materials is curious in light of the recent announcement by the Massachusetts Institute of Technology (MIT), that will make available on the Web a large portion of its instructional materials (such as syllabi, papers, lecture notes) created and used by its professors (http://ocw.mit.edu/index.html). In terms of the standards described here and listed in the references, the MIT project does not qualify as an educational program, nor – obviously – does MIT think it is. In the type of distance education assumed by the standards documents reviewed in this article, the role of technology in permitting access to materials is to foster interactivity among students and between students and teachers. This interactivity is generally left undefined by the documents, and, in many cases, interaction with a computer in a complex branching program that guides the learner in a “step-and-remediation” process will qualify as interactive. On the other hand, most documents mention faculty in one or more of the standards, though generally not under the heading of technology. For example, the American Association for Higher Education says: “Good practice encourages contacts between students and faculty” (AAHE, 2000). The American Federation of Teachers report Distance Education: Guidelines for Good Practice posits: “Close personal interaction should be maintained in distance education courses among students and between students and teachers” (AFT, 2001). The IHE report of March 2000 under “Teaching and Learning” writes: “Student interaction with faculty and other students is an essential characteristic and is facilitated through a variety of technologies including voice mail and / or email” (IHE, 2000). The common underlying assumption of these standards document is that a human guide is available in the form of a faculty member or instructor who leads the student group toward a learning goal. To my knowledge, we have not yet in the US, begun to grapple with quality standards in non instructor-led education. In most cases, the standards under the heading “technology” have yet to identify criteria related to the functions of material access and interactivity, as well as to deeper technical issues such as system maintenance, up-time, redundancy, network access, and so on.

3. Student Services

A third aspect of quality standards concerns neither teaching nor learning, but is instead centered on student services. This domain can be subdivided into the services needed before students’ entrance to a virtual classroom, support during the learning experience, and the continued connection between learners and the institution after the particular course or program has been completed.

Prospective students need to find out, first and foremost, that the learning opportunity exists and to obtain accurate information about it. For a provider organization, the decision to create an e-Learning program must be based, in part, on an assessment of a learning need in an identifiable and reachable group of potential learners. The first of the seven AFT standards mentions that “distance education students should be given advance information about course requirements, equipment, and techniques for succeeding in a distance learning environment...” (AFT, 2001), while the IHE study adds that: “students are informed of and have easy access to information about the programs, including admission, tuition and fees, books and supplies, and student support services” (IHE, 2000). However, none of the standards documents include benchmarks of quality market research and marketing processes as a domain to be included in an evaluation of the educational program. There is a fine line between marketing efforts and providing accurate and timely information to prospective students, and such documents and processes should be included in the assessment of the pre-program student services.

As the “inquirer” becomes a “prospect” in marketing jargon, more hands-on activities occur. The advising role of student services begins, and both practical and academic counseling and advising are frequently mentioned as necessary to a quality e-Learning program. Systems such as toll-free phone numbers with clearly stated hours of operations, database-driven student management software (contact management), advising chat-rooms, and many other tools are used experimentally across the US. In addition, technical training through required or optional pre-program “how to use our virtual facilities” processes, is offered at many institutions. Thirdly, easily accessible and clearly worded statements of time-to-completion, student evaluation, financial aid, stop-out policies, etc., are often cited as examples of quality student services.

An aspect of a quality educational program sometimes subsumed under instructional design, sometimes under teaching and learning, and sometimes under student services, is access to supporting materials. We generally see this as access to an electronically available library, such as the California Digital Library (http://www.cdlib.org/). However, an evaluator might also want to assess the quality of any electronic course packets and reading lists as well as supplementary video and audio recordings under the rubric of supporting materials.

The American image of a “student” relies heavily on campus-centric traditions. Hence, in the US, there is an understandable encouragement of students to become active participants in out-of-class student activities for such potentially self-serving reasons as enhancing degree completion rates, and because researchers (Astin, 1993) report that acculturation into the larger entity of the learning and research body of practice, adds value to the kind and level of learning students’ are able to attain. Many universities currently attempt to forge virtual student-government organizations, student health services available online, career services, and even “virtual football.” As a learner progresses from novice to certificate or degree recipient, virtual alumni associations are also beginning to be mentioned.

4. Instructional Design and Course Development

Many parties are also offering standards for the design and development of e-Learning programs. A report from The Pennsylvania State University titled An Emerging Set of Guidelines for the Design and Development of Distance Education (IDE, 1998) presents the following five aspects of course design with specific principles for each: 1) learning goals and content presentation; 2) interactions; 3) assessment and measurement; 4) instructional media and tools; and 5) learner services and support. While all the standards-proposals describe the need for the design of the virtual classroom to be based on educational outcome objectives, some go as far as to specify that these must be at higher levels of learning such as abstract thinking and critical reasoning (SREB, 2001). For-profit providers of e-Learning content or courseware and the corporations that create and employ e-Learning in their internal workforce, development strategies often include standards of effective and efficient program development processes (project management) in this category (See, for example, Berge, 2001 and Mantyla, 2000-2001).

Given that this study is based on standards proposed primarily by academic institutions and organizations, we have difficulty separating instructional design from instruction. In academe, it is generally the same person, the faculty member, who does it all: determines what the course objectives are, what the learners are to study, when and how students should study, and when and how to conduct assessment. The same faculty member also provides lectures, seminars, and discussions that form the link between the content, its scope, selection, and sequencing, and students’ learning outcomes. However, in e-Learning it is crucial to separate the two roles since it is not given that the person who develops materials for a course will be the one, or the only one, to teach from these.

Design and development of e-Learning, then, is comprised of all the activities that go into getting ready to enable learning to take place. Unlike traditional education, however, e-Learning is heavily front-loaded. Electronic course design takes considerable time, effort, and money, and the results of instructional design decisions are not easily changed. Benchmarks of quality for instructional design as divorced from the process of utilizing that design to facilitate learning (commonly known as “teaching”) tend to state that the design must flow from the objectives – a lofty goal – and that it must enable interactivity. The type and function of this interactivity is not defined and is traditionally left up to the faculty to determine. In addition, we are only beginning to explore how instructional design needs to vary by discipline or by the type of educational outcome objective. Informal discussions occur in many venues of how Web layout and functionality for an electronic course may vary when the objective is to list the steps in performing a multiple regression analysis (considered a lower-level skill); or to evaluate whether the most appropriate statistical techniques have been employed in a research article in a particular field of study (considered a higher-order cognitive practice) (Bloom, 1956 and 1964).

The design imperative of enabling interactivity is based on the assumption that learning occurs through discourse, or at least that the kind of learning opportunities that educational institutions offer occurs through such activity. Interestingly, the development of instructorless CBT does not make this assumption, nor do software-based tutorial programs. However, these tend to have objectives at the Bloom’s taxonomy levels of knowledge, comprehension, and application, rather than the university levels of analysis, synthesis, and evaluation. Can instructorless courses (Schank, 2002) be designed that demonstrably develop learners’ creativity, innovativeness, and abstract thinking? From such studies we will be better able to assess the quality of instructional design separately from instruction.

5. Instruction and Instructors

The fifth domain of quality is that of instruction and instructor services. What is quality instruction at a distance? In addition to the benchmarks of quality instruction in a face-to-face setting (such as depth of knowledge of the instructor, presentation and organizational skills, encouraging attitudes toward student dialog, feedback and guidance, etc.), one issue crops up in most of the standards documents: that the distance learner is mostly solitary. The first principle of good practice from the American Association for Higher Education, for example, states: “Good practice encourages contacts between students and faculty” (AAHE, 2000). In the traditional setting, this occurs in a lecture or seminar where during an hour or two the contacts are between one teacher and several learners, with the possible addition of some one-on-one outside-of-class exchange when brave learners venture to seek out faculty members during office hours. However, unless the instructional design of the electronically mediated course overtly emphasizes the students as a group of learners, individual students are most likely to perceive themselves as interacting within a mutually exclusive student / tutorial relationship with the instructor. It is obvious that this orientation contrasts with the tradition of lecturing to 100, 200, or 1,000 students at one time. Experiments of an instructional team with defined responsibilities comprising the professor, the teaching assistants, and invited guest moderators (e.g., the course Learning Theories and Classroom Practices at the University of California Irvine at http://www.gse.uci.edu/ed173, or at the Open University in the UK http://www.open.ac.uk/frames.html), which employs tutors supporting small groups of learners), are of great interest to universities in coping with the dilemma of class size in e-Learning.

Instructor services, as is the case with student services, can be divided into pre, during, and post course services. Prior to teaching, an instructor must feel comfortable utilizing the media of the course and hence may need training and guidance. During an e-Learning class, personnel must be available at the institution to assist instructors to resolve issues of a technological nature or to mediate difficulties in student and faculty perceptions of what constitutes good instructional practice, such as, for example, reasonable response time. Just as with on-ground instruction, the faculty member must have access to post-course assistance related to evaluation (course, instructor, and student) etc. In addition, as the WICHE guidelines spell out: “Issues of workload, compensation, and intellectual property ownership (should) have been discussed with faculty and agreements worked out” (WICHE, 2001).

6. Delivery

Some standards document a sixth category, program delivery, as a separate domain. This may or may not include student and instructor services. The WICHE guidelines place delivery under “institutional context,” noting such aspects as coordination, oversight, and articulation as parts of this domain. The Southern Regional Education Board (SREB, 2000) provides 15 principles of program management, which include such aspects as monitoring students to ensure academic honesty, content revision and oversight, technical requirements for acceptable access, recourse for appeal if a Web-based course is not delivered as described, and so on. Good program delivery depends on two aspects: 1) defined policies, procedures, responsibilities; and 2) communication, and fair and impartial management. In the best of all possible worlds, program delivery and program administration should be transparent to learners, just as the existence of the power plant on campus generally is.

7. Finances

The headlong rush to develop courses and programs for e-Learning, sometimes fails to heed standard business and accounting practices. The bill is coming due. Many assumed, for example, that the intellectual property fixed in HTML was so valuable that it would have years of shelf-life and could be licensed and / or sold over and over with only minor maintenance. Many also had delusions that the cost of one instructor at regular salary levels could be spread over hundreds of full tuition students and enable a rapid economy of scale. Neither of these assumptions has proven true. Hence, the financial health and responsible business planning of e-Learning programs is a necessary and yet-to-be addressed domain of evaluation standards.

As with all educational programs, there are fixed and variable costs of e-Learning. Universities, which have undertaken the production of full in-house programs from scratch, have often discovered to their dismay that expenditures per course are not inversely proportional to volume of courses to be produced. Instead of a factory model of production, many institutions treat online course development as a cottage industry where each new course is handcrafted. An interesting benefit of this practice is the discussion and nascent development of “learning objects,” the theory being that such molecules of learning can be developed once and then shared and / or licensed to others. Large-scale standards experiments are currently in progress following this line of thought (See http://www.wiadlcolab.org; http://www.adlnet.org, and http://www.imsproject.org). The licensing of content, whether in the form of objects or of complete or partial courses, is generally based on the business model where the content owner receives a per-student royalty in return for the university being able to use that content in a university-approved and offered course. If the intellectual property owner figures it has cost $40,000 to produce the IP and accepts a per-student royalty of $100, the profits begin to arrive with the 401st student. This is a financial risk, of course, and what is more, it may be in opposition to a university’s perceived mission (i.e., most faculty would balk at the idea of using another professor’s lecture notes, yet we have no trouble using someone else’s textbook). However the hybrid model of combining licensed learning objects with university-developed specialty content has promise. This is so, because it enables the university to focus on its primary strength, innovative thought, while licensing or buying the more mundane necessary objects that lead a learner onward, such as self-tests, “bridge” units that connect one topic with the preceding topics, and standard graphs, charts, and diagrams. Many textbook publishers are looking at such a model with interest and are at work disaggregating their traditional bound textbooks into defined, tagged, retrievable, and sharable learning objects.

Some e-Learning programs work on a full-cost recovery model where student tuition is intended to cover all the programs’ costs, while others must work within the funding models (usually average daily attendance) dictated by their respective state. (In the US, it is common for state governments to funnel resources to local schools according to a set formula of funding per capita of the attending student body. Hence, the school only receives funding based on the number of students who show up on any given day.) Either way, the financial management of the distance learning enterprise is an important component of the total assessment of the program’s quality and ability to sustain itself. Most standards documents do not set more specific criteria for financial health than: “There must be financial and administrative commitment” (ACE, 1996) and “The institution’s budgets and policy statements reflect its commitment to the students for whom its electronically offered programs are designed” (WICHE, 2000).

8. Regulatory and Legal Compliance

Regulations governing e-Learning are growing. The two primary sources of regulation in the US are the regional accreditation agencies that have a long history in the US, and the 1990 Federal Americans with Disabilities Act (ADA) requirements (See, for example, detailed specifications of ADA-compliant web design at http://www.usdoj.gov/crt/ada/stdspdf.htm). An accrediting agency, such as WASC (Western Association of Schools and Colleges), or NCASC (North Central Association of Schools and Colleges), has as its charge to set standards of quality practice and to guide institutions toward achieving those standards. In addition to the concerns described above, regional accreditation agencies and schools must follow applicable state and federal laws. The WICHE report states as its first benchmark:

The institution seeks to understand the legal and regulatory requirements of the jurisdictions in which it operates, e.g. requirements for service to those with disabilities, copyright law, state and national requirements for institutions offering educational programs, international restrictions such as export of sensitive information or technologies, etc. (WICHE, 2000, p. 4).

There is considerable discussion within the e-Learning community about how best to ensure ADA compliance. Many believe that e-Learning holds great promise, especially for learners with physical and mental challenges. Suggestions, such as ensuring that every “talking head” or interview segment also has an accompanying text for those who cannot hear the audio or that audio files describe what happens in a visual simulation for those who cannot see it, are being taken seriously. Legal compliance has also focused on intellectual property law, which is in a state of flux. Intellectual property issues are hotly debated both in our US governing bodies and on campuses. Being so new, the e-Learning field is still trying to find its way through the constantly changing legal landscape.

9. Evaluation

Program evaluation is frequently listed as a separate domain in standards documentation. While assessment of student achievement is normally described as part of instructional design and tied to specific course objectives, program evaluation is a meta-activity that incorporates all the aspects of the e-Learning experience. Among its benchmarks, the IHE identified the following three criteria:

1. “The program’s educational effectiveness and teaching / learning process is assessed through an evaluation process that uses several methods and applies specific standards.
2. Data on enrollment, costs, and successful / innovative uses of technology are used to evaluate program effectiveness.
3. Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness.” (IHE, 2000)

The WICHE accreditation document includes these components:

“5d. Overall program effectiveness is determined by such measures as:

1. The extent to which student learning matches intended outcomes, including for degree programs both the goals of general education and the objectives of the major.
2. The extent to which student intent is met.
3. Student retention rates, including variations over time.
4. Student satisfaction, as measured by regular surveys.
5. Faculty satisfaction, as measured by regular surveys and by formal and informal peer review processes.
6. The extent to which access is provided to students not previously served.
7. Measures of the extent to which library and learning resources are used appropriately by the program’s students.
8. Measures of student competence in fundamental skills such as communication, comprehension, and analysis.
9. Cost effectiveness of the program to its students, as compared to campus-based alternatives.”

Program evaluation includes the assessment that students make of their learning experience (point 4 above). At the University of California Irvine, we are experimenting with a Web-based evaluation form comprised of the following seven areas of questions: 1) course content; 2) course design; 3) course materials; 4) instructor and instruction; 5) online student services; 6) overall comments; and 7) a few items asking about demographic information1 .

As always with program evaluation measures and processes, it is critical to build these into the program design from the beginning. That, of course, is in no way unique to e-Learning.

Conclusion and Current Issues

The final section of this article will explore some of the current issues and research needs within the nine sets of standards identified above. To recap, the domains are:

1. Executive commitment
2. Technological infrastructure
3. Student services
4. Design and development
5. Instruction and instructor services
6. Program delivery
7. Financial health
8. Legal and regulatory requirements
9. Program evaluation

What are the issues associated with development of e-Learning programs, as they become an ever-larger segment of the higher education enterprise? Each arena has its own concerns and research agendas.

Executive Commitment

Consumers of higher education rightfully expect universities to be around for a long time. They expect that if an institution offers a program on-ground or online, it will demonstrate the organizational commitment to see the admitted students through. For the e-Learning center to gain such institutional commitment is a human process of relationship management, willingness to meet skepticism and probing questions, and taking the risk of being wrong. Stakeholders and gatekeepers must all be invited to the table along with the converted advocates. It will be advantageous to the higher learning providers to encourage research on the processes of developing e-Learning capabilities in both separate centers of service and fully integrated models. Such studies may reasonably come out of our graduate schools of management.

Technological Infrastructure

The central issue in courseware development at the moment is the potential for developing reusable learning objects, tagging them in a systematic way, storing them in well-designed databases, and retrieving and recombining them with other objects to create customized learning experiences for specific needs. In the learning management arena, technical solutions are required to track student-learning performance. With respect to student services, we are grappling with questions of how to engage learners who have never set foot on campus, how to make them feel part of the university community, and how to advise and guide them through technical interfaces. For content experts and instructors, we are looking for more user-friendly tools to enable them to create excellent learning materials and intellectual meeting spaces easily and quickly. We also see an explosion in the number of people wirelessly attached to their personal digital assistants and who want their learning programs compressed to a screen the size of a palm. And, of course, we need all these technical innovations to interface seamlessly with all legacy systems. All of these issues point to interdisciplinary studies that merge pedagogy with technical tools.

Student Services

Student service departments worry about financial aid regulations and about advising. Since the number of students served continues to be a benchmark of both academic and financial success, the issue of compromise between marketing, which has a tendency toward hyperbole, and accurate information and high program admissions standards is a constant. Another desire of student services is to be able to offer prompt and high quality tutorial assistance to those students who need it.

What are the most effective ways of advising students who never set foot on campus? How do we build a virtual community that they can feel a part of? In general, we are still at a stage where each institution may present “this is how we did it” at local and national conferences, but we do not yet have published qualitative or quantitative studies leading to replicable results. There is a considerable research gap in the arena of student services.

Design and Development

Instructional design for the World Wide Web is based on the temporal, chronological models of speech, where one has to speak one word at a time. Few examples of e-Learning courses are truly non-linear. Yet those programs, which are intended for students under age 30, are enrolling learners who are fully at ease in an avatar-and-bot world. While we are used to controlling learning by requiring that module 1 be completed before the learner has access to module 2, many younger learners have no such predispositions. How do we design non-sequential instruction?

In development, we need faster “time to market.” Rather than sequentially developing courses, we need to outsource components and develop them simultaneously. This requires much higher skills in project specifications and project management than we are used to in universities. We also need multi-skilled staff who can take on intellectual property law one minute and come up with solutions to “make or buy” valid student assessment instruments the next.

The research questions that emerge include those of linearity and non-linearity in enabling learning. What is the optimum level of control and for whom? We hope that graduate schools of education will take up such questions as they study the learning practices that are developing among the younger generations. Secondly, the skills of the new breed of educational manager deserve elucidation as the creation of educational opportunities at a distance increasingly involves multi-disciplinary teams rather than traditional individual faculty members.

Instruction and Instructor Services

The issue of greatest concern, and perhaps the one about which less is known, centers on the question: How we can specify what quality online instruction is beyond what we already know as quality instruction in a face-to-face mode? In general, all the components that the various standards documents describe are those that are commonly accepted as quality instruction. An example can be seen in the seven principles of the American Federation of Teachers. In this quote, I have made a strike-through of the word “distance” to show this equivalence of the first six principles to valid principles for any kind of educational endeavor (the seventh principle is unique to this AFT document):

1. Distance students should be given advance information about course requirements, equipment, and techniques for succeeding in a distance learning environment, as well as technical training and support throughout the course.
2. Close person interaction should be maintained in distance education courses among students and between students and teachers.
3. Equivalent library materials and research opportunities should be made available to distance education students.
4. Assessment of student knowledge, skills, and knowledge should be as rigorous as assessments in classroom-based courses.
5. Academic counseling and advising should be available to distance learning students at the same level it is for students in more traditional campus environments.
6. Academic faculty should shape, approve, and evaluate distance education courses.
7. Full undergraduate degree programs should include classroom-based coursework, with exceptions for students truly unable to participate in classroom education.

How does quality distance instruction differ from quality face-to-face instruction? Anecdotal evidence abounds, but we could use a meta-study approach that can analyze what has been published and draw conclusions from these, albeit scant, documents.

Program Delivery

Program delivery is a process of making the complexity transparent to the learners. One issue often being informally debated is how to manage learners’ expectations. If institutions set, intentionally or unintentionally, the expectation of learners that they will have their questions answered within 20 minutes of emailing their instructors, dissatisfaction is bound to result. Clear communication around expectations and relevant policies is central.

While we do need to protect our students’ right to privacy in an online classroom, it would behoove us to allow researcher “lurkers” to conduct studies in a supervised manner. We need to discover how the student-instructor interactions occur, how such interactions are perceived by both parties, and how to support those that best enable learning. We welcome this research direction.

Financial Health

Financial management of e-Learning is now often one of making a “.com” within a “.edu.” Transitioning from an allowance-model of financial management wherein a department is given a certain dollar amount to spend, to a revenue-generation model, is a major jolt to many university centers. In addition, current practices of development and then delivery of online education have incurred enormous start-up costs and are showing only sporadic return on investment. Attempts in project-based budgeting and accounting, licensing learning objects, sharing risk and reward, and putting chokeholds on development costs, are all part of a steep learning curve for universities.

It is clear that studies of budgetary and financing models of e-Learning are following a moving target. It has been the author’s experience that, while the “going price” of per capita licensing of full-course electronic content was about US$200 only a year ago, it is rapidly descending to the US$50 range. In addition, the pricing of course components, such as learning objects, is quite new to content-providers. Comparative studies are needed of pricing models between the educational “industry” and other component-buying industries, such as clothing manufacturing or high technology equipment assembly.

Legal and Regulatory Requirements

The rapidly changing environment surrounding copyright and intellectual property law means we must be more vigilant than ever. We must balance our hungry need to put our innovative program “out there” against the equal need for the university to be reasonably protected against risk of unintentional errors. An increasing number of institutions of higher learning are adopting well-crafted generic contracts for the licensing of courseware and content and putting into place revised copyright policies. Studies and interpretations of intellectual property rights are extremely valuable to the e-Learning community.

Program Evaluation

The literature on e-Learning program evaluation is naturally skimpy, since few fully developed programs have arrived at a stage where summative evaluation is possible. However, we should soon be seeing articles describing solid attempts and preliminary findings based on in-progress or formative evaluation processes that go beyond the anecdotal. The research body of knowledge of formal program evaluation should be brought to bear on these endeavors.

In conclusion, I believe the distinction between face-to-face and online will soon merge in both quality standard setting and practice. As pedagogy and learning needs drive educational design, every possible mutation of physical and virtual meetings of minds will be created and be grounded in pedagogical purpose. It is a grand time to be an educator.


American Association for Higher Education. (2000). Implementing the Seven Principles: Technology as a Lever. Washington, DC: AAHE

American Federation of Teachers. (2000). Distance Education: Guidelines for Good Practice. Washington, DC: AFT (AFL-CIO)

American Federation of Teachers. (2001). Is Online Education off Course? New AFT Report Proposes Standards for Online Colleges. Press Release. Washington, DC: AFT (AFL-CIO).

Astin, A. W. (1993). What matters in college? The four critical years revisited. San Francisco: Jossey-Bass.

Bates, A. (2000). Managing Technological Change. San Francisco: Jossey-Bass Higher and Adult education Series.

Berge, Z. (2001). Sustaining Distance Training: Integrating Learning Technologies into the Fabric of the Enterprise. San Francisco: Jossey-Bass.

Bindé, J. (1998). Communication and Intelligence: Distance education and culture.” Futures 30(8), 843 – 851.

Bloom, B. (1964). Taxonomy of Educational Objectives: The classification of educational goals. New York: Longman.

Eaton, J. (2000). Core Academic Values, Quality, and Regional Accreditation: The challenge of distance learning. Washington, DC: Council for Higher Education Accreditation.

Gibson, C. C. (1998). Distance Learners in Higher Education. Madison, WI.: Atwood

Gilbert, S. D. (2001). How to be a Successful Online Student. New York: McGraw-Hill

Innovations in Distance Education. (1998). An Emerging Set of Guiding Principles and Practices for the Design and Development of Distance Education. University Park, PA: Pennsylvania State University.

Institute for Higher Education Policy. (2000). Quality on the Line: Benchmarks for success in Internet-based distance education. Washington, DC: IHEP.

Institute for Higher Education Policy. (1999). What’s the Difference? A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education. Washington, DC: IHEP.

Keegan, Desmond. (2000). Distance Training: Taking stock at a time of change. London: Routledge: Falmer.

National Education Association. (2001). Focus on Distance Education. Update 7(2).

Schank, R. (2002). Designing World Class e-Learning. New York: McGraw-Hill.

Smith, S. (1999). An Overview of Internet-Based Education: Challenges facing online education providers. Berkeley, CA.: E-Curriculum Corporation.

Southern Regional Education Board (2000-2001). Essential Principles of Quality: Guidelines for Web-based courses for middle grade and high school students. Atlanta, GA: SREB Educational Technology Cooperative.

Toffler, A. (1970). Future Shock. New York: Random House

University of Illinois. (1999). Teaching at an Internet Distance: The pedagogy of online teaching and learning. Chicago, IL.:Faculty Seminar.

Wahlstrom, C., Williams, B., and Shea, P. (in press). The Successful Distance Learning Student. Belmont, CA: Wadsworth.

Western Interstate Commission for Higher Education. (2001). Best Practices for Electronically Offered Degree and Certificate Programs. Boulder, CO: WICHE.


1. Please contact the author for access to a sample student course evaluation tool.