Corporate managers are constantly looking for more cost-effective ways to deliver training to their employees. E-learning is less expensive than traditional classroom instruction. In addition, many expenses – booking training facilities, travel costs for employees or trainers, plus employee time away from the job – are greatly reduced. However, some firms that have spent large amounts of money on new e-learning efforts have not received the desired economic advantages.
Keywords: e-Learning; corporate; economic benefits
Hall and LeCavalier (2000b) summarized some firms’ economic savings as a result of converting their traditional training delivery methods to e-learning. IBM saved US $200 million in 1999, providing five times the learning at one-third the cost of their previous methods. Using a blend of Web-based (80 percent) and classroom (20 percent) instruction, Ernst and Young reduced training costs by 35 percent while improving consistency and scalability. Rockwell Collins reduced training expenditures by 40 percent with only a 25 percent conversion rate to Web-based training. Many other success stories exist. However, it is also true that some firms that have spent large amounts of money on new e-learning efforts have not received the desired economic advantages.
In addition to generally positive economic benefits, other advantages such as convenience, standardized delivery, self-paced learning, and variety of available content, have made e-learning a high priority for many corporations. Much of the discussion about implementing e-learning has focused on the technology, but as Driscoll (2001b) and others have reminded us, e-learning is not just about the technology, but also many human factors.
There is no doubt that corporations are increasing their emphasis on e-learning. Forrester, an independent research firm that helps companies assess the effect of technology change on their operations, interviewed training managers at 40 Global 2500 companies and found that all but one of them already had online initiatives in place (Dalton 2000). A survey of 500 training directors (Online Learning News, 2001a) clearly shows the new priorities:
ASTD (2002), in its State of the Industry Report, noted that the year 2000 marked a new era of growth for e-learning. The events of September 11, 2001, have only accelerated this growth as organizations cut back on business travel, improve their security, and increase their e-learning efforts.
There is always a focus on the fiscal bottom line in corporate training; the comparatively low costs of e-learning are attractive. Even so, more corporations are looking at such options as blended learning, using more than one method of delivery (e.g., e-learning plus traditional classroom delivery of content, to increase training effectiveness), even if it raises costs. However, Clark (in Online Learning News 2001b) points out that many training managers are not sure how to find the optimal blend for their corporate training programs. He feels they are making decisions based on programs they are familiar with rather than on concrete information about which programs actually produce effective results.
Barron (2001) observes that learning technology providers have been increasingly able to “demonstrate cost-savings and broader benefits, develop integrated offerings, and propose innovative ways of applying e-learning.” However, how do training managers decide which educational products and which learning technology providers actually produce effective results? How do they balance product quality with training costs? As the new corporate adage goes: “Wise training managers realize the bitterness of poor quality remains long after the sweetness of low price has been forgotten.” To justify making decisions about training programs independently of training cost considerations, managers need concrete measures of program effectiveness. While there is no doubt that we see an increasing number of case studies showing success with e-learning, it is still difficult to find solid research measures of learner achievement in the specialized setting of a corporate training program.
When we measure the results of e-learning, do we have to evaluate e-learning differently from traditional training methods? ASTD (2000a) points out that current training evaluation techniques and processes can be expanded to include e-learning as a method of delivery. Indeed, they conclude that the techniques to evaluate e-learning are the same as evaluating other training solutions.
How do we measure the results of e-learning, whatever the delivery method? Using Kirkpatrick’s classic model, any training – traditional or e-learning – can be evaluated at four progressive levels (Kirkpatrick 1979). Level I: Reaction is a measure of learners’ reactions to the course. Level II: Learning is a measure of what they learned. Level III: Transfer is a measure of changes in their behavior when they return to the job after the training program. Level IV: Results is a measure of the business outcomes that occur because they are doing their jobs differently. Phillips (1996) recommends the addition of a fifth level to Kirkpatrick’s model where appropriate. The new Level V is a measure of the Return on Investment (ROI), the cost-benefit ratio of training. In this level, the Level IV data are converted to monetary values and then compared with the cost of the training program.
In spite of all the enthusiasm in corporate training programs for e-learning, an An American Society of Training and Development (ASTD) study found that 67 percent of the training directors interviewed do not measure the effectiveness of their net-based programs at all (2000b). This study found that while 95 percent of surveyed organizations gauged trainees' reactions to courses (e.g., how well they liked the courses) [Level I measure], only three percent of respondents make a real effort to measure the business results of training programs [Level IV measure].
While it is still early to draw solid conclusions about measuring the effectiveness of actual learning that takes place as a result of e-learning – especially within corporate training programs – we can analyze the somewhat controversial results that have come out of mainly academic distance learning programs, using Kirkpatrick’s Four Levels of Evaluation.
Evaluation at this level measures how the participants in a training program feel about their experience. Are they satisfied with what they learned? Do they regard the material as relevant to their work? Do they believe the material will be useful to them on the job? This level, therefore, does not measure learning; it simply measures how well the learners liked the training session.
It is not hard to find learner enthusiasm for e-learning. The majority of 1,002 students who responded to an e-college.com survey said they chose the online format because of the flexibility and convenience of the program. Comments included: “I love that I have the flexibility to continue to hold a full time job.” “To study any time that best suits my busy schedule.” “I travel extensively.” “I was able to work with my instructor, receive tremendous technical support at all hours of the night and gain the same quality content and evaluation as my peers taking the same class on campus.” The survey reports that 75 percent of those students online were employed and 68 percent of the learners worked more than 30 hours per week (ecollege.com 1999). This fact makes this study particularly relevant for corporate trainers who seek to fit e-learning into an already demanding work schedule.
Corporations are beginning to gather more data on how their trainees feel about the use of e-learning technologies. For example, the following results were obtained from an ASTD-Masie Center study involving the experiences of more than 700 e-learners (ASTD 2001):
This question is really an alternate application at Level I evaluation, examining the trainer rather than the trainee. For example, in a recent survey conducted by ecollege.com (1999), 85 percent of the faculty said their students learned equally effectively online as on campus. Some said their students did even better online than in traditional classroom settings. In another TeleEducation study of 130 faculty respondents, 62 percent said their students learned equally effectively in the online environment; 23 percent of faculty stated that their students learned better online; while 90 percent indicated that they were satisfied with online teaching. One faculty comment was: “Online students participate more, perform slightly better than, and are at least as satisfied as their on campus counterparts. From that I conclude that online education appears to be very effective!” (TeleEducation, 2000).
These are qualitative results – both from the learners and instructors – but what about quantitative results?
According to Kirkpatrick, learning is defined as the principles, facts, and techniques that are understood and absorbed by trainees. When trainers measure learning, they try to find out how much the skills, knowledge, or attitudes of their trainees have changed. Measuring learning requires a more rigorous process than a reaction survey. Ideally, both a pretest and posttest are given to trainees to determine how much they learned as a direct result of the training program. While many organizations do not measure at this level, other corporate training centers, such as Sun Corporation’s Network Academy, keep careful track of what employees have learned through the use of both pretests and posttests (Bylinsky, 2000).
A comprehensive research bibliography on e-learning has received much attention. Compiled by Russell (1999), The No Significant Difference Phenomenon provides one of the most frequently quoted rationales for the power of e-learning. This body of research demonstrates that no significant difference can be found no matter what medium is used for learning. In many of these studies, the model is asynchronous learning delivered to the learner on demand. The findings demonstrate that even with no instructor or face-to-face interaction, there are no significant differences in the amount of content learned. A related website, supported by TeleEducation NB, New Brunswick, Canada, includes extracts from more than 355 research reports, summaries, and papers supporting the No Significant Difference phenomenon. This is one time that a finding of no significant differences is actually a compelling factor in favor of e-learning. If corporations can get all of the advantages of e-learning with the same level of results as an instructor-led classroom situation, then the economic advantage for e-learning becomes even stronger.
Wegner, Holloway, and Garton (1999) provide an example of a study showing no significant differences between the test scores of experimental (e-learning) and traditional (classroom-based) students at Southwest Missouri State University. Although there were no statistically significant differences in test scores, this two-semester study yielded qualitative data that indicated that students in the e-learning group had, overall, more positive feelings about their experience than did the control group. This observation is consistent with those found in a number of the “no significant difference” studies.
However, it is becoming more common not to find the same level of results. While some studies show greater benefits in favor of face-to-face delivery, research results consistently demonstrate superior benefits of e-learning in general. In addition to higher performance results, there are other immediate benefits to students such as increased time on task, higher levels of motivation, and reduced test anxiety for many learners. Nettles, et al., (2000) report that, while the majority of the 49 studies they examined reported no significant difference between e-learning and traditional classroom education, nearly 30 percent of the studies report that e-learning programs had positive outcomes based on student preference, improved grades, higher cost effectiveness, and a higher percentage of homework completion.
An alternate website to the No Significant Differences one, also supported by TeleEducation NB, features comparative studies that do show significant differences, most of which report positive results in favor of e-learning. For example, Maki, et al., (2000) evaluated a Web-based psychology course and reported that content knowledge, use of the WWW, and use of computers for academic purposes increased while computer anxiety decreased. Navarro and Shoemaker (1999) reported, “...we see that cyberlearners performed significantly better than the traditional learners. Mean score [final exam] for the cyberlearners was 11.3, while the mean score for traditional learners was 9.8. With a t-test statistic of 3.70, this result was statistically significant at the 99 percent level.”
Along these same lines, a California State University Northridge study reported that e-learners performed 20 percent better than traditional learners (TeleEducation 2000). Nelson (2001) reported a significant difference between the mean grades of 406 university students earned in traditional and distance education classes, where the distance learners outperformed the traditional learners.
In a study within the insurance industry, Redding and Rotzien (1999) report that the online group is the “most successful at cognitive learning as measured by the end of course examinations... The results of the study do provide strong support for the conclusion that online instruction for individuals entering the insurance field is highly effective, and can be more effective than traditional classroom delivered instruction.”
Similar results in support of e-learning came from Asynchronous Learning Networks (ALN) (2001), which reported a summary of empirical studies submitted to them. From the 15 papers in which the effectiveness of ALN was compared to that of traditional classroom instruction, two-thirds reported e-learning to be more effective. The remainder of the papers reported no significant difference.
As an extension of the more usual measures of Level II learning, Jonassen (2001) stressed the crucial need to develop critical thinking and other higher order skills among students using e-learning products. Earlier, Bates (1996) noted that: “the potential for developing higher order skills relevant to a knowledge-based society is a key driver in developing computer-based distance education courses.” Examining how learners engage in higher order thinking is the topic of a research study at Massey University in New Zealand. White (1998) examined strategies of 420 foreign language learners at that university and reported that distance learners made greater use of metacognitive strategies – what individuals know about their own thinking – compared to classroom learners, most notably with regard to strategies of self-management and advance organization and, to a lesser extent, revision. In a study of the infusion of technology in education, Serrano and Alford (2000) conducted research that clearly showed that incorporating technology across the curriculum acts as a catalyst for all learners. They concluded that e-learning empowers students to engage actively in language-content learning tasks and to develop higher-order critical thinking, visualization, and literacy skills.
While developing critical thinking and other higher-order skills is undoubtedly a desirable goal in a purely academic setting, it may be less important in the areas of specialized job-related content delivery or skill-building associated with many types of corporate online training programs. This is yet another evaluation issue that needs to be addressed in this arena.
Even well informed, quantitative learning objectives do not typically indicate how the trainee will transfer that learning to job performance. Changed on-the-job behavior is certainly the main goal of most corporate training programs, but measuring this change is a more complex task than eliciting trainees’ feelings or measuring their direct learning through test scores. In a number of studies included here, there is an assumed connection between measures of behavioral change and the hoped for consequence: solid business results (Level IV), although in most cases, empirical measurement is lacking.
In their overview of the evaluation process, Bregman and Jacobson (2000) discuss the need to measure business results rather than just evaluate trainee test results. They point out that all important business results affect customer satisfaction, either directly or indirectly. Business results that may increase efficiency or help short-term profits – but do not increase customer satisfaction – are obviously bad for business. These authors claim that changes in customer satisfaction due to training of sales or service personnel are easy to measure by asking the customers of the trainees to compile reaction surveys. Generally, reaction sheets for customers get high response rates; therefore, a valid connection between the effects of training on the employee and how the customer feels about that employee can be made. Bregman and Jacobson summarize that a training program succeeds, by definition, when the training changes employees’ behaviors in ways that matter to their customers.
Unilever claims that e-learning helped their sales staff produce more than US$20 million in additional sales (Hoekstra, 2001) – Level IV evaluation. They track the results of their e-training programs by asking course participants to take part in a teleconference several months after the course. Participants are asked to discuss how they have integrated their new skills into their work and to share their best practices – Level III evaluation. Uniacke, the person in charge of Unilever’s training program, points out that many results of e-training programs are difficult to measure. For example, he is convinced many employees do not learn new material, but rather they polish their overall skills and customer interaction techniques – still a significant benefit to the company and its overall bottom line.
As a number of authors have pointed out, it seems that traditional trainers incorporate the first three levels routinely in the design of training programs (e.g., see Boverie, Mulcahy, and Zondlo, 1994). In a more recent report on e-learning evaluation, Hall and LeCavalier (2000 a,b) make a strong case for focusing on Level III with job performance-based measures. Their research study of eleven U.S. and foreign companies helped them identify best practices within these companies, which have significant e-learning success stories. They conclude that the most promising strategy for many companies is to focus on Level III to find out what is really effective within e-learning programs.
Level IV evaluation attempts to measure the results of training as it directly affects a company’s bottom line – a challenging task for many reasons. Kirkpatrick (1999) noted that the number of variables and complicating factors make it difficult, if not impossible, to evaluate the direct impact of training on a business’ bottom line – and this is just as true for e-learning as for traditional training programs.
While reduced costs, higher quality, increased production, and lower rates of employee turnover and absenteeism are the desired results of training programs, most companies do not address this complex evaluation process. However, some companies strive to make the difficult link between training and improved business results.
Some firms are beginning to measure e-learning results for their sales force in terms of increased sales, as in the Unilever case. In another example, Etera, a nursery supply company, uses e-learning to train its national sales force. Their headquarters claims that an Etera-certified dealer who has gone through the online training has 170 percent more sales than an untrained dealer (Zimmerman, 2001).
In a different approach to business results, Bassi’s research (2001) demonstrates that investment in training adds to the value of a company’s shares – a high priority for corporations – and she claims that there is added value regardless of overall market conditions.
To use Phillips’ ROI calculation as an added level to Kirkpatrick’s model requires a lengthy and complex evaluation and calculation process. Using Level IV evaluation data, the results are converted into monetary values and then compared with the cost of the training program to obtain a return on investment.
Phillips (1996) summarizes how Magnavox Electronics Systems Company derives its ROI calculations as it evaluates all five levels of its 18-week literacy program, which covers verbal and math skills for employees. While this is not an e-learning program, it does demonstrate the process of moving through the levels of evaluation, a process that would be equally applicable for the use of e-learning as the delivery method for training content.
In all of the studies being conducted, how systematically are e-learning results being analyzed and are the appropriate correlations being made? When corporations measure e-learning results, are they measuring the right elements? Even when the results are positive in favor of e-learning, are we obtaining and/or measuring quality learning in areas that matter?
In this very new research field, there are challenges to results on both sides of the issue. Joy and Garcia (2000) warned that many of the earlier studies lack scientific validity because the research designs are seriously flawed. This makes many of those results questionable. They point out that if researchers do not carefully control for the most likely factors explaining the variance in student achievement, one may not find significant differences between experimental and control groups.
Joy and Garcia also stress another crucial point – namely, that it is extremely difficult to develop a solid scientific method for comparing the various delivery methods. They point out that even if a legitimate scientific model could be designed to properly control for each independent variable, its usefulness for predicting learning outcomes would, in all likelihood, be extremely limited. This implies that the researcher would have to impose artificial controls to produce true empirical results.
In another careful look at previous research, Saba (2000) noted the fact that many of the original studies, while experimental in nature, were not grounded in a theoretical framework. The researchers simply carried out their experiments in which they compared distance learning with traditional classroom content delivery and reported the statistical results. Perraton (2000) agrees and adds that most previous studies have been in the areas of course or program description, audience analysis, cost-effectiveness, methodology, and social context.
Furthermore, Saba reports that a few researchers frame their studies with increasing attention to methodological issues within the framework of the appropriate theoretical foundations. He cites several researchers – such as Smith and Dillon (1999), Cookson and Chang (1995), Gunawardena and Zittle (1997), and Sherry, Fulford, and Zhang (1998) – as those who not only grounded their studies on theoretical foundations in the field of distance learning, but also used new methods of inquiry, such as discourse analysis and in-depth interviews, to obtain more meaningful learner results.
Saba goes further to point out that analysis of such studies has continued to reveal just how complex the study of distance education is because of the many variables involved in any instructional setting plus other elements such as social, economic, and global issues affecting the field.
For the corporate training arena, Bregman and Jacobson (2000) note that
the additional desired outcome of positive business results is notoriously difficult to measure because of the following factors:
With these caveats about research results in the field of e-learning, how do we arrive at accurate results?
One way to obtain meaningful results is to design more effective assessment methods. According to Driscoll (2001a) “Assessments are the foundation of effective instructional practices and return-on-investment studies. The power of tests and assessments will become exponentially more important with the advent of content management systems and learning management systems.” Indeed, data from assessments should help drive the development of solid content and advanced instructional practices.
As Moore (1999) noted: “One of the few generalizations that can be made about any distance education program – whatever the communications media used and the content level – is that a good monitoring and evaluation system is likely to lead to a successful program, and a poor system is almost certain to lead to failure.” Moore describes the three key features of a good system as follows:
1. The preliminary specification of good learning objectives, with this crucial question at the heart: Did each student produce evidence of having learned what was required as specified in the learning objectives? If not, why not?
2. The construction and handling of assignments, which are the students’ evidence of learning and an important source of feedback for the program.
3. A good data gathering and reporting system and a solid review of all of the data by both instructors and program administrators.
Another way of approaching the attempt to guarantee better results in e-learning programs is to look at content quality measures, i.e., the quality of the online education product itself. The National Education Association and Blackboard Inc. examined case studies of six higher education institutions that provide Internet-based degree programs. Their purpose was to ascertain the degree to which various measures of quality identified in previous studies were actually being incorporated into the policies, procedures, and practices of institutions that have distance education learners. The result was a list of twenty-four benchmarks that they deemed essential to ensuring quality in Internet-based education. These benchmarks were grouped under the categories of institutional support, course development, teaching/learning, course structure, student support, faculty support, and evaluation and assessment (The Institute for Higher Education Policy, 2000).
A new e-learning evaluation method is also gaining support in Canada (ASTD, 2000a). The guidelines, presented in Quality Standards for Evaluating Multimedia and Online Training, have been endorsed by the Canadian government and the Ontario Society of Training and Development. Their model evaluation process was tested at twenty organizations. They first determined an e-learning course’s relevance toward an organization’s needs, followed by analysis of content quality, usability, and instructional design methodology. According to their developer, Lynette Gillis (in ASTD, 2000a), these guidelines provide a robust and comprehensive set of quality criteria that have been shaped by experts in the field of distance education.
Another effort to address issues pertaining to competences and standards for teachers who design and deliver online courses came from an international conference of the Center for Studies in Advanced Learning Technology (CSALT) and the International Board of Standards of Training, Performance, and Instruction. As a result of this meeting, an initial set of competences and performance statements has been formulated and refined (Spector, Ganesan, Goodyear, and de la Teja, 2001).
In the area of e-learning for corporate training, as with university based e-learning, some fundamental questions remain. Burnside (2001) summarized these questions as follows:
1. How does learning actually occur in the human being, individually, in groups, and in organizations?
2. How do we increase students’ capacity for learning?
3. What evidence do we have of the benefits of increased capacity for learning?
4. In what ways can we communicate this effectively in every educational act that we do?
Certainly, those involved in e-learning evaluation must continue to examine evaluation strategies. In addition to Kirkpatrick’s classic four-level model and Phillip’s additional fifth level, a number of alternate evaluation measures are emerging. For example, Holton (1996) recommends an alternative to Kirkpatrick’s model. In his model, the impact of intervening variables such as motivation, trainability, job attitudes, and personal characteristics are considered. Other literature from the traditional training literature is also valuable. For example, Todesco (1997), of The Research Centre, interprets evaluation trends and best practices. Abernathy (2001) challenges us to think outside the evaluation box.
While few people debate the obvious advantages of e-learning, systematic research is needed to confirm that learners are actually acquiring and using the skills that are being taught online, and that e-learning is the best way to achieve the outcomes in a corporate environment. This research must be grounded in solid theoretical precepts to assure that meaningful results are obtained.
Until a more solid research methodology is developed for measuring e-learning results, we can rely on the mainly qualitative feedback from corporations that are using e-learning to deliver their training. Firms praise online training as a cost-effective, convenient, and effective way to deliver corporate education. Early studies seem to demonstrate that e-language-learning in business is a win-win proposition for all – the learner, the corporation, and the customers served by the corporation.
Abernathy, D. (2001). Thinking Outside the Evaluation Box. ASTD Virtual Community. Retrieved February 17, 2002 from: www.astd.org/CMS/templates/index.html
ALN. (2001). Summary of published ALN research studies. WebCenter for Learning Networks Effectiveness Research. Retrieved February 17, 2002 from: www.ALN.org
ASTD. (2002). State of the Industry Report. Retrieved February 17, 2002 from: www.astd.org
ASTD. (2001). State of the Industry Report. Retrieved February 17, 2002 from: www.astd.org
ASTD. (2000a). E-learning evaluation method gains support in Canada, ASTD Learning Circuits, July. Retrieved February 17, 2002 from: www.learningcircuits.org
ASTD. (2000b). Evaluating the Effectiveness and the Return on Investment of E-learning. What Works Online. 2nd quarter. Retrieved February 17, 2002 from: www.astd.org/virtual_community/research
Barron, T. (2001). An e-learning industry update. ASTD Learning Circuits, August. Retrieved February 17, 2002 from: www.learningcircuits.org
Bassi, L. (2001). This Firm Bets on Training. Retrieved February 17, 2002 from: http://www.knowledgeam.com/rds.html
Bates, A. W. (1996). The impact of technological change on open and distance learning. Keynote address at Open Learning: your future depends on it. Queensland Open Learning Network, Brisbane, Queensland, Australia.
Boverie, P., Mulcahy, D. S., and Zondlo, J. A. (1994). The 1994 Annual: Developing human resources. J.W. Pfeiffer (Ed.) San Diego: Pfeiffer and Company.
Bregman, P., and Jacobson, H. (2000). Searching for Answers: Yes, you can measure the business results of training. Training 38(8), 68 – 72.
Burnside, R. (2001). Education ourselves about education – Corporate training comes of age. Corporate Universities International. 7(6), July/August 2001.
Bylinsky, G. (2000, June). Hot new technologies for American factories. Fortune. Retrieved February 17, 2002 from: www.fortune.com/fortune/imt/2000/06/26/elearning.html
Cookson, P. S., and Chang, Y. (1995). The multidimensional audioconferencing classification system (MACS). The American Journal of Distance Education 9(3), p. 18-36.
Dalton, J. P. (2000, August). Online training needs a new course. The Forrester Report, (2000, September 23) Retrieved February 17, 2002 from: http://www.forrester.com/ER/Research/Report/Excerpt/0,1338,10060,FF.html
Driscoll, M. (2001a). Building better e-assessments. ASTD Learning Circuits July. Retrieved February 17, 2002 from: www.learningcircuits.org
Driscoll, M. (2001b). Myths and realities of e-learning. Invited address at ED-MEDIA 2001: World Conference on Educational Multimedia, Hypermedia, and Telecommunications. June, Tampere, Finland.
ecollege.com (1999). Survey Finds Online Education Equal to or Better than On Campus Learning: eCollege.com. Study Shows Positive Results for Online Education Based on Feedback From Students and Faculty. Retrieved February 17, 2002 from: http://www.ecollege.com/company/news/stories/press_2.html
Gunawardena, C. N., and Zittle, F. J. (1997). Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. The American Journal of Distance Education 11(3), 8 – 26.
Hall, B., and LeCavalier J. (2000a). The Case for Level 3. ASTD Learning Circuits, November. Retrieved February 17, 2002 from: http://www.learningcircuits.org/nov2000/hall.html
Hall, B., and LeCavalier J. (2000b). E-Learning Across the Enterprise: The benchmarking study of best practices. Retrieved February 17, 2002 from: http://www.brandon-hall.com/elacenbenstu.html]
Hoekstra, J. (2001). Three in one. Online Learning 5(10), 28 – 32.
Holton, E. (1996). HRD Quarterly 7(1), 5 – 21.
The Institute for Higher Education Policy (2000). Quality on the Line: Benchmarks for success in internet-based distance education. Retrieved February 17, 2002 from: www.IHEP.com/quality
Jonassen, D. (2001). E-learning to solve problems. Keynote address at ED-MEDIA 2001: World Conference on Educational Multimedia, Hypermedia, and Telecommunications. June, Tampere, Finland.
Joy, E., and Garcia, F. (2000). Research paper questions ‘No Significant Difference’ claims. Journal of Asynchronous Learning Networks 6(1), 21 – 39.
Kirkpatrick, D. (1979). Techniques for evaluating training programs. Training and Development Journal 33(6),78 – 92.
Maki, R. H., Maki, W. S., Patterson, M., and Whittaker, P. D. (2000). Evaluation of a Web-based introductory psychology course: Learning and satisfaction in on-line versus lecture courses. Behavior Research Methods, Instruments, and Computer 32, 230 – 39.
Moore, M. G. (1999). Monitoring and evaluation: Editorial. The American Journal of Distance Education 13(2).
Navarro, P., and Shoemaker, J. (1999). Economics in Cyberspace: A comparison study discussion paper. University of California - Irvine, Graduate School of Management.
Nelson, G. (2001). Do no harm: A first measure of effectiveness in small distance education programs. Proceedings of ED-MEDIA 2001: World Conference on Educational Multimedia, Hypermedia, and Telecommunications. June, Tampere, Finland.
Nettles, K., Dziuban C., Cioffe, D., Moskal, P., and Moskal, P. (2000). Technology and Learning: The ‘no significant difference’ phenomenon: A structural analysis of research on technology enhanced instruction. In Dziuban and Moskal (Eds.) Distributed Learning Impact Evaluation. Orlando: University of Central Florida.
Online Learning News. (2001a). Blending and Spending. Online Learning News 4(13). Retrieved February 17, 2002 from: www.vnulearning.com
Online Learning News. (2001b). Gunk and other blender woes, Online Learning News 4(17). Retrieved February 17, 2002 from: www.vnulearning.com
Perraton, H. (2000). Rethinking the research agenda. International Review of Research in Open and Distance Learning 1(1). Retrieved February 17, 2002 from: www.irrodl.org
Phillips, J. (1996). Measuring ROI: The Fifth Level of Evaluation. Technical Skills and Training. April. Retrieved February 17, 2002 from: http://www.astd.org/virtual_community/commevaluation/Phillips.pdf
Redding, T. R., and Rotzien, J. (1999). Comparative analysis of SDL online training with traditional classroom instruction. Presented at the 14th International Symposium on Self-Directed Learning.
Russell, T. L. (1999). The No Significant Difference Phenomenon, 5th ed. Raleigh, NC: North Carolina State University. Retrieved February 17, 2002 from: http://teleeducation.nb.ca/nosignificantdifference/
Saba, F. (2000). Research in distance education: A status report. International Review of Research in Open and Distance Learning 1(1). Retrieved February 17, 2002 from: www.irrodl.org
Serrano, C., and Alford, R. L. (2000). Virtual Languages: An innovative approach to teaching EFL/ESL English as a foreign language on the World Wide Web. In L. Lloyd. (Ed.) Teaching With Technology: Rethinking Tradition. Less Lloyd Medford, NJ.: Information Today, Inc.
Sherry, A. C., Fulford, C. P. and Zhang, S. (1998). Assessing distance learners’ satisfaction with Interaction: A quantitative and a qualitative measure. The American Journal of Distance Education 12(3), 4 –28.
Smith, P. L., and Dillon, C.L. (1999). Comparing distance learning and classroom learning: Conceptual considerations. The American Journal of Distance Education 13(2), 6 – 23.
Spector, J. M., Ganesan, R., Goodyear, P., and de la Teja, I. (2001). Competencies for online teaching and training. Proceedings of ED-MEDIA 2001: World Conference on Educational Multimedia, Hypermedia, and Telecommunications 2001. June, Tampere, Finland.
teleEducation (2000). Is distance education any good? Retrieved February 17, 2002 from: http://teleeducation.nb.ca
Tedesco, A. (1997). From Training Evaluation to Outcome Assessment: What trends and best practices tell us. Retrieved February 17, 2002 from: http://learnet.gc.ca/eng/rescentr/fulltx/outpap.htm
Wegner, S. B., Holloway, K. C., and Garton, E. M. (1999). The effects of internet-based instruction on student learning, Journal of Asynchronous Learning Networks 3(2). Retrieved January 19, 2001 from: http://www.aln.org/alnweb/journal/Vol3_issue2/Wegner.htm
White, C. J. (1997). Effects of mode of study on foreign language learning. Distance Education 18(1), 178 – 196.
Zimmerman, E. (2001). A Competitive Edge. Online Learning 5(10), E2 – E7.