April - 2002

Technical Evaluation Report

10. Evaluating Vendor Supplied Information

Patrick J. Fahy
Centre for Distance Education,
Athabasca University


Distance educators are not normally equipped by their training or experience for the complex task of evaluating technologies. One of the areas of potential disadvantage is in interpreting information provided by vendors themselves, and in relating effectively with sales, marketing and technical representatives. An objective and thorough product evaluation requires that information be selected, and sometimes generated, to aid the process. Vendors may agree to provide additional information, including direct experience with their products, if evaluators know what to ask for and what to expect from vendors.


In evaluating potential distance delivery and support technologies, researchers may have occasion to use information provided by vendors. Historically, education in general and distance education particularly have been considered niche markets by most major technology developers and providers; however, as educational technology investments in general increase, and as distance training in the private sector grows, more vendor resources are being directed to promotion of distance learning applications of all kinds.

Vendor promotion of and publicity about their products, services and achievements creates both challenges and opportunities for distance educators attempting to make intelligent technology choices. Because distance educators are not usually experienced purchasers of technologically complex and expensive systems, and because they usually are not aware of common practices in marketing and sales in these fields, some caveats and an exploration of possible strategies for gaining reliable information to inform the selection process may be helpful. While providing some cautions, this advice might also give individuals who lack marketing or purchasing experience some understanding and some strategies, permitting them to interact on a more equal basis with professional sales and marketing people.

While evaluating the capability of the technology to meet the organization’s needs and expectations, evaluation of software and hardware should also consider the readiness of the organization for adoption of the technology. Welsch (2002) reminds evaluators that innovations of all kinds encounter financial, structural and cultural challenges in any organization, and that an assessment must be made of the organization’s readiness to change in order to successfully incorporate new technologies. The evaluator’s task includes both looking at a technology’s potential usefulness to the organization, and assessing the organization’s readiness to adopt the technology successfully. A mismatch between corporate readiness and the technology’s demands as an innovation could be disastrous for an unprepared organization.

Assumptions and Strategies for Evaluators

A key assumption in the following discussion is that vendor-supplied information can be used effectively in evaluating a technology, if the potential purchaser understands in advance some facts, and, where strategically appropriate, engages in some timely activities.

1. Truth vs. the whole truth in vendor publications. While there should be no misinformation, vendor-supplied information is almost never complete. Vendors know they are not on the witness stand, and they therefore do not feel they have to tell the “whole truth.” (They may be following St. Paul’s insightful advice to the citizens of Corinth: “All things that are true for me may not be expedient.”)

2. Real vs. ideal performance reports. Similar to the above, information provided in marketing and sales publications commonly highlights only the strengths of the products and the company, usually in a non-technical sense, often in comparison with known weaknesses of the competition, and sometimes referring to performance under laboratory (rather than “real-world”) conditions of use.

3. Access to technical information. Related to the above point, evaluators should be able to access quality, objective information from a vendor, consulting, as necessary, appropriate internal and external expert sources. Technical specifications include credible engineering information describing the performance or requirements of the technology under controlled conditions.

4. Reference-site information. Reference sites are customers of the vendor who know (and presumably are happy with) the product, and who are also regarded by the vendor as using the product intelligently, effectively and appropriately. Reference sites can be regarded as demonstrating the most effective potential outcomes likely achievable with the product.

5. Business facts. Information about the company itself, as a corporation, can sometimes show how well managed and accepted the company and its products are in its field. This information is usually only readily available on public companies, which by law must make public certain information about their business dealings. While information on public companies should be entirely accurate (federal regulations apply), the data may be difficult to interpret, and may also be incomplete.

6. The right to pilot-test. Because the acquisition of complex technologies, often unfamiliar in the organization, represents both a major investment and a high risk, the opportunity to pilot-test the product may be of value to some purchasers.

7. Price and support stability. Sales practices in relation to technologies often change as marketing opportunities and products evolve; in some companies, these may also change if sales representatives need to book revenue at the end of a quarter or a fiscal year. To potential purchasers, frequent changes in product packaging or in sales strategies may lead to questions about the product or the company itself.


Vendors competing for business in the highly competitive and often financially lean, even distressed, distance education marketplace, provide product information of different kinds in a variety of forms. As well, the volume and intensity of these kinds of representations may be considerable: marketing and sales expenditures in relation to educational technologies, especially of software products, may comprise a major part of the cost of doing business, sometimes significantly in excess of the cost of the product itself. For example, in 2001, PLATO Learning Inc. reported that its costs for “selling, general and administrative expenses” had increased 30 percent, to 63 percent of revenue, up from 61 percent of revenue in 2000; in 2000, the increase had been 29 percent. Bearing in mind that PLATO is a mature company, with established products and customer based, and therefore less in need of core development and support expenditures, for the same two years “product development and customer support” comprised 11.3 percent and 12.4 percent of revenue, respectively.

Evaluators should know how to judiciously assess and use vendor-provided material, including knowing when to ask for more or different kinds of information. Gathering needed product and organizational data, generating new data where necessary, and helping colleagues see how the data relate to intended outcomes, are core tasks of the technology evaluation process in a training or educational environment. Because of the long-term organizational implications, the evaluation and selection process must generate an accurate and realistic description both of the technologies’ capabilities and of potential impacts on the organization. Vendors may legitimately help in this process if their role and contributions are properly managed by the evaluator.


PLATO Learning Inc. (2002). 2001 Annual Report. Retrieved March 14, 2002, from: www.plato.com

Welsch, E. (2002). Cautious steps ahead. Online Learning 6(1), 20 – 24.

This series of software evaluation reports will continue with reviews of other online collaborative tools.

N.B.Owing to the speed with which Web addresses are changed, the online references cited in this report may be outdated. They can be checked at the Athabasca University software evaluation site: cde.athabascau.ca/softeval/. Italicised product names in this report can be assumed to be registered trademarks.

JPB. Series Editor, Technical Notes