Article: Demonstrating the Value of Technical Communication Products (Part 2 of 6)

The Commerce of Content
Article:
Demonstrating the Effectiveness and Value of Technical Communication Products and Services:
A Four-Level Process (Part 2 of 6)

Project Management | People Management | Business Management | Information Design Models, Processes, and Techniques | Home

Introduction

In this Section
The Quest for Quality
How Trainers Assess Their Effectiveness and Value
Limitations of the Training Model to Technical Communicators

The Quest for Quality

Just like the North American explorers seeking the elusive Northwest Passage, we technical communicators have been looking for a certain statistic that, if we were to find it and measure it, we could prove without a doubt that our work was of high quality and that would demonstrate, without question, the value of a technical communicator’s services to an employer.

We devised many approaches to assessing quality and demonstrating value began in the 1970s and 1980s. They include:

Counting comments on drafts, hoping to establish a correlation between the number of review comments and the quality of communication products.
Assessing conformance to requirements (Bandes, 1986). This is called the industrial definition of quality because it is the same approach used to assess the quality of manufactured parts, like circuit boards.
Devising the U-metric, a list of 100-plus characteristics that are associated with the quality of communication products.

Since the early 1990s, we have generally concluded that no single, unquestionable measure quality and value. We reached that conclusion, in part, because we are increasingly distinguishing between the terms quality and value and recognize that they are not synonymous.

In simple terms, quality is a key attribute that customers use to evaluate products or services....attributes associated with quality vary among products...and firms... (Shetty & Buehler, 1988, 6).

Quality is one of the criteria that people use to assess value. Shetty and Buehler define value as:

Quality

______

Price

That is, people perceive value as a relationship between price and quality. Value, then, addresses the financial dimension of quality.

We also reached this conclusion, in part, through the results of the Redish and Ramey study into the value added by technical communication (1995). They concluded that the value of technical communication is partly demonstrated by the revenues generated or expenses avoided by the communication products we produce. (Communication products is a generic term for the user’s guides, help, references, tutorials and other materials that we produce.) For example, by redesigning a form, a technical communicator working for the U.S. Veterans Administration was able to increase responses and reduce errors, thus significantly reducing mailing and re-work costs. (Daniel, 1995.)

As our field matures, we are increasingly aware of the many characteristics that contribute to quality. Shetty and Buehler note:

Definitions [of quality] usually consider performance features: reliability, conformance, durability, serviceability, aesthetics, and overall reputation or perceived quality (p.6)

Usability, for example, has emerged as a leading characteristic of quality technical communication. In fact, some technical communication products must pass a usability test before organizations will release them to the public. Usability does not assess quality perse, it assesses the effectiveness of technical communication products in meeting the needs of users.

Similarly, we are increasingly realizing that we cannot attribute the quality and value of technical communication products to one contributor, such as the writer or editor but, rather, to the entire team that developed the communication product. (Lay, et al. 1993, Barnum, 1993). Rather than looking for the isolated contribution of a writer or editor, then, we should assess the quality and value of the end product that they collaboratively produced.

Finally, we are realizing that some customers perceive the quality and value of technical communication products not on the products themselves, but on the "customer service" provided to clients by technical communicators. (Clients are the people for whom we produce technical communication products, whether we do so as internal employees or external consultants.) That is, clients often assess the value of technical communication on our ability to serve their needs, such as our ability to meet schedules, work within budgets, and pro-actively inform them about potential problems on a project. Often, clients’ perception of customer service bears no relationship to the assessed effectiveness of the technical communication products we produce for these clients (Carliner & Fredrickson, in press).

Although our understanding of quality and value is maturing, and we have begun to identify characteristics and perceptions that we can assess, we still lack a cohesive framework for considering issues of quality and value. Furthermore, we lack a consistently-used, widely accepted methodology for assessing the quality and value of our work.

In this article, I attempt to provide such a framework. The framework is adapted from a similar framework developed by Kirkpatrick for trainers (1959). First, I present the Kirkpatrick model and identify the limitations in using it to assess the quality and value of technical communication products. Next, I present an adapted model for assessing the quality and value of technical communication products. Last, I address a series of issues related to collecting data about quality and value, such as when to collect the data, how to ensure the credibility of the data, and how to report the data to others. I present this framework in the hope that it will start the process of developing a widely used and accepted approach to assessing the quality and effectiveness of technical communication products.

Top

How Trainers Assess Their Effectiveness and Value

Like technical communicators, trainers are concerned about demonstrating the quality and value of their services. Like ours trainers’ concern partly stems from a need to justify their jobs. Working, as we do, in organizations that hold people accountable, trainers need a method for demonstrating that their work has had an impact on the organizations they work for. Because many training courses, like many technical communication products, are not considered direct product expenses, corporate accounting systems do not offer an easy way of demonstrating impact. Perhaps that impact can be demonstrated by assessing the effectiveness and value added by training courses. But also like us, trainers’ concern for demonstrating the effectiveness and value of their services partly stems from a genuine desire to find out how their work has affected others.

Like technical communicators, trainers also have a number of tools to assess their effectiveness. For example, they have post-class surveys to assess participants’ satisfaction with a class and tests to determine whether participants learned the intended material.

But these assessments do not go far enough. Although participants might have liked a course and learned the intended material, the course still might not have resulted in any significant changes to the way that workers perform their jobs or to the bottom line of the organization. Furthermore, trainers did not have a framework for approaching the various questions about the quality and value of their work.

Then, Kirkpatrick proposed a four-level model for evaluating training courses (which is documented most thoroughly in his 1994 book). This framework assesses various aspects of the effectiveness of training courses. Trainers and their clients can then use the four different assessments to reach conclusions about the overall quality and value of a given training course, as well as its impact on the bottom line of the client who requested the course.

Following are Kirkpatrick’s four levels of evaluation.

Level	Name	Issues Assessed at this Level
I	Reaction	Assesses participants’ initial reactions to a course. This, in turn, offers insights into participants’ satisfaction with a course, a perception of value. Trainers usually assess this through a survey, often called a "smiley sheet." Occasionally, trainers use focus groups and similar methods to receive more specific comments (called qualitative feedback) on the courses. According to the TRAINING magazine annual industry survey, almost 100 percent of all trainers perform "Level I" evaluation.
II	Learning	Assesses the amount of information that participants learned. Trainers usually assess this with a criterion-referenced test. The criteria are objectives for the course: statements developed before a course is developed that explicitly state the skills that participants should be able to perform after taking a course. Because the objectives are the requirements for the course, a Level II evaluation assesses conformance to requirements, or quality.
III	Transfer	Assesses the amount of material that participants actually use in everyday work 6 weeks to 6 months (perhaps longer) after taking the course. This assessment is based on the objectives of the course and assessed through tests, observations, surveys, and interviews with co-workers and supervisors. Like the Level II evaluation, Level III assesses the requirements of the course and can be viewed as a follow-on assessment of quality.
IV	Business results	Assesses the financial impact of the training course on the bottom line of the organization 6 months to 2 years after the course (the actual time varies depending on the context of the course). For many reasons, Level IV is the most difficult level to measure. First, most training courses do not have explicitly written business objectives, such as "this course should reduce support expenses by 20 percent." Second, the methodology for assessing business impact is not yet refined. Some assess this measurement by tracking business measurements, others assess by observations, some by surveys, and still others assess by qualitative measures. Last, after six months or more, evaluators have difficulty solely attributing changed business results to training when changes in personnel, systems, and other factors might also have contributed to business performance. Despite these difficulties in obtaining a measure, over 50 percent of organizations perform this type of evaluation on 50 percent of their courses (TRAINING, 1995). Level IV evaluation is assessment of quality. It does so in financial terms, a perspective different than that of the evaluations at Level II and Level III.

Trainers widely follow this model (although, admittedly, not every organization performs each type of evaluation for every training course). Nearly all organizations perform Level I and II evaluations on most of their courses (TRAINING Industry survey). Many organizations perform Level III and IV evaluations on some of their courses. The framework is followed widely enough that, at conferences and in journals and magazines, trainers refer to these evaluations in a short-hand language: Level 1, Level 2, Level 3, and Level 4.

Top

Limitations of the Training Model to Technical Communicators

Although some aspects of this model transfer well to evaluating technical communication products, other aspects do not. Consider each type of evaluation:

Level 1, Reaction. We, too, can track the satisfaction of our users. Many technical communicators attempt to do so through Reader’s Comment Forms, which we include at the end of some printed communication products, like user’s guides and reference manuals. But a glance at most Reader’s Comment Forms indicates that they do not systematically assess user satisfaction. Instead, they ask users to identify errors.

Furthermore, technical communicators do not seem to include Reader’s Comment Forms with every communication product we produce. For example, the manuals provided with one of the most popular software products, Microsoft Office, do not have Reader’s Comment Forms. This sharply contrasts with the almost universal distribution of "smiley sheets" in training courses.

The problem moves with us online. Online help in most commercial software products rarely offers users with opportunities to comment on their satisfaction or identify errors in the online text. Some websites do, Although websites generally invite readers to "send comments." When users choose that option, the system displays a blank in which to type a message, not a series of specific questions about user satisfaction.

Level 2, Learning. Should we develop training materials, we certainly should measure whether users learned the intended material. But technical communication is a diverse field. Sure, some technical communicators produce training materials like workbooks and tutorials. Others produce marketing materials, such as parts catalogs, proposals, and brochures. And the majority of us produce product and service information, such as user’s guides, help, references, and troubleshooting procedures. Most product and service information is:

Disposable, intended only for one use, such as the procedures for end users to install commercial software (admittedly, technical support staff uses such procedures more frequently)
Only intended for occasional use, such as troubleshooting instructions and replacement parts for manufacturing machinery
Motivational, such as demonstrations that are intended to generate interest among prospective consumers

Once people use the information, they no longer need it or are not likely to use it for a long time, and should not be expected to retain it. For example, casual users of software products should only use troubleshooting procedures once or twice. Measuring retention of information in instances such as these seems inappropriate. (In fact, if customers need to use these procedures more frequently, they are likely to replace the software.)

Instead of measuring learning in all instances, then, technical communicators should measure users’ ability to perform the tasks described in the communication product. That is, if users should be able to install a product, then we should assess how well users actually performed the installation. If users should be able to troubleshoot a problem, then we should assess their speed and accuracy in diagnosing and resolving the problem.

Level 3, Transfer. Most assessments of users’ ability to perform tasks also assess how effectively users can transfer technical information to the task at hand. For those technical communication products that are "disposable," or only intended for occasional use¾ the bulk of technical communication products¾ the two levels can be therefore assessed at one time. For technical communication products intended to train or motivate, a separate assessment of transfer is still appropriate.
Level 4, Business Results. This is the measurement of the value added by technical communication products to the bottom lines of the organizations that commissioned these products. To best assess business results, however, technical communicators would have to begin each project with the objective of having a direct impact on a particular aspect of the bottom line. Such an objective is called a business objective. Thus, before we would begin developing a user’s guide or reference, we would specifically state how the communication product should reduce expenses, generate revenues, or help the organization comply with a governmental, industry-wide, or corporate regulation. In doing so, we would identify measurements we would track to determine whether the product achieved those goals.

In addition to these limitations, the Kirkpatrick model also fails to assess one key measurement: client satisfaction. One key reason for tracking the effectiveness of our products and services is to promote repeat business.

The need to promote repeat business is as essential to the 75 percent of technical communicators members who are employed within organizations as for the 25 percent working in the independent sector (STC membership survey, 1996). Although employed technical communicators will likely receive assignments within their organizations in the short run, their long-term ability to remain employed¾ much less grow¾ within an organization hinges on sustaining good relationships with internal clients. In a downturn or other type of corporate shakeup, technical communicators who have poor relationships with internal clients are likely candidates for dismissal. Through an ongoing assessment of client satisfaction, we might be able to adjust our work to better meet our internal or external clients’ needs.

Top

Sections of this Article