|
The Commerce of Content
|
Project Management | People Management | Business Management | Information Design Models, Processes, and Techniques | Home |
In this Section |
Although the Kirkpatrick model offers a framework for considering the effectiveness and value of technical communication products, it is not suited, as is, to our needs. I therefore suggest the following adaptation of the model for use with technical communication products and services.
| Level | Name | Purpose |
| 1 | User Satisfaction | Assesses how users feel about a given communication product |
| 2 | User Performance | Measures the extent to which users can perform the main tasks |
| 3 | Client Performance | Assesses the extent to which the communication product met the business objectives |
| 4 | Client Satisfaction | Assesses how clients feel about the information development process in general and the experience of working with you in particular |
The following sections describe the purpose of each level and suggest how you might conduct the assessment.
As a survey assesses how participants feel about a given training course, so an assessment of user satisfaction could tell us how users feel about a given communication product. Class surveys in training courses explore how participants felt about the instructors, the facilities, the visuals, the handouts, and similar aspects of the classroom experience. Some also ask participants to assess how much they believed they learned in the course and its value.
A similar survey of user satisfaction could assess whether users can find the information they need in a timely manner, whether that information answers their questions, and their level of satisfaction with the experience of using the communication product. Often, users’ perceptions of their ability to find and use information in a communication product affects their ultimate acceptance of it and of the software, concept, or product described.
Specifically, when collecting data about user satisfaction, we might seek information about:
Readers’ Comment Forms provide a natural starting point for collecting this information. But we would need to redesign these forms so that they ask focused questions to users, rather than simply requesting reports of errors in the communication product. Figure 1 is a sample of such a Reader’s Comment Form for a printed communication product. The same questions can be adapted for online products, videos, and audio tapes.
We also need to more actively invite users to submit their responses to Reader’s Comment Forms. Studies lament the low response rates for Reader’s Comment Forms. To increase response rates on printed materials, we might move the comment form to a more visible location, and give it an inviting design, one that says "mail me back." Online, we might randomly display the comment form when users initially start or close a program to make responding effortless.
Click here to see a sample of a reader's comment form that elicits feedback about user satisfaction.
This type of assessment evaluates whether users can perform the tasks that the communication product describes. But we can’t wait until we’ve developed the communication product to determine what those tasks are. We cannot even wait until we develop an outline for the communication product. We identify these tasks when we develop the requirements for the communication product.
When we state these tasks as requirements for a technical communication product, we should state them in a way that lets us easily assess whether users can perform the tasks. This tasks should be stated, then, in an observable and measurable way. After we develop the communication product, we can observe whether users can perform the task and measure the extent of their performance.
When we state tasks in observable and measurable ways, we would state:
(If these look like learning objectives used by instructional designers, they should.)
When identifying the tasks that users should perform, we distinguish among main tasks and supporting tasks. Main tasks are the tasks users must be able to perform. Supporting tasks are tasks users must be able to perform to complete the main task. For example, suppose the main task is correctly check the spelling of a document using the spellcheck feature. To effectively perform this tasks, users must perform the supporting task of starting spellcheck and recognizing the correct spelling of a word when presented with choices.
User performance is users’ ability to perform these tasks. The way we assess user performance varies, depending on the purpose of the communication product. The following suggests general ways to assess user performance for different types of communication products: training, marketing, sharing scientific information, and using a product, service, or policy.
If the purpose of a communication product is educational, then the we assess user performance by testing users after they use the training materials. The test determines whether users learned the intended information.
We derive test questions directly from the tasks. For example, if the main task (called an objective by instructional designers) indicates that users should be able to name five types of communication products, then the test question should be "Name 5 types of communication products." Similarly, if a main task indicates that users should be able to explain the three benefits of object oriented programming, then test question should be "Explain the three benefits of object-oriented programming."
In more complex situations, we use more complex questions. For example, if users should be able to choose the most appropriate programming technique for a given situation, we should provide users with a brief case study, then ask them to choose a programming technique and explain how they made the choice.
The type of question that you ask on the test is directly related to the action indicated in the objective. Consider these:
| If a Task Begins with this Word | Ask This Type of Question |
| Describe | Essay question, usually beginning with the word describe |
| Explain | Essay question, usually beginning with the word explain |
| Match | Matching |
| Name | Fill in the blank |
If the purpose of a communication product is marketing-related, such as promoting a product or service, or providing users with a catalog from which they can order products, then we assess user performance in one of these ways:
| If Users Are Supposed To | Then Assess Their Performance by |
| Order a product | The accuracy of the order; for example, if users are ordering a spare part for the BelAir model, did they order the correct part or did they order a part for the Biscayne model? |
| Request more information | The type of information they request; is it related to the task? |
If the purpose of your product is sharing scientific information, you use indirect means of assessing user performance. The ultimate purpose of sharing scientific information is that others use it. You track how the scientific information is used. Track citations in other articles and books. Track the creation of derivative works.
If the purpose of your communication product is telling users how to use a product, service, or policy, (as is done by user’s guides, help, references, troubleshooting information, and wizards) then best way of assessing user performance is observing them while they try to perform the task.
Usability tests have traditionally assessed how easily people can use products, services, and policies. In a usability test, technical communicators invite people who represent the intended audience to try to perform the main tasks using the communication product (Duin, 1993). Testers present users with scenarios. Figure 2 shows an example of a scenario. Users must perform certain tasks to complete the scenarios. Observers (usually people other than those who developed the product and the information tested) record:
Users perform these tasks under controlled conditions to ensure the accuracy of the data. Based on users’ performance, technical communicators might make changes to the information so that users can complete the task more quickly and with fewer errors. Figure 3 shows an example of a form used by observers of usability tests.
As testers assess the usability of communication products, they are also assessing users’ performance of main tasks. By testing the main objectives, we also test the supporting objectives because users must be able to perform the supporting objectives before they can perform the main objectives.
Consciously relating the main tasks (the requirements for a technical communication product) with the scenarios used to assess usability enhances the strategic role of usability testing in our development process. By including these scenarios in the plans for technical communication products, we subtly impress upon our clients the significance of usability testing in assessing the effectiveness of our work. Furthermore, by developing the scenarios in advance of the communication product, we can also focus our development efforts on "writing for the test." That helps us avoid the extraneous material that can often creep into our products.
This type of assessment measures the value added by technical communication products. Specifically, it assumes that, before we began developing the communication product, we identified the business goals it was trying to achieve and attempts to measure the extent to which the communication product was able to meet those goals. The business objectives for a given communication product should pertain to one of the following:
These three items are the most basic measures of business performance. Expenses and revenues directly relate to the two main sections of a organization’s financial statement. The financial statement is the ultimate assessment of business performance for most types of organization. Regulations are requirements for staying in business.
Assessing client performance requires that we identify a business measure when we first plan a communication product for a client. Once we identify the measure and the client agrees to it, we must also follow the changes in that measurement. We usually begin doing so before publishing the communication product and continue doing so long after publication. We measure changes over time, rather than at a single point in time, because the purpose of most business goals is facilitating a change over time, such as an increase in revenue or a decrease in expenses. Unless we track the trend in these measurements for a period before publishing the communication product, we cannot, with any credibility, assert that introducing the communication product caused a change in business performance.
Assessing business impact is not an exact science, however. As many people contribute to the success of a technical communication product, so many factors contribute to business success. We should therefore indicate other events that occurred during the time we were tracking the business measurement and that might have influenced any changes.
Consider this example of measuring client performance. The business goal of the technical communication product is keeping support costs to the projected level, $3.4 million for the coming year. We would begin tracking support costs immediately after the project begins. But we should also note other conditions. For example, we should note whether support costs increase before the communication product is published and, if so, how quickly they rise? This unexpected rise in support costs might affect the ability of our technical communication product to contain support costs. Then, continue to track support costs after the communication product is published through the entire year. Note the trend. As you track support costs, also note events that might have had an effect on the measurement. Suppose, for instance, that sales remained flat. The lack of additional demand for support services could just as easily account for the containment of support costs as the publication of a communication product.
Consider the measurement of another business objective: generate $25 million in sales between January and June. We should track revenue between January and June. But we should also track revenue in December and July to determine whether other factors might affect our measurements. Suppose, for example, that, in December¾ the month before the communication product was published¾ sales were unusually weak and sales immediately began lagging in July, when the client stopped distributing your communication product. Then the communication product clearly has had an impact on sales.
Finally, consider the business objective: receive ISO 9000 certification by June of next year. On the surface, the only business measurement should be taken in June of next year. But an organization must pass many intermediate inspections before receiving ISO 9000 certification. Track the intermediate inspections, too, and any comments in them about the information you provided.
In some instances, we can estimate the business results from a usability test. Generally, we can do so if the business objective pertains to reducing expenses and the client already knows how much poor performance of the main task costs. For example, suppose we are redesigning a form so that users complete it more accurately and the client already tracks the number of forms received with errors and how much the errors cost. From the usability test, we can estimate the percentage of forms that will be completed with errors. Then, using the average cost of correcting an error that the client has already determined, we can calculate the cost of handling errors with the new form. Assuming that cost is reduced, we can give a rough estimate of the likely savings the client will realize. Still continue to track actual errors, however, to make sure that the client receives the promised benefit.
In some instances, too, we cannot directly link a communication products with a concrete objective to contain expenses, generate revenues, or comply with a regulation. In such situations, we cannot directly demonstrate client performance.
This type of evaluation assesses clients’ feeling about the information development process in general and the experience of working with you in particular. By carefully tracking our performance in this area, we can assess the likelihood that a given external client will invite us to work with them again and how we might make future experiences with clients more satisfactory to all parties. If we are working with internal clients, we can assess the likelihood of smooth relations on future projects.
In some ways, this can be the easiest type of satisfaction to assess. If a client gives us repeat business, the client seems sufficiently satisfied with our work.
But not every client gives repeat business because they want to, some give it because they have to. The client might be an internal client and required to use our services, no matter how the client feels about the value we provide. Or the client might be an external client and have a one-time-only need for our services. That client might not give direct repeat business because the client does not have any. Unbeknownst to us, however, the client might refer others to us.
Assessing client satisfaction in the absence of repeat business presents a dilemma. Satisfaction surveys, like the one suggested for assessing user satisfaction, serve little good for measuring client satisfaction. Because of the intimacy of the relationship between a technical communicator and a client and, because technical communicators work with so few clients at any given time, clients are not likely to answer such a survey honestly because they might fear we could attribute comments to them. Nor would the results of such a survey have much validity because the sample size would be too small.
Performance reviews often provide an opportunity for candid feedback. Even those working with external clients can request a performance review. In a performance review, both participants review the objectives of the assignment and assess how well they were met. For a technical communication project, clients usually assess us on one or more of the following: our ability to meet schedules, budgets, and quality guidelines, whether users could perform the intended tasks with the information, and whether the communication product contributed to the organization’s bottom line.
In addition, clients must assess us on the perceived quality of our work, such as their affection for our writing styles and assessments of our creativity. Although subjective, this feedback tells us how clients responded to our work.
Sections of this Article 1. Abstract |
Project Management | People Management | Business Management | Information Design Models, Processes, and Techniques | Home
(c) Copyright. 1999, 2000, 2001, 2002. Saul Carliner. All rights reserved.
