Issues to Consider When Collecting and Reporting Data

When technical communicators collect data about the effectiveness and value of communication products, we hope that it will benefit us. And, in the long run, doing so can build our credibility with clients by showing them that we have a well-considered methodology for assessing the effectiveness and value of our work.

But in the short run, merely collecting this information does not guarantee that others will accept it or that we will even be satisfied with the results. The appropriateness of the time and method in which we collect data determines its credibility to the clients to whom we report it. Also, the manner in which we report the data and how we respond to unexpected results are as important as our collecting the data. The next several sections explore these issues.

Top

Before, During, and After: When to Collect Data

If we are sincerely concerned about assessing effectiveness and value, we must recognize that assessment is a process, not an event (Geis & Smith, 1992). Assessment is an ongoing effort that starts before we even begin writing the plan for a communication product and continues long after we publish that communication product.

Before Development Begins

Before we begin a project, we should collect "before" measures of user and client performance. For example, if we are redesigning a form because the client believes that nobody can correctly fill it out, we should track how frequently users actually complete the form in error. Only if we have such a "before" snapshot can we compare readers’ performance with the revised form. And only if we have such a "before" snapshot do clients have tangible evidence of our ability to address their business needs.

During the Development Process: As we develop a communication product, we should do the following:

• Continue to collect data on existing situation (if a replacement product or comparison measure)
• Test the usability of the product to get an idea how well users will perform with the finished communication product and, if possible, what the business results are likely to be
• Manage the expectations (and perceptions) of clients by reporting the status of the project on an ongoing basis
• Evaluate the data we receive thus far and, in response, adjust plans for the communication product (and increase the likeliness of its effectiveness)

After We Publish the Communication Product

Complete the data collection process by:

• Conducting follow up assessments of user performance and client satisfaction
• Tracking business results
• Reporting results to clients

Top

Credibility: Issues in Collecting Data

Merely collecting data does not mean that it provides the answers you seek. For that, we need to ask the right questions of the right people. This is called "credibility." To ensure that our data is credible, we need to make sure that:

• The issues that we thought we were tracking we actually were tracking. That is, if we thought we were collecting data about customer satisfaction, we need to make sure that the questions truly request data about customer satisfaction. For example, suppose we asked users "were you able to find the information you needed?" to assess overall satisfaction. That question only asks users whether they were able to find the information they needed; it does not ask about satisfaction. If we used this question to assess user satisfaction, we could easily arrive at an incorrect conclusion. We could only conclude whether users could find the information they need, not their satisfaction with it.
• That the group from which we collected the data accurately represents the people we are trying to reach. For that to happen, we need a sufficiently large and diverse group of users. For example, if we were to ask the first 10 users of a new software application what they thought of it, and these people happened to work at companies that were selected for special tests of the software (called beta tests) during which they often receive special treatment, we might likely get a different response than if we were to ask a group of users who work for companies that were not included in the Beta tests. If responses came from a group that included users who participated in the test and those who did not in roughly the same percentages that would be found in the general population, then the data would more likely represent the opinions of the user base.
• That the data is properly analyzed. That is, did we use appropriate analysis techniques and draw reasonable conclusions? To make sure we properly analyzed the data, we need to address three issues.
• Did we use appropriate evaluation techniques? For example, we should use the appropriate statistical techniques given the sample size, the type of data, and the level of confidence we seek in the data.
• Did we properly identify the limitations of the data? For example, if we conduct a study on users in North American markets, do we identify this or to generalize the results to the African market?
• Did we draw reasonable conclusions from the data? For example, if we find that experienced users can complete a tutorial in 30 minutes, do we incorrectly conclude that all users can complete the tutorial in the same amount of time?

Choosing the right questions to ask and the right people to ask plays a significant role in collecting credible data. An astute client will ask you questions about both; you need to provide responses that assure your client that you have credible data. If in doubt, contact an experienced market or university researcher before you begin your evaluation program to assist you in formulating questions and choosing people to ask.

Top

Persuading Others: Issues in Presenting the Results

Because business accounting systems do not yet account for the types of benefits from communication products, the data we collect alone does not demonstrate bottom line effectiveness. Our skill at reporting the data is therefore as to building the perception of value as any data that we collect.

Six Ways to Effectively Present the Data

We need to take every opportunity to tell clients how we assess the effectiveness and value of our work and to report that effectiveness to them in a credible way. Specifically, we should consider these opportunities.

1. Information designs. As mentioned earlier, by including observable and measurable tasks and business objectives in our information plans, we set clients’ expectations of the results we plan to achieve.

By including complete evaluation plans, too, we inform clients how to assess the effectiveness of our work. Few of them have such tools at their disposal, by telling clients ourselves, we can set the rules ourselves. We should include drafts of proposed Reader’s Comment Forms and usability scenarios, and list business measurements to track.

2. Project status reports. When regularly distributed through the development process, such reports help us manage expectations in the same way that an effective and complete plan for a communication product helps us set expectations Specifically, the project status report indicates to clients that we can effectively assess editorial, production, and technical matters, alerts them to problems in advance (these, in turn, can affect clients’ perception of our service), and report results of usability tests that occur during the development process.
3. Post-mortems. A post-mortem following a project can not only identify the aspects of the project that did not meet expectations, but those that met or exceeded expectations. We can use feedback from such a session to determine how to more effectively meet and manage client expectations on future projects.
4. Annual reports. We can publish an annual report that tells clients and potential clients about users’ responses to communication products we developed and the business results we have helped clients achieve during the previous year. Not only is this a great means of reminding recent clients how we have helped them, but it helps manage the expectations of other clients.
5. Sales and publicity efforts. Whenever we prepare a brochure or web page about our capabilities or meet directly with a client, we should mention the results our communication products have achieved in with users and clients. More than the intentions, these results demonstrate how we have provided value for previous clients.

Also, when appropriate, provide reader satisfaction levels (if appropriate) and productivity levels (such as the number of pages or screens you produce a day on a given type of project). When asked in informal conversations, most technical communicators do not like to provide productivity levels, claiming such data is confidential. Perhaps we really do not provide productivity levels because we do not know them; many organizations collect productivity data but fail to evaluate it. We track time for the purposes of charging internal or external clients, but do not know how to evaluate this information. By providing clients with an idea of our productivity, we can set realistic expectations about the time and resources needed to complete a project.

6. Informal communications. We should never pass up an opportunity to politely tell a client how we have helped them or others. When doing so, we need to be careful about crossing the boundary between informing and bragging. For those wondering what that boundary is, we can always feel comfortable mentioning the subject when clients introduce it.

Whenever we provide information about the effectiveness and value of communication products, we need to recognize that each client assesses these issues in their own way. If we conduct a thorough evaluation, we have a complete picture and can provide clients with just the data they want. If we provide clients with other data, they will not necessarily be interested unless we preface it with a question like "Would you like to learn about other ways to assess the value of communication products?"

Responding to Unanticipated Results

Although we would like for all of the data we collect to support our belief that our communication products are effective and that we provide value to clients, we must vigilantly look for data that suggests otherwise. Although disappointing, such data can ultimately help us improve our operations if we correctly identify it and promptly respond to it.

Specifically, consider how to respond to these situations:

• Clients reject the data. If this happens, ask clients what data would be convincing to them and, if possible, provide it. For example, one organization provided training on rat reduction in an urban area. The trainers provided extensive data about the effectiveness of the course. Then a client asked how many rats had been eliminated. The trainers did not know because they had not collected that data (Donet, 1996). By failing to have alternative measurements, the trainers looked unprepared and unresponsive.

To avoid such a situation, we might include a sample report in the plan for a communication product to make sure that it provides clients with the information they find useful. If not, we can change the type of information we plan to evaluate before we have actually start collecting data.

• Someone else can provide the same services at a lower cost and a similar level of quality. In such an instance, we need to question our competitiveness in the markets we choose to serve. We could respond by choosing to serve other markets in which we are competitive or looking for ways to lower costs.

We might attempt to ignore this data or, worse, try in vain to find data to demonstrate that the competitors are unworthy. We shouldn’t. Ignoring the data could ultimately cost our companies their business. This data indicates that the competition has raised the standards of service in the field and our best response is raising our standards, too.

• People might lose their jobs. Although the ultimate purpose of collecting data on value is learning how we can be more effective in our jobs, we might also learn through the process of collecting data that some people are ineffective in their jobs.

We could continue to protect them, but, in an economy based on skills and productivity, doing so would not benefit either the worker or the client. The client receives substandard work. The worker might need to develop missing skills or perhaps consider a different line of work. A few moments of "pain" now might sting less than hours of pain later.

Top

Project Management | People Management | Business Management | Information Design Models, Processes, and Techniques | Home

(c) Copyright. 1999, 2000, 2001, 2002. Saul Carliner. All rights reserved.