You are here: Ideas and Issues in E-Learning --> Supplemental Resources for the Book Designing E-Learning --> Chapter 7 --> How to Conduct a Usability Test of the Prototype --> General Procedure for a Usability Test

General Procedure for a Usability Test (Part 2 of 3)

Although the course development team does not test its own work, you should be aware of what happens in a usability test. Usability engineers do the following to prepare for a usability test:

1. Write a test plan. At this time, usability experts do the following:
• Identify the types of, and numbers of, learners (called users in these tests) needed to participate in the test, based on the descriptions of the learners you identified when you conducted the needs analysis.
• Determine whether to conduct the usability test in a usability laboratory or in the field; that is, a real business setting.
• A usability laboratory is a facility that has two main rooms. In one room, users perform their tasks. The room is outfitted with a one-way glass window, video cameras, and microphones. The other room is an observation room, where observers can view users through the one-way glass window, as well as on the video monitor and through the loudspeakers. The observation room usually has video recording equipment, to capture the images of the users in the test, and a large writing table behind the glass window, where an observer can record the users’ actions on an observation sheet or on a computer. Some organizations have their own usability labs, others rent lab space on an as-needed basis. The cost of these facilities should be considered when planning the budget for a project.

The lab provides excellent conditions for observing users, but the conditions under which people use the information product are fully controlled and might not accurately reflect real-world use.

• The field refers to a work setting where the usability test would take place. When conducting tests in the field, usability engineers often outfit the location with video and audio recording equipment to capture actual use of the learning program. Usability engineers might also set up an observation station, from which they can record learners’ actions.

Although the recording equipment is obtrusive, the real world setting gives usability engineers a more realistic insight into how users work with the communication product than might be possible in the contrived situation of the laboratory. Conducting a field test involves receiving permission to use a facility, and travel, costs that must be considered when planning a budget.

• Identify and obtain resources needed to conduct the test. For example, suppose that the online learning program includes an always-available tutor. The usability engineer must arrange for the availability of a tutor during the test.
• Prepare instructions to test subjects. You might have to create accounts and passwords so learners can get into the prototype materials, instructions for starting the online learning, and similar types of instructions.
2. Conduct a trial test. The trial test is primarily intended to make sure that the test works as intended: that the learners understand the test instructions, that all of the information and supplies needed to conduct the test are available, that the test facility (lab or field) actually provides the usability engineer an opportunity to get the intended data, and that test observers have the tools they need to conduct their work.
3. Revise the test. In response to comments from the trial test, make adjustments to the actual test plan. For example, users do not understand the test instructions, the usability engineer rewrites them. If resources are missing during the test, the usability engineer arranges for them to be available.

After the test has been tested and revised, usability engineers conduct the actual test. To do so, they:

1. Schedule the facilities. Usability labs must be scheduled well in advance for use.
2. Schedule people to participate in the test. Because tests are time consuming to observer, a typical usability test has between 3 and 7 participants. Because usability engineers carefully choose participants to reflect the full range of experience of the intended learners, the results of just a few users like this provide a full range of user feedback on the draft.

Usability experts find people in a variety of ways, including:

• Asking the sponsor to identify people who might be interested in participating in the test, then contact the customers to provide them with descriptions of the types of people whom the usability engineers plan to include in the test. If using people outside of your organization as test subjects, the usability engineer typically compensates them, either by paying for the learners’ time or by providing free copies of the learning program after its publication.
• Contacting temporary services to hire people to participate in the test. The temporary services hire people who fit the demographic profile of the intended users.
3. Brief the observers. Observers are people who record each actions of users as they work through the test scenario. Observers are often people who work for the same organization as you but are assigned to other projects and have some spare time while their own projects are out for review.
4. Conduct the test:
1. Ask users to sign non-disclosure agreements. Much of the product and technical information covered by a usability test is confidential; it addresses products that are not yet available to the public. The sponsor does not want information about these products to be leaked before announcement; it could easily kill a competitive advantage the sponsor. The non-disclosure agreement prevents usability test participants from discussing the test or the products used in it.
2. Brief the participants in the test. The extent of the briefing depends on the nature of the test. In some instances, usability engineers tell users the full purpose of the test, in others they do not.
In all instances, however, usability engineers advise test participants that the test is intended to test the learning program, it is not intended to test them. If learners experience frustration with a passage or test question, the assumed culprit is the learning program. Many learners might be apprehensive about using a new course; this advice is intended to put them at ease and help them retain their self-esteem, even if they cannot follow the instructions. The usability engineer also informs participants that the test is being recorded. This is a legal requirement.
3. The usability engineer starts the program, advising participants to speak all of their thoughts aloud as they work. By speaking all of their thoughts, called a speak-aloud protocol, observers can get an insight into the thoughts of the users¾what they liked, what they found frustrating, what they had no problems with.
4. The observer begins recording the following on an observation form, which might be a separate sheet of paper or might be an online form. The observation form requests the following data:
• Participant name (generally, observers only know the first names of participants or a participant number)
• Time the participant started the task
• Time the participant completed the task
• For each action taken (such as entering information), the observer records:
• The time of the action
• Description of the action taken by the participant
• Comments, such as repeating words that the participants said
• “Flag,” to indicate a problem that the usability engineer should consider when reviewing the results

Figure 7W-1 shows an example of an observation form.

User _____________________________________________________ Date _____________________________________________________ Time Started: ____________ Time Completed: ____________ Total number of errors: _______________________________________ Time Action Comments ________ __________________ __________________ ________ __________________ __________________ ________ __________________ __________________

Figure 7W-1: Example of an Observation Form from a Usability Test

5. After each participant finish the prototype unit, the observer or the usability engineer debriefs the participant. In this open-ended interview, the observer or usability engineer ask participants how they felt about the online learning experience in general and this prototype learning product in particular. The observer or usability engineer also asks participants to explain actions that seemed unclear during the observation and to share their frustrations and pleasures with actual actions and tasks.

Although you cannot be a formal observer for the test, most instructional designers and developers like to be “passive” observers of the usability tests of their work. As a passive observer, you do not play a formal role in the test, but you do observe users. Many designers and developers comment that nothing tells them more about making a communication product usable than the experience of actually observing the test.

The discussion continues: How to Respond to the Results of a Usability Test

General Resources on Designing E-Learning | Supplemental Resources for the book Designing E-Learning | Chapter 7 | Links | Glossary
(c) Copyright. 2002. Saul Carliner. All rights reserved.