Orignally appeared in the American Society for Training and Development publication, What Works, March 2002.

Tips for Conducting Level 1 Evaluations of Asynchronous e-Courses

What do learners think of your e-learning courses?

That’s a good question because one of the most vexing challenges in e-learning is getting Level 1 feedback (reaction) from learners. The challenge is two-pronged: encouraging learners to respond and determining what to ask.

Top

Encouraging Learners to Respond

Because the response rate to Level 1 surveys in classroom courses approaches 100 percent, the low response to Level 1 surveys in e-learning courses (sometimes less than 1 percent) is especially troublesome.

Part of the challenge is environmental. Because instructors control the environment and can place subtle pressure on students to complete the survey, high response rates are relatively easy to obtain.

E-learning instructors lack such direct control over learners-especially learners taking asynchronous courses-and need to work harder to encourage students to complete Level 1 reaction forms.

Some techniques for doing so:

When learners complete a course, display a pop-up that says “So we can provide you and future learners with the best possible learning experience, please take a few moments to complete the post-class survey.”

Immediately below, display buttons that say Take the Survey, Remind Me in a Day, and Exit. Those that choose “remind me in a day” should be sent a reminder e-mail, with a direct link to the survey (or the survey embedded in the e-mail message).

Rather than distribute the survey before learners exit a course, send a separate e-mail to each learner immediately after they complete the course. Ask them to respond, and offer an opportunity to receive a premium (such as a gift certificate for a latte or a souvenir tee-shirt) as an incentive to do so within 3 days.

Top

Choosing Questions to Ask

The other issue in Level 1 evaluation for e-learning is asking relevant questions. Following is an annotated sample of a survey that asks questions appropriate to the e-learning environment.

1. In a word, how would you describe this course: ___________________

Logic behind the question: Solicits qualitative feedback as well as assess whether students accidentally transposed the numerical scales (if a student responses “Excellent” to this question, then circles “1” to the next question, chances are the student misread the number system.

2. Using a number, how would you describe this course:

1	2	3	4	5
Abysmal		Average		Outstanding

Logic behind the question: Provides quantitatively-oriented organizations with the numbers they seek.

3. How much did you know about SUBJECT NAME before taking this course:

1	2	3	4	5
Nothing at all		Some		A lot

After

1	2	3	4	5
Nothing at all		Some		A lot

Logic behind the question: Although this question does not assess actual learning, it does assess perceived learning. That is, do learners perceive that they learned something in the e-learning program.

4. How likely are you to use the information taught in this course on the job?

1	2	3	4	5
Not at all		Not sure		Very likely

Logic behind the question: e-learning promises to provide just-in-time skill development for the job, one assessment of its effectiveness is whether learners perceive that they can apply these skills on the job.

5. Describe the experience of taking this course online:

I have taken online courses before (circle one)	Yes No
Rate the quality of the:
Content	1 2 3 4 5 Thrown together OK Superb
Video (Note: only ask if relevant)	1 2 3 4 5 Barely audible OK Crystal clear
Audio (Note: only ask if relevant)	1 2 3 4 5 Barely audible OK Crystal clear
Rate the usefulness of the:
Exercises	1 2 3 4 5 A waste OK Insightful
Linked websites	1 2 3 4 5 Distracting OK Enlightening
Glossary	1 2 3 4 5 Distracting OK Enlightening
Rate the availability of the:
Course	1 2 3 4 5 Perpetually down OK Always available
Instructor (for the purpose of contacting by -mail)	1 2 3 4 5 Missing in action There Dependable
Overall, taking this course was
Convenient	1 2 3 4 5 No way I guess so Totally
Compared to a classroom course	1 2 3 4 5 Much worse The same Much better
I’d take another course online	1 2 3 4 5 I'd rather have surgery without an anesthetic If I must In a heartbeat

Logic behind the question: Ask about various aspects of the course that are important to you, just as you might ask questions about the appearance of the handouts, convenience of the location, and quality of the facilities in a Level 1 evaluation of a classroom course.

6. The best part of this course was: ______________________­_______

____________________________________________________

____________________________________________________

7. This one thing that could improve this course most is: ____________

____________________________________________________

____________________________________________________

Logic behind the question: The last two questions provide qualitative feedback on the course and help prioritize work in a revision. Items mentioned on several responses warrant attention. For example, if 10 learners respond that the exercises are excellent, you should probably incorporate exercises like these in similar courses in the future. Similarly, if 18 learners comment that the pace of the course is slow, the pace should be quickened in the next revision.

Top

Home

other resource websites by saul carliner
models, processes, and techniques of information design | the commerce of content |

(c) Copyright. 2001, 2002. Saul Carliner. All rights reserved.