You are here: Ideas and Issues in E-Learning --> Supplemental Resources for the Book Designing E-Learning --> Chapter 4 --> Tips for Conducting Level 1 Evaluations of Asynchronous e-Courses

Originally appeared in the ASTD Newsletter, What Works, March 2002.

Tips for Conducting Level 1 Evaluations of Asynchronous e-Courses

What do learners think of your e-learning courses?

That’s a good question because one of the most vexing challenges in e-learning is getting feedback on satisfaction from learners. (For those of you who are familiar with Donald Kirkpatrick's four-level evaluation of training programs, this is Level 1--Reaction.)

The challenge is two-pronged: encouraging learners to respond and determining what to ask.

Encouraging Learners to Respond

Part of the challenge in getting learners to respond is admittedly environmental. Because instructors control the environment and can place subtle pressure on students to complete the survey, high response rates are relatively easy to obtain.

E-learning instructors lack such direct control over learners-especially learners taking asynchronous courses-and need to work harder to encourage students to complete Level 1 reaction forms.

Some techniques for doing so:

Choosing Questions to Ask

The other issue in Level 1 evaluation for e-learning is asking relevant questions. Following is an annotated sample of a survey that asks questions appropriate to the e-learning environment.

1. In a word, how would you describe this course: ___________________

Logic behind the question: Solicits qualitative feedback as well as assess whether students accidentally transposed the numerical scales (if a student responses “Excellent” to this question, then circles “1” to the next question, chances are the student misread the number system.

2. Using a number, how would you describe this course:

1	2	3	4	5
Abysmal		Average		Outstanding

Logic behind the question: Provides quantitatively-oriented organizations with the numbers they seek.

3. How much did you know about SUBJECT NAME before taking this course:

1	2	3	4	5
Not at all		Some		A lot

after:

1	2	3	4	5
Not at all		Some		A lot

Logic behind the question: Although this question does not assess actual learning, it does assess perceived learning. That is, do learners perceive that they learned something in the e-learning program.

4. How likely are you to use the information taught in this course on the job?

1	2	3	4	5
Not at all		Not sure		Very likely

Logic behind the question: e-learning promises to provide just-in-time skill development for the job, one assessment of its effectiveness is whether learners perceive that they can apply these skills on the job.

5. Describe the experience of taking this course online:

I have taken online courses before (circle one) Yes No

Rate the quality of:

Content 1 Thrown together 2 3 OK 4 5 Superb Video (Note: only ask if relevant) 1 Barely visible 2 3 OK 4 5 Crystal clear Audio (Note: only ask if relevant) 1 Barely audible 2 3 OK 4 5 Crystal clear

Rate the usefulness of:

Exercises 1 A waste 2 3 OK 4 5 Insightful Linked websites 1 Distracting 2 3 OK 4 5 Enlightening Glossary 1 Distracting 2 3 OK 4 5 Enlightening

Rate the availability of:

Course 1 Perpetually down 2 3 OK 4 5 Always available Instructor (for the purpose of contacting by -mail) 1 Missing in action 2 3 There 4 5 Dependable

Overall, taking the course was:

Convenient 1 No way 2 3 I guess so 4 5 Totally Compared to a classroom course 1 Much worse 2 3 The same 4 5 Much better I would take another online course 1 I would rather have surgery without an anesthetic 2 3 If I must 4 5 In a heartbeat

Logic behind the question: Ask about various aspects of the course that are important to you, just as you might ask questions about the appearance of the handouts, convenience of the location, and quality of the facilities in a Level 1 evaluation of a classroom course.

6. The best part of this course was: ______________________­_______

____________________________________________________

____________________________________________________

7. This one thing that could improve this course most is: ____________

____________________________________________________

____________________________________________________

Logic behind the questions: The last two questions provide qualitative feedback on the course and help prioritize work in a revision. Items mentioned on several responses warrant attention. For example, if 10 learners respond that the exercises are excellent, you should probably incorporate exercises like these in similar courses in the future. Similarly, if 18 learners comment that the pace of the course is slow, the pace should be quickened in the next revision.

General Resources on Designing E-Learning | Supplemental Resources for the book Designing E-Learning | Chapter 11 | Links | Glossary
(c) Copyright. 2002. Saul Carliner. All rights reserved.