Survey Design

Last updated
Save as PDF

Level 1 Evaluations

Kirkpatrick Level 1 evaluations, commonly known as "smile sheets," play a crucial role in assessing participants' reactions to their learning experiences. When constructing an effective evaluation, it's essential to diversify the questions asked. This diversity should encompass both question types and categories.

Incorporating a range of question formats, such as Likert scale queries, open-ended prompts, and multiple-choice items, serves several purposes:

Mitigating Response Biases: by employing different question styles, we minimize the risk of response biases, thereby obtaining more impartial and accurate participant feedback.
Offering a Holistic Perspective: diverse question types provide a more comprehensive understanding of participant reactions, enabling us to capture both positive and negative feedback.
Spotlighting Areas for Enhancement: these varied questions help identify specific areas for improvement within the learning experience.
Enhancing Participant Engagement: by diversifying the survey format, we keep participants engaged and reduce survey fatigue, resulting in more meaningful responses.

Below are sample questions you might consider incorporating into your evaluation:

Category	Sample Questions
Overall Satisfaction	Rate the overall value of the learning experience. Overall, the workshop was a valuable experience for me.
Intent to Apply (i.e., scrap learning)	Indicate the likelihood that you will use what you learned in the next month. I will be able to apply what I learned in the next two weeks.
Net Promoter Score	How likely are you to recommend this webinar to a peer? I am likely to recommend this workshop to my peers.
Impact	After completing this webinar, rate your confidence level in implementing the concepts that were discussed and/or using the tools that were described. As a result of this workshop, I expect to be more effective with my admissions process. This training has a high impact on the ability to perform my job.
Content Relevance	What content was most valuable to you? What did you find most valuable about this training? What content should be added? What changes would you make to improve this training? The majority of the material covered in this workshop was important/relevant to me.
Delivery	The trainer(s) provided feedback on the mastery of the learning outcomes. I am satisfied with the trainer(s) who delivered this workshop. The classroom environment was conducive to learning. How can we be more effective? The pace of the workshop was just right. The workshop's stated objectives were met.
General Comments and Suggestions	Please provide any additional feedback. Additional Comments: Is there anything else you'd like to share with us?

General Guidelines: Writing Survey Questions

When writing survey questions, consider the following:

Because words influence how people perceive things, create a numeric scale and anchor only the ends with words. For example, a Likert scale of 1-7 should only have “1 = strongly disagree” and “7 = strongly agree.” This allows the respondent to interpret the scale as opposed to the writer providing definitions.
Use odd-numbered scales. 7-, 9-, and 11-point scales are best because they provide the respondent with more choices and allow them to pick a neutral response (4, 5, and 6) where necessary. Use an even-numbered scale when you want to force respondents “off the fence.”
Keep question sections (i.e., sets) unlabeled; labels can introduce bias (depending on the survey).
Word about 1/3 of survey questions so that the desired answer is negative. This can create more valid data responses because it forces the respondent to pay attention to each question instead of just marking everything as “strongly agree.”
As a best practice, write all items positively and then find the easiest ones to turn negative. For example, turn “My manager gets to know people” into “My manager doesn’t get to know people.”
Focus on observable behavior; if you can’t see it, don’t ask. For example, the Likert statement “My manager considers whether or not they are knowledgeable about their job(s)” does not account for observable behavior (i.e., “knowledgeable about their job(s)” is not observable).
Design sections to contain a similar number of items and questions that contain a similar number of words; keep comparable sizes close together throughout the survey.
Limit items to a single description of behavior. For example, the Likert statement “My manager provides good feedback and avoids getting emotional” contains two behaviors. Depending on how the reader interprets the questions, it could introduce bias.
Keep demographic information to the end.
Remove the highest and lowest scores from each question/data set when tabulating results.