Relying on “Smile Scores” To Measure Student Learning Is Not a Good Idea

A recent Academic Anonymous post in The Guardian about how student surveys are affecting a young professor’s confidence got me thinking.

Yes, we want students to enjoy our courses. And yes, we want students to find our instructional innovations engaging. But we can’t forget that students’ perceptions of enjoyment or engagement are not measures of instruction’s effectiveness.

I use the term “smile scores” for surveys that ask students how much they enjoyed a piece of instruction – as a reminder of what such surveys do and don’t measure. For example, student evaluations of teaching – a typical measure of students’ perceptions of a course – can be unrelated or even negatively correlated with the effectiveness of instruction. See two studies:

Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors and Student evaluations of teaching (mostly) do not measure teaching effectiveness.

Measuring students’ enjoyment is fairly straightforward with a quick-and-dirty survey, so it might be tempting to measure instructional effectiveness similarly – by simply asking students to rate it. However, research shows we have to be careful here. Students’ self-reports of their own knowledge, skill, or performance are often poorly calibrated. Hence, students’ assessments of their own learning gain and their judgments of the relative effectiveness of instruction also tend to be way off the mark. For example, a 2008 study investigated whether spacing is the “enemy of induction” for learning concepts and categories. The researchers let students experience two different ways to learn various art history styles, gave them a test on the styles, and then asked them which way was better for their learning. Interestingly, 78 percent of the students identified method A as better for learning, but in reality, 78 percent of the students actually did better under method B.

So, when you want a valid way to measure the effectiveness of instruction, don’t use smile scores and don’t just ask students for their opinions on the matter. Instead, consider what students should be able to do by the end of the instruction, and measure

that – ideally by giving students direct performance tasks to complete before and after the instruction. This approach will tell you the instruction’s effectiveness in terms of students’ learning – with a measure you can count on. Now, that’s something to smile about.

[Note: This is a cross-posting of a piece that I wrote for Huffington Post.]