Qualitative Versus Quantitative Usability Testing

Often the question is asked, what is the optimal number of participants to use in a usability (user experience test). Jakob Nielsen (left), the Godfather of Usability, says 20 if you want a statistical confidence interval of +/- 19%, which he considers "ample" for a quantitative test.

"Luckily, you don't have to measure usability to improve it. Usually, it's enough to test with a handful of users and revise the design in the direction indicated by a qualitative analysis of their behavior. When you see several people being stumped by the same design element, you don't really need to know how much the users are being delayed. If it's hurting users, change it or get rid of it.

You can usually run a qualitative study with 5 users, so quantitative studies are about 4 times as expensive. Furthermore, it's easy to get a quantitative study wrong and end up with misleading data. When you collect numbers instead of insights, everything must be exactly right, or you might as well not do the study."

At my usability lab in the 1990's, we preferred to use two of three studies of five each, rather than one study of 15. We staged the studies before and after review/revision cycles so we could see if we had fixed the problems. For the complete Nielsen article, click on the title above. David Orr