How many people do you need to run a user test?
over 6 years ago from Randy Fisher, Design Partner
It depends on the goal of the test, but it's not a lot. Take a look...
over 6 years ago from Randy Fisher, Design Partner
It depends on the goal of the test, but it's not a lot. Take a look...
I follow NNG's approach https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
On my experience i can say that 6 people are enough. If you can have more people available you can divide people in bunch of 5-6 and make round of user tests trying the most critical parts learning from the previous tests.
More people you test with the same set of tasks more time the analysis part will take.
I use 5 as the magic number for qualitative product testing but some stuff I've been re-thinking.
Is 5 still the magic number when you're doing generative research? (i.e finding problems to solve)
Is 5 still the magic number when you have a diverse user group (i.e TV app users)
Is 5 still the right number when you're investigating motivation to do something rather than the ability to do it? (this is probably where quantitative/MVP is preferred)
A good answer I heard, is keep testing until you discover nothing surprising anymore
This is a pretty solid recommendation. For those who are more interested in the details, there is some nuance here.
In most cases, we as UX'ers are talking about Formative (what mistakes will users make) versus Summative (statistically, how many people will succeed).
As you point out, the general consensus is a population size of 5 will deliver the best cost to benefit ratio. There are two scenarios where that doesn't necessarily hold true.
The first, is if you only get one round of usability testing. While unfortunate, it happens–In that case, go for more as it's your only shot.
The second, is when the probability of finding a mistake is very low. The following chart shows the chances of detecting a problem given the sample size and probability of seeing event once (think of dice, d3 versus d20).
You can read the original magic number five article here if you want. Also, since everyone cites Nielsen I think it's prudent to point out that some of the credit should go to Bob Virzi and James Lewis who did a bunch of earlier work on this topic.
I'd be happy to go into the nuance of the topic of Summative sample sizes but that gets deep into statistics which is no fun to write up...
I prefer X people, where X = as many as possible.
5 people is fine for 'Usability Testing', but 'User Testing' is a different beast. You want to run as many tests with as many users as possible, in as many realistic user scenarios as possible, and continue to do so on an ongoing basis.
the more the better
Well, in my agency, we do a lot of user testing (I mean, there is rarely a project without user research and testing).
My recommendations for one-to-one interviews would be : 6 is the extra minimum, 8 is nice and 12 is actually the maximum. above 12, you will rarely find anything interesting or unheard before. most of the time I try to do around 6 to 8 people of the target audience.
of course, it depends of the maturity of the project and what you want to test. Is it the overall flow ? is it tiny graphical details ? or is it information architecture ?
I like the idea of Luca to make rounds of 5-6 testers, it should return interesting things. (Also, if you want to do A/B testing, as many users as possible would be a must !)
Designer News
Where the design community meets.
Designer News is a large, global community of people working or interested in design and technology.
Have feedback?
Login to Comment
You'll need to log in before you can leave a comment.
LoginRegister Today
New accounts can leave comments immediately, and gain full permissions after one week.
Register now