The Problem with 5 Star Rating Methods
Posted on November 29th, 2011
There are all types of rating methods online today. One of the most popular is the 5 star rating method. You’ve seen it everywhere from Yelp, to iTunes, to Amazon, to Netflix. Unfortunately 5 star rating methods face many challenges in their use, and numerous studies have identified flaws with this particular methodology.
The graph below illustrates the main issue with the 5 star rating scale. Most of the ratings are distributed to the 1 and 5 star ratings. Most people will only comment if they really like or don’t like something. The additional options of 2, 3, and 4 stars are unnecessary. After learning this, YouTube moved to a thumbs up / thumbs down rating system. However, taking all of the options out might be a bit much. Having a third, middle (the ‘meh’ preference) option in there is good to balance things out.
The bigger issue with 5 star rating systems is that in almost every case the ratings get averaged. When you average 5 star ratings you get two big problems.
Problem 1:
The higher the average, the higher the item is listed in the scale.
For example:
Product A has 10 ratings, 9 of which are 5 stars and 1 is 1 star. Average rating of 4.6 stars.
Product B has 100 ratings, 85 of which are 5 stars and 15 of which are 1 star. Average rating of 4.4 stars.
Product A will be listed above Product B because it has a higher average. However, Product A has significantly fewer ratings and doesn’t necessarily deserve to be ranked above Product B.
Problem 2:
Averages can be misleading and distract you from important details within specific ratings. Sometimes, just 1 review matters. We can’t find a better way to illustrate this point than with this comic from XKCD. This is a comic pretending to display ratings for a ‘Tornado Guard’ app that will alert you when tornado’s are near. The app has an average rating of 4 starts, but really, it’s only the last review that really matters.
Often times we get asked if the Hively, 3 option rating method (Happy, Satisfied, Unhappy) can be replaced with a 5 star rating system. The reasons above are just a few of the reasons why we think the 3 rating options within Hively are better. The simple smiley faces are universal, easy to understand and keep you focused on each individual rating as they come in – not an average. Give Hively a try yourself. It’s free to sign up and easy to get started. Questions about Hively? Let us know.
View comments on this article
I don’t think a 3-level system addresses the main issues mentioned though; also, the middle option could be seen as unhelpful when trying to make a decision. Binary systems aren’t that terrible. I find that often when I check out reviews, I will read a few positive reviews, and then a couple more very negative reviews. The negative ones are usually the most helpful, because it’s easy to determine whether someone is raising actual concerns, or whether they’re just nagging about irrelevant stuff that doesn’t diminish your personal impression of the product. For the actual concerns, the number of reviewers who voice them gives an idea of how likely the issue is to be an actual problem. So Amazon’s method of featuring a good and a bad review at the top of the section correlates well with the system I’ve found most reliable.
Rating systems with 3, 5 or 10 gradations really only make sense if they’re elaborated on with a well-written, detailed, substantiated review that explains all the advantages and issues clearly, in the process relegating the rating to a mere summary. Because you can’t expect unmoderated public sections to produce useful reviews like that, I think the best move is to limit the masses to a binary “thumbs system,” because that’s generally as much as they can handle, and leave the key to more detailed rating systems only in the hands of trusted writers for moderated or editorial sections.
Thanks for the comment Daniel! I see what you’re saying and think you have some great points.
In your example though, I agree that a binary system might be better for a public forum, especially in terms of its usefulness for a reader of reviews. It seems most people read reviews as you’re suggesting – show me the good and show me the bad.
With Hively being an internally focused system though, the middle (or 3rd option) is useful to those receiving the feedback. Often times this rating type means the customer isn’t mad or upset, but they are reporting an issue they’d like to see resolved.
The following makes no sense to me at all.
“Product B has 100 ratings, 85 of which are 15 stars and 1 is 1 star”.
Shouldn’t there be a total of 100 ratings? (85 + 1 = 86).
And I thought we were working with a 5 star rating system not a 15 star one.
So if you do the math for the corrected version…
99*5=495
1*1=1
495+1 = 496
496/100 = 4.96
So the average rating actually goes up.