Stephan Sokolow's Blog

After writing my previous blog post about calculating an average rating properly, it occurred to me that I’ve never really explained my views on rating systems and their relative merits.

Now, first, let me say that I won’t be getting into the whole objective vs. subject voting mess. (The “It’s a classic and a masterpiece, but I don’t want it showing up in my recommendations” problem.) That’s its own mess. This post is purely about using good UI design to encourage consistent input across a large population of users aside from that issue. That said, let’s get started.

These days, it seems like everyone either has or is planning to move to a simple up/down rating system. YouTube is probably the classic example. My understanding is that they moved from a star rating system to a like/dislike rating system because the former didn’t work out well with the general public. (Some people almost never give out high ratings, some people almost never give out low ones, some people found it too much effort to decide, and the result is a mess, mathematically.)

The fundamental problem with cutting things down to such a two-value system is that it fundamentally doesn’t give you much data to work with and it’s my hypothesis that it also encourages noisy, polarized data above and beyond the obvious, since people don’t typically take the time to rationally analyze their impression of middling content. (In essence, it’s throwing the baby out with the bath water.)

So, if we can’t reduce the number of choices users have by that much, how can we tweak the presentation so that users respond more consistently?

First, let’s look at where star ratings actually come from. According to Wikipedia’s Star (classification) page, the first instance of repeated symbols for ratings was in an 1820 guidebook by Mariana Stark, which used repeated exclamation points. Following that, Murray’s Handbooks for Travellers and the Baedeker Guides replaced the exclamation points with stars.

There’s something very important to notice here. Star ratings originated in the context of highlights! They wouldn’t give some semi-permanent pile of horse manure in a random London back alley a zero-star rating… they’d just omit it!

So, our first step should be to acknowledge that mismatch and bring ourselves into line with human psychology. When we have a low opinion of something, we don’t stop at zero. We use colourful words such as “hate”, “loathe”, “despise”, and “detest”, which express negative emotional value …so, let’s make zero the middle of our scale.

Notice something familiar about that change? We’ve reached upvote/downvote. It’s just a very primitive form of what we’re seeking. People may not agree on how much room to leave on each end of a scale for especially good or bad content, but everyone understands the meaning of a transition point between “like” and “dislike”.

So, what’s next? Well, how about ambivalence. In an upvote/downvote system, users are forced to take sides. There’s no way to express “I don’t really care”, “I have no opinion”, or “Its only noteworthy characteristic is how un-noteworthy it is.” …so we at least want three choices:

Thumbs up from Font Awesome by Dave Gandy.
Modified under a CC-BY-SA 3.0 Unported license.

I’m not a graphic designer, but you get the idea. With the addition of a middle choice, it already represents real human opinions much more effectively.

This is actually the bare minimum I consider for a viable system and, when forced to deal with like/dislike systems I resort to using “abstained from voting” as a means of expressing a third value… and that in itself is a clue.

I coloured the middle values yellow to make them distinct from “unset”, but is that really necessary? What’s the difference between “I viewed it and didn’t vote” and “I viewed it and selected the neutral option”? I’d argue that drawing such a distinction is counter-productive hair-splitting, so I’ll use grey for the neutral option going forward.

So, what’s next? Well, how about degree? Humans aren’t stupid, so I’m not willing to give up on a 5-step rating system yet. If we’ve got a clear and obvious meaning for the middle point, it’s not hard for people to consistently answer the question “Did you like/dislike it a little or a lot?”, so let’s put two choices for like and two for dislike.

Thumbs up from Font Awesome by Dave Gandy.
Modified under a CC-BY-SA 3.0 Unported license.

Using color/brightness on hover and/or selection to “light up” the icons spanning from the selected one to the center can reinforce the understanding that this is a “distance from the center” metric, but having everything grey exacerbates a problem that was growing with the face-based approach.

While a soft smile or frown can serve as a general “like” or “dislike” icon, one can dislike something for many reasons. Emotions like anger, disgust, and extreme sadness all have their own distinct facial expressions. It’s easy to mistake the face-based visualization as a request for a qualitative evaluation of one’s emotional state (anger vs. sadness) rather than a quantitative one.

Furthermore, faces are complex shapes which can be difficult to pick details out of at small sizes. What we need is a set of symbols which are generic, international, and scale well.

Historically, I wouldn’t recommend thumbs, since the meanings of gestures vary so widely around the world, but the thumbs up and down icons seem to have taken on enough of an international meaning online to keep them in the running.

…so what’s the alternative choice? Well, how about plus and minus symbols? Math is international and everyone can understand their meanings in context.

Thumbs up from Font Awesome by Dave Gandy.
Modified under a CC-BY-SA 3.0 Unported license.

Doesn’t that look a lot easier to translate opinions into than a simple row of five stars? …and if not, tooltips can give that little extra boost.

Thumbs up from Font Awesome by Dave Gandy.
Modified under a CC-BY-SA 3.0 Unported license.

(And there also seems to be support for this model from the experts whose salaries depend on doing this sort of thing. Every professionally administered survey I’ve ever taken has incorporated questions with the choices “Strongly Disagree”, “Disagree”, “Neither Agree Nor Disagree”, “Agree”, and “Strongly Agree”.)

In summary, don’t be too quick to sacrifice data and throw out a UI that isn’t working. Sometimes, all it needs is a little tweak.

Bonus Tip: Extra Precision

Let’s suppose that you’re trying to upgrade a system that uses out-of-10 rating or you need to serve a more experienced user base like me (who sometimes feel the need to rate something as being “great” rather than “good” or “excellent”). There is also a way to support this without falling back to the “5 stars to the right of zero” problems that started this whole mess.

The secret ingredient is decimals. Users will have a much easier time if you draw a distinction between normal (integer) and exceptional (decimal) rating precision. In fact, “Rate in the range from -2 to +2… use decimals if you need to” is not only easier than “Rate in the range from 1 to 10”, it’s also more powerful since, if necessary, it can be extended to however much or little decimal precision you need.

In my experience though, it’s so rare for me to desire precision beyond “-2 to +2 in steps of 0.5” than I wouldn’t be concerned with it.

So, how do we produce a UI for this? Well, think about the psychological use of decimals. They’re extra precision that’s not normally needed, so they should be out of the main workflow where users don’t have to fret over them.

There are various ways to accomplish this , but the simplest way to visualize them would be a design inspired by the keys on a piano. Play all the white keys, and you can make perfectly good music in the key of C major… but the black keys are there when you want to do something more advanced.

Thumbs up from Font Awesome by Dave Gandy.
Modified under a CC-BY-SA 3.0 Unported license.

…keeping in mind, of course, that, for mobile use, it’d probably be best to hide the half-step buttons and present some kind of alternative method for entering high-precision information, such as a hamburger button with a popup.

(As zero-centered designs lend themselves best to odd numbers of choices, you’ll have to decide whether the tenth choice should be omitted or added onto either end on a case-by-case basis.)

TL;DR: Read Bayesian ranking of items with up and downvotes or 5 star ratings by Jules Jacobs. I’ll get back to you on how to tune the priors and utility values.

Years ago, when I first had the idea that prompted me to register ficfan.org as a placeholder page, I went looking for a mathematical solution to the problem of bias in story ratings built from very small samples.

Given that this explanation of Wilson scores brought it back to mind, I thought I might as well blog about the topic for anyone who wants to know how to calculate average ratings properly.

I’d recently finished a very basic statistics course and, when I stumbled across How Not To Sort By Average Rating by Evan Miller, I recognized the principle of using the lower bound of a confidence interval, but didn’t know how to generalize it to the multivariate data that is an out-of-5 rating and didn’t find any candidates that were as well-suited to use by a stats novice as Miller’s post.

Since then, that very question got asked on Cross Validated (the StackExchange site for statistics), and a good point was made: The “lower bound of a confidence interval” method will seriously under-estimate things with a low number of ratings.

raegtin gives a good theoretical answer for methods of resolving that problem, but I didn’t yet have the much better stats textbook that’s now in my TODO pile and wasn’t in the mood to soldier through the theory on my own, so I kept looking.

Not longer after that answer, Evan Miller came back with a less hacky solution for up/down ratings which makes good reading if you’re trying to learn the theory without a textbook… but still didn’t meet my “I don’t trust my math. Give me something someone else trusts.” needs.

Now, that said, some people do apply the Wilson score to multi-value data by scaling the range down to between 0 and 1, so a middling rating counts as half an up vote. That’s what this MySQL solution and this Node.JS module do) but it has a big flaw. As Apocalisp pointed out when someone else thought of the idea, it leaves you with 300 3-star ratings being equivalent to 100 5-star ratings. He suggests calculating a Wilson confidence interval for each possible score, then working from there (and I’ve seen it suggested elsewhere), but I wouldn’t feel comfortable with that even if his suggestion had been around earlier, because I couldn’t find a detailed breakdown of why it would produce good results.

Ironically, the oldest resource (What is a better way to sort by a 5 star rating? on StackOverflow) is one I found just recently (Google Fu failure!). Despite that, it’s probably the most useful of the StackOverflow answers.

That said, let’s get on to the aforementioned resources it links.

In 2014, Evan Miller came back with “Ranking Items With Star Ratings“, which is a detailed look at the problem and how to solve it… unfortunately, I was sleep deprived when I encountered it and the massive wall of equations which didn’t end in sample code prompted me to shelve it for later. (Ironically, I can’t evaluate it right now either, because today is the one day this week that I slept terribly and I’m too busy to risk delaying this post until I have time.)

Finally, I came across “Bayesian ranking of items with up and downvotes or 5 star ratings” by Jules Jacobs (written in response to Evan Miller’s improved upvote/downvote code) which is in a form my sleep-fogged brain can handle.

It’s a simple, easy-to-understand explanation for people with minimal background in statistics, it guides you through thinking at the problem from the right direction (eg. what does the utility function really mean?), and comes with Python example code for if you really can’t be bothered to do anything more than copy-paste code.

As for making it efficient, the fact that it’s a modified arithmetic mean allows us to calculate it incrementally as long as we store both the score and the total number of votes with enough precision that we don’t have to worry about rounding errors.

1. Multiply the average by the total vote count to reverse the final step of the process and produce what I’ll call the “expanded average”.
2. Perform the weighting calculations for the new value
3. Add it to the expanded average.
4. Divide the expanded average by the new total number of votes.

This works because of two properties:

• Addition is commutative, so it doesn’t matter which order you sum together the individual ratings.
• The division and multiplication are symmetric, so 1 + 2+ 3 + 4 + 5 and ((1 + 2 + 3 + 4) / 4 * 4) + 5 are mathematically equivalent.

It’s not perfect, since you don’t have a nice averaged number in the range from 0 to 5 to display, but it’s definitely a good start if sorting and graphical visualizations are your goal. (Evan Miller’s approach is probably best if you need to display numbers.)

That leaves only one question: How do you tune your priors and utilities? …I’ll get back to you on that one after I have time to research it.