I posted earlier today on Facebook regarding the topic of ratings collection design systems for beer ratings websites like BrewAdvocate (below, referred to as “BA”), based on my experience designing grading rubrics for teaching and student evaluation purposes. I was posting in the context of others arguing that the ratings systems on various enthusiast rating websites were relatively less meaningful in the fine gradations; that a been with a 97% quality rating may not be “better” than a beer of a 95% rating. I responded as follows:
Yeah, also because people’s rankings are affected by others’ rankings, and by reputation. I can’t remember whether the review/ranking page sits with other rankings available for viewing, but that’s your first sign of an influence leakage. (IIRC BA is that way.) It’d be better if ratings were invisible on the review/ranking page, until after you’ve posted your own. But people probably wouldn’t like it.
Also, the percentage is a ridiculously fine-grained scale for this kind of evaluation. You learn that, teaching. You learn that a grade out of 100 is often too distracting and fine-grained for a student to handle, but also allows too much subjectivity in the evaluation on your side, as a teacher. A+ to F is handy because it outlines very specific value levels, and problem ranges, and you can look at an essay or assignment and clearly slot it into one of those given the quality of the work, as well as it being directly communicative to the student about how well the work met expectations.
Or, you know, there’s pass-fail, or like we had for our public speaking grad requirement: High Pass/Low Pass/Fail. That’s not fine-grained enough for beer evaluations. But it’s better than a 100-point ranking system.
For beer, I think a ten point scale would be more sensible. You’d find people reserving 10 for the life-changing beers, using 9 and 8 for solid, good beers, 7-6 for okay ones, and 5 and below for the crap. It’d be communicative, and less susceptible to the problems.
But of course, we’re talking about the usefulness of the rankings in terms of gauging feedback. Meanwhile, those sites thrive on more social uses of ranking, especially the identity of those doing the ranking. The unnecessarily fine gradations of rankings aren’t useful for beer rating, but they are for people who want to feel like experts and show off what they’ve gotten to tasting.
I would submit this is also precisely why the percentage system is so deeply embedded in formalized education, though it’s a real irony: Continue reading