by fatherwags (Richard Wagner)

Did you ever wonder why book reviewers worth the ink and paper their reviews are published with don’t provide numerical scores for the books they review? Consider that question while I explain why game reviews are so laughingly flawed.

To be honest, I rarely buy a game without looking at a few reviews from sites where I’ve come to know the reviewers and their biases, and with which, in the past, my tastes and opinions seem to match pretty closely with the scores awarded. Of course, it has taken a considerable amount of time to work out a reliable system, but why should it be such a difficult task to figure out who to trust for information about a game experience and maybe a little bit of advice regarding which games might be worth my $60? Granted, it’s $60 Canadian, which is worth about $4.67 in U.S. dollars these days, but it still seems like a pretty good chunk of change to me.

There are all kinds of reasons why I distrust most reviews available in print or on the Internet, but the most significant problem isn’t journalistic or corporate favouritism, it’s the entire system underlying game reviews. From the relativity of numerical scales, to the presumption of objectivity, to the limited time and diversity of play experience to produce a relevant and timely review, these universal conditions make the act of reviewing games somewhat of an exercise in absurdity.

But aside from, or maybe in addition to, these kinds of obvious problems with game reviews, the real defect is actually in the assumptions that underlie the entire system. These are assumptions that no one ever really thinks or talks about, but which actually have a more significant influence on the validity and reliability of games reviews than some corporate trysts or poorly considered editorial policies.

The major flaw with all current review systems is that they try to assign numbers or, as is the case with the current 1Up network and former E.G.M. magazine, letters, to the complex and subjective experience of a playing a game. This is such a ubiquitous assumption about the best way to communicate evaluative information that most people don’t ever really consider how well a number actually represents a game experience.

Why, then, do people use numbers to rate games? For a number of reasons. One is that we are a culture produced by a long-standing belief in the reliability and impartiality of numbers. You can trace it back to the scientific revolution, when the worldviews of people started to shift away from a spiritual focus in favour of a scientific one. It’s a popular belief that science relies on numerical justification for theories and beliefs rather than emotions. Of course, if this were really the case the arguments between scientists wouldn’t be as heated or as prolonged as they often are. Scientists rely on emotion, intuition, and experience to help develop a hypothesis or theory, but in order for it to gain widespread acceptance, there has to be verifiable, objective, observable, repeatable data—often in the form of numbers—to validate the theory.

Forget that 92% of people over the age of 16 know that statistics are woefully untrustworthy and can be manipulated to say just about anything someone wants them to say; we still trust statistics when we see them because our culture is one that is obsessed with numbers; speed limits, prices, times—numbers rule our lives. They give game reviews an air of objectivity and invite trust, which is likely why they were adopted from film reviews in the early days of the videogame industry.

However, numbers aren’t as reliable as one might think. For example, in a recent edition of one prominent videogame magazine, none of the games reviewed were given less than five out of ten despite the reviews of numerous games lacking even a single positive comment. Numerically, a five out of ten is the middle, and would imply an average or adequate game, but when you look at the scores and compare those with the words, the reviewers were pretty clear that these were sub-standard, poor, and, in some cases terrible, games. Yet they still received a five out of ten. How then do you make sense of a numerical system where even obvious truths about numbers are ignored. If, then, a five out of ten is a horrible game, as was clearly the case in this magazine, what number represents an average game?

A number of months ago the 1Up Network decided to change to a letter grade score when reviewing games, in part, to avoid some of the problems with a numerical system, like the ambiguity of the numbers and the inevitably oversimplified and decontextualized use of game ratings on score aggregating websites. Even so, during a discussion about Mr. Lee’s score for 50 Cent: Blood on the Sand on a recent ListenUp! Podcast (February 27, 2009), Garnett Lee, John Davison, David Ellis, and Patrick Klepek, confronted the ambiguity of even letter grades. They couldn’t agree about whether a “C” or a “D” is a bare pass. They abandoned the discussion before confronting other questions that would have suggested how ridiculous a means for reviewing games any kind of grading scale really is, like if a “C” is a pass, what, then, is a “C-minus?” How much more enjoyable is a game that is given a “B-minus” instead of a “C-plus” or a 7.75 instead of a 7.50?

Anyone who is considering these kinds of fine distinctions regarding a score on a scale, and who is not also a teacher, is at least partly insane. It is the acme of lunacy to think that an entire experience can be distilled down to a letter or a number that will then have universal meaning for everyone.

Into this vortex of psychosis, I humbly suggest a saner and more useful system for reviewing games. My vision of an ideal game review is simple: words describe the experience, with reviewers clearly outlining the biases influencing, and limitations on, their knowledge and experience of the game.

First, aside from playing the game itself, words are the next best means of communication available to us when trying to evoke, capture, or express what it’s like to play a game. I don’t feel in numbers, nor do I think in numbers. I, like most people, think in images and words, so when a review uses images and words to share the feelings generated from playing a game, it’s far more effective at expressing the amount of enjoyment provided. If someone were to choose even a single word to review a game, how much more meaningful would that single word be than a single number? Imagine I told you that my review of Fallout 3 is that it’s “empowering.” Now compare that to me telling you that Fallout 3 is a 9. Imagine how lost you would be not knowing what that 9 is marked out of, or not knowing what other games I’ve given a 9 to, or not knowing how to convert that 9 from the metric to the Imperial system. Yet with the word “empowering,” you would know that it makes you feel powerful, which is something you can understand and use to decide whether that feeling is worth your money. The word has meaning but the number is meaningless. That’s not to say that knowing the other games I have called “empowering” wouldn’t also be helpful, but the word still captures an essential aspect of the game experience that “9” can never do.

Second, hearing from reviewers with varying levels of experience with the genre of the game being reviewed, with the franchise of the game being reviewed, and with games similar to the game being reviewed, would provide a broad range of experiences that would appeal to different types of players or consumers. Consider the conflict over the quality of the recent Halo Wars game. Experienced real-time-strategy (RTS) players and those who primarily play RTS’s on computer have complained that the game has been so dumbed down that it’s no longer fun or challenging. On the other hand, many people who haven’t played many RTS’s or who have only played them only on console rave about how much fun Halo Wars is because it’s so easy to pick up and play. Ideally, reviews would be selected so that they represent a variety of experience levels with a genre or a franchise so that people with similar experiences could better understand what they are likely to think about the game. If I’m a hardcore PC RTS player, I know I won’t like Halo Wars because the reviewer who has won StarCraft tournaments in South Korea says that it sucks; if I’ve never played an RTS before but I like the Halo franchise, I know I will likely enjoy the game because the reviewer who has only played first-person shooters enjoyed it.

Another reason the ideal game review would have multiple reviewers is because having more than one opinion provides the opportunity for a more balanced, complete, and accurate representation of a subjective experience. Instead of one person playing through a game with the sole responsibility of speaking authoritatively to everyone who will play the game, we need to learn a lesson from post-modern literary theory: Even if there is a single, absolute truth, we, as subjective and limited human beings, have no way to access it, so the best we can do is to recognize and consider multiple provisional, subjective, and limited truths in order to piece together some version of truth that we find acceptable. The more eyes looking at a game, the more hands playing it, the more minds writing about it, the more complete and accurate is the account of said game.

Third, instead of providing random and irrelevant information about the reviewers the way so many magazines and websites do (five favourite soup restaurants, games currently being played, hated things, etc.), provide significant and relevant information that will help us, as readers and consumers, evaluate the reliability and authority of the reviewers. For example, if a review is being posted for a Resident Evil (RE) game, it would be nice to know how many of the other games in the franchise the reviewers have played, and what they thought of them. It’s not that the perceptions of a rabid RE fanboy/girl are irrelevant and should be ignored when considering a review, it’s just that I would like to know what kinds of pre-existing attitudes and ideas may be at work behind the words that appear in the review. Or, if one of the reviewers of RE5 is a die-hard first-person shooter fan, that’s important information to have as I read the paragraphs ranting about the atrocious and archaic controls—especially as I compare that to another review, from someone who has never really played Western developed first-person shooters but has extensive experience with Japanese developed survival horror franchises, and who  describes the controls as solid.

Of course, not even this simple system is all that simple. The idea of organizations and writers being transparent with their readers about their biases is easier said than done. Removing a longstanding staple of film reviews, and subsequently of game reviews—numbers—would  be met with much anger, no matter how much more informative the resulting reviews might be. Not to mention that the costs of these kinds of reviews would make it prohibitive for print publications, particularly if game publishers respond negatively when they can no longer cite simple scores in the advertisements they place in those print publications.

As I mentioned at the beginning of this article, I refer to reviews from several websites and magazines, but that’s only in an attempt to re-create the kind of review system I’m suggesting here. Even when I do this patching together of reviews from various trusted sources, I’m still provided with a ridiculous number that is supposed to encapsulate and represent everything stated in a several hundred word review. As most intelligent game reviewers note, you can’t just look at the number – you need to read the words to understand why a particular score is given. But if that’s the case, if the number doesn’t adequately replace all of the words, then why use a number at all?

As ridiculous as the current review format is, there are reasons to be hopeful that the games industry can move beyond this bizarre obsession with numbers. With the increasing prominence of online reviews where more words don’t result in more cost; with an increasingly discerning, demanding, and mature audience; and with the rise of independent and enthusiast-driven websites, there exists the possibility for this kind of review format—one that more accurately expresses the relative quality of a given game for a variety of different types of gamers by using words to describe its quality, rather than a number.

Returning to the initial question I asked: “Did you ever wonder why book reviewers worth the ink and paper their reviews are published with don’t provide numerical scores for the books they review?” The answer is simple: people who understand art and literature understand that numbers don’t adequately capture a human experience, and when attempting to express the emotional response to, or the entertainment value of, a piece of art, words are the most effective method of communication we have. That’s why they use words to communicate ideas and experiences rather than numbers; that’s why writers of literature do the same. The borrowing of a convention initially and misguidedly adopted by film critics, and the inability to recognize a most basic truth—that numbers are an utterly inadequate means of expressing human experience—are why game reviews are so laughingly flawed despite the emphasis placed upon them within the videogame industry.