Thunderbolt logo

What’s in a Number?

Scoring

A job, apparently. Or more specifically, a potential position with Bioshock Infinite developer, Irrational Games.

Last week the Take-Two owned development studio posted a listing on Gamasutra’s jobs board, advertising for the position of Design Manager. The post contains a list of understandably demanding desired candidate experiences and skill sets. Nestled at the bottom however is a surprising and unusual requirement, stating that candidates must have a “Credit on at least one game with an 85+ average Metacritic review score.”

It shouldn’t come as any surprise that such numbers have come to influence job prospects within the industry. Gaming reviews aggregator Metacritic has been passively involved in all manner of issues lately, and you don’t have to cast your view far across the web to come across multiple other instances of the site affecting decisions elsewhere.

Recently the creative director and co-owner of Obsidian, Chris Avellone, revealed via Twitter (in a since deleted tweet) that their contract with Bethesda regarding the development of Fallout: New Vegas contained a bonus clause, payable only if the game achieved a Metacritic score of 85+. Likewise EA Sports famously set a Metacritic target of 90 for the developers of their footballing franchise FIFA ahead of its 2010 iteration.

screenshot

The practice of industry executives using Metacritic as a barometer for a game’s quality is well established nowadays, and it’s easy to understand why: Metacritic provides a universally understood, at-a-glance quantification of quality – the number.

But the issue with using Metacritic numbers to inform business practices, and what makes it all very lamentable, is a lack of understanding about what Metacritic scores actually represent on the behalf of those who use it as a metric.

Metacritic draws an average from individual review scores, taken from a number of professional and amateur gaming sites around the internet. Like data from any model, the numbers that Metacritic produces are inherently flawed, and this is for two main reasons.

Firstly the site assigns higher weightings to review scores produced by critics and publications that it deems to be of a higher quality. And secondly, each of these reviews is a subjective opinion, representative of someone’s personal view not the consensus of the general public. The averaging of these numbers loses most of its meaning when you make one person’s opinion more important, and then remove it from the context and commentary that explains it.

Metacritic was only ever intended for use by consumers as a rough guide to what the general critical consensus of a videogame is. That it now informs industry decisions is indicative of just how influential the site has become, and demonstrates laziness and a lack of understanding.

The argument has been made before that there is some logic behind such practices due to the rough correlation between a game’s Metacritic score and its sales figures. But the relationship it is exactly just that – a rough correlation, not a true representation of events. Fallout: New Vegas’s sales figures were 30% higher than Fallout 3’s during the first week of release, and the disparity in Metacritic scores between the two was 7 points, a metaphorical grand canyon of difference when viewed through the prism of the positively skewed videogame review scale.

New Vegas’s total global sales figures may currently be 6.16 million, compared to Fallout 3’s 7.78 million. But for what was essentially a content heavy expansion pack, and considering it hasn’t been out as long as Fallout 3, those figures indicate that New Vegas was a commercially successful and popular title. Why on earth Obsidian’s bonus was determined on the grounds of a Metacritic rating instead of sales figures, the truer picture of a games success and popularity is baffling.

screenshot

In the case of Irrational’s job advertisement the illogicality and (excuse the pun) irrationality of the situation stems from the fact that a game’s Metacritic score is a conglomerative indicator of a videogame development team’s work, not of one person’s skills. And whilst it’s understandable that Irrational wishes to attract a high quality of staff member, specifying such a requirement could filter out some potentially brilliant Design Managers who have been hampered by less than brilliant teams, and inadvertently open the gates to some not so brilliant candidates who have ridden along the success of a brilliant team. It’s like crediting Pinky for the intellectually elaborate plans that Brain came up with or vice versa.

It also seems a tad hypocritical when you consider that some of the studio’s past titles such as Tribes: Vengeance and SWAT 4: The Stetchkov Syndicate only managed a Metacritic score of 83 and 80 respectively.

Metacritic’s influence has extended beyond its intended boundaries and is now used to influence business decisions where it shouldn’t be. EA’s Peter Moore has even admitted that placing Metacritic targets upon developers was a ‘slippery slope’ for the company ahead of EA Sports’ FIFA 11 release.

screenshot

The site itself has nothing to do with how executives and gaming business decision makers have chosen to interpret its data, but it has ended up having an effect on people’s jobs, which it was never designed or intended for. This isn’t a problem within other industries – Metacritic also aggregates film, TV and music review scores as well and those industries have had no such issues.

Gaming is simply too obsessed with the number, an arbitrary metric, without understanding what is actually behind it. It shouldn’t be the promise of a job – that should be determined by the quality of your personal portfolio of work. Nor should it be the incentive of a financial bonus – aren’t there much truer indicators of commercial and public success?

So let’s all take a leaf out of Moore’s book, refute such deplorable practices, and think outside the number. Because when you do think about it, there’s not really much in a number at all.

Editor’s Note: In the interest of full disclosure it should be noted that Thunderbolt is aggregated on Metacritic.

The author of this fine article

is a Staff Writer at Thunderbolt, having joined in April 2012. Get in touch on Twitter @matski53.

Gentle persuasion

You should check out our podcast.