Steam Spy Is Back, But Not As Accurate As Before

Steam Spy, the PC gaming sales-tracking mainstay loved by some developers and loathed by others, was recently presumed dead by its creator, Sergey Galyonkin, after an update to Steam’s privacy defaults. It is now coming back, but it’ll be different.

The old Steam Spy estimated a PC game’s sales by sweeping publicly viewable Steam profiles and taking stock of their game collections. That technique was blocked in early April when Steam’s architects at Valve made it so that users’ profiles no longer automatically show which games players own. The new version of Steam Spy will get around that by embracing machine learning to predict sales based on “coincidental data.”

In a blog post, Galyonkin explained that he’s in the process of switching Steam Spy over to a model rooted in machine learning. He confessed, however, that despite a wealth of data to feed his algorithm, it’s “not very accurate” at this point.

“I have the data for around 70 games from different developers, and for 90% of them, the new Steam Spy is within 10% margin of error,” he wrote, pointing out that the site’s estimate for Frostpunk (252,000 units sold) was basically on the money (250,000 units sold). “But I also saw some crazy outliers, where the difference between the estimates and the real data could be fivefold.”

If there’s a chance the data could be that inaccurate, why bring Steam Spy back at all? Galyonkin credited his decision to “over 200″ messages from developers explaining how Steam Spy impacted their operations. He mentioned an indie company from Berlin that got government funding for its game with the help of Steam Spy’s data, and though he didn’t name the developer in the post, he told Kotaku that it was inbetweengames, creator of noir tactical game All Walls Must Fall

Praise for Steam Spy has always been buffeted by wariness. Earlier this month, many developers were relieved to see it seemingly go, because it empowered armchair analysts (and some harassers) with data they perceived to be accurate—even when it wasn’t. In theory, a new, less accurate model could exacerbate that problem. Galyonkin, though, still thinks Steam Spy is worth bringing back from the brink.

“Armchair analysts do not need Steam Spy data,” he said in an email. “They’re fine with whatever is available: [concurrent players], the number of reviews, even a user score.” He noted that, in his opinion, the most important number for most players is concurrent users in a given game, because that helps them decide if multiplayer games are worth purchasing. Valve, he added, already makes that number public.

As for why developers can still benefit from Steam Spy’s reanimated bones, Galyonkin claimed that they’re already happy to shell out serious cash for services that he thinks are even less accurate.

“Relatively low accuracy doesn’t prevent developers from paying $30,000/year for [an] AppAnnie subscription,” he said, referring to a popular app analytics service, “because having an estimate is usually better when making decisions than having no data at all. Analysts can work with data when they know it’s not entirely accurate as long as they are aware of the limitations and caveats.”

Galyonkin is still fine-tuning the new algorithm, and many of Steam Spy’s old features aren’t even back online yet. Country stats, for example, are off the table right now. Things are early, in other words, and there are still options on the table—some of which involve collaboration with other interested parties.

“I have some offers from people with more recent knowledge in the field of machine learning,” he said. “I hope they’ll help me improve the algorithm.”

You’re reading Steamed, Kotaku’s page dedicated to all things in and around Valve’s wildly popular PC gaming service. Games, culture, community creations, criticism, guides, videos—everything. If you’ve found anything cool/awful on Steam, send us a message to let us know.