clock menu more-arrow no yes

Filed under:

Bad Process Can Still Lead to Reasonable-Looking Results

New, comments


If you've been reading Rufus on Fire for a while, you might recall that I'm a big believer in the notion of Good Process. In short, it's the belief that success more often comes from sound process, and that repeating success stems from repeating that sound process.

The key corollary is that good outcomes do not necessarily come from good process. They can come out of the blue, the result of dumb luck. Decisions are not justified because they work out well.

That brings us to the recent evaluation of the Bobcats published on the Wages of Wins Journal. It feels good to read someone prominent saying the Bobcats have a very good chance to be a playoff team. I've also gone on record as tentatively projecting 40+ wins this upcoming season, so my output isn't all that far off from the WoW output. However, you'll never see me give Wages of Wins any serious credence, because the process by which they calculate player value is hopelessly bad.

The classic takedown was provided by FreeDarko shortly after The Wages of Wins book was published. APBR forum members objected to the WoW process, too, with a lot more detail. (Try wading through that thread.)

When I read the book, I was struck by two major points:

1 -- A good chunk of the book actually addresses baseball and football... and the book's treatment of those sports is laughably bad. As in, nobody ever talks about the baseball or football sections because they were irrelevant and outdated and completely pointless before they were even published. They are plagued by a woeful lack of knowledge about the work already done in those sports by even the most prominent outlets. For example: Football Outsiders and Baseball Prospectus, which, at the time, were probably the two most famous and authoritative sources of "sabermetric" analysis of those two sports.

2 -- That, in turn, alerted me to the silliness in the WoW basketball calculations. Sure, it's a worthwhile point that offensive rebounding specialists don't get the love they probably deserve, but allocating defensive value by minutes played is a lazy and inexplicably baseless decision. That's right: WoW allocates defensive value by determining that if a player played more minutes, he gets more defensive credit. Think I'm twisting their words?

We don't have a measure of defense for each player. What we do have, though, is knowledge about how good the team played defense. We argue that if a player played 15% of the team's minutes, then he is responsible for 15% of the team's defense...

Crazy, right? It completely invalidates any measure of an individual player WoW sets forth. That one of the WoW authors would imply Chris Andersen produced less on defense last season than, say, someone like Glen Davis, because Big Baby played on a team with more wins and had a few more minutes, is absolutely absurd. Even more absurd is justifying this statistical tomfoolery by claiming that Manny Ramirez's poor defense in left field has little to no bearing on the total evaluation of him as a player, a patently incorrect statement (see: FanGraphs).

That's on top of the issues raised by FD, Dan Rosenbaum, and others, the main one being that WoW has little predictive power because it's powered by arbitrary constants on multiple factors. To simplify:

20 = 2a + 4b + 6c

20 = 2(10) + 4(0) + 6(0), where a=10, b=0, c=0

or

20 = 2(2) + 4(4) + 6(0), where a=2, b=4, c=0

or

20 = 2(8) + 4(4) + 6(-2)

Imagine 2, 4, and 6 are the number of rebounds, made free throws, and points our team gained in a season, and 20 is the number of wins. I've just determined the weights that "should" be assigned to each of these. That's essentially what WoW has done, only in more complicated fashion.

So even though WoW comes close to the same conclusion as I do, there's no way I'll cite them as any kind of support.

Apply these principles to moves the Bobcats have made in the past five years as you wish.