One of the most frequently repeated criticisms of defensive stats in general and UZR in particular is that on a number of occasions they aren't very stable on a year-to-year basis. According to UZR, for example, Bobby Abreu dropped from being a -4 run RF in 2007 to a -26 run RF in 2008, while Mark Teixeira jumped from being a -4 run 1B in 2007 to a +10 run 1B in 2008. For a lot of people, this sort of inconsistency seems sufficient to invalidate the entire system.
It's not. For one thing, nobody familiar with defensive metrics would advise arriving at conclusions based on individual seasons (or even individual metrics). It's far better to look at a three- or four-year window in order to gauge a guy's true ability. But for another thing - and this is the bigger point, I think - inconsistency in no way means that defensive stats are bullshit any more than it means offensive or pitching stats are bullshit. Since 2006, Abreu's bat has been worth anywhere from +11 runs to +33 runs, for a range of 22. Teixeira also has a 22-run range. 38 runs for Manny Ramirez. 21 runs for Lance Berkman. 21 runs for Albert Pujols. 34 runs for Jason Bay. And so on and so forth. Are our offensive stats broken?
By tRA, looking over the same span, CC Sabathia has a range of 27 runs. 25 runs for Roy Halladay. 15 runs for Derek Lowe. 29 runs for Dan Haren. 50 runs for Josh Beckett. 30 runs for Carlos Silva. Are our pitching stats broken?
Statistical instability is a part of the game, and just because we don't understand defense as well as we do offense and pitching doesn't mean it shouldn't be susceptible to the same sort of annual fluctuation. Yeah, it's weird to think that Abreu could've dropped from being worth -4 runs in the field in 2007 to -26 runs in 2008 while remaining the same person with the same skillset, but then, he also dropped from being worth 33 runs with the bat in 2006 to 11 runs in 2007 while remaining the same person with the same skillset, too, so who cares? Nobody's accusing wOBA of being a waste of time.
Single-season performances can be inconsistent. At the plate, on the mound, and in the field. That's a fact, and the only people who disagree are the people who don't like what the numbers are telling them. We may not yet have the perfect measure of defensive ability, but we've still got some great stuff, and as long as you interpret the numbers properly, there is valuable and accurate information to be gleaned.