Writing is easier when you don't have to think up an intro.

In terms of things we know we can measure pretty well, Rob Johnson the defender doesn't stand out, at least not in a good way.

• His error rate, for whatever that's worth, is about average. His four in 754.1 innings matches up well with the league-average rate of 6.1 per 1000.


• His arm is decent. He's thrown out 20 of 71 would-be base-stealers - a rate of 28.2% - against a 27.6% league average.


• He kind of sucks at blocking balls. His 2009 rate of passed balls was twice the league average, and the Mariners were a bottom-third team in wild pitches. In addition, I couldn't tell you how many times I saw Johnson flat-out drop a pitch that hit him in the glove.
  

These factors were covered in devil_fingers' evaluation of 2009 catcher defense, and in that study, it was determined that, based on his errors, arm, and ability to receive, Johnson was about two runs below average in the field. Now it's obviously important to remember that understanding performance in a single season isn't the same as understanding true talent, but given these numbers and all of our own observations over the course of the year, I don't see any particular reason to believe that Johnson is a superior defender when it comes to things we can directly measure.

And yet he's developed this whiz-kid reputation. Why? Because, out of guys who caught at least 50 games last season, Johnson posted the lowest CERA - ERA against as a catcher - in the league, at 3.23. This versus Kenji Johjima's CERA of 4.86. Pitchers weren't giving up as many runs throwing to Johnson as they were throwing to Kenji, and this - pretty much this alone - earned Johnson both a lot of playing time and a generally favorable standing among fans.

So given that he doesn't really do anything else well, either standing at the plate or crouching behind it, one's opinion of Rob Johnson comes down to one's opinion of this statistic. Those who like Johnson do so because they feel he helps keep the opponent off the board, while those who don't do so because they don't believe CERA reveals an ability on the catcher's part to control a game. This should be examined.

The absolute first thing we have to do with CERA is consider the pitchers. Catchers don't catch an equivalent distribution of arms, and sure enough, Johnson was catching the bulk of the team's better talent. Felix Hernandez, Erik Bedard, and Jarrod Washburn made up more than half of Johnson's PA's caught, while for Johjima, they made up just 9.4%. For this reason, we should expect their CERA's to be considerably different. The guy who catches the better pitchers is almost always going to post the more flattering rate of runs scored.

However, we wouldn't expect their CERA's to be that different, so clearly there was an additional performance effect. I looked at how the bullpen did throwing to each guy a few weeks ago and, sure enough, their walk and strikeout rates were a little better with Johnson than with Kenji. The picture's a little messier with starting pitchers since the distribution was so skewed, but the take-home message here is that the difference in CERA's was more than simply the difference in pitchers caught.

So with that in mind, the question becomes one about sustainability, about whether or not this sort of thing is reflective of an actual ability and is therefore repeatable on a year-to-year basis. This being beyond my capability, thank goodness for Keith Woolner. All year long, when people have talked about the significance of CERA, others have referred back to work done on the subject a decade ago, and that work was done by Woolner at Baseball Prospectus. Here's the original study, here's a follow-up, and here's a later study done after receiving some criticism from Bill James.

Read those articles. I know they're heavy on the math and somewhat lacking in floral prose, but if you're interested in this subject, then Woolner's work is required reading.

There's too much in there for me to summarize point-by-point. But in the end, Woolner didn't find strong evidence of an effect. He didn't find weak, potentially insignificant evidence of an effect. He found no evidence of an effect. At all. Even just looking at the extremes, the absolute best and worst catchers in Year X, they regressed all the way to the mean in Year X+1. Woolner's ultimate conclusion:

For now, at least, the hypothesis most consistent with the available facts appears to be that catchers do not have a significant effect on pitcher performance.

Based on CERA and its component metrics, Rob Johnson had a much better season of game-calling than Kenji Johjima. But given that this effect has never been shown to be repeatable, we probably shouldn't label it game-calling at all, as doing so implies an ability. We don't actually know if what we're trying to measure exists, and until we do, the most responsible approach is to side with the null hypothesis. There's no proof. There's no proof that Johnson's alleged greatest strength is even the least bit significant, or real.

What's funny is that, if we're just scrounging for as much evidence as we can find, there's a lot more evidence that Kenji was bad than there is that Johnson is good. Kenji was here for four years. In three of those years, his numbers were a lot worse than those of his backup(s), and in the fourth they were about equal. For Johnson, we have one year. 80 games. With that in mind, it makes more sense to suggest that Rob Johnson looked good simply because he wasn't Kenji Johjima. There's more reason to dock the latter than there is to credit the former.

But we needn't focus on that paragraph, because Woolner's work trumps it. Nobody's ever verified that game-calling is a skill, and because of that, if one wants to believe that Johnson calls a great game, then one needs to provide a lot of evidence. Tons of it. Years and years and years' worth. Rob Johnson has started 82 games behind the plate in his Major League career. It is impossible for one to conclude anything about game-calling over a span of 82 games.

Working in Rob Johnson's favor is that pitchers like him. The reason he caught so much of Felix, Bedard, and Washburn is because that's what Felix, Bedard and Washburn wanted. If the pitchers feel like Johnson calls a good game - if the pitchers feel comfortable throwing to him - then that has value. You want your pitchers to feel comfortable when they're going to work. But then Greg Maddux felt most comfortable throwing to pretty much anyone but Javy Lopez, yet of the 12 catchers who caught Maddux for more than 25 games, opponents put up the second-lowest OPS with Lopez behind the plate. Comfort is good, but it's not proof of ability, nor is it a trump card. If you have two guys who're pretty much equal, and a pitcher would rather throw to one of them, then that's fine. But if you have two guys, and one of them is measurably better than the other, then the responsibility falls on the pitcher to feel comfortable with the guy who maximizes the team's chances of winning.

Rob Johnson is a 27 year old Major League catcher who, in his rookie season, became a favorite of some high-level pitchers and coaches. In that regard, he's off to a hell of a start. But a lot of fans have gone so far as to suggest that he's a valuable player, the reasoning being that he keeps the other team off the board. We can't say that. We can't say that and have it mean anything, because that statement has yet to be confirmed. We will know that Rob Johnson is a valuable player if and only if he improves in the areas we know we can measure. And while PITCHfx may allow for someone down the road to show that game calling is a legitimate, repeatable ability, it would really put my mind at ease if Rob Johnson would think about maybe swinging a bat.