Framing the Framing Debate

Defense, the final frontier. Okay, not really, but it makes for a decent lead off. Over the last couple of years, various analysts have made tremendous progress in quantifying defense. We now have some solid estimators in +/- and UZR. Another one, SAFE, might be on the way to accessibility soon as well. Nevertheless, much work remains. Specifically, evaluating the defense of catchers remains a murky field filled with some great ideas but little consensus. I have been coming back to this subject off and on for the last four years or so and my general breakdown for a catcher's defensive responsibilities includes the following four categories.

Fielding range on pop ups, bunts and short ground balls have been covered under David Pinto's PMR for years (2006, 2007, 2008), and they are measurable under practically any defensive system out there. These types of plays, however, are both relatively rare and simple to execute events. That brings us small sample sizes and only minute measurable differences in ability. Specifically, in the case of pop-ups, the vast majority have large enough hang times that every Major League catcher makes a successful play.

Catching here typically has referred to preventing wild pitches and passed balls. To measure that, there are a couple of good methodologies out there. Dan Turkenkopf has a good annual series looking at stopping balls in the dirt (2009 version) and David Gassko has done some good work in the past on wild pitches and passed balls in a more general sense (THT Link). The thought processes in the two linked articles are sound. If Turkenkopf combined his blocks in the dirt with a secondary look at all other pitches, I would be extremely pleased with the overall picture that could provide.

Throwing out base runners is a more complicated system, though still doable. Essentially, it is the same process as John Walsh and I use for measuring outfielder arms. The key here is that it is much more valuable for a catcher to throw out a base runner that is attempting to steal than it is to prevent the attempt in the first place. Attempting a stolen base is usually a losing proposition for the offensive team so catchers that have a reputation for throwing out base runners well (e.g. Ivan Rodriguez) will suffer because runners will attempt to steal less often against them. This damages the defensive team's chances to remove said runners from the base paths. Therefore, you must note how often the catcher sees opportunities to throw runners out.

This brings us to the final point, the mystical cERA. Jeff wrote a well thought out post on the subject (Link) which reminds us that while researchers have shown that, to date, that any cERA effect lies below the level of detection, that threshold is important. It remains high enough that a meaningful difference might be hiding in the noise, much in the way that clutch hitting or a pitcher's ability to influence BABIP does.

Ever since that post, I have been thinking more about the possible aspects of cERA. It has always been my preference to deconstruct complex systems as much as possible and then go from there. What sort of skills, not covered in any of the above three categories, would help the team suppress runs allowed? I came up with three.

There is certainly a coaching aspect that contributes to the catcher-pitcher relationship. I group together tasks such as knowing how to keep the pitcher's mental frame of mind in check or how to spot minor mechanical kinks into this sub-category and dismiss it as far as defense goes. To me, these are coaching skills and should be the topic for another discussion.

There are also skills that I choose to lump under the vague term "game management." Chief among these would be pitch sequencing which I am comfortable giving some credit to the catcher, just not all. After all, the catcher acts as a guide, but it is the pitcher who confirms the pitch and is the one who throws it. I cannot in good conscious sit here, after years of blasting Felix for piping in fastball after fastball, and then claim that catchers deserve full credit for calling pitches. That would be hypocritical and, I feel, an illogical rationale.

The final grouping is one that I feel should actually go under the catching category, but quantifying it remains more elusive. It would encompass all the little things that a catcher can do in order to generate more strike calls. Generally, people refer to this as framing. The goal is to use whatever combination of skill, cajoling or repetition is possible to get an umpire to call a borderline pitch a strike.

People around the net routinely vilified Kenji Johjima for his framing abilities. Personally, I felt it had more to do with a personal bias against him then about anything tangible. Those same people typically lauded Rob Johnson, perhaps only as a reaction to Johjima, for his apparently superb capacity to gather more strike calls.

As I am well known to say, hogwash to subjective statements! Let us get some data in here! Keep reading for more than you probably wanted.

If Rob Johnson actually was so much better at framing pitches than Kenji Johjima then people making that statement should have no worries that the evidence supporting that would be apparent, right? Given the level of vitriol hurled at Johjima's catching style, and the subsequent amount of love heaped on Rob Johnson for his ability, it should be obvious in the data. Before any knuckleheads drag it out, no, I am not talking about the cERA statistic as typically presented.

There are many problems with that statistic. Going over them is a waste of time, but keeping in mind that catchers do not catch the same amount of pitches from each pitcher is the big one. After all, you can put the greatest frame in the world on a Carlos Silva pitch and it does nothing to detract from the fact that Carlos Silva threw it. And a masterpiece from Felix Hernandez can stand by itself.

I am writing of data pertaining to the actual question in hand; did Rob Johnson frame pitches in a better manner than Kenji Johjima did? Was he better at getting pitches called a strike? Critics of Johjima loved to state that he was terrible at getting the low strike called because of his style of dropping his glove downward after receiving. Ergo, if I were to plot every pitch caught by each catcher, there should be a clear difference. The mind, and its selective memory, a powerful thing be. 

Below is such a plot for Kenji Johjima. It is every pitch caught by Johjima in 2009, recorded by Pitch F/X, which the batter took. The red dots represent called strikes, the blue dots, called balls.

I have zoomed in on the important part of the plot; everything outside the zone was an obvious ball. The black box represents a rough approximation of the rulebook strike zone. The grouping of red dots in relation to that zone is not surprising. Several studies of umpire tendencies have shown that they call the strike zone a few inches wider than the actual plate. The width of the zone portrayed here is already wider, stretching an extra inch each way in order to account for "the black." In reality, it appears that pitchers get about two extra inches to each side.

Now, if this were a TV broadcast, this report would be awful because I have given the critics the sound bite they crave right at the start. I will make it even worse in a few sentences since this is text and I get to indulge in sarcasm. Look at the bottom of the strike zone. Do you see all those blue dots? They were right! Johjima consistently costs his pitches the low strike!

That statement is the victim of a classic lack of context. Before anyone trumpets victory, Johjima needs to be compared directly to Rob Johnson. Below is the same plot, under the same constrictions, for Rob Johnson.

Rob Johnson doesn't get the low strike either! Thank you, human umpires. In addition to umpires typically calling a bigger zone on the horizontal axis, they also frequently call the zone too small on the vertical axis. As illustrated, the strike zone should be taller than it is wider. In reality, it is wider (roughly 22 inches) than it is tall (roughly 18 inches).

Time for a quick tangent. Doesn't it appear that Rob Johnson caught against many more left-handed batters then Kenji Johjima? There are certainly a lot more pitches located on the left-handed side of the strike zone. As it turns out, Kenji caught a fraction (less than 1%) more lefty batters.

Now, those are some sweet plots and they make for some compelling evidence I think, but they do not do the whole trick. Visually comparing the two, after all, is just one step better than what the people at the beginning were doing. What would be truly useful is quantifying any significant difference between the two. 

Here is where it begins to get more complicated with statistical jargon. Please do not fret; I will do my best not to lose anyone. I analyzed those plots and created grids, 0.1 inch by 0.1 inch square. Then I calculated an equation (using a kNN process to those that care) to tell me how likely it is that a pitch landing in each grid square would be called a strike.

Are you falling asleep yet? I know it's dry, so here are some colors to distract you. I took those probabilities and contoured them up, heat map style! The black lines represent the boundaries of the 25%, 50% and 75% areas for called strikes. That is, everything inside the middle (50%) circle was more likely to be called a strike than a ball. Kenji is on the left, Rob on the right: 

Now these are certainly pretty and marginally useful just to see where the various likelihoods are, but on their own, they get no closer to what I was talking about before - a quantifiable difference. That is okay though; they were just an intermediary step. The next two steps are the important ones.

First, I finally get around to doing what would seem natural; I subtracted the two plots from each other. For every square in the grid, I took the probability that an umpire would call the pitch a strike for Kenji Johjima and subtracted away the probability that an umpire would call the pitch a strike for Rob Johnson. If the two were equally likely to get a pitch called a strike, the net result would be zero. If the equation predicted Kenji to more likely get a strike call then it would be a positive number (he would have a higher probability), and similarly it would be negative if the probability favored Rob Johnson.

Am I a master of foreshadowing, or what?

My final graph is a plot of said differences. I re-instated the hypothetical strike zone again to provide some reference. The blue dots are grids that hold a preference for Kenji Johjima by at least 10%; the red dots a 10% or greater probability difference for Rob Johnson.

Before passing judgment, I find it satisfying that some patterns are present in this plot. There are indeed areas where it looks like one catcher works the umpire better than the other does. That makes more sense to me than if the differences had been more randomly scattered.

Now, there are clearly more blue dots than red ones, but that's still not a quantifiable answer. For one, Rob Johnson has fewer zones of advantage, but they might be of greater magnitude. In addition, they might come in areas, such as the dots off the left side of the strike zone that see more pitches. The final step is taking this grid and finding out how many pitches in 2009 landed in each spot. To get a total difference in predicted strikes called, it is simply a matter of going square by square and multiplying the number of pitches in that square by the difference in probability for the two catchers.

Based on the above plots, Kenji Johjima's predicted pattern of framing would have resulted in 13.5 additional strikes being called if he had caught all of the pitches studied. Taking Dan Turkenkopf's figure of .161 runs per added strike, that difference is worth 2.05 additional runs for every 10,000 pitches not swung at, which is right about one season's worth from a full time catcher.

Two runs. 

In conclusion, from a thought experiment perspective - which is what this began as - Kenji Johjima looks to have been a better framer of pitches than Rob Johnson was last season, at least by the method that I decided to measure it by. Secondly, that difference was minuscule, even before adjusting for any possible regression.

I am not going to make any overly broad statements about the value in framing pitches across all catchers. I only looked at these two and only this single year. It very well could turn out to be a simple fluke that they ended up so similar. Nevertheless, aside from the obvious Mariner interest, isn't it interesting that two catchers who generated so vastly a different perspective on their framing abilities not only ended up being so close in value, but actually in reverse of the mainstream opinion?

As a last piece, I created two GIFs that flip back and forth between the two comparison graphs so that you can see some of the differences in a different way. Here is the link to the general pitch plots and here is the link to the differing contour maps.