The Problem With Sabermetrics

Does the theory or model have much explanatory power for data sets other than the one I developed it on?

My professors described Astrophysics to me as the only physical science where it was impossible to do experiments. To test hypotheses, you needed to predict a natural event and then wait to see if youid predicted it correctly.

Hell, even relativity gets experiments like Gravity Probe B. What do we get?

So, I’m not so willing to say that baseball analysis isn’t science, because that would mean that Astrophyics isn’t science, and I think Astrophysics is science.

Though, I’m also a fairly strict Popperian, so that might be relevant.

And it really does emphasise how lucky Mariners fans are to have Jeff and Matthew

Though I do miss mean Graham a little.

I still believe that at its best, sabermetrics is interesting, worthwhile and educational. I disagree wholeheartedly with anyone who thinks starts are useless or make the game less fun. However, I also disagree with the direction more recent sabermetric analysis is going, for the reasons I explained in the piece.

I guess all of the above is more obvious if you know about my history with this stat work (obviously you do, Matthew), but I seem to have confused some of the people who weren’t really around for the bulk of my LLing. Hello new people! Much love!

Personally, I think I’ve always been brought up to have a skeptical approach to the world. This probably suggests why analytical blogs appealed to me in the first place. What your writing and the writings of others have bolstered, though, is more how to question things, namely consideration for all contributing factors. I think of it as Seeing ERA, Crediting Defenders.

Concepts like regression to the mean and replacement levels have also bled into my views of politics, society, and pop culture, giving me yet another angle from which to see and possibly better understand things.

once you start thinking about choice in probabilistic terms, it becomes difficult to not apply that to other areas. Though, it does seem to me that having a healthy skepticism of Jenny McCarthy’s claims don’t exactly require an analytical mind.

And I would grossly simplify the situation by dividing it into two groups. One is the group that wants to take the research that’s been done and evangelize with it. The other is the group that wants to keep researching to learn more about the game.

There is a lot more research over the last few years into very fundamental concepts of the game (e.g., what causes the platoon differential, what causes flyballs/groundballs, etc.) than there was 5 or 10 years ago.

There is also a lot more widespread application of the older ideas (e.g. DIPS, fielding metrics based on granular batted ball data), and this shows up everywhere because of fantasy baseball interest.

I mostly don’t find sabermetrics interesting unless I’m doing it for myself. That may be what you’re saying, too. That doesn’t mean I don’t enjoy reading other people’s work, but it’s my own research that gives me the most enjoyment and the most interest in learning more. One of the things I’ve wanted to know for a long time is how much the hit-and-run play affects our valuation of stolen bases and caught stealing. At some point I’ll dig in and mine the data for that answer. There are a ton of other unanswered questions about the game where because of Retrosheet or PITCHf/x the data is now available to find an answer.

I see it as a really exciting time for sabermetics, even though there may be a lot more fluff in the field than there was a few years back.

1. My point about astronomy is not a criticism of philosophy, it’s that basically that whole discipline of astronomy relies on observational data over which they have no control rather than controlled, repeatable experiments. That’s basically the same situation we find ourselves in with analyzing sports. One can go out in the back yard and test a few ideas with a bat and ball, but for most baseball questions that’s not an option, and one must use whatever unrepeatable events the athletes stage.

2. I’m thinking of sabermetrics = “the search for objective knowledge about baseball”, under which definition learning about why the baseball moves how it does, either from the pitcher or off the bat, is very important. I understand that wasn’t the focus of your criticism. However, there is plenty of awful PITCHf/x-related analysis out there. It suffers from all the same problems that you point out above.

3. Thanks for the example. It really frustrates me how much the sabermetric community has pushed back against Colin for that effort as if he is some destroyer of knowledge rather than taking his criticisms to heart and trying to evaluate their validity. (Sean Smith being an exception to that.)

I wonder if that’s truly a bad thing? To me, people like tango and others are the ones who are pushing the field forward in interesting ways. Reading his stuff is incredibly thought-provoking to me because it causes me to actually think about the game in different ways.

To me, FanGraphs is the place for people who are slowly moving away from traditional stats and want to know about how the most famous and new, up-and-coming players are doing and how they are likely to do in the future. After familiarization with the approach, the topics become somewhat routine (boring?) as individuals can do the same level of analysis by themselves more quickly. But until that familiarization occurs, I think I am happy that there is a place like FanGraphs to help people make that transition. I’m sure that there are many for whom baseball is just a casual interest that never fully make the transition and need others to continually do the analysis for them.

For those that feel that FanGraphs isn’t doing interesting work anymore, I think that this is common and had assumed that the answer was to move on to more though-provoking and challenging websites as opposed to giving up on the current trends in sabermetrics all together.

in my own attempts at applying non-linear regressions to some of the current sabermetric concepts, I have ended up with truly absurd results. I’m not convinced this means that more advanced non-linear models wouldn’t be useful, just that it’s likely to be quite difficult.

Is priceless.

Although, some high level math can be very useful and neccesarry. Logistic regressions for example.

the simple regression concept on a non-linear complex system is only used in a best case scenario to see if the data is pliable. No real statistician would just run with it in a simple linear regression on a complex system and stop because it would fail the test phase of their analysis if the data set is partitioned. The real problem is that there are very few true statisticians working on these problems, not people anointed by themselves or others as statisticians.

I love the direction baseball has taken in regards to application of mathematics and models to baseball, but the problem is that there are too many guys that know just enough to not get in trouble doing analysis that is not as thorough as it should be. To compound the problem those results are getting accepted as fact.

This is where my request came from to “please clarify because I think one can read into the criticism whatever one doesn’t like about baseball analysis.”

I see a lot of bad analysis being done with complex multivariate regression.

I do believe that laxtonto’s statement that “modeling is only as useful as the idea at its backbone” is correct and is consistent with one of Graham’s main points. I thought the idea at the heart of what he was saying is that any analysis that loses a connection to how the game works has gone off the track. It’s not the simplicity or the complexity of the model that matters, it’s the explanatory power and the foundation of logic underneath it. Simple regression models can have a lot of explanatory power in baseball and can illuminate a lot if they are driven by thinking that’s well-grounded in how the game works.

but I’m not sure using pejorative terms to classify the traditional or outgoing school of thought is going to encourage open-mindedness regarding new and potentially better theories.

I don’t have a link handy, but you could probably track it down pretty easily.

xFIP no

but perhaps if you looked at the tail ends of the distribution instead of trying to classify the nebulous middle?