Lookout Landing: An SB Nation Community

Navigation: Jump to content areas:


Sports blogs for fans, by fans.
New Blog: Sounder At Heart for Seattle Sounders Fans!

Demuddling some common statistical mixups

So it seems like a lot of people are missing a crucial point in player evaluation:

The stats used to evaluate pitchers are not necessarily the ones we should use to look at hitters.

Analysing run prevention is a different beast to measuring run scoring, and that's because we're trying to solve seperate problems with different constraints. The key concept here is that runs win baseball games, and we should consider the twin goals of calculating present run value and future run value as paramount.

When attempting to measure a player's output in terms of runs scored, we need to figure out the individual contribution to what is a team effort.  Since each player is only 1/9th of a lineup, we can't blindly look at a given player's runs scored/runs batted in. We must determine the components of players that best describe their offensive worth, which is where OBP, SLG, and more advanced metrics come in. These turn out to be reasonably stable year to year.

We run into a different problem with pitching/defence (if everyone will permit me to ignore the difficulties in isolating a pitcher from his fielders for a few minutes). We know how to model runs scored. We take... runs scored against. The correlation here is 1.00, so there is no need to look at components to measure present value of the pitching/defence units. The problem lies in the stability of our run prevention statistics - namely that there barely is any. The components used to evaluate a pitcher should be those that decrease the volatility of our future run value estimate, and therefore they need to be defence independent. This is the province of FIP and tRA.

So when you feel like applying tRA to hitters or OPS against to pitchers, remember the context in which the stats were developed. It's very easy to think that they'll be accurate when applied to the other side of the game, but doing so loses sight of the constraints they were designed to overcome.

2 recs  |  Comment 64 comments

Story-email Email Printer Print

Comments

Display:

I would maybe separate the discussion into two issues:

1) Isolating individual contributions
2) Reducing the volatility of the statistics

With hitters, most reasonable people have moved past team-dependent stats like RBI and runs scored to stats that are more under individual control, like OBP, SLG, wOBP, etc. So 1 isn’t so much of an issue for hitters.

With pitchers, it’s an ongoing battle to convince people that a defense can have a very significant impact on a pitcher’s ERA and that we need to focus more on the batter/pitcher interaction and less on what happens after the fielders get involved. (What I find especially frustrating with this is that many of the same people who cling to ERA will cling to the idea that defense wins championships, but I digress.) So 1 is very much a problem in pitcher evaluation (for most people who follow baseball casually.)

As for #2, a nice side effect of measuring pitcher performance through component stats like strikeouts, walks, fly balls, line drives, etc., is that these stats have a much lower variance than ERA over reasonable periods of observation. That is, even if all teams had equal defenses, ERA would still be a less reliable indicator of performance early in the season. So tRA in one fell swoop helps us both to separate defense from pitching but also helps give us a less variable way to measure pitchers.

With hitters, there’s generally not a huge need to reduce the volatility in season stats, but even then, using component stats to formulate something like PrOPS can give us useful information.

by ubelmann on Dec 3, 2008 10:42 PM PST reply actions   1 recs

Do you think we are going to see

some of the newer stuff like swinging strike %’s integrated into a FIP/tRA-like metric? If we use some of these stats can we reduce the volatility we see in pitchers or are we always going to be doomed to the fact that pitchers are just inherently very unpredictable?

I’m hoping Hit f/x will help us distinguish batters that have fluke seasons and batters that have gotten better. I think this has been one of the traditional debates we always have over young players because most of the stats require a significant sample size to be accurate and if a player only has a few years in the majors it can be difficult to come to any conclusions.

by Edgar for Pres on Dec 3, 2008 10:49 PM PST up reply actions   0 recs

I'd guess that pitchers will always be a little tougher to evaluate than hitters

We have no way of knowing who is going to master a new pitch that helps them, and changes in health can have a huge impact on their performance. And if there’s some good reason that a starting pitcher would have a bad day (has the flue, wife left him, pickup broke down, dog died, etc.) that’s something like 3-4% of his season, whereas if a hitter has a bad day, it’s less than 1% of his season. So the systematic uncertainties are harder to get a handle on for pitchers than for hitters.

That said, I wonder what the year-to-year variance in tRA is compared to the year-to-year variance in wOBA. I always cringe when I see prediction systems tested by comparing a pitcher’s predicted ERA to his actual ERA. (I additionally cringe because they usually use correlation coefficient when it would be more appropriate to use something like a chi-squared statistic.)

by ubelmann on Dec 4, 2008 11:59 AM PST up reply actions   0 recs

Thanks

I think about this kind of stuff all the time. (Really, much more than is probably healthy for me.) I have about a dozen different things I’d like to test out, but thus far I haven’t had the intestinal fortitude to learn some SQL and put together a database so that I can slice the data appropriately.

I really like the stuff that you’re making available over at StatCorner, though, it seems like a good compliment to Baseball Reference and Fangraphs.

by ubelmann on Dec 4, 2008 11:49 AM PST up reply actions   0 recs

If I understand the post correctly

you are saying that applying tRA to batters is wrong because tRA treats all ground balls and flyballs as created equal (which, for a pitcher, they may as well be assuming he’s a major league pitcher) while for a hitter they are not. This is very true, and something people should remember when comparing a player’s BABIP to the league average to say they were inordinately unlucky or fortunate in a given season. Each player has his own expected BABIP, based on power, speed, and contact type.

All the same, statistics like PrOPS are useful because every year there are Beltre type players that beat the snot out of the ball all season long, but due to park factors or prolonged ill luck just don’t get the slash line that would reflect their ability. This is much, less common and much, much less drastic than the pitching equivalent, but should not be ignored.

by Bearskin Rugburn on Dec 4, 2008 7:51 AM PST reply actions   0 recs

The point is essentially to always make sure that there -is- a point

If a stat adds to our understanding of current/future contribution to wins, use it. If not, don’t. Of course, to know which stats would contribute one needs to understand the obstacles to working this information out for both pitchers and batters, which I tried to lay out.

by Graham on Dec 4, 2008 7:56 AM PST up reply actions   0 recs

Do you want to get rid of saves?

This stat doesn’t advance our current/future contribution to wins.

by Edgar for Pres on Dec 4, 2008 11:36 AM PST up reply actions   0 recs

I think the problem isn't that these stats are the worst possible thing created

Its just that we all get so annoyed/pissed when they constantly get used incorrectly that we build up a hatred for them. Instead we should just hate all the stupid people out there that try to claim Ibanez is worth a lot because of his RBI totals, K-Rod is amazing because of his save totals, and Ervin Santana was almost as good as Cliff Lee because of comparable WHIP.

We don’t need to get rid of the stats. We just need to continue to educate people what these stats can’t do and shouldn’t be used for. There are great alternatives out there and anybody with any common sense can pretty easily pick that up.

by Edgar for Pres on Dec 4, 2008 12:06 PM PST up reply actions   0 recs

RBIs, meh.

I think they’re stupid and I don’t see the point in tracking them, but whatever.

Saves though? They don’t need to exist. At all. They should just go away.

by acblue on Dec 4, 2008 1:38 PM PST up reply actions   0 recs

except that saves

are overvalued, and the continuation of the existence of saves allows teams to potentially exploit the compiling of mostly meaningless stats in the effort to get better. See – Koch, Billy.

Can't wait for Ruskell to knock this one out of the park.

by abender20 on Dec 4, 2008 2:54 PM PST up reply actions   0 recs

We seem to have gotten rid of GW RBI

And saves are nearly as ridiculous. If I had to choose one pitching stat to get rid of, though, I would get rid of pitcher wins. It didn’t really make sense as a statistic when pitchers pitched entire games, weren’t necessarily awful hitters, and pitched 50 games a year, and they really don’t make sense now. It just makes zero sense to give a pitcher full credit for a win or loss every game—at that point there’s no room to give hitters any credit for the outcome of the game.

by ubelmann on Dec 4, 2008 12:08 PM PST up reply actions   0 recs

The pitcher win is indeed slightly stupider than the save,

but the save has completely changed the way the game is played, and it has turned managers into idiots. They’re managing to a stat, not to the best way to win a baseball game. I think that’s quite detrimental to the game. Whereas wins are just stupid, saves actually made baseball stupider.

by Teej on Dec 4, 2008 1:03 PM PST up reply actions   0 recs

Do you think

the save is detrimental to the game or do you think stupid managers are detrimental to the game?

by Edgar for Pres on Dec 4, 2008 1:43 PM PST up reply actions   0 recs

People are generally pretty stupid.

Give them less reason to be stupid.

by acblue on Dec 4, 2008 1:52 PM PST up reply actions   0 recs

I think managers feel an obligation to help out their players.

The save has massively inflated the salaries of top relievers, and, unfortunately, the number in a pitcher’s save column is a big deal when it comes time for his next contract.

So while managers surely have to accept some blame for today’s rigid, often-stupid bullpen usage, the stat is what got them there.

by Teej on Dec 4, 2008 2:04 PM PST up reply actions   0 recs

I think that the ill effects of the save on baseball strategy tend to be overstated

Of course it is silly the way that some managers base their strategy around the magical 3-run limit, but most of the rest of the development of modern-day closer usage was kind of inevitable. Managers were eventually going to figure out that it’s easier to find a guy to pitch really well for one inning than it is to find a guy who can pitch really well for three innings.

It’s pretty dumb how some managers manage to the save, but in the end it doesn’t make a huge difference—if it did, more inventive managers would stick out and the guys who managed to the save would lose their jobs. (Also, we shouldn’t just blame the managers—I’m sure there are plenty of closers who really love their save totals and wouldn’t be 100% happy if they had to sacrifice a save opportunity because they had to go multiple innings for the save the day before.)

Regardless of just how big their effect has been, though, I still think they should just go away.

by ubelmann on Dec 4, 2008 2:02 PM PST up reply actions   0 recs

And then you split an infinitive.

I’m so disappointed.

I like using semi-colons; they make me feel smart.

by Llewdor on Dec 4, 2008 4:33 PM PST up reply actions   0 recs

So if you want to get rid of some stats

I want to ask why? I understand the cases against them and I agree most are pretty irrelevant and don’t tell us much about player talent or their underlying performance.

But why do you want to get rid of them? Do you just not want to deal with them more? Are you worried that ignorant people will continue to use them poorly?

Just taking them away so that people do not misuse them is not enough of a reason to get rid of a stat. There are many people who love baseball partly because if its rich (and crazy) use of stats. How has Beltre batted when it has rained on the third thursday of the month? What is the chance a team will win after losing 3 games? All these useless stats are one of the things that makes baseball what it is today. Baseball is not just the real life version of one of the millions of simulations that is churned out on your computer that you can check your answers against. There is much more. It is interesting to know who hit in the most runs. It may not interest you because your motives with stats are very different but that does not mean it is not interesting to the rest of the public.

I understand your annoyance when people use bad stats incorrectly. I agree 100% that its pervasive throughout baseball too from the youngest fans to the oldest front office people. This isn’t a good enough reason to rip away baseball’s quirky obsession with strange stats.

by Edgar for Pres on Dec 4, 2008 12:26 PM PST reply actions   0 recs

The weird things people are obsessed with holds back real analysis.

“Baseball is not just the real life version of one of the millions of simulations that is churned out on your computer that you can check your answers against.”

Baseball is baseball. I’m trying to model baseball. I’m not trying to produce bullshit just to say ooooh look at the pretty stats.

by Graham on Dec 4, 2008 1:22 PM PST up reply actions   0 recs

Some stats are more honest than others

It is interesting to know who hit in the most runs. It may not interest you because your motives with stats are very different but that does not mean it is not interesting to the rest of the public.

RBI, while it can be (and often is) applied poorly to analyze players, at least is what it says it is. It’s not necessarily any more devious than hits. For whatever reason, everyone accepts that we should look at batting average instead of hit totals at the end of the season, but God forbid we try to adjust raw RBI totals for number of opportunities presented to a player. (Which, even then there are problems, but I digress.)

Other stats, like say the Game-Winning RBI, are not really what they purport to be. The GW RBI depends on one run being more than all the rest of them, which I assert is a dubious proposition. The only time an RBI can unequivocally be described as the game-winner is if you have a 1-0 game.

In a similar vein, saves imply first of all that the game was in some kind of danger, when there are quite a few save situations where the closer’s team has an overwhelming probability of victory with even an average pitcher on the mound. Past that, there’s no good reason to why the limits on the save are set in precisely the way that they are. Why not 4 runs instead of 3? Why not 2? Why require 3 innings instead of 2? Why not require 4 innings? Why have a long-save at all? A more honest stat would be something like games finished or even games finished in wins and games finished in losses. At least that would open the argument to “well pitcher X pitched in closer games than pitcher Y.”

Or for pitcher wins/losses, the pitcher is always being given more credit or blame that he deserves. The pitcher didn’t actually win or lose the game and there’s no reason to ascribe something like that to the pitcher. On top of that, the criteria for awarding pitcher wins/losses could easily be changed to something more reasonable (in particular, it never made sense to me that a pitcher who threw 4 good innings but had to leave due to injury should be excluded for consideration for a win) and tradition is the only reason to keep it the way that it is.

I appreciate that there is value in stats that are just for funsies, but it seems like we could easily do that and get rid of two or three especially misleading stats.

by ubelmann on Dec 4, 2008 1:50 PM PST up reply actions   0 recs

There are a lot of stats that have absolutely no use at all.

They exist only because people have an improper understanding of how to evaluate baseball players. That’s their sole purpose. If there hadn’t been a time when people thought RBI totals meant something, RBIs wouldn’t be tracked. So if they don’t serve an evaluative purpose, why keep them around?

by acblue on Dec 4, 2008 1:52 PM PST up reply actions   0 recs

I like keeping track of RBIs

But I sure as hell don’t use them to evaluate players.

Khalilbot

by JI on Dec 4, 2008 2:03 PM PST up reply actions   0 recs

Do you think you would care at all about RBIs if they didn't exist?

I guess that’s my point. I notice them too, but if someone hadn’t once thought you could evaluate a player by keeping track of them, no one would have ever started keeping track of them and you and I wouldn’t think twice.

by acblue on Dec 4, 2008 2:10 PM PST up reply actions   0 recs

but they do exist

And I find it amusing to look at a boxscore and see that a guy drove in 12 runs. Nothing more.

However, I think the sac fly rule should be done away with, and that you should get an RBI when you reach base on an error (if applicable).

Khalilbot

by JI on Dec 4, 2008 2:18 PM PST up reply actions   0 recs

Yes.

Also because I really like WPA/LI and if people were familiar with WPA it would be easier to get them to pay attention to it.

by acblue on Dec 4, 2008 2:36 PM PST up reply actions   0 recs

At the same time

The reason I dislike determining MVPs by WPA is that it’s not so different from using RBI totals. I find it baffling that people were seriously saying Berkman deserved to be the MVP.

Khalilbot

by JI on Dec 4, 2008 2:41 PM PST up reply actions   0 recs

WPA and WPA/LI are quite different

I agree with both of you; to me, it’s pretty odd to use WPA as a sabermetric “alternative” to RBIs or HRs.

But WPA/LI isn’t bad.

by marc w on Dec 4, 2008 2:48 PM PST up reply actions   0 recs

I see the point in using WPA in that way.

It shouldn’t be used for predictive value obviously, but it tells the story in a pretty clear way.

by acblue on Dec 4, 2008 2:51 PM PST up reply actions   0 recs

Maybe there should be a category of stats for storytelling

I mean, yes, it spins a decent yarn. But it’s every bit as context-driven as RBIs are; less dependent on batting order for hitters, but much more biased towards ‘closers’ and the ’pen.

I suppose it’s pretty easy to say that it functions as a wonderful check on sportswriters who claim certain players are ‘clutch’ because they made some play that was used in a Gatorade commercial, or they had a walk-off blooper against the team’s hated rival one time.

by marc w on Dec 4, 2008 2:57 PM PST up reply actions   0 recs

They beauty of WPA though is that it does exactly what it claims to do.

It doesn’t claim to be context-independent, actually the exact opposite. WPA to determine the MVP is the answer for anyone who takes the “valuable” in MVP literally. Without adjusting for defense or context or anything, if you want to know which player provided the most value to his team, fluke or not, then WPA is your answer. And it’s the answer.

by Matthew on Dec 4, 2008 4:12 PM PST up reply actions   0 recs

Totally disagree

Taking the “valuable” in MVP literally should not mean ignoring defensive value, which is precisely what WPA does.

On top of that, WPA adjusts for the importance of a situation within a game but not within a season. Why should we draw the line that way? Surely a game-winning hit against a competitor for your division title is more valuable than a game-winning hit against the Royals, right?

WPA is fun, and for that, I love it. I wouldn’t use it to make judgments about player value, though.

by ubelmann on Dec 4, 2008 4:28 PM PST up reply actions   0 recs

Seconded.

Yes, it claims to do what it says – but no one should use that measure to map to ‘value.’ Unless you can claim with a straight face that Bobby Jenks was a more ‘valuable’ player than Brandon Webb, you’ve got to admit that it’s simply not measuring ‘value.’
I’d agree that it’s not hiding anything, and that it has its place, but I can’t get my head around using it for MVP/Cy Young.

Re-reading your post, Matthew, it seems like my stumbling block is not so much the fact that it ignores defense (which is a pretty big deal), it’s that it’s NOT context-neutral. I think you can make a very similar argument about RBIs; it does what it says on the tin. If that’s what’s most important to you, go for it: it’s the answer.

by marc w on Dec 4, 2008 4:48 PM PST up reply actions   0 recs

In terms of actual production, nuetralizing the context makes no sense.

You would do that in order to obtain a performance-specific value. In terms of how much value a player provided to his team, context is ultimate. That’s what value is. And that’s why many of us hate the term.

by Matthew on Dec 4, 2008 9:06 PM PST up reply actions   0 recs

One small problem I have with using WPA

and I’m not really even sure its a problem. Its just something…

It assumes that every team is made up of entirely average players. When Albert Pujols comes up to bat, on average he will get a positive WPA because he is better than average which is what we usually think about. WPA also assumes that the players before and after Pujols in the lineup are also average. I haven’t really thought about this a lot but it seems like this could really effect the “actual” probability of winning the game.

by Edgar for Pres on Dec 4, 2008 9:46 PM PST up reply actions   0 recs

In terms of how much value a player provided to his team, context is ultimate.

Things I will never agree with: this.

But let me also say that since value has 20 different definitions, it’s just a bad title for the award. I guess, since the award is contract and age neutral, the guy who is just flat out the best player is the one I value the most.

by JI on Dec 4, 2008 10:23 PM PST up reply actions   0 recs

Especially now that I look back through the leaderboards over the past few years.

Once you take position and defense into account it pretty much nails the player that deserved the MVP in both leagues each year.

by acblue on Dec 4, 2008 2:49 PM PST up reply actions   0 recs

Before the dark ages before is ran into advance pitcher evaluation

I honestly have to say that I’d weigh a pitchers WHIP more heavily than I would their ERA, because it seemed clear that while ERAs were subject to randomness, a guy who could keep runners off base and had a lot of strikeouts seemed to more predictable going forward. (Obviously this method causes you to overlooks pitchers with the skillset similar to Brandon Webb). It also seemed reasonable to conclude allowing fewer baserunners led allowing fewer runs, and those who had allowed more baserunners than normal (say 1.3 per inning) but had posted a low ERA had gotten lucky. I know WHIP is stupid, but is caused me to ask “why?” And measuring baserunners allowed, it seems like it would give you a more of an answer to why pitchers give up runs than simply “because they did” which is basically the answer run average gives you.

Khalilbot

by JI on Dec 4, 2008 2:15 PM PST reply actions   0 recs

I'm not saying WHIP isn't pointless as a predictive measure

It is.

But if someone is looking at the “how” of runs scoring, “the pitcher that allowed the lowest on base percentage” is a better place to start than “the pitcher that allowed the fewest runs.” At least it attempts to tackle the process of scoring runs rather than simply accepting the total. That’s why i would say it’s not worse.

Khalilbot

by JI on Dec 4, 2008 2:38 PM PST up reply actions   0 recs

To me, WHIP is just combining two components and spitting out a number

Without and real regard for what they mean.

Try (BB+HBP+1B+2*2B+3*3B+4*HR)IP and I might look at it more favourably as a learning tool.

by Graham on Dec 4, 2008 2:46 PM PST up reply actions   0 recs

I don't think it's so bad

because it means that people are starting respecting OBP more, buy yeah it’s a bastardized version of OPS that needs to go away from any serious discussion.

Khalilbot

by JI on Dec 4, 2008 2:51 PM PST up reply actions   0 recs

Comments For This Post Are Closed


User Tools

By reading a game thread of your own volition you agree to accept all liability for any and all damage done to your delicate sensibilities.
Start posting about the Mariners »

Join SB Nation and dive into communities focused on all your favorite teams.

FanPosts

Community blog posts and discussion.

Recommended FanPosts

Small
FootbaLL III (Nov 7th)

Recent FanPosts

Ichiro_small
Ackley and Triunfel on MLB TV right now
Small
Xbox LLive, Pt 3
Clemente_small
OT on a friday morning 11/6
Curry_small
Curiously wondering what the plan is...
Griffeys_last_game_1_small
Is anyone else going to make fun of this picture?
Sth70021_small
Exploring Hit f/x, Albeit Badly.
Hunter_small
Somewhat OT: Kid dies from aluminum bat, family now rich
Small
OFFTOP: 10/30/09
Commproj08_ajones_small
Sounders Playoffs Gamethread

+ New FanPost All FanPosts >


Sexy People

1_small Graham

Small Matthew

Small Jeff