Black Boxes and Stat Daemons
Jeff's note: I'm a little too...well it's Saturday and I'm disoriented, but I've been told this is another bit of required reading, so I'm giving it a bump. Go Graham go!
Famed science fiction author Arthur C. Clarke's Third Law states, "Any sufficiently advanced technology is indistinguishable from magic." Applied to the advanced baseball metrics we create, it could probably be paraphrased for the average reader (one unfamiliar with deep statistical musings), as, "Any sufficiently advanced baseball statistic is indistinguishable from a load of computer-generated bollocks."
To be fair, it is tricky to fault them for such a mentality. For most people, applying maths to baseball is neither easy nor particularly enjoyable - it takes a scientifically inclined mind to want to bother with this sort of thing. The stats crowd are, in essence, waving computer printouts and clamouring for attention in a space normally reserved for thoughts no deeper than 'Willie Bloomquist is such a gamer'. If I wasn't so fond of statistical analysis myself, I can see how this would be annoying (I used to think Willie was a good young prospect, after all). However, just because most people do not understand advanced statistical stuff doesn't mean they're dumb, or that they're irredeemable. Sometimes it's simply because we're not doing a good enough job of explaining ourselves.
To clarify with an example, one of the favourite targets of the traditional crowd is VORP (or it was a few years ago. Whatever). Why? Well, first, it sounds stupid, but that's neither here nor there. The real reason that VORP causes so much dissension is that it's actually quite difficult to understand cold. For those who aren't aware, VORP is a proprietary tool developed by Keith Woolner that essentially combines the concepts of runs created, replacement level, and positional adjustment (there's a great post on all this up on AN right now). Those are three very important ideas in modern analysis, and if you're aware of them and the basics of how they work then VORP makes some degree of intuitive sense, even if you can't see the equations behind it. If you're not, VORP looks like a box where information is fed in, processed, and spat back out. It could be the random scribblings of stats daemons living in computers for all anyone is aware. For those of you who prefer visuals (and for the sake of making this post look longer):

Black box syndrome is a huge problem. If something new is not explained clearly, concisely and transparently, chances are it's not going to be understood by a huge part of the baseball loving community. This is fine if you're not interested in reaching out to the folks who don't subscribe to sabremetric thinking, but there's a chunk of people open to new thoughts, who are completely capable of following a logical argument to its conclusion. By not providing them with any way of getting there, a black box stat is completely preventing these people from getting involved and interested in the conversation, and this simply fuels the disconnect and hostility between the two camps.
What's the solution? When doing analysis, we should spell out everything we do. Do we need the exact equations? No. Do we need to make it clear where we're making a positional adjustment, or what exactly we're talking about when you say 'replacement level'? Yes. Every single baseball stat (let's ignore Win Shares) is derived logically, and by not presenting data within a logical framework... well, the data without their scaffold are indistinguishable from complete nonsense, and when arguing against a point held dearly by the traditional lot, that's what it will be dismissed as.
There's also a need for increased information accessibility. Whenever a suitably advanced topic comes up in one of my classes at university, it's reviewed, or a reference is given as a means of learning it if one is behind. The same really should go for analytical posts. Jeff links to the win probability explanation in every game recap. Dave Cameron's article on pitching evaluation is an absolutely brilliant resource. It's a pain to have to reference things, yes, but if we're trying to educate people it's sort of on us to ensure that they have everything they need to understand what the hell is going on.
Will we win every battle? Of course not. But by doing more to make research accessible, we can only help the cause (do we have a cause?). We can and should make sabremetrics look a lot more like science, and a lot less like magic.
0 recs |
46 comments
Comments
I nominate this for movement to the front page.
A blog-thingy about the Mariners and stuff.
by BrettJMiller on Feb 15, 2008 5:11 PM PST reply actions 0 recs
Meh.
;-)
Awesome work here, Mr. Smartypants!
by PositivePaul on Feb 15, 2008 5:27 PM PST up reply actions 0 recs
Great stuff...but seriously
"By not providing them with any way of getting there, a black box stat is completely preventing these people from getting involved and interested in the conversation, and this simply fuels the disconnect and hostility between the two camps."
I agree. But what tends to foment hostility more than anything is out and out derision and...er, hostility. How do you square your laudable goal to provide more step-by-step information on the guts of various metrics with your role as LL n00b hazing coordinator?
More seriously, this is something that pisses off so many newbies either here, or at USSM, or at Tango's... the form of debate that many of these folks are used to is academic. The form that newbies are thinking they're walking into is conversational and off-the-cuff. Then, there's the whole 'taking it personally' thing.
From what I've observed, 90% of people won't go and read up on the new stat if they feel their intelligence/knowledge has been maligned. This is difficult because at some level this process is a requirement. Just sayin'.
by marc w on Feb 15, 2008 5:26 PM PST reply actions 0 recs
It's not like I flame ignorance
The thing is, here, we CAN have off the cuff conversation, even in posts where the heavy statistical lifting is going on. The community here is really, really helpful when people ask honest questions in a sensible way, and I don't think we hold people to academic standards.
I'm getting very tired so I don't even know if this comment was coherant. Hopefully it made a modicum of sense.
by Graham on Feb 15, 2008 5:39 PM PST up reply actions 0 recs
how about?
if you're ill-informed (in our opinion) and don't have an open mind --> we will flame you to dust
if you don't put a minimum of effort towards your communications --> we will assume you put equal amount (zero) of effort toward your ideas and thus ignore and flame you
by Matthew on Feb 15, 2008 5:44 PM PST up reply actions 0 recs
makes sense, Graham
If someone wants to really understand how, say, tRA works, they'll ask or they'll read documentation.
If someone thinks that all stats are just stupid and they distract from what's really important (RBIs and runs, baby!), then they're probably not going to put much effort into how tRA differs from xFIP - they will put more effort into mocking the acronym.
This is cynical, I realize. I just think open minded people who find baseball blogs are probably aware that there are advanced metrics in baseball. The problem you face isn't n00bs w/o perfect information; it's people who think the entire premise of analysis is wrong.
I should also point out that my point about academic debate wasn't that LL aspires to some mythic standard of debate where each argument must be supported by footnotes. The point is, academics reviewing a paper can be...harsh. There's a lot of 'this is wrong' and less 'sure! That might work too. I was just thinking that maybe X is better.' Maybe academic isn't the right word. I just think people are shocked when they see 'You're wrong.' And yet, as I said, this is essential (if/when they're wrong). We have off-the-cuff discussions all the time, but we're never going to have a jovial 'Is ERA better than tRA' conversation. I'm not saying we should.
We'll see. At the very least, it'd be nice to have a USSM-style link to offer people when they start talking about WHIP or something.
Ok, now I'm rambling and not making sense either.
by marc w on Feb 15, 2008 6:47 PM PST up reply actions 0 recs
I think you're right
by pdb on Feb 15, 2008 11:05 PM PST up reply actions 0 recs
Yeah, and that's a bad habit of mine
by Graham on Feb 16, 2008 1:39 AM PST up reply actions 0 recs
I wasn't accusing you specifically
It gets to the point, to me, where it's ridiculous - my favorite example is when some less-common metric points out that Given Player is mediocre, and some commenter's (n00b or otherwise) first reaction is "WHY DO YOU HATE GIVEN PLAYER?", which is not at all what was said or implied.
by pdb on Feb 16, 2008 9:50 AM PST up reply actions 0 recs
Right
I really don't think your curt dismissal of dumb posters here is the heart of the problem, or at least, not in 90% of the cases.
For a hell of a lot of people, rigorous analysis and being a baseball fan are actually diametrically opposed. This is stupid; this is life.
by marc w on Feb 16, 2008 4:25 PM PST up reply actions 0 recs
Yeah, you might well be right
by Graham on Feb 16, 2008 4:42 PM PST up reply actions 0 recs
FAO: marc w (outrageously off-topic)
I've just noticed your reply to one of my posts last week on something non-baseball related.
Would love to fill you in but don't want to burn space on here with it all - will happily give you a bit more info but prob best to email me - mark(at)mtedwards.co.uk
Cheers!
Sponsor of Jamie Burke's baseball-reference page
by MarkE on Feb 17, 2008 1:07 PM PST up reply actions 0 recs
We'll see
- Jose Vidro is worthless! PECOTA predicts his VORP to suck.
- What? He has like a .300 batting average! He's an awesome DH.
- BELIEVE ME FOOL
- Fuck you.
by Graham on Feb 16, 2008 1:36 AM PST up reply actions 0 recs
That post was already really long
by davidcameron on Feb 16, 2008 6:32 PM PST up reply actions 0 recs
I figured it was something like that
by Graham on Feb 17, 2008 12:45 AM PST up reply actions 0 recs
A applaud both you
Plus your magic show sucks--just some green boxes and a bunch of words I didn't feel like reading.
by Liebkartoffel on Feb 15, 2008 6:31 PM PST reply actions 0 recs
I like bacon
by wadswerth on Feb 15, 2008 6:39 PM PST reply actions 0 recs
How apropos
by salb918 on Feb 16, 2008 1:48 PM PST reply actions 0 recs
That's a really good piece.
Dave's post on expected Win Values is of a very similar vein, only focusing on the M's. Everyone should go read that too (I know you already have, salb918).
by Graham on Feb 16, 2008 3:02 PM PST up reply actions 0 recs
That was a nice article
by wadswerth on Feb 16, 2008 3:55 PM PST up reply actions 0 recs
I think another huge thing
These statistics are all used to PROJECT what MAY happen in the upcoming season. They are not used as PREDICTIONS, nor should they be taken as guaranteed baselines.
If someone reads a post that says "(Random Player) in 2008 is projected at .314/.422/.645" or whatever, people tend to want to read that as "using my shiny new statistic-ometer, I have generated numbers that guarantee (Random Player) will hit .314/.422/.645, and if he doesn't, then you're an idiot because you don't understand how the numbers came into being".
Which is not at all what the idea behind these numbers is.
by pdb on Feb 16, 2008 2:47 PM PST reply actions 0 recs
Foreign investors are taking over this blog.
by Robert on Feb 17, 2008 2:25 AM PST up reply actions 0 recs
This article
by wadswerth on Feb 16, 2008 5:51 PM PST reply actions 0 recs
Which begs the question:
by Double06 on Feb 16, 2008 6:11 PM PST up reply actions 0 recs
Because otherwise you couldn't read the text
by Graham on Feb 17, 2008 12:43 AM PST up reply actions 0 recs
I wasn't serious
by Double06 on Feb 18, 2008 9:45 PM PST up reply actions 0 recs
You can say more than 'I like bacon'?
by Graham on Feb 17, 2008 12:43 AM PST up reply actions 0 recs
When i feel like it i can.
by wadswerth on Feb 17, 2008 1:36 AM PST up reply actions 0 recs
I've gotta say
With the random bullshit, that is.
by Garces on Feb 16, 2008 8:11 PM PST reply actions 0 recs
Which is to say
by Garces on Feb 16, 2008 8:17 PM PST up reply actions 0 recs
VORP is just RC processed a bunch of times
by Graham on Feb 17, 2008 12:42 AM PST up reply actions 0 recs
The problem with this post is
Let me make this as simple as I can b/c it's late. When a knowledgeable baseball fan like myself looks at the situation without running any mathematical equations, here is what I see. I see a team that last year won 88 real life games with real life players playing against other real life teams. Whether you think they were extremely "lucky" (a topic that these projection models seem to dismiss since luck is not quantifiable), or more realistically, whether they were actually an above average team last year is hardly here nor there. The point is that they won 88.
And by any measure, they seem to be an drastically improved team in 2008 over what they had in 2007. Dramatically upgrading 2/5ths of the starting rotation while marginally downgrading RF and lefty setup guy is an improvement, no matter how much you hate the Bedard trade. Now, it may be that they are indeed a below average team this year, or a 75 win team, or whatever the numbers predict. If that's the case I'm sure you will all have a fine cigar. But for me, if indeed the M's are a good to very good team based on their new additions and previous results, then it seems that the predictive models you are using are wrong.
To clarify, I don't think the analysis is wrong, just the numbers used in the formulas in the first place.
by ryanb on Feb 16, 2008 11:42 PM PST reply actions 0 recs
using 2007 team results
It's simply bad practice. There's too many variables involved. You have to start from scratch.
by Matthew on Feb 17, 2008 12:02 AM PST up reply actions 0 recs
If we are not going to acknowledge past results
by ryanb on Feb 17, 2008 12:12 AM PST up reply actions 0 recs
Past player results are fine
by Jeff on Feb 17, 2008 12:17 AM PST up reply actions 0 recs
Yes, the M's won 93 games the previous year
Of course, the M's will do even better in 2004.
by G_ on Feb 17, 2008 12:22 AM PST up reply actions 0 recs
I'm amazed you could say that with a straight face
by Graham on Feb 17, 2008 12:41 AM PST up reply actions 0 recs
why not?
by ryanb on Feb 17, 2008 12:24 AM PST reply actions 0 recs
Baseball is made up of individual showdowns
Starting from scratch will yield far more accurate results.
by Jeff on Feb 17, 2008 12:29 AM PST up reply actions 0 recs

by 












