The Gambler's Fallacy

Fellow human beings, we are marvels at data processing but flops at data analysis. Specifically for this matter, I will concern myself and you with our predilection toward pattern finding. We are superb at it, a quite useful trait for survival and adaptation to a wide variety of climates. Planting these seeds at this time of the year yields better results than planting six months later. Huzzah! Easier farming leaves more food and more time for breeding. However, it's not all good news because left unchecked, that impulse runs wild and finds patterns in a whole mess of things where it has no business doing so.

Most things we deal with are adequate for pattern hunting. Investigating a pattern in traffic lights can help aid in finding the fastest commute home for instance. It works because there's actually a program controlling those traffic lights. They are not random; there's a design at work, no matter how much you might wonder at the competency of it when stuck in stop-and-go traffic. I believe that most of us (correctly) accept that there's no point in searching for patterns in something that is actually completely random. The tricky part is in getting people to agree that a system in question is actually random, or dominated by randomness.

The Gambler's Fallacy is the belief that in a situation with repeated, statistically independent (i.e. random), events (e.g. flipping a fair coin) that if results are different from your initial expectations (e.g. ten heads in a row) then it is more likely to see results in the opposite direction (e.g. "tails is due") soon. I wrote about the Gambler's Fallacy last year as well but am trying to write this one in a different way so hopefully one of them clicks. If you're still uncomfortable with the concept after this post, please read that one or someone else's attempt.

Falling victim to the fallacy is common in part because we're soft-wired to focus on outcomes and we expect them to average out, too quickly. Having some elementary statistical literacy is incredibly helpful in separating out when to go hunt for patterns and when to just not care and watch some quality Bravo TV programming. The crux is independence. No, not from Britain or from "foreign oil", but from each other*. Two events are statistically independent if the actuality (or not) of one event has no influence on the likeliness of the other event from happening (or not). In theory, what the die lands on, what the coin flips to, what number the ball on the roulette wheel stops at, none of these are affected by what the die, coin or ball finished at in the previous turn. A fair die has a one-sixth chance of landing on four no matter how many times it landed on four in the past. No matter how many times it has landed on four in the past.

*How Seattle

Baseball games are not die rolls* and they are not actually statistically independent. Past results do tell us something about future results. But it is easy to go too far in assuming how much they tell us. Like the evening news, they don't tell us much. Baseball games are overwhelmed by randomness.

*Unless it's Strat-o-matic!

Say you knew the Mariners were a 76-win talent team in the 2012 season. What's usually assumed and omitted is the sample size, in this case 162 games and therefore it's actually a prediction of a winning rate (47%) moreso than a fixed total. If baseball went on strike during the All-Star Break because everyone finally agreed that it deciding World Series home field advantage is just the dumbest idea, you wouldn't* start crowing about how all the forecasts were off on every team's winning totals. You'd understand that they projected rates and that what changed was the number of games (the sample) over which the rate was applied. Continuing the hypothetical, pretend the Mariners — in a manner that changed little to nothing about your opinion of the team's quality — lost their first nine games.

*I hope not. If so, you have problems. This is one of the many problems you have.

They certainly would seem due to win a game. It's not likely that a team would lose nine games in a row (2007 Mariners joke) and it seems even less likely that they would lose ten in a row (2011 Mariners joke). That's true, but once those first nine games are lost, your expectation for the next game (ignoring the context like who's starting, the other team, etc.) should remain at whatever you think the Mariners will perform at for the rest of the season. There's no special period where the Mariners become more likely to win games* in order to "make up" those first nine losses.

*You were going to make a "Except when facing the Astros!" joke weren't you? Did you just skip over the parenthetical about ignoring the opponent? What do you think this is, some sort of place for jokes?

Regression is another concept that gets a little tangled up in this and can lead to some confusion. Shouldn't the Mariners regress back to a 47% winning percentage you might wonder? If you haven't changed your expectation (key part!), then yes. But what regression entails is not a shift in winning percentage going forward; it's an expectation that the Mariners will perform at a 47% clip and that over time (another key part!), more games (samples) will move (regress) their total winning percentage closer to that original prediction (mean). It does not have to happen overnight. For example, the make-believe-Mariners are 0-9 now. If, in 17-game increments, they play 47% baseball (8-9), here's what happens to their record.

0-9 (0%)
8-18 (31%)
16-27 (37%)
24-36 (40%)
...
72-90 (44%)

The Mariners finish the season going 72-81 but started 0-9 for a final record of 72-90, but while they started with a winning percentage of 0%, it regressed toward 47% and finished our sample at 44%. They didn't rebound to finish the season at 47% because, why should they have? Theory says the Mariners should eventually end up at 47%, but it doesn't say it has to get there by a certain period. A baseball season is a totally arbitrary deadline from statistics' perspective. There's nothing intrinsically special about a 162-game sample.

What if we had a bigger sample? We'd expect the Mariners to continue winning 47% of their games and their overall record would continue creeping (regressing) closer and closer to 47% (the mean). Eventually — exactly when depending on your rounding requirement — the Mariners' winning percentage does reach 47%. At minimum it would take a whopping 774 games though. That's almost five whole seasons worth before the Mariners, winning 8 out of 17, overcome their initial nine-game handicap and end up at the expected rate.

A baseball season seems like a long time and 162 games is far more than any other domestic sport plays, but in terms of sampling, 162 is a paltry number when the individual games that make up that sample are so influenced by noise instead of underlying talent.