How ‘survivorship bias’ can cause you to make mistakes

Effective decision making requires a detailed look at data. But it’s very possible you might not be seeing the whole picture.

This video originally appeared in BBC Reel’s Unconventional Guide to Success, a counterintuitive take on how to succeed in work and life.

A desire to learn from the successful is a natural instinct, but this can backfire if we don’t take into account ‘survivorship bias’. 

In simple terms, this comes about when we select only the ‘survivors’ – those that outperformed the rest, whether people, machines or companies – and come to conclusions based on their attributes, without looking more broadly at the whole dataset, including those with similar characteristics that failed to perform as well. 

The most famous example of survivorship bias dates back to World War Two. At the time, the American military asked mathematician Abraham Wald to study how best to protect airplanes from being shot down. The military knew armour would help, but couldn’t protect the whole plane or would be too heavy to fly well. Initially, their plan had been to examine the planes returning from combat, see where they were hit the worst – the wings, around the tail gunner and down the centre of the body – and then reinforce those areas. 

But Wald realised they had fallen prey to survivorship bias, because their analysis was missing a valuable part of the picture: the planes that were hit but that hadn’t made it back. As a result, the military were planning to armour precisely the wrong parts of the planes. The bullet holes they were looking at actually indicated the areas a plane could be hit and keep flying – exactly the areas that didn't need reinforcing.

Once you are familiar with the idea of survivorship bias, you can start spotting it everywhere. For example, a gym might feature those who’ve toned up quickly as a result of going to their facilities but, of course, what they never show is those who signed up but achieved no more than a depleted bank account. Another example is when a company is tempted to copy the strategy of a rival. A policy of “radical transparency” that includes blunt public feedback may work for Netflix and firing the 10% most “under-performing” staff might have been effective at General Electric. But before adopting any of these policies where you work, you should be wary of only looking at their use in successful companies. 

Scientists would seem a group of people that wouldn’t fall prey to the distortions of survivorship bias. But for years, many scientific journals have been criticised for a ‘publication bias’ toward studies that are interesting or that show significant results. Although it’s understandable that journal editors don’t want to use limited space to feature experiments where little happens, the result is an incomplete picture of the evidence. 

Leading by example?

A subtler source of survivorship bias appears when society turns its attention to successful individuals. Often our attention is drawn to people who achieve success ‘despite the odds’ or ‘take big risks’.

For example, a number of today’s billionaires – Bill Gates and Mark Zuckerberg, for example - achieved their success despite never going to or finishing university, a fact that has attracted considerable media attention. The New York Times reported on a “groundswell” of young people choosing not to go to university (the article featured a cartoon with the words, “College is for suckers”) and, in 2011, Silicon Valley entrepreneur Peter Theil launched an ongoing programme that awards $100,000 to young entrepreneurs who want to drop out of school. 

It’s easy to understand the appeal of stories like this. Hearing about successful people defy the odds is encouraging: if they achieved riches without university, it’s easy to believe, then surely I can, too. But this is another example of survivorship bias. If you look at all those who don’t go to college, rather than just the successful examples, a very different picture emerges. In 2018, the employment rate for graduates in the UK was 88%, and 72% for non-graduates. The median annual salary for a graduate was £34,000 ($43,000), and £24,000 ($30,000) for a non-graduate. Although university isn't necessary to be rich, looking at the whole picture rather than just the survivors makes clear that it does help. 

In fact, the danger in basing your understanding of the world on those who have ‘beaten the odds’ or have succeeded taking ‘big risks’ becomes clear if you carefully consider the logic of those phrases. Such people must be unrepresentative of the bigger picture, and thus one should be cautious of emulating them. After all, if everyone was succeeding by taking a big risk, it can’t have been that big a risk, nor the odds that daunting.

We are tempted to think success is due to particular characteristics which can be emulated. Just remember that, sometimes, success can also come down to luck.