What Is Survivorship Bias
During World War II, the U.S. military examined bullet holes on returning bombers and planned to reinforce those damaged areas. Statistician Abraham Wald objected. The areas without bullet holes on returning aircraft were precisely the locations where hits proved fatal, preventing return. This is the classic illustration of survivorship bias.
Survivorship bias is a form of selection bias that arises when only successful cases remain observable while failures vanish from the dataset. Ranking data is open to this bias as well.
Survivorship Bias Hidden in Rankings
Some people aspire to entrepreneurship after studying billionaire rankings. Yet these rankings exclude those who started businesses in the same era and failed. Concluding that entrepreneurship leads to wealth by observing only survivors is the same error as armoring bombers based solely on returning aircraft.
Income rankings suffer similarly. Data from high-income countries is relatively accurate, but in conflict zones and regions of extreme poverty, collecting data is difficult in the first place. Rankings built from "those who could be measured" can leave out the most disadvantaged populations.
Survivorship Bias in Health Data
Articles about "habits shared by centenarians" are perennially popular but represent textbook survivorship bias. People who practiced the same habits but died at 70 never enter the study. Traits common among survivors are not necessarily causes of longevity.
The relationship between BMI and mortality is also affected. Studies of elderly populations sometimes find that slightly overweight individuals live longer, but one explanation is that thin elderly people include those who lost weight due to illness. This does not necessarily mean obesity is healthy; reverse causation may be at work.
Using Rankings While Acknowledging Bias
Completely eliminating survivorship bias is difficult. However, recognizing its existence changes how we interpret data. The questions to ask when examining any ranking are: "Who is not included here?" and "Why are they absent?"
MyRank's rankings are no exception. Anyone using this tool has internet access and the means to use it, which already limits users to a specific stratum of the world population. The ability-test percentiles are built from a reference distribution and users' records, and those on the far side of the digital divide are absent from those records. Maintaining this awareness forms a foundation of data literacy.
Thinking Strategies to Counter Bias
Practical thinking strategies exist to counteract survivorship bias. First, imagine the counterfactual: when observing a success story, estimate how many failed under identical conditions. Second, check the base rate: use the overall success rate as your reference point rather than the success rate among those with a particular attribute.
Third, examine the selection process: consider why specific data reached you and what filtering it underwent. Success stories appearing in social media feeds may have been algorithmically selected for high engagement.