During World War II, fighter planes would come back from battle with bullet holes. The Allies initially sought to strengthen the most commonly damaged parts of the planes to increase combat survivability.
A mathematician, Abraham Wald, pointed out that perhaps the reason certain areas of the planes weren’t covered in bullet holes was that planes that were shot in certain critical areas did not return at all, so their data was missing from the set of available data.
This insight led to the armor being re-enforced on the parts of returning planes where there were no bullet holes. This wisdom was also beneficially applied to the Skyraider during the Korean War.
This shows that the reasons why we are missing certain data may be more meaningful than the available data, itself. In questions of aircraft design, don’t only listen to what the evidence says, listen also to what is not being said.
This is today know as a common data bias, also referred to as the survivorship bias. We look at a certain dataset and derive insights from it usually forgetting that the available data might not give the full picture that needs to be analyzed.
For the same reason people in italy always comment that the old romanic bridges used to be much better then modern ones when they spot an old one that survived almost 1000 years. The problem is that we forget about all the other romanic bridges that did not survive until today….
Another example of a common survivorship bias is the claim “if I can do it, anyone can” that succesfull people, in any field, sometimes tell in interviews and biographic books. In this case the survivorship bias lies in the fact that we are not taking into account how many other people failed in the same process of pursuing the same success.
A common mistake we do in marketing is asking questons to a group of people that does not represent the entire spectrum of people (for example users of our own website). For example you can survey customers as soon as they have completed their order, however the type of questions you ask should not be “how easy/hard have you found the checkout process?”. If you ask them this question you are forgetting that there is another group: the ones that were NOT able to place an order on your website. You are asking the opinon only to the survivors (in this case of the checkout process) so the answer will be biased towards the “i made it” response rather then the “i couln’t make it” remark. It’s also a way to trick the survey results. If you want to have a higher NPS score this month, send an email to the loyal customers asking to fill in the NPS questionnaire 😉
Next week we’ll continue our journey though the different cognitive biases, in the meantime have a good week and happy analysis!
