Two of the biggest political events of 2016, the British exit from the EU and the U.S. elections, have put a big question mark on the value of big data. Pollsters predicted a more than 71% chance that Hillary Clinton would win the elections, versus a 28.6% probability of a Trump victory. Across the Atlantic, the markets had predicted on the day of the referendum an 85% likelihood that Britain would remain in the EU. So what went wrong?
According to the Economist the explanation of the spectacular failure probably lies in the cognitive biases people have. These biases affect the survey respondents as well as analysts, prompting the former to express opinions they don’t actually hold and the latter to interpret data based on faulty assumptions.
The LA times, which was almost the only publication that predicted Trump’s victory, said their polls were successful because they identified and removed the social desirability bias, which was causing Trump supporters to be less comfortable about revealing their vote to telephone pollsters. The analysts’ judgment was probably clouded by confirmation bias, which causes researches to form hypothesis and beliefs and give more importance to facts that confirm their preconceived ideas.