Why Most Pollsters Are Delusional (And Often Dangerous!)

Be wary every time a pollster tells you that they are measuring public opinion.  They are probably delusional and possibly even dangerous.

Just look at the abysmal state of U.S. politics or the environmental movement and you’ll see what I mean.  Everywhere there are charlatans who claim they have “measured public opinion” because they conducted a survey, sent out a push poll, or otherwise asked people how they think or feel about an important topic.

I call these people charlatans because a massive amount of psychological evidence gathered over decades of time all points to the same conclusion — people do not know how they think or feel about a topic!!!  Rather than going through reams of experimental findings that all point in this direction, I will simply offer up this excellent TED Talk by Nobel laureate Daniel Kahneman:

Note how there is a profound difference between how a person experiences the world and how they later remember it.  This is just one example of how the human mind is a complex mesh of unconscious processes that culminate in a very incomplete conscious awareness.  This is why opinion researchers misguidedly talk about “behavior gaps” between what people say they will do and what they actually do.

These gaps are not fundamental to human nature.  They are artifacts of faulty interpretation by the researcher.  So when anyone tries to tell you there is a behavior gap between what consumers say they will purchase (e.g. a green product that is good for the environment) and what they actually purchase, note how the researcher is assuming that what someone said they would do is supposed to be a reliable indicator of future behavior.

The reality is that human minds are structured so that conscious choices are constrained in profound ways by neural computations upstream in the perception process.  By this I mean that their conscious experience is felt as a seamless gestalt that does not include all the preprocessing that was required prior to entering awareness as an integrated whole.  For example, as you look around the room you see objects with coherent meanings (e.g. couches, chairs, people, cats, and so on) but have no awareness of the photons hitting your retinas or the edge and color detection that filled in these shapes for you in the visual cortex of your brain.  All of that neural computation took place outside conscious awareness, and for good reason.  Had you been aware of all that you would quickly become paralyzed with information overload and become incapable of taking effective actions in your environment.

Furthermore, the “higher reasoning” faculties we humans have are recent additions to our cognitive heritage.  Our emotional sensibilities are processed in the middle and lower regions of our brains first, before arising as constraints in the “motivated” reasoning of our much newer neocortex at the top and front of our brains.  What this means is that our reasoning is like the rider on an elephant who sometimes fools herself into believing she is in control.  Until, that is, the elephant decides it wants to do something different.  We are prone to misremember things, be overly critical of information that challenges our preconceived notions, make claims that help us feel like we are morally good people in the eyes of others, and much more — all of which makes survey results less reliable in the absence of robust methodological tools for measuring these biases.

The real tragedy of charlatan pollsters comes out when attempting to make headway on important political issues.  How will the United States take leadership on the climate crisis?  When will we put in place a universal health care system?  Why don’t we have education policies that reflect the best knowledge available about excellent teaching?  Who is going to take the lead on cleansing our federal government of corporate corruption?  Part of the answer to these questions is that too many politicians and activist groups have drank the same Kool Aid as their pollster and believe the delusion of survey results to be a true reflection of public will.

While I have much to say about how we conduct social research differently with legitimacy and rigor, that will need to remain as another blog post.  For now, I just want to be clear that anyone claiming to know what the public is willing and able to do based on the simplistic notion that they merely need to be asked how they feel is going to get us nowhere (as it already has… see the lack of progress on climate, health care, education, and electoral corruption for all the evidence you need).  We have serious challenges before us and must get these things right if we are to restore the faith of our citizens in the prospect of effective government and cultivate the levels of cooperation needed to innovate our way to solutions for the myriad global problems confronting all of humanity today.

Before we can do any of this, we’ll need to get our assumptions about human behavior right.  And so far the majority of pollsters are just not getting the job done.  We can do so much better than this!  And now, thanks to all the advances in cognitive and social science, we have the foundational knowledge from which to correct our misguided assumptions and get it right.

Cognitive Policy Works specializes in providing organizations and individuals with frame analysis, policy briefs, strategic advising, and training.