Telling Lies #7: Confirmation Bias

Female scientist doing Oil and gas research

Are you a Blue Pill or Red Pill kind of person?

Below I envisage the structure required in a research agency to eliminate confirmation bias 100%

One of the great enemies of market research the confirmation bias is also the enemy of good science.  Good science needs to make use of  carefully designed samples, control groups and double blind testing.

Samples must avoid confirmation bias.  If you send out someone to recruit a “random sample” off a busy shopping street with no rules they will come back with an awful lot of people who are of a similar age, ethnicity and social background as themselves.  Without conducting the first experiment they have affected the results.

A control group is a set of people who are tested without having experienced the subject of the test.  In market research we might expose a group to every product except the one we are testing.  In science we might use a placebo, or no treatment at all as a control.

The purpose of the control is to uncover test treatment errors to research.  These are situations where the setting of the test itself (such as a lab, doctors office, research facility etc) impacts on subject behaviour.  The test subjects don’t know they are changing behaviour because they are being tested, but they do.  The testers themselves may also be amending their behaviour, and consequent test results, unintentionally, but significantly.

The purpose of the double blind is to ensure that the tested person does not know if they are in the test group or the control group, and, more importantly, neither does the tester.

Lets say you have a new type of alcohol to test.  You give it to men and women.  They report the impact and complete some tests.

A classic confirmation bias is where a tester belives in advance (due to personal experience, or test assumptions, or any other reason) that the females in the group will react to the alcohol more strongly.  Without intending to falsify results they allow their own confirmation bias to impact on the results.  They “see” more effects in the females and the “edit out” effects in the men.  Their initial assumptions prove prophetic, they find exactly what they expect to find.

Control groups and double blind testing as well as “fair” samples can be expensive to run in market research.  In many cases the market researcher, or scientist, comes under subtle forms of pressure to relax the rules.

When it comes time to write up the findings the person paying the bill can call the tune.  Researchers and scientists are remarkably able to find conclusions that are exactly what their paymaster wants to see.  Commercial company managers like to work with market research companies that “understand our product and market”, and find the results that they hoped the research would find.

None of the lying is overt, but it is systemic.  Each bias builds upon the last one until the research results may land very far from reality.  Bad science!

Even highly rated scientific papers in peer review journals fail the test for confirmation bias.  Follow the money I say.  If it is a university paper they should be declaring their sources of grant funding.  Look up the funding agency, find out what their goals are and read up on the history of the research they funded in the past.  Always follow the money.

The Business Model:

I have a vision for unbiased research.  On the one hand are funding clients, companies, foundations etc who submit both the question, and the funding.  On the other hand are the front line scientists or market researchers.  In the middle are three layers of administration, with chinese walls between them.

Layer 1 is the business engine and has nothing to do with the research.  Layer 1 accepts the research applications.  Their job is to anonymise the research question and distance it from the source of the funding.

Layer 1 makes a submission to Layer 2.  The client on the job is Layer 1.  The research question is carefully set out to meet the real client needs, but never identifes the client.  Layer 2 receives a budget and a research task.  They make a proposal on how to carry out this task with the available budget.  They specify the sample, the test type, the control etc.  They submit a detailed proposal to Layer 1.  They are the research “pitch team”.

Layer 1 submits the proposal to the client, who approves or rejects the research.  If the research is approved Layer 1 gives approval to Layer 2.

Layer 2 engages a team from Layer 3 to carry out the research.  They are the field research admin team.  They pull from a pool of qualified front line researchers or scientists to perform the work required.  Layer 3 assembles the results and prepares the analysis.  They return the results, finding, limitations, conclusions etc to Layer 2.

Layer 2 run a cross check to ensure all research objectives have been met.  All being well the final report is given back to Layer 1.  Now you have a piece of research with no confirmation bias.  You present the findings, unvarnished to the client.

Then, and only then, if they want the findings “brought to life” will you pull together a team from all layers to make a presentation to the client.

This is all very bureaucratic and very expensive.  It gives absolutely no comfort to clients that they will get the results they are looking for.   But they will get proper research.

The one proviso of this model is that your organisation needs to be large and have so many clients that making “educated guesses” about who your client on any job is would be a waste of time.

 

Insight

It may seem like a basic question, but when I ask “what is an insight?” I get a plethora of answers.  So I set out to answer the question myself.

When I started my career it was very much in the realm of data processing and information retrieval.  Data as the raw material and information as the analysed and summarised outcome.   I then moved to the qualitative side of market research back in the days when we called it research.

Somewhere along the way data processing was seen as too old fashioned, and everybody wanted “data mining” and “big data expertise”.  Qualitative market research findings were no longer sexy.  They had to be “Insights”.

Over the years I have seen a lot of simple data, summarised information, behavioural observation, behavioural understanding and product improvement which were presented as “GROUND BREAKING INSIGHTS”.

So here is my simple view of the world:  “If it doesn’t change consumer behaviour it’s not an insight.”

The short version of this article is that Insights need to be behavioural, emotional, true (credible), relevant, original, ownable and measurable.

The World of Compromise

We live in a world of limits and compromises.  There are many things we would like to do better.  When somebody shows us a better way we adopt it quickly.  Many of the greatest inventions in history are so obvious once seen that a common reaction to them is “Why didn’t I think of that?”

The insight comes from seeing how people behave and understanding that their behaviour is a compromise from the ideal.  The inventor then leverages the insight to produce the product that changes behaviour.

Recently I worked on a project with a major packaging manufacturer.  They spent a day in a busy bakery observing workers in action.  They noticed two things in particular.

  • Existing product (a frozen part-baked bread range) was stored in large cartons that were very heavy to lift. They needed to be removed from the freezer to open them.
  • Once lifted out of the freezer the staff were reluctant to put them back in. Product left in the hot kitchen began to thaw and spoilage rates were high.

The engineers set about addressing these two insights.  They designed a new freezer carton which could be opened in-situ in the freezer.  The staff could remove only the product they needed for immediate baking.  This innovation changed how the staff in the kitchen behaved.  It made life much easier for them, so it qualifies as insight.

The change also reduced the levels of product spoilage.  This is process improvement (but not necessarily insight).  It improves profit levels for the client.

There is a nice roundedness to this outcome.  The Client makes more profit, and is consequently more likely to work with the packaging company again.  The staff have an easier time in work, so they are happy with the change.  The customers of the bakery are less likely to receive a sub-optimal product, so they will enjoy their bread and come back for more.

The world of needs and wants

Anyone who studies marketing 101 learns about Maslow’s Hierarchy of Needs.  As you rise on the pyramid you move from needs to wants.  If you are purely needs driven then you are unfortunate in modern society.  Most humans have moved beyond a daily struggle for Water, Food, Shelter and Security.

We live in a world of choice, which is good in one way and bad in another.  Everyone, even the richest billionaire, faces resource constraints.  In simple terms there is more “stuff” out there than we can afford.  If you want it all, and want it now you will be disappointed.  You have to make trade-offs.

The early trade-offs are easy.  Do you eat food this week or do you take a spa day?  Starving people don’t take spa days.  If you are needs driven then the need wins out.

In a wants driven society there are many people who forgo food and trade it off for a day at a spa.  They call it a diet, or a detox.  They don’t “need” food, so it becomes something they can trade off.

In the world of needs and wants “Insights” are clues to how trade-offs will work.  This is the realm of Economic Behaviourism.  It is a weird and wonderful place where people frequently make sub-optimal decisions which make no sense on paper.

In this world your best clue that you are dealing with actual “insights” is emotion.  Insights are born in the Freudian Id, what popular psychologists refer to as the Inner Child, the Primitive Brain or the Lizard Brain.

If your research uncovers useful data you will see people nod sagaciously as they consider how to use the findings in the business.  They will see the relevance of your findings to others, but not usually to themselves.

If your research is insightful your audience will be excited, emotional and immersed.  You will hear phrases such as “that is sooooo true!”  “That is so me!”  “I do that all the time” etc.  It is real, truthful and personal in a way that data and information never are.

Paying the piper

Insights are fantastic as long as the client can use them to make money.  As a result there are a few boxes you have to tick when you present your insights:

Relevance:  they have to change consumer behaviour in relation to your client’s product.

Originality:  there is no advantage to being the second company to leverage an insight.

Ownership:  if your client can own the insight territory this has potential for huge market share gains.  Most innovations are easily copied by the competiton.  Branding is less easy to copy.  Insights and Branding are two peas in the same pod.

Measurability:  I have seen brilliant insights that have come to nothing because they could not be applied to the customer record data.  It is pointless having something that changes the lives of 25 year old female insurance buyers if the client does not collect customer age and gender in the sales process.

Truth is a voyage of exploration

Vasco

On December 16th in 1497 Vasco da Gama passed the mark set by Bartolomeo Dias in 1488 and rounded the Cape of Good Hope, sailing into the Indian Ocean and history.

Each Portugese explorer tried to build on the knowledge and achievements of his predecessor and stretch the boundary of the known world further with each voyage.

The outcome of decades of exploration was to open the door for da Gama to exploit the Indian Spice Route.

Academic research adopts the same approach. When you write a research thesis you begin with a review of all the available literature in your field. You establish the base of existing knowledge and then you seek to expand it further with your research project. If we do not know what already exists then we waste time reinventing the wheel.

When we educate oppressed people to the possibilities that exist in other societies we open the door for them to liberate themselves. A regime that limits knowledge is a regime ruled by fear of the possible. An open information policy is a sign of a healty regime.

In the modern world you can quickly identify which regimes are open by looking at their approach to regulating the internet. That will give you a very strong indication of how they control freedom of the press and the education of the people.

A nation that shrouds itself in a blanket of ignorance is protecting the vested interests of the current regime. No such nation can sustain this stance into the long term. The truth will emerge and will set the people free.

The child is not dead; by Ingrid Jonker

The child is not dead
The child lifts his fists against his mother
Who shouts Afrika ! shouts the breath
Of freedom and the veld
In the locations of the cordoned heart

The child lifts his fists against his father
in the march of the generations
who shouts Afrika ! shout the breath
of righteousness and blood
in the streets of his embattled pride

The child is not dead not at Langa nor at Nyanga
not at Orlando nor at Sharpeville
nor at the police station at Philippi
where he lies with a bullet through his brain

The child is the dark shadow of the soldiers
on guard with rifles Saracens and batons
the child is present at all assemblies and law-givings
the child peers through the windows of houses and into the hearts of mothers
this child who just wanted to play in the sun at Nyanga is everywhere
the child grown to a man treks through all Africa

the child grown into a giant journeys through the whole world
Without a pass

Listen Hear

Talktohand

Do you practice listening?  If you don’t then you probably aren’t listening very well, unless you are one of those strange and wonderful creatures born with the natural propensity to listen actively to what others are saying.  For most of us frail humans active listening is a chore.  It is easier by far to listen to the sage wisdom that spews from our own consciousness.

However, brilliant though you may be, there is only one of you.  The collected wisdom of mankind does not reside in your head, so from time to time make the effort to listen to others.  They may be saying something important.

To help you here are some techniques for turning people on, and for turning people off.

1.  Active listening body language

Just the way you sit or stand can encourage, or discourage the person who is speaking.  To be an active listener you need to monitor your body language to make sure your posture is open and welcoming.  You should make good eye contact.  Lean forward, angle your head slightly.  Avoid folding your arms and crossing your legs as these are physical barriers.  Don’t lean or sit back (seems like you are in judgement), and avoid rubbing your chin or steepling your fingers.

2.  Non-vocalised encouragement

Nodding is good, you are saying “Yes!  Go on!”  Shaking your head in appropriate places is also powerful, as you can be saying “so sad” or “how unbelievable” etc.  More difficult to gauge is making physical contact, as it is highly cultural, contextual and personal.  In the right situation holding a hand, touching a shoulder or giving a hug can engender great empathy, trust and an outpouring of emotion.  In the wrong situation it will rapidly shut down communication and cause offense.  It could even get you arrested.

3.  Vocalised encouragement

A far safer area of encouragement than tactile reinforcement is verbal reinforcement.  This can range from simply making encouraging noises (Ah Ha, Um, Oh), through actual words (Yes, I see, and then?) to full blown sentences (So what did you say to him then?)

4.  Driving the narrative

Part of active listening is allowing the other person to tell their story.  Very often we hear the beginning of a story, which reminds us of something that happened to us, and we launch into telling our story.  The original story is lost along the way.  The active listener is focused on hearing the whole story.  To help it along they may drive the narrative forward.  I find the best way to do this, without changing the story, is to repeat the last sentence you heard.  Try it, it is very powerful.  It reinforces to the narrator that you are paying attention, you are interested and you want to hear more.

5.  Colouring in the narrative

Some people tell a great story, and others need help.  If a story is very dry, it may be because it is devoid of adjective, descriptor, emotional context etc.  You can help the narrator to build the story with some well-placed cues to colour in the landscape.  Examples might be along the lines of:  Who was with you?  How did that make you feel at the time?  What was your first reaction?  Did they seem to be under pressure?  Did you have a meal in the café? Etc.

If you want them to build and build, then preface these cues with the words “Yes, And…”

6.  Killing the narrative

The opposite of “Yes, And…”  is “Yes, but…”.  When you use the word “but” everything positive that you just said is cancelled out.  If you want to stop someone speaking there are many ways to do it.  Disinterested body language, absent eye contact, tell them they are wrong, laugh inappropriately in the wrong place.  Some can be subtle and others are obvious.  But sometimes you need to keep someone quiet.  If you are actively listening and someone else tries to interrupt you generally get the message across with a blocking gesture, hand up, palm out towards the interruption.  A slow, loose handed version acts as a “Shhhhh” whereas a firm, flat handed version, held for a second, with full eye contact, is an emphatic “shut up”.  This can then be reinforced by turning to the original speaker, and saying “Go on, you said”….and repeat their last sentence.

In group situations (focus groups, meetings etc) when one person becomes annoying and will not allow anyone else to speak it is perfectly acceptable to be very straight with them along the lines of  “We know your opinion, now we want to hear the opinions of others, can you be quiet and listen for a while now please?”

Come to think of it, people are always saying that to me.  I wonder why?

Trumpet Player; by Langston Hughes

The Negro
With the trumpet at his lips
Has dark moons of weariness
Beneath his eyes
where the smoldering memory
of slave ships
Blazed to the crack of whips
about thighs

The negro
with the trumpet at his lips
has a head of vibrant hair
tamed down,
patent-leathered now
until it gleams
like jet—
were jet a crown

the music
from the trumpet at his lips
is honey
mixed with liquid fire
the rhythm
from the trumpet at his lips
is ecstasy
distilled from old desire—

Desire
that is longing for the moon
where the moonlight’s but a spotlight
in his eyes,
desire
that is longing for the sea
where the sea’s a bar-glass
sucker size

The Negro
with the trumpet at his lips
whose jacket
Has a fine one-button roll,
does not know
upon what riff the music slips

It’s hypodermic needle
to his soul
but softly
as the tune comes from his throat
trouble
mellows to a golden note