Telling Lies #7: Confirmation Bias

Female scientist doing Oil and gas research

Are you a Blue Pill or Red Pill kind of person?

Below I envisage the structure required in a research agency to eliminate confirmation bias 100%

One of the great enemies of market research the confirmation bias is also the enemy of good science.  Good science needs to make use of  carefully designed samples, control groups and double blind testing.

Samples must avoid confirmation bias.  If you send out someone to recruit a “random sample” off a busy shopping street with no rules they will come back with an awful lot of people who are of a similar age, ethnicity and social background as themselves.  Without conducting the first experiment they have affected the results.

A control group is a set of people who are tested without having experienced the subject of the test.  In market research we might expose a group to every product except the one we are testing.  In science we might use a placebo, or no treatment at all as a control.

The purpose of the control is to uncover test treatment errors to research.  These are situations where the setting of the test itself (such as a lab, doctors office, research facility etc) impacts on subject behaviour.  The test subjects don’t know they are changing behaviour because they are being tested, but they do.  The testers themselves may also be amending their behaviour, and consequent test results, unintentionally, but significantly.

The purpose of the double blind is to ensure that the tested person does not know if they are in the test group or the control group, and, more importantly, neither does the tester.

Lets say you have a new type of alcohol to test.  You give it to men and women.  They report the impact and complete some tests.

A classic confirmation bias is where a tester belives in advance (due to personal experience, or test assumptions, or any other reason) that the females in the group will react to the alcohol more strongly.  Without intending to falsify results they allow their own confirmation bias to impact on the results.  They “see” more effects in the females and the “edit out” effects in the men.  Their initial assumptions prove prophetic, they find exactly what they expect to find.

Control groups and double blind testing as well as “fair” samples can be expensive to run in market research.  In many cases the market researcher, or scientist, comes under subtle forms of pressure to relax the rules.

When it comes time to write up the findings the person paying the bill can call the tune.  Researchers and scientists are remarkably able to find conclusions that are exactly what their paymaster wants to see.  Commercial company managers like to work with market research companies that “understand our product and market”, and find the results that they hoped the research would find.

None of the lying is overt, but it is systemic.  Each bias builds upon the last one until the research results may land very far from reality.  Bad science!

Even highly rated scientific papers in peer review journals fail the test for confirmation bias.  Follow the money I say.  If it is a university paper they should be declaring their sources of grant funding.  Look up the funding agency, find out what their goals are and read up on the history of the research they funded in the past.  Always follow the money.

The Business Model:

I have a vision for unbiased research.  On the one hand are funding clients, companies, foundations etc who submit both the question, and the funding.  On the other hand are the front line scientists or market researchers.  In the middle are three layers of administration, with chinese walls between them.

Layer 1 is the business engine and has nothing to do with the research.  Layer 1 accepts the research applications.  Their job is to anonymise the research question and distance it from the source of the funding.

Layer 1 makes a submission to Layer 2.  The client on the job is Layer 1.  The research question is carefully set out to meet the real client needs, but never identifes the client.  Layer 2 receives a budget and a research task.  They make a proposal on how to carry out this task with the available budget.  They specify the sample, the test type, the control etc.  They submit a detailed proposal to Layer 1.  They are the research “pitch team”.

Layer 1 submits the proposal to the client, who approves or rejects the research.  If the research is approved Layer 1 gives approval to Layer 2.

Layer 2 engages a team from Layer 3 to carry out the research.  They are the field research admin team.  They pull from a pool of qualified front line researchers or scientists to perform the work required.  Layer 3 assembles the results and prepares the analysis.  They return the results, finding, limitations, conclusions etc to Layer 2.

Layer 2 run a cross check to ensure all research objectives have been met.  All being well the final report is given back to Layer 1.  Now you have a piece of research with no confirmation bias.  You present the findings, unvarnished to the client.

Then, and only then, if they want the findings “brought to life” will you pull together a team from all layers to make a presentation to the client.

This is all very bureaucratic and very expensive.  It gives absolutely no comfort to clients that they will get the results they are looking for.   But they will get proper research.

The one proviso of this model is that your organisation needs to be large and have so many clients that making “educated guesses” about who your client on any job is would be a waste of time.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s