4 Distortive Market Research Biases to Avoid

When conducting market research projects for our portfolio companies, OpenView Labs is expected to get to conclusive results at expansion-stage speed. Our portfolio companies want actionable insights to help them execute, and they want them yesterday.

This powerful sense of urgency makes it tempting for early-stage companies to cut corners in market research to get to results faster, and cutting corners only exacerbates the biases that are natural in these projects.

As much as market researchers like to consider themselves scientists, there is a fundamental difference between the social sciences (market research included) and controlled experiments. In market research, it’s not an option to reach a completely unbiased result; the only question is how much bias to tolerate and how to properly qualify your results to account for it.

The best option is to organize the project to minimize bias. When we can’t eliminate a bias from our project, we at least highlight the directional effect when we present our results to stakeholders: “Our research showed this, but here’s why it might not be completely representative.”

In particular, there are four common biases we make sure to look out for when conducting quantitative and qualitative market research projects:

1) Sample Bias Towards Customers

The Bias: Customers are the easiest interviews to schedule, they’re pre-qualified for relevancy, and there is plenty of information on them readily available in their CRM. It’s natural that they’d be the first targets when conducting interviews or surveys for buyer or user research.

The drawback from relying too heavily on customer input is that your results may be too complimentary to your own product and too tolerant of your company’s weaknesses. If your company’s strength is customer service but has poor integration with legacy systems, your customer base is likely to be more sensitive to customer support and have few legacy systems. That doesn’t mean these characteristics are representative of your target market though, and suggesting so can lead your company or client down the wrong path.

The Solution: We typically try to stratify our sample between three groups of market participants: cold prospects, current customers (or partners), and lost customers. While the mix depends on the goals of the project, a health balance insures you have a balanced perspective on your strengths an weaknesses.

2) Self-Selection Bias

The Bias: Our interview requests or surveys usually have response rates between 1% and 25%, depending on the specifics of the request. The fact that someone responded when many of their peers neglected to can say a lot about the person. Often, they have a good relationship or perception of your company. If there’s compensation involved, they may be the most price-sensitive in your sample. Regardless, it introduces bias into your data set that must be addressed.

The Solution: This bias is impossible to eradicate, since you can’t force a response. Still, there are ways to mitigate its effect. Consider putting more resources towards less responsive groups (such as higher-level respondents or lost customers), either by increasing the total pool of requests or the compensation offered. Don’t just rely on the low-hanging fruit.

3) Confirmation Bias

The Bias: In order to organize a concise interview or survey, you’ll have to go in with some idea of what you want to validate or invalidate. This can be problematic if your questions lead an interviewee to confirm what you already thought. For instance, if you believe price is the most important consideration in your interviewee’s purchasing decision, you may be tempted to ask, “How important was price in your purchasing decision?” to which they’ll almost definitely reply “very.”

The Solution: Lead with more open-ended questions that give the interviewee a chance to surprise you. Prompt them with possible answers and ask more specific questions later in the interview. Make sure to record whether an answer was prompted or unprompted.

4) Bias towards memorable interviews

The Bias: A typical primary research project for OpenView Labs involves 10-20 interviews, often spread out over a few weeks. Since memory research has shown a universal tendency to recall the first and last events of a sequence more clearly than the middle ones, your memory may be slanted towards a subset of your responses. Relying on your memory when presenting the results can in turn bias your results towards this subset as well.

The Solution: Your memory isn’t perfect so rely on it as little as possible. The more standardized your results are when you record them, the easier it will be to systematically analyze them, giving equal weight to all responses. Anecdotal results can be useful in building a narrative, but should always be grounded by hard data.

*****

Check out this link for more on market research biases.

For additional help, see my colleague Brandon Hickie’s tips on writing interview guides.