Are you using your data correctly?
You may think you’re a data driven organisation, but are you using data to correctly inform product development, growth and optimisation? Are you using it to support the decisions you’ve already decided? Or are you truly using it to lead you to product and customer experience enhancements?
Chris Gibbins, our Chief Experience Officer was at MeasureFest Brighton where he made our case that a worryingly common pitfall in product management and growth is caused by a form of confirmation bias within the discovery phase.
A starting point for our shared mantra at Creative CX is that we believe its human nature to jump to conclusions and ignore evidence that doesn’t support our beliefs. Complementing this, people will actively seek evidence that (only) supports their own beliefs.
A good example of this is the 1969 moon landing. Over 50 years later, a long-standing conspiracy theory is that the moon landings were faked and Neil Armstrong only ever set foot on a studio set.
Theorists point to a lack of stars, and the US flag moving.
As it is, there are specific explanations that counter each of these observations.
Nevertheless, this is a conspiracy theory that persists to this day.
What is Confirmation Bias?
This is an example of Confirmation Bias, in which a majority of facts are discounted and only a small proportion of evidence that supports a starting assertion is used.
This is a big problem in experimentation and it can gain traction in an organisation very easily.
A client team can have an idea, then naturally look for insights. At this point, they may cherry pick insights that back up their hypothesis, before A/B testing.
A typical example of a cherry picked data insight which is in fact ‘reverse causation’:
As with the moon landings, there will be a reason for this finding: Visitors who are already bought in and interested, and this far into a product journey, will of course more likely watch a video. They are already invested and simply want to find out more.
How Confirmation Bias leads to low success rates
As we noted before, this sort of thinking can very easily gain traction in an organisation.
Teams know they need data to get their ideas prioritised.
And because there is so much data around, it’s easy to find data that supports a specific hypothesis.
This has made confirmation bias more prevalent, as data teams focus on validating ideas from the product team leaving a lack of resource for true discovery research.
In short, teams are missing out on impactful discoveries as they focus only on validation.
At Creative CX one of our Strategy Consultants, Tom Painter, coined an acronym for this repeated behaviour – FlLTH – derived from ‘Forcing Insights & Learnings To Hypotheses.’
How a ‘Discovery First’ approach paid off for this UK hotel website
At Creative CX we are firm advocates of a ‘Discovery First’ approach, where data is used to help set a course, rather than justify already determined outcomes.
Data leads us, from which we ideate a hypothesis, which we then A/B test.
We applied this mantra to positive effect for our UK Hotel chain client.
Prior to carrying out our Opportunity Audit discovery research no one thought there were opportunities to improve the Log in page.
The page already possessed several examples of best practice, with a fairly low exit rate.
Creative CX’s Opportunity Audit discovery research uncovered 101 opportunities, which we then prioritised for investigation – no conclusions were made at this point.
One of these opportunities related to the Log In page.
Usability testing uncovered a serious issue for the customer that wasn’t previously spotted:
In any e-commerce business, Checkout is a familiar component, coming at the end of a visit and replicating the real-life retail journey as a customer heads to Checkout with their basket of goods.
However, for a hotel this can be confusing.
On arrival, a guest checks into a hotel. On departure, they check out.
How then can a (potential) guest checkout when they haven’t booked, let alone checked into a hotel?
In hindsight, this observation seems obvious.
However, many leading hotel and travel websites use Checkout as part of their journey making the case that simply copying competitor best practices isn’t always the smartest thing to do.
With a strong problem statement and hypothesis, we were able to run a powerful A/B/n Test that explored 4 different variations to solve this problem. And the winning variation resulted in a statistically significant uplift in booking conversion rate, which was a fantastic result for the business.
We would never have realised this improvement without Discovery First research.
How to make ‘Discovery First’ part of your organisational thinking
- Conduct regular Discovery Research & Analysis projects throughout the year
- Follow-up with problem-focussed Ideation workshops to ensure a greater % ideas are data-driven
- Make sure your prioritisation framework gives higher scores to data-driven ideas than FILTHy ones!
- Track where the ideas come from & their success rates when A/B tested
Watch the full video of the talk here: https://www.youtube.com/watch?v=yB-znQyoujM
To find out more about how Creative CX’s Discovery First approach could work for your organisation, get in touch.
