On May 14th, our team had the pleasure of attending and sponsoring Quantum Metric’s LEAP into London 2024.
This event, themed ‘Simplicity reimagined. Efficiency redefined,’ was hosted in a stunning venue, providing a perfect backdrop for a day filled with insightful talks and networking opportunities.
We were particularly looking forward to kicking off the morning with an expert-led instruction session by Paul Wainwright, our Lead Strategy Consultant, who delivered a talk on “Insights-Led Product Experimentation.”
Insights-Led Product Experimentation talk
Paul delivered an engaging talk focusing on the crucial role of combining qualitative and quantitative data to identify customer needs and better understand the journey. Ensuring product teams are tackling the right issues through experimentation. The session was a deep dive into the challenges companies face when experimenting, and the practical strategies to overcome these hurdles.
Challenges in experimentation
While every organisation has their own unique challenges when it comes to running experiments, their objectives when engaging Creative CX can broadly be divided into three groups:
- Those with little A/B Testing experience who want to establish an Experimentation programme.
- Teams that are currently A/B Testing either as part of a small centralised CRO/Optimisation team, or simply validating features in the product teams, but want to ramp up their velocity and enable more teams to test – often by outsourcing development resources.
- Businesses who want to improve the quality of the Experimentation programme that they run – to increase win rates and overall impact.
Of course, none of these groups are mutually exclusive, particularly when it comes to increasing both the number and quality of experiments that are run.
In recent years, Creative CX has seen a big shift in the makeup of its clients, now working predominantly with Product and Optimisation teams, rather than Marketing. As a direct result, a lot of work that Paul and the team do is focused on enabling internal teams to become confident to run their own tests, and particularly to increase experiment quality.
There’s a whole host of factors that teams are quick to identify as the issues hindering their experiment quality, such as:
- Inability to build complex experiments.
- Platform issues and limitations.
- Poor comprehension of statistics.
- Difficulty in analysing experiments.
- The need to report ROI.
- Poor experimentation governance.
While these process improvements are relatively straightforward to identify, the more challenging aspect is implementing cultural change. Deciding which ideas to test, and how to prioritise them often requires external guidance.
Improving the experimentation process
A common place to start making improvements is by first mapping out what your process looks like, then identifying opportunities to improve.
Paul showed the following common workflow diagram and talked through how Creative CX often helps to improve the quality and efficiency of these processes.
While improving these steps can deliver significant improvements, an often neglected step is in selecting which problems the teams select to solve.
Delivering winning experiments
From years of supporting programmes for a wide range of businesses, Creative CX has identified two key factors that influence the success of experiments:
- Quality of Execution: High-quality design, usability, and code.
- Problem Understanding: Ensuring a complete understanding of the problem.
Which problems are selected for experimentation can often be biased by a few factors, such as the classic HIPPO, or too much focus on the competition. A troubling trend we observe across the industry is that of confirmation bias powered by data. Teams often start with an idea and hypothesis, then seek insights and learnings to back up their preconceived notions.
Example of confirmation bias
- Hypothesis: “Full screen auto-playing video on product details pages will enhance user experience.”
- Insight 1: “People who watch product videos have a conversion rate 4X higher.” (Reverse causation)
- Insight 2: “15% of users say videos are important.” …But what about the other 85%?
To highlight the issue of this bias, Creative CX ran an internal competition to coin a term that could be used to describe this behaviour. Senior Strategy Consultant Tom won the competition with his suggestion: “Forcing Insights & Learnings to Hypotheses’ (FILTH). It was a funny, yet insightful way to emphasise the pitfalls of confirmation bias in experimentation.
Preventing FILTH
Within his presentation, Paul outlined four strategies to prevent confirmation bias from contaminating experiments:
- Bridge silos containing customer research – Analytics, UX research, product, and marketing teams need to collaborate seamlessly.
- Schedule continuous research – Implement continuous opportunity discovery through user research, usability testing, data analysis, and ideation.
- Triangulate data points – Enhance accuracy and gain comprehensive insights by triangulating data from various sources.
- Prioritise problems first – Focus on identifying and prioritising problems before jumping to potential solutions.
Conclusion
Paul’s talk provided invaluable insights into the critical elements of successful product experimentation. By emphasising the importance of combining qualitative and quantitative data, he highlighted how understanding customer needs and journeys is essential for effective problem-solving. The session addressed common challenges in experimentation and presented practical strategies for improvement, particularly in overcoming biases and ensuring high-quality execution.
Creative CX’s approach to improving experimentation processes, from mapping workflows to enhancing problem understanding, offers a clear path for businesses seeking to refine their experimentation efforts. By preventing confirmation bias through strategies like bridging silos, scheduling continuous research, triangulating data, and prioritising problems, companies can achieve more accurate and impactful results. Ultimately, the workshop underscored that the key to delivering winning experiments lies in rigorous, unbiased analysis and a deep understanding of the problems at hand.
If you’re interested in learning more about how to enhance your experimentation efforts and achieve better results, feel free to get in touch with us.
We want to extend a massive than you to Quantum Metric for organising such a fantastic event. Our team thoroughly enjoyed the engaging talks, and some of us definitely enjoyed ourselves in the amazing after-party, which was the perfect celebration of experimentation and a great chance to connect with new people.
