We are back for another roundup of my favourite conference of the year – Experimentation Elite! Different from last year, this time the conference was held at the Hilton hotel near Birmingham NEC, and was expertly hosted by our very own Chris Gibbins. The event took place over a couple of days, with the first day dedicated to workshops and the TLC summit, which I was honoured to be a part of.
Test and Learn Community Summit
On the first day we gathered for the TLC Summit, the first of its kind, which I was fortunate enough to be a part of. Six of us presented our short talks (maximum 10 minutes) to a panel of inspirational judges. Our talks were wide ranging, but, I felt, still suited well to each other. Kelly Wortham has created something special with the TLC, and I really recommend joining it and taking part in the discussions. We will be sharing more about this in a separate blog, so keep an eye out for a more detailed look at the TLC summit.
Els Aerts: 7 Deadly Sins of User Research
The second day of the conference kicked off with an engaging talk by this year’s Experimentation Legend, Els Aerts, who delved into the “7 Deadly Sins of User Research.” Els emphasised the growing importance of user research and the respect it commands in today’s data-driven world. She began with a striking analogy: bad user research is like mushrooms – some can be deadly.
Sin 1: Not Thinking Business First
Els stressed the need for intentionality in user research. It must be tied to business KPIs, ensuring actionable insights that can drive both learning and customer satisfaction.
Sin 2: Not Using the Right Research Method
Matching questions to appropriate research methods is crucial. Els warned against common mistakes, such as asking quantitative questions in user testing.
Sin 3: Biassing Your Research
Acknowledging that complete objectivity is impossible, Els highlighted the importance of being aware of biases, such as sampling and response biases, and taking steps to mitigate them.
Sin 4: Jumping to Conclusions
User research isn’t just about statistics. Els advised looking beyond percentages and recognising patterns, even if they are small.
Sin 5: Overlooking Smaller Patterns in Surveys
Small patterns can reveal significant insights. For example, beyond the obvious reasons for not upgrading, subtle feedback can uncover deeper issues.
Sin 6: Laziness
While AI can assist in research, Els cautioned against relying on it blindly.
Sin 7: Taking Research at Face Value
Els concluded by urging critical approach to research data, emphasising the importance of understanding the full context of data.
Stuart McMillan: Site Speed for CROs
Following Els, Stuart McMillan took the stage to discuss “Site Speed for CROs.” He explained how site speed impacts both testing and overall performance, underlining the importance of optimising web performance.
Key metrics for Google Judgement:
- LCP (Largest Contentful Paint): Must be better than 2.5 seconds.
- INP (Interaction to Next Paint): Must be better than 200 milliseconds.
- CLS (Cumulative Layout Shift): Must be better than 0.1.
Stuart underscored the importance of education teams about web performance, empowering them with tools and knowledge to influence site speed, and inspiring them with best practices are essential steps.
He also mentioned that the implementation of the back/forward cache can enhance user experience by caching pages efficiently. Managing resources through effective cache management and utilising new technologies like compression dictionaries and speculation rules further optimise performance.
Additionally, Stuart addressed the potential negative effects of A/B testing on Site Speed, potentially adding extra bytes and causing layout shifts, and provided strategies to mitigate these impacts.
Ellie Hughes: What to Do When Experimentation is a Dirty Word
Ellie Hughes‘ talk, “What To Do When Experimentation Is A Dirty Word,” explored the challenges of fostering a culture of experimentation within organisations. Ellie described how an experimentation culture should be driven by curiosity and data, rather than opinions. Inclusivity is key; everyone in the organisation should feel empowered to commission tests, ensuring all experiments adhere to ethical standards.
Creating a Culture of Experimentation
Maintaining high standards can be demanding, but it is crucial for staying competitive. Ellie shared the Red Queen hypothesis from ‘Alice in Wonderland’ as a metaphor for the constant need to innovate just to maintain equilibrium. She emphasised the importance of storytelling and change management in overcoming resistance to experimentation.
Solutions for Fostering Experimentation
- Sell Your Story: Use data to make your case compelling.
- Manage Change: Identify barriers to action and address them.
- Embrace Failure: Allow people to make mistakes and learn from them.
Stewart Ehoff: Beyond Testing: Scaling Success with Experimentation Systems & Operating Models
Stewart Ehoff‘s session on scaling experimentation was both insightful and entertaining, especially with his use of Dragon Ball Z analogies to drive home his points. Stewart stressed the importance of integrating experimentation into the product development process, rather than treating it as a standalone activity.
He illustrated that teams often start with experimentation as a “thing” outside the product backlog. However, to achieve true scale and impact, experimentation must be embedded into the daily workflows of product squads. This integration reduces friction, enhances collaboration, and ensures that insights from experiments directly inform product development.
Stewart shared his experience at RS, where they developed an experimentation operating system that included ticket tracking for tests, detailed metadata collection, and integration with UX and analytics. He highlighted the transformation from a centralised model to a centre of excellence, enabling various teams to conduct their own experiments while maintaining a cohesive strategy.
Paula Sappington: Research & Experimentation: Shift Left to Win Right
Paula Sappington from Hilton shared insights on integrating user research into the experimentation process. She explained Hilton’s journey from ad-hoc testing to a human-centred design approach, significantly improving their win rates.
User research significantly improves experiment wins, reducing risks related to usability, value, feasibility, and business impact. Combining quantitative and qualitative research provides reliable signals and contextual clarity, enhancing decision-making.
Paula emphasised the importance of exposing everyone to research findings and integrating qualitative and quantitative insights early in the design process. Involving stakeholders in the research process ensures that insights are aligned with business objectives and drives more effective outcomes.
She highlighted four types of risk (usability, value, feasibility, and business) and the need to address these through continuous testing and refinement.
Bhavik Patel: Not Another AB Testing Tool
Bhav‘s talk was a last-minute stand-in for David Maneheim’s originally planned talk. Despite the short notice, Bhav delivered an interesting presentation. Effective experimentation requires more than just following a simple set of steps. Bhav emphasised the importance of a strategic measurement framework. This framework defines what to measure and why, aligning experimentation with business objectives.
Building this framework involves defining the business model and objectives, developing KPIs and metric trees, implementing data collection and event schemas, and measuring baseline metrics. This structures approach ensures that experimentation efforts are targeted, measurable, and impactful.
Luke Carthy: Mastering GA4 for Optimising Conversion
Google Analytics 4 (GA4) is a powerful, yet complex tool that offers greater flexibility compared to Universal Analytics (UA). Luke Carthy explained that while UA was designed for marketers, GA4 is tailored for analysts, providing more detailed data and better customisation.
He compared GA4 to a supermarket – complex but flexible – and highlighted its advantage over Universal Analytics.
GA4’s emphasis on capturing all error messages and their impacts allows for a deeper understanding of user behaviour and site performance. A case study on Clarks demonstrated how GA4 was used to identify and fix poor-performing search items, showcasing its practical applications and benefits in optimising conversion rates.
Bjarn Bruenberg: OKRs to Strategic Experimentation
Bjarn‘s talk was a real fan favourite – bringing out a lightsaber, talking about using the force and challenging our own Chris Gibbins in a game of Rock, Paper, Scissors. Bjarn used these elements to highlight the importance of linking business OKRs to experimentation strategy, to overcome the strategy gap and improve the perception of experimentation internally.
He explained the importance of aligning decisions with financial outcomes through visual mapping of revenue to KPIs and OKRs, laid out in an easy to understand visual map, followed by easily recognisable OKR trees.
The overarching message was that understanding the opportunity space before devising solutions is crucial, and to use these trees to uncover similar opportunities within your business. This involves identifying customer needs and pain points through research, brainstorming solutions, and tracking their implementation. Their framework ensures that experimentation is strategically aligned with business goals and drives meaningful results.
Erin Weigel: Come in, We’re Open
In Erin‘s talk, she explored the benefits of drawing parallels between physical and digital customer service that can enhance user experience, drawing on our own past experiences within the physical space to run successful digital experiments.
Erin discussed how competing for visibility on platforms like Google and Facebook is akin to attracting foot fall on a busy street. First impressions are crucial, and using visual cues can help attract and retain customers. Similarly, brand alignment is important – if you do manage to secure a visitor to your site, they need to understand your offering and what makes you different from your competitors early in their visit, or they may leave.
Erin went on to stress that ensuring seamless navigation and engagement is vital, much like her beloved store, IKEA’s physical store layout. Motivating customers with incentives such as discounts can encourage purchases, making the digital shopping experience as effective and enjoyable as the physical one.
Finally, Erin took us back to her youth, and the excitement of receiving coupons in the post for her favourite store, but never remembering to take them with her. Luckily, they had the same vouchers in store, which made for a great user experience. The digital equivalent of which is presenting visitors with discounts on your website, rather than forcing them to search for them, and even automatically applying them so your customer doesn’t need to remember the code themself.
I loved Erin’s constant allusions to the real world examples, and connecting those into the digital experience – it got me thinking of my own experiences, and how they may apply to the digital world!
Shagun Aulakh: Tales from the Trenches
Shagun‘s presentation offered a candid look at the real-world challenges and successes encountered with a Centre of Excellence team, starting with the controversial point that they don’t always work, and you shouldn’t aim for one. The key to Shagun’s talk revolved around really thinking through the idea of a CoE and planning ahead of diving in – embedding experimentation within organisations requires strategic planning and execution. Shagun posed key questions about the fit and importance of experimentation within the company’s goals and the structure of the teams involved.
I found Shagun’s assessment of centralised and decentralised approaches really insightful; A centralised approach to experimentation provides stability and control, but may lack the flexibility of a decentralised approach. Conversely, decentralising experimentation allows for more individuality and flexibility, but requires higher maturity. Collaboration, evolving models, and quick wins are essential steps to demonstrate value and build momentum, ensuring that experimentation becomes an integral part of the organisation’s culture.
Shagun presented many tools for this, including visual maps, skills and timelines. One of the diagrams that really stood out to me, as it wasn’t something I’ve seen before, was a “talent dashboard” – mapping out your team and their strengths in various areas to understand where you may benefit from additional training or hires, as well as who is best places to lead certain tasks.
To Conclude
These talks were all well delivered and curated, reinforcing key insights and practical advice. It was fascinating to see all these concepts explored across various roles and industries, demonstrating their impact.
Experimentation Elite Summer 2024 was an inspiring experience that celebrated innovation from our community. We had a fantastic time and can’t wait for the next one in December that will be hosted in London.
It’s a great place to learn from others, hear what people in the industry are struggling with, and of course, networking with leaders from across the Experimentation world, fostering invaluable connections and insights.
