The first ‘Experimentation Elite’ event of the year took place on the 28th June in central London, and we were glad to once more sponsor this fantastic event, which brings together CRO and experimentation specialists across a range of industries. As an attendee of this extraordinary event, I had the privilege of immersing myself in the forefront of experimentation and emerging thoughts. From a thought-provoking keynote on the impact of AI, to an enlightening discussion on the impact of the interaction effect, Experimentation Elite provided an invaluable platform for industry experts, researchers, and practitioners to share their knowledge, experiences, and visions of the future.
In this blog post, I will highlight the twelve most compelling and enlightening insights I gained from the speakers at the event – stay tuned for an exploration of the latest trends, innovative strategies, and profound revelations that emerged from Experimentation Elite – an event that pushes boundaries, fosters collaboration, and sets the stage for the future of experimentation in a rapidly evolving technological landscape.
Craig Sullivan: Optimise or Die
The conference opened with a fantastic keynote address from Craig Sullivan, who had closed out the previous event in December 2022. His talk was very relevant to current technological discussions; focused on the topic of job security in relation to AI, specifically focusing on large language models (LLMs) and their capabilities.
Craig is always energetic and engaging, and this talk was no different – taking us through the current landscape of AI, demonstrating the way big companies are currently using AI as well as highlighting examples from 50 years ago. Overall, the talk explored both the hype and reality surrounding LLMs and AI, and the underlying message was simple and reassuring; AI tools are meant to augment human capabilities, rather than replace us entirely, enhancing productivity and filling gaps rather than creating new ones.
Or, as Craig put it, “AI isn’t going to replace you – another person using AI to become more productive will.”
Erin Weigel: Design for Impact. A participatory model for conversion design
Craig was followed by the fantastic Erin Weigel, who discussed the concept of conversion design, its importance, and provided an overview of the conversion design process.
Erin’s talk was incredibly informative, but my key takeaway was the consideration of Conversion design as more than just visual aesthetics; it encompasses the purpose, target audience, problem-solving, functionality, and construction of a design.
As Erin said, “design is seen as the manifestation of intent”, requiring a combination of design, science, and business considerations. The ultimate goal is not just to create different designs but to make improvements that enhance the user experience and achieve better outcomes.
Erin also highlighted the importance of iterative testing, focusing on the quality of results rather than the quantity, although this isn’t always the easiest concept to highlight to a wider business.
Luke Frake: Optimising Optimization
The final talk before our first refreshment break was Luke Frake from Spotify. Not only did Luke produce some excellent visuals using MidJourney, but he also very clearly highlighted the importance of program metrics, areas where improvements can be made, and emphasised the significance of prioritising people and celebrating success.
Most memorably for me, Luke equated the idea of iterating on one hypothesis to visiting a beach – there are only so many clothes you can take off before your embarrassment outweighs your comfort, and in the same way there are only so many times you can follow a single hypothesis before new problems will start to outweigh the one you are trying to solve.
Luís Trindade: All-in for server-side experimentation?
Luís gave a very informative talk on the need for careful consideration when choosing between client-side and server-side approaches, and the benefits of feature toggles, including risk mitigation, targeting capabilities, dynamic configuration, and enhanced experimentation. By leveraging feature toggles, teams can achieve faster development cycles, reduced errors, and increased flexibility in feature activation.
For me, the main takeaway was the necessity of choosing the right tools your team, but also starting with the right mindset.
David Mannheim: Personalisation Squares: Choose your own Adventure
David kicked off his talk in a very personal way, showing a clip of his 15 minutes of fame on TV, before sharing his Disney obsession with us. While his talk was engaging and entertaining, it also highlighted some very key points of personalisation that are worth considering.
While personalisation has been a bit of a buzz word for the last few years, it has started to lose it’s meaning – with large companies forgetting about the person. Communication has focused more on relevant content, rather than content which resonates with the audience, and many companies are now using personal information to serve personalised ads and sale opportunities, rather than improving the experience for the customer.
Overall, David’s talk emphasised the complexities and challenges of personalisation, the importance of genuine care, and the need to understand customer intent; the future of personalisation lies in building meaningful relationships and connecting with customers on a personal level, as people don’t want to be sold to – they want to be seen.
Kenda Macdonald: Conscious conversion – The art and science of banishing the cognitive bias that kill conversions
The final slot before lunch is a risky one, as many audience members are often more focused on the prospect of food rather than the information they are receiving. Fortunately, Kenda was full of energy and did a fantastic job of keeping the room engaged in her talk and the topic at hand.
Kenda discussed the idea of System 1 and System 2 thinking “Thinking Fast and Slow” as Daniel Kahneman coined it), and how this relates to decision making and bias, whether intentional or not. It was particularly memorable due to her reference of ‘minions’ (the yellow characters), as well as how it’s important to market to this style of thinking rather than the System 2, ‘Gru’, style. By leaning into consciousness rather than coercion, keeping messaging ‘minion-friendly’ (simple and relatable), providing explicit information, and highlighting key points, audiences will find it easier to understand the content, and therefore will feel more favourable and increase the chance of parting with their money.
Chris Gibbins & Natasha Senior: Horizontal Experimentation for Seamless Customer Journeys
As risky as the pre-lunch slot can be, the post-lunch slot is equally tricky to get right, or else you risk sending the audience into a food coma! The honour of this slot went to our very own Chris Gibbins, along with the fantastic Natasha Senior from Sky, with whom we’ve been working very closely.
Chris and Tash took us through the challenges Sky has faced with vertical teams and how they are aiming to overcome this via horizontal experimentation, and cross-journey teams. Notably, they highlighted the importance of establishing technical foundations, focusing on end-to-end journey research and optimization, and avoiding gatekeeping within experimentation. By keeping these aims in mind, the program aims to improve the seamlessness of journeys, foster a strong experimentation culture, and drive impactful results through horizontal experimentation, although challenges in tech, politics, and departmental coordination need to be addressed to facilitate these improvements.
Ruben De Boer: Habits, personal growth & company-wide experimentation
Ruben’s talk emphasised the importance of personal and team growth in creating a better world through experimentation. He stressed the importance of focusing on both hard and soft skills to drive an optimised experimentation program, which applies to teams as well as individuals.
For the individual, forming habits can increase your chances of making long term change. Utilising the idea of SMART and STEP, for your System 2 and System 1 brain respectively, you may find these habits easier to create and follow through.
For the team, Ruben posited that it is key to address both how and why change is important. Making it easy, granting autonomy, and providing a sense of purpose contribute to team motivation, as well as tweaking the physical and social environment, establishing rules and norms, and tying rewards to experimentation results are also effective strategies.
Combining both of these principles into becoming a likeable authority is key. To lead effectively, it is important to be passionate yet humble, approachable, show interest, understand and align with others, build trust, and avoid directly telling someone they are wrong.
Devon Boyd & Paul Wilkins: Optimising beyond conversion
Devon, from AB Tasty, and Paul, from Conversio, took to the stage to echo similar sentiments from previous talks, focusing on the importance of understanding the customer and their motivations to effectively improve the digital experience. However,they also put their own twist on this idea, demonstrating some of their own case studies that went beyond traditional digital experiences and encompassed omnichannel journeys. One particular case study that stood out to me was the inclusion of a bespoke delivery option for a high-end fashion brand, where a concierge would hand deliver the clothes, as well as some additional pieces for further upsell/cross sell opportunities, who would then wait so the customer could immediately return any unwanted items. Although the feasibility and desire for the service is limited, the experiment helped to demonstrate the importance of delivery considerations for the consumer.
Devon and Paul’s case studies demonstrated the effectiveness of personalized experiences, diverse product representation, and unique delivery propositions in improving conversion rates, average order values, and customer satisfaction. By focusing on customer-centric approaches, businesses can navigate the current trends, overcome challenges, and achieve better outcomes in their experimentation programs. Ultimately, only by understanding your customer’s emotions can you make changes that matter to them.
André Vieira: Optimising Key Moments in Customer Journeys – A Practical Blueprint
André presented one of the most polished presentations of the day, which he had apparently created less than 24 hours previously after the original file had been corrupted. The talk was full of information, presented in the form of a clear plan laying out the key moments in customer journeys.
According to André, optimizing customer journeys requires understanding and addressing the barriers that customers may encounter along their path. The A/C/R (Acquisition/Conversion/Retention) Funnel framework highlighted the importance of context in the customer journey and the need to consider multiple touchpoints, and the concept of momentum as a crucial component for a smooth journey, avoiding either rushing or frustrating the customer.
This led to the topic of friction, which can be addressed by creating a Friction Barrier Map; where heuristic analysis, user testing studies and other qualitative research methods can help classify the identified friction barriers and how to address them.
By following this mental model and blueprint, businesses can effectively identify and address barriers in customer journeys, leading to smoother experiences and improved conversion rates.
Lukas Vermeer: How To Run Many Tests At Once: Interaction Avoidance and Detection
The day ended with the esteemed Lukas Vermeer, who began with some meta analysis of his own talks, as well as the common structure of many other talks (the so-called ‘Hero’s journey’). This segwayed neatly into the topic of his keynote as this is something he is often asked about in reference to scaling experimentation – “what about the interaction effects?”
Lukas’ response was resounding, and reassuring; “Not a problem.”
Interaction effects are generally rare and severe ones are easily avoidable or detectable. There are a number of ways to prevent them, which you may already be doing (e.g running A/B/n tests rather than just A/B), which means that the problem with interaction effects comes more from worrying about them, rather than the effects themselves. This worry can undermine trust in experimentation and reduce the velocity of your program, having a larger impact than a potential interaction effect (which would likely be caught).
In Summary: The future of experimentation
The previous Experimentation Elite event in December had many talks about improving experimentation culture within a company, so these talks were very different! It was clear to see the common theme throughout the talks, which all danced around the personal side of optimisation and putting the customer first. It feels like experimentation is going back to basics from this perspective, whilst retaining the same robust practices and blueprints that have been developed throughout the years. The inclusion of more psychological research will only further this knowledge and improve the way we are optimising.
