Last week, a group of us from Creative CX attended Experimentation Elite Winter 2024 in London.
The day was packed with inspiring talks, thought provoking discussions and an amazing sense of community.
This year, you might have noticed our brightly coloured pink stand, proudly promoting our Problem-First Experimentation approach and championing the idea of Experimenting Everywhere!
At the stand, we ran an exciting competition asking attendees: Where else in your business could experimentation reveal untapped potential?
We were so impressed by the creativity shared, with participants pitching non-digital experiments they had worked on, or theorised.
In the end, Craig Sullivan and Oli Spencer’s innovative experiences and ideas took the top spots, earning them £250 Amazon gift vouchers each.
Throughout the day, the sessions explored everything from AI’s evolving role in experimentation to practical approaches for scaling testing and humanising the customer experience.
A special shoutout goes to our CXO, Chris Gibbins, who returned as MC. As part of his introductory, he opened the day with the “Pac-Man rule,” a reminder to always leave a gap in conversations to invite others in.
It was a simple, yet powerful idea that stuck with me throughout the day, encouraging inclusivity and the opportunity to meet new people.
If you couldn’t make it, here’s a recap of the talks with some standout moments, key themes, and takeaways from the day that I have no doubt will continue to shape the future of experimentation.
The Role of AI in Experimentation with Stephen Pavlovich
Stephen Pavlovich, CEO of Conversion, was the keynote of the day and started the talks discussing the evolving role of AI in experimentation.
Quoting Sam Altman’s claim that AI could take over 95% of marketing tasks, Stephen raised an important question: Should AI replicate what humans can already do, or take on the challenges we can’t?
He illustrated this idea with the story of the Mechanical Turk, an 18th-century chess-playing automaton secretly powered by a hidden human operator.
Stephen’s example reminded us that technology is most impactful when humans are in the loop, guiding and amplifying its capabilities.
Stephen shared how Conversion is blending human creativity with AI to assist with experimentation.
Tools like Conversion.com’s Confidence AI use machine learning to prioritise experiments and improve win rates, while Radius acts like a “second brain,” making it easy to find insights without losing track of the source material.
He also discussed the “Lever framework,” a structured way to scale and prioritise experiments based on user behaviour and available traffic.
However, Stephen emphasised that human intuition remains important. AI may assist with tasks like transcribing usability videos, but it can’t yet interpret subtle human reactions like a frown or eye roll.
His message was clear that AI isn’t here to replace us, it amplified human potential, letting optimisers focus more on strategy and creativity.
Building Genuine Customer Loyalty with Abi Hough
Abi Hough brought humour, heart, and humanity to her talk, Have I Got Your Retention?, blending personal stories and her sharp wit into how brands can grow genuine customer loyalty.
She shared an example of a project with a plant shop struggling to understand its customers.
Using the Keller Brand Equity Model, Abi walked us through a pyramid of customer connection, from basic recognition (who are you?) to deeper emotional loyalty (what about you and me?).
She also highlighted the dangers of viewing customers purely as data points: “we sit in our ivory towers made of data, splitting people into segments.”
Abi’s undercover research revealed the plant shop’s core audience: Boomers – who value loyalty, competitive pricing, and personalised experiences.
She shared actionable strategies to engage this demographic including a plant finder tool for the “plant murderers,” user-generated content like “get the look” to spark inspiration and calendar opportunities which helped to build community.
She stressed the importance of fostering long-term emotional loyalty through relational cohesion and transactional value, rather than short-term wins.
Her four R’s of Retention served as a memorable takeaway:
- Resonate: Build emotional connections.
- Relationships: Cultivate loyalty over time.
- Really Understand: Know your audience deeply.
- Reality: Stay grounded in what works.
Retention, Abi reminded us, isn’t about quick fixes. It’s about building lasting relationships and helping customers fall in love with your brand.
The Future of eCommerce Experimentation with Valentin Radu
Valentin Radu took the stage with the tricky pre-break slot, but his engaging insights kept the audience intrigued.
Drawing on his extensive CRO experience, he predicted how the future of experimentation in eCommerce lies in evolving both our methods and our mindset.
To boost eCommerce revenue, he explained, we must change customer behaviour – and to do that effectively, we need to experiment on how we actually run the experiments.
Valentin contrasted old approaches with new, customer-centric strategies:
- Old: focusing solely on visitor data and measuring success by conversion rates and revenue per visitor.
- New: Incorporating both visitor and customer behaviour data, emphasising customer lifetime value, and leveraging AI for ongoing customer experience monitoring.
Valentin also introduced the concept of evolving from the standard eCommerce flywheel (money – demand generation – customers) to the advanced flywheel including word of mouth – retention – feedback – better experiences and products. This holistic approach created more meaningful and lasting connections with customers.
Using the JTBD timeline, Valentin highlighted the importance of understanding the entire customer journey, from initial passive interest to post-purchase reflection. By looking into individual customer triggers, habits, and anxieties, businesses can refine their messaging and offerings to more effectively meet customer needs.
Perhaps the best standout point was his breakdown of customer types, from ‘Don Juans’ to ‘Soulmates’ and how RFM segmentation can identify customer behaviours. By combining this qualitative insight with quantitative validation, businesses can tailor their strategies to maximise CLV.
Valentin closed by exploring the symbiotic relationship between AI and humans in experimentation – a recurring theme throughout this year’s conference!
Understanding User Behaviour on Different Platforms with Dr. Hannah Tickle
A talk from Dr. Hannah Tickle is always a highlight, and this one didn’t disappoint.
With her background in experimental psychology – “working with brains before trains” – Hannah explored how context influences user behaviour across devices.
She explained that mobile apps, mobile web, and desktop are not just different platforms, but distinct environments with unique user motivations.
Apps, for example, are often downloaded by highly engaged users and cater to quick frictionless purchases
Hannah brought this to life with a quote from our CXO Chris Gibbins: “Even whilst running, I was able to search for the right destination, choose the correct ticket type, complete the purchase, scan the ticket at the barrier, and jump on the train.”
This perfectly captured the urgency and efficiency required by on-the-go commuters.
The session took an interactive turn, featuring a screen time exercise, demonstrating how deeply embedded our devices are in our lives.
When it comes to experimentation, Hannah encouraged this approach: test FAST, test LIGHT, test BIG.
Finally, hitting the “goldilocks spot” of information is key -not too much, not too little, but just the right amount for the platform and context.
This means carefully considering where and when to present the information depending on whether users are on the app, mobile web, or desktop.
Hannah’s talk was a perfect blend of practical insights, humour, and thought-provoking ideas, prompting us to rethink how we approach experimentation across different platforms and user behaviours.
Embracing AI in Creative Problem Solving with Iqbal Ali
Iqbal Ali’s talk was not only interesting, but also visually engaging, featuring many of his iconic comic-style illustrations, which added a fun and memorable twist to the session.
Iqbal raised an important point about how we often love to solve a problem.
He noted that “we pass on ideas like viruses”, instead of focusing on the actual issue at hand because there’s a lack of direct connection to the problem.
This particularly resonated with me as we’ve been helping our clients to implement a problem-first prioritisation approach.
This methodology focuses on prioritising problems first before jumping to solutions, ensuring that the focus remains on actually solving the customer problem.
You can read more about our problem-first approach here.
To reinforce this idea, Iqbal referenced an insightful quote from Einstein: “If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.”
Carrying on the theme of AI for the day, he also explored how AI can accelerate workflows and spark creativity.
By using AI, Iqbal noted that teams are able to generate more ideas and reduce solution bias, leading to a broader diversity and better quality of ideas.
However, during the Q&A Iqbal stressed the importance of validating AI output, as AI can sometimes “hallucinate” ideas that need to be checked.
His session wrapped up with a call to embrace AI in a balanced way – iterating, validating, and interacting to unlock real solutions.
Unconventional Copywriting with Eden Bidani
Eden Bidani’s talk on Unusual Copywriting Approaches That Drive Unusually Good Results was exactly the kind of talk every copywriter should attend.
She tore up the rulebook on traditional copywriting and showed us how doing the unexpected can deliver real results.
From the start, Eden tackled the common advice to “dumb down” your writing for accessibility.
Instead, she argued that copy should be tailored to the audience’s expertise.
In one example, she explained how using jargon-heavy language for a highly technical client actually worked better than simplifying it.
The people reading the copy were experts in their field, and they didn’t want to be talked down to. It was definitely a bold move – and it paid off!
The main takeaway here is that “The only predictable human behaviour is that we’re unpredictable.”
When everyone else is following the same tired template, why not stand out and surprise your audience? It’s all about testing the waters and experimenting with your copy, in the words of Eden: THIS IS WHY WE TEST!
She also touched on the power (and potential danger) of social proofing. People like to see that others trust a brand, but when it’s overdone, it can feel insincere. To support this theory, Eden showed us an old ad from Listerine that clearly played with some fun copy by plastering “I hate it, but I love it” bold on the front of the ad.
Then came possibly my favourite part of the talk, the concept of Pattern Breaks.
Eden challenged us to “show, don’t tell,” and proved her point by showing us how most brands’ copy looks almost identical – just rearranged phrases.
Instead of using the recycled content that floats around, Eden recommended us to start building a file of inspirational examples where doing things differently led to big results, like Patagonia’s “Don’t buy this jacket” campaign. This kind of bold, authentic message is the kind of pattern break that works as the messaging connects to the brand messaging and ethos.
During the Q&A section, Eden mentioned that whilst AI might be all the rage at the moment, relying too heavily on it could result in stale, repetitive copy.
To truly stand out, we need to embrace creativity and break the pattern.
LATAM’s Experimentation Journey with Rafael Buitrago & Marcus Vinicius Gonzaga Garcia
Rafael Buitrago, Experimentation Subject Matter Expert and Marcus Vinicius Gonzaga Garcia, Data Science and ML Team Lead, both at LATAM Airlines, shared how LATAM tripled its experiments per month – a transformation that was equal parts technical shift and cultural change.
Starting with the backdrop of a 90-year-old airline operating across 148 destinations, they shared the challenges of modernising legacy systems including outdated code, and culture.
Six months after the retirement of Google Optimize, LATAM faced an urgent need to prioritise experimentation without a dedicated platform.
This led to the adoption of a solution that wasn’t focused on tools, but also embedding experimentation into the company.
Moving from testing solutions to testing hypotheses became a key part of their strategy, supported by a centralised, collaborative governance structure that encouraged teams across the company.
Key roles, including product and business champions, subject matter experts, applied scientists, and data leads, were important in encouraging cross-functional experimentation.
The goal for Rafael and Marcus was to emphasise that experimentation isn’t reserved for product teams, it should involve everyone.
Marcus and Rafael also encouraged organisations to view vendors not as service providers but as true partners.
The pair also mapped out LATAM’s evolution in stages: crawl, walk, run, and fly (I particularly enjoyed the “fly” reference) – representing increasing levels of experimentation maturity.
Currently situated between the walk and run phase, LATAM has already expanded experimentation to multiple teams, implemented standard frameworks.
Rafael and Marcus’s talk showed what’s possible when experimentation becomes a mindset rather than just a method.
Creating Human-Centred Digital Experiences with Luiza de Lange
Luiza de Lange’s session reminded us that at the core of digital experiences, we all have one client: the customer.
Her approach challenged us to move beyond the surface-level interactions and focus on creating experiences that are simple, accessible, and genuinely great to interact with.
“To humanise something is to care,” she explained, quoting Frederick F. Reichheld: “Our primary purpose is to enrich the lives of our customers.”
Luiza also highlighted the impact of biases and bots, with bots accounting for over half of 2023’s internet traffic. While not all bots are harmful, they can distort analytics if not properly managed.
Beyond bots, cognitive biases like the confirmation bias and pattern-seeking tendencies can cloud judgement. Luiza recommended tools like the Lighthouse plug-in for accessibility checks to counteract these issues.
Accessibility emerged as a key theme for Luiza’s talk with shocking statistics that 70% of digital content remains inaccessible, and 71% of users with disabilities will abandon a website if they face difficulties.
With legislation like the EU Accessibility Act coming into effect by 2025, Luiza emphasised that it’s time for businesses to take this seriously and prioritise inclusive designs now – or risk legal penalties and missed opportunities.
Luiza’s session was more than a lesson in best practices, it was a call to care deeply about the experiences we design for customers.
Her talk reminded us that putting the human at the heart of digital experiences isn’t just good practice, it’s essential for progress.
Optimising Customer Journeys with André Vieira
André Vieira’s talk was exactly the boost of energy we all needed to wrap up the day before heading into networking drinks.
Focused on the customer journey optimisation (CJO), André broke down the process of mapping and improving every interaction a customer has with a business. He stated that one of the main keys to this is a clear system.
André’s CJO ‘Blueprint’ highlighted steps like setting goals, mapping context, gathering feedback, and identifying friction points.
André also introduced the four levels of CJO maturity:
- Foundational
- Observational
- Predictive
- Transformative
He explained how businesses at any stage can apply these principles, whilst scaling complexity as they grow.
A frustrating headphone purchase that André shared served as the perfect example of how overlooking the details in a customer journey can lead to confusion, frustration, and missed opportunities.
Highlighting the importance of tools like segmentation analysis, heatmaps, and A/B testing to uncover insights and optimise experiences, noting that the best journeys don’t just function – they make us feel great too.
As a wrap up to the end of the day André definitely left us with some sharp reminders:
Customer experience focused KPIs should always be a priority, journeys need to deliver more than just functionality. It was a high-energy finish to the day and a strong reminder of what it takes to create standout experiences.
See You Next Time…
What made Experimentation Elite stand out wasn’t just the impressive lineup of speakers, it was the sense of community that we’ve built around us and something to feel incredibly proud of.
The competition was definitely a highlight for us, with Craig and Oli’s standout ideas showcasing how creative and forward-thinking our great community is.
The conversations this year were fantastic, and we’re already excited to see what new ideas will come at the next Experimentation Elite in Birmingham, set for June 4th and 5th, 2025.
Can’t wait to see what’s next!
For anyone who is interested, our stands and pop-ups were provided by KGKgenix, a brand who shares our commitment to sustainability.
We were so impressed by the quality of their products, which are made in the UK using durable, recyclable corrugated cardboard and eco-friendly inks.
Their displays are built to last and can be reused, which perfectly fit with our environmental values without compromising on quality.
