Building an Experimentation Centre of Excellence (CoE) for YNAP, a two year journey

Background

Yoox Net-A-Porter Group (YNAP) is a prominent online luxury fashion retailer renowned for its visionary approach to the digital shopping experience. YNAP operates through four key brands; YOOX, Net-A-Porter, Mr Porter, and The Outnet.

YOOX offers a diverse range of luxury and contemporary fashion from multiple brands, while Net-A-Porter is celebrated as a premier destination for high-end designer apparel, accessories, and beauty products. Mr Porter caters specifically to men, offering an extensive selection of sophisticated fashion and lifestyle items. Lastly, The Outnet presents fashion enthusiasts with a curated collection of discounted luxury fashion from leading designers.

The Engagement

YNAP was considering a new experimentation platform to provide greater testing capabilities and performance improvements. They engaged CCX to help them evaluate the new platform.

CCX demonstrated the value and performance of the new platform by successfully building a complex client-side A/B test to a very high standard. YNAP were impressed because the previous supplier had been unable to build this particular A/B test.

Following the successful proof of concept CCX was selected to onboard the tool across all of its websites and Apps, and to support the four brands with a revitalised Optimisation programme.

CCX provided full end-to-end Experimentation services working in close collaboration with the two main points of contact in the Optimisation team.

Kicking things off with an Opportunity Audit with usability testing and data analysis, CCX uncovered many opportunities across key user journeys which fed into ideation sessions and a new refreshed roadmap.

By improving the quality of hypotheses thanks to the triangulated quant and qual data insights from the opportunity audit, and by increasing the velocity A/B tests thanks to new processes and improved development provided by CCX, the team were able to achieve excellent results.

However, there was a limit to the amount of impact possible within the current operating model of a Centralised Optimisation Team working separately from the Product Teams.

While CCX helped validate a handful of the product teams’ features, most feature releases, page redesigns and App updates were still being released without A/B Testing.

One validation A/B test of a so-called ‘no-brainer’ solution ended up being particularly impactful, but not in the intended way. This unexpected result highlighted to the business the importance of testing every idea even if they were ‘no-brainers’.

In addition to a few showcase A/B Test examples raising the prominence of the Optimisation Team, a change in senior leadership also helped to elevate the importance of experimentation.

It was now crystal clear that every team on every digital platform should be innovating, optimising and continually improving the customer experience through Experimentation.

The Challenge

The Centralised Operating Model has its place for many organisations and can be the best way to get started with experimentation, prove the value, get teams on board with the idea of experimentation and get extra investment for a more substantial approach.

However, there are serious limitations with this structure many of which were affecting YNAP, including:

• The Optimisation Team were trying to improve the same parts of the journey that the Product Teams were responsible for, causing clashes in terms of releases and objectives.
• Product teams were being asked to launch features without the opportunity to run A/B tests.
• Product teams were driven by output rather than outcomes. I.e. they were more like feature teams rather than empowered product teams.
• Product teams were unable to report accurately on how well new features or redesigns had performed because most of these were not A/B Tested.
• Designers were hesitant to consider multiple solutions to problems and were playing it too safe. Without the safety net of A/B testing, how could they take bigger risks?
• Developer and tech teams had some bias and mistrust against A/B testing and the validity of results.
• The whole organisation was missing out on the full potential of Product Experimentation at scale.

Due to CCX’s collaborative partnership with YNAP, our previous experience of scaling experimentation programmes, onboarding product teams onto server-side testing and also helping large organisations to move to more effective operating models such as Centre of Excellence’s, CCX was given the opportunity to take on this challenge.

The solution

CCX worked with YNAP to review the different possible operating models. After discussing all of the details it was finally agreed that a Centre of Excellence model would be the best fit for their current setup and the most effective way to achieve their objectives for scaling experimentation and driving more impact.

The team were currently run as a centralised model, but as discussed this has inherent challenges which prevent scaling. Conversely, completely decentralised models can cause issues with ownership, consistency of processes and overall governance. CCX didn’t want experimentation to become the Wild West which is what can happen with complete decentralisation!

The CoE Experimentation Team model empowers each Product Team with the autonomy to test all potential ideas, designing and building their own experiments. Enabling them to focus on outcomes over output, but with close support and guidance from the CoE team.

As CCX moved forward together, it was critically important to gather senior buy-in at this stage, which we were pleased to be able to do.

Combining this top-down drive for impact with a bottom-up empowerment of the teams supercharged the delivery of the solution, and enabled us to make cultural change.

The Delivery

Once we had architected the solution, delivery took the form of a 5 pillar Centre of Excellence (CoE).

By producing a program of work in each of these key areas, we were able to break the project down into clearly scheduled and achievable work streams:

• Tooling & Platform
• Process & Governance
• Training & Culture
• Programme Maturity
• Support/Tech Strategy

Training & Culture spotlight

A particular focus for this project was the delivery of meaningful training which could impact culture and drive lasting change to how experimentation is conducted.

Our consultants delivered sessions based on experimentation strategies, as well as technical and platform training sessions.

This project was delivered at a time of easing lockdown restrictions, making it necessary to handle training remotely. The rollout was international and met with great feedback.

As part of the sessions, we were also pleased to leave behind a knowledge base which included large-scale written resources and over 180 pages of value-added intellectual property. These resources included how-to guides and best-practice advice designed to leave the customer as self-sufficient as possible.

Documenting a New Model is Not Enough

It’s important to note that running training sessions and creating supporting documentation in isolation is not enough to ensure long-term change.

Critical to the success of this training was the Support programme delivered by the CCX CoE Support team that went through 3 key phases:

1. Pro-actively engaging all Product Teams, to challenge and validate their approaches and experiment design.
2. Regular check-ins with Product Teams to help with any questions and continue validation.
3. Drop-in sessions to help with any questions, plus a focus on measuring Team and programme maturity and identifying areas for continuous improvement.

From cynicism to advocacy

As with any long-term project, there are bumps in the road to navigate, and these twists and turns form the unique shape of the implementation.

Initially, we were very focused on aligning priorities and making sure we were able to continue the momentum of the project across multiple departments.

In order to achieve this, we had to think outside the box and find new ways to increase buy-in. One of the highlights of the project was the use of gamification during the bi-monthly “Experimentation days”. We used these group sessions to bring Product together and win advocacy within the most important teams.

We asked project managers to be problem-orientated, presenting their experiments as a way to solve problems rather than simply test features out. Each team member pitched their ideas to the group, with prizes for the winners.

Later these results were presented in product forums, critically, with the output built into the roadmap.

Mission accomplished

The project was able to achieve its most ambitious stretch targets. Moving 98% of experimentation to server-side, enabling product teams across the full breadth of the company, and ultimately delivering a significant measurable uplift in revenue.

The project touched more than 180 members of staff, from the top to bottom of the organisation, and was able to test more than 5.8X more ideas than before our engagement, saving money by avoiding poor decisions, and bringing in additional business to drive ROI.

Then in December of 2022, the team had the pleasure of taking to the stage with Ibrahim from YNAP to tell a very engaged audience about this journey, what we learned and what we achieved along the way.

The CCX Strategy Consultants and Technical Consultants continue to work within the CoE team providing ongoing support for all their product teams, helping the organisation to continually improve their product experimentation capabilities.