Experimentation can result in some pretty incredible changes, changes that improve the experience for your customers and your profit margins. But while the experimentation process is exciting and its results insightful, effectively managing an experimentation programme and its lifecycle is demanding and can often be viewed as a little dull. Yet this is a crucial element of experimentation and one that can determine the success of your experimentation programme. Without it, you are never going to be able to scale your programme or achieve a high velocity which ultimately determines the success of your programme.
Having a defined lifecycle separates elite experimentation teams from the rest. Figuring out a lifecycle that best suits you and your team can be a long process, but it is certainly a rewarding one. At CCX we have a lot of experience in helping our clients define a workflow and governance that suits their internal business and overcome the challenges they face. Below are some tips and tricks that we have used in the past.
But first, what is our experimentation lifecycle process?
At CCX, we follow a standardised lifecycle process for each experiment – with a few adaptations if the experiment or the client we are working with calls for it. Our current seven-step process helps provide us with a structured approach to experimentation, ensuring we can deliver results and insights to our clients in a timely and efficient manner.
Whilst our lifecycle may look simple, it encompasses all the different steps involved in the experimentation process.
Benefits of defining an experimentation lifecycle
We’ve already touched upon some of the benefits pre-defining an experiment lifecycle brings, but there are plenty more reasons you should create a standardised process across your team. Mistakes brought about during experimentation within your process can often undermine the experiments themselves, invalidating the results of past, present and future experiments.
A defined lifecycle allows you to outline the who, when, what and how of our experiments. From conception to completion, being clear on the following is key to any successful experiment:
- Who is involved within the experiment?
- What are the objectives of the experiment?
- What learnings will we gather from this experiment?
- Is our experiment aligned with the wider company goals?
So how do we make sure we answer these questions?
Rather than creating our own system to answer these all-important questions, we use the RACI matrix, which provides clarity about each experiment.
RACI Matrix
A RACI matrix is a project management chart that allows us to assign roles and responsibilities for each task or decision within the experiment. It maps out the roles involved in each project and defines each team member’s level of responsibility.
RACI stands for Responsible, Accountable, Consulted, Informed. Each letter in the acronym represents a level of task responsibility, as below:
| Responsible | Performs the work to complete the task |
| Accountable | Delegates work and is the last one to review the task or deliverable before it is deemed complete |
| Consulted | Provides input based on how it will impact their future project work or expertise on the deliverable itself |
| Informed | Kept in the loop on project progress, but not required to be involved in the details of the task |
So, for example, we might decide that the current experiment would work best using the following levels of task responsibility at each stage.
It also helps ensure that even when you’re working on multiple experiments at once, you’ll be able to know at a glance which elements need your attention most. This is important as your experimentation programme’s maturity grows and more teams across different departments get involved.While this process may seem overwhelming and time consuming, it ensures all those necessary in experimentation understand their role. Even if not directly involved, it helps everyone in the team to adopt an experimental mindset which can help lay the foundations for a culture of experimentation within your business. It does so because it keeps everyone involved in the project informed, even if they aren’t directly involved in the experimentation process, which helps everyone in the team adopt an experimental mindset.
Seven top tips for successfully managing an experiment lifecycle
But with so many moving parts involved in the experiment process, which components should you focus on? Here are our top tips for managing any successful experiment lifecycle.
1. Define your process early on
Ideally, the workflow and management process should be defined before any of your experiments even begin. This helps you start off on the right foot. You should also get into the habit of tailoring your process to each experiment before you start each new project.
In this way, you can rest assured that you have a clearly defined process before you even begin. Whilst this might seem like an unnecessarily painstaking step at first, it’ll become second nature for everyone involved to follow the process and refer back to the workflow or RACI matrix when necessary. This will ultimately mean that less time is wasted while you experiment – meaning more time to focus on strategic elements of your tests.
2. Strategically plan who is involved in the experiment
As mentioned earlier, not all stakeholders should be involved in every single task. This is often a waste of time and resources for all parties involved.
Think of it this way: if you give three people one task, they’ll complete the task in three different ways based on what they think is right. This not only highlights the importance of limiting the parties involved in each experiment, but it also shows just how important it is to have a specific process in place to ensure each task is being handled efficiently. Without a process in place, disorganisation, confusion and inefficiency will continue unimpeded.
We aren’t saying you have to follow the above workflow we’ve created to a tee, but it is important to define everyone’s responsibilities and their roles throughout each experiment.
The great thing about being involved in experimentation is that it doesn’t have to be limited to just your experiments. The process is something that you can optimise as your experimentation programme matures. Once successful process changes are made, document them as a single source of truth then hit repeat.
3. Match your process to your governance structure
It is likely that your experimentation will clearly reflect your internal governance model and structure, which identifies decision makers in the business, the hierarchy of power and processes for certain scenarios. Even if this governance structure is used more as a guide than a rigid set of rules, having something in place is important to ensuring your experimentation programme runs as efficiently as possible.
Whether you’ve created a centralised or decentralised organisation, or even a centre of excellence, the structure you choose allows you to spread a culture of experimentation, getting more teams involved in such a way that it will help create a test-and-learn environment.
4. Set programme metrics to keep on track
What are programme metrics?
Programme metrics are standard measurements for monitoring, controlling and evaluating a programme. They can include budget, quality, test velocity, number of wins, number of team members involved and more. Choosing the right metrics in experimentation allows you to ensure that the success of your programme doesn’t rely purely on the success of your experimentation or the ROI you gain.
Examples of good programme metrics are:
- Velocity
- Number of variations tested per experiment
Velocity
Velocity is measured by the number of experiments that run. It tells you how many experiments and ideas in your organisation are validated with data. High-velocity testing is the idea that rapid testing and experimentation is the key to major growth. Essentially, the philosophy suggests that the more tests you run, the more likely the learnings from these tests will turn into iterations and gather more success.
High-tempo testing is actually more complex than it sounds, but this is the basic idea underpinning the movement. Some successful companies that follow this philosophy include Facebook, Airbnb and Booking.com.
Number of variations tested per experiment
The more tests you run, the more you learn. So it makes sense to constantly try and run as many tests as possible, right? In actual fact, this is a lot harder to put into practice, mainly because you’re usually working with a limited budget and resources.
This means you can’t simply run multiple experiments for the sake of velocity or start recklessly testing too many variations. With a sound experimentation workflow in place that helps reduce inefficiencies, this can free up time for you to test multiple variations.
This is a great way for you to perform high-velocity testing even when your focus is narrow or you want to measure the impact different variables have on your users.
5. Manage stakeholder expectations
This is a key driver in any programme and isn’t unique to experiments. Experiment stakeholders will have a number of different expectations regarding the experimentation programme, from budgets to test results. A little bit of education can go a long way with managing internal expectations.
For example, running stakeholders through the process of how experiments go live will help them understand the effort and resources that go into delivery, means you can set realistic expectations from the get-go. In addition to this, if stakeholders understand the value gained not just from positive experimentation results but also from new customer learnings, this will help them understand how valuable your experiment insights are.
6. Maintain constant visibility and transparency
You should always aim to keep communications with stakeholders open and honest, especially with regard to the project’s performance. This includes how you’re allocating resources and any potential risks that could jeopardise the project.
As an Experimentation, CRO or Experience Optimisation team, demonstrating the value of our work is highly important and we find that the easiest way to do this is to maintain visibility and transparency with stakeholders and team members. As experimentation programmes often require a high level of investment, it’s all the more important to do this.
For example, you might warn stakeholders of the programme’s pitfalls but let them know how you plan to avoid them. Scheduling frequent status meetings also helps keep everyone informed about the project. We personally do this by:
- Having Slack integrations in place to keep all stakeholders up to date.
- Regularly scheduling status calls, both internally and with clients.
- Holding daily internal stand up meetings to update each other on any deliverables we’re working on. This helps improve internal visibility.
7. If all else fails, call in the experts
As you can see, there are plenty of things to consider when you’re starting out on the experimentation journey. Once you start your own experimentation programme, the exciting discoveries are often underpinned and only allowed due to the effective experimentation programme you have working away in the background.
If you have any questions or you’d like help with starting your own experimentation programme, feel free to reach out to us on LinkedIn or use the form below. At CreativeCX, we love talking about all things experimentation, so no need to be shy!
