When it comes to experimentation and conversion rate optimisation (CRO), we often see people relying too heavily on their instincts, abandoning logic and data in favour of their gut feelings. But really, nothing in experimentation is certain until tested. This realisation automatically makes you question everything you want to change about your website. This means experimentation should be approached like a scientific experiment that follows three core steps; identify a problem, form a hypothesis, and test that hypothesis.
But when it comes to experimentation, should you value the problem statement over the hypothesis? Or vice versa?
Which is more important: the problem statement or hypothesis?
At CreativeCX, we actually place equal importance on the problem statement and the hypothesis. This ensures that we consider both the customer problem that needs to be solved, as well as the business objectives.
All too often, we see companies either neglect the problem statement and hypothesis entirely or favour one over the other.
But weakness in either of these elements can seriously hinder the success of your experimentation programme.
So how can you structure both statements in such a way that you get the most out of your experiment?
The problem statement
First of all, what is the problem statement? A problem statement is a concise description of an issue that needs to be addressed or improved upon. In the case of digital products and services, this should be related to a problem that the customer has.
In any experiment, the problem statement should always come first. Without a problem, you have no real reason to conduct the experiment or understanding on what to conduct an experiment on.
The problem statement guides the strategic direction of your experiment while ensuring that you are always focusing on the customer.
How do we identify customer problems?
The data and research that you undertake will help you identify customer problems, either for your current customers or for your target audience. Identifying the pain points of your customer’s online behaviour should ideally come from multiple sources of data and research. This enables you to triangulate insights so that you can build a more complete picture of the problem, whilst also gaining an understanding of the magnitude of the issue. When starting your research process, you’ll probably find yourself having more questions than answers at the beginning. That’s fine; in fact, it’s normal at this early stage of the experiment.
Rather than letting this put you off, it is better to dig deeper, ask more questions and achieve a greater understanding of the customer problem before trying to find a solution. A greater understanding of the problem and how it’s affecting your customers will lead to better solutions and a higher win rate with your experiments. With this in mind, it is wise to collaborate with other teams within your business – preferably members of your CX and UX departments – who may be able to share relevant customer insights that they have discovered through their own research.
Once you have sufficient data, it is likely you will start to identify problem themes, which will help you understand the wider issues your customers are facing. This is where we start to create a clear problem statement.
How do you craft a clear problem statement?
A clear problem statement should help you identify what the problem is and the data that backs up your claim. At CreativeCX, we organise each problem statement as follows:
We believe [state the problem identified] because [state the supporting data].
Let’s demonstrate with an example. We work with an e-commerce company that sells women’s loungewear. Through our research, we discovered the following two pieces of data:
Usability testing showed users were moving back and forth between the product details page and the basket page to edit their selected size.
Website data showed only 2% of customers engage with the “size guide” text link on the product details page.
Based on this analysis, we have inferred a problem: users are struggling to understand which size they should choose. Through this, we are able to make the following problem statement:
We believe that users are struggling to understand which size they should choose because our data shows that users are editing their selected size multiple times on the basket and product details page and only 2% of customers engage with the sizing guide.
Can you see how much better this statement is compared to the following:
We think we have an issue with users understanding which size would fit them best.
Here are our top three questions we suggest you keep in mind when writing a problem statement:
Is my problem statement focused on my customers?
Is my problem statement clear and precise?
What data do I have to back up this problem?
As you can see in the examples above, our first example answers all three questions while the second statement falls short on questions two and three.
Whilst your problem statement identifies the problem you hope to solve, the hypothesis helps you decide on how you will try to solve it.
The hypothesis statement
The hypothesis: you’ve probably come across this word years ago in a science class, and its meaning remains the same even in this context. Essentially, the hypothesis statement is a prediction for what you think will happen if you take a certain type of action to resolve a problem.
The hypothesis usually identifies what is going to be changed and the action’s potential outcome, as well as why you think the change will have that particular result.
Creating a hypothesis is a key part of any quality experiment and shouldn’t be rushed. Rushing over this critical step could mean that you miss out on key actions or insights further down the line.
Similar to the problem statement, the hypothesis should be precisely constructed. Having a vague hypothesis may actually be a sign that your problem statement isn’t as clear as you originally thought. An unclear problem statement or hypothesis could, in turn, result in your proposed solutions not having the desired or expected results.
How do you write a clear hypothesis?
There are many ways to write a strong hypothesis. At CreativeCX, we structure ours using the following formula:
By [state experiment change], we believe [user behaviour change], solving [state problem]. We expect to see [expected results].
Now, some may say this will create a hypothesis that is too lengthy. However, this structure clearly incorporates three key elements of an experiment: the problem we are trying to solve, the specific execution, and the expected result. More importantly, it strikes a balance between focusing on the business goals you want to achieve and optimising your customers’ online experience.
Let’s go back to our previous example. There might be multiple solutions to solving this sizing problem, all of which would require a different hypothesis. However, our problem statement has allowed us to identify that we need to increase user awareness of the sizing guide on the product page.
We have identified this as a top priority, so our hypothesis would be as follows:
By increasing link prominence for the sizing guide, we believe more customers will interact with the link, solving sizing uncertainty. We expect to see an increase in customers engaging with the size guide, as well as an increase in customers progressing from the basket to checkout.
Again, whilst this is lengthy, it is also precise. It clearly defines the experiment’s aim with both the business and its customers in mind.
Compare this to the following:
Making the sizing guide link larger will improve our profits.
Here are our top three questions to bear in mind when you’re writing a hypothesis:
Is my hypothesis a statement or a testable question?
Is it clear and precise?
Is my hypothesis human-friendly and keeping the customer in mind?
As you can see, whilst our first example considered all three questions, the second is relatively vague and doesn’t relate to the customer at all.
What do I do once I have written a problem statement and hypothesis?
With your concise problem statement and hypothesis, you should have a great foundation for your experiment. The next step looks at designing your experiment, not in terms of actual visual designs, but what solutions you will be testing in a hope to validate your hypothesis and gain as much learnings on your customers as possible.
Look out for our future blog around how best to design your experiments and be creative with your potential variations.
If you have any questions about topics that have been covered in this blog or you’d like help with your experimentation or CRO programme, please don’t hesitate to reach out to us.
