August 23, 2019
Lean encourages us to base products on user data rather than opinions and to test ideas by deciding as late as possible. Experimentation is a key ingredient in discovery, it’s also an easy way to learn and test assumptions.
I love running experiments and learning how users interact with a product. In this article, I want to show you how easy it is to run experiments.
To begin you’re going to need a problem that requires solving. This could be you want more user to download your app or you want to increase user retention. Either way, you’ve got a problem with your product.
Next you’ll want to gather your team and come up with as many ideas as possible for solving the problem. You’ll need to put these ideas on a board and prioritise them. I generally prioritise mine based on the riskiest assumptions but you might want to prioritise them on what you deem is easy to test but provides high value.
You’re going to need somewhere to keep track of experiments. I generally use a Trello board but you can use a whiteboard. It really doesn’t matter what you use as long as it works for you. Click here to view the Trello board.
The Trello board that I use follows a typical Kanban approach in which experiments move from left to right on the board. An experiment is represented by card. Each card consists of a title, owner, description and a link to an experiment document (I’ll explain this in more detail later).
The board consists of the following lanes:
This lane consists of a prioritised list of the ideas for experiments that myself and team have come up with. At first each experiment is no more than just an idea but by the time its ready to move to the “Next Up” lane it will be fleshed out in more detail.
This lane consists of the experiments that have are ready to be run. This means that I’ve created a experiment document for it and everything is ready for the experiment to run.
This lane consists of the experiments that are currently running. It’s very important to monitor the experiments.
This lane consists of the experiments that have finished. We consider an experiment to be done once the results have been documented, recommendations are presented and retrospective has been completed.
Here’s an example of a Lean Experiments board that I quickly created using Trello. It only takes 15 minutes to create one and is super easy to use.
For each experiment I want to run I use the following experiment template to fill in its details. Click here to view the template.
The template consists of the following categories:
Provide a super awesome name for the experiment. Don’t just go with “Experiment 1.b”, yawn! 😴 It should be meaningful, useful and easy to remember.
The learning goal should clearly identify the purpose of the experiment. It is much broader than a hypothesis. If you’re unsure of the clear purpose of the experiment then you’re probably best off not running it in the first place.
Describe what you’re testing. For this reason, the hypothesis must be clear, measurable and it also must be falsifiable. If the hypothesis is good then it will be much easier to write the experiment.
Describe the plan which is a clear description of the steps required to perform the experiment. Only provide enough information for anyone reading the document to understand the experiment. If you need a diagram to help someone understand it, then include it.
Define the metric/s that will be measured to try validate the hypothesis? The metric is generally quantitative data but it could also be qualitative data.
The Fail Condition is the resulting measurement which would convince us beyond any reasonable doubt that our hypothesis is invalid. If this happens our experiment has failed and you should stop it immediately.
Specify the start date, end date and retrospective date of the experiment. It’s important to start and end the experiment on the dates specified. It’s also good to have a quick retro after every experiment so that you can reflect on how to improve.
Once the experiment is finished you will need to document your findings and the results. Try to be clear about the results.
Based on the results of the experiment what are your recommendations. Remember some experiments will be inconclusive so it’s good to recommend what the next steps should be.
Now that you’ve got a template you should be able to start creating your own experiments.
Here are few of my guidelines when running experiments.
Do regularly brainstorm new experiments to run.
Do rerun past experiments. Some experiments will be impacted more by seasonality than others.
Do run experiments that you think are stupid. Even stupid experiments will have interesting results.
Do try to keep experiments short and easy to run. We want our experiments to deliver as fast as possible.
Do let your team know which experiments are currently running.
Don’t change details of the experiment whilst its running. Any changes could affect the experiment.
Don’t be upset if the experiment fails. Experiments are used to amplify learning so you will learn something whether the experiment fails or succeeds.
Don’t keep the learnings from experiments to yourself. Share them with your team and colleagues.
With the above in mind, I think you should be able to introduce experiments into your product and start to amplify learning. Go forth and create experiments that drive innovation! ⚗️
If you enjoyed this article then you may want to read my other articles.
Fill in the form below, and I'll get back to you within 24 hours.