How to create experiments that drive innovation

Lean encourages us to base products on user data rather than opinions and to test ideas by deciding as late as possible. Experimentation is a key ingredient in discovery, it’s also an easy way to learn and test assumptions.

I love running experiments and learning how users interact with a product. In this article, I want to show you how easy it is to run experiments.

‍

It all begins with a problem

To begin you’re going to need a problem that requires solving. This could be you want more user to download your app or you want to increase user retention. Either way, you’ve got a problem with your product.

Next you’ll want to gather your team and come up with as many ideas as possible for solving the problem. You’ll need to put these ideas on a board and prioritise them. I generally prioritise mine based on the riskiest assumptions but you might want to prioritise them on what you deem is easy to test but provides high value.

‍

Keeping track of experiments

You’re going to need somewhere to keep track of experiments. I generally use a Trello board but you can use a whiteboard. It really doesn’t matter what you use as long as it works for you. Click here to view the Trello board.

‍

The Trello board that I use follows a typical Kanban approach in which experiments move from left to right on the board. An experiment is represented by card. Each card consists of a title, owner, description and a link to an experiment document (I’ll explain this in more detail later).

The board consists of the following lanes:

Backlog

This lane consists of a prioritised list of the ideas for experiments that myself and team have come up with. At first each experiment is no more than just an idea but by the time its ready to move to the “Next Up” lane it will be fleshed out in more detail.

Next Up

This lane consists of the experiments that have are ready to be run. This means that I’ve created a experiment document for it and everything is ready for the experiment to run.

Doing

This lane consists of the experiments that are currently running. It’s very important to monitor the experiments.

Done

This lane consists of the experiments that have finished. We consider an experiment to be done once the results have been documented, recommendations are presented and retrospective has been completed.

Here’s an example of a Lean Experiments board that I quickly created using Trello. It only takes 15 minutes to create one and is super easy to use.

‍

A template to run innovative experiments

For each experiment I want to run I use the following experiment template to fill in its details. Click here to view the template.

The template consists of the following categories:

Name

Provide a super awesome name for the experiment. Don’t just go with “Experiment 1.b”, yawn! 😴 It should be meaningful, useful and easy to remember.

Learning Goal

The learning goal should clearly identify the purpose of the experiment. It is much broader than a hypothesis. If you’re unsure of the clear purpose of the experiment then you’re probably best off not running it in the first place.

Hypothesis

Describe what you’re testing. For this reason, the hypothesis must be clear, measurable and it also must be falsifiable. If the hypothesis is good then it will be much easier to write the experiment.

Experiment

Describe the plan which is a clear description of the steps required to perform the experiment. Only provide enough information for anyone reading the document to understand the experiment. If you need a diagram to help someone understand it, then include it.

Metrics

Define the metric/s that will be measured to try validate the hypothesis? The metric is generally quantitative data but it could also be qualitative data.

Fail Condition

The Fail Condition is the resulting measurement which would convince us beyond any reasonable doubt that our hypothesis is invalid. If this happens our experiment has failed and you should stop it immediately.

Timeframe

Specify the start date, end date and retrospective date of the experiment. It’s important to start and end the experiment on the dates specified. It’s also good to have a quick retro after every experiment so that you can reflect on how to improve.

Results

Once the experiment is finished you will need to document your findings and the results. Try to be clear about the results.

Recommendations

Based on the results of the experiment what are your recommendations. Remember some experiments will be inconclusive so it’s good to recommend what the next steps should be.

Now that you’ve got a template you should be able to start creating your own experiments.

‍

Some do’s and don’ts

Here are few of my guidelines when running experiments.

Do regularly brainstorm new experiments to run.

Do rerun past experiments. Some experiments will be impacted more by seasonality than others.

Do run experiments that you think are stupid. Even stupid experiments will have interesting results.

Do try to keep experiments short and easy to run. We want our experiments to deliver as fast as possible.

Do let your team know which experiments are currently running.

Don’t change details of the experiment whilst its running. Any changes could affect the experiment.

Don’t be upset if the experiment fails. Experiments are used to amplify learning so you will learn something whether the experiment fails or succeeds.

Don’t keep the learnings from experiments to yourself. Share them with your team and colleagues.

‍

With the above in mind, I think you should be able to introduce experiments into your product and start to amplify learning. Go forth and create experiments that drive innovation! ⚗️