Experiment design: test the riskiest assumption before you build
Run this method in IdeaHone
Creates a space with the entities and scoring criteria of this method, ready for your next session.
Most innovation tests are not tests. A team "tries the refill station in one store for a while", looks at it after a month, and decides based on how it feels. If the numbers are good, it was a success. If they are bad, it was too early, the weather was wrong, the signage was small. Nothing was ever going to stop the idea, because nobody said in advance what would. A real experiment fixes three things before it starts: what you believe, how you will know, and what you will do with each possible result.
The experiment card
Write one card per assumption. It fits on one screen and takes 20 minutes with the people who will act on the result.
- We believe that regular customers will bring their own bottle.
- We will know when they refill at the station.
- Audience: regulars of the Mitte store.
- Method: concierge, a staff member refills by hand for two weeks.
- Metric and threshold: at least 20 refills per 100 receipts.
- If we reach it, we roll out to three stores.
- If we miss it, we stop the station.
- Owner and date: Anna checks it on 31 March.
The last two lines are the ones teams skip, and the ones that matter most. Once the threshold and the consequences are written down, the result decides. Afterwards, nobody can move the line.
Designing it in 20 minutes
- Start from the riskiest assumption. The one that is important and has the least evidence: the top right of an assumption map. Testing a safe assumption feels productive and teaches nothing.
- Write the belief as a sentence about behaviour. "Customers like refills" cannot fail. "Regular customers will bring their own bottle" can.
- Name the signal you could observe. What would you see, count or hear if the belief were true? If you cannot observe it, rewrite the belief.
- Pick the cheapest method that produces the signal. Talk to people when you need to know why, use a landing page or a concierge when you need to know whether they act. Build last.
- Set the threshold before anyone sees a result. A number and a direction: at least 20, at most 5 minutes. Pick the number at which you would honestly change your plan.
- Write what you do on each side. Above: the next step, usually a bigger test. Below: stop, or change the idea. The below line becomes the kill criterion of the decision.
- Give it an owner and an end date. Two weeks is a good default. The owner records the measured value on that date, and the card says what happens next.
Choosing the method
| Method | Answers | Cost |
|---|---|---|
| Customer interviews | Why people do what they do | A few days |
| Survey | How many people say something | Days, but says, not does |
| Landing page | Whether people sign up before it exists | A day and some traffic |
| Prototype | Where people get stuck | Days to a week |
| Concierge | Whether the value works, done by hand | One to two weeks |
| Wizard of Oz | Whether they use it when it looks real | One to two weeks |
| A/B test | Which of two versions works better | Needs enough traffic |
| Pilot | Whether it works in real conditions | Weeks, and real money |
If two methods could answer the question, take the one you can start this week. A small test that runs beats a perfect one that waits for a budget.
Where experiments go wrong
- The threshold comes after the result. "11 refills is actually not bad for a start." If the line can move, it was never a test. Write it first, and record why if you still decide differently.
- Testing the comfortable assumption. Teams test what they expect to pass. The assumption that would kill the idea is the one to test first, because the answer saves the most money.
- No decision attached. A result without "then we" is a report. Every card says what happens above and below the threshold.
- Pivot without a record. Changing the idea after a missed threshold is often right. Record it as a decision with the result and the reason, then design the next card, so the team can see later why it changed course.
Score and prioritise ideas with your team
IdeaHone scores ideas against your criteria, shows them on a matrix and keeps the reasons, so the decision is clear and traceable.
What is experiment design in product innovation?
Turning an assumption into a small test with a fixed hypothesis, an observable signal, a method, a metric with a threshold, and a decision for each side of the threshold, all set before the test starts.
What is a good hypothesis format?
"We believe that [behaviour of a specific group]. We will know when [observable signal]." It has to be possible for the belief to be wrong, otherwise it is not worth testing.
How do you set the threshold?
Ask what result would honestly change your plan. Use the number at which you would stop, not the one you hope for, and write it down before the first data comes in.
What is a pivot?
A decision to stop testing the current form of an idea and try a different one for the same problem. Record it like any decision, with the measured result and the reason, and design the next experiment straight away.
