How to Create Tiny, Fast Experiments (5:21)
Here is the big question we get asked all the time: How do I design a good experiment?
Let’s take a high-level look at the three components of a good experiment. It will maximize for speed, learning, and focus. Without all three of these things, your experiment will not be optimal, and it might fall apart.
If you are focused and moving quickly but not planning for learning, then you are likely not actually making progress, because you are not finding out the thing that you need to about your business to move it forward.
If you are not optimizing for speed, then you risk running out of resources including time.
Lastly, is the danger of not optimizing your focus. Many times companies will prematurely focus on the wrong part of their business.
Let’s move away from what not to do and return to the question: “How do I run an effective experiment?”
First of all, you need to identify one key metric. What is the thing that you are trying to learn? And how can you quantify that?
The next piece of this is seeking to craft an experiment that does a very, very small thing to optimize for learning quickly. We are huge advocates of running tiny, fast experiments. What is the smallest experiment that you can do that helps you learn the thing that you need to know?
The next piece is formulating a falsifiable hypothesis. What this means is that you need to have a repeatable action with an expected measurable outcome. And by doing this, you will know whether you are right or wrong about sort of the thing that you hoped would happen. This is the the quantifiable piece that helps tell you with data if you are moving in the right direction. With a tiny, fast experiment that tests a hypothesis, we are trying to validate qualitatively (testing with customers), but verify quantitatively (measuring outcomes).
The last piece here is ensuring that you can correlate results to specific actions. You need to be creating repeatable processes so that you can do things over and over again and have expected outcomes. Otherwise, building your business turns into a guessing game.
It is also important that your experiment focus on your riskiest assumption. Your riskiest assumption is what must be true for your business to be successful. As founders and entrepreneurs, we have a lot of ideas and assumptions that we believe to be true about our business. Your job now is to go through and validate which assumptions are true and which ones are false. The best way to start is to define which assumption is the most risky, or rather, which assumption needs to be true or else your business is going to shift dramatically.
If you are having a hard time figuring out what your riskiest assumption is, return to the Founder Roadmap, and look at each stage. If you are having trouble answering any of those questions in any stage with data, then likely there is an assumption that exists there that can be tested. Probably it is your riskiest assumption.
Finally, let’s look at a simple formula that you can use to write an effective hypothesis for each test: When I do [insert repeatable action], I expect [hypothesized result] to happen with [measurable frequency].
For example, if I interview 10 potential early adopters, six will verbalize my problem statement. Interviewing customers is the repeatable action that you can take. What is the result that you hope to happen? You hope the people that you interview will verbalize your problem statement. That's the outcome. The last piece of the hypothesis defines how you will measure it and what you hope to be the outcome: Six out of 10 customers will verbalize my problem statement. If six out of 10 customers do so, you validate that this assumption is true. By using this formula, you are able to always create a hypothesis that helps you learn and helps tell you if you are going in the right direction.