Build Measure Learn
Hopefully you’ve read some of the earlier articles in this series and now understand the importance of the minimum viable product. You can see that it plays a crucial part in testing your business hypotheses and their underlying assumptions. Heck, you even realise that the minimum viable product will help define your target customer and understand what they really want. Now it is time to learn how you can do this.
Plan-Do-Study-Act
This sounds familiar, doesn’t it? Well, it should. This is the methodology taught to UK medical graduates for undertaking quality improvement projects. Unsurprisingly, both the Plan-Do-Study-Act methodology and the Build-Measure-Learn feedback loop share common roots with the System of Profound Knowledge and Lean Manufacturing. If it works in healthcare and manufacturing, it is probably going to work with start-ups. Let’s explore how the Build-Measure-Learn methodology works.
Leap-of-Faith Assumptions
Every new business is built upon the foundation of assumptions. Before you land your first ever customer, you’ve probably imagined a product that you will sell to them. Your first assumption is that someone will like the product and they will want to pay for it. The next assumption you have is that more than one person will like the product and you will soon have a queue out the door, thus keeping you busy and paying the bills. These two assumptions are the riskiest elements of any start-up. They are known as the Value Hypothesis and the Growth Hypotheses respectively. Everything depends upon these two leap-of-faith assumptions.
The Value Hypothesis
You can think of value for the customer as something which provides a benefit to them. The more your product or service helps a customer, the more value you are providing them. Read more about the value hypothesis.
The Growth Hypothesis
Types of growth include the sticky engine, the paid engine and the viral engine. Read more about engines of growth.
A short warning: Companies with a working engine of growth can easily be misled. This is because the company is continuing to grow, but they might not truly understand why their company is growing and may falsely attribute it to one of their features. For these companies it is vital to understand the engine of growth in order to create experiments that will improve growth.
The Cycle: Build-Measure-Learn
The build-measure-learn feedback loop lies at the core of any lean start-up. Every single time you make a change to your product (such as introducing a new feature), you are conducting a new experiment. Your product is the test substrate, and your customers interact with it to produce data. The feedback they generate, both quantitative and qualitative data, will shine a light onto your value and growth hypotheses, thereby letting you assess your leap-of-faith assumptions. This provides the evidence you need to determine whether your business will succeed or fail. Every single experiment you create should test one aspect of at least one of these hypotheses.
The build-measure-learn feedback loop systematically breaks down your business plan into its component parts and tests each part using a scientific model. This model enables a start-up to measure where it is right now and confront the hard truths about what it must do to achieve its overall vision. The creation of further experiments enables the start-up to learn how to move the real data closer towards the ideal numbers found within the business plan. There are three steps to the build-measure-learn feedback loop. We will cover each in turn.
Build
Measure
Learn
Build
As soon as you have established you leap-of-faith assumptions, it is important to enter the build phase as fast as possible with a minimum viable product. This is a version of your product that will enable a full turn of the build-measure-learn feedback loop. The perfect minimum viable product requires the minimum effort to achieve a full turn and should take the least amount of development time. Importantly, you must be able to measure the impact of the minimum viable product and it must be seen by customers.
It is vital that you test the riskiest assumptions first. If you cannot find a way to mitigate these risks, then there is no point in testing the others because your start-up would be doomed to fail from the outset.
When you build your minimum viable product, you should start with a clear baseline actionable metric in mind and a hypothesis about what will improve that metric. Then use a set of experiments to test that hypothesis. It is possible to create one minimum viable product that tests many assumptions, but you can also create many minimum viable products that test one assumption at a time.
Measure
When you make several changes at once, how do you know the changes you made are related to the results you are seeing? How can you draw the right lessons from those changes? The answer is you can’t. But there is an easy way to do this. Test one change at a time!
Once you have determined the hypothesis to be tested and produced a minimum viable product capable of doing this, you enter the measurement phase. This is where you show your product to the customer and determine whether efforts are leading to progress or whether they are wasteful. Progress is measured by a method known as innovation accounting and requires the collection of actionable metrics. Both are described below.
Learn
By this point you have run your experiment. This means your latest product or feature has been put to the test and has been offered to real life customers. You have gathered data and are now able to compare it to the hypothesis you created before you launched. So, was it a success and why does this matter?
Learning is vital for every start-up. There are often many incorrect preconceptions that drive your decisions, and these preconceptions may even lead to a strategy that is fundamentally flawed. You need to learn which elements of your strategy are working, who your customers are and what your customers really want. This is where validated learning comes into play.
Validated learning is a rigorous method for demonstrating that a team has uncovered valuable truths about a start-up’s present and future business prospects. It is how you know you are making progress within the uncertain environment in which your start-up operates.
Validation is found through evidence gained during the build-measure steps. As you adjust your product and strategy to meet the desires of your customer, you will find that your quantitative results change, and you can systematically figure out the right things to build. Validated learning demonstrates that your development efforts are leading to success.
The Virtuous Cycle
After completing these three steps, the build-measure-learn feedback loop then repeats. You continue with your hypothesis, refine it and expand it. If your hypothesis was completely wrong, you simply create a new hypothesis. Using this updated hypothesis, you must decide:
- What you need to learn
- What you need to measure to ensure you make validated learning
- What product to build to run the experiment and gain the necessary measurement
Innovation Accounting
This is a quantitative method based on learning milestones which are used to assess progress accurately and objectively. Innovation accounting is used to see if your engine tuning is having the desired effect. It is a standard upon which you base your decisions, including, but not limited to, product prioritisation, deciding which customers to target and whether to pivot or persevere. There are three milestones that make up innovation accounting. Their role is to ensure there is sufficient data to make a pivot or persevere decision. The milestones are:
The Minimum Viable Product
This is used to establish real data on where the company is right now.
Engine Tuning
This moves the start-up from the baseline towards the ideal. It may take many experiments to optimise the product (value hypothesis) or enhance a given driver of growth (growth hypothesis).
Over time the baseline numbers will converge to something of the ideal in the business plan. If you fail to do this, the ideal will get further away.
Pivot or Persevere
If you are making good progress, this means the learning is appropriate and effective. Thus, you should persevere. If you are not making good progress, your current strategy is flawed and needs a serious change.
Enacting a change in strategy is known as a pivot and involves starting the process all over again. If the pivot is successful, the engine tuning activities are more productive after the pivot than before.
Actionable and Vanity Metrics
Vanity metrics are of little use to the start-up. They are traditional gross metrics that established businesses rely upon. An example would be the total number of customers. Vanity metrics often show a hockey stick growth but they don’t tell you anything about the progress you are making. In fact, it is impossible to judge improvement using them and for this reason they will not help improve a start-up’s prospects. Please note that innovation accounting will not work if it is being misled by vanity metrics.
On the other hand, actionable metrics help analyse customer behaviour in ways that supports innovation accounting. They can be used to judge how the business is progressing and the learning milestones along the way. Examples of an actionable metrics for the growth hypothesis include registration rates, activation rates, retention rates and referral rates. Examples of actionable metrics for the value hypothesis include download rates, repeat usage rates and customer lifetime value.
The Decision: Pivot or Persevere
As you run your experiments, you will eventually reach a point where a decision needs to be made. You must decide whether to pivot or persevere. If your leap-of-faith assumptions turned out to be false, now is a good time to make a new strategic hypothesis. This is known as a pivot. Early pivots are efficient because they save time and lead to less waste. Learn more about pivots in our article.
As you accelerate the build-measure-learn feedback loop, the time between each pivot and persevere decision gets shorter and shorter. This is important because it means you are learning more about your customers, market and strategy.
References
- The Lean Startup; Eric Ries
- The $100 Startup; Chris Guillebeau