What are experiments?

Modified on Mon, 02 Sep 2019 at 07:02 AM

One of the key elements of the Lean Startup methodology is the idea of experiments. Experiments are the tests you run to prove (or disprove) your assumptions about your product or service, before you go to the trouble and expense of actually building it. This approach lies at the heart of the Lean Startup method, and is designed to avoid unnecessary costs and wasted time.


The principle of an experiment is to turn your assumptions about the product or service you are looking to build into more certain knowledge, by testing them with real potential customers or users. By testing in the wild, you are able to learn much more than through traditional market research techniques, such as asking customers what they think. Customers are notorious for not knowing what they want or how much they might want it. Furthermore, if you are working on something truly innovative, it can be difficult for customers to understand whether they might want it, or how much they might pay for it, until they can actually see or experience the product.


As with scientific experiments, before you can test your assumptions you have to express them clearly and succinctly – these are your hypotheses (more on hypotheses). Hypotheses range from fundamental assumptions about the business model itself (e.g. ‘I believe people will be willing to buy books online’) to ones about the value of the product or service (e.g. ‘I believe people will be willing to pay a premium to have extra leg room on a flight’) right down to smaller details about the product or service. In theory, you could be testing and learning about your product all the way through the process, so that every element has been validated with potential users before being built or developed. In the practice of innovation, it is most important to test your fundamental business model or idea, and the price people are willing to pay for it, before you invest too much time or energy in developing the idea.


The Lean Startup focuses most of its attention on one particular form of experiment: the Minimum Viable Product, or MVP.


The Minimum Viable Product

"A minimum viable product (MVP) helps entrepreneurs start the process of learning as quickly as possible. It is not necessarily the smallest product imaginable, though; it is simply the fastest way to get through the Build-Measure-Learn feedback loop with the minimum amount of effort." - Ries, E. (2011)  The Lean Startup: How Constant Innovation Creates Radically Successful Businesses

An MVP can take many forms, depending on the type of product or service you are developing, and there are many case studies and examples available online (see Further Reading for some suggestions). Ries discusses two forms of MVP in his book: the Video MVP and the Concierge MVP.


The Video MVP

The video MVP is useful for products or services where it's not possible or too expensive to give customers access to a working version. This might be software that would require complex installation or integration with existing systems, for example, or where building even a demo version would take many hours. Ries uses the example of Dropbox, which used an explainer video to describe what their product did without having to build a complex piece of software. Buffer went even more minimalist with a simple landing page that explained what Buffer would be, and used clicks to the pricing page to gauge interest. (Ries does not explicitly discuss landing page MVPs in his book, but these have been separated out as a different type by subsequent writers.)

Concierge MVP

The Concierge MVP is where you use manual processes in order to gauge interest in the service, before building the automated version. Ries uses the example of Zappos, whose founder started by posting photographs of shoes from shoe stores on a website, and then physically going to buy them and posting them to the customer whenever he made a sale. In this way, he was able to see whether people would be willing to buy shoes online. Another great example of a Concierge MVP is Stitch Fix, the online personal stylist. Originally set up by Katrina Lake from her flat, the first Stitch Fix orders were personally assembled by her to see if customers would pay for this kind of fashion advice. As the company has grown, it uses algorithms in combination with stylists to achieve results at scale.

Wizard of Oz MVP

A variation of the Concierge MVP, in a Wizard of Oz MVP the fact that the process is manual is hidden from the customer. This avoids the issue of customers having to move from what they feel is a personal service to one that is more automated and therefore lower quality. In fact, as the product or service evolves and becomes more automated, they should perceive a rise in quality due to an increase in the speed of delivery. But companies need to be careful with this type of MVP, as the discovery of the humans behind the curtain can erode trust, particuarly if the truth is concealed from all stakeholders


Other types of experiments

As well as the MVP, innovation and lean practitioners have access to a huge number of potential experiments that can be used to test hypotheses. Many of these have been cross-fertilised from other disciplines such as UX, marketing and digital, among others, and can be used to test an entire business model or idea, or a tiny element of a product or service. Selecting the right type of experiment for your hypothesis requires careful consideration of exactly what you are trying to test, and what the results will show you.

Further Reading

The Lean Startup

Forbes - Five pitfalls of running Lean Startup experiments

This Week in Startups - Eric Ries discusses experiments

The Lean Enterprise Institute - Why Start up experiments are hard to design

Allen Cheng - Lean Startup Experiments

The Real Startup Book - assembled by practitioners and available under Creative Commons

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article