For the last year and across multiple stints before that, I’ve worked toward getting pre-product-market-fit startups (products and services) off the ground.
What I’m going to tell you about is the dog food I eat everyday. The problems teams run into as they go through the starting-up process. Not all the problems—thank goodness!—but a particular kind of them.
Here’s what I add to the conversation:
- Justifying an action bias while building startups is easy. But that busy-ness can be misguided.
- Hypothesis-driven startup-building is rooted in the scientific method. But most of us aren’t scientists.
- Construct testable hypotheses before spending resources on experiments—this advice is hard to follow because hypotheses require deep immersion into the customer’s problem.
- Testable hypotheses are a shield against your own cognitive biases. If you cannot fail a test, you cannot learn.
- If you follow the scientific method, there are no failed experiments. You will learn every time.
And here’s the best quote from the piece, from Eric Ries in The Lean Startup
…if the plan is to see what happens, a team is guaranteed to succeed—at seeing what happens—but won’t necessarily gain validated learning. This is one of the most important lessons of the scientific method: if you cannot fail, you cannot learn.
If you’re a founder, an operator in a startup, or someone taking ideas to commercialization at your company, this is for you. The link to the essay is in the first comment. I invite you to read and subscribe, if you like.
I launch startups for a living. For the last year and across multiple stints before that, I’ve worked toward getting pre-product-market-fit startups (products and services) off the ground. Just this year itself, I’ve spent almost all my time trying to explore the future for two such initiatives.
What I’m going to tell you about is the dog food I eat everyday. The problems teams run into as they go through the starting-up process. If you’re a founder, an operator in a startup, or someone taking ideas to commercialization at your company, this is for you.
Before we dive in, here’s what I’m adding to the conversation:
- Justifying an action bias while building startups is easy. But that busy-ness is often misguided.
- Hypothesis-driven startup-building is rooted in the scientific method. But most of us aren’t scientists.
- Spend time defining the problem and constructing testable hypotheses before spending resources on experiments—this advice is hard to follow because trying out many things is a lot easier than coming up with falsifiable predictions before trying out anything.
- Testable hypotheses are a shield against your own cognitive biases. If you cannot fail a test, you cannot learn.
- If you follow the scientific method, there are no failed experiments. You will learn every time.
The scientific method and startup-building
In a recent talk, Brian Chesky of Airbnb said, to put a finger on the thing, that he made a rule at Airbnb forbidding any A/B testing without a hypothesis.
A/B tests randomly divide similar customers into a control group that sees the existing product and a treatment group that sees a product with at least one change. The test, if done well, tells you if the modifications yield a statistically significant performance improvement. So what’s all this Chesky says about having a hypothesis at hand prior?
To get what Chesky meant and why that is so powerful, we need to first get the scientific method. The upside to doing this well is that once you understand the underlying thing, you will be able to apply it to not just new launches but to solving most meaningful business problems.
The scientific method goes something like this:
- Define problem statement
- State hypotheses
- Design and run experiments to test hypotheses
- Observe experimental results
- Draw conclusions from the results and confirm/disconfirm said hypotheses
Undergirding the precise thinking that sets a scientist apart is the repeated and iterative application of the above sequence of steps in a loop. This method mirrors the process of idea validation for entrepreneurs. In startup parlance, this sequence translates into:
- Define product vision
- Translate vision into testable hypotheses
- Run tests
- Observe results
- Make decision: persevere, pivot, or perish
>>>>> gd2md-html alert: inline image link here (to
images/image35.jpg). Store image on your image server and adjust path/filename/extension if necessary.
(Back to top)(Next alert)
>>>>>
The problems with Just Do It! entrepreneurship
I first encountered the term Just Do It! in the second-year course material (Launching Technology Ventures) at Harvard Business School. It was used to describe an improv way of launching startups: skip the vision, drop the hypotheses, jump straight into experimental action.
Those who follow the Just-Do-It! school of entrepreneurship swear by action. The approach has the clear benefit of not letting founders get sucked into endless planning and theorizing. It can push them from uncertainty to opportunity.
But there’s something in its shadow that needs sunlight. As someone who launches new initiatives for a living, I see it everyday.
When you run experiments without a hypothesis to test, your decision to Persevere, Pivot, or Perish will be compromised.
👉If a month does well, you’ll do more of the same things. If it does poorly, you’ll stop doing those things.
👉You’ll ask for and observe all data, giving everything the same attention and making connections where there should be none.
👉You’ll go with early customer feedback and add a bunch of features. But these are power users with outlying needs you’ve listened to (previous point). Now you’ve an over-engineered product/service that doesn’t offer value for money for your actual target customer base.
Continue like this and soon you’ve run out of time and money. Before you know it, you have to now make a decision on the future of your startup without validating nearly as much as you could have.
Where did you trip up?
You did not define the business problem or construct testable hypotheses before running tests. In business parlance, you skipped strategy and jumped on to tactics.
This happens because you did not have a vision or a set of ideas that could be validated. You didn’t have a hunch to answer for your experiments.
Falsifiability—the special power of hypotheses
Eric Ries has this to say in The Lean Startup:
…if the plan is to see what happens, a team is guaranteed to succeed—at seeing what happens—but won’t necessarily gain validated learning. This is one of the most important lessons of the scientific method: if you cannot fail, you cannot learn.
As a newsletter writer a topic I often think about is How can I make this newsletter more valuable for my readers, which translates into ‘How can I increase the subscriber base of my newsletter?’
A testable hypothesis to answer this question would be:
Changing the CTA from Subscribe to A Sea of Knowledge Awaits You On the Other Side of This Click will improve weekly subscriptions by 20% for the next month.
It is easy to see when this hypothesis could be rejected. The impact of the predicted change is measurable. But it is almost impossible to fail with a hypothesis that says, ‘My subscriber base will grow through catchy CTAs.’
As long as there is a non-zero rate of subscription, this loosely worded statement will prove true, whether the rate is very low or very high. Heads I win, tails you lose.
Where’s that playbook for startup ideas and testable hypotheses?
So, running tests cannot be the starting point for a new venture. You don’t want to waste your time and money that way. You need to first define the problem and then construct testable hypotheses.
In his essay How to Get Startup Ideas, Paul Graham says:
The way to get startup ideas is not to try to think of startup ideas. It’s to look for problems, preferably problems you have yourself.
That almost doesn’t make sense. Graham elaborates:
Why do so many founders build things no one wants? Because they begin by trying to think of startup ideas. That m.o. [modus operandus] is doubly dangerous: it doesn't merely yield few good ideas; it yields bad ideas that sound plausible enough to fool you into working on them.
This is only one reason why big incumbents find it hard to innovate. They also more often lack the setup that allows for deep and total immersion into a problem. They’re more often conflicted among competing priorities. They’re more often stocked up with a one-size-fits-all yardstick for new and existing ventures. Those tasked with innovation are more often than not caught up in showing management the size of the opportunity than in probing the depth of the customer’s pain point.
And if you don’t have a clear view of the problem, you’re not going to come up with insightful hypotheses.
Hypotheses don’t emerge out of thin air, though sometimes they may give such an impression. They may strike someone on the outside as the result of a flash of inspiration but that demonstration is rooted in deep empathy for the customer. Wondering about the life of the target customer, working out her desperation, and forming a view about how to create value for her are hard things—all of these feed into what appears as intuition, inspiration, revelation.
At a startup unit I was leading, we came up with one hypothesis in six months. That’s six months worth of time and capital spent on one bet. Even if that hypothesis was right—and it was—our dullness in coming up with testable theories had us leave the iterative way of the lean startup method and settle for early optimization. Where we could have imagined a degustation, we ended up with a single-course unlimited _biriyani _meal.
No such thing as failed experiments
‘I have not failed,’ Isaac Newton had declared. ‘I have just found 10,000 ways that won’t work.’ As much as it sounds like a pithy byte designed for social media, Newton was hinting at a basic truth.
Experiments, by definition, cannot fail. If done well, they will yield results that will persuade the experimenter one way or the other. Yet, if you’re familiar with the corporate context, you will know that this definition does not apply to the corporate method. Because failed experiments are common and gigantic failed experiments just as much.
Wait a minute! I used the Just Do It! entrepreneurial style earlier to show you how experiments can waste limited resources and now I’m saying there’s no such thing as a failed experiment. What’s missing?
In the 1974 classic Zen and the Art of Motorcycle Maintenance, author Robert Pirsig observes:
...experimentation, is sometimes thought of by romantics as all of science itself because that’s the only part with much visual surface. They see lots of test tubes and bizarre equipment and people running around making discoveries. They do not see the experiment as part of a larger intellectual process and so often confuse experiments with demonstrations.
>>>>> gd2md-html alert: inline image link here (to
images/image36.jpg). Store image on your image server and adjust path/filename/extension if necessary.
(Back to top)(Next alert)
>>>>>
This trope of scientists running around with test tubes in a lab assumes the scientists know what they’re looking for. It’s not hard to guess why. Doing experiments means doing things (demonstrations). Coming up with hypotheses is sitting and twiddling your thumbs. More from Pirsig on this:
_An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question, when the data it produces don’t prove anything one way or the other. _
What then happens in corporate settings? We deem an experiment a failure if it falls short of predicted results. But the basis—an insight, a sense—on which we made our prediction is fuzzy. There’s no reason that we have put forward that will prove us right or not.
Testing the startup foundations
If we don’t have a sense of how we will be right about the world, we’ll see whatever we’re hoping to see.
All founders and operators: hypotheses define the boundaries for your exploration in uncertainty. They save you from taking a wrong turn by putting up signposts at every corner. Without them you can hurtle ahead at speed, foolishly convinced you know the map only to find yourself lost and spent. Applying the scientific method to startup-building may make you feel like you are being too careful but it is your best chance to not be tricked into thinking you know something that you actually don’t.
- The business models of startups can be broken down into four main components: (1) consumer value proposition; (2) go to market; (3) profit formula (which incorporates market size and unit economics); and (4) the product, technology and operations.
- When you introduce different functions into the founding team, they bring different lens into the problem. But the founder still disproportionately guides the decision-making. For example, a strategy guy will focus on the market size, a marketing guy will be drawn to GTM, and so on.