How Can you Test Business Ideas? Interview with David J. Bland

If you ever read any of my content, you know that OKR is about delivering value, and that good OKRs measure outcomes and pass the “So What?” TestThe whole point of using OKR is to agree on the outcomes we want to achieve and then let the teams test different ideas quickly to find which ones will work.

 Whenever a team says that they can’t measure outcomes in a single quarter, it’s because they don’t know how to run experiments. 

They don’t have an OKR problem. They have an experiment problem.

So if you are using OKR correctly, teams won’t have a fixed list of deliverables to implement. That is why OKR fails without experiments.  Unless employees learn  how to set up and structure experiments, they will waste a lot of time and money going in the wrong direction.

Thankfully,  David J. Bland teamed up with Alex Osterwalder, the creator of the Business Model Canvas, to tackle this topic in his new book, Testing Business Ideas. My blog editor Melissa Suzuno recently sat down with David to learn more about the book and David’s advice on experimenting in a business setting.

Tell us a little bit about yourself and how you got started on your journey with experiments.

David: I went to school for design and then I ended up joining a startup right out of college, where I did a little bit of everything—design, front-end development, web development, leading teams, closing deals. This experience was really influential in my career. When you’re at a startup, you’re trying to search for the business model, the right customers, fit. Now, as a consultant, I try to use everything I learned at startups to handle uncertainty and how you test your way through things. No matter what type of company I’m working with, we focus on identifying the riskiest thing they have and how to test that so they don’t find out they’re wrong much later and fail slowly.

Can you tell us a little bit about your book and give us a quick overview of what it’s about?

David: I was approached by Alex Osterwalder who created the Business Model Canvas and the Value Prop Canvas. He said, “There’s this need in the market where people aren’t sure what experiments to run and when they should run them.” Our goal was not to build a giant encyclopedia of them, but to help people understand how to choose an experiment and what it should look like. We started there and we essentially ended up with this list of experiments. We have 44 in the book and we applied a taxonomy to them, to help people understand whether an experiment will help them determine the type of risk—if something is desirable, viable, or feasible.

The idea is that someone could say, “I have this kind of risk. What kind of experiments are available to me?” And the book will help them see which experiments they can use and how to run them.

The core of the book is the list of experiments, but then the wrapper around that content addresses questions like: What kind of team do I need? How do I set that up? What kind of ceremonies does this involve? How do I make this a repeatable process?

Why are experiments so crucial to achieving high performance?

David:  Usually when you’re starting anything new—if you frame this as discovery versus validation—you’re doing probably a lot of discovery just to understand if there’s a problem there. And then as you learn more, you want to be able to put things in front of people and get feedback on them and then get higher and higher strength of evidence. 

It’s not just whether they want it, but it’s also will they pay for it? And can you do it? And so over time that never really stops. Even when you have your business and it’s running, it’s not just running on autopilot. There’s always this element of, okay, well if we wanted to add a feature to that, what would that look like? How would we test our way through that before we just throw in more features? 

Once you get going and you find some customers and you have some traction, you don’t just stop experimenting. It’s not a phase. It’s something that you need to continue to do. You’ll want to think about what are the things in your roadmap or strategy that you feel really nervous about. What can you do to learn about those quicker?

In a modern view of OKR, managers should not tell teams what features to build or projects to work on, but instead they should agree on the outcomes that they want to achieve. In your opinion, what role do experiments play in that type of scenario?

David: With outcomes, it’s essentially where you are focused in your business model. For example, are you focused on the fit between your value prop and the customer segment? Are you focused on the channels and how you can grow those channels? Are you focused on building the relationships? Are you focused on price testing and what your revenue looks like? Are you focused on backstage where you need resources and activities and how much that’s going to cost? All of those are outcome-based in the sense of, if we’re trying to hit this price point by delivering our value at this low cost, what’s the smallest test we can run to see if that’s true.

If this is what we predict will happen, we can run some experiments and see how far off we are. Are we close to hitting that number or hitting the outcome? With regards to business modeling, you’re always trying to focus on what are the outcomes and what’s the evidence you can use to inform your strategy. You don’t want to just give a team a list of experiments to go run. Instead you want to say, we’re trying to test this outcome to see if this is possible for us. What are some ways we might go about doing that?

It’s nice because it puts some guardrails on it, but it doesn’t necessarily dictate to them, here’s the 20 experiments you’re going to run. It’s more of use your creativity and your skill and your team to figure out, okay, well if we were trying to achieve this outcome, what are some different experiments we’d run? And then how would we report out the progress toward the outcome. Then you go back to your stakeholders with your hypotheses. You can explain the experiments you ran, the outcomes you observed, and what you recommend next. It becomes more of a conversation where you’re giving an account of how you made progress and not necessarily just being held accountable to running a bunch of experiments.

Felipe’s take: In my experience, when teams are learning OKR and experiments, they start by focusing on achieving the OKRs themselves and not on testing the business model. Validating the business model is usually for more experienced teams or for teams who are launching something completely new. So most people start with the situation that David describes in the second and third paragraphs above.

What are some of the most common mistakes that people make when testing business ideas?

David: It usually comes back to biases—there are three big biases that come up time and time again. One is confirmation bias. It’s this idea that you’re just confirming your beliefs in theories. And so the problem, the risk there is prematurely validating your hypothesis by setting the bar really low.

Overconfidence bias comes up time and time again. Your perception of your judgment is actually much greater than it really is, so you have excessive confidence in your own answers and might question whether you even need to experiment. 

Finally, there’s experimenter’s bias. When you’re performing the research, you may have unconsciously influenced the results. This is when we usually discard anything that doesn’t agree with our hypothesis. One of the ways you can kind of address that is to be more diverse and invite different perspectives into the process when you’re synthesizing the data. Don’t just do it by yourself. Running multiple experiments helps there, too.

How can companies get better about embracing a culture of experimentation?

David: I feel like there’s this gap where the companies that are embracing experimentation are getting further and further ahead of the companies that aren’t willing to do it at all. The gap is only getting bigger and that’s going to be a problem because some of these companies aren’t going to be able to catch up or they’re late adopters to experimentation and they’re going to have a really hard time. 

To change this, it’s very much about the culture of your company and your mindset. If you have something that you feel like you’ve tried to solve a few different ways and it just hasn’t worked, that’s usually because you don’t understand the problem very well. So I recommend thinking deeply about the problem, trying to run experiments, and generating some more evidence about what the solution could be. A lot of times, we don’t understand the problem enough and we’ve jumped to the solution too early.

For leaders, the best thing they can do is try to design an environment in their organization where this can occur. That’s things like giving people time to work on this, not micromanaging, leading with questions instead of leading with answers all the time. And that really challenges the core leadership because if you feel like you always have to know the answer, you’re probably going to have a hard time saying, we’ll just go experiment and learn because you’re going to have this inherent desire to always say what the answer could be. 

If there’s one thing that you’d like people to take away from your book, what is it?

David: To be open to the idea of being wrong. I think the way we were taught in schools, at least the last couple of decades, has been very much rewarded for being right. There’s usually one single right answer. But in your organization and your business—and your life for that matter—there’s rarely one right answer. They’re just choices. 

I would love for people to feel like they can actually apply their creativity and it’s okay if they’re wrong. It’s not at the end of the world. It’s really a small test that you’re running. How do you learn from that and try again? 

We actually created a job story for the book (using the Jobs to Be Done framework). For the reader, the job to be done for the book was: “When I have a business idea, I want to be able to rapidly test it so that I don’t build something nobody wants.”

Testing Business Ideas will be released on November 12. You can find the book here.