Today’s interview is with Stefan Thomke, who is the William Barclay Harding Professor of Business Administration at Harvard Business School. Stefan joins me today to talk about his new book, Experimentation Works: The Surprising Power of Business Experiments, what does experimentation mean, what does it look like, what are the benefits, why does it matter for customer service and experience leaders and practitioners.

This interview follows on from my recent interview – Emerging leadership lessons from the pandemic and what we need from leaders now – Interview with Rachel Neaman – and is number 356 in the series of interviews with authors and business leaders that are doing great things, providing valuable insights, helping businesses innovate and delivering great service and experience to both their customers and their employees.

Here’s the highlights of my chat with Stefan:

• Stefan wrote his first book about experimentation in business and innovation in 2003 called Experimentation Matters. That book was more about how experimentation technologies are changing the way product development and R&D is being done.
• However, experimentation has moved out of product engineering and R&D and it is being used in lots of different settings, especially online.
• Experimentation Works is about how experimentation is something that a whole organization needs to embrace. It’s a mindset, it’s an approach to decision making and to innovation.
• Large scale experimentation is used by lots different companies but the highest profile ones include Microsoft, Google, Netflix and Booking.com.
• These organizations run tens of thousands of experiments annually on you and I. We just don’t know that they are doing it.
• As a sign of their success these organizations have significantly out-performed the S&P 500 over the last few years.
• The ability to run large scale experimentation is a significant factor in their performance.
• Stefan tells a story about how some Microsoft employees ran out of patience with a program manager who considered their idea low priority. However, they ran their experiment which was to do with their search engine and how ad headlines are displayed. The idea itself could sound trivial but the tiny change their were testing resulted in an increase of more than $100 million of revenue in the first year and it didn’t hurt any key user experience metrics.
• In the online world, small changes can make a huge difference.
• Ideally in experiments you want to test an idea or hypothesis to see how well or how badly it works. But, in doing that you would ideally want to hold other variables constant. In practice that is hard.
• Randomization helps with that and is a technique that takes all of the other causes and distributes them randomly across different treatments. That means that the other causes are still present and they may still change, but they don’t bias results of the experiment one way or the other.
• To adopt this type of practice requires a total change of attitude. It’s a different way of making decisions. It’s a different way of innovating and there are several cultural elements that are necessary to make this work at large scale.
• One of those things is to be able to cultivate curiosity. Everyone in an organization from the leadership on down needs to value surprises.
• You have this mindset that curiosity prevails and people will see failures not as costly mistakes, but they see it as opportunities for learning.
• Another aspect is that data trump opinions. If the decision is already made, why run an experiment?
• In an experimentation organization, the results of the experiment must prevail even when they clash with strong opinions and no matter whose opinions they are.
• The problem, of course, is human nature. We tend to a happily accept good results, that confirm our biases, but we challenge and thoroughly investigate bad results that go against our assumption. And so, we have to really keep ourselves disciplined.
• But that does not mean following data and results blindly. Sometimes you may get results that conflict with your strategy or may violate a contractual commitment that you have.
• When we run an experiment we’re adding transparency to decision making.
• Experimentation organizations empower people to run their own experiments.
• At Booking.com, anybody can launch an experiment on millions of customers without management’s permission.
• They run their experiments of the one platform. Whenever somebody wants to launch a new experiment, they broadcast the experiment to everybody else in the organization for feedback, they also check their experiment database for details of what people have done before and what happened.
• And, anybody at booking.com can stop any experiment at any time. That’s what they call the nuclear option.
• Running experiments on this sort of scale requires operating with ethical responsibility.
• The stages of experimentation maturity are Awareness, Belief, Commitment, Diffusion and Embedded.
• In the environment that we find ourselves in experimentation is the only way forwards.
• Richard Feynman said “It doesn’t make any difference how beautiful your guess is, it doesn’t make any difference how smart you are, who made the guess or what his name is. If it disagrees with the experiment, it’s wrong. That’s all there is to it.”
• Stefan’s Punk CX word(s): DIY, Empowerment, Experimentation.
• Stefan’s Punk CX brand: Booking.com
• One booking.com executive told Stefan: “Large scale testing is not a technical thing. It’s a cultural thing that you should fully embrace.”
• He raised two interesting questions for CEOs:
  • How willing are you to be confronted every day by how wrong you are?
  • How much autonomy are you willing to give to the people who work for you?
• And, if the answer is no to these questions, it’s never gonna work.

About Stefan

Stefan Thomke, a leading authority on the management of innovation, is the William Barclay Harding Professor of Business Administration at Harvard Business School. He is author of the books Experimentation Works: The Surprising Power of Business Experiments (HBR Press, 2020) and Experimentation Matters: Unlocking the Potential of New Technologies for Innovation (Harvard Business Review Press, 2003), as well as over one hundred articles, case studies and notes in books and journals, such as Harvard Business Review, Management Science and MIT Sloan Management Review. Thomke has taught in and chaired numerous executive education programs, both at Harvard Business School and in companies around the world. He is also a frequent conference speaker and advisor to global business leaders.

Grab a copy of Stefan’s new book here, check out his faculty page at HBS, and connect with him open LinkedIn here.

Image by ar130405 from Pixabay