Mistakes in Evidence Based Management (and how to avoid them)

If doctors practiced medicine the way many companies practice management, there would be far more sick and dead patients, and many more doctors would be in jail. This is because medicine, unlike business, has a long and proven history of relying on evidence to make decisions.

Despite the hyperbole around big data – it’s the new oil, haven’t you heard? – the rise of data analytics and more; many decisions inside organisations are made on anything but evidence.

Instead, decisions are based on:

  • Hope
  • Fear
  • What others seem to be doing
  • What senior leaders have done before and believe has worked in the past
  • Dearly held ideologies

This doesn’t seem very empirical, does it?

According to Stanford Professors Pfeffer & Sutton, these are top three poor decision-making practices:

  1. Casual Benchmarking
  2. Doing what worked in the past
  3. Following deeply held yet unexamined ideologies

Below we review each of these traps, as well as tips to avoid them.

Casual Benchmarking

Benchmarking can be an incredibly powerful tool. Any single performance is largely measured against others for context. If we had never timed anyone run the 100 metre sprint, we wouldn’t have context for Usain Bolt’s elite performance.

The problem lies in how “casually” this is typically done. The logic behind what works at top performers, why it works, and what will work elsewhere is barely unravelled. This results in mindless imitation. If you were to observe Bolt was wearing Nike shoes, had short-cropped hair, and started his day with yoga and a granola bar, simply imitating this would not see your own running performance increase.

Yet this is the approach companies often take.

When United Airlines decided in 1994 to compete with Southwest in the intra-California market-place, the company tried to imitate Southwest. United put its gate staff and flight attendants in casual clothes. It flew only Boeing 737’s. It renamed the service (“Shuttle by United”). It stopped serving food and increased frequency of the flights.

Much like this author’s best Usain Bolt impression, these changes had little effect on performance. “Shuttle” was ultimately shut-down, a dismal failure.

Similar imitations have been done by numerous auto manufacturers, trying to ‘lift and shift’ the production capabilities of Toyota. Despite decades trying to replicate their Lean methods, most still lag behind Toyota in productivity.

Copycat activities can easily arise within software companies too.

Sam, Product Manager for PetSmart, was called into a meeting with a group of executives. The group was facing lagging sales, and wanted to review the competition.

Executive A: “Our competitor has a 4-tier pricing strategy, whilst we only offer modules.”

Executive B: “Tiered pricing has been best practice for awhile now. Just look at what Atlassian has done, and they are a $40B+ company.”

Executive C: “I think it’s clear we need to change our pricing.”

This kind of gut-based decision making will be all too familiar to software professionals. It stems from a psychological bias known as availability heuristic. This is when readily information is given more weight. Little additional thought or research will be done into what complementary activities those companies perform or how they approach sales. The competitor could be building additional trust in the sales pipeline by running consulting workshops, creating valuable content marking and industry-based certifications, strong referral programs and loyalty discounts, or amazing customer success that drives NPS. Simply changing the licensing model is not going to drive results at PetSmart.

The root cause for the ineffectiveness of casual benchmarking is that performance is often akin to an iceberg. Companies and consultants – in their haste to copy – are only taking activities at the surface level (the visible tip of the iceberg). The most important activities – strategies and thinking – are being done below the surface. 

Many activities are simply copied without asking the why behind these activities and how they might drive performance. It is often the thinking behind the actions that is driving results. Both SouthWest and Toyota had deeply held management philosophies. These total quality management systems and KPI’s around continuous improvement pervaded throughout their entire organisations. It guided not just current activities, but decisions around future activities also. The ability to make these same future decisions was lacking by imitators.

Strategies are often ‘lifted and shifted’ from completely different markets, categories & segments. Start-up culture is drowning in founders pitching their idea as the next “Uber for X” or “Airbnb for Y”. Desperate for innovation, established companies will also chase this fool’s gold, attempting to establish the next ‘App Store’ or customer destination portal, with very little evidence being required to spur them on.

Tip: Don’t benchmark what others do, benchmark how they think.

Before you copy others, ask yourself:

  • Is the success you observe by the benchmarking target attributable to the practice you seek to copy?
  • Sam could ask if PetSmart’s competitor was truly winning due to a tiered pricing structure, or something else?
  • Why is that particular practice linked to performance improvement? What is the logic? If you can’t explain the underlying logic or theory of why something should enhance performance, you are likely engaging in superstitious learning and may be copying something that is irrelevant or even damaging.
  • What are the downsides and disadvantages to implementing the practice, even if it is a good idea? Are there ways of mitigating these problems, perhaps ways your target uses that you aren’t seeing?

Doing what worked in the past

How many times has a new executive joined your company, only to start rolling out things they did at their last company? This is particularly pervasive for activities like sales commission, performance reviews, KPI design and time tracking. These ‘feel’ like generic problems that can be solved with generic solutions. These same templates seem to roll around every season like the flu.

This is something you would hope to never see in medicine.

Imagine walking into a routine medical check-up, and with little examination the doctor declaring she was going to take out your appendix. When you question her diagnosis, she replies confidently “I took out the appendix of my last patient and that worked out well.”

The cure needs to be tailored to the problem, not applied universally.

Yet as humans we need these mental shortcuts. We need to learn from past mistakes and successes, to avoid making them again in future or wasting time reinventing the wheel. So how can we balance our need to leverage past knowledge and mistakes, whilst avoiding ill-fitting solutions?

Before you use what worked in the past, ask yourself:

  • Are you sure that the practice that you are about to repeat is associated with the past success? Correlation does not equal causation; the previous activity may not have been the reason.
  • Is this new situation – the business, the technology, the customers, the business model, the competitive environment – so similar to past situations that what worked in the past will work in the new setting?
  • Why do you think the past practice you intend to use again has been effective? If you cannot unpack the logic of why things have worked, it is unlikely you will be able to determine whether or not they will work this time

Following Deeply Held, Yet Unexamined Ideologies

When people are overly influenced by deeply held ideologies or beliefs – causing their organisation to adopt some management practice not because it is based on sound logic or hard facts, but because managers “believe” it works, or it matches their (sometimes flawed) assumptions about what propels people and organisations to be successful.

There are many examples of ‘common business sense’ that have little statistical evidence, yet remain central to company strategies and decisions.

Stock Options for Executives to Increase Performance

The use (and defense) of stock options as a compensation strategy is a great example of belief trumping evidence, to the detriment of organisations. In the early years of the new millennium, there was an unprecedented wave of corporate bankruptcies and financial scandals. Senior executives lied about their company’s performance, even as they sold stock and left pension funds and other investors holding worthless paper. Experts and evidence now place a large part of the blame for financial scandals on the excessive use of stock options and stock-based compensation.

“At the very least, options tended to promote short-term focus. And at worst they promoted fraudulent activity to manipulate earnings” – Carol Bowie, director of governance research services at Investor Research Centre

A review of 220 studies concluded that equity ownership had no consistent effects on financial performance. Another massive study by the National Bureau of Economic Research reported most schemes designed to align managerial and shareholder expectations failed to do so; instead operating as devices to enrich senior management.

Yet executives, particularly those in the technology space, remain unconvinced. They wax lyrical about how it encourages 80-work weeks, promotes frugal spending and a sense of ownership. And of course they can look at this all very objectively…

First Mover Advantage

“It’s better to be first than it is better.”

Al Ries – The 22 Immutable Laws of Marketing

Existing empirical evidence is actually mixed and unclear as to whether such an advantage as first-mover advantage exists. Many “success stories” purported to support the first-mover advantage turn out to be false. Amazon was not the first online bookstore. The iPod was not the first MP3 player.

In fact, for new products and segments, these first movers do a lot of the heavy lifting to convince such a market is legitimate. They battle through the innovator and early adopter phases only to have later entrants swoop in and steal the market. For more on this topic, see the chapter of the same name.

Given the nuances, complexity and evidence to the contrary, why then does the legend of the ‘first mover advantage’ continue to be so compelling?

The answer is because it would please us if it were true.

From a very young age, we are taught that coming in first is better. That someone should be rewarded for their original ideas. Thus it is more than just intuitive that this concept be correct, but something deeper.

Such beliefs rooted in ideology or in cultural values are “sticky”. They resist disconfirming evidence and persist in affecting judgements and choice, regardless or not they are true.

To counteract this phenomena, ask yourself:

  • “Is my preference for a particular management practice solely or mostly because it fits with my intuitions about people and organisations?”
  • “Am I requiring the same level of proof and the same amount of data regardless of whether or not the issue is one I believe in?”
  • “And, most important, are my colleagues and I allowing our beliefs to cloud our willingness to gather and consider data that may be pertinent to our choices?”


Pfeffer, J. and Sutton, R. (2014). Hard Facts, Dangerous Half-Truths, and Total Nonsense. Boston: Harvard Business Review Press.