(2024-01-08) Chin Becoming Data Driven From First Principles

Cedric Chin: Becoming Data Driven, From First Principles. I’ve increasingly come to appreciate that you must earn the right to criticise becoming data driven.

At the other end of the spectrum, there are many screeds about why becoming ‘data driven’ is bad, about how we should be ‘data informed’ instead, about why “just looking at the numbers” can lead one astray.

We can all quote stories about misaligned incentives, or bad financial abstractions, or stories about warped organisational behaviour — all enabled by or caused by dumb usage of data. It’s too easy to throw shade at the very idea of becoming data driven.

it’s really the principles of Statistical Process Control (and/or Continuous Improvement; pick your poison) that you want to internalise, because it is those principles that led to the WBR, and it is through those principles that you can come up with equivalently powerful mechanisms of your own.

This is a controversial topic, and for good reason. On the one hand, you might say “of course we should be data driven — that’s just what good businesspeople do!” Notice the expectation embedded in that sentence. Never mind that few of us, if any, have ever been taught basic data literacy

Why Become Data Driven?

I’m going to attempt something ambitious with this essay: I will explain how one might go from ‘understanding variation’ to the structure of the Amazon-style Weekly Business Review (WBR).

This piece will set up for the next two essays in the Becoming Data Driven series, which will describe the actual practice of the WBR. And though it might not seem like it, I believe this piece is actually the more important one

when you open a business dashboard in your company, do you feel a little confused? Like you don’t know what to do?

W Edwards Deming believed that there was no such thing as truth in business. He argued that there is only knowledge. (2023-04-04-ChinThereIsNoTruthInBusinessOnlyKnowledge)

Why is This Hard?

If they don’t hit their targets, they make up some spiel about how “if you hit 100% of your OKRs you’re not being ambitious enough” (or they just, you know, doctor the numbers). They’re never entirely sure how it all fits together.

Deming and colleagues — the aforementioned pioneers of SPC — believed... knowledge is evaluated based on predictive validity alone. (thinking in bets)

So if being data driven is so important, why is it so rare?

The obstacle is that the chart wiggles.

Deming taught basic data literacy.

when you hire a new software engineer, or you launch a marketing campaign, or you change your sales incentive scheme, you are in effect saying “I predict that these changes will lead to good outcomes A, B and C. I also believe that there will not be any bad consequences.” But how would you know?

the purpose of data is to give you a causal model of your business in your head. If reality changes, the figures (metrics) in your business should show it.

They are, as statistician Donald Wheeler likes to say, “a style of thinking with a few tools attached.”

In learning these methods, these operators earned the right to recognise when data is used badly. And they are able to do so credibly because they have a viable alternative to these practices.

The purpose of data is knowledge. Knowledge is ‘theories or models that allow you to predict the outcomes of your business actions.’

Most people do not use data like this.

what is the purpose of data?

what is the biggest obstacle to becoming data driven?

We shall define knowledge as “theories or models that allow you to predict the outcomes of your business actions.”

The ideas we will examine in this essay are very simple, very easy, and surprisingly niche given their power.

Most business charts will look like a crayon scribble by a two-year-old. This is more problematic than you might think.

Here are three examples to illustrate this

You Don’t Know If You’ve Successfully Improved

You make a change to the way Marketing Qualified Leads are vetted and wait a few weeks. Then you take a look at your data:... Did it work? Hmm. Maybe wait a few more weeks?.... Did it work? Did it fail? Hmm. Perhaps it’s made an impact on some other metric? You take a look at a few other sales metrics but they’re all equally wiggly. You discern no clear pattern; you chalk it up to a “maaaybe?” and move on to the next idea. You don’t get any feedback on your moves. You’re like a blind archer, shooting arrows into the dark.

This is one reason people don’t close their loops. (open loop)

You Waste Time Chasing Noise

Your boss opens up the sales meeting and says “Sales is 12% down for the month. This is very bad! We’re no longer on track to hit our quarterly targets!”

you can’t find anything wrong. Perhaps this is just normal variation?

So you make up some explanation, he accepts it, and then next month the number goes up again and everyone breathes a sigh of relief.

Sometimes You Set Dumb SLAs

You’re in charge of data infrastructure. You have a maximum latency SLA (Service Level Agreement) for some of your servers

Every two months or so, the latency for a key service violates your SLA. You get yelled at. “WHAT WENT WRONG?” your boss sends over the channel, tagging you.

It doesn’t cross your mind that perhaps the process’s natural wiggling will — for some small % of the time — violate the SLA.

The problem is compounded by the fact that once every year or so, there is some root cause: some service goes awry; some engineer trips up during a deploy

The real solution to your ‘regular two month yelling’ isn’t to investigate routine variation; the solution is to completely rethink the process so that the entire range of variation lies below the SLA.

All Roads Lead to the Same Outcomes

The eventual outcome of all three scenarios is identical: you stop using data as an input to drive decision making. And why should you? The feedback you get is not clear.

There are a hundred articles about how to set up a data warehouse but not one about how business users should read charts.

*If these metrics become part of some goal-setting system, then another predictable outcome will occur. Joiner’s Rule says: When people are pressured to meet a target value there are three ways they can proceed:

1. They can work to improve the system
2. They can distort the system
3. Or they can distort the data* ((2023-01-23) Chin Goodharts Law Isn't As Useful As You Might Think)

What are some common methods to solve for this? Some people attempt to use a moving average:

What if, say, the moving average after a change just happens to fluctuates up, you declare a successful change, and then a few weeks later it fluctuates back down? Does that represent a real shift? Hmm.

Or perhaps you should draw a linear trend line through the data?

But the issue with a linear trend is that it takes very little to change the trend!

This challenge that I’ve just demonstrated to you are all symptoms of the same problem. They are a problem of understanding variation. A wiggling chart is difficult to read.

A large change is not necessarily worth investigating, and a small change is not necessarily benign

What you want to know is if the change is exceptional — if it’s larger than historical variance would indicate.

What If It Doesn’t Have To Be This Way?

Let’s say that you have a tool that tells you, damn near certain, “yes, a change has occurred.” By implication, this magical tool is also able to tell you: “no, this change is just routine variation, you may ignore it.”

when you attempt to improve a process, you will know when things work

This seems like a small thing, but it leads to a number of non-obvious, knock-on effects

You come into possession of this magical tool. The magical tool is a dumb charting technique

You throw it at some metric that you care about, like website visitors. You try a bunch of things to improve the number of visitors to your website. Most of your efforts fail

You keep trying new changes, and this magical tool keeps telling you “no, there’s no change” repeatedly, and you start to lose faith

At some point, one of two things will happen: either you make a change and then suddenly your tool screams SOMETHING CHANGED! Or — out of nowhere — something unexpected happens, and some subset of your metrics jump, and your tool screams SOMETHING CHANGED, and you scramble to investigate

You discover the thing that caused your website visitors to spike. The next week, you do the thing again, and the metric spikes again. You stop doing it, and your tool tells you “SOMETHING CHANGED” — your metric returns to the previous level of variation

Congratulations: you’ve just discovered controllable input metrics.

You repeat steps 1 through 4, and over a series of months, you begin to discover more and more control factors that influence site visitors. (Spoiler alert: there typically aren’t that many. But the few that you discover truly work.)

At this point you say “wait a minute, what else can this be used for? Can it be used for … revenue?”

You start to apply this to revenue, and the same thing happens

you begin to systematically drive revenue up because you actually know what affects it.

At this point you are convinced this methodology works, so you start applying it to everything in your company. Hiring problems? You instrument your hiring process and throw this magical tool at it

Costs? You start instrumenting various business costs and throw this tool at it, investigating instances where there are exceptional spikes or drops

Marketing performance? You instrument various parts of your marketing activities, and watch closely to see if changes in various marketing activities cause follow-on changes in downstream metrics

You realise that every goddamn metric is connected to every other goddamn metric in your company... You realise that it makes sense to bring leaders from every department together to look at company metrics, so they can see how changes flow through the numbers from various departments and out to financial metrics out the other end.

At this point you have the same causal model of the business in your heads. You have, in Deming’s parlance, knowledge

Some weeks you add new metrics in response to new control factors you’ve discovered, and in other weeks you remove metrics that have stopped being predictive

Borders between departments break down. Executives collaborate.

People begin to internalise “hey, I can try new things, and can get relatively good feedback about the changes I try!” Eventually someone gets the bright idea to let software engineering leaders in on this meeting, because they can change the product in response to the causal model of the business

Mary, who has tried to get A/B testing to take off in the product department for the past two years, finally gets the green light to get an A/B testing program off the ground. Execs at the highest level finally see A/B testing for what it is: yet another way to pursue knowledge.

The Trick

Have you noticed the trick yet?

it’s not the magical tool that’s important in this story. It’s the cultural change that results from the tool that is more important.

We’ve talked about one tool. Now let’s talk about the style of thinking it enables.

one of the earlier entries in the Data Driven Series is titled “Operational Excellence is the Pursuit of Knowledge.” Let’s call this the ‘process control worldview’: every business is a process, and processes may be decomposed into multiple smaller processes

Most people approach data with an ‘optimization worldview’. (Special thanks to Ergest Xheblati and Shachar Meir for this observation). Crudely speaking, they think in terms of “make number go up.

I will set a Big Hairy Audacious Goal.” (Never mind that nobody knows how to hit the goal, setting the goal is inspirational and inspiration is all you need).

The process control worldview is different. It says: “Here is a process. Your job is to discover all the control factors that affect this process

Your job is to figure out what you can control that affects the process, and then systematically pursue that.”

Uncovering the causal structure of your business informs your business intuition; it doesn’t stand in opposition to making large discontinuous bets.

Amazon, for instance, has a famous OP1/OP2 goal-setting cycle that is similar to OKR practices in many other large companies; they just happen to also have a strong process control approach to their metrics. Early Amazon exec Colin Bryar told me that it was always important for leadership to incentivise the right controllable input metrics, and to avoid punishing employees when leadership selected the wrong input metrics

it’s important to understand that the process control worldview opens the door to a few vastly more interesting organisational setups.

why stop at a goal? What if you design an org structure that incentivises workers to constantly search for the pragmatic limit, and then pursue improvements right up to that limit? (continuous improvement)

• If you have a culture of yearly OKRs, your workers might not to be incentivised to exceed the goals you’ve set out for them. After all, next year’s goals will simply be set a little higher.

It takes a different kind of incentive structure to create a culture of ceaseless improvement, pushing outcomes across the company — from transmission tolerances right up to financial profits — all the way against the pragmatic limit, making tradeoffs with other important output metrics

these ideas have led to the Toyota Production System, which in turn has led to the discovery of Process Power — one of Hamilton Helmer’s 7 Powers (i.e. there are only seven sustainable competitive advantages in business and Process Power is one of them

The Magical Tool

What is this magical tool? How does it work? How do you use it?

The magical tool is something called a Process Behaviour Chart

The PBC goes by many names: in the 1930s to 50s, it was called the ‘Shewhart chart’, named after its inventor — Deming’s mentor — the statistician Walter Shewhart. From the 50s onwards it was mostly called the ‘Process Control Chart

Of all the PBCs, the most ubiquitous (and useful!) chart is the ‘XmR chart’.

Using the XmR chart to Characterise Process Behaviour

The XmR chart tells you two big things. First, it characterises process behaviour

it is able to tell us one other thing. It tells us when exceptional variation has occurred.

Using the XmR Chart to Detect Exceptional Variation

means that something meaningful has changed

SPC teaches us that you cannot improve an unpredictable process. Why? Well, an unpredictable process means that there is some exogenous factor that you’re not accounting for, that will interfere with your attempts to improve your process. Go figure out what that factor is first, and then change your process to account for it.

How does the XmR chart tell you these things? It has a set of rules. Historically there have been many rules. Wheeler has simplified it down to just three.

• Rule 1: When a Point is Outside the Limit Lines: source of special variation and you should investigate
• Rule 2: When Three out of Four Consecutive Points are Nearer to a Limit than the Centre Line: When three out of four consecutive points are nearer to a limit than to the centre line, this is a moderate source of special variation and you should investigate
• Rule 3: When Eight Points Lie on One Side of the Centre Line

The Intuition Behind XmR Charts

What I want to do here is to give you a rough intuition for why XmR charts work.

The intuition goes like this: all processes show some amount of routine variation, yes?

work for most probability distributions that you would find in the wild.

The XmR chart does this detection by estimating three sigma around a centre line.

Sidenote: it is important to note that you should not use a standard deviation calculation to get your limit lines. A standard deviation calculation assumes that all the variation observed is drawn from one probability distribution … which defeats the whole point of an XmR chart!

some implications that fall out of this approach

• For starters, you do not need a lot of data to use XmR charts.
  • In 1992 Donald Wheeler taught these methods to a Japanese nightclub, and had the waitresses, bartenders, and managers learn this approach to improve their business. It worked
• Second, XmR charts do not require you to be statistically sophisticated
• Third, and finally, I’ve been using XmR charts for about six months now, and I can confirm that they work for a very large number of business processes

Moving Beyond the XmR Chart

At some point, after you’ve gone down this path long enough, after you’ve transformed your organisation to think in this way, you should discover that you no longer need to use them.

my team and I are not able to do without XmR charts. But Toyota no longer uses them.

the Amazon WBR may draw from the process control worldview, but it does not use process behaviour charts of any type

So what are they good for?

The XmR chart is most potent as a cultural transformation lever
Wheeler likes to say that XmR charts are the beginning of knowledge. What he means by this, I think, is simple: the XmR chart is the quickest, most effective way to create org-wide understanding of variation

You will begin to look askance at those who use the optimization worldview. Wheeler has observed that those who successfully adopt PBCs are quite likely to find their way to Continuous Improvement.

They are not strictly necessary. Colin told me that if you looked at operational data often enough, and for long enough, you would be able to recognise the difference between routine and special variation

This tacit understanding of variation came up again and again as I sought out and interviewed data-driven operators from multiple companies over the past year. (tacit knowledge)

Eric Nehrlich was part of a team that got Google’s revenue forecasting error rate down from 10-20% to ~0.5% over a period of a few years. Half our interview was about his experience: Eric Nehrlich: Yeah, I think the main thing people would get wrong is panicking when the number is below the forecast... we should try to be 50% above, 50% below. We should expect to be below the forecast 50% of the time.

this gets back to the question of what purpose is the forecast meant to serve? If it's meant to be accurate, you have to be prepared that there's gonna be negative days. And sometimes big negative days because, as we know from variation, one out of a hundred days is gonna be three sigma away.

It's educating yourself on that and then educating your stakeholders. Like this is what we expect. We expect some above, some below

So why is this an open secret?

I think there are two contributing factors: the first is that nobody seems to teach this — at least not beyond the factory floor.

Sure, Deming was willing to teach his methods to anyone who listened, and he knew goddamn well they worked beyond manufacturing — but then he mostly taught them to manufacturing companies and then he died

Which leaves us with the second factor: if nobody teaches this, then a tacit understanding of variation only emerges if you have the opportunity to look at lots of data at work! This is incredibly depressing. It implies that you cannot become data driven in a company with no prevailing culture of looking at data

Wrapping Up

You have to get as lucky as Jeff Bezos did, when Amazon hired manufacturing savant Jeff Wilke from AlliedSignal. And then you had to be smart enough, and curious enough, to notice that Wilke was reorganising Amazon’s warehouses into factories — Fulfilment Centers, he called them — and that he was applying all of these ideas to solve Amazon’s distribution problems

The key argument I want to reiterate is that the XmR chart is not the point — the pursuit of knowledge is the point.

When you leave the XmR chart behind, what does the ‘pursuit of knowledge’ really mean? It means seeking out all the causal factors that power your business

When you interview customers rigorously, you are actually pursuing knowledge

When Amazon switched to Free Cash Flow as its primary financial measure under CFO Warren Jenson, they built a tool that would tell employees the FCF impact of every product decision they made

When Alan Mulally forced every Ford executive to attend a weekly Business Plan Review, he did so with the intent that they got the end-to-end feel of the various parts of the business... Mulally used this to force incompetent, politicking execs out. For the execs who remained, this meeting gave them a causal understanding of Ford.

When Bernard Paulson began systematically isolating each production unit in Koch Industries’s Pine Bend refinery, he was looking for knowledge... Koch HQ did not set revenue targets; they focused on long-term profit growth.