(2024-08-28) Xmrit Course

Cedric Chin has a free email course on using XmR charts, esp with his Xmrit tool. https://xmrit.com/

[Day 2] The Basics of Using XmR Charts

the three rules of XmR charts.

An XmR chart consists of two charts: the X chart (X here means the variable, or metric, you’re interested in), and the MR chart (which tracks the ‘Moving Range’, or the absolute differences between each successive point in the X chart).

You’ll notice a few properties of an XmR chart.

First, you’ll notice that there are ‘limit lines’ — one limit line above the chart, and another below the chart. Broadly speaking, these lines tell you what to expect: that if nothing changes, your metric should wiggle between the two lines.

If your metric just wiggles between the two lines, we will call this routine variation

If your metric is doing something abnormal — that is, something unexpected is happening, or your metric itself is changing — this is exceptional variation

XmR charts come with a few rules for identifying exceptional variation. Exceptional variation is considered a ‘signal’

Donald Wheeler simplified the rules down to just three.

These three rules are the rules that are built into Xmrit’s free tool.

Rule 1: The Process Limit Rule

Rule one states that if a point lies outside the limit lines (the blue lines), on either the X chart or the MR chart, something unusual is going on

Rule 2: The Quartile Limit Rule

if you have a run of three out of four successive points that is closer to the limit lines (blue dotted lines) than the centre line (red dotted line), then this is a moderate source of exceptional variation

Rule 3: Runs of Eight

if you have data points in a row on one side of the average (red) line, this is a weak source of special variation and you should investigate

Some Tricks

First, signals can show up at smaller time scales that don’t show up at larger time scales.

measure only as quickly as you think your process can change

your company’s sales process can’t change on a daily basis (you’re not changing sales training or onboarding every day are you?)

Second, signals are best when they’re recent.

The weaker the signal, the more difficult it will be to find the root cause

Our point: XmR charts are best used operationally, with your looking at the charts each week. That way you can investigate when a signal is still fresh.

The short, technical explanation for what the XmR chart does is that it is a homogeneity test: it checks that every data point in a time series is independent and indentically distributed.

What the XmR chart does is that it detects the presence of more than one probability distribution in the variation of a given time series.

The limit lines on an XmR chart estimate 3 sigma around the centre line

• see other source for discussion/equations and constant-values for calculating Control Lines (aka Limit Lines)

This is Rule One

Rule Two detects a shift between 1.5 and 2 sigma. Rule Three detects a shift in the neighbourhood of 1 sigma.

XmR charts is a type of Process Behaviour Chart (PBC). It is, in fact, the simplest PBC available

they have gone by other names, such as the ‘Shewhart Chart’

[Day 3] XmR Charts Should Make You Think Different

XmR charts work for any metric that displays routine variation.

if you are not sure if an XmR chart would help, you should just give it a try. Plot your business metric on an XmR chart with our free tool and eyeball it. You can usually tell — within seconds — if it’s a good idea.

When you step on a weighing scale in the morning, you know that your weight will not stay the same from day to day.

Here’s something that you might already recognise. In business, it is common to be presented with metrics like:
Sales is down 12% YTD (year to date).
Sales is up 15% MoM (month-on-month — that is, compared to last month).

is it meaningful for me to tell you that your weight is 3% higher WoW?

The real question you should be asking is “is this change meaningful?” Meaningful requires you to ask for more than just a percentage change — you want to know what the historical wiggling looks like

let’s say that sales has done badly last month. Your sales figures are 20% below the previous month’s number. Should you yell at your sales force?

What if it’s routine variation?

*However, at this point some of you might shoot back “Well, salespeople have agency! They have control over their own performance! We should scold them, so that they work harder!”

And the truth is that this will work … but up to a point. Your sales people may get slightly better numbers next month if you yell at them*

In truth, systems with people in them also show routine variation

you should work to permanently increase the full range of routine variation you observe.

[Day 4] Other Ways to Use XmR Charts

Today, let’s talk about other uses of the XmR chart. We shall describe four.

how do I know if a metric is actionable or not?

With XmR charts:
You know what is ‘normal’ for a given process.
You know what to investigate (investigate only when you see one of the three rules!)
More importantly, you know what not to investigate.

if you investigate exceptional variation and:
Find something negative, you can ask yourself “how do I prevent this from happening in the future?” Or … If you find something positive, you can ask: “how can I make more of this happen in the future?” Which results in your uncovering what truly affects your numbers. And if you repeat this often enough, for enough business processes, you will begin to build a ‘causal model’ of your business in your head. You will begin to know how your business truly works.

With XmR charts, you are now able to check if your process improvement has actually improved some set of numbers you care about.

XmR charts ALSO tell you how to improve a process.

The logic is simple:
A process is predictable if it only shows routine variation. If a process is predictable, it is already performing as well as it possibly can. Your job is to fundamentally rethink the process.
A process is unpredictable if it shows routine and exceptional variation. If a process is unpredictable, your job is to investigate (and possibly remove) the points of exceptional variation.

[Day 5] What to do about Variation?

Today, we’re going to talk about the two types of variation, and then talk about them in the context of business metric improvement.

*If you are regularly looking at a business metric, you are typically doing so for one of two reasons:

You are actively working on it, trying to get it to improve. Or … You’re checking on it to make sure it doesn’t degrade.*

What do you need to do when you’ve discovered a source of exceptional variation? Well, if it is a positive change, then you should see if you can do more of it. If it is a negative change, then you might want to change your process to prevent it from happening again.

Sometimes, though, you will discover that the exceptional variation is the result of a one-time event that you have no control over:

investigating and identifying root causes improves your understanding of your business. It is important to have that discussion. A weekly metrics review meeting where you discuss exceptional variation with your team is an excellent opportunity to build that understanding together.

On the other hand, let’s say that you’re dealing with a predictable process. What does it mean to fundamentally rethink the process?

Well, depending on the process, it might mean changing one step

The point here is to try. Take a guess; make a change

Often, your changes might not result in any process improvements — at least, nothing detectable on an XmR chart. But the goal is to keep trying until you discover something that does.

The Two Signs of Process Improvement

You can reduce the variation (narrow the limit lines on your XmR chart). Or you can shift the entire band of variation up (or down).

In manufacturing, the usual recommendation is to 1) reduce the variation first, and 2) then shift the entire band of variation.

The first argument is that there is a cost associated with variation

The second reason is that if you reduce variation before you attempt to shift the process, it will be easier to see when process improvement has occurred

Of course, this might not be possible

Wrapping Up

you will be surprised by the kinds of systems that show routine variation

[Day 6] How Many Data Points?

A common question you’ll have at this point is “how many data points should I wait before I can trust what my XmR chart tells me?”

This question is so important that we’ve included it in our User Manual. (Relevant section linked).

The rule of thumb goes as follows:
6 data points is the absolute minimum.
Between 6 to 12 data points: the limit lines begin to gel.
Between 12 to 20 data points: the limit lines begin to harden.
More than 20 data points: there is typically marginal benefit to waiting for more data.

If you want to establish limit lines for a stable process — use 15 to 20 data points. You’ll want to lock your limit lines, and only recalculate them when you see a permanent change in your process.

If you are introducing a process change and want to see if it’s actually affected your process, 6 to 8 data points may be enough!

In Xmrit, place a divider at the point when you change a process, and then wait to observe subsequent process behaviour

Finally, give a thought to the time grain that you’re looking at. As a general rule of thumb, anything more than a year and a half ago (say, 18 months) may be irrelevant for your current use case.

[Day 7] When do XmR Charts Fail?

There are actually four major situations in which XmR charts do badly. In this free email course, we’re going to talk about two of the most common ways XmR charts fail.

The first — and most famous — failure mode is known as ‘chunky data’. Chunky data is defined as ‘the distance between the possible values becomes too large’.

if you round the measurements to just one decimal place

Let’s take a more realistic example. The following is an example from Donald Wheeler. Let’s say that you’re monitoring a production process for a plastic knob. The measurements you have are recorded to the nearest 0.01 of an inch.

With chunky data, you’re going to get lots of false alarms.

you have chunky data when you have “less than four different values on the Moving Range chart”.

Don’t round off and reduce precision unnecessarily!

Some types of data are naturally chunky. For instance, some events are rare.

In such a situation, you should count the number of days between accidents instead, and plot that on an XmR chart.

you can imagine other metrics that might be naturally chunky — for instance, ‘number of articles that end up on the Hacker News or Reddit front page’ — which should be 0 most of the time, and then 1 perhaps once — or twice! — a year.

Failure Mode 2: A Combination of Two Different Processes

XmR charts have about a 3% false alarm rate for most types of data you will get in the real world. However, if you are measuring a stream of numbers taken from a combination of two processes, you should expect this false alarm rate to go up to something as high as 11%.

Well, take the following metric — Net New Subscribers — as an example

This is a combination of two different metrics: new newsletter subscribers, and newsletter churn.

The net result? Always separate your processes and chart them separately. see related (2022-12-08) Schmidt Strategy Systems Thinking And Being Wrong

[Day 8] When Can You Trust Your Limit Lines?

A question that commonly comes up is: “I’ve observed exceptional variation in my metrics. Can I trust the limit lines when exceptional variation is present?”

This questions touches on slightly more advanced topics.

The short answer is no: you cannot.

This gets at a fundamental assumption with the way XmR charts are supposed to be used.

If you observe exceptional variation in your data, you should investigate and remove those sources of exceptional variation first!

Only when your process is predictable (shows only routine variation) should you trust in the limit lines.

When our data contain signals we are open to the mistake of missing these signals. It is only when our data do not contain signals that we are open to the mistake of getting false alarms.

You need not be concerned with the probability of false alarms until you have finally learned how to operate your process predictably.

This has a practical implication, though: if you observe a one-off event that is exceptional, you should remove it from your limit lines calculation. Doing so will allow you to establish trustworthy limit lines.

Eight in a row

A slightly different action is required if you observe eight data points in a row on one side of the centre line. (This is Rule 3 of the three rules for detecting exceptional variation).

this is a clear sign that your process has shifted to new behaviour.

You should investigate to find out what happened.

Follow-up questions I had

• what if your series is trending? Answer: transform (plot the "errors" from the interpolation)
• what if your series has seasonality (month-of-year, even day-of-week)? Answer: basically the same idea: you're creating a simple prediction model, and plotting the errors.