How to measure customer retention honestly: control groups, programme-level holdouts, and who should own the comparison.

A retention number can have a perfect definition and still say nothing about whether your work caused it.

In short: A retention rate describes what happened, not what you caused. Control groups answer the causal question for one campaign at a time, provided the test is sized and its rules fixed before it starts. But campaign wins do not add up: reporting by credit, short windows, the selection of winners and decay all push the sum of local lifts ahead of the truth, and overlap between campaigns can push it either way. Only a standing holdout, a small group that receives none of the discretionary retention programme, shows what the programme as a whole adds. Running one costs the net revenue the programme would have earned on those customers, and you only learn how much by running it, so someone has to decide to pay that price knowingly and protect it afterwards.

For the sake of example, picture a retention team that ran ten campaigns last year, each against its own control group, each a verified win. Added up, the reported lifts come to around eight points of retention. A standing holdout, a comparison many businesses never run (yes, you’d be surprised), says the whole programme is worth about two. (Bear in mind the figures are illustrative, drawn from patterns I’ve seen in real programmes; the pattern is common.) Every number in that story can be honest (and not help the operator make better decisions), and the rest of this piece explains how.

Why “Everyone Owns Retention” Means No One Does argued that retention needs one owner, and that the owner’s first act is to define what a retained customer is: which activity counts, over what window, and whether a customer who only returns when paid to is retained or rented. It also set aside a second question, which is how to measure against that definition honestly. This is where we tackle that question.

Let’s break it down. A definition fixes what is being counted. It does not show whether anyone’s work changed the count. A business can hold a precise, agreed definition of retention and a dashboard everyone trusts, and still be unable to say whether its retention effort is doing anything at all. What is missing is a comparison.

A retention rate describes what happened. It cannot say what you caused.

Retention (not as a definition but as a number or goal) is measured after the fact: of the customers active in one period, how many were still active in the next. That is a description. The question every retention programme implicitly claims to answer is a different one: how many of those customers would have left if the programme had not existed? And this often can significantly skew the organisation’s view of the success (or lack thereof) of its “retention actions”, since that counterfactual is never observed directly, and most reporting skips it.

The omission produces errors in both directions. Say a win-back/reactivation campaign goes to customers who have been inactive for 60 days, and 12% of them return within a month. It reads as a result and some degree of success. However, some lapsed customers return regardless: whether due to a seasonal pattern, a payday, a habit resurfacing or bouncing between operators. Now, when a comparable group that received nothing from the same campaign returned at 9%, the campaign’s effect is in reality three points, not twelve. In the other direction, retention can fall simply because market demand softened while a retention programme was genuinely helping, and the dashboard shows the same decline either way. We can see how this can easily skew what is seen as a successful programme and an ineffectual one, and the consequent ineffective use of resources.

Attribution rules are the usual answer, and they are necessary. They decide who gets credit when several campaigns touched the same customer, and they should be fixed in advance, not argued after the result (which happens more often than one believes). But they answer a different question. Attribution assigns the credit; it cannot say whether the credit is deserved. Perfect attribution would still hand all twelve points to the campaign. Without a comparison, a number can be claimed by whoever touched the customer last, or blamed on whatever changed most recently, which is the selective claiming the ownership article described.

The comparison also prices the waste.

The same arithmetic changes the economics. Let’s put numbers on the example above: take 10,000 customers who received the campaign and a control group of 1,000. In the campaign group, 1,200 return. In the control group, 9% do, which scales to 900 of 10,000. So about 900 of those customers would have come back anyway without any action, and suddenly three quarters of the people who returned did not need the campaign to do it. This means that if an incentive went to every returning customer, then in expectation three quarters of that spend landed on customers who would have returned regardless (there is no telling which ones), and the real cost of each additional reactivation is four times the reported cost per reactivation (1,200 returns against 300 incremental ones).

None of this makes the campaign a failure. Three points of real lift may be well worth paying for. What this actually does is make the price of each return visible, and move the question from “did customers return?” to “what did each additional return cost, and was it worth it?” A business that reports only the twelve cannot ask it (and will most likely just repeat the campaign, believing it is on the right track).

A control group answers the question for one campaign.

The standard tool is a control group: a randomly selected share of the target audience is held back from the campaign, then compared with the group that received it. Simple enough, although three clarifications keep the claim honest.

First, the control group receives nothing from this campaign, not nothing from the business. It still gets the rest of the programme and every transactional message, so the 9% above is the return rate without this campaign, not the return rate without any CRM. That is a narrower claim than it can sound, and it is the right one for the question being asked.

Second, this is not a laboratory, in the sense that there is no absolute (or near absolute) sterility. A customer’s life events, a competitor’s promotion and a payday all affect the result, and none of it can be controlled. What randomisation does is balance those influences: it spreads them across both groups, so on average they reach treated and control customers alike and largely cancel out of the comparison. It cannot remove the noise they add, or the fact that a result obtained under one set of conditions may not hold under another. A test result is an estimate with a range, not a fact.

Third, a low lift cannot say why it is low. Timing, offer, frequency and audience all sit inside those three points, and a weak result is consistent with any of them being wrong. If contacting customers at day 30 instead of day 60 is the hypothesis, that is a separate cell with its own control, not an inference from the first test.

A control group is also only as honest as the rules it runs under, and rules written after the result is known are not rules. So, size the test before launch: with a small audience only a large effect shows through the noise, and a test that cannot see the effect it is looking for is not worth running. (An underpowered test that does cross the line tends to overstate what it found, because only the lucky readings get there.) Fix the sample and the horizon in advance and read the result once, because checking a live dashboard and stopping at the first significant reading inflates false positives and produces winners that fade on rollout. Stop early only for harm, compliance or operational problems, or when the pre-set sample is arithmetically out of reach. Record a test that could not detect an effect as inconclusive, not negative, because “did not work” buries ideas that were never properly tested. And leave a running test alone.

Local wins do not add up.

Let’s return to the pattern from the opening. Each of the ten campaigns was tested against its own control group, each was declared a win, and each report was accurate. The lifts sum to around eight points, and a standing holdout, which we will come to, puts the programme as a whole at around two.

No campaign misreported anything, and the sum still overshoots. Part of the gap comes from how results are reported and selected, and part from what a campaign-level control cannot see:

• Credit-based reporting counts one return several times. Where a lift is reported as returns within a window after a touch, instead of being read from a randomised control, a customer who returns after a win-back may also have received a loyalty offer and a product announcement in the same fortnight, and each campaign claims the return. Randomised tests net this out within each test; the inflation enters in the reporting that sits on top of them.
• Short windows flatter. A lift read at day 14 is not a lift at day 60. Some of it is borrowed from later in the same customer’s year, and some of it disappears once the incentive ends (the rented customers from the opening that we have also discussed in previous pieces).
• Winners are selected. The campaigns that clear the bar are reported and rolled out; the ones that do not are dropped. Estimates that cross the line are biased upward, because only the lucky readings of a noisy test reach it, so the survivors’ reported lifts run larger on average than their true effect.
• Gains decay, and conditions shift. Audiences change and novelty fades, so a lift measured in spring may not exist in autumn. A test is a snapshot.
• Overlap and shared load are invisible to a single test. A campaign test measures the campaign’s effect at the contact load the other campaigns already create. Lifts from overlapping campaigns are not additive either: two offers can substitute for each other, or reinforce each other. Only a group that receives none of the programme shows what the programme’s total volume does to the base. This is why the sum of the parts can land on either side of the whole, depending on design and context.

The mechanisms above mostly push the same way, which is why the sum of local lifts tends to run ahead of the truth. It is a claim about the programme, not a measurement of it. It is also routinely the figure that gets quoted.

A standing holdout shows what the programme adds.

The comparison that closes the gap operates one level up. A small, randomly selected group of customers receives none of the discretionary retention programme (no lifecycle journeys, no promotions, no win-backs), while everyone else receives it as designed. Retention, by the shared definition, is then compared between the two groups over months, not weeks.

Campaign-level controls are the default in most platforms, commonly set at 5% to 10% of a target group. A programme-level holdout usually has to be designed and kept deliberately, and in my experience it is less common than it should be. It answers three things nothing else can:

• Whether the programme as a whole adds anything, and how much, including the overlap and the contact load that local tests cannot see.
• Whether past gains still hold. A holdout runs in the present, so a lift that has decayed shows up as a gap that has narrowed.
• What the programme is worth, which is the most direct basis for deciding what it should cost.

Scope matters. The holdout withholds the discretionary layer only. Transactional and mandatory communication, such as billing, security and order or account notices, still reaches everyone, and should. So the holdout measures what the optional programme adds on top of the baseline every customer receives. It is not a measure of the value of contact as such, and anyone reporting it should say so.

The design borrows its rules from the local tests, applied over a longer window: random assignment, a size set by the baseline rate and the smallest effect worth detecting, a fixed duration, and a refresh schedule agreed in advance so the group stays comparable. One rule needs adapting. “Read once” suits a test with a short horizon; a holdout that runs for months needs review points on a calendar, with what each review may and may not change written down before the first result arrives. A holdout run without those rules is a larger version of the same mistakes.

The cost is proportional. A holdout of, say, 2% of the base forgoes about 2% of what the programme adds, net of the incentives and sending costs not spent on those customers, for as long as it runs; if the programme adds nothing, it costs nothing. (Whether 2% is enough is a power question, covered below.) The cost can fall as evidence builds: some platforms automatically shrink a campaign’s control when the lift is repeatedly confirmed, leaving less revenue on the table. At programme level, resizing should happen at the pre-set reviews, and the group should never reach zero, because the evidence is never finished (if the programme is ongoing). A lift confirmed in spring can decay by autumn, and a holdout too small to show a narrowing gap cannot tell you.

Two integrity points worth mentioning. Holdout customers who, for example, contact support stay in the analysis: excluding people for what they did after assignment biases the comparison, and the programme can itself cause support contact. Everyone is analysed as assigned. What does contaminate the holdout is a person delivering a retention offer, such as a save-desk agent discounting a holdout customer who threatens to cancel. Flag holdout membership where offers can be made, or log those cases and report them. A leak of this kind narrows the measured gap, so the result understates what the programme adds, and a business could be led to cut a programme that is working. It should be disclosed.

The two layers do different jobs. Local tests tell a team what to change. The holdout tells the business whether the changing pays. When the holdout gap is smaller than the sum of the local lifts, the discrepancy is a finding in itself, and the test log is where to look for the reason. Keep a dated record of what changed and when, so that a narrowing gap can be matched to the change that preceded it.

Someone has to own the comparison.

A holdout has a cost, of course. Customers in the group are deliberately not contacted, and some of them would have generated revenue from the programme they are not receiving. That is a trade-off, and the rule from the ownership article applies to it: it has to be decided, not discovered. The decision belongs to the owner of retention or to the executive team, depending on the stakes, made with the cost understood in advance. A campaign team that finds the revenue gap after the fact will, reasonably, close the holdout.

Ownership also matters for another reason. In organisations where no one owns retention, credit flows inward and blame flows outward: every team’s explanation is locally plausible, and nothing forces them to reconcile. An uncompared number is what allows that, because every team can report a gain against no baseline and so every team can claim it. A standing holdout is a number no team produces for itself. Someone has to hold it, protect it from being shut down when a quarter looks tight, fix the rules before the results arrive, and report on a schedule what it shows, including when it shows less than the dashboards claimed. That is the owner’s role, doing one specific job: giving the organisation a single comparison against which the rest of its reporting can be checked.

Whether that owner has the standing to hold the line when the answer is inconvenient is a separate (and quite pertinent) question.

Where honest measurement runs out.

None of this is free of limits, and a piece about measurement should state them.

• Holding customers out costs money. The cost is real, and a business may decide not to pay it. That is a legitimate decision, provided it is taken knowingly and the resulting numbers are treated as unverified.
• Small bases cannot detect small effects. If the base is small, the holdout needed to detect a realistic effect may be larger than the business can afford to withhold. As an illustration, detecting a two-point effect takes roughly 1,800 customers in the holdout at a 9% baseline and about 4,700 at a 60% baseline, so a 2% holdout only works on bases of roughly 90,000 to 235,000 customers. The honest position, when the numbers do not work, is to say the effect cannot be measured at this scale and to treat programme decisions as judgements, not findings.
• The holdout inherits the definition. It gives a precise answer to whatever retention means in your business. If the definition counts the wrong activity or the wrong window, the answer is precise and beside the point.
• The stack may not allow it. Campaign-level controls usually come with the platform. If it cannot hold out a random group reliably across all discretionary contact, or the data will not join into a single view of the customer, the binding constraint is the tool, not the discipline. Working out which constraint is actually binding is a diagnosis in its own right, and it comes before any decision to change the tooling.

The point

A retention rate tells a business where it is. It takes a comparison to tell it whether its own work is the reason. Campaign-level controls provide that for one campaign at a time, and they should be run properly. The programme-level holdout provides it for the work as a whole, and it is the one many businesses do not hold, because it costs something now, belongs to no team, and sometimes shows less than the dashboards claim.

That last property is what makes it worth having.

Frequently asked

What is a holdout group in retention? A small, randomly selected group of customers that receives none of the discretionary retention programme for an extended period, used as a baseline. Comparing its retention with everyone else’s shows what the programme adds beyond what would have happened anyway.

How is a holdout different from the control group in a campaign test? A control group belongs to one campaign and is compared over a short window, while still receiving the rest of the programme. A holdout sits outside the whole programme and is compared over months, so it captures what a single test cannot: the combined effect of overlapping campaigns, the programme’s total contact load, and decay over time.

How large should a holdout be? There is no universal size. It depends on the baseline retention rate, the smallest effect worth detecting, and how much risk of a wrong answer the business accepts. Work it out before starting; as an illustration, a two-point effect takes roughly 1,800 to 4,700 customers in the holdout depending on the baseline. If the required size is more than the business can afford to withhold, the effect cannot be measured at that scale.

Does a holdout mean some customers get worse service? No. Transactional and mandatory communication, such as billing, security and order notices, still goes to everyone. The holdout withholds only the optional retention programme, so it measures what that layer adds on top of the baseline.

Should holdout customers who contact support be excluded? No. Excluding people for what they did after assignment biases the comparison. Analyse everyone as assigned, and flag or log any case where a person delivers a retention offer to a holdout customer, since that is the real contamination.

Is a campaign-level control group enough? For judging one campaign, yes, if it is sized and run by rules fixed in advance. For judging the programme, no: the sum of campaign-level lifts tends to run ahead of what the programme adds, and only a programme-level holdout shows the whole.