How do you tell whether a change on a collection or sales floor really worked, or is just noise?

The short answer

Compare the change against a group that didn’t get it, in the same weeks. That is the only way to separate the change from everything else that moved at the same time.

Agree three things before you start: what you will split, what you will measure, and how long the test runs. Where you can, keep it blind: the people doing the work shouldn’t know which group an account, lead or person is in. Then, when the results come in, check that the gap between the two groups is bigger than the gap you see when nothing has changed at all.

Why before and after misleads

In our experience the usual way to judge a change is to compare the month after it with the month before, and that comparison is almost always mixed up with something else: new placements or new leads, the season, a leaver or a new starter, a shift in the economy. Any of them can move results more than the change did. Why collection results swing from month to month.

There is a second trap. In our view, changes are often made right after a bad month, and a bad month tends to be followed by a better one anyway. Michael Mauboussin’s book on skill and luck, The Success Equation (2012), describes this: where luck plays a part, unusually bad results tend to move back towards the average. Coach the collectors who had the worst month, and in our experience many of them will improve next month whatever the coaching did.

Split by what the change touches

The test is the same whatever the change: one group gets it, one doesn’t, both run in the same weeks. What you split depends on what the change touches.

• Changes to how work is routed or dialled. A new routing rule, call order, dialer setting, text or letter. Split the accounts or leads at random as they arrive, and let every collector or agent work both groups. How to prove a routing change raised collections and, for sales floors, how to prove a lead routing change raised sales set this out step by step.
• Changes to people. Coaching, a new script, a new training course, an incentive. Split the collectors or agents at random into two groups, and give the change to one. Both groups should work the same kinds of account or lead in the same weeks.
• Watch for leaks. People talk. In our experience, if half the floor gets a new script, the other half will often start using the best lines within days. Where that is likely, split by team or by site instead; in our view the gap will then tend to look smaller than it really is.

Keep people blind to the groups where you can

Where it is possible, in our view the strongest test is a blind one: the people doing the work don’t know which group an account, lead or person is in. That is how a BestPair test runs: collectors work accounts from both the match group and the control group without being told which is which.

In medical trials this is called blinding. Kenneth Schulz and David Grimes (The Lancet, 2002) describe it as keeping trial participants, the people giving the treatment or the people collecting outcome data unaware of who got what, so that they will not be influenced by it. They note that terms such as single blind and double blind mean different things to different people, and recommend saying plainly who was blinded, and how.

On a collection or sales floor, in our view, that means:

• Don’t mark the groups on the screen. In our experience a collector or agent who knows an account is in the test group may work it harder, or less hard, and the gap then measures the knowing as much as the change.
• Keep team leads out of it too, where you can. If supervisors know which accounts or people are in which group, in our view they may steer effort or coaching towards one side without meaning to.
• Don’t share group results with the floor until the end. In our experience a floor that watches the score week by week starts to play to it.
• Take the result from the system, not from people’s notes. In our view payments and sales recorded in the payment or order system leave little room for anyone to tilt the score, whoever knows the groups.

It isn’t always possible. Changes to people, such as coaching or a new script, can’t be hidden from the people who get them. Then blind what you can: agree the measure before the start, take the result from the system, keep the comparison group’s work unchanged, and read the later weeks as well as the first, when the novelty has worn off.

Know how much your results wobble on their own

Before you can call a gap real, you need to know how big a gap appears when nothing has changed. A simple way to find out: split your accounts or your people at random into two groups, change nothing, and watch how far apart the two groups drift week by week. Testers call this an A/A test, and Ron Kohavi, Diane Tang and Ya Xu give it a chapter of its own in their book on running controlled experiments (Cambridge University Press, 2020). The book is written for websites and apps, but in our view the idea carries straight over to a collection or sales floor.

Once you know your floor’s normal wobble, a result means something when the gap between the groups is clearly bigger than that wobble, and when it holds week after week rather than in one lucky week.

What spoils a test

• Stopping when it looks good. In our view a gap that appears early and disappears in the later weeks was never there. Agree the length before the start, and judge at the end.
• Changing two things at once. If a new script and a new dialer setting start the same week, you can’t tell which one moved the result.
• Moving accounts or people between groups. Keep every account, lead or person in the group they started in, whether or not the change was followed on a given call.
• Measuring the wrong thing. Promises that never pay, or sales that are cancelled, can make a change look better than it is. Score the outcome the business is paid on. How to check a vendor’s lift claim on your own data lists the same questions for anyone selling you a change.
• The first-weeks effect. In our experience people often work differently when something is new and someone is watching. Read the later weeks of a test as well as the first ones.

Where agent matching fits

Agent matching is one change among many, and it is tested the same way. BestPair sends each account to the eligible collector who has done best with accounts like it, from the floor’s own call and payment history, inside the rules and workload limits the agency sets, and measures the result against a control group. Any gain is the gap between the two groups in the same weeks, on the measure agreed before the start. Does holding back a control group cost collections?

Sources

• Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing: Ron Kohavi, Diane Tang and Ya Xu, Cambridge University Press, 2020; a guide to running controlled tests on websites and apps, with a chapter on the A/A test.
• The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing: Michael J. Mauboussin, Harvard Business Review Press, 2012.
• Blinding in randomised trials: hiding who got what: Kenneth F. Schulz and David A. Grimes, The Lancet 359, 2002.