How do I check a vendor’s reported lift against my own numbers?

What to ask for, in order

Every reported lift is a comparison. Before you weigh the number, find out what it was compared with and how.

1. A control group. Which accounts did not get the change, and how many. A result with nothing alongside it describes what happened and cannot show what the change added.
2. How the split was made. At random, account by account, before anyone dialed. Stephen Leonard’s 2006 piece in insideARM on champion and challenger testing describes the method collection floors have used for letters: two statistically random sets of accounts, one treatment on each, so that a difference in results can be put down to the treatment.
3. The same weeks. Both groups worked in the same period, by the same collectors, on the same dialer.
4. The measure and its denominator. Dollars collected per account contacted, per right-party contact or per account placed? Promises or payments? Gross, or net of reversals? Over what window after the call? RPC, promise-to-pay and kept rates shows how far the same name can drift.
5. The period. Which months, and whether any month or campaign was left out, and why.
6. Adherence. How often the change actually reached the call. A test where the new routing was followed on few calls measures less of it than it could.
7. Who calculated it, and from what. The vendor, your team or an outside party; from which files; and whether you can see them.

Red flags

• Before and after. Tax season, new placements, staff changes and a week of absences all move collections. A gain against last year could come from any of them.
• A starting point picked for being bad. Barnett, van der Pols and Dobson, writing in the International Journal of Epidemiology in 2005, explain regression to the mean: unusually high or low measurements tend to be followed by ones nearer the average, and the effect is more noticeable when follow-up is examined only on a group selected on its starting value. A pilot that starts on the weakest team or after the worst month can show a gain with nothing changed.
• Chosen months or campaigns. Ask for every month of the test, including the first.
• Subsets that were not random. "Where the recommendation was followed", "our best agents" or "accounts we worked" are groups somebody or something selected, so they differ from the rest before any change.
• A percentage on its own. Ask for both groups’ figures and the number of accounts behind each.
• Another agency’s result. It comes from a different book, mix of paper and floor. Experian’s Insights blog (Experian sells agency allocation software) made the like-for-like point in 2014 when writing about comparing collection agencies: compare like accounts placed at the same time.

How to rerun it on your own data

1. Get the assignment list. Account ID, group and the date each account was assigned. In our view this is the single most useful thing to ask for, because without it nothing can be rechecked.
2. Join it to your own payments. Use your collection system’s payment file, keyed by account ID, not a summary from the vendor. How payments are credited and how dialer and payment data join.
3. Count accounts contacted in each group from your call records, using your own right-party contact codes.
4. Work out dollars collected per account contacted for each group, net of reversals, over the window agreed at the start. Compare the two.
5. Check the groups were alike at the start. Balance, age of debt and placement date should look much the same in both if the split was random.
6. Check every assigned account is counted, whether or not the change was followed on its calls.
7. Read it month by month, and set the gap against how much your months move on their own.

The same questions apply to BestPair

Ask BestPair the same questions; how to prove a routing change worked, what a control group costs and how agent matching is priced set out its control group, measure and invoices. Once in any twelve-month period an agency may have the calculations reviewed by an independent accountant, at the agency’s cost unless the review finds an overcharge of more than 5%, in which case BestPair refunds it and pays the reasonable cost of the review.

Sources

• insideARM, Champion-Challenging Letter Strategies, Part 1 Stephen Leonard, 14 December 2006: champion and challenger testing on two random sets of accounts.
• Barnett, van der Pols and Dobson, Regression to the mean: what it is and how to deal with it International Journal of Epidemiology, 2005 (abstract).
• Experian Insights, Can You Compare the Performance of Your Collection Agencies? A 2014 post on a vendor’s blog about comparing agencies on like accounts placed at the same time.