Is it the collector or the accounts? How do you tell skill from luck on a collection floor?

The short answer

Usually it is both. A collector’s results are their skill, plus the accounts they were given, plus chance. Over a few weeks on a small number of accounts, the accounts and chance can swamp the skill: one collector draws a run of fresh, large balances and tops the board; another draws aged accounts that have already been through two agencies and sits at the bottom, whatever their effort. The quickest test is to look at everyone on the same kind of account: if nobody does well on it, it is the accounts; if one collector stands out where the rest struggle, it is probably the collector. Beyond that, compare collectors on similar accounts, over longer periods, against what accounts like theirs usually return. And be careful what you rank people on, because a board that ranks the paper and the luck can do more harm than good.

Why results swing

• The accounts. Balance, age of the debt and how many agencies have already worked it all change what an account is likely to pay. In our view a collector’s collections are the collector plus the accounts they were given. What a fair collector scorecard measures.
• Small numbers. In our experience a collector may reach only a small number of consumers on any one kind of account in a month. A handful of payments either way moves their numbers a lot.
• Chance, and the swing back. Where luck plays a large part, unusually good and unusually bad results tend to drift back towards the average. Michael Mauboussin’s book on skill and luck, The Success Equation (2012), covers this: in most fields skill and luck are tangled together, few people are good at telling them apart, and results tend to revert towards the mean. A collector praised for a great month, or warned after a bad one, will often move back towards their usual level whatever the manager does.

How to tell skill from the accounts

1. Start with the obvious test. Look at how everyone does on the same kind of account. If nobody is doing well on it, it is almost certainly the accounts. If one collector stands out on accounts the rest of the floor struggles with, it is probably the collector.
2. Compare like with like. Look at each collector’s results on similar accounts, by balance band, age and placement, not just their overall total.
3. Look over months, not weeks. Read the trend, and pool similar kinds of account where there are too few of one kind to judge.
4. Compare with the floor on the same paper. A collector who is ahead of the floor’s results on accounts like theirs is doing well, even if their raw total is low.
5. Look at conversion, not just dollars. Right-party contacts that turn into payments or kept promises tell you more about the collector than raw collections do. What RPC conversion is.
6. Use a random split for big decisions. In our view, before promoting, letting go or changing how accounts are shared, compare results on accounts that were split at random, so neither side had better paper. How to prove a change worked.

In our view, done this way, the floor usually finds that most collectors are strong on some kinds of account and weaker on others, rather than simply good or bad. Why collectors convert at different rates.

What leaderboards do to a floor

Many floors put a ranking on the wall to spur people on. Research on rankings, at work and in online tasks, suggests it often doesn’t work that way:

• Rankings can lower effort. Wharton’s Iwan Barankay ran an online field experiment with workers doing paid image tasks on Amazon Mechanical Turk, some of whom were told how they ranked against their peers. As Knowledge at Wharton reported (2010), managers tend to assume comparison makes people work harder, but "in fact, the opposite happens".
• Public rankings can be worse than private ones. In a randomised experiment at a Bangladeshi sweater factory, Anik Ashraf found that privately ranked workers worked harder after good news and less hard after bad news, and that public ranking led to lower net effort than private ranking, because of a strong preference not to outperform friends (2018).
• Feedback on what people control can help. Rainer Rilke’s field experiment at a software company, described in a seminar abstract, started from the idea that employees struggle to link relative feedback to their own effort. Relative feedback based on inputs, measures closely tied to effort, raised the performance of low performers more than high performers (2021).

None of these studies was on a collection floor. In our view the lesson carries over: a board that ranks collectors on raw collections ranks the paper and the luck as much as the people, and the collectors at the bottom, often the newest, learn that effort doesn’t count.

Keeping collectors going when results feel like luck

• Show each collector against accounts like theirs. And against their own past results, so improvement is visible even on hard paper.
• Coach on the kinds of account they struggle with. Not general advice. Most collectors are behind on some kinds of account and ahead on others.
• Share accounts fairly. No feedback system can make up for a split that hands some collectors the best paper. How to share accounts fairly across collectors.
• Celebrate progress, not just totals. A first kept arrangement, a better week on aged accounts, a consumer brought back into contact. People keep going when they can see their work come to something. Why collectors quit, and how to reduce turnover.

Where agent matching fits

Agent matching changes which accounts each collector gets, which is the part of a collector’s results that is about the paper rather than the person. BestPair sends each account to the eligible collector who has done best with accounts like it, from the floor’s own call and payment history, inside the rules and workload limits the agency sets, and measures the result against a control group. In our view that takes some of the luck of the paper out of the picture; we haven’t measured its effect on morale. What predictive pairing is.

Sources

• The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing: Michael J. Mauboussin, Harvard Business Review Press, 2012.
• Ranking Employees: Why Comparing Workers to Their Peers Can Often Backfire: Knowledge at Wharton, 18 August 2010, on Iwan Barankay’s field experiment on rankings.
• Do Performance Ranks Increase Productivity? Evidence from a Field Experiment: Anik Ashraf, 2018, a randomised experiment at a Bangladeshi sweater factory; UK government research output.
• Input-based relative performance feedback: evidence from a field experiment: Rainer Michael Rilke, WHU, seminar abstract, 13 July 2021.