Same book, same agents, 25% more cash per account reached: a live collections test

Transcript

The impact actually came from the amounts that got paid.

My name is Kevin Daly, and today I’m going to be presenting a telco case study in the collection space. This is a North American case, and we’ve got some cool results that I want to share with you, and got some insights that hopefully you’re going to find interesting. And we’ve applied the BestPair model, and for those of you who don’t know, BestPair is an allocation algorithm using pairing logic.

So basically, we figure out what the agent is good at, and we try and give them more of that and less of the stuff that they’re not so great at. In this particular case, we had 170,000 accounts, and we split them evenly between a control group and a BestPair allocation group. And I want to run through with you how it went and what we learned.

So firstly, some key concepts that I want to describe is that your best agent isn’t necessarily your best agent in every single example. Sometimes your best agent overall, of course, you get some of that better than others. But at an account level, you might find that you have specialization with some of the agents. I think we all know if you ask the agents and you say to them, so who do you like to talk to? What’s your preferred type of customer? You don’t necessarily get the same answer from every single person. And we’re trying to play on that, but we’re using numbers to prove it. So we’re looking for what the agent is really good at and want to give them more of that and less of something else. So half is random control, and the other half has been treated with the BestPair product.

So it’s important to realize that we’re not choosing who’s going to get called. The dialer is doing what the dialer does. So once they segmented those two pieces, they reshuffled, loaded up into the dialer randomly. So it’s not like half the agents are getting control, and half are getting BestPair. No, every agent is getting a mixed bag of both and would randomly appear into each group. The agents aren’t aware that this is even happening in the background. So it’s not like they know and they’re treating one group differently. Only myself and the data guy actually knows which is which. So it couldn’t be that you take, for example, only the control group and say, we’re going to send SMSes to them. It wouldn’t be possible. No one knows which is in that particular case. So the team started out at 15 agents and then grew up to 20. That’s a little smaller than what we would normally like. Normally, the minimum that we would like is going to be 25 or more. So the way in which we’ve constructed this, it is a double-blind, as I’ve described. And what we’re going to do, what we have done, is that we’re looking to ultimately prove that BestPair works, not theoretically prove. The way in which we need to prove this is, does it collect more money? And that’s ultimately going to be the test. What happened on the control group? What happened on the match group? And if we’ve got a great theoretical product, but can’t implement it, that’s wasted time for everyone. In the results, we have trimmed off the top 1%,

the highest payments on the 1%. We didn’t want to distort the results from one group to another. So ultimately, what we’re measuring is phone call takes place as a right-party contact, so this is only a right-party contact, did money hit the till 30 days later. So what happened?

So there was a 25% increase in the amount of money that got paid in the BestPair group compared to the control group.

So the BestPair group per right -party contact collected $98.76, and the control group collected $79.17. That’s roughly 25%. The actual amount that hit the till was $23,000 more on this test,

and that was directly as a result of the BestPair allocation.

So if we look at some of the other numbers, it’s super interesting when you dive down into some of the detail, dive down into some of the detail here. The amount of accounts that paid on either side wasn’t a huge difference.

So in other words, the insight that we’re getting here is that we actually made a difference predominantly on how much got paid, not necessarily on the probability of getting paid. Yeah, there was an increase in the probability, but the impact actually came from the amount that got paid, which is pretty interesting.

If you start to drill down into some of the detail, you actually start to see something else emerging. On these smaller accounts, there’s actually very little difference between the BestPair group and the control group. That’s in this use case. That doesn’t mean that that’ll be the case for other tests that we do, but in this case, it was the larger accounts where our impact was greater. This particular book has got quite a fat tail. In other words, a lot of the debt is sitting in large outstanding amounts, although the vast majority of them are small amounts,

which is good news for this type of approach. So if you look at the control group on bigger accounts, bigger accounts in this case with debts over $1,000, one in five said yes and made a payment.

Under the BestPair group, for those accounts bigger than $1,000, one in three said yes. So the impact is definitely around those type of debts that are older and bigger, in other words, where there’s a need for some trust, a need for some conversation between the agent, a need for persuasion, frankly, between the customer and the agent. What became super clear is that the amount of times that the right agent or the agent that we assigned actually had the conversation with the client was only 50% of the time. We call that adherence rate, and that is how often did the dialer adhere. Now, some of that is purely structural. If that cohort of agents was not available when the customer answered, this was outbound, in an outbound scenario, then it has to go to someone, and therefore, it can’t adhere. We were expecting that, but we were only expecting about 20% did not adhere. In this case, 50% did not adhere. So there were some structural issues, a little frustrating, but there were some structural issues in the way the dialer script was executing, subsequently being fixed. So that’s the conclusion. The conclusion is that this worked super well, all super excited about what this could look like. This is one telco book on one agency. It’s not a forecast for your floor. Your own data will tell us what this should produce for you. If this is something that you want to find out a little more about, if you haven’t done so already, go to bestpair.ai, super keen to have a chat to you to see whether this is something that could work in your environment.

Book a call with me, and let’s see if that’s something that would be useful.