Better credit data: what the evidence can tell Charlie Bucket

A thin credit file can leave a lender without useful information about repayment, increasing the appeal of other data sources. Credit Garden’s review series begins with that tension: when does additional information improve a decision, and when does it simply introduce a new proxy for advantage?

One influential study gives a concrete starting point. Berg, Burg, Gombović and Puri studied roughly 250,000 purchases at a German online retailer. Customers received goods before paying, which made the setting useful for studying default prediction. The July 2019 manuscript compares a bureau score with a model using digital traces such as device and browsing information.

What the comparison actually measured

The authors’ manuscript, page 4, reports an area under the receiver operating characteristic curve, or AUC, of 68.3% for the bureau score, 69.6% for digital footprints and 73.6% for the combined model. AUC describes how well a score ranks cases with different outcomes. A higher number here does not mean that the model gives every customer an accurate probability of default.

The combined score exceeded the bureau-only score by 5.3 percentage points in that comparison. That difference describes risk ranking. It does not measure a reduction in loan losses. The point estimates shown in the chart do not include confidence intervals, because intervals for this plotted comparison have not been extracted for this article.

These results are evidence from the named retailer and version of the research. They are not a Charlie Bucket benchmark. The later journal publication is On the Rise of FinTechs: Credit Scoring Using Digital Footprints in the Review of Financial Studies, DOI 10.1093/rfs/hhz099. We retain the manuscript date beside the chart so the reader can trace the displayed numbers to the inspected source.

Three questions for a Caribbean credit model

The first is whether the model can rank risk. A score should help distinguish applicants more likely to repay from those less likely to repay under the outcome definition being used. The comparison needs information available at the application date, with the evaluation rules fixed before testing. Later collections activity or repayment records would give a model information a real decision-maker did not have.

The second is whether its probabilities are calibrated. If a model assigns a group a particular risk level, the observed rate over the defined period should be consistent with that prediction. A system can rank borrowers reasonably well and still understate risk across the portfolio.

The third is whether the decision improves outcomes. A lender might reduce defaults by refusing more people. That could lower losses while worsening access for the customers the programme is intended to serve. Approval rates, pricing, repayment stress and appeals therefore belong beside predictive performance in an evaluation.

A BIS study of two US fintech small-business lenders found that their internal scores predicted later performance more accurately than the traditional scores examined. The authors also studied where lending occurred. This broader design matters: prediction and access are related questions, but they require different evidence. It still does not establish what a model will do in Jamaica or another Caribbean market.

Why a useful signal can still be a poor choice

The digital-footprint authors explicitly discuss privacy and the possibility that behavioural traces proxy for sensitive characteristics. The fact that a variable is accessible does not establish that a lender should use it. A device can carry information about household resources, while a browsing pattern may change when a website or advertising strategy changes.

For Charlie Bucket, we are examining what candidate information adds under local conditions and whether its use is appropriate. We need to compare it with a simpler scorecard, test performance over time and examine who gains or loses access. A borrower should be able to understand the relevant reason for a decision and correct an inaccurate record.

That applies to an agent assistant too. An agent can help a reviewer find the policy or source data supporting an explanation. It should not turn a statistical association into an accusation about a person’s character, or fill a missing income field with a plausible-sounding estimate.

Financial well-being is a separate outcome

Credit Garden’s financial-health programme asks what happens after a decision. Does the repayment schedule fit the borrower’s cash flow? Can the borrower absorb an unexpected expense? Is repeated borrowing improving their position or keeping them under pressure?

The CFPB questionnaire provides an example of measuring people’s experience of their finances. It has ten questions and uses US comparison data. Its existence does not validate a Caribbean version. Local testing needs to establish whether the questions and score mean what we intend for the people using them.

How we will build the next review

This article is a focused evidence review, not a new meta-analysis. For the planned synthesis, we will specify the population, lending product, outcome horizon and comparison before extracting results. Studies of short-term retail purchases, unsecured consumer loans and business lending should not be averaged into one universal claim about AI credit scoring.

Where studies report comparable differences in AUC on the same test population, a synthesis needs the uncertainty of each difference and the dependence between competing models. Where those details are unavailable, we will report a structured comparison and the missing information. We will also identify repeated datasets so that several papers about the same borrowers do not count as several independent populations.

1. Use a dated application snapshot and a fixed repayment outcome when defining a local comparison.
2. Compare the candidate model with an interpretable scorecard under the same data and tuning conditions.
3. Report access, losses, explanation errors and borrower outcomes alongside discrimination and calibration. Keep early-stage process tests separate from claims about mature loan performance.

The result we are working towards is a clear account of where additional data helps, which customers it helps and what the improvement costs. That is the standard Charlie Bucket will need to meet before a research finding becomes a claim about Credit Garden’s service.