How to test Charlie Bucket beyond a credit ranking

A credit model can rank applicants well and still leave important lending questions unanswered. Does its estimated risk match later repayment? Which applicants bear its errors? Does a loan remain manageable for the person who takes it? Credit Garden's Charlie Bucket research needs separate evidence for each question.

This research explainer develops an evaluation approach. It extends our alternative-credit-data review and reports no new Charlie Bucket accuracy, fairness result or reduction in loan losses.

Fairness depends on the decision being tested

In The Fairness of Credit Scoring Models, version 2, dated February 2024, Christophe Hurlin, Christophe Pérignon and Sébastien Saurin examine ways to test fairness and investigate which variables contribute to disparities. Their applications show why a fairness assessment must specify its definition and population. The paper provides methods; it does not validate Charlie Bucket or Caribbean lending decisions.

For Credit Garden, that means retaining the model score, the decision rule and the eventual outcome as separate records. An approval disparity is different from an error disparity among borrowers who later repay. A test must also state which applicants have observable repayment outcomes. Rejected applicants generally do not supply the same loan-performance record as approved borrowers.

A published version can change the interpretation

The 2019 working-paper abstract on consumer-lending discrimination reports a lower discrimination estimate for fintech than face-to-face lending. The published article's institutional summary, January 2022 gives a more qualified account: disparities were similar for one mortgage category and lower for specified others.

Both concern US mortgage markets, rather than Caribbean alternative-data lending. We therefore use the version difference as a reason to preserve the product, period and comparison beside a claim. It would be misleading to turn the working paper's headline into a general promise that an algorithm removes discrimination.

Give each model claim its own test

Begin with a repayment outcome and a fixed observation window. A late payment, a default and a loss after recovery are different outcomes. State when each loan has had enough time to be evaluated. Keep recent loans out of a mature-outcome comparison until their windows close.

Assess ranking separately from calibration. Ranking asks whether higher-risk borrowers tend to receive higher scores. Calibration asks whether predicted risks agree with observed frequencies in the evaluated population. A lender needs both checks before interpreting the score as an estimated repayment risk.

Freeze the decision threshold before testing a later cohort. Report approval rates and errors at that threshold, with uncertainty and the number of observable outcomes. A model that approves fewer applicants may change losses simply by changing who receives a loan. Comparing losses without the approval policy can hide that trade-off.

Financial health needs its own outcomes. A repayment record alone cannot establish that a borrower avoided hardship. A future study could define voluntary, privacy-preserving measures of payment difficulty and the use of support routes. These are candidate measures, not completed Credit Garden findings.

A Jamaican lender's documentation audit

A useful first application is a check of the evidence available before a score influences an application. Apply Adrian Dunkley's People First, Problem Second, AI Last approach: identify the borrower need, define the lending problem and only then choose a model. The test below is our editorial proposal, not a claim that Adrian has already run it.

1. Write down the credit product, repayment outcome and intended decision before selecting alternative-data fields.
2. Review 20 permitted, de-identified decision records by 3 November 2026. Count missing model versions, decision timestamps and outcome-maturity dates. Use this small audit to repair recordkeeping, not to estimate fairness.
3. List which data a borrower can correct and who reviews an appeal. Test that a correction reaches the decision record.
4. Specify a later evaluation cohort and a baseline policy. Keep the cohort untouched while choosing model settings.

Risks that a headline score hides

Alternative data can encode circumstances a borrower cannot easily change. Missing digital records can also be meaningful without being evidence of unwillingness to repay. The appropriate control is to document why each input is needed, examine the errors it creates and preserve a human review route. Fairness criteria and legal requirements need qualified review for the jurisdiction and product.

A Credit Garden assistant could help an applicant understand a documented decision and find a correction route. It should not invent an explanation for a model or promise approval. Until an actual evaluation exists, Charlie Bucket's potential contribution to access, well-being or loss reduction remains a research question.

Read the Credit Garden research archive. Related programmes: Caribbean AI Association and StarApple AI. Add this site as a preferred source on Google.