RatioLogo
Back

The Calibration Gap in Lending AI

What if the math used to approve your next loan is perfect at ranking who is "risky," but fundamentally wrong about the actual odds of them paying it back? In the high-stakes world of consumer lending, this isn't a hypothetical question.

Banks have long relied on a metric called discrimination—the ability to tell a "good" borrower from a "bad" one. But a landmark study reveals a critical flaw in the most advanced models.

A Landmark Study on Model Flaws

A study of 18 independent datasets from banks and online lenders found that AI models are often uncalibrated. They might know you are riskier than your neighbor, but they can't accurately pin down if your chance of default is 5% or 15%.

For the average person, this "calibration gap" can mean the difference between a fair interest rate and a rejected application.

The Research Methodology

Researchers Pedro G. Fonseca and Hugo D. Lopes tackled this problem by auditing 162 experimental runs. They tested how Machine Learning (ML) models like Random Forests and Gradient Boosting handle real-world, "noisy" loan data.

Key Experimental Setup

The team employed a structured, chronological data split to mirror real-world lending scenarios:

  • 60% Training: Used to initially train the ML models.
  • 20% Calibration: A dedicated set used specifically to fine-tune the model's probability outputs.
  • 20% Test: Used for the final, unbiased evaluation of performance.

The primary metric for success was the Brier Score Loss, a rigorous measure that treats the accuracy of a probability as a sacred value. A lower Brier Score means the predicted probabilities are more reliable.

The Path to Better AI

The results provide a new roadmap for financial AI, shifting focus from traditional methods to advanced calibration techniques.

The Industry Standard vs. The New Guard

  • Logistic Regression: The industry's traditional "old guard" method.
  • Non-Parametric Models: The study found that when these complex models (like Gradient Boosting) are properly tuned, they are far superior.

The Gold Standard for Recalibration

Isotonic Regression emerged as the most effective technique. Its impact was significant:

  • It improved Logistic Regression results in over 75% of cases.
  • It allowed complex ML models to consistently outperform traditional methods.

The Challenge of "Calibration Paradoxes"

The path to better math is not straightforward. The team discovered several critical pitfalls that challenge easy implementation.

Key Paradoxes Uncovered

  1. The Performance Illusion: Sophisticated models often look perfectly calibrated on paper (training data) but fall apart when faced with new, "Recent" test data.
  2. The Volatility of Common Techniques: A widely used technique called Platt (Sigmoid) Scaling was found to be surprisingly volatile. It actually decreased performance in approximately 25% of cases for some models, making it a risky choice.

Conclusion & Key Warning

The study concludes with a breakthrough insight and a crucial warning against over-automation.

"Results show that when the dataset is treated as a time series, the use of re-calibration with Isotonic Regression is able to improve the long term calibration better than the alternative methods," the authors noted.

Critical Limitations & The Human Element

Despite its success, Isotonic Regression is not a magic bullet. The researchers noted important caveats:

  • It requires significant data volume to avoid overfitting.
  • Their "naive" split-set approach may not capture all the nuances that more advanced cross-validation could reveal.

This underscores that model development still requires expert oversight and cannot be fully automated.

Final Takeaway: In the future of lending, the most important question isn’t just "Who is riskier?"—it’s "Exactly how much risk do they carry?" Accurate probability calibration is the key to fairer, more reliable financial decisions.

Based on: Calibration of Machine Learning Classifiers for Probability of Default Modelling by Pedro G. Fonseca and Hugo D. Lopes (James Finance/CrowdProcess Inc., October 24th, 2017).