Comparative Analytics of Machine Learning and Traditional Models in Mortgage Credit Risk: Evidence from Freddie Mac Data

Abstract

This paper compares the performance of traditional models and machine learning (ML) models for mortgage credit risk prediction using the Freddie Mac Single-Family Loan-Level Dataset. A sample of 100,000 loans from 2022 and 2023 was used to train and test logistic regression models (with LASSO, Ridge, and Elastic Net penalties), a generalized additive model (GAM), and three boosting ML models (XGBoost, CatBoost, and LightGBM). All models used the same training and test data splits, ensuring a fair comparison. The models were evaluated using ROC-AUC, Precision-Recall metrics, Brier scores, F1 scores, and calibration measures. Among the logistic regression models tested, engineered logistic regression had the highest ROC-AUC, 0.7956. The GAM had the highest F1-score of 0.1542, with the lowest ECE (0.000703) and Brier score (0.02283) among all models tested. The CatBoost model had the highest ROC-AUC among all models tested at 0.7988, but also the worst calibration (ECE = 0.3386, Brier = 0.1835). Furthermore, the traditional logistic regression models outperform the ML boosting models in Brier score, with values of 0.0228 - 0.0229 compared to 0.1797 - 0.1853 for the boosting models. Finally, the PSI interest rate of 3.121 for the period from 2022 to 2023 shows the need to recalibrate these models over time. Overall, the results suggest that there may be little benefit to using ML models for credit risk prediction tasks. However, CatBoost may be useful for creating early warning systems for mortgage loan defaults. At the same time, GAMs may be useful for estimating losses on mortgage loans in default and reporting that information to regulatory authorities.