What is it about?
In big-data credit scoring, the most complex model is not necessarily the best—performance, interpretability, computational efficiency, and classification risk must be considered together. Credit scoring increasingly uses large and diverse datasets from open banking and digital financial platforms. These data can improve personal credit evaluation, but they also create challenges related to data quality, computational efficiency, and model risk. This study demonstrates how PySpark can be used to implement and compare several statistical and machine-learning models for credit-risk assessment at scale. The models include logistic regression, decision trees, random forests, neural networks, and support vector machines. The results show that more complex machine-learning methods do not automatically produce better credit-scoring decisions. In this application, logistic regression provided a favourable balance of predictive performance, lower false-negative rates, computational efficiency, and interpretability.
Featured Image
Photo by Markus Winkler on Unsplash
Why is it important?
Financial institutions increasingly have access to large volumes of customer and transaction data, but using more data and more sophisticated algorithms does not necessarily lead to better lending decisions. This study shows the importance of evaluating credit-scoring models not only by predictive accuracy, but also by computational cost, false-negative risk, and interpretability. A simpler model may sometimes provide a better overall solution than a more complex machine-learning algorithm. The findings are relevant for banks, fintech companies, and risk managers seeking scalable and understandable credit-risk models for big-data environments.
Perspectives
The growth of big data creates new opportunities for credit-risk analysis, but it also increases the temptation to assume that more complex algorithms are always superior. Our results suggest that model selection should instead consider the complete decision-making context. In credit scoring, missing a genuinely risky borrower can be especially costly, while models must also remain computationally feasible and sufficiently understandable for practical risk management. PySpark provides a useful environment for scaling statistical and machine-learning methods to larger datasets. However, efficient computation should be combined with careful attention to data quality, model risk, interpretability, and the consequences of classification errors.
Prof. Afshin Ashofteh
Universidade Nova de Lisboa
Read the Original
This page is a summary of: Big Data for Credit Risk Analysis: Efficient Machine Learning Models Using PySpark, January 2023, Springer Science + Business Media,
DOI: 10.1007/978-3-031-40055-1_14.
You can read the full text:
Resources
Contributors
The following have contributed to this page
