Despite its emphasis on credit-scoring/rating model validation, Basel II does not impose any standards on the process. This article presents a methodology for validating credit-scoring and P D models, focusing on retail and small business portfolios. A future article will address corporate portfolios.

Always a good idea, development of a systematic, enterprise-wide method to continuously validate credit-scoring/rating models nonetheless received a major shove from Basel II. As we've all come to know, the Accord requires qualifying banks to have robust systems for validating the accuracy and consistency of rating systems and processes. Further, banks need to be able to estimate the risk components, namely, probability of default (PD), loss given default (LGD), and exposure at default (EAD).

The validation process also is important for corporate governance purposes. It can help detect deterioration in a model's performance, which could affect existing risk-tolerance limits and economic capital allocation. The process can also assist in maintaining the loss/revenue objective associated with the implementation of a scoring model.

This article presents a methodology that can serve both purposes--validating credit-scoring models used for customer adjudication and validating the estimation of the risk components. Application and behavior scores may be used as input for pooling retail portfolios as well as for estimating the risk components.

According to a 2003 ISDA-RMA survey, the range of available data has caused banks to employ a range of validation techniques, resulting in key differences in the techniques used for corporate versus retail portfolios. This articles focuses on credit-scoring models for retail and small business (typically less than $200,000 credit) portfolios.

Methodology

Credit-scoring models are usually static in that they do not account for the time to delinquency or default and are built from a couple of point-in-time snapshots. There are various reasons that could cause actual performance of a scoring model to deviate from its expected performance--that is, performance at the time the scoring model was developed. For example, a scoring model might lose its predictive power during a recession if the characteristics entered into the model or the underlying customer population are sensitive to the economic cycle. In such cases, the distribution of the input characteristics could shift. Also, a scoring model may continue to rank-order the population and provide acceptable discriminant power, yet fail to produce desired performance because the scores (probabilities) from the model have lost their calibration with respect to the actual probabilities in the current population. If cutoff scores are used for adjudication, adjustments of those cutoff scores may be necessary.

Three diagnostic techniques for monitoring the performance of credit-scoring models can help us check for deterioration. The first technique can detect shifts in the score distributions of the development and current populations. The second technique can detect changes in the rank-ordering power of the model. Both techniques are presented with tests for assessing statistically significant changes. The third technique can be used to explain any possible misbehavior identified from the application of the first two techniques, by examining the characteristics input to the model as potential causes of that misbehavior. The application of the third technique is therefore conditional on the outcome from the first two techniques. In combination, these three techniques enable us to build an early warning system for detecting deterioration in credit-scoring models.

The first step is to define the data required for the validation process. Figure 1 shows an example of such data. For a given credit product, collect data for the through-the-door population of applicants over the past K months (for example, K=18). The data includes the model characteristics, adjudication outcome, and, as required, the credit performance of approved applicants. Accurate comparisons between the current and development populations require us to use the same data definitions as those used at the time of model development. Based on these definitions, filter out any applicants that were excluded at development, such as applicants who were manually adjudicated. Then divide the population into accepted/rejected, and divide the accepted applicants into "goods," "bads," and "indeterminates." The latter group could be, say, accepted applicants with fewer than six months' performance history or, in the case of revolving products, accepted applicants whose credit remains uncashed.

[FIGURE 1 OMITTED]

Think of the through-the-door population and its good/bad subpopulation as "current," to be compared

with their counterpart populations at the time of model development ("development" populations). At this point, there are three steps in the validation process.