Written by Can Soypak, Principal Consultant
Background
The European banking regulatory framework (Specifically the Capital Requirements Regulation - CRR), allows the institutions to use two different methods to calculate their credit risk related capital requirements: standardized approach or internal rating-based (IRB) approach.
The implementation of IRB approach requires that the institutions develop models, which produce their own estimates for certain parameters of the formula used to calculate the credit risk related capital requirements: the probability of default (PD), loss given default (LGD) and credit conversion factor (CCF).
If an institution has been granted the permission to calculate their risk-weighted exposure amounts using the IRB Approach, they also have to have robust systems in place to validate the accuracy of these internally estimated risk parameters, the data quality, model governance, etc. (Article 185 of CRR). Furthermore, internal validation results must be properly documented along with the validation methods (Article 187 of CRR). This internal validation carried out by the institutions is also the first point of reference for supervisors whilst performing regular model monitoring or when new IRB models are approved, existing models are confirmed etc.
As a result, each IRB institution performs regular internal validation of its internal models. These regular validations should be performed with at least annual frequency and after every significant model change. However, institutions deploy different validation metrics especially in the quantitative validation of internally estimated risk parameters (usually referred to as back-testing). Since different validation metrics are based on different assumptions (e.g. regarding distribution of risk parameters or correlation between defaults), the accuracy of internally estimated risk parameters is not easily comparable between different institutions.
To reduce this variation in the internal validation tests, the European Central Bank (ECB) has instructed all institutions under the Single Supervisory Mechanism (SSM) to publish supplementary validation reports using common validation metrics. To this end, ECB prepared a reporting template that all SSM institutions need to fill at least with annual frequency for all internally estimated risk parameters. Furthermore, ECB has also published a paper explaining in detail how this template should be filled and how these common validation metrics should be calculated (such as AUC, gAUC, Jeffrey’s test, etc.). These ECB instructions for validation are requested by supervisory authorities to supplement the regular internal validations of IRB models.
Institutions’ challenges and goals
In their effort to implement these supplementary reporting requirements, institutions have already started to incorporate the validation metrics defined in ECB instructions into their internal validation reports. It is common that institutions treat the ECB instructions as the minimum reporting standard and expand the internal validation reports by including additional test metrics on top of the commonly used validation metrics. This action is also undertaken by non-SSM institutions as some local regulators are already asking the non-SSM institutions to also comply at least with the reporting requirements from ECB instructions.
However, this process can be challenging; the institutions are not familiar with some of the test metrics suggested by ECB instructions. Specifically, the test metrics such as Jeffrey’s test (predictive ability test for PD) or standard deviation of gAUC (discriminatory power for LGD/CCF) are not common metrics used currently in internal validation process. Hence, institutions need to compute these test metrics in a reliable manner and incorporate them into their validation reports.
Furthermore, institutions are taking steps to automatize the internal annual validation process including the population of the ECB reporting template with the hopes of reducing the invested time for the internal annual validation process. This automation can also be expanded beyond the supplementary ECB reporting template and some institutions are already using this opportunity to also streamline the internal validation reporting process.
Finalyse validation toolkit
Explaining the process:
To assist the institutions with their new challenges in the internal validation for credit risk, a group of Finalyse consultants (experienced in IRB regulations, credit risk validation and programming) has developed the Finalyse validation toolkit. Finalyse validation toolkit includes the complete list of validation metrics that are required in ECB instructions for validation. Furthermore, it automatically reports the test results and qualitative portfolio information in the required format from the ECB reporting template and thus automates the supplementary validation process end-to-end in a user-friendly manner.
Additionally, Finalyse has purposely built a lean tool that can be easily expanded and adjusted with additional test metrics to suit institution’s preferences and internal validation standards. Furthermore, the toolkit is also offered in 3 different programming languages (SAS, R and Python) which allows it to be adapted without any extra effort and without any change in the client’s IT infrastructure.
Benefits of Finalyse validation toolkit
At Finalyse, we have made a deliberate effort for a tool to be as useful and user friendly as possible. Our plan was to develop a tool that would be so beneficial to the institutions, that they would seek to adopt it for their internal model validation, even if they are not strictly obliged to follow the ECB rules. Therefore, we sought to make sure that our tool:
- Can reliably produce all test metrics requested by ECB instructions using the programming language preferred by our client (R, Python or SAS)
- Can create model validation samples that are aligned with ECB requirements and that include built-in dictionaries
- Uses the accurate and up-to-date definition for credit risk parameters in the calculation of test metrics
- Automatically produces supplementary validation reports as required by ECB instructions for validation.
- Further expands on the ECB supplementary validation reports to create automatic internal validation reports by incorporating additional metrics not requested by ECB, but useful for the purpose of internal validation, into the tool
Conclusion
The Finalyse validation toolkit will help IRB institutions to streamline the periodical validation process for their credit risk parameters considering the new ECB instructions regarding the definition of validation samples and test metrics. As IRB institutions have already started expanding their internal validation standards according to these new ECB instructions, this toolkit will also be helpful in the internal validation process in general.
The toolkit also offers an error-free computation of a wide-range of quantitative and qualitative validation metrics and automatically produces validation reports in line with ECB instructions. This way, Finalyse validation toolkit will also allow banks to spare more resources on additional validation tasks such as data quality checks or to implement additional validation tests at more granular levels.