Supervisory handbook on the validation of IRB rating systems

0
EBA Regular Use
SUPERVISORY HANDBOOK
ON THE VALIDATION OF RATING SYSTEMS UNDER
THE INTERNAL RATINGS BASED APPROACH
EBA/REP/2023/29

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 1 Content Executive summary 3 List of figures and boxes 4 Abbreviations 5

Introduction: overview of the validation handbook 7
1.1 Legal status of the supervisory handbook 7
1.2 Specificities of the validation in the regulatory framework 8
1.3 Structure of the supervisory handbook 11
General principles for the validation framework 15
2.1 Scope and objectives of the validation 15
2.2 The validation function as a second layer of defence, between CRCU and Internal Audit 17
2.3 Validation policy and validation report 20
2.4 Validation tasks 23
Validation content 27
3.1 Assessment of the core model performance 27
3.1.1 Risk differentiation 29
3.1.2 Risk quantification 39
3.1.3 Other specific points 50
Specificities related to the validation of defaulted exposures’ risk parameters 50 Specificities related to the validation of credit risk mitigation 52 Specificities related to the validation of the slotting approach 56
3.2 Assessment of the modelling environment 60
3.2.1 Data quality and maintenance 60
3.2.2 IT implementation of the rating systems 63
First validation activities 66
4.1 General requirements 66
4.2 Specificities of the first validation regarding the core model performance 68
4.2.1 Risk differentiation 68
4.2.2 Risk quantification 71
4.2.3 Other specific points 71
Specificities related to the validation of defaulted exposures’ risk parameters 72 Specificities related to the validation of credit risk mitigation 72 Specificities related to the validation of the slotting approach 72
4.3 Specificities of the first validation regarding the modelling environment 72
4.3.1 Data quality and maintenance 72

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 2
4.3.2 IT implementation of the rating systems 72
5. On-going validation activities 74
5.1 Scope of application 74
5.2 Minimum regular validation tasks regarding the core model performance 81
5.2.1 Risk differentiation 81
5.2.2 Risk quantification 83
5.2.3 Other specific points 84
Specificities related to the validation of defaulted exposures’ risk parameters 84 Specificities related to the validation of the slotting approach 84
5.3 Minimum regular validation tasks regarding the modelling environment 85
5.3.1 Data quality and maintenance 85
5.3.2 IT implementation of the rating systems 85
6. Focus on specific validation challenges 86
6.1 Focus 1: validation in the context of the use of external data 86
6.2 Focus 2: validation in the context of outsourcing of validation tasks 88
6.3 Focus 3: validation in the context of data scarcity 92

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 3 Executive summary The task of the EBA to develop and maintain a supervisory handbook derives from Article 8(1)(aa) of the EBA Regulation 1 which stipulates that the EBA shall ‘develop and maintain an up-to-date Union supervisory handbook on the supervision of financial institutions in the Union which is to set out supervisory best practices and high-quality methodologies and processes and takes into account, inter alia, changing business practices and business models and the size of financial institutions and of markets’. In the context of the Internal Ratings Based Approach (IRB Approach), the EBA has already clarified a number of requirements, aiming at reducing the risk-weighted exposure amounts (RWEA) unjustified variability stemming from different supervisory and bankspecific practices. In this context, this handbook complements the previous regulatory products published under the roadmap to repair IRB models as regards supervisory practices. 2 In particular, the validation of the IRB rating systems is an essential step to ensure a robust measurement of credit risk within the IRB Approach, such that it allows for the highest risk sensitivity, but also ensures comparability across institutions. It is foreseen as an activity to be performed by an independent function (the ‘validation function’), which is expected to challenge the main methodological choices done by the credit risk control unit (CRCU) and assess regularly and empirically the performance of the rating system. As such, a general description of the activities and objective of the validation function is provided in Article 185 of the Regulation (EU) No 575/2013 (CRR), as well as in Chapter 3 of the Commission Delegated Regulation (CDR) (EU) 2022/439 (CDR on assessment methodology). Nevertheless, the EBA has identified some heterogeneity in the expectations of competent authorities (CA) relative to the validation function. While the validation methods, procedures and concrete analyses are expected to be tailored to the specificities of the rating systems, the objective and areas on which the validation function is expected to form an opinion on should be harmonised. Consequently, the handbook provides additional clarity on best supervisory practices which the CAs are expected to give consideration to when performing their supervisory activities and developing their own expectations on the validation of IRB rating systems. It fully leverages on guidance from the IRB repair program. It clarifies the specificities of the validation in the context of the prudential framework, provides an overview of the validation framework as well as a description of the areas whereby the validation function is expected to form an opinion on, without prescribing any specific methodology to get this opinion. As such, the handbook touches upon both the pure model performance assessment, in terms of risk differentiation and risk
quantification, and the modelling environment. It highlights some key differences between the first validation activities and the ones performed on an on-going basis, and discusses further the validation challenges related to the use of external data in the model development, the outsourcing of validation task and the validation in the context of data scarcity. 1 Regulation (EU) No 1093/2010 establishing a European Supervisory Authority (European Banking Authority) 2 https://www.eba.europa.eu/eba-publishes-report-on-progress-made-on-its-roadmap-to-repair-irb-models

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 4 List of figures and boxes FIGURES
Figure 1: Structure of the validation handbook 11
Figure 2 : Schematic view of the development sample versus Validation samples 37
Figure 3: Schematic view of best practice of uses of validation samples between the CRCU and the
validation function 70
CONTEXT BOXES
Context box 1: assessment of the process-related aspects of rating system changes by IA 26 Context box 2: accuracy in the development sample and conservatism in the application portfolio of the rating assignment 29 Context box 3: the different forms of human judgment 34 Context box 4: specific cases with conservative requirements in the CRR 44 Context box 5: EBA Supervisory Benchmarking 49 Context box 6: the data quality framework and its application in the IRB framework 60 Context box 7: the review of estimates 74 FOCUS BOXES Focus box 1: validation cycle in relation to the interaction with the CA 16 Focus box 2: validation sample and validation data set 21 Focus box 3: quantitative thresholds and comparative analyses 22 Focus box 4: accuracy of the rating assignment in the context of model (re-)development 35 Focus box 5: accuracy of the rating assignment in the context of model (re-) development 40 Focus box 6: specific challenger analyses for the modelling choices under the slotting approach 58 Focus box 7: IT implementation and assignment of parameter estimates 64 Focus box 8: samples used during the validation of new or changed aspects of a rating system 66 Focus box 9: specific additional analyses expected during the full validation 79 INTERACTION BOXES Interaction box 1: structural independence of the validation function vis-a-vis the CRCU 17 Interaction box 2: assessment of the rating system 24 Interaction box 3: assessment of the definition of default 28 Interaction box 4: assessment of the human judgment in the assignment process 31 Interaction box 5: statistical assessment of the model between CRCU and validation function 36 Interaction box 6: assessment of the CRM 52 Interaction box 7: other validation function in FCP 55 Interaction box 8: assessment of the data quality between the different functions 61 Interaction box 9: assessment of the IT implementation of the rating system 64 Interaction box 10: interaction between the CRCU and the validation function during the first validation activities 67 Interaction box 11: statistical assessment of the model between CRCU and validation function during the first validation 70 Interaction box 12: interaction between the CRCU and the validation function during the regular validation 76

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 5 Abbreviations CA Competent authority CDR Commission Delegated Regulation CF Credit conversion factor CRCU Credit risk control unit CRD Capital Requirements Directive (Directive 2013/36/EU) CRM Credit risk mitigation CRR Capital Requirements Regulation (Regulation (EU) 2013/575) DR Default rate EBA European Banking Authority EL Expected loss EU European Union FCP Funded credit protection GL Guidelines GL on CRM Guidelines on Credit Risk Mitigation for institutions applying the IRB Approach with own estimates of LGDs (EBA/GL/2020/05) GL on downturn LGD estimation Guidelines for the estimation of LGD appropriate for an economic downturn (‘Downturn LGD estimation’) (EBA/GL/2019/03) GL on Internal governance Guidelines on internal governance under Directive 2013/36/EU (EBA/GL/2021/05) GL on outsourcing Guidelines on outsourcing arrangements (EBA/GL/2019/02) GL on PD and LGD estimation Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16) IRB Internal ratings based IA Internal Audit LGD Loss given default MoC Margin of conservatism OOS Out-of-sample OOT Out-of-time PD Probability of default RDS Reference data set RTS Regulatory Technical Standards CDR on assessment methodology Commission Delegated Regulation (EU) 2022/439

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 6 CDR on model changes Commission Delegated Regulation (EU) No 529/2014 CDR on slotting approach Commission Delegated Regulation (EU) 2021/598 RW Risk weight RWEA Risk-weighted exposure amounts Slotting approach Supervisory Slotting Criteria Approach UFCP Unfunded credit protection

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 7
1. Introduction: overview of the
validation handbook
1.1 Legal status of the supervisory handbook
1. Background. In the context of the ‘Internal Ratings Based’ Approach, the EBA has previously
published under the roadmap to repair IRB models a number of requirements, aiming at reducing the RWEA unjustified variability stemming from different supervisory and bankspecific practices. The validation of IRB rating systems is an essential step to ensure a robust measurement of credit risk within the IRB Approach and, consequently, competent authorities usually have a (published or unpublished) set of expectations for the internal validation of IRB rating systems as part of their supervisory practice. This supervisory set of expectations is in particular reflected in the supervisory approval process and the on-going supervision of rating systems used under the IRB Approach. In the past, the EBA has identified heterogeneous supervisory practices in this regard. With the validation handbook, the EBA aims to achieve a harmonised supervisory understanding and harmonised supervisory practices by providing an outline of best supervisory practices on expectations than can be put on the institution’s implementation of the requirements for validation of IRB rating systems.
2. Mandate. The task of the EBA to develop and maintain a supervisory handbook derives from
Article 8(1)(aa) of the EBA Regulation 3 which stipulates that the EBA shall ‘develop and maintain
an up-to-date Union supervisory handbook on the supervision of financial institutions in the Union which is to set out supervisory best practices and high-quality methodologies and processes and takes into account, inter alia, changing business practices and business models and the size of financial institutions and of markets’. In addition, Article 29(2), second subparagraph, of the EBA regulation specifies that ’For the purpose of establishing a common supervisory culture, the Authority shall develop and maintain an up-to-date Union supervisory handbook on the supervision of financial institutions in the Union, which duly takes into account the nature, scale and complexity of risks, business practices, business models and the size of financial institutions and of markets.’ Therefore, the supervisory handbook should cover all matters which are within EBA's remit with the aim to set out best supervisory practices rather than provide further specifications for the application of the legislation.
3. Content of the handbook. The validation handbook details the best supervisory practices,
which the CAs are expected to give consideration to when performing their supervisory activities and developing their own expectations on the validation of IRB rating systems. More precisely, with the validation handbook, the EBA aims to promote convergence of CA’s approaches and practices by providing good and best practices observed both within institutions, i.e. in terms of validation framework implemented, as well as within supervisors, 3 Regulation (EU) No 1093/2010 establishing a European Supervisory Authority (European Banking Authority)

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 8 i.e. in terms of best supervisory practices and ‘expectations’ derived from existing regulatory requirements. These expectations should be understood as a best supervisory practice of the expectation to a sound IRB validation.
4. The supervisory handbook vis-à-vis EBA GL. Both EBA GL and the supervisory handbook are of
non-binding nature, of general application and acts of Union law whose validity can be determined only by the Union courts, in a preliminary ruling. 4 However, unlike guidelines, the supervisory handbook is not addressed directly to financial institutions, but to CAs, and neither limit their judgment-led supervision, nor the supervisory assessment of the individual cases. As no ‘comply or explain’ mechanism is applicable to the handbook, any departure from it can be justified merely on the needs of judgment-led supervision. In terms of harmonisation effects, the handbook will predominantly be used as a benchmark of convergence during peer and other reviews.
1.2 Specificities of the validation in the regulatory framework
5. General definitions. The general definition of the term ‘model validation’ is known in various
fields such as computer science, engineering and finance; as different as these disciplines are, model validation always refers to one of the key assessments undertaken to verify that a model is working as expected. In general, ‘model risk’ can be described as the potential for adverse consequences of decisions based on incorrect or misused model results and reports. Against this background, point 11 of Article 3(1) of the Capital Requirements Directive (Directive 2013/36/EU - CRD) defines this risk as the risk of a potential loss ‘an institution may incur, as a consequence of decisions that could be principally based on the output of internal models, due to errors in the development, implementation or use of such models’. Thus, the main task of the model validation process is to prevent models from producing inadequate results, by effectively challenging them and by assessing and identifying possible assumptions, limitations and shortcomings.
6. Previous work on the validation in context of the Internal Ratings-Based Approach (IRB
Approach). The scope and objectives of validation have already been described by various other initiatives. This handbook is bringing together the perspectives provided by these various initiatives, also leveraging on good and best practices observed by CA. Prior to this handbook, the ‘CEBS Guidelines on the implementation, validation and assessment of Advanced Measurement and Internal Ratings Based Approaches’5 (CEBS Guidelines 10) provided clarification on the validation activities for IRB Approach in the European Union (EU). This guidance was based on the Newsletter 4 on validation 6 from the Basel Committee on Banking 4 Therefore, the handbook provides general supervisory ‘expectations’, for the consideration of CAs, which are linked to already existing regulatory requirements. It also contains a set of what is considered as ‘best practices’ based on supervisory experience, as well as ‘good practices’ observed in institutions. 5 https://www.eba.europa.eu/sites/default/documents/files/documents/10180/16094/525151b9-ea22-42b2-bd28- 00e35a0add7e/GL10.pdf?retry=1 6 https://www.bis.org/publ/bcbs_nl4.pdf

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 9 Supervision and listed six principles that validation of the IRB Approach should follow. 7 In the United States of America, the Board of Governors of the Federal Reserve System issued an ‘SR letter’ dealing with the ‘supervisory guidance on model risk management’. 8 This letter provides guidance on effective model risk management, where model validation plays a critical role. It builds on a previous bulletin issued by the Office of the Comptroller of the Currency in 2000, which outlines key model validation principles and expectations on sound model validation processes which was updated in 2011 and included in the 2011 Federal Reserve System newsletter.
7. The IRB validation beyond a model validation. In all these previous publications, there is a
consensus that the validation of IRB rating systems goes beyond the pure concept of model validation. The validation of IRB rating systems is not limited to the proper functioning of a model from a statistical perspective, but also includes the assessment of the data quality, the structure of the rating system and its correct application. It includes the set of policies, processes and procedures put in place to assess the accuracy and performance of the rating systems 9 on the institution-specific portfolios and to verify that the models 10 used by the institutions work properly.
8. The IRB validation in the EU framework. This understanding is reflected in the EU regulation,
11 and further elaborated in Chapter 3 of the CDR on assessment methodology 12 which clarifies the interaction between the validation of IRB rating systems and the internal governance, as well as risk oversight in general. As such, the internal validation framework is not only limited to the tasks and organisation of the validation function: the regulation includes minimum requirements on the senior management and management body, 13 the internal reporting, 14 the 7 Principle 1: Validation is fundamentally about assessing the predictive ability of a bank’s risk estimates and the use of ratings in credit processes; Principle 2: The bank has primary responsibility for validation; Principle 3: Validation is an iterative process; Principle 4: There is no single validation method; Principle 5: Validation should encompass both quantitative and qualitative elements; Principle 6: Validation processes and outcomes should be subject to independent review. 8 https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf 9 A ‘rating system’ is defined in the Article 142(1), point (1) of the Capital Requirements Regulation (Regulation (EU) No 575/2013 -CRR) as ‘the methods, processes, controls, data collection and IT systems that support the assessment of credit risk, the assignment of exposures to rating grades or pools, and the quantification of default and loss estimates that have been developed for a certain type of exposures’. 10 ‘PD model’ and ‘LGD model’ are defined in section 2.4 of the Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16 - Guidelines on PD and LGD estimation). While not provided in these guidelines, the definition of a CF model can be inferred using Article 4(1), point (56) of the CRR, as ‘All data and methods used as part of a rating system within the meaning of Article 142(1), point (1) of the CRR, which relate to the differentiation and quantification of own estimates of CF which are used to assess the level of currently undrawn amount of a commitment that could be drawn and that would therefore be outstanding at default, to the currently undrawn amount of the commitment, for each facility covered by that model.’ 11 Such as Articles 144(1)(f) and 185 of the CRR. 12 These requirements only apply indirectly to institutions as the CDR on assessment methodology provides the scope of assessment criteria and the methods to be applied by CAs. 13 Article 14 of the CDR on assessment methodology 14 Article 15 of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 10 interaction with CRCU 15 and the internal audit (IA). 16 In addition, while the validation function is not separately mentioned in the Guidelines on internal governance, 17 it can be viewed as an internal control function (section 19) given the tasks it performs.
9. The IRB validation through multiple layers of defence. The scope of activities to be performed
in the context of the validation of IRB rating systems has led to a specific set of governance and organisational requirements. In particular, the assessment of the model performance is performed by several functions, each of them with its own perspective. In this respect, the CRCU has an ‘active participation in the design or selection, implementation and validation of models used in the rating process’ 18 and as such is the first function to analyse and validate the model. However, the EU regulation requires in addition institutions to set up a specific independent validation function with its own responsibilities. Following the background and rationale of the Final draft regulatory technical standard on assessment methodology for IRB, 19 the independence of the validation function from the CRCU is essential ’in order to allow for an objective assessment of the rating systems, a limited incentive to disguise the model deficiencies and weaknesses, as well as a fresh view on the rating systems by people not involved in the development process’. 15 Article 16 of the CDR on assessment methodology 16 Article 17 of the CDR on assessment methodology 17 EBA/GL/2021/05 18 Article 190(2)(f) of the CRR 19 EBA/RTS/2016/03

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 11
1.3 Structure of the supervisory handbook
Figure 1: Structure of the validation handbook
10. Structure of the handbook (illustrated in Figure 1). A general description of the requirements
applicable to the validation function and on the tasks to be performed is given in the sections [2 General ] and [3 Validation content]respectively. In practice, the description of the validation tasks is organised in two sections. The tasks related to the model performance assessment are developed in section [3.1 Assessment of the core model performance] and the ones dealing with the modelling environment (i.e. the data quality and the IT implementation of the rating system) are further described in section [3.2 Assessment of the modelling environment]. The presentation of the core performance assessment is split in three parts:

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 12 a. Performance of the rating system - Risk differentiation:
Consistency and comprehensiveness of the rating assignment
1. Documentation for consistency
2. Comprehensiveness and conservatism for nonstandard ratings
Accuracy of the rating assignment
Discriminatory power Homogeneity &
Heterogeneity
Input data
1. Data quality
2. Completeness of the RDS
3. Data preparation (including estimations)
4. Representativeness
Methodological choices
1. Risk drivers
2. Functional forms and human judgment
3. Definition of grades or pools
Statistical tests 1. Scope and level of application
2. Various economic conditions
Validation challengers
1. Impact of overrides
2. Number of overrides
3. Stability of the ratings
4. Monotonicity of the DR
5. External data sources
6. Concentration in rating grades
b. Performance of the rating system - Risk quantification Input data
1. Data quality
2. Completeness of the RDS
3. Data preparation (review of the exclusions and realised LGD floored at 0%)
4. Representativeness (challenge adjustments)
Methodological choices
PD LGD Conservatism Downturn (DT)
1. General calibration methodology
2. Average DR (Overlapping windows
only relevant for PD)
3. LRA (including for LGD treatment of
Incomplete work-out)
4. Calibration segment and type
5. Appropriate adjustments
1. 180(1)(c) and
181(1)(c)(d) of the CRR
2. Quantification
for each MoC category
3. Aggregation of
MoC categories
1. Period for
the
Economic
DT
2. DT LGD
Validation challengers
1. Compare DR with PD and similar analysis for LGD and CF – 185(b) CRR
2. Other quantitative validation tools (best estimates) – 185(c) CRR
3. External data sources

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 13
c. Performance of the rating system - Other specific points
Defaulted exposures
1. RDS: reference dates, realised LGDs and data requirements
2. ELBE: MoC, economic conditions and SCRA
3. LGD in default: relation with LGD non defaulted and ELBE
CRM
1. RDS: source and allocation of cash flows, recoveries from collateral
2. Level of validation
3. Meaningful recognition (no double counting)
FCP UFCP
1. On-balance sheet netting and
master netting agreement
2. Adverse dependency
1. Choice of the approach
2. Recognition of multiple CRM
Use of multiple CRM
Slotting approach
1. Assessment of the assignment process
2. Assessment of the input data
3. Assessment of the modelling choices
4. Quantitative and challenger analyses
11. Difference between the first and on-going validation. While the section [3 Validation content]
describes the areas to be assessed by the validation in a general manner, the actual tasks to be performed may differ depending on the position in the validation cycle (as further described in focus box [1]):
a. The first validation of a rating system is part of the institution’s assessment conducted before submitting the application to the CA. Section [4 First validation activities] provides guidance with respect to the specific aspects of these validation activities regarding a first validation. b. On the contrary, on-going validation activities are to be conducted once the rating system has been approved by the CA in accordance with Article 143 of the CRR. Details on on-going validation activities are described in Section [5 On-going validation activities].
12. Focus sections. This handbook also covers some specific areas where the validation function
may face specific challenges:
a. Some models are developed on a broader scope than the scope of application. The specificities of the validation of this type of models are further discussed in the section [6.1 Focus 1: validation in the context of the use of external data] b. Some of the operational tasks of the validation function can be outsourced both internally or externally. However, the responsibility for these tasks remains within the validation function. This particular point is further described in the section [6.2 Focus 2: validation in the context of outsourcing of validation tasks];
c. The validation of the so-called low-default portfolio, or more generally low-data
portfolio, can result in challenges for the use of some validation tools which require a minimum number of observations to be conclusive. Further thoughts on the

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 14 validation of these models are given in the section [6.3 Focus 3: validation in the context of data scarcity].
13. Validation of CF estimates. This handbook contains less guidance on the validation of own
estimates of CF, given that these parameters were not explicitly part of the EBA IRB repair program. However and as a general remark, it is considered as best practice to have similar validation techniques in place as for the LGD risk parameter, especially when it comes to the validation of the downturn component (as mentioned in paragraph [44]) and the treatment of extreme realised values (as mentioned in paragraph [36.a]).

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 15
2. General principles for the validation
framework
2.1 Scope and objectives of the validation
14. Scope of the validation. To satisfy the requirements of Part III, Title II, Chapter 3 of the CRR,
the validation activities should be conducted at each level where a CA has granted an approval for a rating system (or is expected to do so in the context of an initial validation of a new rating system). Therefore, in the case where a rating system has received the approval on a consolidated as well as sub-consolidated and/or individual basis, the internal validation should be performed at all of these levels. While more than one validation function may be involved in the validation of a rating system, in particular in the context of outsourcing and/or when a rating system is used by different legal entities, the responsibility of the validation tasks and the validation objectives mentioned in paragraph [17] should be retained by the validation function of the entity at the level of which the rating system has been approved. As mentioned in paragraphs [18] and [25], the validation function’s resources and framework are expected to be commensurate with the complexity and materiality of the rating system.
15. Involvement of several entities in the validation through outsourcing. The degree to which
one validation function can leverage on the validation activities of another validation function is further described in section [6.2 Focus 2: validation in the context of outsourcing of validation tasks], and in particular in paragraph [135].
16. Involvement of several entities in the validation through a common rating system. In the case
where a rating system is used at different levels of a group, the validation functions of the involved entities are expected to share their findings. A good practice observed in institutions is to form an opinion on a single shared set of possible recommendations on the corrective actions against any identified model deficiency or under-estimation of risk parameters. In particular, the validation functions may come to an agreement on whether a deficiency identified at a certain level is an indication of a general deficiency of the rating system at group level, taking into account the assessment of paragraph [17], along with a common understanding of the possibilities on how these might be remediated in line with paragraph [19]. In any case, institutions are required to ensure the sufficient capitalisation at all relevant levels (consolidated, sub-consolidated and/or individual basis), taking into account the assessment of the validation functions.
17. Objective of the validation.
20 The validation function is expected to form an opinion on whether the final rating system developed by the CRCU meets the regulatory requirements and the internal expectations on the quality of the IRB models. It is expected to then communicate 20 Article 185(a) of the CRR, Article 11 of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 16 its opinion to the CRCU, the senior management and the management body, as part of the corporate governance as mentioned in paragraph [19]. 21 To this end, the validation function is expected to provide:
a. A list of all the deficiencies identified, along with an assessment of their materiality and severity (e.g. via quantitative impact), such that it can be used by the CRCU to prepare a prioritisation plan for their resolution; b. An assessment of the consequences of the combination of these deficiencies on the overall performance of the rating system, along with the consequences in terms of usability of the rating system for regulatory purposes;
c. An assessment of the level of confidence in the results of its assessments, in particular
when lack of data can be considered as an impediment to the robustness of the statistical tests. This objective and independent assessment is in particular essential in order to ensure an effective interaction with the CA, as further explained in Focus Box [1]. FOCUS BOX 1: VALIDATION CYCLE IN RELATION TO THE INTERACTION WITH THE CA Validation activities are continuously performed during the full life cycle of an institution’s rating system. These activities aim at giving confidence to the CA that the IRB rating system is working in an appropriate way, either with a view of a first approval, or for its continued use. The first validation and related validation activities take place during or subsequently to the model development in order to assess the regulatory compliance and performance of the rating system, in view of providing an approval under Article 143 of the CRR. The first validation aims at ensuring the appropriateness of the rating system once being used for own funds requirements and internal risk management. At the same time, it also ensures that the newly developed rating system is ready for a supervisory (e.g. on-site) assessment (with necessary changes to the rating system implemented by CRCU following the validation function’s assessment). 22 As such, an important focus point in the first validation is the methodological choices taken by the CRCU. The first validation of the rating system can be used as a starting point for the on-going validation and the related validation activities that are required to be conducted after regulatory approval was granted. The on-going validation activities aim at ensuring an effective challenge for the adequate model performance and appropriateness of the rating system for IRB purposes on an on-going basis, 21 Article 189 of the CRR. For the rest of this handbook, this identification and communication of the deficiencies will be referred to as ‘form an opinion on’. 22 Article 144 (1)(f) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 17 subsequently to a first validation. As such, the outcome of the on-going validation will typically be taken into account in the on-going supervisory assessment performed by the CA. These ongoing validation activities differ from the first validation as they benefit from additional data and observations and have at their disposal previous conclusions from the first validation. As a consequence, for some specific tasks, the assessment from the validation function can be based largely on its previous conclusions.
2.2 The validation function as a second layer of defence, between
CRCU and Internal Audit
18. Independence of the validation function vis-à-vis the CRCU. The validation function assesses
the final rating system developed by the CRCU as a second layer of defence, i.e. it challenges in an independent manner the model design and methodological choices used by the CRCU during the model development. Thus, the independence of the validation function is crucial to prevent any conflict of interest, as well as ensure no subordination in relation to the CRCU. This independence is ensured by two means:
a. The structural independence, ensured via the organisational setup (see Interaction box [1]). In this regard, it is expected that large and complex institutions apply the setup which provides the highest level of independence of the validation function (Point 1 of Interaction box [1]). 23 However, as further described in paragraph [23.d], the validation function can leverage to some extent on the work performed by the CRCU. b. The sufficient resource allocation. In this regard, it is expected that the number, seniority and expertise of the validation staff is commensurate with the complexity and materiality of the rating systems under the scope of validation of the validation function, 24 such that the validation function can still effectively challenge the work of the CRCU. INTERACTION BOX 1: STRUCTURAL INDEPENDENCE OF THE VALIDATION FUNCTION VIS-A-VIS THE CRCU
Article 10 of the CDR on assessment methodology provides three different types of setups within
the institution’s organisational structure which can be allowed, depending on the nature, size and scale of the institution and the complexity of the risks inherent to its business model:

The validation function is in a unit separated from the CRCU and both units report to
different members of the senior management;
23 Article 10(5) of the CDR on assessment methodology 24 Articles 10(2)(a), 10(3)(a), 10(4)(a) and 12(b) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 18
2. The validation function is in a unit separated from the CRCU, but both units report to the
same member of the senior management;
3. The validation function is not in a unit separated from the CRCU, i.e. no separated
validation unit exists, but the staff performing the validation function is different from the staff responsible for the design and development of the rating system, and from the staff responsible for the credit risk control function.
19. Communication of the findings and recommendations. The management body owns the
responsibility for all material aspects of the rating and estimation processes and senior management shall have a good understanding of the rating system designs and operations and shall ensure, on an on-going basis, that rating systems are operating properly. 25 In this context, the outcome of the validation function’s analyses, in the form of findings and recommendations, is expected to allow senior management to understand the identified model deficiencies and be in a position to decide on a remediation action plan as well as to have a good understanding of how these deficiencies are addressed in the risk estimates. Effectively, a key product of the validation function is the validation report, as further described in paragraph [24]. 26 In addition, while the validation function should perform its assessments independently from the CRCU (i.e. independently identify and report deficiencies and shortcomings) it is nonetheless expected to have a good understanding on the issues detected and on the possibilities on how these might be remediated. In any case, the findings and recommendations of the validation function should not displace the responsibility for the design or selection, implementation, oversight and performance of the rating systems, which should remain within CRCU, and the validation function should always remain critical on any changes implemented on the rating system. 27
20. The IA as a separate function. When it comes to the IA function, in accordance with Article 191
of the CRR, the ‘Internal audit or another comparable independent auditing unit shall review at least annually the institution’s rating systems and its operations’. 28 This requirement is complemented in the CDR on assessment methodology, 29 which de facto sets the IA and validation functions as independent functions within institution’s governance structure, which constitute different levels of defence and should not be merged into a single function. Hence, the roles and responsibilities of the IA and the validation function should be clearly defined, such that all relevant tasks necessary for the evaluation of the rating system are performed, as discussed in paragraph [22], and cannot be transferred between each other. This requires the existence of an effective separation between the staff of the IA and the validation function. 25 Article 189 of the CRR 26 Article 13 of the CDR on assessment methodology 27 For example, the validation function could recommend redeveloping part of or in full the rating system, or recommend a recalibration or the introduction of additional MoC. 28 In the rest of the handbook, the IA refers to either the internal audit or another comparable independent auditing unit. 29 Article 17(1)(a)(iv) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 19
21. The role of IA in the assessment of the validation function. In addition, according to paragraph
155 of the Guidelines on internal governance 30 ‘The risk management framework should be subject to independent internal review, e.g. performed by the internal audit function’. Consequently, as the validation function is part of an institution’s risk management framework, the IA function should have an independent opinion on the institution’s validation function, which encompasses:
a. The independence of the validation function, mentioned in paragraph [18]: i.e. the setup of the validation function (e.g. whether it has a sufficient number of resources) and its independence (in relation to the CRCU as well as to the personnel and management function responsible for originating or renewing exposures); 31 b. The institution’s validation policy, mentioned in paragraph [23], i.e. the scope and suitability of the validation tasks in terms of assessment of the rating systems, including the documentation to be produced. This is of particular relevance in the context of a change in the validation methodology or processes, as this may entail a categorisation of the rating system change as material, 32 and in this case the assessment of the IA is part of the application package sent to CAs; 33
c. The adherence of the validation function to the validation policy during the
performance of the validation tasks; d. The comprehensiveness and clarity of the conclusions of the validation function and the related documentation produced, including the validation report mentioned in paragraph [24]; e. The appropriateness and timeliness of the follow up of the validation function’s findings mentioned in paragraph [19] by the institution 34 (and, where relevant, of the findings raised by the CA).
22. Interaction between the IA and the validation function in the assessment of rating systems.
On top of the assessment of the validation function, the IA should review the adherence to all requirements applicable to the institution’s rating systems. 35 This assessment encompasses:
a. A high-level perspective of the institution’s rating systems, which includes in particular an overview of the rating systems and related risks to ensure the adequacy of own funds requirements (this includes the assessment of model risk, including the 30 EBA/GL/2021/05 31 Article 10(1)(a) of the CDR on assessment methodology 32 Annex I, Part II, section 1 point 4 of the CDR on model changes 33 Article 8(1)(e) of the Commission Delegated Regulation (EU) No 529/2014 (CDR on model changes): report of the institutions’ independent review 34 Article 13(c) of the CDR on IRB assessment methodology 35 Article 191 of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 20 review of the classification of the materiality and complexity of the rating systems as further discussed in Context Box [1], as well as the corporate governance 36 and use test 37 fulfilment); b. An overview of all the operations related to rating systems, which includes in particular an annual review of the performance of each rating system. However, to avoid duplication of tasks between these two functions and ensure an appropriate challenge by the IA, the independent opinion that the IA forms on the institution’s rating systems can take into consideration the analyses performed by the validation function, where appropriate. In any case, all the necessary tasks to form an opinion on the institution’s rating systems should be performed and the IA should be responsible of the assessment of their completeness (i.e. absence of gaps due to the distribution of tasks between the different internal control functions);
c. A detailed assessment of the elements not assessed in depth by the validation
function. Some of these elements are further clarified in ‘interaction boxes’ in other parts of the handbook. In practical terms, some elements may not be reviewed in depth by the validation function before the assessment of the IA as part of the overview mentioned in paragraph [22.b]. For example, the validation function may not be responsible for the detailed review of the proper implementation of each rating system, which includes in particular the integrity of the rating system and rating grades assignment process, or the correct calculation of own funds requirements (e.g. allocation of each exposure to the proper exposure class, correct application of PD and LGD input floors, calculation of the maturity, IT implementation of the rating system).
2.3 Validation policy and validation report
23. Validation policy.
38 The validation policy documents the validation framework, i.e. the roles, responsibilities, processes and content of the validation activities that are expected to be performed in a sufficiently precise manner such that a third party is able to gain a good understanding of the tasks the validation function will perform. A good practice observed in institutions is to document the validation policy in a single document. In particular, the validation policy is expected to include:
a. A description of how the validation function forms its opinion on the accuracy and consistency of the rating system as a whole. 39 This implies that the validation policy is expected to describe the aggregation methodology to be used across the different 36 Article 189 of the CRR, Chapter 3 - section 3 of the CDR on assessment methodology 37 Articles 144(1)(b), 145, 171(1)(c), 172(1)(a), 172(1)(c), 172(2) and 175(3) of the CRR, Chapter 4 of the CDR on assessment methodology 38 Articles 9(3)(c) and 12 of the CDR on assessment methodology, section 4.2.2 of the Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16 - GL on PD and LGD estimation) 39 Article 185(a) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 21 analyses, in particular where quantitative tests are performed as further described in paragraph [25.b]; b. A description of the data collection and selection process underlying the construction of all the data sets used for the purpose of validation. This ‘validation data set’ contains, but is not limited to, the validation samples as described in Focus Box [2], in addition to the reference data set (RDS) used by the CRCU during the estimation of the risk parameters or the review of estimates; FOCUS BOX 2: VALIDATION SAMPLES AND VALIDATION DATA SET The validation samples used for the performance assessment, i.e. for running the quantitative tests as further described in paragraph [25.b] validation challengers, contain all information to allow for all relevant types of validation analyses. This includes observations covering a time period that is as long as possible, a different level of consolidation (at sub-consolidated and/or individual level, where relevant) and at the level of scope of application of the individual PD and LGD models. In addition to the validation samples, other data is expected to be used by the validation function, wherever benefit can be gained from this additional information. The set of all data sets used by the validation function for the validation of a rating system constitutes the ‘validation data set’. This includes in particular the development data or other data used by the CRCU, or data gathered from independent data sources (e.g. for benchmarking purposes).
c. The list of the analyses to be performed (as further described in paragraph [25]), along
with a description of their purposes and possible limitations (i.e. underlying assumptions and theory), their scope of application (i.e. data sets on which they are applied and compulsory or discretionary use), their envisaged frequency (including if relevant for first, regular and/or full validation), the methodology to derive the scores mentioned in paragraph [17.b] where relevant, and the associated findings and recommendations mentioned in paragraph [19]. As further developed in the section [4 First validation activities] and section [5 On-going validation ], the validation framework is expected to take into account the specificities of the model life cycle. For the quantitative tests as further described in paragraph [25.b], the documentation is expected to include a high-level description of the expected data preparation process, the computations to be performed, the fixed targets and tolerance thresholds (see Focus Box [3]), as well as the potential (neither exhaustive nor mandatory) qualitative analyses to be conducted to complement the assessment;

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 22 FOCUS BOX 3: QUANTITATIVE THRESHOLDS AND COMPARATIVE ANALYSES In practical terms, while absolute thresholds can generally be considered as adequate backstops, it is usually helpful to complement them by an ad-hoc comparative analysis (such as with results of previous years as mentioned above). For the on-going validations, the comparison between the latest results of the validation and the ones observed in the previous years (and in particular the ones observed during the first validation) can be used to detect a trend (in particular in the case of a deterioration) in the model performance. In the context of the validation of rating system changes, typically the model performance of the new model has improved compared to the current version. Hence, the results of the key performance metrics of the new model can reasonably be expected to be better than the ones performed using the current model in place. Consequently, if the model performance of the new model has not improved compared to the current version, the validation function is expected to conduct a more in-depth analysis to fully understand the reasons. In the context of newly introduced rating systems, a relative comparison may be harder to find. While absolute thresholds may be deemed sufficient, other alternatives are however possible. One possible way is to compare the results of key performance metrics with the ones calculated for other rating systems (e.g. other exposure classes), for instance in the context of a roll out where relevant. Another possibility might be to use the differentiation provided by the Standardised Approach (via the different allocation of exposures into different risk weight (RW) buckets), the loss given default (LGD) and conversion factor (CF) regulatory values in the case of the introduction of the use of own LGD or CF estimates for non-retail exposures or the supervisory slotting criteria approach (slotting approach) for specialised lending exposures, and ensure that the risk differentiation provided by the IRB parameters leads to better results (this approach produces less relevant results for retail exposures). d. The conditions under which the validation function may leverage on the work performed by the CRCU (e.g. by reviewing the work performed by CRCU instead of performing its own calculation). As further developed in the section [4 First validation activities] and section [5 On-going validation ], the degree of leverage on the work performed by the CRCU may be different depending on the position in the model life cycle; e. The main content, frequency and recipients of the validation reports.
24. Validation report.
40 A key component of the communication of the validation opinion on the rating system is the validation report. Its structure is left to the own judgment of the validation function in order to optimise the communication of its opinion, and as such is not expected to 40 Articles 9(2)(c), (d), 9(3)(d), (e) and 13 of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 23 be harmonised between institutions (and is in particular not required to follow the structure of this validation handbook), nor necessarily between different rating systems for a given institution. In practical terms, the validation report is expected to be understandable by a knowledgeable third party and describes how the validation policy has been applied to a particular rating system (the validation report can refer to the validation policy while providing the validation result for a particular rating system), and as such details:
a. The rating system version that was subject to validation and, where relevant, a description of the on-going model development activities, i.e. performed since the last validation or planned in a short-term horizon. In this context, the validation report is expected to provide the opinion of the validation function on the rating system changes, including their materiality assessment, since the last validation report, and the changes themselves; b. All the relevant tests performed to challenge the rating system, along with a description of the validation data preparation and the related data quality of validation samples used mentioned in paragraph [23.b], including their sizes (e.g. in terms of number of exposures and number of years), and a comparison vis-a-vis the application portfolio and overlap with the RDS (e.g. development sample or calibration sample);
c. The outcomes of the validation analyses, which are expected to be verifiable by other
internal functions or external parties (e.g. the CRCU, the IA or the CA). In this regard, the validation function is expected to express a clear opinion on the performance of the rating system as mentioned in paragraph [17], and to determine a categorisation of the findings and the relevant recommendations in accordance with their materiality. In this context, a good practice observed in institutionsis to communicate these results in the form of scores (e.g. traffic light approach); d. A good practice observed in institutions is that the report includes a comparison between the latest results of the validation and the ones observed in the previous years (as a result of the analyses performed as per paragraph [25]), as well as the highlighting of the previously identified deficiencies, along with their severity, and a description of how they have been addressed.
2.4 Validation tasks
25. Type of analyses to assess the accuracy and consistency of rating systems. Institutions shall
have robust systems in place to validate the accuracy and consistency of rating systems, processes and the estimation of all relevant risk parameters. 41 The validation methods and procedures should be appropriate to the nature, degree of complexity and range of application of the rating systems and the data availability, and as such all relevant validation techniques 41 Article 185(a) of the CRR, Article 12(a) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 24 are expected to be used. In particular, institutions are expected to include quantitative as well as qualitative methods that are commensurate with the complexity and materiality of the rating system (in terms of volume and riskiness). 42 In particular, the areas described in section [3 Validation content] do not prevent institutions from developing additional tools and methods. In practice, for the assessment of the model performance described in section[3.1 Assessment of the core model performance], the type of analyses performed by the validation function are mainly, on the one hand, an assessment of the work done by the CRCU, and in particular of the different (modelling) choices taken during the development of the rating system, and, on the other hand, the development of own empirical validation challengers, in particular using new set of data not used during the development of the rating system:
a. Assessment of CRCU’s work and related documentation. 43 To the end of gaining a good understanding of the CRCU work, the validation function is expected to review and challenge the steps performed during model development and risk quantification (or potentially the review of estimates during the on-going validation), respectively, as well as the decisions made during these processes. In this context, as a good practice observed in institutions, the validation function reproduces the documented steps of the risk parameters’ estimation and / or reviews directly the code, and in any case is expected to check if the methodological and technical documentation of the rating system related to the validation function’s scope of assessment is usable and understandable by any third parties. 44 The usability implies that a third party is able to replicate the estimation of risk parameters and arrive at the same results, such that the validation function can independently assess the rating system. As a matter of fact, in addition to being used by the CA for the authorisation of the rating system, the documentation is expected to be used by the validation function to assess the model specifications and monitor their changes (see Interaction Box [2]), on the top of using metrics measuring the pure performance of the model. INTERACTION BOX 2: ASSESSMENT OF THE RATING SYSTEM DOCUMENTATION While the validation function assesses the content of the documentation, in particular during its assessment of the rating system developed by the CRCU, it is not necessarily expected to perform a regulatory compliance check based on the documentation it receives. 45 It is considered as best practice for institutionsto have at least the documentation of the technical IRB modelling aspects of compliance with applicable laws, rules, regulations and standards checked by the validation function. 42 Article 12(b) of the CDR on assessment methodology 43 Article 11(2)(a) of the CDR on assessment methodology 44 Articles 144(1)(e), 171(1)(b), 175 of the CRR, Article 3(1)(d) of the CDR on assessment methodology 45 E.g. assessment of the completeness of the documentation on the design and operation details of rating systems as per Article 31 of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 25 When it comes to the review of the non-technical aspects of the documentation, different practices have been observed (e.g. documentation proper track-change history, proper labelling). It is therefore not necessarily the validation function which is responsible for these checks, but in any case, it should be clear to all involved internal control functions 46 and properly documented who is responsible for ensuring compliance with which aspects of applicable laws, rules, regulations and standards, as discussed in paragraph [20]. However, as further mentioned in paragraph [79], the validation function is expected to review the documentation submitted to CAs in the context of a material change of a rating system (i.e. the application package that documents the change of the rating system). 47 b. Assessment via validation challengers. As mentioned in paragraph [23.a], the validation function is expected to form an opinion on the accuracy and consistency of the rating system as a whole. 48 This implies that, in the case where statistical tests are performed at a more granular level (e.g. for specific years or specific grades or pools), or when multiple tests or metrics are calculated, the validation function is expected to develop an aggregation methodology to deliver an overall assessment of the performance of the rating system as a whole. During this aggregation, specific attention is expected to be retained on the results of the assessment where a high share of obligors and exposure values are observed (in the development and application portfolio).
26. Consistency of the validation tasks in the performance assessment. The assessment of the
performance of the rating systems should be performed ‘consistently and meaningfully’, 49 and as such the institution should define and implement validation methods and procedures that are consistent across rating systems as well as over time. 50 It is considered as best practice ensure to the changes in the validation policy are recorded and highlighted, both for the methods and the data used (data source and periods covered). Nevertheless, institutions are expected to seek for state-of-the art validation techniques as well as to develop a targeted validation framework (e.g. processes, tests or frequencies) for a specific type of portfolios or rating systems with specific challenges, including using data appropriate to the portfolio. 51 Particular validation challenges are further discussed in the section [6 Focus on specific validation challenges].
27. Assessment of the materiality of a model change or extension. In the context of the
development activities that occurred since the last validation, the validation function should review the materiality of all rating system changes and extensions, and their combined 46 Paragraph 169 of the GL on internal governance 47 Article 8 of the CDR on model changes 48 Article 185(a) of the CRR 49 Article 185(a) of the CRR 50 Article 185(d) of the CRR, Article 12(d) of the CDR on assessment methodology 51 Article 185(c) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 26 effects. 52 This includes a qualitative assessment, using the relevant annexes of the CDR on model changes, and the quantitative assessment, using the thresholds defined in Article 4(1)(c) of the CDR on model changes. This quantitative assessment requires the institution to be in a position to estimate the own funds requirements resulting from its updated risk parameter estimates, as well as to be in a position to estimate the own funds requirements resulting from the last version of the model before the implementation of the change. CONTEXT BOX 1: ASSESSMENT OF THE PROCESS-RELATED ASPECTS OF RATING SYSTEM CHANGES BY IA The IA is expected to review the process-related aspects of model changes as regard to their identification, notification and classification (i.e. change in the interpretation and implementation of the requirements from the CDR on model changes). In any case, the IA is expected to check the process ensuring that one material extension or change is not split into several changes or extensions of lower materiality. 53 In any case, the register of rating systems should cover all changes and extensions, and include their impact as part of the description of the change category assigned in accordance with the CDR on model changes. 54 52 Article 11(2)(d) of the CDR on assessment methodology 53 Article 3(3) of the CDR on model changes 54 Article 32(2)(c) and (d) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 27
3. Validation content
28. Purpose of this section. This section recalls the main dimensions on which the validation
function is expected to have an opinion, without listing the exact validation tasks and analyses to be conducted to form such opinions. In any case, as mentioned in paragraph [25], this section does not prevent institutions from developing additional tools and methods. This is because, as stated in paragraph [26], the validation framework to be defined by each institution (e.g. processes, tests or frequencies) is expected to be tailored to the specificities of a given type of exposures or rating system. 55
3.1 Assessment of the core model performance
29. Dimensions of the assessment of the core model performance. One of the objectives of the
validation function is to assess the core performance of the rating system. 56 As such, this assessment can be broken down using the structure of the CRR, i.e. distinguishing between the performance in terms of risk differentiation and risk quantification:
57 a. Risk differentiation: The model should allow for a meaningful differentiation of risk 58 to ensure grouping of sufficiently homogenous exposures into the same grade or pool. To this end, the validation of a model is expected to evaluate in particular its discriminatory power, as well as the homogeneity within and heterogeneity across grades or pools. 59 b. Risk quantification: The estimates should meet all regulatory requirements. 60 To this end, the validation of the risk parameter estimates should include a comparison of realised DR with estimated PDs for each grade or pool, and analogous analysis for LGDs and CFs where institutions received permission to use own estimates for those risk parameters, 61 taking into account the rating philosophy. 62 For LGD and CF estimates, this should include an assessment of their appropriateness for an 55 Article 142(1) point (1) and (2) of the CRR 56 Article 11(2)(c) of the CDR on assessment methodology 57 Articles 144(1)(a) and 185(a) of the CRR 58 Article 170(1) and (3) of the CRR 59 Articles 170(1)(a),(d) and 170(3)(c) of the CRR, Articles 36 of the CDR on assessment methodology and paragraphs 69 and 130 of the GL on PD and LGD estimation. 60 Articles 178 to 184 of the CRR 61 Article 185(b) of the CRR. For the rest of the handbook, unless specified otherwise, the requirements on the LGD and CF models apply only to institutions which have received permission to use own estimates of those risk parameters for the respective type of exposures (i.e. within the respective rating system). 62 Article 12(f) of the CDR on assessment methodology, paragraph 66(c) of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 28 economic downturn where those estimates are more conservative than the long-run average. 63
30. Calculation of IRB metrics. The validation function is expected to assess the performance of
the rating system using regulatory definitions. To this end, the IRB metrics used to assess the core model performance are expected to be calculated according to the regulatory requirements (and it is expected that the related data requirements are fulfilled). 64 This includes the calculation of the realised DR, 65 the economic loss and realised LGD 66 as well asthe realised CF, 67 while the definition of default may be treated separately (see Interaction Box [3]). In practice, the validation function is expected to form an opinion on the compliance of the IRB metrics used by CRCU as part of the model development, risk quantification and the review of estimates. When the validation function does calculate these IRB metrics by itself to run the statistical tests and validation challengers, it is expected to compare its own IRB metrics as well as the results of the analyses with those that are derived by the CRCU. INTERACTION BOX 3: ASSESSMENT OF THE DEFINITION OF DEFAULT The assessment of the definition of default may involve multiple analyses, which are not necessarily conducted by the same function:

The validation function may not necessarily perform the assessment of the
compliance of the internal criteria used for the identification of defaulted exposures with the regulation 68 . Nevertheless, the responsibility of the review for these assessments should be clear to all involved internal control functions;
The validation function may not necessarily perform the assessment of the correct
implementation of the default definition, and in particular the documentation, the implementation in the IT systems 69 and the identification and monitoring of the technical past due situations, 70 Nevertheless, the responsibility of the review for these assessments should be clear to all involved internal control functions; 63 Articles 181(1)(b) and 182(1)(b) of the CRR, Commission Delegated Regulation (EU) 2021/930, GL on downturn LGD estimation and GL on LGD estimation. 64 Sections 5.3.1, 6.1.2 and 7.1.3 of the GL on PD and LGD estimation 65 Article 4(1) point (78) of the CRR, Article 46(1) of the CDR on assessment methodology, section 5.3.2 of the GL on PD and LGD estimation, Q&A_2019/4599 66 Articles 5 point (2), 175(3), 181(1)(i), 182(1)(c) of the CRR, Article 48(h), 49 and 54(d) of the CDR on assessment methodology, sections 6.3.1 and 7.3.1 of the GL on PD and LGD estimation 67 Article 182(1)(c) of the CRR, Article 48(h) and 54(d) of the CDR on assessment methodology and section 6.3.1.2 of the GL on PD and LGD estimation 68 Articles 175(3) and 178 of the CRR, Commission Delegated Regulation (EU) No 2018/171, Chapter 6 of the CDR on assessment methodology and the Guidelines on the application of the definition of default (EBA/GL/2016/07) 69 Chapter 10 of the GL on the definition of default 70 Paragraphs 23 and 24 of the GL on the definition of default

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 29
3. The definition of default may trigger some issues in the model development or risk
quantification (e.g. representativeness, reconstruction of historical time series), which are on the other side expected to be assessed by the validation function, including the related appropriate adjustments and margin of conservatism (MoC). As such, the validation function is expected to review the documentation related to the definition of the default in order to understand the changes impacting the RDS, as well as in the context of a material rating system change as part of the review of the documentation submitted to the CA (see Interaction Box [2]).
3.1.1 Risk differentiation
31. Dimensions of the assessment of the risk differentiation. The validation function is expected
to form an opinion on two dimensions (see Context Box [2]) to assess the risk differentiation performance of a model:
a. The consistency and comprehensiveness of the rating assignment process; b. The accuracy of the rating assignment in the model development.
32. Outcome of the assessment of the model. In any case, the validation function is expected to
be confident enough that all the deficiencies observed in the risk differentiation are sufficiently 71 Article 171(2) of the CRR, section 8.1 of the GL on PD and LGD estimation 72 Article 171(2) of the CRR, paragraph 74 of the GL on PD and LGD estimation 73 To note, this conservative rating assignment is different to the conservatism added to the estimates (the latter being so called ‘margin of conservatism’). CONTEXT BOX 2: ACCURACY IN THE DEVELOPMENT SAMPLE AND CONSERVATISM IN THE APPLICATION PORTFOLIO OF THE RATING ASSIGNMENT [Q&A 2021/5761, Q&A 2019/5029] To ensure that RWEA are calculated in a conservative way, IRB models generally need to be applied in a conservative manner, i.e. the rating assignment process itself is required to be conservative when relying on insufficient information. 71 This requirement is frequently implemented by e.g. using conservative assumption(s) in case of a lack of information or missing risk drivers. In contrast to the application, the model development should ensure that the risk quantification is based on an accurate rating assignment. 72 As such, a conservative rating assignment 73 could lead to a biased default rate calculation and risk parameter estimation when used subsequently for the calibration sample. This aspect is further developed in the Focus Boxes [3] and [4].

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 30 limited according to its criteria defined in the validation policy (including the fixed targets and tolerance thresholds mentioned in paragraph [23.c] and the Focus Box [2]). 74
33. Assessment of the consistency and comprehensiveness of the rating assignment process. In
order to validate the on-going rating assignment process, the validation function is expected to review the framework used for the rating assignment process, such that:
a. The rating assignment process is adequately documented and understandable by a third party, such that it can be performed in a consistent manner (see Interaction Box [4]), both in terms of definition of the scope of application of the rating system as well as in terms of definition of rating criteria (including the assignment to a ranking method and to a calibration segment); 75 b. The assignment process is performed in a comprehensive manner. 76 In practice, the validation function is expected to analyse the policy for the treatment of those cases where the obligor or facility could not be assigned to an obligor grade or pool based on the ‘standard‘ rating assignment and assess the materiality of these cases in the application portfolio (along with the resolution plan proposed by the CRCU). 77 This includes missing ratings and cases where the assignment was based on outdated or missing data, or where the assignment could not be renewed in time (outdated ratings). A good practice observed in institutions is to make the assessment of materiality in terms of exposure value and RWEA, as well as in terms of number of obligors or facilities to monitor the magnitude of the deficiencies. 74 As explained in the Context Box [2], deficiencies or lack in performance in the risk differentiation can generally not be covered directly by additional conservatism. It is nevertheless likely that a lack of homogeneity within grades or pools will indirectly lead to a higher uncertainty at grades or pools level in the risk quantification process and will increase the MoC added to the best estimates. 75 Articles 142(1)(1) and (2) and 171(1) of the CRR, Articles 24(1)(a), (c), (d), 31(2)(b), 32(2)(a) of the CDR on assessment methodology, section 4.1 of the GL on PD and LGD estimation. 76 Articles 24(1) and 25(2) of the CDR on assessment methodology 77 Article 171(2), 172(2) and 173(1) of the CRR, Article25(3) of the CDR on assessment methodology and section 8.1 and paragraph 75 of the GL on PD and LGD estimation. N.B.: these cases are not expected to be dealt with the standardised approach, but instead rather through a conservative IRB treatment (i.e. via a downgrade of the exposure).

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 31
34. Dimensions of assessment of the accuracy of the rating assignment. IRB models are in practice
based on either statistical models or other mechanical methods, in order to assign exposures to obligors or facilities grades or pools. In this context, the validation function should assess the input data, challenge all methodological choices used during the risk differentiation, and perform statistical tests on the model performance, in order to form an opinion on two dimensions of the accuracy of the rating assignment, namely:
78 Article 173 of the CRR
79 Articles 173(1)(b) and 173(2) of the CRR, Article 25(2) of the CDR on assessment methodology INTERACTION BOX 4: ASSESSMENT OF THE HUMAN JUDGMENT IN THE ASSIGNMENT PROCESS The validation function is expected to review the incorporation of subjective data in the model (i.e. the specification of the model) for the assignment of the exposures to grades or pools. This assessment is expected to be conducted along two dimensions:
1. The clarity of the definitions, processes and criteria defined by the CRCU to ensure the
consistency of the rating assignment, as described in paragraph [33.a];
2. The integration of the human judgment in the overall rating assignment, as mentioned
in paragraph [36.b].
On the other hand, the validation function may not necessarily perform the evaluation of the actual implementation (e.g. integrity of the assignment process). 78 Nevertheless, the responsibility of the review for these assessments should be clear to all involved internal control functions. This includes:
1. Governance aspects, such as the assessment of the independence of staff and
management responsible for the final approval of the assignment from the ones responsible for the origination or renewal of exposures;
2. Application aspects, such as the evaluation of the consistency between the framework
and the actual implementation of the human judgment (i.e. the application of the model specifications). In addition, the IA may assess the rating assignment review and in particular the frequency and adequacy of the assignment process (in order to ensure that all outdated ratings are properly identified as such). 79

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 32 a. The discriminatory power of the model, i.e. its capacity to efficiently discriminate riskier obligors or facilities from less risky ones, based on the difference in the level of default (for the PD), loss given default (for the LGD) and conversion (for the CF) risk; 80 b. The homogeneity within each grade or pool, in terms of default, loss given default and conversion risk, and the heterogeneity between grades or pools, in terms of distributions’ overlaps of default, loss given default and conversion risk between all grades or pools. 81
35. Assessment of the input data. The assessment of the input data is expected to include:
a. An opinion on the data quality of the full RDS, as part of the validation function activities described in section [3.2.1]; b. A review of the completeness of the RDS, 82 in terms of scope (obligors, facilities and default identification) and information (values of the risk drivers at the relevant dates, data necessary for calculating realised DR, realised LGD and realised CF and any other relevant data used in the risk differentiation); 83
c. A review of all the procedures applied to the data used for the development of the
model, including data collection, data cleansing, data processing (e.g. normalisation, treatment of collinearity) and data estimation (e.g. cash flow projections used for specialised lending). A good practice observed in institutions is to complement the review of the framework used for these estimated input data with back-testing comparisons between these estimations (including the projections which go beyond the one-year time-horizon) and the subsequently realised values (out-of-time (OOT) validation tests); d. The analysis of the representativeness of the development sample vis-à-vis the application portfolio. 84 The validation function is expected to include in its assessment the dimensions mentioned in the GL on PD and LGD estimation: the scope of application, the definition of default, the distribution of relevant risk characteristics as well as the lending standards and recovery policies. 85 This assessment should be done in accordance with the process and methods defined in the validation policy. 86 80 Article 170(1) and (3) of the CRR 81 Articles 170(1)(a), (d) and 170(3)(c) of the CRR, Articles 36 of the CDR on assessment methodology and paragraphs 69 and 130 of the GL on PD and LGD estimation 82 Article 174(b) of the CRR 83 Sections 5.2.1, 6.1.2 and 6.2.1 of the GL on PD and LGD estimation 84 Article 174(c) of the CRR 85 Article 37(2) of the CDR on assessment methodology and section 4.2.3 of the GL on PD and LGD estimation 86 Section 4.2.2 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 33 It is expected to subsequently evaluate the measures taken by the CRCU to deal with deficiencies in these areas.
36. Assessment of the modelling choices and specifications. The assessment of the modelling
choices and specifications 87 should ensure that the chosen input variables form a reasonable and effective basis for the resulting predictions and that the model does not have any material bias. 88 As such, the validation function is expected to have a good understanding of the documentation and the features of the model, including its scope of application, limitations and weaknesses, main and alternative assumptions or approaches to those finally chosen, in order to effectively challenge them. 89 In this context, the validation function is expected to assess:
a. The selection process and related outcomes of risk drivers and rating criteria in terms of predictive power, such that all relevant information is taken into account. 90 In practice, the choices are expected to be consistent with the results of the statistical methods further described in paragraph [37] 91 and with business expectations. 92 This analysis is expected to include a review of:

The minimum list of potential risk drivers to be considered mentioned
in the regulation; 93
Extremely high values of realised LGDs, as it could require the
identification of specific risk drivers;
94
Where external ratings are used as primary risk driver, whether all
relevant information has been considered; 95
The use of third-party ratings,
96 by challenging the appropriateness of the use of rating transfers, the related use of overrides (as further discussed below) or the related use of risk drivers;
The framework for the treatment of connected clients, ensuring in
addition that cases where the obligors are assigned to a better grade 87 Article 174(d) of the CRR 88 Article 174(a) of the CRR 89 Article 175(4)(a) of the CRR, Article 38 of the CDR on assessment methodology 90 Article 171(2) of the CRR, Article 24(1)(e) and (f) of the CDR on assessment methodology 91 Article 33(1)(c) of the CDR on assessment methodology 92 Article 171(1)(c) of the CRR, Article 33(1)(b) of the CDR on assessment methodology 93 Article 170(4) of the CRR, Article 33(2) of the CDR on assessment methodology, sections 5.2.2 and 6.2.1 of the GL on PD and LGD estimation. 94 Paragraph 162 of the GL on PD and LGD estimation 95 Article 171(2) of the CRR 96 Section 5.2.3 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 34 than their parent entities are intended to be exceptional and duly justified. 97 b. Any functional form or ‘hyperparameters’ 98 used in the model development to aggregate all the risk drivers. This includes how the statistical model and human judgement (see Context Box [3]) are combined to derive the final assignment of exposures to grades or pools. 99 This assessment is expected to comprise an evaluation of the theoretical framework (such that only additional information not considered in the statistical model is incorporated via human judgment in a consistent way – see Interaction Box [4]) 100 and the comprehensiveness of its documentation. 101 CONTEXT BOX 3: THE DIFFERENT FORMS OF HUMAN JUDGMENT The human judgement refers to three particular notions:

The human judgement applied in the development of the model;
102
The human judgement applied in the process of assignment of exposures to grades
or pools, in the form of subjective input data (such as qualitative variables based on an expert-based assessment); 103
The human judgement in the form of overrides, either of input or outputs, of the
assignment process of exposures to grades or pools.
104
c. How obligor and facility grades or pools are defined, such that the methodology used
to define grades or pools ensures the homogeneity of obligors and exposures assigned to the same grade or pool over time. 105 In particular, the validation function is expected to assess whether:
97 Article 172(1)(d) of the CRR and Article 24(3) of the CDR on assessment methodology 98 See [Discussion Paper on Machine Learning] pages 16 and 17 99 Article 174(e) and Article 31(5)(d) of the CDR on assessment methodology 100 Article 39(b) and (d) of the CDR on assessment methodology 101 Article 39(c)(i) and (d) of the CDR on assessment methodology 102 Article 39(a) and (d) of the CDR on assessment methodology, paragraph 35(a) the GL on PD and LGD estimation 103 Paragraph 201(a) the GL on PD and LGD estimation 104 Article 24(2) and 39(b), (c) and (d) of the CDR on assessment methodology, paragraph 201(a) and (b) of the GL on PD and LGD estimation 105 Article 170(1) and (3)(c) of the CRR and Article 36 of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 35

The definition of grades or pools is sufficiently clear, and the rating
scale is not too granular in order to allow for a consistent assignment of obligors or facilities posing similar risksto the same grade or pool; 106
The number of rating grades meets the regulatory requirements in
terms of minimum number: the obligor rating scale for exposures to corporates, institutions and central governments and central banks shall have a minimum of seven grades for non-defaulted obligors and one for defaulted obligors and at least four grades for non-defaulted obligors and at least one grade for defaulted obligors for specialised lending exposures treated under the slotting approach; 107
The number of rating grades is not excessive: the number of exposures
in a given grade or pool is sufficient to allow for meaningful quantification and validation of the default or loss characteristics at the grade or pool level; 108 as such, a high number of rating grades can be an indication of a lack of heterogeneity between grades or pools. FOCUS BOX 4: ACCURACY OF THE RATING ASSIGNMENT IN THE CONTEXT OF MODEL (RE-) DEVELOPMENT [Q&A 2021/5761 AND Q&A 2019/5029] In the context of a model development, the rating assignment of exposures in past years may involve some operational challenges, in particular when the model cannot retrospectively be run in a fully automated manner (e.g. use of human judgement in the form of qualitative variables or overrides). In this context, the validation function is expected to assess the assumptions and limitations of the approach chosen by the CRCU to determine the rating assignment of past exposures, and get an opinion of their impact on the risk quantification (in terms of bias or additional uncertainty). In practice, this may involve:

If an institution chooses to build the risk quantification on retrospectively calculated
ratings (based consistently on the model to be calibrated), the validation function is expected to assess whether these retrospectively performed rating assignments were incorporating conservative adjustments in the ratings. For example, information on overrides or other forms of human judgement may not be available and be derived via assumptions from the overrides performed as part of the old model; 106 Article 171(1)(c) of the CRR 107 Articles 170(1) and 170(2) of the CRR, Article 34(1)(a) of the CDR on assessment methodology. To note, this assessment is also related to the validation challenger provided in paragraph [38.f]. 108 Article 170(3)(b) of the CRR, Article 34(1)(c) and (d) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 36
2. If an institution chooses to base the risk quantification on historically performed ratings
(even potentially stemming from different versions of a rating model), the validation function is expected to check if appropriate measures were taken in model development, or risk quantification respectively, to remove conservative assumptions from historical ratings. This analysis should then be considered for the assessment of the appropriateness of the MoC as further described in the Focus Box [4] in the section risk quantification.
37. Quantitative validation challenger analyses. The empirical assessment of the model
performance is expected to be based on rigorous statistical tests(see Interaction Box [5]). These tests are expected to be documented in the validation policy, as mentioned in paragraph [23], be sound and adequate (including using IRB metrics as defined in the regulation as mentioned in paragraph [30]) and consider all available data (see Focus Box [1]). In addition, they are expected to:
a. Cover the dimensions mentioned in paragraph [34]. As such, they are expected to be conducted at all relevant levels. In this context, it is considered as best practice to complement the empirical assessment based on the final rating by a deep dive analysis based on the intermediate steps of the model (e.g. qualitative or quantitative sub modules, evaluation before and after overrides) for the dimension in paragraph [34.a]. For the evaluation of the selection of risk drivers and final ranking, the evaluation is expected to be performed in particular at calibration segment level (when used). For the evaluation of the homogeneity and heterogeneity, the evaluation is expected to be performed within (homogeneity) and across (heterogeneity) grades; b. Allow for an evaluation of the performance of the model under various economic conditions. 109 INTERACTION BOX 5: STATISTICAL ASSESSMENT OF THE MODEL BETWEEN CRCU AND VALIDATION FUNCTION Institutions shall establish a rigorous statistical process including out-of-time (OOT) and out-ofsample (OOS) performance tests for validating the model. 110 In particular, institutions need to develop robust models to allow for stable model use across time (and thus to a certain extent across changing environment or economic conditions). As such, these tests are based on two different samples, as illustrated in a schematic view in Figure 2. 109 Article 35 of the CDR on assessment methodology 110 Article 175(4)(b) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 37
Figure 2 : Schematic view of the development sample versus Validation samples
These tests are expected to be used by the CRCU as part of the model development. However, as further developed in the sections [4 First validation activities] and [5 On-going validation ], the validation function is expected to or may have to perform additional tests to form its opinion on the performance of the model. NB: as these tests are related to the development of the model (risk differentiation), the above requirements are without prejudice to the need to use all relevant data when it comes to the risk quantification. 111
38. Validation challengers. In addition to these statistical tools, the validation function is expected
to assess the following:
a. The impact of overrides on the performance of the rating assignment process. To this end, the validation function is expected to assess the performance of the model before and after overrides; 112 b. The number of overrides applied on the model outcomes. This could indicate a weakness in the rating model in terms of effectiveness to consider all relevant information. 113 To this end, the validation function is expected to assess their materiality (in terms of number of obligors or facilities, their exposure value and their related RWEA) for the application portfolio, and review the threshold set as maximum acceptable rate of overrides for the model; 114 111 Articles 179, 180, 181 and 182 of the CRR 112 Article 172(3) of the CRR, Article 24(2)(d) of the CDR on assessment methodology, paragraphs 206 and 207 of the GL on PD and LGD estimation 113 Articles 170(4) and 172(1) of the CRR, Article 24(2)(d) and (e) of the CDR on assessment methodology 114 Paragraph 205 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 38
c. The stability of the ratings assigned to individual obligors or facilities (using for
instance migration matrices) in relation to the economic cycle, to understand the core feature of the model with respect to the rating philosophy. 115 The outcome of this analysis is expected to be compared to the expected outcome due to the rating philosophy. 116 In case of material deviation, this could be an indication of a deficiency in the model, such as missing risk drivers or inadequate definition for grades or pools (i.e. lack of homogeneity or heterogeneity). In addition, the validation function is expected to be aware of the rating philosophy and rating stability properties of the model, and their adequacy for the respective scope of application, the risk quantification methodologies used 117 and their impact on the stability of risk parameters. 118 The result of this analysis is expected to be considered for back-testing purposes, as mentioned in paragraphs [46] and [47]; d. The relationship between obligor grades in terms of the level of default risk. 119 In particular, this assessment can be done by analysing the monotonicity of the one-year DR or long-run average DR. The validation is expected to have a good understanding of the reasons for the non-monotonicity and is expected to take into account this analysis in particular when assessing the discriminatory power of the model. A similar analysis can be conducted for the realised LGD or realised CF in the case where rating grades are used; e. Other relevant external data sources, where available. 120 For this purpose, where a sufficient number of external ratings is available, it is considered as best practice to use them as a challenger. As such, the comparison with the ranking derived from the external ratings is expected to be used as a tool to search for potential weaknesses in terms of effectiveness to consider all relevant information; f. The potential concentration in rating grades, which if unwarranted, could be an indication of a lack of homogeneity within grades or pools and therefore of missing risk drivers. 121 In addition, concentration in rating grades can give rise to data scarcity related issues, which are further discussed in section [6.3 Focus 3: validation in the context of data scarcity]. 115 Article 12(f) of the CDR on assessment methodology, section 5.2.4 of the GL on PD and LGD estimation 116 In case of the application of paragraph 90 of the GL on PD and LGD estimation, the calibration can have an impact on the rating assignment and should therefore be considered. 117 Paragraph 66(c) of the GL on PD and LGD estimation 118 Article 11(2)(c) of the CDR on assessment methodology 119 Article 170(1)(c) of the CRR 120 Article 185(c) of the CRR 121 Articles 170(1)(d), 170(1)(f) and 170(3)(b) of the CRR, Article 34(1)(b) of the CDR on assessment methodology, paragraph 87(c) of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 39
3.1.2 Risk quantification
39. Dimensions of the assessment of the risk quantification. The validation function should assess
the input data, challenge all methodological choices used during the risk quantification and perform statistical tests between estimates and observed data, in order to form an opinion on the three dimensions of the risk quantification performance of the model:
a. The accuracy of the best estimates 122 in terms of alignment with the long-run averages per grades or pools, in relation with the observed realised DR, 123 LGD 124 and CF respectively; 125 b. The conservatism of the risk estimates, taking into account in particular the quantification of the MoC; 126
c. For the LGD and CF parameters, the appropriateness of the estimates for an economic
downturn, if those are more conservative than the long-run average. 127
40. Assessment of the input data. The input data used for the risk quantification is expected to be
reviewed to ensure that any uncertainty related to a deficiency is sufficiently covered through a MoC. 128 It is expected to include:
a. An opinion on the data quality of the full RDS, as part of the validation function activities described in [section 3.2.1]; b. A review of the completeness of the RDS, in terms of historical experience and empirical evidence in order to check that all the available data was considered for the risk quantification, as well as in terms of scope (obligors, facilities and default identification) and information (values of the risk drivers at the relevant dates, data necessary for calculating the realised DR, realised LGD and realised CF and any other relevant data used in risk differentiation or risk quantification); 129
c. A review of all the procedures for data collection and data cleansing applied to the
data used by the rating system and the compliance of the data preparation with the regulatory requirements. 130 For these years used for the risk quantification, the validation function is expected to check that all observations have been taken into 122 Paragraph 38 of the GL on PD and LGD estimation 123 Article 180(1)(a) and Article 180(2)(a) of the CRR 124 Article 181(1)(a) of the CRR 125 Article 182(1)(a) of the CRR 126 Article 179(f) of the CRR, section 4.4.3 of the GL on PD and LGD estimation 127 Articles 181(1)(b) and 182(1)(b) of the CRR 128 Section 4.4, and in particular paragraph 37(a) of the GL on PD and LGD estimation 129 Article 174(b) of the CRR, Article 42(1)(a) of the CDR on assessment methodology, sections 5.3.1 and 6.1.2 of the GL on PD and LGD estimation 130 Article 11(2)(a) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 40 account, with the exception of exclusions in the specific circumstances mentioned in the regulation (i.e. wrong rating model assignment or default identification), 131 and that those exclusions and data cleansing are duly documented. 132 In particular, the treatment of the cases with non-standard or outdated ratings (as referred to in paragraph [33.b]) are expected to be carefully reviewed, as further described in the Focus Box [4]. With respect to the LGD parameters, the realised LGDs of the cases with no loss or with a positive outcome should be floored at 0% for the purpose of the calculation of the observed average LGD and the estimation of the long-run average LGD. 133 FOCUS BOX 5: ACCURACY OF THE RATING ASSIGNMENT IN THE CONTEXT OF MODEL (RE-) DEVELOPMENT [Q&A 2021/5761 AND Q&A 2019/5029] A key input during the risk quantification is the rating assigned to each obligor or facility based on the model developed during the risk differentiation. As mentioned in the previous section, the rating assignment used for the risk quantification should be as accurate as possible to ensure homogenous grades or pools. In particular, in the context of model development, the rating assignment of obligors or facilities of the past years should not include the additional conservatism added due to insufficient information (as per section 8.1 of the GL on PD and LGD estimation), and may also require some assumptions (with hence associated limitations) by the CRCU to determine the rating assignment of past obligors or facilities when human judgement was involved. In this context, the validation function is expected to assess the materiality of the cases with nonstandard or outdated ratings, as well as the severity of the deficiency (in terms of magnitude of the uncertainty on the real rating of the obligor or facility) in the calibration segment, and check that any related uncertainty is sufficiently covered by a MoC. 134 d. A review of the representativeness 135 of the data used for the risk quantification. The validation function is expected to develop statistical tests or metrics for this task, 136 and check that any related uncertainty is sufficiently covered by a MoC. 137 As such, the validation function is expected to have an opinion on the representativeness of the samples used for risk quantification (i.e. samples used to calculate long-run131 Paragraph 71 of the GL on PD and LGD estimation 132 Article 42(1)(e) of the CDR on assessment methodology, sections 5.3.1, 6.3.2 and paragraph 163 of the GL on PD and LGD estimation 133 Paragraph 160 of the GL on PD and LGD estimation 134 Article 46(6) and (7) of the CDR on assessment methodology, paragraphs 37(a) and 75 of the GL on PD and LGD estimation 135 NB: the representativeness in the context of the risk quantification is also referred to as ‘comparability’, for instance in Annex 1, Part 1, section 1, paragraph 1(c)(ii) of the CDR on model
changes. 136 Paragraph 17 of the GL on PD and LGD estimation 137 Paragraph 37(a) of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 41 averages and calibration samples – if different) vis-à-vis the application portfolio. The representativeness should be assessed in terms of scope of application, definition of default, distribution of relevant risk characteristics, the current and foreseeable economic or market conditions, lending standards and recovery policies. 138 This assessment is expected to be performed at the calibration segment level; in the specific case of a risk quantification based on the long-run average DR at the level of grade or pool, 139 the assessment of the representativenessis expected to factor in the properties of the rating assignment.
41. Assessment of the methodological choices for PD best estimates. In order to challenge the
methodological choices used to derive the PD best estimates in relation to the long-run average DR per grades or pools, the validation function is expected to assess:
a. The choice of the general calibration methodology, 140 based on the assessment mentioned in paragraph [40] of input data for both internal data 141 and external data (as further discussed in section [6.1 Focus 1: validation in the context of the use of external data]). As such, the validation function is expected to check that the approaches used are appropriate for the portfolio and the available data, and in particular the use of a second-best approach, such as a strong reliance on external data, or a calibration for either retail or purchased corporate receivables based on an estimate of total losses, is justified by insufficient internal data. Last, the validation function should check that the use of continuous PD estimates meets the requirements set out in the regulation; 142 b. The choice of the approach used to calculate the observed average one-year DRs. 143 In practice, this refers to the choice of overlapping versus non-overlapping one-year time windows, which should be appropriately justified; 144
c. The choices underlying the calculation of the long-run-average DR,
145 and in particular the length of the historical period used; 146 138 Article 42(2) of the CDR on assessment methodology, section 4.2.4 of the GL on PD and LGD estimation 139 Paragraph 92(a) of the GL on PD and LGD estimation 140 Articles 160(2), 180(1)(a), (c), (d), (e), (f), (g), 180(2)(a), (b), (c), (d), paragraph 91 of the GL on PD and LGD estimation 141 See in particular paragraph 95 of the GL on PD and LGD estimation where institutions derive PD estimates from the estimates of losses and LGDs in accordance with Articles 161(2) and 180(2)(b) of the CRR. 142 Paragraph 96 of the GL on PD and LGD estimation 143 Section 5.3.3 of the GL on PD and LGD estimation 144 Paragraph 80 of the EBA GL on PD and LGD estimation 145 Section 5.3.4 of the GL on PD and LGD estimation 146 Articles 180(1)(h) and 180(2)(e) of the CRR, Articles 42(1)(d) and 45 of the CDR on assessment methodology, section
5.3.4 of the GL on PD and LGD estimation. In addition, as per Article 180(2)(e) of the CRR and Article 45(c) of the CDR on
assessment methodology, the validation function is expected to review the choice of not giving equal importance to historic data for retail exposures, which should be justified by better predictions of loss rates.The better prediction is to be understood in terms of capturing changes in a specific policy or business changes rather than to capture changes in macro-economic conditions.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 42 d. The choices underlying the calibration to the long-run average DR. This includes the choice of calibration segments and calibration type, 147 the choice (e.g. length) of the calibration sample within each calibration segment 148 and the associated hypothesis to arrive at the final estimates 149 considering the rating philosophy (and as such the analysis performed under paragraph [38.c]); 150 e. The existence and accuracy of any appropriate adjustment, which should result in a better estimate of the risk parameter; 151 in particular, the validation function is expected to review the impact of any correction based on the input data mentioned in paragraph [40] as well as the representativeness of the historical observation period 152 and the related impact of any adjustments performed in case of nonrepresentativeness of the likely range of variability of DRs used to derive PD estimates. 153
42. Assessment of the methodological choices for LGD best estimates. In order to challenge the
methodological choices used to derive the LGD best estimates in relation to the long-run average realised LGD per grades or pools, the validation function is expected to assess:
a. The choice of the general calibration methodology 154 based on the assessment mentioned in paragraph [40] of input data for both internal data 155 and external data (as further discussed in section [6.1 Focus 1: validation in the context of the use of external data]). As such, the validation function is expected to check that the approaches used are appropriate for the portfolio and the available data, and in particular the use of a second-best approach, such as a strong reliance on external data, or a calibration for either retail or purchased corporate receivables based on an estimate of total losses, 156 is justified by insufficient internal data, and in any case the use of external data in the form of market prices of financial instruments is only used to supplement internal experience. 157 In addition, the number of closed recovery processes should be sufficient to provide robust LGD estimates; 158 147 Paragraph 92 of the GL on PD and LGD estimation 148 Paragraph 88 of the GL on Downturn LGD estimation 149Articles 46(4)(a), (b) and (c), 46(9) of the CDR on assessment methodology, sections 5.3.5 and 6.3.3 of the GL on PD and LGD estimation 150 Paragraph 66 of the GL on PD and LGD estimation 151 Section 4.4.2 of the GL on PD and LGD estimation 152 Article 46(2) of the CDR on assessment methodology, paragraphs 83 and 94 of the GL on PD and LGD estimation 153 Article 46(3) of the CDR on assessment methodology, paragraphs 84, 85 and 86 of the GL on PD and LGD estimation 154 Articles 181(1) and 181(2) of the CRR, Article 48 of the CDR on assessment methodology 155 See in particular paragraph 95 of the GL on PD and LGD estimation where institutions derive PD estimates from the estimates of losses and LGDs in accordance with Articles 161(2) and 180(2)(b) of the CRR. 156 Articles 161(2) and 181(2)(a) of the CRR, paragraph 103 of the GL on PD and LGD estimation 157 Paragraph 102 of the GL on PD and LGD estimation 158 Paragraphs 147(b) and 148 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 43 b. The choices underlying the calculation of the long-run-average LGD. This includes the proper calculation of the arithmetic average, 159 and the choices made for the treatment of incomplete recovery processes to calculate the observed average loss rate. 160 In particular, it is expected that the validation function evaluates the following:

Choice of the length of the period for the maximum recovery process
used to estimate future recoveries; 161
Choice of the method and data set used for the estimation of future
costs and recoveries on these exposures, and the related MoC to ensure that the relevant information is taken into account in a conservative manner; 162
Impact of any adjustment for massive disposals,
163 in order to ensure that it does not result in any estimation bias on other exposures which would not be subject to such massive disposals. 164
c. The choices underlying the calibration to the long-run average LGD. This includes the
length of the historical period used 165 and the choice of calibration segments (for the LGD estimates, this choice interacts with the quantification of downturn LGD 166 and should therefore be evaluated in this context, taking also into account the existence of high values of realised LGDs 167), and calibration type; 168 d. The existence and accuracy of any appropriate adjustment, which should result in a better estimate of the risk parameter. 169 In particular, the validation function is expected to review the impact of any correction based on the input data mentioned in paragraph [40]. When assessing the representativeness of the historical observation period, the validation function is expected to check that adjustments 159 Article 48(b) of the CDR on assessment methodology, paragraph 150 of the GL on PD and LGD estimation. In addition, as per Articles 181(2) of the CRR and 47(c) of the CDR on assessment methodology and paragraphs 151 and 152 of the GL on PD and LGD estimation, the validation function is expected to review the choice of not giving equal importance to historic data for retail exposures, which should be justified by better predictions of loss rates.The better prediction is to be understood in terms of capturing changes in a specific policy or business changes rather than to capture changes in macro-economic conditions. 160 Section 6.3.2.3 of the GL on PD and LGD estimation 161 Paragraphs 155 and 156 of the GL on PD and LGD estimation 162 Paragraphs 153 and 159 of the GL on PD and LGD estimation 163 Article 500 of the CRR 164 See in particular Q&A 2019_4819 and Q&A 2019_4824 165 Articles 181(1)(j) and 181(2) of the CRR, Articles 42(1)(d), and 47(a) and (b) of the CDR on assessment methodology, sections 6.3.2.1 of the GL on PD and LGD estimation 166 Paragraph 14 of the GL on Downturn LGD estimation 167 Paragraph 162 of the GL on PD and LGD estimation 168 Paragraph 161 of the GL on PD and LGD estimation 169 Section 4.4.2 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 44 made on the basis of the changes expected in the foreseeable future do not lead to a decrease in the estimates of LGD parameter. 170
43. Assessment of the methodological choices for conservatism and MoC. In order to challenge
the methodological choices used to derive conservative estimates and quantify and aggregate the MoC, the validation function is expected to assess whether:
a. The two conservative requirements mentioned in the CRR (see Context Box [4]) are implemented in the risk estimates (in the risk quantification or in the rating assignment process in the application of the model); CONTEXT BOX 4: SPECIFIC CASES WITH CONSERVATIVE REQUIREMENTS IN THE CRR The CRR explicitly mentions several areas in which institutions should ensure sufficient prudence (on the top of generic requirements on the assignment to grade or pools when less information is available and MoC):

for exposures to corporates, institutions, central governments and central banks, where
the obligors are highly leveraged or where the obligors’ assets are predominantly traded assets, to ensure that the PDs reflect the performance of the underlying assets in periods of stressed volatility; 171
in the case of LGD, where there is a significant degree of dependence between the risk of
the obligor and that of the credit protection or provider of credit protection 172 as well as currency mismatches between the obligation and the credit protection. 173 For these two cases, the regulation does not require an explicit methodology of implementation. Therefore, the exact role of the validation function may differ depending on the approach chosen by the CRCU. In any case, the validation function is expected to check that the conservative requirements are implemented and assess its theoretical implementation. However, if the conservatism is implemented directly in the application of the model, the correct final implementation can be assessed by other internal control functions. b. The quantification of Category A and Category B MoC 174 is meaningful to cover for the uncertainty related to all identified deficiencies related to the estimation of risk parameters 175 (and in particular related to the deficiencies identified on the input data 170 Paragraph 164 of the GL on PD and LGD estimation 171 Article 180(1)(a) of the CRR, Article 46(5) of the CDR on assessment methodology 172 Article 181(1)(c) of the CRR, Article 48(f) of the CDR on assessment methodology 173 Article 181(1)(d) of the CRR 174 The categories are defined in the section 4.4.1 of the GL on PD and LGD estimation. 175 The estimation of risk parameters is defined in paragraph 8 of the GL on PD and LGD estimation.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 45 in paragraph [40] and the adjustments mentioned in paragraphs [41.e] and [42.d]), and Category C MoC covers the general estimation error. 176 In addition, for CF estimates, larger MoC should be incorporated where a stronger positive correlation can reasonably be expected between the default frequency and the magnitude of CF; 177
c. The aggregation of MoC within each category is meaningful,
178 the aggregation between categories is additive, 179 the MoC is applied on the best estimate of the risk parameter, 180 and each category does not lead to a decrease of the risk parameter estimates. 181
44. Assessment of the methodological choices for downturn risk parameter estimates. In order
to challenge the methodological choices used to derive LGD and CF estimates appropriate to an economic downturn, the validation function is expected to assess:
a. The methodology used to identify the nature of the economic downturn (in particular, the choice of economic factors), 182 its severity (in particular, the length of the historical period considered), 183 and its duration (in particular, that the downturn period spans a 12-month period, or that any length of a downturn period exceeding 12 months is properly justified), 184 according to the CDR on downturn; b. For the downturn LGD estimates, 185 the methodology chosen for the estimation, 186 the comparison with long-run averages, 187 the sensitivity of downturn LGD estimates to changes in economic cycles, 188 the aggregation of the impacts from intermediate 176 Article 44 of the CDR on assessment methodology and section 4.4 of the GL on PD and LGD estimation 177 Article 182(1)(c) of the CRR 178 Paragraph 43 and 44 of the GL on PD and LGD estimation 179 Paragraph 45 of the GL on PD and LGD estimation 180 Paragraph 46 of the GL on PD and LGD estimation, with the exception of the ELBE which should be void of any Margin of Conservatism according to paragraph 182 of the GL on PD and LGD estimation 181 Paragraph 47 of the GL on PD and LGD estimation 182 Article 2 if the CDR on downturn 183 Article 3 if the CDR on downturn 184 Article 4 if the CDR on downturn. To note, this duration is used for the purpose of identifying the downturn, while the duration used for the risk quantification is further assessed in paragraph [44.b]. 185 Article 50 of the CDR on assessment methodology 186 Section 4.3 of the GL on Downturn LGD estimation 187 Paragraph 16 of the GL on Downturn LGD estimation, the floor at the long run average is also applicable to the CF as per Article 182(1)(b) of the CRR 188 Paragraph 17 of the GL on Downturn LGD estimation. More generally, the validation function is expected to assess whether the downturn estimates do not fluctuate with the economic cycle.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 46 parameters, 189 the calculation of the reference value 190 and the estimation process for defaulted exposures. 191 In addition:

In the case the institution uses the downturn LGD estimation based on
observed impact, the validation function is expected to challenge the analysis requested by paragraph 27 of the Guidelines for the estimation of LGD appropriate for an economic downturn (‘Downturn LGD estimation’) 192 (GL on Downturn LGD estimation), its incorporation in the calibration of the downturn 193 and the MoC applied in case where no impact of a downturn period is observed in the institution’s relevant loss data; 194
In the case the institution uses the downturn LGD estimation based on
estimated impact,
195 the validation function is expected to challenge the choice of the methodology used by the institution (‘Haircut approach’ versus ‘Extrapolation approach’), the use of intermediate parameter in the calibration of the downturn 196 and the MoC applied in case of a lack of data. 197

Validation challengers. The validation function is expected to develop and use various
statistical tools, to ground its opinion on the model performance on empirical evidence, making particular use of the most recent validation sample (and more generally of any data available, see Focus Box [1] and section [6.3 Focus 3: validation in the context of data scarcity]) and using IRB metrics as defined in the regulation as mentioned in paragraph [30]. The analyses that are expected to be performed by the validation function include the back-testing of risk estimates, as further described in paragraph [46], the assessment of the accuracy of model prediction and best estimate calibration, as further described in paragraph [47] as well as the benchmarking analysis, as further described in paragraph [48]. In this context, it is considered as best practice to complement the empirical assessment based on the final risk estimate by a deep dive analysis based on the intermediate steps of the estimation (i.e. from the risk differentiation steps, e.g. before and after application of specific sub modules, before and after overrides, as well as from the risk quantification steps, e.g. before and after adjustment(s) of the long run average, before and after Appropriate adjustment and MoC). 189 Paragraph 26(a) of the GL on Downturn LGD estimation 190 Paragraphs 18 and 19, section 8 of the GL on Downturn LGD estimation 191 Paragraph 21(b) and (c) of the GL on Downturn LGD estimation 192 EBA/GL/2019/03 193 Paragraph 28 of the GL on Downturn LGD estimation 194 Paragraph 29 of the GL on Downturn LGD estimation 195 Section 6 of the GL on Downturn LGD estimation 196 Paragraph 34 of the GL on Downturn LGD estimation 197 Paragraph 35 of the GL on Downturn LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 47
46. Back-testing of risk parameter estimates. In order to form this opinion on the appropriateness
of the risk parameter estimates, the validation function should compare the realised DR with the estimated PD for each grade or pool and perform analogous analysis for LGD and CF estimates (‘back-testing’), 198 considering the rating philosophy of the model as mentioned in paragraph [38.c]. 199 This comparison is expected to be used to challenge the level of the final risk parameters used for the RWEA calculation (while the assessment of accuracy of the model is rather assessed in paragraph [47]), and as such the LGD and CF estimates used for this comparison are the estimates used for the RWEA calculation, i.e. the ones appropriate for an economic downturn if more conservative than the long-run average. 200 In practice this means the following:
a. For this assessment, the validation function should make use of historical data that cover a period as long as possible. 201 In this context, it is considered as best practice to use not only the full historical data in one test, but also to consider multiple subperiods separately for this purpose (e.g. for any single year - including the latest 12- months period, most recent three years, most recent five years, one economic cycle, etc.); b. For the back-testing of PD estimates, where the realised one-year DR in a grade or pool falls outside the expected range for that grade or pool, the validation function is expected to analyse the deficiency in accordance with point [d]. In this context, it is considered as best practice to consider the deviation in light of:

Whether this deviation happened during an extreme year considering
the business cycle (i.e. bad year in case of DR exceeding PD estimates and vice versa) as reflected by economic indicators that are relevant for the considered type of exposures; 202
The rating philosophy,
203 i.e. how the business cycle interacts with the related systematic variability in default experience; 204
The results for other sub-periods as mentioned in point [a];
When defining the expected range, particular care is expected to ensure no double counting between the statistical uncertainties from the estimation (covered by the 198 Article 185(b) of the CRR 199 Article 12(f) of the CDR on Assessment methodology, paragraph 66(c) of the GL on PD and LGD estimation 200 Articles 181(1)(b) and 182(1)(b) of the CRR 201 Article 185(b) of the CRR 202 Paragraph 83(b) of the GL on PD and LGD estimation 203 Where rating assignment process is highly sensitive to economic conditions, grade assignment will change significantly while default rates of each grade remain relative stable. Contrary when the assignment is less sensitive to economic conditions, the yearly default rates per grade will capture the cyclicality of the economic conditions. 204 Article 12(f) of the CDR on Assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 48 MoC) and the one from the realised DR (accounted for in the confidence interval). As a result, a one-year DR higher than the regulatory parameter is expected to be analysed with particular scrutiny;
c. For the back-testing of LGD estimates, the validation is expected to perform three
checks: (i) one comparing the LGD estimates with the realised LGDs using only closed cases (including cases for which the maximum period of the recovery process has been reached), 205 (ii) one using all cases (i.e., incorporating some estimations of future recoveries on incomplete cases), and (iii) a comparison of the estimation of future costs and recoveries on incomplete cases versus their realisation. 206 In case where the realised LGD in a grade or pool falls outside the expected range for that grade or pool, the validation function is expected to analyse the deficiency in accordance with point [d]; When defining the expected range, particular care is expected to ensure no double counting between the statistical uncertainties from the estimation (covered by the MoC) and the one from the average realised LGD (accounted for in the confidence interval). As a result, a realised LGD higher than the regulatory parameter, especially if observed outside of downturn or worse economic conditions, is expected to be analysed with particular scrutiny; d. Where the results of these analyses show an inappropriate level of the regulatory parameter in question (either in the form of values outside of the expected range as mentioned in paragraphs b and c, or where realised values continue to be higher than expected values, and in particular higher than the regulatory parameter), appropriate actions are expected (e.g. in the context of the review of estimates), considering a high severity in terms of deficiency (i.e. to be resolved in the short term). 207 The analyses are expected to take into account the confidence level of the back-testing results, as mentioned in paragraph [17.c].
47. Accuracy of model prediction and best estimate calibration. In addition, the validation
function is expected to assess the accuracy of the model prediction using other quantitative tools, considering in particular the rating philosophy of the model as mentioned in paragraph [38.c]:
208 a. For this assessment, the validation function is expected to make use of historical data that cover a period as long as possible. 209 In this context, it is considered as a best practice to use not only the full historical data in one test, but also multiple sub205 Paragraph 156 of the GL on PD and LGD estimation 206 Paragraph 158(b) of the GL on PD and LGD estimation 207 Article 185(e) of the CRR, Chapter 9, and in particular paragraph 217(c), of the GL on PD and LGD estimation 208 Article 185(c) of the CRR, Article 12(f) of the CDR on Assessment methodology, paragraph 66(c) of the GL on PD and LGD estimation 209 Article 185(c) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 49 periods (e.g. most recent year, most recent three years, most recent five years, one economic cycle) separately for this purpose; b. These other quantitative tools are expected to include a back-testing of the PD best estimates (i.e. without any conservative adjustment) for each grade or pool, to assess the accuracy of the model predictions. In this context, it is considered as best practice to assess the distance between the observed DR and the PD best estimates in a similar way as for the assessment with the PD estimates described in paragraph [46.b];
c. These other quantitative tools are also expected to include a back-testing of the LGD
and CF best estimates for each grade or pool, as well as of the final long-run average estimates when the back-testing of LGD and CF estimates was performed on the parameters appropriate for an economic downturn. For the LGD estimates, a good practice observed in institution is for the validation function to perform as an additional check a comparison of the LGD estimates with the realised LGDs using only closed cases and excluding cases for which the maximum period of the recovery process has been reached; d. For this assessment, a good practice observed in institutions is to complement the tests based on the final rating grades assignment used for the own funds calculation (see Context Box [2], Focus Boxes [3] and [4]) by other tests based on rating grades without additional conservatism (e.g. by excluding obligors or facilities with a ‘nonstandard’ rating or assigning them to rating grades without conservative assumption); e. Where the results of these analyses show an inappropriate level of model predictions for the parameter in question, appropriate actions are expected (e.g. in the context of the review of estimates framework). The analyses are expected to take into account the confidence level of the back-testing results, as mentioned in paragraph [17.c].
48. External data sources. Last, the validation function should perform an analysis based on
relevant external data sources, where available.
210 For this purpose, the DR associated to external rating grades may be used as challenger for low default portfolios, as well as benchmarks provided by the EBA on the EBA benchmarking portfolios (see Context Box [5]). CONTEXT BOX 5: EBA SUPERVISORY BENCHMARKING Since 2015, the EBA has been conducting an annual supervisory benchmarking exercise for credit risk models. The underlying framework is mandated by Article 78 of the CRD, which requires CAs to conduct an annual assessment of the quality of internal approaches used for the calculation of own funds requirements. To assist CAs in this assessment, the EBA calculates and distributes 210 Article 185(c) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 50 benchmark values against which individual institutions’ risk parameters can be compared. These benchmark values are based on data submitted by institutions as laid out in EU Regulation 2016/2070, which specifies the benchmarking portfolios (for all exposures, i.e. for both lowdefault portfolios and high-default portfolios), templates and definitions to be used as part of the annual supervisory benchmarking exercises. By nature, these benchmarks are not at the level of grades or pools (which are institution-specific), but at the level of portfolios.
3.1.3 Other specific points
49. Scope of the section. This section describes the specificities of three aspects, which are
complementing the expectations described in the general section. As such, the usual validation activities described in the previous section are expected to be performed. This section deals with the validation of rating systems in relation to:
a. Defaulted exposures, i.e. the methodology used to derive LGD in-default and expected loss best estimates (ELBE) for defaulted exposures; b. Credit risk mitigation (CRM) techniques;
c. Exposures risk weighted according to the slotting approach.
Specificities related to the validation of defaulted exposures’ risk parameters
50. Applicability of all requirements to defaulted exposures. The validation of the LGD within a
rating system encompasses a specific review for defaulted exposures, i.e. validation of LGD indefault and ELBE. 211 As such, all the activities related to the LGD parameter mentioned previously on the risk differentiation and risk quantification are expected to be performed, 212 but using the reference dates instead of the dates of default (for instance for the back-testing). Therefore, these risk parameters entailsome specific validation tasks, even though the RDS and the methods used should be generally consistent with the ones used for non-defaulted exposures. 213
51. Additional requirements on the RDS related to the estimation for defaulted exposures. As
opposed to the LGD estimation for non-defaulted exposures, additional relevant information from and after the default should be taken into account, 214 and therefore may entail a specific validation of both risk differentiation and risk quantification. 215 In particular, on top of assessing the RDS along with the general requirements for non-defaulted exposures, the validation function is expected to:
211 Article 51 of the CDR on Assessment methodology, Chapter 7 of the GL on PD and LGD estimation 212 Paragraph 167 of the GL on PD and LGD estimation 213 Section 7.1.1 and paragraph 175 of the GL on PD and LGD estimation 214 Article 171(2) of the CRR, Section 7.1.3 of the GL on PD and LGD estimation 215 Sections 7.2 and 7.3 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 51 a. Challenge the way reference dates have been defined (such that it is consistent with the recovery patterns observed); 216 b. Check that realised LGDs related to each of these reference dates have been appropriately calculated. In this context, compared to non-defaulted exposures, the realised LGDs of defaulted exposures requires some re-calculation to account for, in particular, the treatment of the capitalised fees and interests, the discounting of cash flows and drawings and the treatment of partial write-offs; 217
c. Check that the related data requirements are fulfilled.
218
52. Specific analyses for ELBE. The validation function is expected to check that the ELBE
parameters:
a. Neither include MoC,
219 nor any other adjustments for conservatism; b. Are sufficiently reflective of the current economic circumstances. 220 In this context, the validation function is expected to check that the long-run average LGD is not unduly adjusted if one of the conditions of paragraph 184 of the GL on PD and LGD estimation is met and thereby current economic conditions are already reflected, 221 and to challenge any adjustment related to economic conditions; 222
c. Rely on specific credit risk adjustment models only if all the LGD estimation
requirements are met (including the definition of the economic loss), 223 or rely on individually assessed specific credit risk adjustments via overrides if the accuracy of the ELBE is improved and reflective of the definition of the economic loss. 224 In general, the validation function is expected to check that any situations where the specific credit risk adjustments exceed the ELBE amounts are justified adequately. 225
53. Specific analyses for LGD in-default. The validation function is expected to check:
a. The consistency between defaulted and non-defaulted estimates, including aspects related to the downturn nature of the estimate. 226 In this context, the validation function is expected to identify and challenge any systematic deviations in the LGD 216 Section 7.1.2 of the GL on PD and LGD estimation 217 Section 7.3.1 of the GL on PD and LGD estimation 218 Section 7.1.3 of the GL on PD and LGD estimation 219 Paragraph 182 of the GL on PD and LGD estimation 220 Article 181(1)(h) of the CRR, section 7.3.2 of the GL on PD and LGD estimation 221 Paragraph 184 of the GL on PD and LGD estimation 222 Paragraph 185 of the GL on PD and LGD estimation 223 Paragraph 186 of the GL on PD and LGD estimation 224 Paragraph 187 of the GL on PD and LGD estimation 225 Article 51(2)(f) of the CDR on assessment methodology, paragraph 188 of the GL on PD and LGD estimation 226 Section 4.2 of the GL on Downturn LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 52 estimates applied immediately before and after the default. 227 This assessment is expected to be performed on the basis of a comparison between the average LGD estimates immediately before and after default at a comparable granular level. This implies that, in the case where post-default information is factored in LGD in-default estimates, this comparison is performed at an intermediate aggregated level; b. The relation between the LGD in-default and the ELBE. As such, the LGD in-default should always be higher than the ELBE, in particular when the ELBE is derived via specific credit risk adjustments 228 and/or via overrides 229 to ensure their consistent application. The validation function is expected to check that the difference between the LGD in-default and the ELBE covers for the effects of a downturn (and as such the validation function is expected to perform the checks described in paragraph [44]), MoC, and potential additional unexpected loss if relevant. 230 Specificities related to the validation of credit risk mitigation
54. General requirements on the appropriate recognition related to the CRM. The validation
function is expected to check that the LGD estimates reflect the collection and recovery policy. 231 Therefore, generally speaking, all general requirements apply to the exposures benefitting from a CRM. As such, the validation function is expected to check the recognition of CRM in own estimates, as part of the assessment of the model performance, and therefore perform the validation activities described previously in this section. However, these are complemented by additional specific requirements to ensure a prudent and consistent recognition of the CRM effects, some of them being performed in cooperation with other internal control functions (see Interaction Box [6]). INTERACTION BOX 6: ASSESSMENT OF THE CRM The validation function is primarily responsible for the review of the modelling aspects of the CRM, i.e. where risk parameters are estimated. On the other hand, the validation function may not necessarily be responsible for the review of the non-modelling aspects of the CRM. Nevertheless, the responsibility of the review for the assessments of the non-modelling aspects of the CRM should be clear to all involved internal control functions. This relates in particular to the implementation of the model and the calculation of own funds requirements, such as:

The application of the regulatory requirements to derive own funds requirements if an
institution has not received the permission of the CA to use own LGD estimates for exposures to corporates, institutions, central governments and central banks pursuant to 227 Paragraph 169 of the GL on PD and LGD estimation 228 Paragraph 191 of the GL on PD and LGD estimation 229 Paragraph 192 of the GL on PD and LGD estimation 230 Article 181(1)(h), paragraph 193 of the GL on PD and LGD estimation 231 Paragraph 105 of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 53
Article 143 of the CRR. This includes cases where the recognition of the funded credit
protection (FCP) or unfunded credit protection (UFCP) involves the use of regulatory parameters;
2. The pure calculation of the RW floor when a UFCP is used; 232
3. The proper identification of the eligible CRM 233 in the application of the model (while the
validation is expected to check the proper treatment of the CRM in the model development);
4. The allocation of CRM related to multiple facilities across different (models or) regulatory
approaches (i.e. between the use of own estimates or regulatory values);
5. The recognition of CRM in a consistent manner,
234 which implies a review of the actual implementation of the model, including the assessment of the integrity of the assignment process (see Interaction Box [4]).
55. Additional requirements on the validation of the RDS related to the CRM: The use and
recognition of CRM come along with specific requirements on the observations themselves. In particular, the validation function is expected to check that:
a. The source of the recovery cash flows is properly identified. In the context of CRM, this requires a comprehensive identification of CRM, regardless of their eligibility. 235 The validation function is expected to ensure that the RDS as well as more recent data used by the validation function contains the information on the eligibility of the CRM for each exposure, as part of the assessment described in paragraphs [32] and [35.b]. As such, the RDS and the validation data should be sufficiently detailed to allow for a specific analysis on the use of CRM as potential risk drivers; 236 b. The allocation of recovery cash flows to specific facilities is consistent with the internal recovery and collection process of the institution. 237 This includes the allocation, sequence and recognition of FCP and UFCP, and more generally the overall traceability of the recoveries as described in paragraph [73];
c. The recoveries from collateral and, in particular, the value of repossession which should
adequately reflect the value of the repossessed collateral, is consistent with any established 232 Articles 161(3) and 164(2) of the CRR, paragraphs 42 to 47 of the Guidelines on Credit Risk Mitigation for institutions applying the IRB Approach with own estimates of LGDs (EBA/GL/2020/05 - GL on CRM) 233 Section 5 of the GL on CRM. This assessment is expected to be bi-directional, i.e. an eligible CRM should meet all the eligibility requirements, while a non-eligible CRM should not meet at least one eligibility requirement. 234 Paragraph 46(c) of the GL on CRM 235 Paragraph 33 of the GL on CRM 236 Paragraphs 105, 109(k) and 121(a) of the GL on PD and LGD estimation 237 Paragraphs 37-38 and 46(a) of the GL on CRM

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 54 internal requirements for collateral management, legal certainty and risk management; in addition, it is expected to check the appropriateness of the estimation of the haircut applied to the repossession value, including its related MoC. 238
56. Validation level for the validation of estimates related to the CRM: Any validation activities
on exposures with CRM are expected to be performed at least at the same level (e.g. obligor or facility level) than validation activities on exposures without CRM. Nevertheless, this does not prevent the validation function from performing additional tests to challenge the modelling choices. For instance, additional statistical tests at the obligor level (or even several obligors, in the case of a specific CRM which benefits to several obligors such as in the case of a mortgage for a household) may be used to challenge the policy of allocation of costs and recovery cash flows mentioned in paragraph [55.b], 239 which is especially relevant in the context of obligors with multiple facilities with different degrees of protection (and hence different levels of risk). The level of validation is expected to also be adjusted in the case of use of a substitution approach, where the guaranteed and non-guaranteed parts have to be validated separately. 240
57. General requirements on the appropriate recognition related to the CRM: Last, the validation
function is expected to check that there is no double counting in the recognition of any CRM in the estimates (i.e. that the effects of the same credit protection are not recognised more than once, 241 for instance both in the exposure value and the risk estimates). 242
58. FCP – On-balance sheet netting and Master netting agreement: The non-recognition of a CRM
is expected to be carefully assessed in the case of master netting agreements or on-balance sheet netting. In this context, the validation function is expected to check that the realised LGD is calculated using the fully-adjusted exposure value as the outstanding amount and that no cash flows from netting should be included as recoveries after default in the economic loss. 243
59. FCP - Validation of recognition of collateral in estimates: In the recognition of the effect of
FCP, the validation function is expected to check that cases with adverse dependency between the risk of the obligor and that of the collateral or collateral provider are identified and dealt with an appropriate level of conservatism as pointed out in the Context Box [2]. In addition, the validation function is expected to challenge the methodology used to estimate LGD in the cases where the collateral is a liability of the obligor, a subordinated position of the institution (in relation to the collateral) or is a movable collateral with likely different geographic locations. 244 238 Paragraph 117 of the GL on PD and LGD estimation 239 Paragraph 112 of the GL on PD and LGD estimation 240 Paragraphs 37 and 38 of the GL on CRM 241 Paragraph 46(b) of the GL on CRM 242 Article 181(1)(g) of the CRR 243 Paragraph 29 of the GL on CRM 244 Article 181(1)(e) of the CRR, paragraph 30 of the GL on CRM

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 55 INTERACTION BOX 7: SPECIFIC ESTIMATION TECHNIQUES FOR FCP The recognition of FCP may involve specific estimation techniques, which, while not being necessarily purely related to the estimation of credit risk per se, is expected to nonetheless be validated by an independent function (i.e., not necessarily the credit risk validation function). This includes the estimation of volatility adjustments under the own estimates approach 245 as well as the use of internal models approach for the master netting agreements. 246
60. UFCP - Choice of the approach: The recognition of the effect of UFCP can be performed
according to different methods. 247 Generally speaking the methods used are expected to be consistent, 248 in particular in the case of multiple credit protection. In addition, the validation function is expected to check that the requirements for the use of a particular approach are met; this includes on the one hand the eligibility, cost and non-defaulted status criteria 249 and back-testing requirements on the expected loss (EL) 250 for the substitution of risk parameters approach (and the non-application of this approach under the slotting approach), 251 and on the other hand the ‘fall-back approach’ criterion for the override approach.
61. Use of multiple CRM – risk parameters: With respect to the validation of risk parameters, the
use of multiple CRM can bring additional modelling challenges, which are expected to be checked by the validation function. This includes the estimation of the LGD of a comparable direct exposure to each of the guarantors along with the effect of other existing unfunded credit protection for the RW floor, 252 as well as the estimation of the LGD of a comparable direct exposure to the guarantor that includes the effect of additional unfunded or funded credit protection in the context of both the RW floor and the substitution of risk parameter approach, 253 where the ‘comparable exposure’ is expected to be effectively challenged.
62. Consistency of the metrics: the validation function is expected to check the consistency
between the back-testing metrics (e.g. realised loss rates), the model design, and the final risk parameters applied for the RWEA calculation, in terms of allocation of collateral to specific exposures. 245 Article 225 of the CRR 246 Article 221 of the CRR 247 Paragraph 31 of the GL on CRM 248 Paragraph 32 of the GL on CRM 249 Paragraph 36 of the GL on CRM 250 Paragraph 40 of the GL on CRM 251 Q&A 3295 252 Paragraphs 45(b) of the GL on CRM 253 Paragraphs 47(b) of the GL on CRM

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 56 Specificities related to the validation of the slotting approach
63. Adaptation of the requirements for the slotting approach. For specialised lending exposures
where no partial use of the standardised approach is used and in respect of which an institution is not able to estimate PDs or the institutions' PD estimates do not meet the regulatory requirements, the regulation prescribes the use of the so-called ‘slotting approach’. 254 As such, this approach does not rely on the usual risk parameters (i.e. PD and LGD) and is more prescriptive compared to the usual approaches (i.e. via the CDR on the slotting approach instead of the usual risk differentiation and via the prescribed RWs and ELs instead of the usual risk quantification). Therefore, these differences entail some adaptations of the validation tasks and related best practices, in particular when it comes to the performance assessment.
64. Assessment of the assignment process. As part of the assessment of the rating assignment
process described in paragraph [33], the validation function is expected to assess the consistency and comprehensiveness of the rating assignment process. In particular, the consistency of the assignment process is expected to be assessed both in terms of assignment of:
a. Corporate exposures to the sub-exposure class ‘specialised lending exposures’ (if not covered by other internal control functions) and to a class of specialised lending exposures (‘project finance’, ‘real estate’, ‘object financing’, ‘commodities financing). 255 Where the assignment to the specialised lending sub-exposure class is not assessed by other internal control functions, the validation function is expected to ensure that only exposures assigned to corporates following the assigning sequence 256 are classified as specialised lending exposures, and that the process for identifying specialised lending exposures 257 allows for a consistent and replicable identification. In all cases, the validation function is expected to check that the assignment of exposures to a class of specialised lending exposures is also implemented in a consistent and replicable manner; b. Specialised lending exposures into a particular slotting approach (i.e. rating system 258), such that all exposures covered by one approach have a credit risk assessment in a similar manner.
65. Assessment of the input data. As part of the assessment of the input data as described in
paragraph [35], a good practice is to assess the representativeness separately between exposures with a maturity shorter than 2.5 years from those with a longer maturity. In addition, the validation function is expected to check the evolution of the distribution of the features 254 Articles 153(5) and 158(6) of the CRR 255 Article 1 of the CDR on slotting approach 256 Article 61 of the CDR on assessment methodology 257 Article 147(8) of the CRR 258 Recital 1 of the CDR on slotting approach

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 57 (i.e. sub factors and sub-factor components) that receive a high or low weight in the aggregation methodology mentioned in paragraph [67]. 259
66. Assessment of the modelling choices – selection of relevant information and rating criteria.
The validation function is expected to review the selection of relevant information and rating criteria as per paragraph [36.a], including that the relevant information and rating criteria considered are consistent with the factors, sub-factors and sub-factor components mentioned in the regulation. 260 In particular, the validation function is expected to challenge any deviation from the definition of the sub-factor components as defined in the Commission Delegated Regulation (EU) 2021/598 (CDR on slotting approach), the inclusion of additional relevant information into a selected sub factor component 261 and the dismissal of any sub factors or subfactor components for a type of specialised lending exposures. 262
67. Assessment of the modelling choices – aggregation of relevant information. As part of the
assessment of the aggregation methodology described in paragraph [36.b], the validation function is expected to review:
a. The allocation of each sub-factor component into a category. 263 This includes an assessment of the cut-off values used when quantitative metrics (e.g. financial ratios) are used; b. The determination of the category for each sub-factor, in particular when based on the aggregation of the relevant sub-factor components’ category; 264
c. The determination of the category for each factor, i.e. the aggregation methodology
of the relevant sub-factors’ category; 265 d. The determination of the final category for the exposure, i.e. the weights used to aggregate the relevant factors’ category. 266
68. Assessment of the modelling choices – definition of obligor and facility grades. As mentioned
in paragraph [36.c], the minimum number of grades should match the number of categories prescribed in the CRR. 267 259 For instance, the validation function may have to check the evolution of the proportion of balloon payments to assess whether the weight associated with the sub-factor component ‘amortisation schedule’ is still appropriate. 260 Article 153(5) of the CRR, annexes of the CDR on slotting approach 261 Articles 3(3) and 6(1)(b) of the CDR on slotting approach 262 Articles 3(4) and 6(1)(c) of the CDR on slotting approach 263 Article 3(2)(a) of the CDR on slotting approach 264 Articles 3(1), (2)b and 4 of the CDR on slotting approach 265 Article 2(1) of the CDR on slotting approach 266 Article 2(2), (3), (4) of the CDR on slotting approach 267 Article 170(2) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 58
69. Quantitative analyses. As for any other IRB model, the predictive power of the slotting
approach is expected to be assessed by the validation function. However, the empirical assessment mentioned in paragraph [37] requires some adjustment, as the slotting approach does not require a rating assignment to have an obligor rating scale which reflects exclusively the default risk of an obligor. Hence the dimensions mentioned in paragraph [34] may not be fully appropriate. Instead, these analyses may be conducted via specific challenger analyses (see Focus Box [5]).
70. Validation challengers. As part of the challenger analyses mentioned in paragraph [38], the
validation function is expected to perform the following:
a. Challenge the use of overrides other than ‘input overrides’ as described in paragraph [38.a] and [38.b], in order to assess whether the use of overrides is not an indication of disregarding relevant information in the assessment of a sub-factor that could have been considered jointly with a sub-factor to which it most closely corresponds (via an ‘input override’); 268 b. Use other external data sources, where available;
c. Assess the monotonicity of the observed loss rates per slotting category;
d. Assess the concentration of exposures per slotting category by type of specialised lending exposures which if unwarranted could be an indication of missing risk drivers. FOCUS BOX 6: SPECIFIC CHALLENGER ANALYSES FOR THE MODELLING CHOICES UNDER THE SLOTTING APPROACH When assessing the aggregation methodology under the slotting approach, the following possibilities have been observed:

For the allocation of each sub-factor component into a category, a possible analysis is to
challenge the human judgment (see Interaction Box [4]) and estimation methodology used in the case where estimation projections are required (e.g. analysis of the cash flow projections). In particular, this can entail some back-testing analyses with longer time horizon than the usual one-year time horizon considered for other metrics, since the cash flow projections can be performed for the full lifetime of the exposure (e.g. back-testing of the estimated value of an aircraft’s engine over n years). In addition, a good practice observed in institutions is to challenge cases where the same rating assignment process is used across different classes of specialised lending exposures (for instance leveraging on the challenger analysis of overrides as mentioned in paragraph [38.b]) or when 268 Recital 8 and Article 3(3) of the CDR on slotting approach. ‘Input overrides’ are required to take into account each of any additional risk drivers and consider it jointly with the sub-factor of the specialised lending exposure framework which most closely corresponds to the risk driver.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 59 different rating assignment processes are used within the same class of exposure of specialised lending exposures;
2. For the aggregation of the sub-factor components’ category and sub-factors’ category,
which is less prescriptive than the aggregation of the factors’ category (weighted average), possible analyses include an assessment of:
a. The sensitivity of the aggregation to the model components (e.g. weights if a linear aggregation is used), by looking at the magnitude of the migrations between the categories due to a small change in the model components used; b. Whether each sub-factor and sub-factor component has an influence in the aggregation. 269
3. For the aggregation of the factor category, which is constrained by the CDR on slotting
approach (weighted average), a possible analysis is to assess the sensitivity of the aggregation to weights used by looking at the magnitude of the migrations between the categories due to a small change in the weights used.
4. For the challenger analysis, the monotonicity analysis on loss rates can be complemented
by a monotonicity analysis on default and delinquency rates. This analysis can also be performed at sub-factor and sub-factor components level to challenge their categorisation, where it is not predetermined by the CDR on slotting approach.
5. For the challenger analysis, the validation function can complement the concentration
analysis with an assessment of the concentration at the sub-factor and / or sub-factor component level, where their categorisation is not predetermined by the CDR on slotting approach, as part of the analysis mentioned in paragraph [65]”). When assessing the performance of the model in terms of final rating assignments, while no risk quantification is required under the slotting approach, it is considered as best practice to assess the general consistency between the (number and exposure-weighted) average losses observed on the exposures assigned to a particular category and the related regulatory parameters 270 as other quantitative validation tools. 271 Such analysis is expected to be generally consistent with the back-testing analysis performed for the LGD mentioned in paragraph [46.c], however taking into account also non-defaulted exposures using a 0% loss rate and with some different level of calculation. In addition, this analysis could be complemented via challengers derived from the 269 In other words, for any sub-factor and sub-factor component, there exists at least one particular configuration (a particular allocation of categories to other sub-factors or sub-factor components respectively) where two different categories for the sub-factor or sub-factor component considered lead to two different categories for the factor or subfactor respectively. 270 Article 158(6) of the CRR 271 Article 185(c) of the CRR

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 60 observed default rates; the results of these tests could then be analysed taking into account the realised loss on these defaults. More generally, in the context of specialised lending exposures, more emphasis on the consistency between the modelling choices and the transactions characteristics and business expectations may be appropriate. In addition, where data is scarce, the assessment of the slotting approach is also expected to consider section [6.3 Focus 3: validation in the context of data scarcity] in order to deal with data scarcity issues.
3.2 Assessment of the modelling environment
71. Scope of this section. As noted in the introduction, the validation function is not only
responsible for analysing the model performance from a pure statistical perspective, but is also expected to assess the development and production environment. 272 As such, the validation function is expected to be involved in the data quality assessment (before the core performance assessment) and check the proper IT implementation of the rating system (after the core performance assessment).
3.2.1 Data quality and maintenance
72. The data quality framework. To ensure a proper assessment of the data quality and
maintenance,
273 the data quality framework (see Context box [6]) should define clearly the policies, roles and responsibilities in data processing and data quality management. CONTEXT BOX 6: THE DATA QUALITY FRAMEWORK AND ITS APPLICATION IN THE IRB FRAMEWORK Institutions should ensure that information technology and data governance are adequate to support the broad management of financial risks, including bank-wide data aggregation, consistent data quality, implementation and monitoring of the IRB rating systems. This has been developed in the BCBS principles, 274where it is specified that institutions should have in place a strong data governance framework based on sound IT infrastructure. In particular, in relation to the governance, the Board and senior management should review and approve the group risk data aggregation and risk reporting framework and ensure that adequate resources are deployed. Senior management should be fully aware of and understand the limitations that prevent full risk data aggregation, in terms of coverage (e.g. risks not captured or subsidiaries not included), in technical terms (e.g. model performance indicators or degree of 272 Article 11(2)(a) and (b) of the CDR on assessment methodology 273 Chapter 12 of the CDR on assessment methodology 274 BCBS 239: Principles for effective risk data aggregation and risk reporting (bis.org).

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 61 reliance on manual processes) or in legal terms (legal impediments to data sharing across jurisdictions). The institution’s data quality framework should cover all relevant data quality dimensions, 275 Including completeness, accuracy, consistency, timeliness, uniqueness, validity, and traceability, and should cover the full data life cycle, from data entry to reporting, and should encompass both historical data and current application databases. Therefore, to ensure the quality of the data used for credit risk measurement and management processes, institutions should establish and implement an effective data quality management framework that is formalised in a set of policies and procedures. When it comes to the application of this data quality framework in the modelling, each institution should have in place a process for vetting data inputs into a model, which includes an assessment of the accuracy, completeness and appropriateness of the data. 276 This framework should be applicable to all data used in IRB-related processes, i.e. internal data, external data and pooled data, 277 if any. It should include the assessment of the RDS regarding input data, in particular characteristics and risk drivers, but also the data for the application of the model as well as the output data like the risk parameter estimations and the RWs as discussed in the next section. Hereby not only particular elements should be considered, but also their interactions such as the relationship or mapping used to produce IRB parameter estimates (e.g. consistency between PD and LGD in terms of default recording). INTERACTION BOX 8: ASSESSMENT OF THE DATA QUALITY BETWEEN THE DIFFERENT FUNCTIONS Although institution’s IT functions are ultimately responsible for supporting the operation of the systems for data collection, processing, transformation and storage during the entire life of the data, the validation function is expected to have sufficient knowledge of the IT infrastructure of the institution and of all data quality checks performed with respect to IRB rating systems and their inputs, outputs and the calculation of the own funds requirements. In practice, it is important to make the distinction between the tasks expected to be performed by the validation function and the ones expected to be performed by other relevant functions of the institution, such as the IA or the IT functions. 275 Article 73 of the CDR on assessment methodology. 276 Article 174(b) of the CRR. The appropriateness is however not a data quality dimension per se and is therefore rather checked as part of other assessments (e.g. assessment of the representativeness in paragraph [35.d]). 277 Article 72(1)(a) of the CDR on assessment methodology.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 62 As such, the data quality in the IRB rating systems is usually assessed by different functions with a different perspective:
1. The institution’s independent data quality function, which is expected to be a second line
of defence and to ensure the quality of the data itself that flows between the different systems and databases supporting the IRB modelling and application (including reporting), as defined in the data quality framework;
2. The CRCU, which is expected to assess the data quality of the (historical) data for the
estimation of the risk parameters (and in particular for the quantification of any potential appropriate adjustment applied to the risk estimates and/or the MoC);
3. The validation function, which is expected to check the quality of two datasets:
a. The data it uses for the validation activities, such that it can be used effectively to perform analyses on the rating system developed by the CRCU; b. The data used by the CRCU (and potentially manipulated) for the estimation of the risk parameters. This includes the data extracted by the CRCU from a database, where the validation function is expected to detect errors in the data, as this extraction is part of the full modelling process. 278 In this context, the validation function is expected to compare the outcome of its data quality assessment with the one performed by the CRCU in point [2] as part of its challenge of the rating system developed by the CRCU (and in particular the quantification of any appropriate adjustment applied to the risk estimates and the MoC);
4. The IA, which is expected to be the third line of defence, particularly with regard to IT and
data quality review. 279
73. Dimensions assessed by the validation function for data quality. The validation function is
expected to form an opinion on the data quality dimensions (completeness, accuracy, consistency, timeliness, uniqueness, validity and traceability) and data quality processes relevant for the IRB modelling. 280 The outcome of this assessment is expected to be an opinion from the validation function on the data quality used for IRB modelling. In case the validation function detects any errors in this data, the identified deficiencies are expected to be reported and addressed in the validation report as mentioned in paragraph [24]. 281 As mentioned in the Interaction Box [8], the validation function is expected to check whether the identified data 278 Paragraph 8 of the GL on PD and LGD estimation 279 See The internal audit function in banks (BCBS, June 2012). 280 Article 73 of the CDR on assessment methodology 281 Article 189(2)2 of the CRR.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 63 quality deficiencies impact the estimation of risk parameters and lead to a bias in the quantification of those parameters or to an increased uncertainty to be addressed by a MoC.
74. Tools used to assess the data quality. In order to form an opinion on the quality of data used
by the rating system (e.g. the IRB relevant data do not have material quality errors), the validation function is expected to check that all the relevant IRB data (including internal, external or pooled data) used in model development and risk quantification, is encompassed by the institution’s data quality framework. In order to perform this check, the validation function is expected to have access to the relevant data quality management reports submitted to the institution’s senior management. 282 In addition, the validation function is expected to also assess how frequently the conservatism referred to in section 8.1 of the GL on PD and LGD estimation is being used, as this might be an indication of data-quality-related issues.
75. Accessibility of the data for the data maintenance. The validation function is expected to have
access of all the relevant IRB data and their respective databasesindependently from the CRCU, as well as sufficient data extraction and manipulation capabilities. 283 To this end, the IT infrastructure that supports the data used for IRB-modelling purposes and the relevant data sources and flows should be appropriately documented and understandable by a third party, 284 to ensure that the validation function is in a position to conduct an effective and independent challenging of the model development and use. Therefore, the validation function is expected to assess:
a. The completeness and readability of the global map of databases involved in the calculation systems used for the purposes of the IRB Approach, of the documentation supporting the relevant processes for data extraction and transformation, and the specifications of the modelling databases; 285 b. The data flow diagram showing a map of the key applications, databases and IT components involved in the application of the IRB Approach and related to rating systems. 286
3.2.2 IT implementation of the rating systems
76. Adequacy of the implementation. The validation function is expected to verify the adequacy
of the implementation of internal ratings and risk parameters in IT systems. 287 To this end, the validation function is at least (see Interaction Box [9]) expected to:
282 As defined in Article 74(2) of the CDR on assessment methodology.
283 To ensure that the validation function is in a position to conduct an effective and independent challenging of the model development and use. To this end, the data should be available, as referred to in Article 42(1)(b) of the CDR on assessment methodology. 284 Article 75 of the CDR on assessment methodology 285 Article 74(1)(a) of the CDR on assessment methodology 286 Article 75(1)(b) of the CDR on assessment methodology 287 Article 11(2)(b) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 64 a. Analyse the relevant functional documentation, 288 for example, by checking if the functional documentation of the IT systems (e.g. business specifications) supporting the rating system is consistent with the rating system documentation. However, the adequacy of the translation of business/functional requirements defined for the systems supporting the IRB modelling and application into IT specifications can be performed by another independent function, clearly identified in internal policies as mentioned in paragraph [22.b]; b. Ensure that the implementation of the rating system in the relevant IT systems is compliant with and reproduces exactly the documented rating system under review. 289 To this end, the validation function is expected to check whether the necessary user acceptance tests (UATs) relating to the rating system have been performed and concluded with no material errors. For those tests, a good practice observed in institutions is to set up ‘walk-through’ sessions between the validation function and the IT department for a better understanding of the IT documentation related to the testing of the rating system implementation. INTERACTION BOX 9: IT IMPLEMENTATION OF THE RATING SYSTEM The validation function may not be expected to assess the correct implementation of the rating system and calculation of own funds requirements beyond the elements mentioned above (including, for instance, the assignment of the exposures to the proper exposure class). Instead, this can be performed by the IA, which can leverage on the methods described in Article 67(4), 68 and 71 of the CDR on assessment methodology. FOCUS BOX 7: IT IMPLEMENTATION AND ASSIGNMENT OF PARAMETER ESTIMATES Different forms of IT implementation approaches have been observed, especially when it comes to the assignment of the parameter estimates. The two most prominent are:

The model engine not only assigns exposures to grades or pools, but also assigns the
parameter estimate. In other words, the IT system in which the rating assignment process has been implemented also adds the information about the parameter estimate (PD, LGD, CF, ELBE, RW, EL) to the respective obligor/facility/exposure. E.g. model engine output: “Obligor [XYZ]”; “Grade [ABC]”; “PD [NN]” 288 Article 67(4)(a) of the CDR on assessment methodology 289 Article 31(8)(b) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 65
2. The model engine assigns the exposures to grades or pools, and the parameter estimates
are added at a later stage, often in the RWEA calculation engine, through a steering
table.
E.g. model engine output: “Obligor [XYZ]”; “Rating System [UVW]”; “Grade [ABC]” => steering table: “Rating System [UVW]” + “Grade [ABC]” = “PD [NN]” The first implementation approach has been observed stand-alone or as a sub-system of the RWEA calculation engine. When validating the IT implementation, the validation function is expected to verify not only that the correct rating assignment process has been implemented, but also that the correct parameter estimates are assigned to each grade or pool. Therefore, under the second implementation mentioned, besides the model engine, also the mapping table assigning the parameter estimate is expected to be reviewed.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 66
4. First validation activities
4.1 General requirements
77. Scope of application of this section. This section deals with the ‘first validation’, i.e. the
validation of new rating systems without any previous authorisation from CA pursuant to Article 143 of the CRR (‘newly introduced models’). Moreover, it interacts with section [5 On-going validation ] in the case of changed rating systems, as the validation function faces a similar situation as in the first validation, it is expected to perform the activities described in this
section at least for the aspects that have been directly or indirectly affected by the rating
system change..
FOCUS BOX 8: SAMPLES USED DURING THE VALIDATION OF NEW RATING SYSTEMS OR OF CHANGED ASPECTS OF A RATING SYSTEM Two types of samples are relevant in the context of the validation of new or of changed aspects of rating systems:
1. The ‘performance metric samples’, whereby the obligors or facilities are assigned to
grades or pools and the empirical metrics (e.g. defaults) can be observed to assess the core model performance, to the extent possible (e.g. in sample). This can refer, for example, to the development sample(s) or the calibration sample(s) used by the CRCU;
2. The ‘application test samples’, whereby the obligors or facilities are assigned to grades
or pools, but there has not been enough time to observe the empirical realisations to assess the model. This can refer for example to the application portfolio. The requirements in the context of the first validation may differ due to different situations with respect to the application of the model. As a matter of fact, the scope of exposures available for the first validation with ratings assigned to grades or pools according to the full rating system under validation (‘application test sample’) is expected to be different:
1. In the context of the very first validation of a new rating system, an institution shall have
actually been using a rating system broadly in line with the requirements of the IRB Approach for at least three years prior to its application to use the IRB Approach. 290 Therefore, it is expected that at the time of first validation, the vast majority of exposures under the scope of the rating system has been assigned to grades or pools according to the ‘full’ rating process under application (i.e. including with the human judgement 290 Article 145 of the CRR, Article 22 of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 67 applied in the process of assignment of exposures to grades or pools, or in the form of overrides);
2. In the context of a validation of rating system changes, it is expected that the institution
falls within one of the following two situations:
a. All the exposures of the relevant range of application are re-rated according to the new rating system. This is in particular expected if the rating is fully automatic, with no human judgement applied in the process of assignment of exposures to grades or pools, or in the form of overrides; b. A representative sample of exposures 291 is re-rated according to the ‘full’ rating process (i.e. including with the human judgement applied in the process of assignment of exposures to grades or pools, or in the form of overrides).
78. Objective and specificities of the first validation. One of the most important topics to be
addressed in the first validation are the methodological choices related to the model design and risk quantification. This is because, firstly modelling and risk quantification decisions are assessed by the validation function for the first time, and secondly because only few new data will be available and thus standard statistical measures (e.g. for discriminatory power) will in most cases not provide much new information compared to what was performed by the CRCU. For these reasons, a focus point for the validation function during the first validation is the assessment and challenging of the modelling and calibration choices, including the modelling assumptions and model limitations. Unless specified otherwise, all the dimensions of the validation are expected to be performed during the first validation in a comprehensive and independent manner (see Interaction Box [10]). The rest of this section further describes some specificities of this assessment related to the elements covered in section [3 Validation content]. In addition, in case of a model change, the validation function is expected to compare the performance of the new models with the previous ones in order to assess whether the changes have led to a documented improvement of the performance and consistent implementation. The validation function is also expected to assess, in the case of a change in the risk differentiation, the appropriateness of the risk estimates, in particular if no new recalibration has been performed and the previously estimated risk parameters are intended to be used. INTERACTION BOX 10: INTERACTION BETWEEN THE CRCU AND THE VALIDATION FUNCTION DURING A FIRST VALIDATION While the validation function should be informed about the analyses made and conclusions reached by the CRCU, the validation function should always analyse the results independently 291 Article 3(2)(b) of the CRD on model changes. The exact definition of this sample can be determined on a case-by-case basis, taking into account the degree of expert judgments involved in the rating and the minimum number of exposures necessary to form an opinion on the compliance with regulatory requirements.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 68 and come to its own independent conclusions. In contrast to on-going validations, where it can leverage to some extent on the analyses performed by the CRCU, in the case of the first validation, the validation function is expected to challenge and complement the CRCU analyses. This entails performing own data extraction and analyses using as much as possible new data available since the model development and risk quantification, and challenge the data extractions performed by the CRCU. In any case, the validation function is expected to perform additional tests whenever it considers that the actions performed by the CRCU do not cover all angles of all the potential deficiencies. 292
79. The first validation in connection with the CA’s approval of a rating system. Any rating systems
submitted for an approval from the CA should have been properly validated before the rating system is used for the calculation of own funds requirements and for internal risk management purposes. 293 In this context, it is expected that sufficient time has been dedicated for an identification and evaluation of any potential deficiencies, 294 such that all necessary changes have been performed before the submission for an approval to the CA. 295 In practical terms, it is expected that a rating system submitted for an approval from CAs is not assessed by the validation function as materially incompliant with the regulation (i.e. no material deficiencies nor material number of non-material deficiencies are yet to have been rectified). In addition, in the context of extensions and changes to the IRB Approach, the validation should review the materiality classification performed by the CRCU mentioned in paragraph [27]. In the case of extensions and changes classified as requiring CAs' approval or notification, it is expected that the validation function reviews the documentation that is going to be submitted to the CA, as mentioned in interaction box [2]. 296
4.2 Specificities of the first validation regarding the core model
performance
80. First validation of the IRB metrics. The validation function is expected to check the correct
calculation of the IRB metrics according to the regulatory requirements as mentioned in paragraph [30] for both newly introduced rating systems as well as rating system changes.
4.2.1 Risk differentiation
81. First validation of the rating assignment process. With respect to the evaluation of the rating
assignment process, the validation function is expected to perform the analyses described in paragraph [33]:
292 The validation function may request the CRCU instead to perform these additional tests in order to complete its analysis. 293 Article 11(4) of the CDR on assessment methodology 294 Article 12(c) of the CDR on assessment methodology 295 Article 144(1)(f) of the CRR 296 Article 8 of the CDR on model changes in conjunction with Article 11(2)(d) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 69 a. The evaluation of the documentation referred to in point [33.a] is expected to be focused on the assessment of whether the documentation allows for a consistent application of the human judgement in the rating assignment; b. The evaluation of the materiality referred to in point [33.b] is expected to be performed in the context of a model change on the ‘application test sample’, leaving aside the analysis of outdated ratings. 297 In the context of the first validation of a new rating system, the evaluation is expected on all dimensions and using the most recent application portfolio.
82. First validation of the input data. With respect to the assessment of the input data, all the
dimensions are expected to be assessed. In particular, the analysis of the representativeness described in paragraph [35.d] is expected to be conducted between the development sample and the current application portfolio at the time of the first validation and assessing also the activities performed by the CRCU as part of the model (re)development.
83. First validation of the modelling choices. With respect to the assessment of the modelling
choices and specifications described in paragraph [36], which is of particular importance in the first validation, the validation function is expected to assess all the elements mentioned in that paragraph. In particular, for the elements mentioned in point a, the assessment of the use of third-party ratings and the cases of obligors’ assignment to a better grade than their parent entities are expected to be assessed on the most recent application sample or, in the context of a model change, on the ‘application test sample’;
84. Quantitative analyses in the first validation. With respect to the empirical assessment of the
model performance described in paragraph [37], the validation function is expected to pay particular attention to the robustness of the model. To this end, during the first validation, the validation function is expected to:
a. Assess if due measures were taken in the model development to avoid overfitting and to ensure that the model is able to perform also under a changing economic environment. In particular, during the first validation, the validation function is expected to check that OOT and OOS-testing was performed as part of the model development; b. Perform additional tests to have an independent view on the performance of the model, as described in the Interaction Box [11]. In particular, where a sufficient amount of more recent data as in model development is available (in the case where a significant period of time has elapsed between the cut-off date used for model development and the validation date), the validation function is expected to perform at least an OOT-validation using that data. 297 Article 25(2)(b) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 70 INTERACTION BOX 11: STATISTICAL ASSESSMENT OF THE MODEL BETWEEN CRCU AND VALIDATION FUNCTION DURING THE FIRST VALIDATION A good practice observed in institutions is to ensure a validation based on OOS and OOT samples which have not been used for the model development (i.e. for the model design, but the CRCU may have used the OOS and OOT samples itself for the validation of the final model developed) by the CRCU, to ensure a complete independence of the assessment of the model (see Figure 3). It is however not expected that the validation is restricted to these OOS and OOT samples, i.e. additional validation tests based on the full data set can complement the analysis. In any case, the validation function should have at its disposal the empirical assessment conducted by the CRCU of the performance of the model on OOT and OOS samples. However, the model validation based on OOT and OOS samples may trigger specific challenges in the context of data scarcity, which are further discussed in section [6.3 Focus 3: validation in the context of data scarcity].
Figure 3: Schematic view of best practice of uses of validation samples between the CRCU and the validation function
85. Validation challengers in the first validation and specific aspects for validation in case of
rating system changes. For the purpose of challenging the accuracy of the rating assignment, the validation function is generally expected to perform all the analyses described in paragraph [38]. In particular, the following considerations can be taken into account:

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 71 a. The analysis mentioned in points [38.a] is not expected to be performed in the case of a material rating system change in case the historical overrides cannot be considered as meaningful for the new rating system; b. The analysis mentioned in point [38.b] can be done on the most recent application sample or, in the context of a model change, on the ‘application test sample’;
c. The analysis mentioned in point [38.c] can be performed using the backward
simulated ratings where possible in case of model changes; d. The analysis mentioned in point [38.d] is expected to be performed on the RDS used for risk quantification; e. (The analyses mentioned in point [38.e] are expected to be performed;) f. The analyses mentioned in point [38.f] are expected to be performed on the RDS used for risk quantification and on the application portfolio;
4.2.2 Risk quantification
86. First validation of the risk quantification. With respect to the assessment of the risk
quantification, all the analyses described are expected to be performed. To this end, during the first validation, the validation function is expected to:
a. Assess the input data as well as any methodology choices implemented by the CRCU, as mentioned in paragraphs [40], [41], [42], [43] and [44]; b. Perform additional tests to have an independent view on the risk quantification, as mentioned in paragraphs [46] and [47]. In particular, where a sufficient amount of more recent data as used in the model development is available (in the case where a significant period of time has elapsed between the cut-off date used for risk quantification and the validation date), the validation function is expected to perform the quantitative tests taking also into account that data;
c. (The analyses mentioned in paragraph [48] are expected to be performed;)
4.2.3 Other specific points
87. First validation of other specific points. As mentioned in paragraph [49], this section describes
the specificities of three aspects, which are complementing the expectations described in the general section. As such, the usual first validation activities described in the previous section are expected to be performed.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 72 Specificities related to the validation of defaulted exposures’ risk parameters
88. First validation of the defaulted exposures’ risk parameters. With respect to the assessment
of the risk parameters for defaulted exposures, all the analysis described in section [3.1.3] are expected to be performed. Specificities related to the validation of credit risk mitigation
89. First validation of the CRM. With respect to the assessment of the CRM, all the analyses
described in section [3.1.3] are expected to be performed.
Specificities related to the validation of the slotting approach
90. First validation of the slotting approach. With respect to the assessment of the slotting
approach, all the analyses described in section [3.1.3] are expected to be performed.
4.3 Specificities of the first validation regarding the modelling
environment
4.3.1 Data quality and maintenance
91. First validation of the data quality and maintenance. With respect to the assessment of the
quality of data, as mentioned in section [3.2.1 Data quality and maintenance], the validation function is expected to conduct its analysis along two dimensions:
a. On the RDS for the modelling development, as mentioned in paragraphs [35.a] and [40.a]; b. For the application of the model, in conjunction with the analyses described in section [4.3.2], however, taking into account that these processes might not be fully implemented in the production environment at the time of first validation to the extent they substitute currently implemented processes.
4.3.2 IT implementation of the rating systems
92. First validation of the adequacy of the implementation. For the purposes of first validation,
points [76.a] and [76.b] are expected to be applied, to ensure that the model in production reproduces the business and functional requirements defined by the new or changed model. Point b is expected to be applied with regard to the development environment 298 of the new model. This assessment is expected to be performed on the basis of the specific test-related documentation, such as a general policy for IT implementation testing, individual user testing procedures, model change requests, list of detected anomalies and error logs. 298 In some cases, this may include the test environment.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 73
93. Documentation for the first validation of the IT implementation. On this last aspect, when
submitting a material rating system change, the rating system under validation is typically in a development environment and has not been set into a production environment. Nevertheless, the documentation package submitted to the validation function for the first validation is expected to include sufficient technical documentation, especially the IT functional documentation (e.g. business specifications), allowing the validation function to form an opinion on the integrity of the rating system implemented in the development environment and to be implemented in the production environment.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 74
5. On-going validation activities
5.1 Scope of application
94. Scope of application of this section. This section deals with the on-going validation activities,
as further described in paragraphs [97] to [100]. As a result of the aim of the on-going validation, there is also a regular interaction between this section and section [4 First validation activities], as the more in-depth analyses described there should be used and leveraged on whenever warranted. CONTEXT BOX 7: THE REVIEW OF ESTIMATES As part of the on-going model use, institutions are required to perform on a regular basis a review of their estimates. 299 The Chapter 9 of the GL on PD and LGD estimation has further clarified the two types of review that are supposed to be performed:
1. A ‘regular’ review of estimates, which should be conducted at least on a yearly basis,
which is linked to the requirements in Article 179(1)(c) of the CRR and are further clarified in Article 43 of the CDR on assessment methodology and paragraphs 217 and 218 of the GL on PD and LGD estimation.
2. A ‘full’ review of estimates, with a frequency that should be adjusted depending on the
materiality of the rating system, and which is linked to the requirements in Article 190(2) of the CRR and are further clarified in paragraphs 219 and 220 of the GL on PD and LGD estimation. A good practice observed in institutions is to conduct this full review of estimate not less frequently than every three years for material rating systems, such that this assessment can be used by the CA when performing its regular review to assess whether the rating system is based on well-developed and up-to-date techniques and practices. 300 The interaction between the CRCU and the validation function on the review of the estimates is further discussed in the Interaction Box [12].
95. Specificities of the on-going validation. On-going validation activities differ from the activities
performed as part of the first validation in two ways:
299 Articles 179(1)(c) and 190(2) of the CRR
300 Articles 101(1) and (2) of the CRD

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 75 a. They benefit from additional data and observations, which can be used to further challenge the model; 301 b. They benefit from the conclusions from previous validation activities (including the first validation) and other additional sources. More generally, during the on-going validation activities, the validation function is expected to be aware of any deficiencies identified by other functions, such as the CRCU and the IA.
96. Objectives of the on-going validation. As mentioned in Focus Box [1], the on-going validation
aims at ensuring the adequate model performance and appropriateness of the model for IRB purposes on an on-going basis. In particular, the objectives of the subsequent validation are threefold:
a. On the empirical side, the validation function should form an opinion on the performance of the model over time. For this assessment, the validation function should use in particular the new data available along with its previous assessments and conclusions. In particular, as mentioned in the Focus Box [2], the validation is expected to compare the results using the latest data available, with the ones observed in the previous time periods. Especially, the reference dates used for any statistical tests are expected to be consistent from one period to the other to allow for a robust comparison of the results over time. In case where the validation is based on a different period in time, such that the new data available do not cover a full year (i.e. where on-going validation activities are performed more frequently than on a yearly basis or on an ad-hoc basis), a good practice observed in institutions consists in comparing the validation results with the outcome of the analysis performed on the previous year with a consistent reference date; b. On the methodological side, the validation function is expected to assess the identified deficiencies over time, i.e. to verify that all planned changes have been implemented 302 and that any negative evolution of the deficiencies detected does not lead to a rating system being materially incompliant with the regulatory requirements. As further detailed below, unless specified otherwise, for the methodological assessments the validation function may rely on its previous assessments;
c. In case of a rating system change, as mentioned in paragraphs [27] and [77], the
validation function should check the materiality of rating system changes that occurred since its last review. Additionally, the related validation activities described in the section [4 First validation activities] are expected to be performed at least for those aspects that have been directly or indirectly affected by the rating system change, however noting that for non-material changes to a rating system the review 301 To note: the same principle also applies to the CRCU during its review of estimates. 302 Articles 23(2)(f), 30(2)(f), 41(2)(f), 67(3)(g), 72(2)(f), 74(2)(a) and 75(4)(a) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 76 of the changes could be performed during the regular (yearly) validation activities, and the result of the validation function’s assessment would be communicated to the CA via the usual validation report;
97. Types of activities performed during the on-going validation. Two types of on-going validation
activities can be distinguished:
a. Regular validation activities, either in the context of a yearly exercise as further elaborated in paragraph [98], or performed in the context of a full validation as further elaborated in paragraph [100]; b. Ad hoc validation activities as further elaborated in paragraph [99].
98. Regular validation activities. In case where the estimation of risk parameters is unchanged
compared to the first validation (i.e. neither material nor non-material changes to the rating system have occurred), the on-going validation can be focused on periodical validation activities and analyses of model performance by running more quantitative tests and/or validation challengers. Such activities typically include tasks where the basic computation can frequently be highly automated. Sub-sections [5.2] and [5.3] provide further details on these periodical activities. 303 As mentioned in paragraph [23], these activities are expected to be documented in the validation policy. For such analyses, the validation function may rely in some circumstances on the CRCU analyses (see Interaction Box [12]), as well as on the result of the assessment conducted in a previous exercise. It is nonetheless expected that the validation report produced during the regular validation is understandable by a third party and communicates the key messages in an efficient manner, as mentioned in paragraph [24]. INTERACTION BOX 12: INTERACTION BETWEEN THE CRCU AND THE VALIDATION FUNCTION DURING REGULAR VALIDATION ACTIVITIES As introduced in Context Box [6], some of the tasks performed by the CRCU and the validation function are typically very similar and some analyses might even be identical to validation. As a matter of fact, in their regular reviews of estimates, the institution’s CRCU may take into account the analyses of the independent validation function where such results are up to date. 304 When it comes to the validation activities and their interaction with the analyses performed in a recent regular review of estimates (i.e. the review conducted on a yearly basis by the CRCU), the validation may rely to some extent on the analysis performed by the CRCU (if the regular review 303 NB: these sections provide ‘minimum tasks’ to be performed in the context of a yearly exercise. As such, this does not prevent institutions from defining a validation framework with more tasks to be performed during this exercise (and in particular, the tasks provided in focus box [9] should be considered as indicative). 304 NB: on the contrary, the CRCU is expected to perform the full review of estimate independently from the validation function.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 77 is performed by the CRCU in the first place), depending on the materiality of the rating system and the validation activities. As such, two degrees of leverage are possible:
1. Leverage on CRCU analyses with independent conclusions. In these specific cases, the
validation function can review and challenge the analyses performed by the CRCU but, in contrast to the general expectations during the first validation, may decide not to perform additional tests if the CRCU tests are deemed adequate in terms of input data, specifications and implementation (i.e. they are void of operational errors). However, as for the first validation, the validation function is expected to analyse the results independently, come to its own independent conclusions and perform additional tests whenever it considers that the actions performed by the CRCU do not cover all angles of all the potential deficiencies;
2. CRCU analyses complemented by own analyses. In the other cases, the interaction
between the CRCU and the validation function is expected to be similar to the one described in the first validation, whereby the validation function is expected to complement the analyses performed by the CRCU with its own additional analysis. As such, the validation function can leverage on the CRCU activities along the following lines:
1. For the assessment of the data quality, the validation function can leverage on the CRCU
assessment of the data quality management reports submitted to institution’s senior management, but it is expected to complement the CRCU assessment by its own analysis;
2. For the assessment of the representativeness in terms of both risk differentiation and risk
quantification, the validation function can form its opinion by leveraging on the CRCU analyses with its own independent conclusions;
3. For the assessment of the appropriateness of the risk differentiation not related to the
assessment of the stability of the ratings, the validation function can form its opinion by leveraging on the CRCU analyses with its own independent conclusions;
4. For the assessment of the rating philosophy and the appropriateness of the risk
quantification (i.e. back-testing analyses), the validation function can leverage on the CRCU assessment but is expected to complement the CRCU assessment by its own analysis. In the particular case where the validation tools are maintained by the validation function (and used by the CRCU), or if the tools have been developed by the CRCU but reviewed and certified by the validation function, the outcomes of the tools can be considered as being derived from the validation.
99. Ad hoc validation activities. Ad hoc analyses complement the regular validation activities
where necessary. As such, the validation policy is expected to further specify the type of those

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 78 ad hoc analyses to be performed to the extent possible, as further discussed in paragraph [23]. Institutions are expected to assess at least the following situations with regard to the need for an ad hoc analysis complementing the regular validation:
a. Where regular analyses indicate potential model deficiencies (e.g. representativeness issues, issues with the discriminatory power, back-testing issues or an increase of the override rate); b. In order to assess specific aspects of the rating system changes, making use of the analyses described in section [4 First validation] covering the aspects affected by the changes;
c. Where validation reports have listed own conclusions, findings and recommendations
that require follow-up; d. In the context of the remediation activities for supervisory findings; In addition, further ad hoc activities can also be triggered, independently of the regular validation cycle, to be performed without delay. In this context, examples of events which can be considered as potential triggers for such an ad hoc validation are large external occurrences influencing the obligors and facilities such as specific-crisis and corresponding measures implemented where it would not be prudent to wait until the next regular validation to perform a review.
100. Full validation. A full validation complements the regular validation by covering
methodological aspects for which the underlying assumptions used at the time of model development may no longer be considered verified, hence adding to the regular activities a less frequent, but nevertheless periodical review of the rating system, in a more comprehensive manner. This full validation includes an assessment of alternative modelling possibilities (i.e. of the choices mentioned in paragraphs [36], [41], [42], [43], [44], [51], [52], [53], [63], and [66]), which are expected to ensure that several viable alternatives have been investigated. As such, a good practice observed in institutions is to conduct periodic independent challenging analysis via ’challenger models’, as well as an assessment of the new available data and current market ‘best practices’. Moreover, the validation function is expected to review the specific analyses not performed during the regular validation (i.e. in cases where the validation function used the results of its previous assessment) as further elaborated in Focus Box [9]. In performing this comprehensive review, the validation function may take into account the analyses performed in a sufficiently recent full review of estimates conducted by the CRCU as further described in Interaction Box [12]. The definition of an appropriate frequency of the full validation is expected to be part of the validation policy, such that any new material deficiency is detected within a reasonable time period. As a good practice, institutions can perform the full validation every 3 years.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 79 FOCUS BOX 9: SPECIFIC ANALYSES EXPECTED DURING THE FULL VALIDATION The institutions are expected to find an appropriate split between those validation activities to be performed in a more frequent manner as regular validation activities on the one hand and those that can be performed in the full validation, only. Thus, for some types of analysis, the full validation serves as a backstop and the respective areas should be assessed in each full validation at the latest. In practice, this is typically relevant for the following validation activities:
Analyses related to the assessment of the materiality of model changes:
In addition to the materiality assessment performed in accordance with the CDR on model changes and described in paragraph [96.c], a good practice observed in institutions is to have an opinion on the materiality of the combined non-material changes that occurred since the last review by the CAs. As such, a possible approach may be to calculate (or estimate) the total impact of all changes resulting from the updated risk parameter estimates, i.e. to compare the own funds requirements resulting from the updated risk parameter estimates with the ones resulting from the last version of the rating system authorised by the CA (i.e. the version of the rating system without taking into account non-material rating system changes). In case of a material difference between those two, the validation function would investigate the driver(s) behind this difference and elaborate on them in the validation report. Analyses related to the risk differentiation methodology:
1. The modelling choices and underlying assumptions, including the selection of risk drivers
(point [45.a]), the functional form of the risk differentiation (point [45.b]), the definition of grades or pools (point [45.c], focusing on the granularity), both for non-defaulted and defaulted exposures in the scope of application (point [59]) are still applicable;
2. For the slotting approach, that the use of the slotting approach cannot be replaced by
appropriate estimates of PDs, as referred to in paragraph [72], the scope of the slotting approaches is still appropriate (point [75.a]), and whether all relevant information is considered (point [75.b]). Analyses related to the risk quantification methodology:
1. The conditions for the chosen general calibration methodology (points [41.a] and [42.a],
especially when a particular approach is justified by insufficient internal data) are still applicable;
2. The conditions for the chosen calculation of the observed default rates (point [41.b]) are
still applicable;
3. The conditions for the chosen calculation of the long-run-average (points [41.c] and
[42.b]) are still applicable. For the LGD, the validation function is expected to assess

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 80 whether the inclusion of newly closed recovery processes leads to a significant change in the estimation related to the non-closed recovery cases;
4. The conditions for the chosen methodology of calibration to the long-run-average (points
[41.d] and [42.c]) are still applicable. In the case where, during previous validation, the minimum years were not reached (i.e. the transitional period envisaged in the CRR was used), the validation function is expected to check that there has been a recalibration of the rating system and as a consequence, the core model performance regarding the risk quantification is expected to be assessed according to the section [4 First validation activities];
5. The deficiencies related to any appropriate adjustments (points [41.e] and [42.d]) are, to
the extent possible, in a state of actively being resolved and where they have not been, the adjustments are still appropriate; 305
6. If the MoC is still appropriate (point [43.b]) given more recent data and developments.
Analyses related to the effect of the economic downturn on the risk parameters:
The validation function is expected to consider the effects of recent changes in the economic conditions on the current portfolio and to review:
1. The adequacy of the downturn period used for the estimation, i.e. whether any new
observations would qualify as a downturn period or whether a change in the application portfolio would justify a change in the selection of the downturn period (point [53.a]);
2. The adequacy of the method chosen for the estimation. In particular, the validation
function is expected to check that any new data cannot be used to change the selected methodology of estimation or enhance the adequacy of its outcomes (point [53.b]). Analyses related to the risk parameters for defaulted exposures:
1. In relation to the RDS, (point [60.a]), whether changes in the recovery policy or processes
are not lessening the accuracy of estimates via the way reference dates are defined (for instance if reference dates are defined as fixed time horizons after default and recovery practices change);
2. In relation to the ELBE, whether the adjustment related to the economic conditions (if
applied) remains appropriate (point [61.a]), and the comparison with specific credit risk adjustments on the last years’ observations (point [61.b]);
3. In relation to the LGD in default, whether any (new) systematic deviations in the LGD
estimates applied immediately before and after the default are observed (point [62.a]), 305 Paragraph 40 of the EBA GLs on PD and LGD estimation.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 81 and whether the LGD in default is always higher than the ELBE on the last years’ observations. Analyses related to the incorporation of credit risk mitigation in risk parameters:
1. That the allocation of recovery cash flows is still consistent with current recovery and
collection processes (point [64.b]);
2. Whether the use of the most recent data would lead to materially different recovery
estimates based on the analysis of the estimated recoveries from collateral (point [55.c]);
3. The additional tests described in paragraph [56] with the most recent data deemed
relevant during the first validation to check that the modelling choices and hypotheses are still relevant (e.g., these tests could check empirically that collateral agreements have not changed over time);
4. Whether the approach chosen to recognise UFCP described in paragraph [60] remains
appropriate, and in particular whether the most recent data show a material breach in the conditions for the use of a particular approach (e.g. increase in the costs or changes in the back-testing results of the EL). The validation function is expected to assess the potential occurrence of any possible misalignments between the policy and its implementation relative to the use of overrides.
5.2 Minimum regular validation tasks regarding the core model
performance
101. Regular validation of the IRB metrics. For the assessment of the calculation of the metrics
mentioned in paragraph [30], the validation function can use its previous assessment of their implementation.
5.2.1 Risk differentiation
102. Regular validation of the rating assignment process. With respect to the assessment of
the rating assignment process described in paragraph [33] the validation function can use its previous assessment of the documentation of the assignment process(point a) and of the policy for the treatment of non-standard rating (point b). However, the validation function is expected to assess the potential occurrence of deficiencies detected in the actual implementation of the model (e.g. it is expected to know the conclusions of IA), and is expected to perform the materiality assessment of non-standard ratings on the latest years available.
103. Regular validation of the input data. With respect to the analysis of input data described
in paragraph [35]:

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 82 a. For the assessment of the data quality described in point [35.a], the validation function can use its previous assessments. As mentioned in the Interaction Box [12], the validation function is expected to assess the data quality management reports submitted to institution’s senior management directly; b. For the assessment of the completeness of the RDS described in point [35.b], the validation function can use its previous assessment;
c. For the procedures applied to the data used for the development of the model
described in point [35.c], the validation function can use its previous assessment. However, a good practice observed in institutions is to perform the back-testing comparisons of the data estimation using the new data available; d. For the analysis of the representativeness of the data used for the model development described in point [35.d], the validation function is expected to perform this assessment with respect to the application portfolio. For this analysis, the validation function can take into account the analyses performed by the CRCU in the context of paragraph 218 (a) of the GL on PD and LGD estimation as further described in the Interaction Box [12].
104. Regular validation of the modelling choices. With respect to the assessment of the
modelling choices and specifications described in paragraph [36], the validation function can use its previous assessments.
105. Statistical tests in the regular validation. With respect to the statistical tests mentioned in
paragraph [37], the validation function can take into account the analyses performed by the CRCU in the context of paragraph 218 (b) of the GL on PD and LGD estimation as further described in the Interaction Box [12], and is expected to form an opinion on:
a. The performance of the model on the sample composed of only the new data available since the last validation (as part of OOT tests), b. The evolution of the performance over time (for instance by performing tests separately for each year of observation) and in particular with the performance reached in the model development (see Focus Box [2]).
106. Validation challengers in the regular validation. For the purpose of challenging the
accuracy of the rating assignment, all the analyses described in paragraph [38], are expected to be conducted during the regular validations. The validation function can take into account the analyses performed by the CRCU in the context of paragraph 205 of the GL on PD and LGD estimation (point [38.b]) and paragraphs 206 and 207 (point [38.a]), as further described in the Interaction Box [12].

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 83
5.2.2 Risk quantification
107. Regular validation of the input data. With respect to the assessment of the input data used
for the risk quantification, the validation function is expected to review the elements described in paragraph [40]:
a. For the analysis of the data quality described in point [40.a], the validation function can use its previous assessments. As mentioned in the Interaction Box [12], the validation function is expected to assess the data quality management reports submitted to institution’s senior management directly; b. For the assessment of the completeness of the RDS described in point [40.b], the validation function can use its previous assessments;
c. For the assessment of the procedures for data collection and data cleansing described in
point [40.c], the validation function can use its previous assessments; d. For the analysis of the representativeness of the data used for the risk quantification described in point [40.d], the validation function is expected to perform this assessment with respect to the application portfolio. For this analysis the validation function can take into account the analyses performed by the CRCU in the context of Article 218 (a) of the GL on PD and LGD estimation as further described in the Interaction Box [12]. With this assessment the validation function is expected to challenge the continuous appropriateness of all appropriate adjustments applied on the risk estimates due to a lack of representativeness (mentioned in paragraphs [41.e] and [42.d]), in light of the new data available.
108. Regular validation of the methodological choices – best estimates. With respect to the
assessment of the methodological choices for PD best estimates described in paragraph [41] and for LGD best estimates described in paragraph [42], the validation may rely on its previous assessments.
109. Regular validation of the methodological choices – MoC. With respect to the assessment
of the methodological choices for MoC incorporated in the risk estimates as referred to in paragraph [43], the validation function may rely fully on its previous assessments.
110. Regular validation of the methodological choices – downturn. With respect to the
assessment of the inclusion of an economic downturn in LGD and CF estimates described in paragraph [44], the validation function may rely on its previous assessments.
111. Validation challengers in the regular validation. With respect to the statistical tests used
to challenge the risk quantification of the model, all the analysis in paragraph [45] should be performed, in particular using also the new available data. In this context, the validation function is expected to assess the evolution of the different adjustments and MoCs in relation to the evolution of the corresponding deficiencies and uncertainties, also leveraging on the

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 84 checks of paragraph [107.d].For the PD estimates, this includes an assessment of the period of the likely range of variability of defaults rates and the mix of good and bad years. For this evaluation, the validation function can leverage on the analysis from the CRCU as further described in the Interaction Box [12]. In particular, the validation function is expected to review the CRCU assessment of whether the use of the most recent data in the risk quantification would lead to materially different risk estimates (long-run-average and downturn estimates). 306
5.2.3 Other specific points
112. Regular validation of other specific points. As mentioned in paragraph [49], this section
describes the specificities of three aspects, which are complementing the expectations described in the general section. As such, the usual activities performed during the regular validations and described in the previous section are expected to be performed. Specificities related to the validation of defaulted exposures’ risk parameters
113. Regular validation of the defaulted exposures’ risk parameters. With respect to the
general assessment of the risk parameters for defaulted exposures, as mentioned in paragraph [50], the validation function is expected to follow a similar approach to that for non-defaulted exposures. In the context of the regular validation, this implies an assessment of whether new systematic deviations between realisations and estimates were observed on the most recent data. On the other hand, with respect to the assessment mentioned in paragraphs [51], [52] and [53], the validation function may rely on its previous assessments. Specificities related to the validation of credit risk mitigation
114. Regular validation of the CRM. With respect to the assessment of the specificities of the
incorporation of the CRM in the risk parameters, the validation function may rely on its previous assessments. Specificities related to the validation of the slotting approach
115. Regular validation of the slotting approach. The specificities of the regular validation of
the slotting approach are similar to the ones of other rating systems described in Section [5.2.1]. In particular, using the most recent data available, the validation function is expected to focus on:
a. The assessment of representativeness of exposures (in particular, if there is a change in the granting or renewal of loans when it comes to maturity and bullet payments); b. The specific challenger analyses as described in the Focus Box [5];
c. The general challenger analyses mentioned in paragraph [38].
306 Paragraph 218(c) of the GL on PD and LGD estimation

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 85
5.3 Minimum regular validation tasks regarding the modelling
environment
5.3.1 Data quality and maintenance
116. Regular validation of the data quality and maintenance. As mentioned in the Interaction
Box [12], for the assessment of the data quality, the validation function is expected to assess the data quality management reports submitted to the institution’s senior management directly such that it is aware of any new deficiency. In addition, the validation function is expected to check how the previously identified deficiencies have been treated and addressed by the CRCU, in particular, if there was an appropriate adjustment applied to the risk estimates and if these have been appropriately addressed by a MoC. 307
117. Monitoring of the assignment process in the regular validation. In addition, the validation
function is expected to monitor the comprehensiveness of the assignment process, as developed in paragraph [33.b], as a downward trend could be an indication of new data quality issues.
5.3.2 IT implementation of the rating systems
118. Regular validation of the implementation. With respect to the assessment of the adequacy
of the implementation mentioned in paragraph [76], the validation function can rely on its previous assessment, to the extent that the definitions of business/functional requirements for the systems supporting the IRB modelling and application are unchanged. 307 Paragraph 37 of the EBA GLs on PD and LGD estimation.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 86
6. Focus on specific validation
challenges
6.1 Focus 1: validation in the context of the use of external data
119. Scope of this section. This section is dedicated to the specific situation where a rating
system is developed on a broader range of exposures than it is afterwards applied (i.e. with additional obligor or facilities added to the RDS vis-à-vis the application and historical portfolios). To note, the use of external data to supplement the development and/or application of the model without adding obligors or facilities (e.g. via the supply of data used as risk drivers) is not covered in this section. Consequently, typical cases addressed in this
section are:
a. Case 1: The development of the rating system is based on both internal and purchased external data stored in the internal systems, i.e. to which the institution has access to; b. Case 2: The development of the rating system is based on internal data, as well as on external data to which the institution does not have access to. In practice, this can be the case when the rating system is developed:

Case 2a: at group level while used at stand-alone level of multiple
subsidiaries (i.e. this can be seen as a pool model from the perspective of each individual subsidiary) or
Case 2b: externally based on pooled data of several institutions not
belonging to the same group (in which case the participating institutions will share data and use a rating model with common features).

General requirements on the validation when using external data. In all these cases, the
validation of the rating system is not expected to materially differ from the validation of other rating systems but entails some specificities. As such, the validation of a rating system which is built on external data is expected to follow the following five principles.
Principle 1 – appropriateness of the use of external data (representativeness): the use of
external data entails some specific risks, and may not always be appropriate. In particular, the representativeness for risk differentiation (as mentioned in paragraph [35.d]) and risk quantification (as mentioned in paragraph [40.d]) is expected to be carefully assessed vis-a-vis the individual entity’s application portfolio. Especially, the CRR provides further requirements when a mapping to the rating grades of an external credit assessment institution or a similar

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 87 organisation 308 or when pooled data is used. 309 Based on this opinion on the representativeness, the validation function is expected to challenge the appropriateness of the external data used, and to carefully review the quantification of the (Category A) MoC. 310
122. Principle 2 – access to data: The validation function keeps the responsibility of the
objectives described in paragraph [17], and is expected to be in a position to challenge methodological choices related to the development of the rating system and to perform additional quantitative analyses. In the cases 2a and 2b, this implies that the validation function has the possibility to request any further analyses from the data provider (the third party managing the data pool may assist the institution in its validation activities by performing those tasks of validation which require access to the pooled data 311

this can be viewed as an
outsourcing of operational tasks as further described in section [6.2 Focus 2: validation in the context of outsourcing of validation tasks]).

Principle 3 – methodological choices’ assessment: When assessing the methodological
choices mentioned in paragraphs [36], [41], [42], [43] and [44], the validation function is expected to assess whether any bias has been introduced due to the duplication of observations on the same obligors or facilities used in the risk quantification.
Principle 4 – performance assessment: Even if the rating system has been developed using
external data,
312 the quantitative evaluation of its performance (i.e. tests mentioned in paragraphs [37], 313 [38] and [45], [46] and [47]) is expected to be performed first on the internal data. In addition, in the case where external data is used to circumvent data scarcity issues, the performance assessment of the rating system based on internal data can be complemented by an assessment of the performance using all data available. Challenges associated with the scarcity of data are further described in section [6.3 Focus 3: validation in the context of data scarcity].
Principle 5 – data quality: The external data is not expected to be treated differently than
internal data in terms of data quality assessment from the moment where it is stored in the internal system of the institution. As such, all the expectations mentioned in section [3.2.1 Data quality and maintenance] and more generally in paragraphs [35] and [40] apply. In addition, the validation function is expected to form an opinion on the data quality framework of the data provider. 308 Article 180(1)(f) of the CRR 309 Article 179(2) of the CRR 310 Paragraph 37(a)(viii) of the GL on PD and LGD estimation 311 Article 4(3) of the CDR on assessment methodology 312 To note, in risk quantification the use of internal data is mandatory. 313 In particular, OOT and OOS requirements are also expected to be respected when using external data.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 88
6.2 Focus 2: validation in the context of outsourcing of validation
tasks
126. Scope of this section and relevant regulatory requirements. Where an institution takes
the decision to start the process of outsourcing certain operational tasks of the validation function, it is expected to perform a comprehensive analysis of its compliance with all the regulatory requirements on outsourcing. In particular, some requirements arise through the Guidelines on outsourcing arrangements 314 (GL on Outsourcing), including the setting up of an outsourcing policy in which the outsourcing arrangements are carefully planned. 315 This plan is expected to account for the fact that, according to the CDR on model changes, any changes to the validation methodology and/or validation processes of existing IRB models have to be assessed and subsequently notified to the CA. In addition, the GL on Outsourcing define critical or important functions. 316 As mentioned in paragraph [8], the validation function can be viewed as an internal control function, and hence be subject to the requirements applicable to the critical or important functions introduced in the GL on Outsourcing.
127. Non-transferability of the responsibility. The independence of all validation activities,
regardless of them being outsourced or not, needs to be ensured by the institution (including in the case of intragroup outsourcing). In this context, only ‘operational tasks of internal control functions’ can be outsourced to anyone outside of the validation function, as also prescribed in the Guidelines on internal governance under Directive 2013/36/EU 317 (GL on Internal Governance) 318 (see paragraph [135] for more details on inter-bank specifics). More specifically, in the context of the validation of IRB rating systems, the operational tasks are limited to tasks which are sufficiently specified by the validation function such that their outcomes are not dependent on the party performing them. Hence, the validation function of the respective institution should always retain the responsibility of the opinion on the rating system (and the related components of this opinion mentioned in paragraph [17]). As such, the validation function remains responsible of its validation policy (in particular its soundness), of the correct implementation of the validation methodology and ofthe final assessment on the rating system (including the follow-up of the validation function’s findings by the institution and, where relevant, of the findings raised by the CA as mentioned in paragraph [96.b]). This should be evaluated by IA along the usual dimensions mentioned in paragraph [21].
128. Involvement of the senior management and management body. 319 As a result, the
management of the validation function will remain responsible for all validation activities. Therefore, all changes to validation methodologies and/or validation processes and validation 314 EBA/GL/2019/02 315 Paragraph 42(c) of the GL on Outsourcing 316 Paragraph 29 of the GL on Outsourcing 317 EBA/GL/2021/05 318 Paragraph 170 of the GL on Internal Governance 319 Paragraph 35 of the GL on Outsourcing

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 89 reports should be understood and approved by the senior management and the members of the management body (or the designated committee thereof).
129. Assessment of the outsourcing providers. In addition to the regulatory requirements, the
outsourcing policy of the institution plays an important role in order to ensure a sufficient degree of knowledge and resources of the validation function such that it can perform all the validation activities (as mentioned in paragraph [18]) in an independent manner with the help of the outsourcing provider. The outsourcing policy should take into account the nature of the outsourcing providers, 320 including:
a. Whether they are authorised by a CA; b. Whether they are within the same group (intragroup outsourcing arrangements), outside of the group but from an entity being part of the same institutional protection scheme or completely unrelated;
c. Whether they are located within a Member State or a third country;
d. Whether the service provider performs some sub-outsourcing; 321 e. Whether they are currently performing model development or CRCU tasks. A good practice observed in institutions is, when the service provider to which operational validation tasks have been outsourced also performs CRCU activities, to have these tasks and activities performed by an independent unit.
130. Communication with the CA. While all planned outsourcing of operational tasks of the
validation function has to be communicated to the CA in a timely manner, 322 it is advised that the discussion process between the institution and the CA starts as early as possible, if possible as part of the pre-outsourcing analysis, 323 especially in cases where there are plans to outsource operational tasks of the validation to a service provider that is:
a. Not authorised by a CA of a Member State; b. Outside the group; or
c. Located in third countries.
This is very much interlinked with the institution’s ability to oversee the service provider and to manage the risks and the pertaining business continuity measures. 324 320 Paragraph 43 of the GL on Outsourcing 321 Section 13.1 of the GL on Outsourcing 322 Paragraph 58 of the GL on Outsourcing 323 Section 12 of the GL on Outsourcing 324 Paragraph 44 of the GL on Outsourcing

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 90
131. Transparency of outsourcing. Outsourcing requires to retain a clear and transparent
organisational framework and structure.
325 To that end, any outsourcing of operational tasks of the validation function should be properly documented. Where operational tasks of the validation were outsourced, the respective validation reports should carry the logo and name of the institution and of the third party performing the outsourced tasks. Additionally, it should be clearly identifiable, which operational tasks have been performed by the service provider, for example via service level agreements.
132. Access and inspection in the context of outsourcing. As already established at the
beginning of the chapter, the validation function falls into the category ’critical or important function’. 326 It is therefore necessary that the institution as well as the CA (including resolution authorities, and any other person appointed by them or the CA) have full access to the service provider and unrestricted rights of inspection and auditing related to the outsourced operational tasks. 327
133. Quality of outsourced operational tasks. Institutions should monitor the performance of
the service providers on an ongoing basis whenever it comes to the outsourcing arrangements regarding the operational tasks of the validation function. 328 To that end, it is also necessary to ensure that the service providers all meet appropriate performance and quality standards. 329
134. Business continuity in the context of outsourcing. When outsourcing operational tasks of
the validation function, the existence of a business continuity plan is required. 330 Therefore, the institution must be able to either transfer the function to alternative service providers or reintegrate the function within an appropriate time frame, including the operational tasks related to the validation for which this transfer or re-integration should be possible at the latest by the next validation of the affected rating system(s). 331 This should be part of the exit plan and strategy. 332
135. Intragroup outsourcing. When it comes to outsourcing within a banking group (e.g. as in
the case mentioned in paragraph [15]) it is also not possible to outsource any part of the validation function with the exception of operational tasks. Hence, while it is possible to perform certain operational tasks centralised through the use of service level agreements or other contractual arrangements, the responsibilities mentioned in paragraph [127] are still retained by each individual validation function. However, where a validation function 325 Paragraph 39(b) of the GL on Outsourcing 326 Paragraph 29(b) of the GL on Outsourcing 327 Paragraph 87 of the GL on Outsourcing 328 Section 14 of the GL on Outsourcing 329 Paragraph 104 of the GL on Outsourcing 330 Section 9 of the GL on Outsourcing 331 Paragraph 40(f) of the GL on Outsourcing 332 Section 15 of the GL on Outsourcing

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 91 outsources operational tasks to a validation function of a consolidating entity to which the CRR requirements apply, the following requirements should apply simultaneously:
a. The independence of the validation function (paragraph [129]) of the consolidating entity can be verified most efficiently via the reports from IA of the consolidating entity as well as pertaining reports by the CA to which the outsourcing entity should have full access to. 333 b. The transparency of the outsourcing (paragraph [131]) needs to be fully adhered to.
c. The unrestricted rights of inspection and audit (paragraph [132]) could be ensured by
the contractually ensured possibility of the outsourcing entity to trigger an investigation by the IA of the consolidating entity in a timely manner. d. The quality of the outsourced tasks (paragraph [133]) can be verified by the validation function of the outsourcing entity through an assessment of the reports from the IA of the consolidating entity as well as relevant CA reports. Additionally, the validation function should report deficiencies identified in relation to the outsourced operational tasks to the internal function responsible for the monitoring of the outsourcing. 334 e. Business continuity (paragraph [134]) can be ensured via the business continuity plan of the consolidating entity. In addition, in the case of continuous quality or timeliness issues with regard to the outsourced tasks (e.g. issues with one of the previous points described in this paragraph), the outsourcing entity should be able to transfer the function to alternative service providers or reintegrate the function within an appropriately short time frame. 333 Paragraph 22(b) of the GL on Outsourcing 334 Paragraph 104 of the GL on Outsourcing

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 92
6.3 Focus 3: validation in the context of data scarcity
136. Scope of this section. In the context of this section, data scarcity refers to the lack of a
sufficient number of observations on the empirical realisation of ‘risk metrics’, i.e. defaults, realised LGDs and realised CFs.
137. Adaptation of the validation policy in the context of data scarcity. The validation of ratings
systems in a context of data scarcity brings some additional challenges when it comes to the statistical tests mentioned in paragraph [25.b]. As such, the validation policy is expected to provide in particular:
a. Specific metrics or tolerances defined. 335 In this context, the validation function is expected to pay special attention to the interpretation of the results obtained for the application of statistical challengers or tools. This includes the different assumptions underlying the application of the challengers, in particular the constraints of some statistical tests. In any case, statistical uncertainty stemming from data scarcity is expected to be treated conservatively when it comes to drawing conclusions. Therefore, broad confidence intervals are expected to be complemented by a more logical or judgmental interpretation of the results; b. A description of complementary analyses that are foreseen to supplement quantitative measures, such as descriptive statistics or visual analyses (e.g. based on graphical analyses, such as boxplots or histograms);
c. When a rating system is developed on a broader range of exposures than it is
afterwards applied, the validation policy is expected to be adjusted as further discussed in section [6.1 Focus 1: validation in the context of the use of external data].
138. Specific assessment of the risk differentiation in the context of data scarcity. The
validation function is expected to verify:
a. Whether the main risk drivers of the observed defaults and losses are appropriately reflected in the model, as part of the analysis mentioned in paragraph [36.a], by analysing observed individual defaults (or at least a sample of them where the number of defaults makes a comprehensive analysis unduly burdensome). However, the model is not expected to be fully adjusted to fit a small number of observations (i.e. in order to avoid overfitting); b. The adequacy of the number of rating grades and pools in relation to the available data to allow for a quantification and validation of the default and loss characteristics at grade or pool level, as mentioned in paragraph [36.c]. In this context, an excessive granularity may indicate a lack of heterogeneity between grades or pools. 335 Article 12(c) of the CDR on assessment methodology

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 93
139. Specific assessment of the risk quantification in the context of data scarcity. In case a
second-best approach is used for either retail or purchased corporate receivable based on an estimate of total losses as mentioned in paragraph [41.a] and [42.a], the backtesting is expected to be performed on the estimated parameters as well as on the ‘intermediate parameters’ (i.e. total losses).
140. Examples of alternative validation approaches in the context of data scarcity. Where it is
not feasible to apply certain statistical tests, examples of good practices observed in institutions to complement validation analyses or approaches are:
a. Risk differentiation: comparison with internal credit expert ranking (e.g. blind rank ordering tests, whereby the ranking produced by the model is assessed against the ranking produced by credit experts); 336 b. Risk differentiation- OOT and OOS validation samples. When it comes to the assessment on OOT and OOS samples, in the context of data scarcity, it may become more challenging to dismiss some data for the model development (to leave it for the validation). Nevertheless, the validation function is expected to conduct sufficient analyses in order to have sufficient confidence that the developed model does not suffer from overfitting and that its performance is preserved over different economic conditions. As such, when choosing the approach, the validation function is expected to find the appropriate balance between, on the one side, gaining sufficient confidence in the assessment of the model performance, and on the other side, not unnecessarily impeding the model robustness. As such, several alternative approaches have been observed during the first validation:

Complement the tests performed by the CRCU with in sample tests
and qualitative analysis (such as with the one mentioned above);
Conduct the validation solely based on either an OOT or an OOS
sample, using data not used at all by the CRCU for the model development;
Leverage on the analyses performed by the CRCU, where the CRCU has
assessed the performance of the model via OOT and OOS samples only during intermediate steps, but has used the whole sample to train the final model;
Other approaches based on simulations or bootstrapping techniques.
In the case where the validation function uses any of the alternative approaches, it is nevertheless expected to perform an appropriate OOT analysis during the regular validation. 336 In case the validation relies on such an approach, a key aspect is to ensure that the ranking produced by credit experts is derived independently from the ranking produced by the model assessed.

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 94
c. Risk differentiation and risk quantification: aggregation of data from different
observation periods or consideration of analyses based on multi-year periods; d. Risk differentiation and risk quantification: data enhancements (e.g. early arrears definition of default or an extension of the default horizon); e. Risk differentiation and risk quantification: testing with external benchmarks (e.g. external ratings or market driven metrics such as bond spreads).

SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 95 EUROPEAN BANKING AUTHORITY Tour Europlaza, 20 avenue André Prothin CS 30154 92927 Paris La Défense CEDEX, FRANCE Tel. +33 1 86 52 70 00 E-mail: info@eba.europa.eu https://eba.europa.eu

Lineage: No known changes

Similar documents from other regulators