G9-2022 - Matters related to the credit risk models of banks

370 Helen Joseph Street Pretoria 0002
+27 12 313 3911 / 0861 12 7272 www.resbank.co.za Ref.: 15/8/1/2 G9/2022 To: All banks, branches of foreign institutions, controlling companies, eligible institutions and auditors of the banks or controlling companies Guidance Note issued in terms of section 6(5) of the Banks Act 94 of 1990 Executive summary This guidance note covers various matters related to the credit risk models of banks using the internal ratings-based (IRB) approach (IRB banks), to calculate minimum required capital and reserve funds for their credit risk exposures. Its purpose is to articulate the Prudential Authority’s (PA) expectations regarding the implementation of the various IRB minimum requirements specified in the Regulations relating to Banks (Regulations) and to also outline factors to be taken into account by IRB banks in their model development, validation and on-going monitoring activities. The guidance set out in this Guidance Note will form an integral part of the PA’s assessment of model change applications and ongoing assessment of IRB banks’ overall compliance with the minimum requirements specified in the Regulations.

1. Introduction
   1.1. In 2021, the PA issued a discussion paper titled “Policy guidance on credit risk
   models related matters”, requesting comments from IRB banks on various matters related to the IRB minimum requirements specified in the Regulations and their practical effect on the development, validation and on-going monitoring activities of IRB banks.
   1.2. The topics covered in this guidance note flow directly from the discussion paper
   and are intended to outline the PA’s expectations regarding the practical implementation of the IRB minimum requirements specified in the Regulations.
   1.3. It is the PA’s expectation that all IRB banks should take account of factors outlined
   in this Guidance Note when developing and validating their credit risk models, prior to submission to the PA for approval.
   1.4. The guidance provided in this Guidance Note is however not exhaustive and
   accordingly, the PA also expects IRB banks to consider other factors that may be relevant to their credit models in line with the respective requirements specified in regulations 23(11) and 23(13) of the Regulations.

2. Model documentation quality
   2.1. As part of its model review process, the PA relies extensively on model
   development and validation documents compiled by IRB banks. It is therefore critical that these documents are prepared to the highest standards, and when assessed independently, provide evidence of compliance with the IRB minimum requirements specified in the Regulations, validation and approval procedures that are in line with IRB banks’ internal policies and processes.
   2.2. Different portfolios will require different considerations with regard to the type and
   amount of qualitative and quantitative information included in the documents. However, IRB banks should consider amongst other things, the nature and complexity of the relevant portfolios. For example, residential portfolios-given their data rich characteristics-will generally be overweight on quantitative information compared to low default portfolios such as property funds where elaborate qualitative descriptions may be warranted.
   2.3. IRB banks are encouraged to include guidelines on documentation standards in
   their model development and validation policies to ensure bank-wide consistency.
   2.4. When assessing whether the information is sufficient, a key consideration is the
   extent to which external parties, without the benefit of internal experience, will gain an understanding and appreciation of the risk characteristics of the portfolios covered by the model and the technical approaches employed to develop and validate the model based on what is documented.
   2.5. To this end, IRB banks should take into account the following factors when
   finalising their documents prior to submission to the PA:
   2.5.1. Documentation of portfolio characteristics
   2.5.1.1. Descriptions of the portfolios and business segments covered by the models and
   related regulatory asset class in which the portfolios are reported should be clearly documented. These descriptions should include underlying counterparty segments and typical products offered in these portfolios. In cases where the models cover portfolios residing in multiple business units, descriptions of key differences-if any￾and similarities between the various sub-portfolios should be included in the documents.
   2.5.1.2. Quantitative metrics that banks should consider, include historical average risk
   weights, probability of default (PD), loss given default (LGD) and exposure at default (EAD) estimates-and where relavant, a breakdown between performing and non-performing portions of the portfolio.
   2.5.1.3. For newly developed models, this history may not be available or the time series
   may be short. In such cases, more recent data such as default and loss frequencies may be included to complement the available parameter estimates history.
   2.5.1.4. In most cases, regulatory asset classification may be obvious, whereas in other
   cases it may not be. Moreover, some models may cover multiple exposures that map into multiple asset classes. Accordingly, documents should provide a

breakdown of risk parameters across various asset classes within the portfolios. This breakdown should also include, for each sub-portfolio, average risk weights, and where possible, a split between performing and non-performing loans.
2.5.1.5. In addition, and in cases where it is feasible, rating grade distributions of EAD
should be included. Although the Regulations prescribe a minimum number of rating grades, the PA recognises that some banks may use grades that are relatively more granular, in which case banks must exercise judgement in terms of the level of granularity they include in the documents to, for instance, demonstrate absence of undue concentrations.
2.5.2. Documentation considerations of judgement-based models
2.5.2.1. Credit risk models in one form or another rely on the inputs of internal and external
experts. This is partly to ensure that their scope is clearly defined from the start, representative development datasets are used, and upon implementation, to also ensure that they are used in line with their development specifications. External expertise is most often leveraged to ensure that models employ best practices. The involvement of experts is even more important for low default portfolios given the limited data available to employ statistical and other quantitative methodologies.
2.5.2.2. Documentation should therefore include a description of the processes followed to
engage experts, and how their involvement impacted various elements of the development and validation processes.
2.5.2.3. It will not be sufficient, for instance, to merely document that consultations with
experts motivated certain modelling choices or risk drivers, without providing any evidence to support those choices and why IRB banks consider these decisions and choices appropriate for their portfolios. Accordingly, documents in this regard should reflect in sufficient detail the experts consulted, the nature of the consultations and their ultimate impact on the relevant models.
2.5.2.4. In the case where final risk drivers and calibration levels are a combination of data
and expert judgement, an elaborative description of this process should be included. It may even be useful to include as annexures some of the minutes or records of these expert meetings and interactions to provide evidence of the complexities of the discussions and debates during the development process.
2.5.3. Documentation of new models versus changes to existing models
2.5.3.1. Changes to existing models will require different considerations in terms of
documentation compared to newly developed models. For existing models, it is important to clearly document the distinction between those aspects of the models that are changing versus those that remain unchanged. This will focus internal approval, but also the PA’s review and approval process only for amendments.
2.5.3.2. Similarly, validation approaches may differ between new and existing models,
with some banks following different levels of hierachies in terms of depth. This difference should also be made clear and demonstrated to be in line with internal validation policies and in compliance with validation requirements specified in the Regulations.

2.5.3.3. Similar considerations should guide the documentation of impact assessments,
especially in cases where multiple changes to risk parameters are being effected. In this regard, the PA requires information related to, amongst other things, the impact on risk parameters, RWAs and capital, both in isolation and cumulatively. The same goes for multiple methodological changes, in respect of which the impact must be shown separately and cumulatively.
2.5.3.4. There are however cases where a breakdown of the impact separately for each
methodological change is not possible, at least prior to rating an entire portfolio with the approved model. In such cases, the information documented should, to the extent possible, be on a best efforts basis, and the PA will then expect the post approval confirmation of the impact to include the detailed capital impact breakdown.
2.5.3.5. Some information is better placed in annexures, including development codes. It
will suffice in this case to summarise results as opposed to including detailed descriptions of the coding process. It will also be useful to include descriptions of abbreviations in the beginning of the documents for ease of reference.
2.5.4. Records of approval committees and evidence of challenge
2.5.4.1. Directive 2 of 2014 (D2/2014) requires IRB banks to submit a record of
proceedings or minutes of all approval committees of their model change applications. These records serve two primary functions: Firstly, they provide the PA with evidence that the model changes were taken through all relevant committees prior to submission. Secondly, they provide evidence that the model changes were discussed and adequately challenged.
2.5.4.2. The second function presents more nuanced implications on the structure and
content of these approval records. In other words, banks must consider how best to reflect the respective complexities of the discussions at these meetings, particularly given their technical nature.
2.5.4.3. Relevant considerations in this regard include the proper documentation of the
concerns raised by members of the committees. In some cases conditional approval may be granted, which will either result in immediate action items for development and/or validation teams or for noting in next reviews on recalibrations. This in turn may require development and/or validation teams to amend their reports to take account of these concerns.
2.5.4.4. These reports should therefore reflect these discussions and evidence that either
the concerns were not sufficiently material to warrant a decline or model development and/or validation teams performed additional work to adress the concerns prior to submission to the PA. Alternatively, if decisions result in actions for future redevelopments and recalibrations, these should be accompanied by timelines for remediation.
2.5.4.5. In principle, the PA has no objection to the use of round robin approvals, but is
however of the view that these should be used sparingly and discouraged particularly for material models and material changes as prescribed in D2/2014.

The PA nonetheless accepts that these type of approvals may be necessary in cases where, for example, a committee approved models subject to being provided with additional information or data, with no additional work required from development and validation teams. In those cases, the PA will still expect the chairs and members of these committees to assess whether round robin approvals are appropriate, or whether additional engagements may be warranted.
2.5.4.6. In this regard, the PA considers as inappropriate a justification of a round robin
approval merely to avoid delays and deviation from timelines of planned submissions to other higher committees or the PA. Sufficient evidence of a rigorous approval process must be clearly documented.
3. LGD estimates for defaulted loans
3.1. The PA wishes to draw IRB banks’ attention to regulation 23(13)(b)(v)(C)(xiii)(aa)
read with regulation 23(13)(d)(i) of the Regulations. These sections amongst other things require banks to estimate LGDs for defaulted loans in order to reflect the possibility that the bank may have to recognsie additional, unexpected losses during the recovery period.
3.2. In addition, regulation 23(11)(d)(ii)(B) requires IRB banks to compare their LGD
estimates for defaulted loans with the best estimate of their expected loss (BEEL) -where BEEL should equal at least specific credit impairments- to determine the amount of capital for defaulted loans. There are two pertinent implications of these regulatory requirements:
3.2.1. The LGD methodologies for defaulted loans are within the scope of the regulatory
capital models and therefore subject to the IRB minimum requirements specified in the Regulations. This in turn means that any changes must be submitted to the PA for approval prior to the implementation thereof, in line with the requirements specified in D2/2014.
3.2.2. It is however not necessarily a requirement that IRB banks must develop separate
models, at least from performing LGDs. Rather, the requirement does however expect LGD estimates assigned to defaulted loans to incorporate additional considerations relating to the uncertainties around the recovery process. Common methodlogies which the PA has approved in the past, include those that use performing LGDs as a starting base, and making adjustments to account for features specific to defaulted loans such as time in default. The net effect is that the longer a defaulted loan stays in the recovery process the higher the LGD estimate assigned and therefore the resulting RWAs and capital requirements, at least relative to performing loans.
3.2.3. The PA also wishes to draw IRB banks’ attention to the RWA and capital impact
numbers submitted in the cover note accompanying the model change applications. The PA noted inconsistencies amongst banks in terms of how this
table is completed, with some banks for instance recording zero LGDs and zero
capital impact for defaulted loans. In other cases, the BEEL is reported as being the same as the expected loss for performing loans.
3.2.4. In this regard, the PA wishes to reiterate that this table must be completed in line

with the requirements specified in regulations 23(11) and 23(13) of the Regulations. In cases where banks report zero LGD numbers for these loans, the PA will expect justifications. The PA also expects the validation unit to verify the accuracy of these impact numbers prior to submission and thereafter with the post implementation confirmation.
4. LGD overrides
4.1. Essentially, an LGD override is a process whereby model users replace or adjust
model generated LGD estimates in the capital calculation. This process may be necessary in cases where model outputs are outside the range of possible outcomes. In other cases this may be based on a review by rating analysts -of information not incorporated in the model- which may then suggest that the estimates are out of line with the underlying risk profile of the portfolios. Overrides may also be necessary due to structural shifts in the portfolio, with consequent loss in model predictive power. In other cases however, the overrides may be the result of user beliefs or views that are not empirically justifiable.
4.2. IRB banks should also take note that adjustments to LGD estimates that form part
of the model or are embedded in the model development process will not be regarded as overrides. An example in this case will be where a minimum floor is imposed on estimates produced by the model.
4.3. That said, the PA in principle does not object to LGD overrides, but is however
of the view that these should be kept to an absolute minimum and only effected to mitigate for limitations in the model. This in turn means that a persistent and high number of overrides most likely suggest that users place limited reliance on the model outputs and therefore may indicate a need for corrective action by development teams.
4.4. The PA expects IRB banks to continously monitor and perform regular analysis
to determine whether overrides are within acceptable ranges and more generally, whether the models still provide reliable LGD estimates that are in line with the expectations of users.
4.5. Rigorous approval processes provide an important control mechanism for
ensuring that overrides are performed within acceptable ranges, for the right reasons and within properly delegated authorities. Whilst the PA has observed that IRB banks have in place proper governance processes, discussions at these approval committees are enriched and robust if these committees are composed of the right mix of personnel, from business units and senior management and other independent personnel. Therefore, IRB banks should continously ensure that these approval committees are performing in line with their mandates and the composition is adequate.
4.6. The PA will not prescribe override monitoring mechanisms, but will expect IRB
banks to put in place tools to track overrides against internal thresholds, and to produce regular reports to relevant stakeholders across the bank. An example of a tracking mechanism, shown below, could include a classification of overrides into several categories. The PA will in the near term complement these internal monitoring with its own data collection efforts, with the recent IRB data and self-

assessment templates being one of the initiatives.
Table 1: Overrides monitoring
Outcome Threshold Description
Satisfactory Override rate ≤ XX% Overrides are within an acceptable range Satisfactory with limitations XX%<Override rate ≤YY% Overrides are within acceptable ranges, but closer monitoring is recommended Unsatisfactory Override rate≥ ZZ% Overrides outside acceptable range
4.7. Other analytical tools may incude an assessment of the extent to which overrides
increase or reduce overall portfolio LGD estimates, and whether these result broadly in more or less conservatism in final estimates compared to actual experience.
5. Quality and representativeness of development datasets
5.1. Limited or a complete lack of default and loss data for low default portfolios often
makes the development of robust and reliable credit risk models challenging, and as a result, banks most often rely on external data sources and expert judgement to develop models for low default portfolios.
5.2. Indeed, regulation 23(11) and regulation 23(12) of the Regulations allow IRB banks
to use external data. This is however subject to compliance with various minimum requirements, including the avoidance of undue bias because of inconsistencies between external and internal data sources, particularly regarding rating criteria, risk characteristics and definition of default.
5.3. In cases where banks source external data, it is important that rigorous
assessments are performed to ensure that the data comply with the regulatory requirements, but more generally to ensure reliability in terms of quality and representativeness.
5.4. Quality in this regard includes getting comfort that data owners have in place
validation tools to check quality prior to the commencement of the development process. For instance, if data is collected by a consortium from various banks, validation tools must ensure that submissions from participating banks are consistent across various dimensions, including products, default and loss definitions.
5.5. Moreover, missing and outliers in datasets may in some cases point to issues of
quality, but in other cases may point to unique bank specific or jurisdictional features. All these factors should accordingly be taken into account by banks when deciding whether the datasets are acceptable.
5.6. Representativeness assessments include both qualitative and quantitative tests,
and although not necessarily exhaustive, these should include:

5.6.1. Assessment of lending standards of exposures underlying these datasets against
internal portfolios. Other relevant characteristics include comparing distributions of obligors across industries and geographies. The PA however admits that these assessments are fraught with practical challenges. For one, the number of data fields available from data sources may be limited and anonymised. Moreover, assessing lending standards may be hampered by confidentiality issues.
5.6.2. These issues may admittedly constrain the assessment in a significant way, and
in such cases the PA will not object to quantitative comparisons of at least subsets of plausible criteria or dimensions. Nonetheless, the PA will still expect banks to obtain, from external vendors, sufficient information about the nature and sources of the data and/or participating institutions, to be in a position to assess as reasonably as possible, whether the data or portions of it represent similar exposures or characteristics to their own portfolios.
5.6.3. In cases where a mixture of external and internal data is used, IRB banks should
ensure that the definitions used in combining the different sources are consistent.
5.6.4. External sources often use data gathering methods that do not always align to
regulatory requirements. In this regard, IRB banks should also ensure that appropriate adjustments are made to datasets to achieve a reasonable degree of consistency.
5.6.5. In cases where the inconsistencies between internal and external data sources
are significant, IRB banks may instead select to use external datasets for model development. In other words, use the external datasets for risk driver selections, estimation of weights etc. and thereafter uses internal datasets for calibration purposes. Internal datasets may also be used to inform any additional adjustments to models to bring them in line with internal experiences and regulatory requirements.
5.6.6. IRB banks should also have internal thresholds on acceptable rates of missing
data to ensure that datasets used in the modelling contain sufficient and meaningful data fields.
5.6.7. Definition of default has a direct bearing on the classification of exposures into
default and non-default status and the composition of the datasets flowing in the model development process. Apart from the 90 days past due requirement, regulation 67 of the Regulations outlines other considerations for flagging default events.
5.6.8. Accordingly, many features are likely to give rise to bank-specific variations
compared to external sources. Notable examples include a requirement for “banks to consent to a distressed restructuring”, or “the obligor has applied for or has been placed in bankruptcy”. This is even more pertinent for wholesale exposures, where the default flagging process is governed by internal policies and processes that in most cases result in a default event well before the 90-days past due trigger.
5.6.9. In this regard, the PA will look for evidence demonstrating a rigorous engagement
with the differences in definitions between internal and external data sources. In

deciding whether to use external data, banks should also consider necessary adjustment to the external data to create consistencies with internal experiences and regulatory requirements.
5.6.10. Industry and risk characteristics, IRB banks must recognise that market events
affect different portfolios in different ways. This in turn may result in diverse composition of default datasets between internal and external sources in cases of different industry compositions. For instance, if banks’ internal datasets are overweight to the mining sector, then external datasets with overweight composition to information technology may not be appropriate. Therefore, consideration should be given to the bias likely to arise if external datasets exhibiting such differences are introduced into the models.
5.6.11. Conservative overlays: Reliance on external data poses some risk, and this is
true even if a bank can demonstrate that it is representative of its portfolio(s). Conservative adjustments in various elements of the model are therefore critical to mitigate these risks. It also means endeavoring to err on the side of caution when dealing with any uncertainties, or concerns about the external data. However, conservative adjustments should never be used to cover fundamental underlying weaknesses either in the data or the resulting model methodology, but merely to mitigate risks and gaps in the data that may impact on the final performance of the model.
5.6.12. Governance and stakeholders’ involvement: Model development and
implementation are cross-cutting activities, involving various personnel across the bank. Those responsible for data must ensure that the correct data is available, whereas development and validation teams are respectively responsible for building and assuring acceptable performance. There are also business units, with their insights into the process, to ensure that the outcomes are in line with business experiences. The governance and approval processes provide oversight mechanism over the activities, and in many respects serve as coordinating forums to ensure timely and efficient delivery of sound and reliable models.
5.6.13. Accordingly, all these stakeholders have a role to play when it comes to decisions
on the degree of reliance development teams ought to place on external data and also in giving their inputs on relevant adjustments. In this regard, the PA expects evidence that the decision with regards to these data sources is the product of engagement and robust discussions with all relevant stakeholders.
6. The definition and incorporation of downturn in LGDs and EADs
6.1. Regulation 23(13)(b)(v) of the Regulations requires IRB banks to estimate LGD
ratios “for all relevant facilities and asset classes which shall incorporate all relevant and material information, including conditions relating to an economic downturn where such information is necessary to duly capture the relevant risk”.
6.2. The requirement to incorporate economic downturn introduces a caveat, which
some IRB banks have often interpreted to suggest that there may be instances where loss severities do not exhibit cyclical variability and by extension their LGD

estimates may not differ materially from long run average LGD estimates.
6.3. Flowing from this view, is then the conclusion that downturn adjustments to LGD
estimates are therefore not required. Whilst this interpretation may hold in some cases, it is however important that IRB banks present empirical evidence in support of this.
6.4. After all, it is possible that different conclusions may be reached depending on
the data and analytical approaches used. In this regard, the PA has observed different data and analytical approaches even within the same bank for different portfolios. Some banks for instance compare long run default and loss frequencies with various macroeconomic or other credit indices to assess correlations. Other banks select particular periods, for example, of highest default frequency and assess that against macroeconomic or credit indices’ variables to assess if there are any correlations and specifically whether this correlation corresponds to periods of economic downturn. Other banks include accounting metrics in their analysis, for example, non-performing loans (NPL) coverage ratios. Moreover, some banks rely on macroeconomic variables, whereas others construct credit indices that take into account more granular industry variables relevant to their portfolios. Moreover, methodologically, these analytical approaches lie on a continuum ranging from technical and mathematically based to expert judgement-based approaches.
6.5. The PA does not consider it appropriate to express any preference on any of
these methodologies or variables, suffice to state that IRB banks should apply a high standard in their analysis before concluding that there is no need for downturn adjustments. That in other words means that the downturn adjustments should be the default position unless compelling empirical evidence can be produced to justify otherwise. In this regard, the PA will expect to see evidence demonstrating that IRB banks considered various analytical approaches and a broad range of variables and data to support their conclusions.
6.6. In cases where banks apply a similar methodology across all models,
assessments of their appropriateness must be performed per model, since what works for some models or portfolios may not work for others.
6.7. In principle also, the PA expects IRB banks to develop their own methodologies
to take account of internal experiences and the structure of portfolios. However, the PA accepts that data limitations-especially for low default portfolios-are likely to make the development of these methodologies challenging, in which case empirical evidence may be sufficient to demonstrate that the Federal Reserve Formula (Fed Formula) is appropriately conservative against actual experience. Whilst the Fed Formula is a crude adjustment with little to no conceptual basis, the PA nonetheless accepts that data limitations may make this the only viable option for some wholesale portfolios and will therefore not object to its use if sufficiently justified.
6.8. Nonetheless, several factors will guide the PA’s considerations of downturn
adjustments during its model reviews:
6.8.1. Different portfolios will react differently to economic cycles, such that they will in

turn exhibit different “downturn”, default and loss dynamics. In other words, some portfolios may react immediately to cycle swings compared to others that may react with a lag. Different analytical approaches may therefore be warranted depending on portfolio compositions. The PA will therefore look for evidence that a diverse range of underlying portfolio characteristics were assessed to arrive at a conclusion that a downturn adjustment is not necessary, and in cases where an adjustment has been included, factors informing the magnitude of that adjustment.
6.8.2. There is certainly an added complexity of complying with this requirement when
using external data, but the PA will nonetheless expect IRB banks to make every effort to understand the composition of the external data sources along the lines of industry and country compositions and the default and loss trends flowing out of these dynamics, and in turn arriving at an appropriate adjustment of internal estimates based on insights gleaned from the external data.
6.8.3. The analysis and adjustments must also be commensurate with the materiality
and complexity of the portfolios. For instance, retail portfolios whilst material in size, are otherwise data rich and therefore analytics in these cases will be largely empirically driven, arguably less complex and at an aggregated level compared to wholesale portfolios where multiple dimensions may require considerations and therefore granular and relatively more complex analytical approaches that combine quantitative and qualitative factors.
7. The treatment of stale ratings
7.1. The Regulations require IRB banks to regularly review and refresh ratings, but at
least annually, to keep them current and up to date, but more importantly to reflect any relevant information that has an impact on the credit risk of exposures as and when it becomes available. It is accordingly sensible that when up to date information is not readily available, IRB banks default to punitive ratings to mitigate possible underestimation of credit risk.
7.2. The PA has however observed diverse approaches in terms of the treatment of
stale ratings. Some banks default to punitive internal ratings, whereas others apply the standardised approach. However, the size of these exposures is generally immaterial at an aggregate level, although often significant for certain portfolios, such that they result in notable disparities in RWAs across banks.
7.3. Similar to the guidance on override rates, the PA expects IRB banks to put in place
portfolio level thresholds to monitor portfolios against, and to identify portfolios that consistently breach these thresholds and if warranted, to then institute corrective actions. In addition, these monitoring mechanisms should consist of portfolio level breakdowns to provide granular information that may point for instance to systems issues around data collection. These reports should be shared regularly with relevant stakeholders, and action items agreed to have in place clear timelines for remediation.
8. Validation
8.1. Regulations 23(11)(b)(v)(H)(vii) and 23(13)(a)(i) of the Regulations require IRB
banks to regularly -but at least once a year- subject the credit risk models used in

the RWA calculation to a validation process to assess methodological soundness and robustness, completeness, accuracy and appropriateness.
8.2. Moreover, regulation 39(7) of the Regulations requires this validation process to
be the responsibility of an independent function. This function must be functionally independent from personnel and management functions, business units and lines responsible for the origination of credit exposures. These requirements have several practical implications:
8.2.1. Firstly, it means that the function must be adequately resourced with suitably
qualified and experienced personnel. It also means that it must enjoy a sufficient amount of independence and credibility, so that it is in a position to provide reliable and credible independent assessments to senior management, board of directors and the PA on the overall credibility of the credit models used by IRB banks.
8.2.2. Independence and credibility in this regard refer amongst other things, to the
validation unit’s ability to define their work plans, agree timelines with model owners and users and thereafter carry out their validation activities without any undue interference or pressure.
8.2.3. It also refers to the validation unit’s ability to independently arrive at conclusions
on model performance, make findings where warranted and recommend remedial actions with timelines. By extension, it refers to the extent to which business units and model development teams adhere to these timelines.
8.2.4. Above all, it refers to the validation unit’s representation or participation on all key
committees and governance structures. What this means in practice will differ between banks, although as a minimum the common practice is for validation units to have membership of technical forums, and participation or representation in senior committees (executive and even designated committees).
8.2.5. Escalation mechanisms are also an important measure of effectiveness. In other
words, these units should have clearly documented channels for escalating issues of material concern regarding the development, validation and implementation of models to relevant senior management and board committees.
8.3. Secondly, it requires policies that set out in clear terms roles and responsibilities
of validation units. For instance, in some banks, the responsibilities for validating all models-regulatory and otherwise-rest solely with the units, whereas in other banks, these are shared with other units. Clearly defined roles and accountability not only reduce possible duplications, but also the risks of some models falling out of validation scope.
8.4. In terms of validation techniques, the validation performed should include a wide
range of qualitative and quantitative tests to assess various aspects of the data and its inputs. Table 2 summarises some of the dimensions and quantitative measures used widely by IRB banks. The list is not necessarily exhaustive, and banks may wish to add additional dimensions and tests. The PA does not expect IRB banks to use all the tests in the table, and some tests may work better and produce useful insights for some models compared to others. In addition, the PA is cognisant that for low default portfolios a significant number of these tests may

not produce satisfactory results.
Dimension Description of intended test Measures Data quality Review data quality for completeness and accuracy Missing rates Outliers Model design Model design methodology and compliance with regulatory requirements Length of development datasets Adherence to definition of default Incorporation of downturn conditions Model usage Review of internal usage of model, including override rates and re-rating frequencies Override rates Number of unrated and stale ratings Model uncertainty Review of model under extreme conditions or given changes in input data Stressing certain variables or testing the models using data covering periods of stress Calibration Ability to estimate correctly Herfindal Index Chi-Square test Brier score test Mean square error Goodness of fit Binomial test Wilcoxon Singed Ranked Test Stability Review stability of model and its variables Population stability index (PSI)
8.5. In terms of data quality:
8.5.1. The validation unit should assess amongst other things, completeness and
relevance. Completeness in this case means not excluding datasets that will result in unfavorable outcomes, either in terms of higher default or loss estimates. In cases where development teams intend to exclude certain periods or portions of the data, reasons for such exclusions must be documented and empirically justified. An assurance should also be given by the validation unit that these exclusions will not result in any undue bias in the final models.
8.5.2. In some cases, datasets may sit on multiple systems, possibly stored in different
formats and configurations. This has consistency and quality implications when performing aggregations for model development purposes. It remains the primary responsibility of development teams to ensure that any consistencies, be it in terms of definitions, time periods or even quality, are adequately addressed prior to commencement of development. Validation units in turn should provide independent assurance that the aggregation process results in consistent and reliable datasets.
8.5.3. Any transformations or adjustments must be assessed for sensibility and possible
impact on credibility of the resulting datasets.
8.6. In terms of model design:

8.6.1. Assessments should include tests around rating philosophy, that is, whether PD
estimates are through the cycle in line with regulatory requirements, as well as the appropriateness of downturn adjustments for LGDs. Adjustments of PDs and LGDs in this regard is admittedly not an exact science, and IRB banks should accordingly use various approaches ranging from quantitative to judgement￾based approaches. That in turn means that various validation approaches and factors will guide the assessments. Some banks may use bank-wide approaches, whereas other model specific approaches. There are pros and cons to each - including for instance a lack of data for low default portfolios- and the fact that bank-wide methodologies may be appropriate for some models and not for others. Regardless of approach, model specific assessments should always be performed to ensure that appropriate adjustments are made and comply with the letter and spirit of the respective requirements specified in the Regulations.
8.6.2. Real estate models rely on multiple inputs and relatively complex methodologies.
Data is often a challenge and development teams rely on expert judgement or external data. What is more, even though data may be available, it sometimes does not support underlying assumptions of the models. An example is property growth rates inputs, that are generally assumed to correlate with macroeconomic variables, although empirical evidence is not always conclusive in this regard. Vacancy rates is another important input, yet data to model this accurately is often sparse in some banks, ultimately with a lot of reliance placed on simulations for instance Brownian motion, most often with little to no justification of its appropriateness.
8.6.3. The PA expects validation to assess these various elements, and this invariably
requires the use of non-traditional validation tools that are a combination of quantitative and qualitative tests. This in turn calls for the expansion of the validation toolkit that is more relevant to real estate models and more generally low default portfolio models. A common test used by some banks is a correlation assessment of model outputs and expert rankings. Benchmarking against external data sources and hypothetical portfolios is also gaining some prominence. Pillar 3 disclosures also provide a useful source of peer comparison data, although consistency should call for caution.
9. Acknowledgement of receipt
9.1. Kindly ensure that a copy of this guidance note is made available to your
institution’s independent auditors. The attached acknowledgment of receipt, duly completed and signed by the Chief Executive Officer of the institution should be returned to the PA at the earliest convenience of the aforementioned signatory. Fundi Tshazibana Chief Executive Officer Date:
The previous guidance note issued was Guidance Note 8/2022, dated 20 June 2022. 29/07/22