16 October 2023
ESMA91-372-2811

Opinion

CCP back testing requirements under Article 49 of Regulation (EU) 648/2012 and Article 49 of Delegated Regulation (EU) 153/2013

1 Legal basis

1. ESMA’s competence to deliver an opinion to competent authorities is based on Article 29(1)(a) of Regulation (EC) No 1095/2010 of the European Parliament and of the Council of 24 November 2010 establishing a European Supervisory Authority (European Securities and Markets Authority) 1 (ESMA Regulation).

2. Pursuant to Article 29(1)(a) of ESMA Regulation, ESMA shall provide opinions to competent authorities for the purpose of building a common Union supervisory culture and consistent supervisory practices, as well as ensuring uniform procedures and consistent approaches throughout the Union.

3. In accordance with Article 23a (1) of Regulation (EU) 648/2012 of the European Parliament and of the Council of 4 July 2012 on OTC derivatives, central counterparties and trade repositories 2 (EMIR), ESMA shall fulfil a coordination role between competent authorities and across colleges with a view to building a common supervisory culture and consistent supervisory practices, ensuring uniform procedures and consistent approaches, and strengthening consistency in supervisory outcomes, especially with regard to supervisory areas which have a cross-border dimension or a possible cross-border impact.

2 Background

4. In accordance with paragraph 1 of Article 49 of EMIR, “A CCP shall regularly review the models and parameters adopted to calculate its margin requirements, default fund contributions, collateral requirements and other risk control mechanisms. It shall subject the models to rigorous and frequent stress tests to assess their resilience in extreme but plausible market conditions and shall perform back tests to assess the reliability of the methodology adopted” (emphasis added).

5. Article 49 of the Commission Delegated Regulation (EU) 153/2013 (“the Delegated Regulation”) further specifies the type of tests to be undertaken for different classes of financial instruments and portfolios. In particular, Article 49(1) of the Delegated Regulation requires CCP to assess their margin coverage through back testing, i.e., by performing “an ex-post comparison of observed outcomes with expected outcomes derived from the use of margin models”.

[1] OJ L 331, 15.12.2010, p. 84–119, as amended by Regulation (EU) 2019/2175 of the European Parliament and of the Council of 18 December 2019.

[2] OJ L 201, 27.7.2012, p. 1–59, as amended by Regulation (EU) 2019/2099 of the European Parliament and of the Council of 23 October 2019 amending Regulation (EU) No 648/2012 as regards the procedures and authorities involved for the authorisation of CCPs and requirements for the recognition of third-country CCPs.

ESMA - 201-203 rue de Bercy - CS 80910 - 75589 Paris Cedex 12 - France - www.esma.europa.eu

6. The above-described back testing performed for the purpose of Article 49 of the Delegated Regulation is usually referred as margin adequacy back testing. These back tests are key to assess whether the CCP has collected enough financial resources to meet its coverage requirement at a given point in time.

7. Furthermore, Article 47 of the Delegated Regulation requires CCPs to conduct a comprehensive validation of its models and methodologies. In particular, paragraph 3 of Article 47 specifies that a comprehensive validation shall at least include an evaluation of the conceptual soundness of the models and framework, including developmental supporting evidence, a review of the adequacy and appropriateness of the models, and an analysis of the outcomes of testing results.

8. These back tests performed for the purpose of validation in accordance with Article 47 of the Delegated Regulation are usually referred to as core model back testing. They are particularly needed to assess the statistical performance of the margin model, e.g., to establish that an existing or proposed risk model is sound and that the statistical forecasting models are accurate. These tests help informing the CCP’s model validation, either where the CCP applies for a significant model change in accordance with Article 49 of EMIR, or where the CCP performs a periodic review of its models.

9. Further to the above, ESMA also notes that when performing back testing, CCPs should usually distinguish between two types of back tests:

a. Unit back tests, which assess the performance of the model for one given security or contract. It must be noted that tests over small theoretical portfolios representing typical3 trading strategies of spread positions (e.g. curve positions, basis positions, etc…) are also part of the set of unit back tests, as they demonstrate the performance of the model on these types of curve/basis/… positions.

b. Portfolio back tests, which assess the performance of the model for production portfolios (i.e., actual positions of the CCP’s clearing members), or, where relevant, theoretical portfolios.

10. The aim of this opinion is to bring some clarifications in relation to the implementation and use of back tests across EU CCPs, whether they are used as core model back tests (section 3.1) or margin adequacy back tests (section 3.2). In particular, this opinion aims at clarifying:

• The purpose and structure of each type of back test (core model vs. margin adequacy back test)

• Whether unit back tests and portfolio back tests are relevant for core model back testing (resp. margin adequacy back testing)

• Whether and how margin add-ons should be included in the back testing framework.

  11. ESMA is of the view that the above-mentioned clarifications in respect of Article 49 of EMIR and the Delegated Regulation would contribute to the common Union supervisory culture and consistent supervisory practices regarding CCPs, including in respect of the assessment by national competent authorities of the CCPs risk management models, and would ensure consistent approaches throughout the Union in respect of supervision of CCPs.

[3] By using the word “typical”, it is meant that the theoretical portfolios represent the risk factors involved in the product cleared, rather than an attempt at predicting future production portfolios.

3 Opinion

12. This opinion aims to clarify CCPs back testing practices under EMIR, and in particular where back testing is performed for the purpose of core model back testing for the purpose of model validation in accordance with Article 47 of the Delegated Regulation or for margin adequacy back testing in accordance with Article 49 of the Delegated Regulation. It aims at harmonising back testing practices across authorised EU CCPs but does not intend to limit CCPs' freedom of modelling, nor to prescribe a specific margin model.

3.1 Core model back testing

13. Core model back testing is essential to demonstrate that a model targeting a given confidence interval is achieving this performance, by comparing price shocks against the core margin model. It should evaluate how accurately the statistical models forecast the tails of the loss distributions, identify potential model weaknesses, evaluate the underlying assumptions and increase confidence that the models will be robust under changing market conditions.

14. Core model back testing should also evaluate exceedances (or exceptions), i.e. the number of observations where the losses at margin account level would have exceeded the margins held for this account. The frequency of exceptions should be consistent with the percentile tested.

15. However, the models are not expected to pass back tests (i.e., meet or exceed the expected confidence interval) on every contract or unit portfolio. Still, exceptions should not be clustered, i.e. concentrated within a subset of units with similar risk profile, as this would indicate the model is not capturing a certain risk factor that may materialise in trading positions. ESMA considers that as a minimum the failure rate should be consistent with the expected type-I (false positive) error rate of the chosen test, and the failures should not be concentrated within a subset of units with similar risk profiles, which would indicate model failure for that specific type of risk.

16. In addition, ESMA notes that the requirement for the confidence interval applies to all possible portfolios, including portfolios composed by one security or derivative contract4 . If the model is calibrated for a given interval, the output of the unit back test should be in line with the targeted confidence interval.

17. Finally, ESMA notes that CCPs generally rely on margin add-ons to reflect uncertainties or potential effects that are not accounted for in the core margin models. These add-ons usually cover market liquidity risk, wrong-way risk, concentration risk, credit event risk, or credit-risk add-ons that can be charged based on the credit-worthiness of a clearing participant.

18. Considering of all the above, ESMA is of the opinion that:

a. The practice consisting in relying solely on portfolio back-tests when conducting the assessment of a margin model is not sufficient to demonstrate its appropriateness. When

[4] For the avoidance of doubt this does not mean that portfolio margining is not accepted.

performing a margin model validation in accordance with Article 47 of the Delegated Regulation 5 a CCP should rely both on unit back tests and on portfolio back tests.

b. The results of unit back tests should demonstrate compliance with the targeted confidence interval at a unit level. For the avoidance of doubt, this means that the percentage of exceptions should be evaluated per unit back test not by grouping all observations across the unit back tests. There can be unit back tests showing more exceptions than expected under the required confidence interval, however the number of such exceptions should be coherent with statistical tests and should not be clustered in a specific type of unit positions.

c. Unit back-tests should include single positions as well as simple combinations of securities/derivatives representative of risk factors, such as curve trades, calendar spreads, call spreads, etc. When the number of underlying securities / points on a curve / ISIN per issuer is very high, the CCP may run the unit back tests on a representative subset of underlying securities / points in the curve / ISINs. However, the selection of the unit back tests should avoid introducing a bias (such as avoiding a part of the curve, a part of the possible option strikes…).

d. Margin add-ons covering risks which are not included in the historical price data and which are not based on a statistical model should either be removed (where not directly embedded in the CCP’s initial margin model) or consistently accounted for when assessing the performance of unit back tests and portfolio back tests used for the core model back testing. Such add-ons include but are not limited to the following:

i. add-ons for transactions or liquidation costs (representing bid-offers) when using historical mid-prices for the core model back testing;

ii. concentration charges / transaction costs stemming from large positions (i.e. the impact of liquidating the large position on the market beyond the regular bid-offer);

iii. add-ons to capture the risk of jump-to-default in a CDS portfolio when assessing the performance of the margin component targeting spread movements;

iv. add-ons used to limit portfolio offsets in line with Art 27(4) of the Delegated Regulation;

v. wrong-way risk charges;

In the context of core model back testing, the CCP should consider excluding the additional margins required as a result of the APC adjustment in accordance with Article 28 of the Delegated Regulation. The CCP should document and justify such determination in accordance with Article 47 of the Delegated Regulation.

Components or add-ons of the margin model that cannot be directly measured and tested through back-testing and which reflect particular portfolio characteristics, such as liquidity or concentration – should nevertheless be subjected to detailed validation for conceptual soundness and adequacy.

[5] For example as supporting evidence for the validation of a significant change in accordance with Article 49(1) of EMIR, or as part of the annual validation of models

3.2 Margin adequacy back testing

19. Margin adequacy back testing evaluates whether the CCP had sufficient initial margin to cover its losses on a specific historical day assuming a clearing member defaults. In other words, margin adequacy back testing evaluates whether the margins collected by the CCP are sufficient to comply with the EMIR minimum requirements and the CCP’s own risk appetite and provide empirical evidence on a regular basis that the CCP is meeting these requirements.

20. When performing margin adequacy back testing:

a. A CCP should use actual portfolios that are guaranteed by the CCP including the effect of transactions which are not yet settled, i.e. trades novated between the last margin payment and the time of the theoretical default need to be accounted for.

b. A CCP should consider only the margin requirement of the clearing member as of the time of the theoretical default. Any voluntary or excess collateral held by the CCP should be ignored.

c. Losses on the portfolios should be calculated consistently with the default timing assumptions. The change in portfolio value should be taken from the timestamp of the market data that was used to calculate variation margins (VM) for the last successful VM call until the time the CCP has liquidated the portfolio as the CCP is not covered by VM for market moves between these times, even if the VM is paid successfully in the morning.

d. As for core model back testing (see paragraph 18(d)), margin add-ons should be excluded from the margin adequacy back testing, unless they account for risks which are reflected in the price history.