DORA Update 5: Testing Digital Operational Resilience

SUPERVISION DORA UPDATE 5 SEPTEMBER 2024 Getting Started with DORA: Testing Digital Operational Resilience In brief This is the fifth edition in a series of AFM publications on the Digital Operational Resilience Act (DORA). This series is intended for all enterprises that must comply with this European regulation from 2025 onwards. In this edition, we focus on testing digital operational resilience. This allows enterprises to analyze their current status in this area and determine any further steps they may need to take to comply with the regulation.

1. Introduction 1 A financial entity that is not a trading platform, central counterparty, transaction register or central securities depository, and where fewer than 10 persons are employed and where the annual turnover and/or annual total balance sheet do not exceed €2 million DORA aims to ensure that financial institutions better manage ICT risks and thereby become more resilient to cyber threats and ICT disruptions. To this end, the regulation describes various requirements in the field of ICT, including for testing digital operational resilience. Enterprises can already analyze whether they comply with the DORA requirements in this regard, and then take action (if necessary). To comply with DORA by 17 January 2025, it is necessary to start implementing it now. In previous editions, the DORA requirements for managing ICT risks (including those of third-party providers) and ICT incident management were discussed. DORA expects financial institutions to take sufficient measures and establish processes that help improve information security and cyber resilience. To ensure that these measures are adequate, it is important that ICT tools and systems are tested regularly to expose any vulnerabilities and shortcomings. By regularly testing the resilience of ICT tools and systems, enterprises can ensure the continuity of important and critical functions in the event of a disruption. The requirements for testing digital operational resilience are described in Articles 24 to 27 (Chapter IV) of the regulation. Article 24 describes the general requirements applicable to carrying out tests. This includes, among other things, how organizations must draw up a test program, how often tests must be carried out, and how findings must be followed up. These requirements apply to all enterprises subject to DORA, with the exception of micro-enterprises1. Micro-enterprises are expected to adopt a risk-based approach. Article 25 describes which tests can be used for testing ICT tools and systems. Articles 26 and 27 of the regulation describe the requirements for advanced testing based on threat-led penetration testing (TLPT), where multiple important or critical functions are tested (in the production environment). These articles describe, among other things, the scope of these tests, the role of supervisors, and how it is determined which financial institutions must carry out TLPT. In addition, the requirements for testers are described for carrying out TLPT. The requirements for TLPT are further elaborated in the Regulatory Technical Standard (RTS)2. This provides more detailed information on the criteria on the basis of which enterprises can be designated for TLPT and the requirements that apply to the use of internal testers. Furthermore, the RTS explains the TLPT process. This process is based on the TIBER-EU framework. The AFM has been accompanying TIBER tests for financial enterprises since 2021, in collaboration with DNB3. In this DORA update, we will delve deeper into the general requirements for testing digital operational resilience and where organizations should start now to be able to comply with DORA. We will also discuss the requirements regarding TLPT and the process for such tests. 2 See Regulatory Technical Standards (RTS) and Implementing Technical Standards (ITS) 3 See TIBER-NL programme (afm.nl) Table 1 Supplementary elaborations Subject Completed RTS for Article 26 (11) Advanced testing of ICT tools, systems and processes based on TLPT Submitted to the EC in the meantime

Getting Started with DORA: Testing Digital Operational Resilience 2 SUPERVISION DORA UPDATE 5 2. Getting Started with Testing Digital Operational Resilience 4 For more information on the ICT risk management framework, see DORA update 3 (Publications (afm.nl)) 5 The requirements in Chapter IV of the regulation must be applied in proportion to the size and overall risk profile of the enterprise and to the nature, scale and complexity of their services, activities and transactions. Testing ICT tools and systems Enterprises can already start with: • Drawing up a risk-based program for testing digital operational resilience; • Executing the test program. Article 24 of the regulation describes the general requirements for testing digital operational resilience. To systematically assess the resilience of ICT systems and services, enterprises must draw up, execute and regularly evaluate a test program. This test program must be part of the ICT risk management framework4 and contains the tests, practices, methodologies and tools that are carried out regularly to assess the organization's ICT systems, tools and processes. The assessment looks at the processes established to detect and handle ICT-related incidents in a timely manner. In addition, enterprises must themselves assess the extent to which they are capable of recognizing vulnerabilities and shortcomings in digital resilience. Finally, the tests provide insight into the extent to which the organization can implement timely remedial measures that minimize the duration and impact of a disruption. When determining the tests, the enterprise must take into account, among other things, the (changing) landscape of ICT risks, specific ICT risks for the enterprise and the critical nature of the ICT systems and services. Organizations are expected to test their ICT systems and applications that support critical or important functions at least once a year. These tests can be carried out by both internal and external testers. It is important here that (potential) conflicts of interest are prevented during the design and execution phases of the test. When internal testers are used, enterprises must take sufficient measures to ensure that the testers have no interest in the results of the tests. Furthermore, financial institutions must establish policies and procedures to classify, prioritize and follow up on all shortcomings that have emerged during the execution of the tests. They must also check whether all identified vulnerabilities and shortcomings have been fully addressed. Article 25 of the regulation describes the tests that enterprises can carry out to test their ICT systems and applications. Here, enterprises determine which tests are relevant in accordance with the principle of proportionality5. Examples of tests are: • Vulnerability scans. Here, the security of ICT systems and tools is assessed and vulnerabilities (often via automated scans) are identified; • Gap analyses, where the current state of the ICT systems and tools is compared with the desired state. Based on these analyses, it can be determined which systems do and do not meet the requirements; • Physical security assessments. Think here of tests to determine whether people can gain unauthorized access to certain locations (such as offices, data centers, etc.); • Source code assessments where developers who did not write the code themselves check the source code before it goes to the production environment; • Compatibility tests: a form of testing where the operation of the software is tested in different environments (software/hardware platforms, networks, browsers, etc.); • End-to-end testing: a form of testing where the application is tested from start to finish to verify that all components also work in realistic scenarios; • Penetration tests: tests where the (often external) tester tries to exploit vulnerabilities to gain access to the system. The above requirements do not apply to micro-enterprises. For micro-enterprises, a risk-based approach applies, taking into account on the one hand the amount of resources and time required to carry out the tests, and on the other hand the urgencies, the type of risk, the critical nature of the ICT system and any other relevant factors. Advanced testing of ICT tools, systems and processes For some enterprises, in addition to the (general) requirements for testing ICT systems and tools, additional requirements apply regarding testing digital operational resilience. These institutions must carry out an advanced test once every three years6 using threat-led penetration testing (TLPT). TLPT involves an extensive test in which the tactics, techniques and procedures used in practice by threat actors (such as hackers) are simulated7. This allows the cyber resilience of the financial institution to be tested in a controlled, institution-specific and intelligence-driven manner. Such a test forms an extended form of red-teaming, partly due to the involvement of the supervisor. To successfully carry out TLPT, it is important that the enterprise views this test as an opportunity to learn and expose any vulnerabilities. Furthermore, TLPT is demanding, which is why it is important for enterprises to allocate sufficient resources and personnel during the various phases of the test. Each TLPT must relate to the critical or important business functions of the enterprise and must be carried out on the (live) production systems that support these business functions. For this, it is important to first identify the ICT systems, processes, tools and (outsourced) services that support critical or important functions. The enterprise then determines which critical or important functions must be part of the test. Before the scope of the test can be finalized, it must be validated by the TLPT authority overseeing the test. In the Netherlands, this will be done by the AFM or DNB, depending on the competent authority that granted the license. When third-party ICT service providers fall within the scope of a TLPT, the enterprise must take sufficient measures to ensure the participation of the third-party providers in the TLPT. In the event that participation in TLPT would have a negative effect on the quality of the third-party provider's services to organizations not subject to DORA, the external service provider may be excluded from the scope of the enterprise's TLPT. In that case, the third-party provider must itself appoint an external tester to carry out a pooled TLPT (or pooled test), in which multiple financial institutions to which ICT services are provided are involved. This pooled test must cover all ICT services outsourced to the third party and support critical or important functions of the various financial entities. To ensure that TLPT is carried out correctly, there are a number of requirements for the testers (or red team). Enterprises must use testers who are certified in a Member State by a certification body and/or comply with formal codes of conduct/ethical frameworks. In addition, testers must possess sufficient technical and organizational capabilities and be able to demonstrate relevant knowledge in the field of threat intelligence, penetration testing and red-teaming. Finally, testers must be able to guarantee the independent execution of the test and the confidentiality of information (such as test results). In the event that internal testers are used, the enterprise must additionally allocate sufficient resources to prevent conflicts of interest, and the use of internal testers must be approved by the TLPT authority overseeing the test. The requirements for the use of internal testers are further elaborated in Chapter IV of the RTS. For the designation of enterprises that must carry out TLPT, the TLPT authority determines which financial entities are designated to perform TLPT. Depending on the licensing authority, the AFM or DNB will be responsible for the designation (and supervision) of TLPT. Some financial institutions may be designated by both the AFM and DNB. In that case, it may be decided that a joint test is carried out, with both supervisors involved. The criteria for designating enterprises are described in Chapter II of the RTS. The TLPT authority takes proportionality into account here. Institutions can be designated for TLPT on the basis of 'hard (or quantitative) criteria'. These include, for example, trading platforms with a certain market share at national or European level. These hard criteria are described in Article 2(1) of the RTS. Institutions not designated on the basis of quantitative criteria may still be designated to be required to carry out TLPT on the basis of their ICT risk profile, systemic character and impact on the stability of the financial sector. More concretely, institutions can be designated on the basis of the following (predominantly qualitative) factors: • Systemic character and impact-related factors:

• The size of the institution;
• The degree and nature of the interconnectedness of the institution with other financial institutions in the financial sector; 8 Small and non-interconnected investment firms, payment institutions exempted under Directive (EU) 2015/2366; institutions exempted under Directive 2013/36/EU for which Member States have decided not to apply the option referred to in Article 2(4) of DORA; electronic money institutions exempted under Directive 2009/110/EC, and small occupational pension institutions.
• The importance of the services offered;
• The substitutability of the services offered;
• The complexity of the institution's business model;
• Whether the institution is part of a group using shared ICT systems. • ICT risk-related factors:
• The risk profile and threat landscape of the institution;
• The extent to which critical, important or supporting business functions depend on ICT systems and processes;
• The complexity of the institution's ICT architecture;
• Outcomes of any supervisory investigations relevant to the assessment of the ICT maturity of the financial entity;
• The maturity of ICT business continuity plans and ICT recovery and response plans;
• The maturity of operational ICT security detection and mitigation measures;
• Whether the institution is part of a group using shared ICT systems. Finally, micro-enterprises and enterprises for which the simplified ICT risk management framework8 applies cannot be designated for TLPT. In the next section, we will delve deeper into the TLPT process and the different roles of the involved parties. Table 2 Supplementary elaborations Description Completed RTS for Article 26 (11) Advanced testing of ICT tools, systems and processes based on TLPT Submitted to the EC in the meantime

Getting Started with DORA: Testing Digital Operational Resilience 6 SUPERVISION DORA UPDATE 5 TLPT process To ensure that each TLPT is carried out correctly, a number of requirements regarding the test methodology and the TLPT process have been elaborated in Chapter III of the RTS. To begin, enterprises assess the risks associated with carrying out TLPT. Sufficient measures must be taken to prevent these risks from materializing as a result of the testing activities. This concerns risks regarding: • granting access to sensitive data to external parties; • non-compliance with TLPT requirements; • crisis/incident management; • disruptions to critical activities and processes; • loss of data as a result of the testing activities; • failure to fully restore systems affected by the test. Before the testing activities can be carried out, it must be clear what the role of everyone involved in the execution of the TLPT is. From the authority overseeing the TLPT (for example, the AFM or DNB), a test manager must be appointed who oversees the testing activities and ensures that all requirements are met during execution. In addition, at least one deputy must be appointed who can take over the tasks of the test manager if necessary. The authority overseeing the TLPT is expected to be involved in all phases of the testing activities and to provide timely feedback, validation or approval when needed. The enterprise must ensure that it has a team of employees involved in the execution of the test (the control team or white team), one of whom is appointed as team leader. The control team is kept informed of every finding resulting from the TLPT. This applies to findings from both personnel within the organization itself and from external service providers. Any follow-up actions for incidents arising from the test are handled by the team itself, and information on the progress of the testing activities must be shared with the test managers when requested. Finally, sufficient measures must be taken to guarantee the confidentiality of the TLPT. Access to information about the TLPT is thus limited to the control team, the management body of the enterprise, the TLPT authority, the threat intelligence providers and the testers. Here, the threat intelligence providers are external specialists who collect and analyze data on current threats and develop realistic scenarios based on this. The testers consist of (external) ethical hackers (or red team) who try to gain access to the enterprise's production systems. The blue team consists of employees of the enterprise itself, who try to protect the network and ICT systems from external attacks. The blue team is not involved in the test and is therefore not aware of the test. Preparation phase Once the enterprise receives a notification from the TLPT authority, it can begin preparing for the test. During the preparation phase, the enterprise carries out the risk assessment, looking at the risks associated with carrying out a test on the production environment of systems that support important and critical business functions. In addition, before the start of the test, a project charter9, the contact details of the control team leader and information on the use of internal or external testers must be provided to the TLPT authority. Furthermore, information is shared with the TLPT authority regarding the communication channels during the execution of the testing activities and the codename used for the test. This information must be shared with the test managers no later than three months after the notification from the TLPT authority. Enterprises designated by the AFM for TLPT can submit the required reports via the AFM portal10. Once this documentation has been approved by the test managers, the enterprise can set up the control team, which supports the team leader in preparing for the test. Once the composition of the control team has been approved by the TLPT authority, the enterprise determines which critical and important business functions will be included in the execution of the testing activities. This takes into account, among other things, the importance of the function for the enterprise and the stability of the financial sector, the substitutability of the function, the interconnectedness with other functions and the geographical location of the function. Once the scope of the test has been determined, it must be approved by the management body and shared with the test managers11. This report must be shared no later than six months after the notification from the TLPT authority. Once the submitted reports have been approved by the test managers, they can be shared with the testers and threat intelligence providers. The enterprise ensures that both the red team and the threat intelligence providers are contracted before the start of the testing phase. Testing phase The testing phase can be divided into two parts: threat intelligence and red-teaming. Once the scope of the test has been approved by the TLPT authority, the threat intelligence provider must conduct research on the enterprise and threats and vulnerabilities relevant to the enterprise. During this research, they collect data by identifying and analyzing cyber threats in the sector and identifying existing and potential vulnerabilities that can be exploited during the test. For this step, the threat intelligence providers may consult with the control team and the test managers overseeing the test. Based on this analysis, the threat intelligence provider will draw up a number of scenarios that can be used during the execution of the test. The team leader of the control team then selects... [text cuts off at "scope sp"]