Increased Requirements for Banks in Light of Operational Problems in Easter 2011

CIRCULAR
CIRCULAR:
20/2011
DATE: 15.06.2011
APPLIES TO: Banks
FINANS TILSYNET
Postboks 1187 Sentrum
0107 Oslo

Increased Requirements for Banks in Light of Operational Problems in Easter 2011 2 | Finanstilsynet

1 Introduction
Finanstilsynet has evaluated the events in Easter 2011 that affected banks' payment services in the card area, with subsequent serious consequences for the banks' customers. Based on reports from the banks and their suppliers, Finanstilsynet has obtained an overview of the cause of the error, cascading errors, and consequences of the events. 140,000 customers were affected in various ways by the operational problems, either when using cards in the banks' ATMs or at the banks' EFT/POS solutions at merchant locations. In the period from Wednesday, April 20, 2011, until all corrections were made on Wednesday, April 27, over 200,000 transactions were rejected and 240,000 reservations had to be deleted.

In addition to a description of the events, the circular provides information on which measures Finanstilsynet considers necessary as part of preventing such situations from recurring.

2 Details on the events in Easter 2011
The main cause of the events lies in a change carried out by EDB ErgoGroup ASA (EDB) in the autumn of 2010. A contingency test revealed a need to upgrade a central component (IssuerGatewayServer) in the card system solution. The solution was set up as a redundant system with a primary and a secondary server. When carrying out the change, only the primary server was upgraded due to lack of access to upgrade equipment for the secondary server. Serious errors were identified, both in the handling of the change and in the responsible technical unit at EDB, which did not follow up on the deficiency correctly. When a fault occurred on the primary server (hardware) on Wednesday, April 20, 2011, it turned out that the secondary server did not have sufficient capacity to handle the traffic volume.

The error resulted in a number of cascading errors when using cards in ATMs and shops:

• Cards were rejected in ATMs. This was due to a lack of response to the request for acceptance from the ATM system for withdrawals, which was caused by sluggishness in the IGWS solution.
• Cards were rejected at merchant locations, due to a lack of response to the request for acceptance of the payment at the merchant location, which was caused by sluggishness in the IGWS solution.
• Cards (same payment transaction) were attempted to be used again by the customer, with the consequence that either the capital transaction was updated twice, or an "advice message" (basis for updating the available amount to the pseudo-system (reservation of amount)) was generated twice.
• A large volume of advice messages were, due to the problems at EDB, standing in a queue at Nets Norway AS (Nets). When these were later updated at EDB, the capital transactions (EFT part) had already been updated, and subsequent reservation of the amounts should not have occurred. For the banks' customers, this is perceived as double posting/reduction of available balance.

Problems at merchant locations and ATMs started on Wednesday, April 20, 2011, at 10:00, and lasted until 17:00 the same day. Consequences of the error for the banks' customers and corrections of cascading errors continued in practice until Wednesday, April 27, when the last corrections were carried out.

Increased Requirements for Banks in Light of Operational Problems in Easter 2011 Finanstilsynet | 3

EDB's crisis team was quickly established to handle the deviation. Initially, the work was mainly concentrated on solving the fault on the relevant IGWS server. The IGWS solution is a component that is part of a longer transaction chain. EDB's crisis team did not have the necessary insight into the service areas and all elements that are part of the transaction chain. They were therefore unable to prevent the extent of the cascading errors. The error handling shows that the contingency organization must also consist of representatives from the banks and from other cooperating suppliers, in this case Nets.

A contingency organization with participation from banks and Nets could have limited the consequences of the error that occurred. Establishing a contingency setup that also includes banks and trading partners is a responsibility that lies with the banks.

Nets takes care of important functional areas for the banks' card systems. Since EDB takes care of important functional areas in the same areas, it is necessary with better cooperation between EDB, Nets, and the banks. Consequences of the error that occurred at EDB were made visible both at the merchant locations and at Nets, which takes care of tasks related to the network, collection, and settlement. Nets followed the established procedures managed by the Financial Industry's Joint Organization (FNO) and the Banks' Standardization Office (BSK) on behalf of the banks. Nets monitored the transaction traffic through its monitoring and follow-up systems and thus had information about the problems that accumulated. On this basis, Nets could have given a clearer warning both to the banks and to EDB about the extent of the problem and the risk that advice messages would reach EDB only after the capital transactions had been settled. This means that there is also a need at Nets to have a contingency organization that includes representatives from the banks and important suppliers that Nets must cooperate with. According to what Finanstilsynet knows, FNO will review the current regulations for BankAxept ("The Blue Book") so that this can be adapted to a crisis situation better than the current regulations.

3 Measures and Follow-up
3.1 Banks' Responsibility
The operational problems highlight vulnerabilities in the transaction chain and show the necessity for banks to take clearer responsibility for the part of the transaction chain operated by external suppliers.

Banks are responsible for all activities that constitute the bank's business. This is particularly clarified in the Regulation on Risk Management and Internal Control and in the ICT Regulation, which emphasize the banks' responsibility also for tasks that are outsourced. This responsibility applies fully even if the error has actually occurred with the supplier. The question is whether errors, such as in this current case, could have been limited if the banks had exercised their responsibility to a greater extent and in a more active manner.

Necessary measures:
Banks must set concrete requirements for the suppliers and their work, and ensure that the work is carried out in accordance with the agreement, current guidelines, and applicable regulations, through the exercise of active management and control of the deliveries.

Increased Requirements for Banks in Light of Operational Problems in Easter 2011 4 | Finanstilsynet

3.2 Mapping of Critical Components
It is of essential importance for the banks' business that the banks have identified the components in their ICT infrastructure that represent critical functions so that payment systems and customer account operations have sufficient availability. Reference is made to the ICT Regulation's § 3 (Risk Analysis), § 10 (Requirements for Continuity), and § 11 (Operational Disruption and Disaster Preparedness), which set requirements in this area.

Necessary measures:

• Banks must map and document critical components in their ICT infrastructure and other necessary elements that are part of the transaction chain, including the components located with external suppliers. The documentation must include an assessment of risk, security of continuity, and how contingency is secured.
• The bank's internal audit¹ must confirm that the mapping and documentation have been carried out in a sound manner.
• The documentation, together with the internal audit's confirmation, must be presented to the bank's board. Finanstilsynet assumes that the risk assessment related to critical components in the ICT infrastructure is included as part of the annual assessment of the risk situation, cf. Regulation on Risk Management and Internal Control § 8.
• A copy of the documentation after the review in 2011 and the internal audit's confirmation must be sent to Finanstilsynet by December 31, 2011.

3.3 Coordinated Contingency
It is not satisfactory that banks and important suppliers do not have coordinated contingency. The events in Easter 2011 show that the individual supplier does not have sufficient insight to limit the consequences of a serious event. Precise and timely information in the relevant channels can be decisive for handling a problem. With regard to requirements for contingency setups, reference is made to the ICT Regulation's § 11 (Operational Disruption and Disaster Preparedness).

Banks must ensure that the contingency is coordinated with the suppliers' contingency organizations where this is considered appropriate (supplier that has significance for the bank's business), and that exercises are conducted to ensure that the total contingency organization functions.

Each bank provides Finanstilsynet with a description of how this is resolved by December 31, 2011.

¹ Applies to those banks that have internal audit.

Increased Requirements for Banks in Light of Operational Problems in Easter 2011 Finanstilsynet | 5

3.4 Change Control
Necessary measures:
Banks must evaluate how they can, in a suitable manner and to a greater extent than today, participate in change handling at suppliers. This applies where the banks' solutions are directly affected by the changes, and for critical components that can directly affect the banks' payment systems and/or customer/account area. Reference is made here to the ICT Regulation's § 9 (Deviation and Change Handling).

A description of how each bank will ensure more direct participation in change handling must be sent to Finanstilsynet by December 31, 2011.

Emil R. Steffensen
Anne Merethe Bellamy

Contact persons:
Section Chief Frank Robert Berg, tel. 22 93 98 47, e-mail: frank.robert.berg@finanstilsynet.no
Supervisory Advisor Stig Ulstein, tel. 22 93 99 66, e-mail: stig.ulstein@finanstilsynet.no

POST@FINANSTILSYNET.NO
WWW.FINANSTILSYNET.NO
Finanstilsynet
Postboks 1187 Sentrum
0107 Oslo