Although it’s been more than a decade since financial services firms have had to undertake stress tests and scenario analysis for operational risk, regular errors in both approach and execution continue to be made. Below are 7 of the top mistakes firms are making:
1. Confusing stress testing and scenarios
These really are two different types of exercises. Stress testing explores the impact of a single factor or risk by itself. In contrast, scenarios analysis is about understanding the impact of multiple factors, or risks, occurring within a short time.
The Prudential Regulation Authority’s Consultation Paper 1/15 spells this out. Unfortunately, many firms and regulators confuse the terms, for example, by saying scenario analysis when they mean single factor testing.
2. Asking SMEs what the worst year in 100 years looks like
It’s perfectly reasonable to ask subject matter experts (SMEs) what the impact is likely to be if the IT infrastructure goes down within a five year or 25 year horizon. That’s because those time frames are within the scope of an individual’s career.
However, asking someone about a 1-in-100-year event is not productive, simply because most people are not capable of conceptualizing that. Some regulators have actually asked their firms to stop asking SMEs about 1-in-100-year events on that basis. Instead, use multifactor scenario modelling with conditional probabilities, to arrive at something with a 1-in-200 year probability of happening.
3. Including scenarios and stress tests in the RCSA
Don’t do this. The risk and control self-assessment (RCSA) should be focused only on a short-term time horizon risks – risks that could happen in one, three or five years.
Trying to talk about longer time horizons in RCSAs just the confuses people trying to complete the RCSA. Work in severe but plausible events that are 1-in-25 years or longer in a separate exercise.
4. Not documenting the scenario / documenting it too much
Some people will do the scenario testing and fail to document it. For the regulator, if it’s not documented, it didn’t happen. So – document, document, document! On the other hand, don’t write a Hollywood screenplay of several hundred pages either.
It’s easy to see how the documentation can get long – for example, describing a typical multifactor scenario of the IT system going down, the disaster recovery site not working, and then someone with a gambling problem taking advantage of the situation and committing fraud. Remember it’s important to communicate concisely and clearly with the regulators, so keep the documentation down to what the regulator really needs to know.
5. Developing scenarios or stress tests that really aren’t (because they are not very exceptional)
One of the main reasons that regulators want senior management to engage with severe but plausible events through a scenario analysis program is to sensitize management to these kinds of things happening – to encourage them to think about what they would do. This is part of pushing organizations to be more operationally resilient.
Using scenarios that are not exceptional will not stretch management’s thinking in the way regulators are looking for.
6. Failing to consider secondary effects on the full risk profile of the firm
Secondary effects occur when controls that mitigate other risks – as well as one of the scenario risks – break down and create a loss. Many firms don’t look at secondary effects but regulators give the firms that do a big pat on the back. For example, when the IT system goes down and as a result an internal fraud happens, controls must have failed for those risk events to have occurred.
Firms should look at those failing controls and ask themselves, where else do those failing controls impact the firm? What other risk events could result? If the fraud indicates that ethics training has failed, what other kinds of risk events could occur as a result of ethics training failure?
7. Developing scenarios without input from the business
It’s common for the risk team to develop scenarios without input from either the business or senior management. This is a big mistake. Not only will regulators criticize the firm for doing this, but it’s a big missed opportunity when it comes to building operational resiliency. Op risk teams need to work with these stakeholders. The scenario-building exercise will help managers think more carefully about what could happen to a firm during a big loss event, and build operational resiliency within the business. Op risk teams should view their opportunity to engage in this way as career-enhancing.
Firms seeking to understand what types of scenarios they should be building a part of these exercises should look at regulatory documents for ideas about what may be of concern.
For example, in July 2009 the Basel Committee on Banking Supervision published Results from the 2008 Loss Data Collection Exercise for Operational Risk. Within that paper, a lot of single factor stress tests that firms were using – and continue to use today – are discussed in the tables. They can be combined to create a scenario analysis program.
In short, for firms to achieve maximum benefit from either stress testing or scenario analysis – both for the business and for its regulatory relationships – it’s important to build a program on a solid understanding of the right approaches.
To learn more about stress testing and scenario analysis, including consulting, training, and software, contact RiskLogix.
