How common key indicator errors can increase risk and reduce resiliency.
This is the second in a series of four blogs about the ways in which common shortcuts can undermine overall operational risk management success within organizations. You can access the first blog here Risk and Control Self Assessments.
Everyone loves metrics, and particularly key indicators such as key performance indicators (KPIs), key risk indicators (KRIs), and key control indicators (KCIs). In a perfect world, you’ll be using risk management software and each data point should provide the organization with importance intelligence. However, most operational risk teams wind up burdened with hundreds of KPIs, KRIs and KCIs, and then find they cannot see the forest for the trees. At some point, most operational risk teams will find themselves needing to sit down and shake up their stock of key indicators. And that’s where the temptation to use shortcuts can arise. Common shortcut mistakes include:
- Deeming the KPIs to be KRIs – Often the business will have a list of KPIs that they track for operational and financial purposes. There can be a temptation to just adopt all of these KPIs as KRIs, and call it a day. However, KRIs are just what they say they are – and so they need to be identified with a specific risk. For example, a KPI that everyone looks at is profitability, and this is an excellent KPI. However, it tells the operational risk team almost nothing specific about the way the firm is managing any of its operational risks or controls. Operational risk teams should challenge the use of KPIs as KRIs by asking which specific risk or control a KPI relates to.
- Regarding KCIs as KRIs – Once again, it’s important to distinguish between what is a KRI and what is a KCI, and to treat them differently. KCIs are vitally important for an operational risk program – they measure the strength of the control environment. Treating them as KRIs leads to failing to consider the control environment on its own. KCIs should be paired with a relevant KRI, for maximum operational risk framework robustness. As well, they should be tested regularly through the control testing that the operational risk team performs.
- Using the data to set the thresholds – Although using the data might seem to be a straight-forward way to set the thresholds of key indicators, it is deeply problematic. For example, some operational risk teams will look at the data for a key indicator and say to themselves, “we’re generally where we want to be, but sometimes we’re not so that must be amber, and remember that really big one two years ago, that must be red.” However, this thinking doesn’t really capture what is happening on the ground within the business. In fact, the process the key indicator is capturing may be experiencing problems, so what looks like “green” from the data might actually be amber or red for the team. It’s important to have a conversation with the business for each KRI and KCI being used, to set the thresholds according to what is right for the business.
- Believing all KRIs are early warning signals – Certainly some KRIs are early warning signals. In particular, those KRIs which relate to the likelihood of a risk happening can fall into this category. However, those KRIs which are indicators of the size of the impact of a risk event are not early warning indicators – they simply indicate the possible magnitude of the impact. As a result, impact KRIs are more of a lagging indicator. An example of a good operationa risk early warning signal is a staff turnover metric. Staff leaving a settlements processing team leads to a loss of operational history as well as the need to train new staff. Existing staff can become demoralized and not train new team members properly – and so risk can rise. Therefore, indicators of likelihood are good early warning signals because the right ones can provide an insight that a risk is more likely to happen. However, in this example, the number of settlement fails would be an impact indicator – it is an indicator of the impact of losing key staff – and not providing an early warning of anything.
- Tossing out indicators that aren’t predictive – Most financial services organizations have far too many key indicators – they tend to accumulate over time. However, because there is a commonly-held belief that all key indicators are early warning signals, some operationa risk teams have simply put other types of key indicators in the bin. This is a big mistake. For example, KRIs that suggest the potential impact of a risk event, as well as KCIs are important for understanding operational resilience. Instead, operationa risk teams seeking to reduce the number of key indicators should look towards industry best practice and indicator effectiveness as benchmarks.
In short, it’s important to take a more thoughtful approach to restructuring an ecosystem of key indicators. Taking shortcuts can reduce the ability of certain KRIs to be predictive, and damage the firms’ understanding of its operational resiliency, through other KRIs and KCIs, as well.
