In search of a Model to Measure Effectiveness

This paper sets out what effectiveness could mean in terms of a Bank’s response to fighting financial crime and how this can be measured and tested. It argues for a new focus on effectiveness as a better way to measure and assess financial crime compliance, as an alternative to the current approach of testing through checklists and file reviews to determine compliance with regulatory expectations.

Many leading FI’s believe compliance with regulatory rulebooks, whilst important, cannot be equated with fighting financial crime. Regulations are imposed largely as a reaction to past events, whereas fighting financial crime needs to learn from the past but look to the present and to the future in order to be effective. I believe one of the ways we can modernise the fight against financial crime is to put “effectiveness” at the heart of what’s important and move beyond pure “regulatory compliance.” This also means moving beyond the current examination approach dictated by policy makers and carried out by regulators, to a more comprehensive assessment of “effectiveness” based on “objective criteria” that is focussed on desired outcomes, set by a combination of policy makers, regulators and law enforcement.

Many FI’s are starting to measure the “effectiveness” of their own financial crime programme with policy makers, regulators and law enforcement in mind. Some are also considering if they need to measure the negative or unintended consequences of Financial Crime programmes too.

What does this Model look like?

It starts by undertaking and repeating periodically, a comprehensive risk assessment to understand the particular risks and threats faced. Then comes the design and calibration of a risk based series of controls to effectively manage and mitigate those risks, recognising that avoiding risk and managing risk is never going to produce a risk-free environment. Where risks remain, these should be within an established risk appetite and controls should operate within established tolerances. In this way we can bring discipline and a performance measurement approach to a so called “non-financial” risk type, learning from the more mature “financial risk” types such as credit, market or fraud risk. Indeed, there may even be synergies in managing such risks holistically. In measuring financial crime risk, you can measure “inherent risk” and “controls effectiveness” as follows.

Inherent Risk 

Inherent risk is the level of financial crime risk introduced by the business a Bank undertakes, prior to the application of controls. Inherent risk is calculated using a scale which rates client risk, product risk; channels risk and geographic risk. Once each of these are risk assessed, the risk ratings are weighted and aggregated first by “assessment unit” and then at overall Group level, thereby providing an enterprise-wide inherent risk assessment, rich with information to form the basis of actions designed to either reduce, manage or mitigate those risks.

Inherent risk models where client, product, channels and geography risk ratings are designed to interact to identify cumulative risk are likely to provide better results. After measuring inherent financial crime risks, two main questions arise. The first is whether and/or to what extent are the inherent financial crime risks that have been identified are within this risk appetite? The second, (if the first question is answered in the affirmative) is to ask what controls are required to bring down the net or residual risk to an acceptable level, i.e. net risk / risk appetite.

An additional area of interest might be to apply data analytics to the data that is used to measure inherent risk. For example, by interrogating the data, it may be possible to identify as yet unknown risks, for example, large books of business may have an increased likelihood of certain types of financial crime, whereas smaller portfolios may have increased concentrations of risk which could potentially produce a greater negative impact.

Controls effectiveness 1 – Preventative and Detective

Controls are employed to mitigate inherent financial crime risks. Even low inherent financial crime risks require effective controls. The greater the inherent risk, the more important it is to have effective controls. The purpose of effective controls is to reduce net or residual risk to acceptable levels, i.e. within risk appetite.

Controls are generally described as “preventative” or “detective.” Preventative Controls are proactive and emphasise “quality.” Detective Controls either uncover errors or inaccuracies in the operation of preventative controls or evidence that they are functioning as intended.

Controls are “effective” if they are operating as intended, either to avoid negative outcomes or to increase the likelihood of avoiding negative outcomes. Controls can still be “effective” even when unwanted outcomes occur provided these are relatively small or rare.

For example, most controls, especially those involving staff members, will operate with an acceptance of some level of human error. Even where automation is preferred, errors can and will still occur. Allowing for exceptions or tolerances in the results from control tests is reasonable, although these should be proposed and accepted thoughtfully.

There is no industry standard when it comes to control exception levels and tolerances and individual financial institutions have no access to comparative data on which to benchmark. In the absence of available information, we have set our own exceptions and tolerances using professional judgement and experience, as challenged and ultimately accepted (or amended) by our financial crime oversight governance bodies.

Whilst professional judgement and experience is important, it’s unlikely any exception and or tolerance is going to be right long term without operational information validating those thresholds. We started in some cases with zero exception and or tolerance levels and in others with much more liberal ones. In both cases over time, as we gained experience trying to achieve the best outcomes, we gained insight into what was possible. When we felt as though we had reached a sweet spot between inputs and outputs and were still comfortable with the net risk as a result, then we knew we had found a place where we could rightly argue it was reasonable to accept exceptions and/or set tolerances to operate controls to this level. Still further improvements in a process, in people and/or technology and data could reduce exception or tolerance levels still further. Over time it’s likely that gains can be made which could further improve the management of net or residual risks. Applying this approach to controls is agnostic to the control, whether that be to Key Controls such as CDD, PEP, Name or Transaction Screening, List Management, Transaction Monitoring or Investigations & SAR Filing or to any other control.

Controls effectiveness 2- Enabling and Correcting Controls

These well-established key controls play a critical role in directly preventing and detecting financial crime. But “enabling” and “correcting” controls are essential to making an internal control system effective and sustainable. Enabling controls include governance; assurance and testing; established three lines of defence; a functioning risk assessment; comprehensive training and awareness; the right number and quality of people and resources; quality management information and reporting; effective project and change management; specialist technology support; and an authentic tone from the top. Correcting controls are important to ensure a proportionate response is taken once an unwanted risk has materialised e.g. disciplinary action; accountability reviews; root cause analyses; lessons learned reviews; and the publication of codes of conduct.

Effectiveness based on Results

A comprehensive performance management approach, measuring and reporting on our inherent risks and on internal controls using reasonable risk based tolerances, is a powerful tool for appraising a programme’s effectiveness. Additionally, an important element to consider, beyond the management of the programme itself, is the net effect in positive terms as far as law enforcement is concerned. Tracking and measuring positive results to law enforcement is not easy and more difficult absent feedback from law enforcement. Nevertheless, there is feedback, possibly received as letters of recognition, awards and appreciation for actionable intelligence and investigation support (which are increasing), or, following SAR filing, the receipt of production orders or direct contact with investigation officers.

We advocate for a mature and robust performance management framework which can provide Regulators with a way to measure Bank effectiveness and, with Regulators and Law Enforcement working together to provide an overall performance assessment, this approach can surpass and ultimately replace the current onsite compliance testing regime in place today.

Currently there is no requirement that Regulators take into account assistance provided to law enforcement, and whilst incremental regulations wouldn’t be the answer, greater appreciation by Regulators of Bank efforts in this area could further incentivise Banks to focus efforts in this direction.

Another area to focus performance measurement could be on unintended consequences, such as financial exclusion and access to banking

A Bank cannot outsource the design, management and measurement of its anti-financial crime programme to a regulatory rulebook, but must instead discover for itself the risks it faces and an effective way of managing those risks within a reasonable risk appetite supported by risk limits, exceptions and tolerances.

Conclusion 

Better measurement and reporting on our financial crime risks and the effectiveness of our approach enables stakeholders such as Boards to have greater input and responsibility for approving and monitoring those risks and the measures taken to mitigate them. It is also an opportunity to modernise regulatory supervision, moving from testing compliance with regulation to accepting or challenging institutions on their particular approaches, be that on their risk appetite, permitted thresholds or tolerances, or outcomes for law enforcement. By embracing performance measurement, we have a means to measure effectiveness which is a win-win for everybody,2 except the financial criminals!

For more details on Modernising the fight against financial crime see Part 1 & Part 3

This paper is derived from an extract from a paper authored by John Cusack and published by Standard Chartered Bank in August 2018, and as supplemented here.