A growing reliance on artificial intelligence for decision-making is pushing financial institutions, regulators and supervisors to clarify the sources of risk and how to control them. Some of these risks were already present, if marginal, in the usual model risk management framework; others did not exist at all. As machine learning becomes widespread and industrialised across banks and insurers, issues such as interpretability and dynamic monitoring, robustness, ethics, bias and fairness require specific attention.
Although all these topics are active fields of academic research and business innovation, their rigorous analysis from a model risk perspective is still in its early stages. Close collaboration between academics, regulatory experts and private-sector professionals can accelerate pragmatic answers to many important questions. How can we interpret the outputs of black-box models? How can we monitor machine learning models over time? When and why do they drift? How can we control the discrimination introduced by algorithms? How can we protect decisions from changes in input data, or from falsified data?
This short paper is based on Emerton Data research and analysis. It provides an introduction to the new issues raised by machine learning risk and ethics, with a focus on insurance and, more broadly, financial services, probably the most mature sectors, even though these issues will soon affect every industry.
Introduction
Financial services are, by nature, built around flows of information and data. Quantitative decision-making based on data has long been common practice in the sector. A major change is under way: machine learning and artificial intelligence are moving financial services from traditional modelling, where experts built models from theories and assumptions, to a new framework in which black-box models are trained to predict a target output from observed inputs. Models therefore rely much less on expert assumptions and much more on input data. To some extent, machine learning can be seen as the automation of model building, and the usual model risk is expected to shift significantly. Our objective here is to explore the nature of the main shifts in model risk, and the maturity of the methods and tools available to manage them.
After clarifying the definitions of model and model risk, we discuss why machine learning has a strong impact on model risk, and explore what we see as the two major changes at stake: interpretability and dynamic monitoring on the one hand, fairness and bias on the other.
Model risk and machine learning
A model is a process that relies on statistical, financial, mathematical and economic techniques and theories, as well as assumptions, to turn input data into quantitative estimates for decision-making (Exhibit 1). A huge number of models serve many purposes: scoring credit risk, pricing insurance policies, defining investment strategies, improving CRM in marketing and communication, reducing costs and optimising claims processes, detecting money laundering, and so on.
Exhibit 1: What is a model?
Models are simplifications of reality; they are never perfect. Various metrics can assess a model’s quality, depending on its purpose. Performance metrics are well established: in a fraud detection model, for instance, the objective might be to minimise the number of fraudulent cases classified as non-fraudulent (the false negative rate). But many other quality criteria matter, such as robustness to outliers, stability with non-stationary inputs (fraud behaviour evolves over time) and non-discriminatory behaviour.
Model risk is a subset of operational risk. It arises when a model used to predict outputs (such as claims frequency, claims severity or elasticity) fails or performs poorly, leading to inadequate decisions (pricing, in the previous example) that translate into costs or losses.
It occurs for two main reasons:
- fundamental modelling errors: models rely on theories and assumptions that simplify more complex phenomena, and these approximations compromise the reliability and integrity of their outputs. The quality of inputs is also decisive: incorrect or unrepresentative inputs lead to a defective model;
- inappropriate use: a model is designed for a specific purpose, in a predefined environment. Applying it in a different environment from the one it was trained for can make it inadequate. Calibration errors or data issues can have the same undesirable effect.
Model risk can be addressed with an effective model risk management (MRM) framework. The framework suggested by the US Federal Reserve is described in Exhibit 2. Its main objective is to secure the development and validation process across the whole institution. It covers risk identification and assessment, risk measurement and mitigation, and risk monitoring and reporting.
Exhibit 2: Model risk management, good practices.
Financial institutions have relied heavily on models for a long time. This dependency is growing with machine learning, through the new applications it enables and the automation of traditional modelling tasks. The industrialised use of machine learning therefore means more models and less business modelling expertise.
Machine learning models use fewer a priori assumptions, are less constrained and need less expert modelling during development (Exhibit 3). Their complexity improves performance by capturing complex interactions between variables, such as multilinearities and non-linearities. They can therefore reduce modelling risk by detecting correlations that humans might have underestimated or overlooked. But they also create new risks.
Exhibit 3: From traditional algorithm design to machine learning algorithms.
Take the canonical example of actuaries modelling risk to improve the price segmentation of insurance policies. These models are critical: any underestimation of future claims costs leads to lower prices and, through adverse selection (many underpriced risks buying the policy), can ultimately bankrupt the insurer. Traditional modelling is time-consuming and based on many expert assumptions. Machine learning could drastically speed up the process and improve performance. But how can we make sure such a model is robust, when there is no control over expert assumptions? Not to mention the interpretability challenges often imposed by supervisors. The role of modelling cannot be overlooked. An effective approach, now under way, is to combine machine learning with business modelling.
This is just one example among the many use cases where machine learning is now in production: instant quotes, fraud detection, claims settlement, litigation scores, deep triangles in reserving, marketing models, client targeting. In each case, new sources of risk emerge and existing risks from traditional modelling change, requiring an update of the model risk management framework. Rather than listing every modified or new risk, we look more closely at two major risks of machine learning models: interpretability and dynamic monitoring, and fairness and bias.
Interpretability and dynamic monitoring
Interpretability is the degree to which a human can understand the cause of a decision.
Interpretability and dynamic monitoring are among the major challenges of machine learning, at several levels. Supervisors often require algorithms to be audited to make sure they comply with rules. Internally, it is key, especially during AI adoption, that the organisation understands the decision-making process so that it can audit and monitor decisions. Insurers must also give individual customers a meaningful explanation of decisions, and insights on how to improve their score, for example. The same applies to banks that must justify a credit refusal to their customers.
There are three levels of interpretability: algorithm transparency, global interpretability and local interpretability. They answer, respectively: “How does the algorithm create the model?”, “How does the trained model make predictions?” and “Why does the model make a given prediction for a given instance?”. For global interpretability, an interpretable surrogate model can be trained to explain a complex black-box model, but this approach is limited by performance issues. An alternative, with many recent applications, is to understand the decision-making process behind a particular prediction through local interpretability. For instance, if a client’s credit request is denied and they ask how to improve their score, a financial institution can train a more interpretable local surrogate model to understand how the score is computed and how to improve it. The best-known local interpretability methods are LIME and SHAP. They go in the right direction, but probably still lack the efficiency needed for massive adoption.
Last but not least, hundreds of models run in production, and the relationship between input space and target space rarely stays constant. Either the source generating the data is not stationary, or the concept to be learned changes over time: fraud behaviour evolves, and may even adapt to fraud detection models while they run. Either way, a model must be robust to these shifts, by detecting, understanding and handling them appropriately. It is therefore necessary to understand and characterise the change, rather than simply modifying the model after a loss of performance. This is part of the model life cycle: understanding when and why a model becomes obsolete, and how to fix it. Of course, this relies heavily on interpretability: it is hard to fix a model you do not understand.
Fairness and bias
Dealing with bias and fairness is another major challenge. In many use cases involving people (credit scoring, insurance pricing, or models that take employees’ actions into account), models are expected to reach a certain level of fairness, to avoid any uncontrolled discrimination against groups of people. The European Convention for the Protection of Human Rights and Fundamental Freedoms prohibits “discrimination on any ground such as sex, race, colour, […] national or social origin”, and later instruments add grounds such as genetic features, age and sexual orientation. Avoiding discrimination can directly conflict with objectives such as price segmentation. There are two major sources of unfairness (Exhibit 4): for classic tabular data, unfairness can come from the rows (bias) or from the columns (information retrieval).
Exhibit 4: Two potential sources of unfairness in tabular data.
The classic example in insurance is avoiding gender discrimination. If the gender variable is excluded from a model, then, all else being equal, the output is the same for both genders. However, gender is usually correlated with other variables, and the model will learn from these correlations, which act as proxies for gender. It ends up discriminating as soon as its outputs are compared on real sets of women and men. A clear fairness policy must therefore be defined. Regulators understand that discrimination is not always negative, and are starting to distinguish intended discrimination (prohibiting the direct use of information such as gender or religion in models) from discrimination as a result (controlling and tolerating observed differences in model outputs across groups).
Two classes of approaches to fairness are emerging in the academic literature, and to a lesser extent in banks and insurers:
- Fairness by design, which takes fairness and prejudice criteria into account when designing and training machine learning models (for instance, by adding a fairness term to the model’s objective);
- Fairness in use, which instead focuses on how model outputs are used, to satisfy fairness criteria (e.g. diversity quotas in recruitment in the US).
A biased training dataset is another major cause of unfairness. If a dataset discriminates against a group of people, the machine learning model will reproduce the bias and often amplify it. Take human resources: if a recruiter unintentionally discriminates against a group, a model trained on that recruiter’s past selections will reproduce or amplify the bias. One challenge is learning to detect biases in a dataset and deciding which are harmful, and to what degree. Dataset certifications may appear in the near future.
Conclusion
Model risk has long been identified and controlled in banks and financial services. Since the sub-prime crisis, banks have strengthened their risk management frameworks, supported by new regulations and financial supervision.
With machine learning in production, model risk is changing. It is a growing concern that is receiving special attention.
Despite recent work on new sources of risk, it is not clear that financial services have reached the maturity needed to properly assess and control machine learning risks. The interpretability of black-box models is well identified but still poorly addressed, even though it is clearly key to the dynamic monitoring of hundreds of models, often retrained very frequently. Fairness is probably not yet well defined, but it is critical to controlling the bias and discrimination that machine learning models can produce.
Further reading
- A reference note on model risk management (financial supervision): SR 11-7, Guidance on Model Risk Management, Federal Reserve System.
- A recent paper on model risk fundamentals (risk management): B. Hassani, Model risk management: from epistemology to corporate governance, Journal of Risk Model Validation, 2019.
- A risk management perspective on AI and machine learning systems: A. Bogroff, D. Guegan, Artificial Intelligence, Data, Ethics: An Holistic Approach for Risks and Regulation, preprint, 2019.
- The summary of the French regulator’s task force on AI in the financial sector (in French): O. Fliche, S. Yang, Intelligence artificielle : enjeux pour le secteur financier, 2018.
- AXA’s research department on these topics (several papers online): AXA REV Research.
- On fairness by design: T. Kamishima, S. Akaho, J. Sakuma, Fairness-aware Learning through Regularization Approach, IEEE International Conference on Data Mining workshops, 2011.
- On fairness in use: J. Kleinberg et al., Algorithmic Fairness, AEA Papers and Proceedings, 2018.
- The reference paper on LIME: M. Ribeiro, S. Singh, C. Guestrin, “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, 2016.
- The reference paper on SHAP: S. M. Lundberg, S.-I. Lee, A Unified Approach to Interpreting Model Predictions, Advances in Neural Information Processing Systems, 2017.
