Trust, Context and, Regulation: Achieving More Explainable AI in Financial Services

Download Slides

This presentation seeks to advance the thinking on how financial services firms can implement a framework that supports explainable artificial intelligence (AI), thus building trust among consumers, shareholders and other stakeholders, and helping ensure compliance with emerging regulatory and ethical norms.

The use of AI in financial services continues to grow. Especially nowadays with the global COVID-19 pandemic, industry take up is increasing and use cases are expanding out from the back office and into customer-facing applications. We also see a shift towards more complex models giving more accurate and deeper insights. This expansion brings many opportunities for industry to improve efficiency, better manage risk and provide exciting new products and services to customers.

However, to take full advantage of this opportunity, there needs to be trust. As with all innovations, ethical considerations must keep pace with technological development. Building trust requires transparency and communication. Indeed, this is a topic of growing regulatory and government interest in many countries. Transparency and communication with customers have long been key considerations for financial services but AI will require new approaches and techniques if explanations are to be meaningful. Effective explanations will also require a degree of subtlety; given the huge potential range of use cases, close attention to the context of each will be key.

Alongside this, consumer education as to how and why AI is being used is increasingly important. Achieving effective explanations will require firms to have a clear AI strategy and robust governance, and to engage effectively with colleagues from a range of functions, including data science, compliance, audit, business and senior management, and even ethicists. It will also require ongoing work, with limits yet to be resolved in the state of the art of explaining AI and with ‘best practice’ sure to evolve. More research and thinking will be needed, especially now with the change in ways of working, not just from firms but also from regulators, government and think tanks.

This presentation will focus on why and to what extent explainability of AI outputs is needed, the challenges to achieving this and potential ways to apply the latest guidance. It will also provide technical financial services use cases to explore potential approaches to different types of explanations, according to the context and type of model. We consider not just common existing uses of AI, but also emerging or possible uses.

Speakers: Ansgar Keone and Walter McCahon


– Great, so thanks for joining us today for this presentation on the Joint UK Finance, EY paper on AI explainability, trust, context and regulation, focused on financial services. So, I am Walter Walter McCahon from UK Finance, presenting on the paper today with Ansgar Koene from EY.

– Welcome everyone, I’m looking forward to a great discussion after this presentation.

– [Walter] Great. So, to start things off, why do we do a paper on explaining AI in financial services as well in the UK? And if there is a growing adoption of artificial intelligence as shown in quite a range of statistics that are out there. And also unfortunately highlighted by a joint survey last year by the UK’s financial services regulators. And not only that, but there is also a growing public interest in artificial intelligence. So, we’ve seen some high profile cases where AI applications have perhaps gone wrong, not so much in financial services, but in various other sectors in the UK. And as shown on the slide, there have been incidents with the use of algorithms aspiring, for example, local councils in relations welfare, and in relation to the scaling of university entrance exam grades, which we’ve seen in the United Kingdom. And we note also that there are some estimates that in some cases AI models will still be producing erroneous outcomes for some time to come. So, with that context in mind, in our paper we try to bring together core technical regulatory and governance considerations So, before we go into some of the detail, let’s just do a quick overview of what we cover. So, broadly the paper looks inside the growing interest, both public and regulatory in artificial intelligence and its risks. So, this raises of course, the importance of maintaining trust in one’s AI systems in order to manage regulatory expectations and to avoid reputational damage. And then our key considerations to get right include AI fairness and AI explainability, very broadly speaking systems that are auditable and for which the outcomes and functioning can be sufficiently explained to customers to other stakeholders. So, explainability is really the focus of the paper, there Some wider considerations, such as trust and fairness, which are intimately linked, also discussed. So, in the paper, we look at some of the challenges to achieving appropriate levels of explainability together with some tools for tackling these challenges. We leveraged the most comprehensive guidance that we know of on this issue on AI explainability, that being the best practice guidance from the UK Information Commissioner. this’s a helpful tool to think through a useful and a comprehensive approach the firms can take. review also processes and governance that will help firms in practice. And we explore some use cases trying to bring together regulatory, technical, and customer considerations. And we’ll look at a couple of those quickly a little bit later in the presentation. So, at the risk of ruining the story, there are some key takeaways to call out upfront. So, first of all, the importance of thinking about making AI trustworthy, really from the beginning of the project, rather than trying to essentially finish the project and then make by patching on some explainability right at the end. We also highlight the importance of thinking carefully about context. So, the stakeholders you need to communicate with use case, et cetera, and setting up of governance. So, including the particular governance to help you identify priority models and use cases that you’ll need to focus on. So, a quick overview done, let’s now look at some of the paper and a little bit more detailed. So, first up you mentioned some of the key challenges and the particular one, which many people will be the challenge of the likely trade off between accuracy and explainability for different types of model. As we can see pictured here on the left side of the slide, and we suggest four steps that can help firms navigate this in practice. So, first of all, determine what is in scope. So, we note that AI is not always clearly defined, I think it’s one of those terms which is… Everybody who tries to define it eventually struggles with, quickly start struggles within banks. So, firms might want to consider what other types algorithms and analytics should be brought within the explainability framework. And next to build up good governance and clear about accountability. So, there’ll be a lot of different teams within a firm that likely to be needed and the role of each. And the ultimate responsibilities will need to be clear. Identify the priority models for explanations. So, especially for firms that use a lot of algorithms or a lot of models, you probably can’t solve for them all at once. So, setting clear priorities and working through them will be key considering factors like customer impacts, regulatory interest, and reputational risks. And lastly, determining what it exactly it is that you need to explain a number of different answers depending on the situation. This is clearly a key consideration. So, I’ll now hand over to Ansgar to look at this in a little bit more detail.

– Thank you very much, Walter. So, clearly, you know, when thinking about explainability of AI systems, particularly within mind, the idea of trying to increase trust both within your own organization and also among customers and other parties that you engage with. It’s important to start off by realizing that AI is an umbrella term. So it’s going to cover quite a different range of various types of algorithmic systems, each of which provide different types of challenges when it comes to explaining what the system is actually doing. On the other hand, it’s also important to think about the fact that, when you’re trying to explain the algorithmic system, from the point of view of building up trust in what is happening, we need to think about different stages of the life cycle of this system and each of which has different requirements relating to it. So, the initial part of thinking about the business case, why this type of algorithmic system is even being introduced into your organization and the kind of governance that is built around it will raise certain aspects of explanation. Then there’s a question about the types of data and processing that’s going to be used, the modeling choices that are being made and the ways in which outcomes are being analyzed. And very importantly also, the way in which the system is being monitored after it is deployed. All of these stages within the life cycle of an algorithmic system bring different points to the four that need to be included in explanations. The other aspect that is very important to take into account, even from the start of thinking about how to explain the AI systems, is what is it really that, what type of explanation is it that is needed for your particular use case in your particular stakeholders that you’re engaging with? So the ICO, the UK data protection authority, in their report earlier this year, nicely highlighted six different dimensions of types of explanations. So, one is talking about the rationale. What is it that the AI system is really meant to be achieving? Another is a responsibility, who has responsibility for which parts of the decision-making process? Explanation around the data that is being used. Why is this particular kind of data being used? Why is the state had considered to be relevant for producing a decision on this particular, for the particular use case? Fairness questions are very important. We’ve seen a lot of the debates around the use of AI hinges on the question of whether the systems are biased or not, questions about, what kind of fairness? There are very many different dimensions of fairness. So, there needs to be an explanation as to the choices and the ways in which fairness is being assessed. Safety and performance. Key indicators that need to be monitored. So, there needs to be also an explanation as to how the safety is being monitored, how it is being guaranteed, what kind of performance is being optimized for, all of these are things that the stakeholders going to be interested in, whether it is the regulator or the client, or some other party that you’re engaging with? And finally, what type of impact is this AI system and it’s kind of decisions actually going to have on the individual? One of the things that a lot of parties will want to know before they even start engaging with you and your use of an AI system. So, with these in mind, there’s the question about how to go about, start to implement explainability. And again, referring to the work that was coming out of the ICO, they very nicely highlight the sort of two key dimensions that need to be considered. One is the types of early stages within the development process, where you need to start considering about the explanations that need to be made. So, when considering that, you need to be clear about the domain where the system is going to be used, what the use case exactly is. What kind of an impact you’re hoping to achieve on the individual and what kind of impacts might happen that are unintended? These need to be considered when thinking about what the priorities are around the explanation. So, thinking of the previous list, for instance, is the rationale going to be a key thing that you want to be explaining, or is the data set one? Then there’s questions about, thinking about the explanation from the very beginning during the pre-processing of data, and also thinking about explanations during the system builds so that it isn’t an afterthought. Besides these questions about how to get at the information that you need for the explanations, is also very important to think about how the explanation is going to be delivered. Technical details can be very important for certain parties. Regulators might want to have these, but for many parties, these kinds of levels of detail we’ll just have to escape. We’ll get in the way of a clear communication of what is of core importance. And there’s a question about making sure that you are customer facing staff is able to convey the explanations. It’s no good if only the data scientists have an understanding of the explanation. And then finally, it’s important to consider how to build and present the explanations. The ICO, for instance, talks about using a layered approach where you present certain parts of the explanation. Then only if the stakeholders asking for more, do you present the additional piece of explanation building it up so that nobody gets overwhelmed with the information. So, Walter, I think you wanted to start off with the use cases.

– Yeah, thanks Ansgar. So, we wanted to explore some of the the practical case studies that can help eliminate these key considerations that we’ve been discussing so far. And clearly there are going to be a lot of different valid approaches that firms could be taking, but we wanted to explore some of these in the paper. So, having looked at some global case studies that were instructive, the first example that we chose to focus on was credit decisions and the risk of bias, or at least the perception of bias. So, first question is. What is it exactly that should be explained? As Ansgar mentioned, there’s always a risk of overwhelming customers explaining too much or giving information that’s too complex or not what they’re looking for causing frustration and not helping in that overall goal of enhancing the trust in your AI. So, it’s helpful to think about the information needs of the customer or of the consumer. So, thinking about these as a sort of a series of questions. Why was I declined, or perhaps why did I receive that particular credit limit? So in practice, it’s probably only where a customer doesn’t like the decision that detailed questions of this kind going to be asked if the customer the loan that they were looking for, perhaps wasn’t happy with how the credit limits and consumers may want to know what can do to improve their chances of getting a favorable lending decision in the future, whether that’s reducing debt or other actions that they might be able to take. However, at the same time, firms are going to have to think about making sure that they don’t enable consumers to game the system by just tweaking some of their data points without fundamentally addressing So, explanations would need to be designed carefully in order to mitigate against their risk while still providing some useful information to the consumer so they can take steps to improve their standing. And built into this, there will of course be a range of regulatory requirements to weave together as we know financial services as a heavily regulated sector everywhere. So, likely to have some GDPR considerations such as Article 22 in relation to automated decision making, which could be applicable. And depending on the jurisdiction, there will no doubt be other regulations and industry guidelines that require certain facts to be explained to the customer and when engaging in lending. And so, all of these as be drawn together as part of the AI explanation. So, next, customers could challenge and say that this doesn’t seem fair. happen again if the customers in fact not happy with the decision, they could challenge its fairness, they could perceive bias involved with it. It was a discriminatory decision of some sort. And there have been some cases internationally where there’ve been perceptions at least that the creative decisions were not made on a gender neutral to this risk. Next, why looking there? So, there are some interesting consumer credit use cases emerging that leverage unconventional data like social media as a data source, for example, and for those firms that are looking perhaps to do this, there’s certainly some benefits for consumers here, particularly those that might have a file, where they might be more able to show their and get access to more affordable credit by providing or by agreeing to have this kind of unconventional data about them factored into decisions. But these firms would want to make that they’re being very clear about what these data sources are. Obviously, quite bad news for a customer to be shocked after a decision is made and to challenge the fact that perhaps unexpected data was looked at. Then I think you got this wrong and sort of building on comments earlier, essentially customer’s unhappy, they’re likely to want to know it just in case, how they could go about challenging the decision, perhaps having it looked at again. So, with those points in mind, it’s useful to go through some of the technical considerations as well. So, in addition to customer information needs as a part of looking at those information needs and making sure there’s a reached. It’s important to use the most interpretable model type that can achieve the desired accuracy requirements. So, if you need a high level of interpretability of the new regression, perhaps random trees model could be more appropriate than some of the alternatives, even there might be some trade-off in terms of accuracy, next documentation of the data and the model selection justification’s you can show why you made the decisions that you did in that particular year unconventional data or unexpected models are being used. And next, ensuring that you have clear internal fairness criteria and controls to meet those. So, for example, perhaps could be the same false positive rate for different demographics. It could be again, different adjustments approaches to this, but making sure that you have a clear choice that you’ve made and that you’ve documented your thinking. And then in terms of building all of this together into the customer journey, mentioned earlier, the layering of explanations. So, we thought that this could potentially start with a basic rationale explanation. So, the main factors behind the decision in working in the considerations from industry guidance relating to lending more generally, and so employing counterfactual analysis to see how the decision might have turned out differently to provide that initial basic rationale expert explanation being ready to perhaps provide a more comprehensive one if needed, and also a light touch responsibility explanation. So, showing customers where they can go, if they want to make queries or complaints, showing which part of the business is responsible for that. And if asked, being ready to explain how fairness is insured and quite as a difficult element of that, making sure that the explanation is understandable to the customer. So, there’s these issues of fairness and explainability, tightly connected as discussed, and they’re quite difficult. So, I’ll now hand over to Ansgar to look at this in a little bit more detail.

– Thank you, Walter. So, looking in more detail at the role of explainability in order to establish trust, and specifically in relation to algorithmic bias in machine learning validation. It’s important to, first of all, think about the different kinds of dimensions that really come into play here. So, on the one hand, we have explainability playing a role as part of model transparency, making sure that the stakeholders, the users, the customers, have clarity as to what it is that is actually happening around them. And on the other hand, questions around fairness, providing people with the levels of information that they need in order to have assurance that what measure of fairness is being applied here, how that fairness is, is being guaranteed and really understanding how model fairness is being provided. So, on the model transparency side, it’s really important to think about this from the human kind of perspective of the stakeholder that’s meant to receive this explanation. So, will they be able to understand the decision framework, will they be able to understand and consistently interpret how model prediction outcomes are being generated? So, with these kinds of goals in mind, around explaining in order to provide transparency about how the system works, some of the key things to consider are the types of features that were considered to be important. So, in the case of credit assessment, for instance, features would be things like whether or not you are somebody who has, as a house owner that is paying a mortgage, whether or not you’re as somebody who in steady employment, those kinds of things. What are the features that are actually being used? In other aspects could be, clarity about how the inputs actually lead to the outputs. Is it possible to provide some kind of a conceptual model around why these particular features that are being looked at are relevant for the assessment that’s being made? Is it possible to provide some kind of a visualization around how the system works? Obviously, this will not be going into the fine details of how the system is operating, and to a large extent, frequently, you wouldn’t even want to provide those because that would enable people to game the system. However, it is important to be able to communicate the essence of how the system works so that people can have a sense of whether or not this is an appropriate way of being assessed by this automated system which connects to the question about model fairness. How is the model being guaranteed to provide a fair assessment of this individual in this particular case? And it’s important in that sense to highlight that there are different ways of assessing fairness, individual fairness versus group fairness, fairness of process versus fairness of outcome i.e is everybody being treated in the same way, or is the treatment potentially not exactly the same, because you are trying to compensate for underlying disparities between groups in order to make sure that as far as outcomes are concerned, the fairness is established. So, there are different ways in which fairness can be approached. And it is important to be able to communicate which of these methods is being used and why that particular methodology is considered to be relevant for the particular applications space that is being done. So, questions around the process of… Beginning with clear and documented statement of the fairness definition being able to communicate this is how fairness has been defined in our use case. And this is why we argue that it is an appropriate definition for our use case. If there is a sense that the process hasn’t led to the correct kinds of outcome, what kind of methodologies are there to remediate, to interfere in intersect in the process in order to make adjustments? And how are the systems being optimized? Is it for one type of fairness or for another? So, being… Providing clarity around the various different ways in which fairness can be assessed that this is often not a… There’s only one way of doing it, but rather being clear as to what the design decisions were and why these were considered to be the appropriate ones will be important aspects to be able to communicate to customers and users. So, to summarize, there are a number of key takeaways regarding the way in which explanation around AI decision making should be considered and should be communicated, both from the business side and also from a regulatory kind of perspective. So, in the business side, it’s important to have clarity around the governance of the AI system. So, what is the purpose and what is the context for which the system is being used? What are the scoring mechanisms that are being applied? Different kinds of models will have different kinds of prioritizations around the types of explainability that are required in certain use cases. In order to gain more explainability, there will be need to sacrifice some accuracy. So, we’ll that… What kind of priority needs to be made in a certain kind of application space. Which connects to the procedures for trade-offs that have to be made. Trade-offs of explainability versus accuracy between AI trust attributes. So, knowing the limits of AI explanations. So, it may be important to communicate to both internally to the stakeholders within the organization, but also to the external stakeholders there customers, why certain decisions were being made, why perhaps a choice was being made to use a deep learning system that is less explainable, but how certain advantages regarding the ability to cope with complex situations versus other cases where perhaps the trade off needs to be made in a different direction. So, effective explanations need to be appropriate for the particular user, whether they’re internal stakeholders, whether they’re external stakeholders, and in order to have comprehensive trust in the AI principles, it’s needed to create and maintain trust and explainability in the longterm. So, that means, clarity as to the way in which the core processes working, why the types of decision factors that were being included are relevant to this particular case. This may require extra effort to explore why, for instance, a deep learning system has generated certain types of associations between inputs and outputs. On the regulatory side, we need to be thinking about explainability requirements that need to match the oversight requirements. So, depending in which secretor you are operating, there will be different requirements as to the ability to explain in where at various levels of detail how certain decisions were being reached. Meaningful explanation would depend on the specific AI application, but also on the sector where it’s being used. And so, there’s an aim for common standards, aligning with data ethics and AI guidance across different authorities and to prioritize applying AI specific guidance to existing rules and regulations. On and all the there’s an important need for aligning around, not just definitions and terminology but also around which types of transparency, which types of assessment methods are going to be considered acceptable practice within various different domains. This is still a space that is under development. So, it will be important for any organization to have clarity as to what kind of decision trade-offs were being made in their choices so that they will be able to engage with the regulators as these things are being worked out. So, thank you very much. This is this set of different authors who contributed to our work, a joint publication of EY and UK Finance. And you will find the link for this on the next slide. So, this is a publication that came out earlier this year. The full paper is available through the link that you can see up on the slide here, which is through the UK Finance website. And as with all of these talks, please don’t forget to rate and review this session. And now we welcome questions.

Watch more Data + AI sessions here
Try Databricks for free
« back
About Walter McCahon

UK Finance

Walter leads UK Finance’s work on data protection, privacy and data ethics, coordinating collaboration among member firms’ subject matter experts on policy and regulatory issues affecting the industry.

Before working at UK Finance he worked on financial sector and data protection law reforms at the British Bankers’ Association, New Zealand Bankers’ Association and for the New Zealand government.

About Ansgar Keone

Ernst & Young LLP

Dr. Ansgar Koene is Global AI Ethics and Regulatory Leader at EY (Ernst & Young) where he supports the AI Lab’s Policy activities on Trusted AI. He is also a Senior Research Fellow at the Horizon Digital Economy Research institute (University of Nottingham) where he contributes to the policy impact and public engagement activities of the institute and the ReEnTrust and UnBias projects. As part of this work Ansgar has provided evidence to parliamentary inquiries, co-authored a report on Bias in Algorithmic Decision-Making for the Centre for Data Ethics and Innovation, and was lead author of a Science Technology Options Assessment report for the European Parliament on “a Governance Framework for Algorithmic Accountability and Transparency.”

Ansgar chairs the IEEE P7003 Standard for Algorithmic Bias Considerations working group, was the Bias Focus Group leader for the IEEE Ethics Certification Program for Autonomous and Intelligent Systems (ECPAIS), and participates in the IEEE standards P2089 “Age Appropriate Digital Services Framework” and IEEE P2863 “Governance of AI”. Other standards development work includes participation in the ISO/IEC JTC1 SC42 Artificial Intelligence activities and the CEN-Cenelec Focus Group for AI.

He is a trustee for the 5Rgiths foundation for the Rights of Young People Online and is part of the 5Rights Digital Futures Committee.

Ansgar has a multi-disciplinary research background, having worked and published on topics ranging from Policy and Governance of Algorithmic Systems (AI), data-privacy, AI Ethics, AI Standards, bio-inspired Robotics, AI and Computational Neuroscience to experimental Human Behaviour/Perception studies. He holds an MSc in Electrical Engineering and a PhD in Computational Neuroscience.