Proceed With Caution: Explainability, AI, And The Risks Of Regulatory Compliance

Proceed With Caution: Explainability, AI, And The Risks Of Regulatory Compliance

Ciarán Daly

February 26, 2019

7 Min Read
Photograph of head of man wearing a headset to answer calls

by Jelani Harper

"The choice for organizations using AI models for customer decisions is clear: either use less accurate, more transparent models for full compliance, or chance much more accurate, opaque ones resulting in increased productivity—and compliance problems."

SAN FRANCISCO - The tension between the most accurate manifestations of artificial intelligence today and the explainability and interpretability issues plaguing them has never been greater.

The precision of these advanced machine learning and deep neural networks is almost beyond dispute. However, operationalizing them for increased revenues, customer satisfaction, and the attainment of business objectives requires an articulation of their intricacies to regulators—which is far from straightforward.

"There’s so much additional value in using these techniques that we’re trying to come up with ways that make them explainable: even though you can’t take those levels and all the lines going back and forth; you can’t unpack them to say ‘this is exactly what happened'," says David Wallace, SAS Global Financial Services Marketing Manager.

Unfortunately, that’s just what regulators want to know. This partly accounts for the expansion of regulations in scope and severity over the past several years. In finance alone, the Federal Reserve Board’s SR 11-7, the Targeted Review of Internal Models, sanction screening, Anti-Money Laundering regulations, Know Your Customer, various anti-fraud mandates, the Bank Secrecy Act and more either compel organizations towards complex AI models, or demand explainability in order to do so.

There are also similar regulations in verticals like healthcare and insurance pertaining to explainable model results. That's not to mention the numerous universal regulations, spanning from the General Data Protection Regulation to regional mandates like the California Consumer Privacy Act (widely considered a regional parallel to GDPR).

The choice for organizations using AI models for customer decisions is clear: either use less accurate, more transparent models for full compliance, or chance much more accurate, opaque ones resulting in increased productivity—and compliance problems.

Nonetheless, the distinct possibility of a third alternative has recently emerged, one in which the compliance issues of the latter approach are counterbalanced by interpretability and explainability so that there’s what Ilknur Kabul, SAS Senior Manager of AI and Machine Learning R&D calls fair, accountable, transparent and explainable AI and machine learning. This, she admits, is "not easy to get".

However, those that succeed in deciphering the labyrinth of interpretability, explainability, and ethical AI will have the most powerful, regulatory-compliant predictive analytics models on the planet.

Related: AI Matters For Dynamic Retail Pricing - Here's Why


Despite the numerous overlapping definitions of explainability and interpretability, these terms are not synonymous.

Firstly, interpretability is a critical prerequisite for explainability in that it utilises a mathematical understanding of the numerical outputs of machine learning models. According to Kabul, interpretability in data science not only requires an understanding of the inner layers of complicated neural networks, but also a need to communicate the results obtained from the model to the organisation's upper management.

Of the myriad ways to increase interpretability, the most common is to simply adjust the weights and measures for the inputs to evaluate their effect on the outputs, drawing logical inferences in the process. Other means of interpretability revolve around surrogate modeling (training simpler models with inputs and outputs from complex ones, which can involve techniques like LIME) and visually plotting relationships between specific variables and the model’s output (with techniques like ICE and Partial Dependent Plots).

Regardless of which of these methods is used, the most competitive interpretability mechanisms rely on graphic approaches that bestow an added dimension to the quantifiable inputs and outputs. Such platforms utilize a “visual way of pulling data in, having the system itself compare different modeling techniques, [then] you look at the one that has the greatest lift—the technical measurements that provide the answer to which one has the highest predictive power,” Wallace said. These graphic approaches provide a host of functionality for honing in on specific aspects of models, inputs, variables, their stability and more to illustrate the preceding means of interpretability.

Related: Modernizing Natural Language Processing With Deep Learning


A statistical understanding of the inner workings of the most accurate machine learning models is the foundation for explainability—which involves issuing full verbal explanations about how black box models function.

Explainability is required to inform both regulators and customers about the results of models, especially when they’re used to determine financial offers, insurance policies, healthcare strategies, and other results impacting customers.

According to SAS AI and Language Analytics Strategist Mary Beth Moore, explainability requires articulating why weight was given to one input versus another, whether this was done automatically, and, if a data scientist opened up the neural network to change the weightings, why this was done. Graphic mechanisms effectively assist with explainability by providing visuals to demonstrate how models are operating.

Another facet of explainability relates to rules that further clarify points of understanding. Such rules can fortify the meaning of explanations to satisfy customers and regulators. Once organizations have issued interpretability and explainability, they can then take the output of the ML algorithms and turn them into explainable rules, explains Franz CEO Jans Aasman. “Then you can say well, the reason I’m not giving you this loan is because of these factors.”

This use of rules not only aids explainability, but is also influential in customer relations pertaining to the results of complicated AI models. “Now, instead of just applying the formula, you can also use additional rules,” Aasman offers. “Of course you have rules for how you want to deal with customers, and you apply the rules and then you can use continuous machine learning to see if your actions were positive or negative for the bank or for your customer.”


Ethics is implicitly central to explainable AI for regulatory compliance, which is frequently pushed to the forefront of discussions about these technologies.

"You need to figure out if you can predict from the signals in your data whether or not someone is going to be at financial risk.”

- Jans Aasman, CEO, Franz

From an enterprise perspective, explainable AI is necessary for deploying the opaque machine learning models with the greatest accuracy (and therefore the most value) for mission critical endeavors like Know Your Customer or offering competitive products and services to consumers. It’s also a requisite for satisfying regulators in various verticals. However, explainability is often viewed as the initial step in using deep neural networks and other intricate models in an ethically defensible means for actually improving society, instead of exacerbating it.

"You need to figure out if you can predict from the signals in your data whether or not someone is going to be at financial risk,” Aasman ventures. “But then you also have to be able to figure that out in rules so that you can make sure that you don’t have racist rules, or improper rules that are against compliance, or discriminate against women.”

In this respect, explainable AI not only assists organizations in maintaining regulatory compliance, but also in adjusting various policies, practices, and rules that might contribute to unethical situations such as biased treatment of customers.

Model Risk Management

Deep neural networks and other forms of opaque AI models represent the pinnacle of the predictive power of this technology today. However, their deployment is circumscribed by the capability to consistently interpret and explain their results in an unbiased, ethical manner.

The impact of regulations is critical to this situation. On the one hand, mandates such as Anti-Money Laundering and Know Your Customer can be considerably assisted by black box techniques. Yet, if they’re deployed and organizations can’t explicate their findings, these firms are liable for all sorts of penalties pertaining to model risk management and other forms of risk.

Interpretability, explainability, and the various techniques discussed above for facilitating them are therefore the means of unleashing the full predictive prowess of advanced machine learning. Without them, AI technologies designed to enlighten the enterprise—to empower and effect competitive advantage—can potentially consume it in a flood of regulatory penalties and less accurate, more transparent algorithms.

Jelani Harper is an editorial consultant servicing the information technology market, specializing in data-driven applications focused on semantic technologies, data governance and analytics.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like