Explainable AI: emerging practices to ensure responsible transparency
Written by Jens Eriksvik
AI increasingly governs critical decision-making processes. But relying on inscrutable systems becomes increasingly problematic. AI systems are often black boxes characterized by opacity.
However, there are emerging practices to address this. With AI's increasing importance to businesses, there is an increased need for transparency and comprehensibility - a need to understand “the reasoning” of an AI system. The field of “explainable AI” (XAI) aims to give AI models interpretability, and thereby to enable stakeholders to understand AI decision-making.
The development of XAI follows the emergence of regulatory frameworks, e.g. the European Union's AI Act. This legislative framework underscores the role of explainability, particularly in AI applications deemed high-risk. Analogous regulatory initiatives are manifesting worldwide, rendering XAI not merely a discretionary best practice but a legal imperative for certain applications.
XAI is about making complex AI models transparent, to create an understanding of why AI models make certain decisions. It contains a range of techniques, from analyzing a model's behavior to highlighting key data points, to create visibility for the inner workings of a model. This not only builds trust but allows for improvement and debugging of AI models for better performance.
XAI methodologies
A diverse array of XAI methodologies exists. There is no one-size-fits-all approach, rather each method is tailored to address distinct requirements.
Model-agnostic methods: These versatile techniques operate independently of specific model architectures, analyzing their behavior to explain predictions.
Model-specific methods: These are methods tailored to particular model types, they leverage knowledge of the internal structures to furnish explanations congruent with the model's architecture.
Feature importance methods: This methodology explains the data features pivotal in shaping the model's decision-making process, thereby offering insights into its decision rationale.
Basically, these methods create an understanding of how AI models work through an investigative approach, identifying key characteristics of the models to determine which pieces of data and/or inputs had the most significant impact on the model output. E.g. feature attribution and counterfactual explanations facilitate this process by pinpointing the pivotal data points and exploring alternative scenarios.
XAI analytical paradigms
XAI represents a dynamic field characterized by innovation and refinement. Researchers continually develop novel methodologies while refining existing techniques. The trajectory of XAI is expected to veer towards seamless integration into the entire AI development lifecycle, thereby ensuring transparency from the early stages of model development.
In XAI, there are two main mathematical methodologies prevalent today: SHAP (Shapley Additive exPlanations) which is a methodological framework that employs the concept of Shapley values to apportion prediction credits among various input features, thereby affording insights into their relative importance. The second one is LIME (Local Interpretable Model-agnostic Explanations) which constructs simplified, interpretable models centered around specific predictions. LIME facilitates the comprehension of localized decision logic within AI systems.
Example scenario: Explaining why a drowsiness determination algorithm reaches a certain conclusion
Let's imagine a driver drowsiness detection model using a camera to monitor the driver's face. XAI could be used to explain the model's prediction, thereby creating several benefits: a) The explanation helps the driver understand why the system flagged drowsiness and potentially adjust their behavior, b) By analyzing feature importance, developers can identify if the model is over-reliant on a single feature and improve its overall accuracy, and c) transparency in the model's reasoning fosters trust in the driver assistance system.
Scenario: the model predicts the driver is drowsy.
Possible XAI techniques:
Feature importance: XAI analyzes the input features (e.g., eye closure percentage, head pose, yawning frequency) and identifies which features most influenced the drowsiness prediction. Explanation: "The model predicted drowsiness primarily because of the high eye closure percentage detected in recent frames."
Feature map visualization (if using a deep learning model): XAI visualizes which areas of the driver's face (eyes, mouth) captured the model's attention during prediction. Explanation: "The model focused heavily on your eyelids during the analysis, suggesting possible closure contributing to the drowsiness prediction."
Counterfactual explanations: XAI virtually alters the driver's image (e.g., opens eyes slightly) and re-evaluates the model's prediction. Explanation: "If you opened your eyes a bit wider, the model might have predicted reduced drowsiness risk."
XAI for business
XAI is key in any business AI strategy. It has the potential to create a competitive advantage by creating better-performing AI algorithms and driving trustworthiness. When people understand how AI works, they're more likely to trust it. This builds stronger relationships with customers and other stakeholders.
Identify high-risk AI applications: Identify AI applications where the stakes are high and apply XAI methods to drive transparency and mitigate risks.
Invest in expertise: Given the intricacies inherent in XAI, prioritize investments in building internal expertise or forging strategic partnerships with XAI specialists. This should be part of a cross-functional AI hub/CoE together with technical experts, legal experts, and business owners.
XAI by design - integrate XAI practices in AI development: Integrate XAI principles across the entire AI development lifecycle to ensure that transparency and explainability are integrated from the start of deployment.
By embracing XAI, businesses lay a robust foundation for the responsible adoption of AI technologies, driving trust and confidence among stakeholders and laying the foundation for continuous improvement of algorithm performance.