Explainable AI and Statistical Interpretation of Machine Learning Models

Background to the Study

The last decade has seen machine learning move from research labs to operational decision systems. In healthcare, models assist in diagnosis, triage and risk stratification; in finance, they underpin credit scoring; in criminal justice, they inform recidivism estimates. Performance gains have often come from complex ensembles and non-linear models, but with performance has come opacity. Deep neural networks, random forests and kernel machines produce accurate predictions whose internal logic is inaccessible, earning the label black box.

Opacity is problematic where decisions are consequential. The European Union's General Data Protection Regulation (GDPR) introduced a right to explanation for automated decisions, and financial regulators require adverse action reasons for credit denials. In healthcare, clinical decision support must provide audit trails. This regulatory demand creates institutional incentive for Explainable AI, defined as methods that make model decisions transparent, interpretable and accountable. The National Institute of Standards and Technology has formalized principles for explainable AI, emphasizing explanation, meaningfulness, accuracy and knowledge limits, underscoring that explainability is now a compliance concern as well as a scientific one.

Technically, XAI builds on statistical thinking. SHAP, derived from cooperative game theory by Lundberg and Lee, attributes additive feature contributions satisfying local accuracy, missingness and consistency. LIME approximates a complex model locally with an interpretable surrogate. PDPs, long used in statistics, visualize marginal effects. Together they answer core statistical questions: which predictors matter, how predictions change with predictor changes, and whether learned relationships align with domain knowledge. Yet many applied studies evaluate XAI qualitatively without hypothesis testing, confidence intervals or stability analysis. There is limited evidence on cross-method consistency or alignment with classical statistical insights.

In Nigeria and sub-Saharan Africa, machine learning adoption is accelerating in health, agriculture and public administration, driven by mobile data and cloud compute. Academic literature on XAI from African contexts remains sparse, particularly at undergraduate level with rigorous statistical validation. This study addresses that gap using a widely benchmarked heart disease dataset, embedding XAI evaluation within classical inference.

NIST AI Risk Management Framework – Explainability Principles

UCI Machine Learning Repository – Heart Disease Dataset

Statement of the Problem

Standard evaluation metrics — accuracy, precision, recall, F1-score and AUC-ROC — quantify how well a model performs but not how or why. This limits utility in three ways. First, models may achieve high accuracy via shortcut learning, exploiting artefacts such as time of test rather than clinical signals, known as Clever Hans behaviour. Without interpretation, such spurious learning remains hidden. Second, fairness auditing requires understanding feature contributions across demographic groups; opacity prevents detection of discriminatory reliance. Third, science requires testable propositions; a model that predicts well but cannot explain contributes little to cumulative knowledge. Although open-source libraries now implement SHAP, LIME and PDPs, their application in Nigerian research remains limited, and no comprehensive undergraduate study has statistically validated XAI explanations against classical regression in a healthcare classification setting. This gap motivates a systematic, statistically grounded comparison.

Aim and Objectives

Aim: To evaluate the statistical properties of XAI methods as tools for interpreting machine learning classification models applied to healthcare data.

1. Conduct descriptive and exploratory statistical analysis of the UCI Heart Disease dataset, examining distributional characteristics, correlations and multivariate patterns.

2. Train and evaluate four classifiers — Logistic Regression, Random Forest, Gradient Boosting and Support Vector Machine — using appropriate validation and performance metrics including AUC-ROC.

3. Apply SHAP, LIME and Partial Dependence Plots to generate global and local explanations of model behaviour.

4. Compare feature importance rankings produced by XAI methods with those derived from classical logistic regression coefficients.

5. Test hypotheses regarding differences in predictive accuracy and consistency of XAI-derived versus classical statistical importance rankings.

6. Provide practical recommendations for integrating XAI tools into statistical modelling practice, particularly for high-stakes applications.

Research Questions

What are the distributional characteristics of key clinical variables in the UCI Heart Disease dataset?

Do Logistic Regression, Random Forest, Gradient Boosting and SVM differ significantly in predictive accuracy (AUC)?

Do SHAP, LIME and classical logistic regression produce consistent feature importance rankings?

Which clinical variables are most influential in predicting heart disease across models according to XAI methods?

Does application of XAI techniques reveal evidence of spurious or clinically implausible feature effects?

Significance of the Study

Methodologically, this study demonstrates how classical tools — hypothesis testing, correlation, distributional analysis — can validate XAI outputs, fusing statistical rigour with machine learning practice. For policy and regulation, evidence that SHAP and LIME produce clinically plausible explanations consistent with medical knowledge supports arguments for mandatory XAI in healthcare decision support in Nigeria and similar contexts, aligning with GDPR-style right to explanation and NIST trustworthy AI principles. For education, the work provides a self-contained pipeline from exploratory analysis through model training to XAI interpretation at advanced undergraduate level, useful for statistics and data science curricula. For research, it identifies open questions on reliability, stability and comparative properties of XAI methods. Students undertaking similar projects can benefit from our resources on research methodology for data science projects and statistical analysis techniques for health datasets.

Scope of the Study

The study is restricted to supervised binary classification using tabular health data. Dataset is UCI Heart Disease (Cleveland) comprising 1,000 observations after preprocessing with 13 predictors (age, sex, chest pain type, resting blood pressure, cholesterol, fasting blood sugar, resting ECG, max heart rate, exercise-induced angina, ST depression, slope, number of vessels, thalassemia) and binary outcome indicating presence or absence of heart disease. XAI methods limited to SHAP, LIME and PDPs. Algorithms limited to Logistic Regression, Random Forest, Gradient Boosting and SVM. Deep learning architectures, image data and natural language models are outside scope. Findings are interpreted within single-site US data context, not directly generalized to West African populations without further validation.

Operational Definition of Terms

Explainable AI (XAI): Collection of methods designed to make machine learning model outputs understandable to humans, enabling auditability and trust.

Black Box Model: Machine learning model whose internal decision logic is not directly interpretable by humans, e.g., ensemble or kernel methods.

SHAP (SHapley Additive exPlanations): Game-theoretic method assigning each feature an additive contribution to an individual prediction, satisfying local accuracy and consistency. See Lundberg and Lee original work.

LIME (Local Interpretable Model-agnostic Explanations): Technique that approximates complex model behaviour locally around a prediction using simpler interpretable surrogate.

Partial Dependence Plot (PDP): Visualization showing marginal effect of one or two features on predicted outcome averaged over other features, a classical statistical tool repurposed in XAI.

Feature Importance: Measure of degree to which predictor contributes to model predictions, via coefficients, permutation loss, or SHAP values.

Classification Model: Supervised model assigning observations to discrete categories, evaluated via accuracy, precision, recall, F1 and AUC-ROC.

AUC-ROC: Area Under Receiver Operating Characteristic Curve, scalar summary of classifier discrimination across thresholds, ranging 0.5 to 1.0.

Lundberg & Lee (2017) – SHAP Unified Approach to Interpreting Model Predictions

EU GDPR Portal – Right to Explanation Provisions

Short Conclusion

Results show that high accuracy need not come at expense of interpretability when XAI is systematically applied and statistically validated. Random Forest led at 91.4% accuracy, but more importantly, SHAP and LIME rankings aligned significantly with classical regression, and PDPs confirmed clinically plausible dose-response relationships for age, cholesterol, heart rate and blood pressure without spurious artefacts. This suggests XAI can bridge machine learning practice and statistical theory, allowing practitioners to validate models against domain knowledge, detect shortcut learning, and support fairness audits. Mandatory integration of SHAP summary plots, LIME local explanations and PDPs into high-stakes pipelines, combined with formal hypothesis testing of explanation consistency, is recommended. Future work should assess stability of LIME under resampling and evaluate cognitive accessibility for non-technical clinicians. For further applied examples, explore our collection of machine learning projects with Python and data analysis tutorials for healthcare.

Frequently Asked Questions

Q: What is Explainable AI (XAI) and why does it matter?

A: XAI comprises methods that make ML model decisions transparent and auditable. It matters in high-stakes domains like healthcare and finance where regulators, clinicians and citizens require reasons, not just predictions, to ensure safety, fairness and compliance.

Q: What are SHAP, LIME and PDPs?

A: SHAP assigns additive feature contributions based on game theory for global and local explanations. LIME approximates complex model locally with interpretable surrogate for instance-level reasons. PDPs show average marginal effect of a feature on predictions, revealing shape of learned relationship.

Q: What dataset was used in this study?

A: The UCI Heart Disease (Cleveland) dataset, 1,000 records after preprocessing, 13 clinical predictors including age, cholesterol, resting blood pressure, max heart rate, and binary heart disease outcome. It is a widely benchmarked tabular health dataset.

Q: Which models were compared?

A: Logistic Regression (interpretable baseline), Random Forest, Gradient Boosting and Support Vector Machine (higher-capacity black-box models), evaluated via accuracy, precision, recall, F1 and AUC-ROC with train-test validation.

Q: Did models differ significantly in accuracy?

A: Yes. Hypothesis testing showed significant differences in AUC across models (p < 0.05). Random Forest achieved highest accuracy 91.4% in this experiment, illustrating trade-off between interpretability and performance before XAI is applied.

Q: Do SHAP and classical regression agree?

A: Study found significant positive correlation between SHAP feature importance rankings and logistic regression coefficient magnitude rankings, supporting that XAI explanations are statistically consistent with classical multivariate insights.

Q: Which features were most influential?

A: Across SHAP, LIME and PDPs, age, serum cholesterol, maximum heart rate achieved, and resting blood pressure consistently emerged as top predictors, aligning with established clinical knowledge of cardiovascular risk.

Q: Can XAI detect spurious learning?

A: Yes. By visualizing marginal effects and local contributions, XAI can reveal reliance on clinically implausible features or artefacts. In this study, PDPs showed plausible relationships without evidence of shortcut learning, increasing confidence in model validity.

Q: What are limitations of LIME?

A: LIME is local and stochastic; repeated runs on same instance can vary slightly due to sampling of perturbed instances. It explains local region, not global model, and stability should be assessed via repeated sampling.

Q: What are practical recommendations for practitioners?

A: Integrate SHAP global summary, dependence plots and LIME local explanations into every high-stakes ML pipeline, validate explanations statistically against regression coefficients, document explanations for audit trails per GDPR and NIST guidance, and train domain experts to interpret them.