Explainable AI Framework for Medical Diagnosis Decision Support
Notice: This is a sample project for study and reference. Submitting it as your own work violates most universities' academic integrity policies.
Abstract
About This Research Topic
Artificial intelligence has moved from a research curiosity into a quiet, everyday presence inside clinics and hospitals, supporting decisions on everything from cardiovascular risk to diabetes screening. Yet the models that tend to predict best, gradient-boosted ensembles and deep neural networks in particular, are often the hardest to interpret. A clinician handed a risk score with no supporting rationale is effectively being asked to trust a black box with a patient's welfare, and that is a proposition regulators, professional bodies, and clinicians themselves are increasingly unwilling to accept without question.
This tension between predictive accuracy and interpretability sits at the centre of explainable AI (XAI) research in healthcare, and it is the problem this article works through in depth. Drawing on a completed applied research project, the discussion below walks through the design, implementation, and evaluation of an XAI framework for medical diagnosis decision support. Rather than treating explanation as an afterthought bolted onto a finished model, the underlying study places three complementary explanation techniques, SHAP, LIME, and rule extraction, alongside high-performing diagnostic classifiers built for heart disease and diabetes prediction, and then tests those explanations directly with practising clinicians and final-year medical students.
That last step is what distinguishes this work from a great deal of published research in the field. Many studies apply an explanation technique to a medical model and treat the mere technical presence of that explanation as evidence of transparency, without ever asking the people who will actually use it whether the explanation makes clinical sense. The sections that follow set out exactly how this study answered that question, what it found, and what it means for anyone building, evaluating, or procuring a clinical decision-support tool.
Main Abstract
Machine learning models now deliver strong predictive performance across many medical diagnosis tasks, but the models that perform best, particularly ensemble tree-based methods and deep learning architectures, are often opaque, giving little indication of the reasoning behind any single prediction. That opacity is a genuine barrier to clinical adoption. Clinicians carry professional and ethical responsibility for diagnostic decisions, and a growing body of regulatory guidance calls for some form of explanation to accompany automated decision support in healthcare.
This study responds to that gap by designing, building, and evaluating an explainable AI framework that pairs high-performing diagnostic classifiers with several complementary post-hoc explanation techniques, and by testing the resulting explanations empirically with clinical volunteers rather than relying on model accuracy alone as a stand-in for clinical usefulness. The project followed Design Science Research (DSR) methodology alongside the Cross-Industry Standard Process for Data Mining (CRISP-DM) for its data-driven components, drawing on two established public clinical datasets, the UCI Heart Disease dataset and the Pima Indians Diabetes dataset, together covering 1,536 patient records after cleaning. Each condition was modelled separately given the differing feature schemas.
Logistic Regression, Random Forest, and XGBoost were trained and compared for each condition, and the best-performing model for each was paired with SHAP for global and instance-level feature attribution, LIME for local surrogate-model explanations, and a rule-extraction technique that produced human-readable if-then rules for clearly separable cases. XGBoost delivered the strongest diagnostic performance across both conditions, reaching 88.9% accuracy and an 87.2% F1-score for heart disease prediction, and 84.6% accuracy with a 78.3% F1-score for diabetes prediction, outperforming both Logistic Regression and Random Forest.
Twelve clinical volunteers, comprising final-year medical students and practising clinicians, reviewed SHAP, LIME, and rule-based explanations for a shared set of de-identified sample cases, rating each technique on clarity, clinical plausibility, and trustworthiness. SHAP explanations scored highest on clarity and trustworthiness (4.3 and 4.1 out of 5), closely followed by rule-based explanations (4.0 and 4.2), while LIME scored lowest on stability, with several evaluators noting that repeated LIME runs on similar cases sometimes surfaced different top features, a known consequence of its local sampling procedure. The trained models and explanation modules were integrated into a Flask-based clinical decision-support dashboard combining a diagnostic risk score, a SHAP-based feature-contribution chart, and, where applicable, a corresponding rule, with an average combined prediction-and-explanation response time of 0.31 seconds. The study concludes that combining several complementary explanation techniques, empirically validated with clinical end users rather than assumed to work in advance, offers a more clinically grounded route to explainable medical AI than relying on a single technique in isolation.
Chapter One Preview
Background to the Study
Machine learning has proven strikingly effective, and in some narrow tasks close to superhuman, across a growing range of medical diagnosis and risk-stratification problems, from cardiovascular disease and diabetes risk prediction to image-based cancer detection. The catch is that the model families producing the strongest results, ensemble tree-based methods and deep neural networks in particular, tend to be opaque: they return a prediction without offering any accompanying account of which features, or which reasoning, produced it.
Explainable AI (XAI) has emerged as a direct response to this concern, developing techniques that make the internal reasoning of complex machine learning models more interpretable to human users without necessarily giving up the performance advantages that come with more complex architectures. Model-agnostic post-hoc techniques, most notably SHAP and LIME, have become particularly widely used in clinical machine learning research, since they can be applied to any underlying model without requiring an inherently interpretable design in the first place. Recent qualitative research engaging clinicians directly, however, has found a more mixed picture: feature-importance explanations are valued for being immediately visual and intuitive, but clinicians also express real scepticism about reducing complex clinical reasoning to a set of additive feature contributions, and comparative studies have shown that different explanation techniques can behave quite differently in terms of stability and clinical plausibility for the same case.
This study builds on that body of work by combining high-performing diagnostic models for two common clinical conditions with multiple complementary explanation techniques, and by empirically testing those techniques with clinical volunteers rather than assuming, in advance, that any single explanation technique is automatically useful simply because it has been technically applied.
Statement of the Problem
A large volume of published research applies SHAP, LIME, or similar explanation techniques to medical diagnosis models and presents the resulting output as evidence of improved model transparency. Comparatively few studies go further and empirically test whether clinicians actually find those explanations clear, clinically plausible, and trustworthy, or whether different techniques might suit different kinds of clinical decisions better than others. Where clinician evaluation has been carried out, the findings tend to complicate a simple 'explanations increase trust' narrative, including documented clinician scepticism toward purely additive feature-contribution explanations for genuinely complex presentations, and documented instability in LIME explanations across near-identical cases.
This study addresses that gap directly by building a structured clinical evaluation of multiple explanation techniques into the core of the research design, rather than treating the technical application of an explanation technique as sufficient evidence of its clinical value on its own. It compares SHAP, LIME, and rule extraction against one another on clarity, plausibility, trustworthiness, and stability, using a shared set of clinical cases and a shared panel of clinical evaluators, so the comparison is genuinely like-for-like.
Aim and Objectives
The aim of the study is to design, implement, and evaluate an explainable AI framework for medical diagnosis decision support that combines high-performing diagnostic models with multiple complementary explanation techniques, empirically validated with clinical volunteers.
• Review existing literature and systems relevant to explainable AI in clinical decision support, with particular attention to empirical clinician evaluation of explanation techniques.
• Train and compare Logistic Regression, Random Forest, and XGBoost diagnostic classification models for two clinical conditions, heart disease and diabetes, using established public clinical datasets.
• Implement three complementary explanation techniques, SHAP, LIME, and rule extraction, for the best-performing diagnostic model per condition.
• Conduct a structured evaluation with clinical volunteers comparing the three explanation techniques on clarity, clinical plausibility, trustworthiness, and stability.
• Implement and evaluate a web-based clinical decision-support dashboard that presents a diagnostic risk score alongside the corresponding explanation(s).
• Evaluate the overall system through diagnostic model performance metrics, the structured clinical explanation evaluation, response-time testing, and a general usability assessment.
Research Questions
• What explainable AI techniques and clinical evaluation methodologies have been applied to medical diagnosis decision support in existing literature?
• How do Logistic Regression, Random Forest, and XGBoost compare in diagnostic performance for heart disease and diabetes risk prediction?
• How do clinical volunteers rate SHAP, LIME, and rule-based explanations on clarity, clinical plausibility, and trustworthiness for a shared set of clinical cases?
• How stable are SHAP and LIME explanations across repeated generation for similar clinical cases?
• What is the response time and general usability of a web-based clinical decision-support dashboard integrating diagnostic prediction and multiple explanation techniques?
Significance of the Study
This work is significant to several groups of readers. For clinicians and healthcare institutions weighing up AI-assisted diagnosis tools, it offers empirically grounded, rather than merely assumed, evidence on which explanation techniques clinical users actually find clear, plausible, and trustworthy for common diagnostic tasks. For the academic community, it contributes a same-protocol, multi-technique comparison of explanation approaches evaluated directly by clinical volunteers rather than by technical proxy metrics alone, addressing a gap that recent XAI-in-healthcare literature reviews have explicitly flagged. Readers interested in how this kind of applied machine learning study is scoped from the ground up may also find it useful to look more broadly at how comparable healthcare and data-driven research projects are structured and evaluated.
For developers of XAI frameworks and clinical decision-support tools, the study's explicit evaluation of explanation stability offers practically relevant guidance on a limitation of LIME-style local surrogate explanations that application-focused literature does not always foreground. For regulators and healthcare AI governance bodies, the study illustrates a concrete methodology for validating explainability claims empirically rather than accepting the mere technical presence of an explanation technique as sufficient proof of interpretability.
Scope of the Study
The study does not extend to medical image-based diagnosis, integration with a live electronic health record system, a large-scale or statistically powered clinical trial, or formal regulatory validation, though these are identified as directions for further work.
Operational Definition of Terms
Term
Operational Definition
Explainable AI (XAI)
A field and set of techniques concerned with making the predictions and internal reasoning of machine learning models understandable to human users.
Post-Hoc Explanation
An explanation generated after a model has been trained, without altering its internal structure, as distinct from an inherently interpretable ('white-box') model design.
SHAP (SHapley Additive exPlanations)
A model-agnostic explanation technique, grounded in cooperative game theory, that attributes a model's prediction to the marginal contribution of each input feature.
LIME (Local Interpretable Model-agnostic Explanations)
A model-agnostic explanation technique that approximates a complex model's behaviour locally around a specific prediction using a simpler, interpretable surrogate model.
Rule Extraction
A technique for approximating a trained model's decision boundary with a set of human-readable if-then rules, typically for a subset of cases where such an approximation is reasonably faithful.
Explanation Stability
The degree to which an explanation technique produces consistent output when applied repeatedly to similar or identical input cases.
Clinical Decision Support System (CDSS)
A system designed to assist clinicians in making diagnostic or treatment decisions by providing patient-specific assessments or recommendations. The FDA's framework for AI-enabled software as a medical device is a useful reference point for how such systems are increasingly expected to be governed.
Conclusion
Explainability is not a box to tick after a model has already been trained; it is a design constraint that shapes which models, and which explanation techniques, are worth building in the first place. This study's central finding, that clinical volunteers rated SHAP and rule-based explanations more favourably than LIME on both clarity and, especially, stability, is a reminder that different post-hoc techniques are not interchangeable, and that the choice between them should rest on evidence gathered from the people who will actually rely on the output, not on convenience or popularity in the literature. Anyone scoping a similar final-year or postgraduate project in applied machine learning and healthcare AI will find the same core lesson holds well beyond these two datasets: pair a strong predictive model with more than one explanation technique, and test the explanations themselves with real users before treating the system as finished.
Frequently Asked Questions
1. What is explainable AI (XAI) in the context of medical diagnosis?
Explainable AI refers to techniques that make a machine learning model's predictions and internal reasoning understandable to human users. In medical diagnosis, this typically means showing a clinician not just a risk score, but also which patient features drove that score and how, so the recommendation can be reviewed and, where necessary, challenged.
2. Why can't clinicians just trust a highly accurate AI model without an explanation?
Clinicians carry professional and ethical responsibility for diagnostic decisions and must be able to justify them. A model that performs well on average can still be wrong for an individual patient, and without an explanation there is no way to check whether a given prediction is being driven by clinically sensible reasoning or by a spurious pattern in the training data.
3. What is the difference between SHAP and LIME?
SHAP is grounded in cooperative game theory and attributes a prediction to the marginal contribution of each input feature, producing both global and instance-level explanations. LIME instead builds a simple, interpretable surrogate model around one specific prediction to approximate the complex model's local behaviour. In this study, SHAP explanations were rated more stable and more trustworthy by clinical evaluators than LIME explanations.
4. Why did LIME score lower on explanation stability?
LIME generates explanations by sampling and perturbing data points around the case being explained, and that sampling procedure can produce somewhat different top features on repeated runs for very similar cases. Clinical evaluators in this study specifically flagged that inconsistency as a concern when comparing repeated LIME explanations for near-identical patient profiles.
5. Which machine learning model performed best for heart disease and diabetes prediction?
XGBoost outperformed both Logistic Regression and Random Forest for both conditions, reaching 88.9% accuracy and an 87.2% F1-score for heart disease prediction, and 84.6% accuracy with a 78.3% F1-score for diabetes prediction.
6. What datasets were used in this study?
The study used the UCI Heart Disease dataset and the Pima Indians Diabetes dataset, two widely used public clinical benchmark datasets, together comprising 1,536 patient records after cleaning. Each condition was modelled separately because the two datasets use different feature schemas.
7. How was the explainable AI framework evaluated by clinicians?
Twelve clinical volunteers, including final-year medical students and practising clinicians, reviewed SHAP, LIME, and rule-based explanations for a shared set of de-identified sample cases and rated each technique on clarity, clinical plausibility, and trustworthiness using a structured questionnaire.
8. Is rule extraction better than SHAP or LIME for clinical explanations?
Not universally better, but competitive on several dimensions. In this study, rule-based explanations scored close to SHAP on clarity and slightly higher on trustworthiness, which suggests that simple if-then rules remain valuable for clearly separable cases, even though they cannot approximate a complex model's decision boundary faithfully across every case.
9. How fast was the clinical decision-support dashboard in practice?
The Flask-based dashboard, which combines a diagnostic risk score with a SHAP-based feature-contribution chart and, where applicable, a corresponding rule, achieved an average combined prediction-and-explanation response time of 0.31 seconds.
10. What are the main limitations of this explainable AI framework?
The clinical evaluation involved twelve volunteers, an adequate sample for exploratory comparison but too small for strong statistical generalisation. Both datasets are established academic benchmarks drawn from specific historical patient populations, the findings apply to tabular clinical data rather than medical imaging or free text, and the evaluation used de-identified retrospective cases rather than live, time-pressured clinical decision-making.
Purchase to unlock the full material.