Agentic AI Framework For Explainable Clinical Decision Support Using Multimodal Electronic Health Records And Medical Imaging

Abstract

Although the efficiency of clinical decision support system has been greatly increased by integrating health care analytics technologies, typical AI models remain as black-box with limited interpretability and lacking integration of heterogeneous clinical data sources. We propose a new Hierarchical Agentic Explainable Multimodal Reasoning Network (HAEMR-Net), which is a set of intelligent agents-based AI architecture to achieve both explainability and accuracy with the integration of multimodal Electronic Health Records (EHRs) and medical imaging. In the proposed architecture, four collaborative intelligent agents are developed: (i) Data Harmonization Agent for data preprocessing, (ii) Clinical Knowledge Reasoning Agent for reasoning based on clinical knowledge graphs and context information, (iii) Explainability Agent for visualising the feature-attribution and attention maps via SHAP value and attention mechanisms, and (iv) Decision Validation Agent for verifying model decisions using confidence score estimation and rule-based analysis. We conducted extensive experimental evaluations on a multimodal EHRs dataset containing 15,240 patient records along with corresponding diagnostic images, retrieved from publicly accessible healthcare repositories. The experiments compared HAEMR-Net with existing multimodal models in terms of accuracy, precision, recall, F1-score, Area Under the Receiver Operating Characteristic Curve (AUC) and Explanation Fidelity. The results show that HAEMR-Net achieve promising results by reaching 97.4% accuracy, 96.9% precision, 97.1% recall, 97.0% F1-score, and 98.2% AUC, outperforming other mainstream deep learning and transformer-based multimodal models in major evaluation indicators by 3.8-6.5%. Additionally, the explanation component improves trust of clinical diagnosis via the production of feature-level explanation and region-based justification, with an Explanation Fidelity score of 95.6%. Our agentic model successfully tackles multimodal challenges by seamlessly integrating heterogeneous data sources and enhances explainability of AI-based healthcare system, representing a novel, effective and interpretable solution for next-generation intelligent health care.