Multi-Modal Deep Learning Framework for Integrated Clinical and Imaging Data-Based Disease Diagnosis

Abstract

The growing use of artificial intelligence in medical care has had a considerable positive impact on the process of diagnosing the disease, but most of the methods used so far utilize only one type of data, resulting in a disjuncture in the analysis and poor clinical decision-making. Specifically, the complementary diagnostic information is not easily captured due to the independent use of medical imaging and clinical data. To solve this issue, this paper presents a new multi-mode deep learning framework that incorporates heterogeneous sources of data to diagnose a disease better. The proposed architecture is a convolutional neural network (CNN)-based imaging branch that can extract spatial features of medical images with a machine learning/deep learning-based clinical branch that can handle structured patient data, such as demographic, physiological, and laboratory measurements. All these modality features are combined through an intermediate feature fusion approach to form a single representation, which is then sent to a fully connected classification layer to formulate the final prediction. The framework is tested on benchmark datasets that contain imaging and clinical features, and the performance is compared to the performance of single-modality and traditional machine learning models. We find that, as demonstrated by experimental results, the proposed multi-modal method provides superior performance in diagnostic, with high accuracy, sensitivity, specificity, and area under the ROC curve (AUC) in comparison to the baseline methods. Also, the model is more robust and better generalizes to different data distributions. The results emphasize the usefulness of using multi-source medical data to provide a comprehensive diagnosis of the disease and emphasize the opportunities of the described framework to the real-life clinical decision-support systems. This strategy opens the door of the next-generation intelligent healthcare solutions to use multi-mode data in making medical diagnosis accurate, reliable and scalable.