Can saliency maps flag chest radiograph classifier errors at a new hospital? An external validation study

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Same question Cited by this paper Used in depth by this paper Not relevant

   Benchmarking saliency methods for chest X-ray interpretation

   Adriel Saporta, Xiaotong Gui, Ashwin Agrawal et al. · 2022 · Nature Machine Intelligence · cited 325 times · Not summarized yet

   Benchmarking saliency methods for chest X-ray interpretation tests how accurately saliency maps localize radiographic findings, while this study tests localization and error detection across hospitals.

   Abstract

   Saliency methods, which produce heat maps that highlight the areas of the medical image that influence model prediction, are often presented to clinicians as an aid in diagnostic decision-making. However, rigorous investigation of the accuracy and reliability of these strategies is necessary before they are integrated into the clinical setting. In this work, we quantitatively evaluate seven saliency methods, including Grad-CAM, across multiple neural network architectures using two evaluation metrics. We establish the first human benchmark for chest X-ray segmentation in a multilabel classification set-up, and examine under what clinical conditions saliency maps might be more prone to failur…

2. Very close Same question Not relevant

   Assessing the validity of saliency maps for abnormality localization in medical imaging

   N. Arun, N. Gaw, Praveer Singh et al. · 2020 · arXiv.org · preprint, not peer reviewed · cited 24 times · Not summarized yet

   Assessing the validity of saliency maps for abnormality localization in medical imaging evaluates Grad-CAM localization on RSNA pneumonia images, while this study measures anatomical localization on chest radiographs.

   Abstract

   Saliency maps have become a widely used method to assess which areas of the input image are most pertinent to the prediction of a trained neural network. However, in the context of medical imaging, there is no study to our knowledge that has examined the efficacy of these techniques and quantified them using overlap with ground truth bounding boxes. In this work, we explored the credibility of the various existing saliency map methods on the RSNA Pneumonia dataset. We found that GradCAM was the most sensitive to model parameter and label randomization, and was highly agnostic to model architecture.

3. Very close Same question Cited by this paper Not relevant

   Revisiting the Trustworthiness of Saliency Methods in Radiology AI.

   Jia-Jin Zhang, Hanqing Chao, Giridhar Dasegowda et al. · 2023 · Radiology: Artificial Intelligence · cited 80 times · Not summarized yet

   Revisiting the Trustworthiness of Saliency Methods in Radiology AI tests saliency robustness to input perturbations on chest radiographs, while this study tests anatomical plausibility and error detection under hospital shift.

   Abstract

   Purpose To determine whether saliency maps in radiology artificial intelligence (AI) are vulnerable to subtle perturbations of the input, which could lead to misleading interpretations, using prediction-saliency correlation (PSC) for evaluating the sensitivity and robustness of saliency methods. Materials and Methods In this retrospective study, locally trained deep learning models and a research prototype provided by a commercial vendor were systematically evaluated on 191 229 chest radiographs from the CheXpert dataset and 7022 MR images from a human brain tumor classification dataset. Two radiologists performed a reader study on 270 chest radiograph pairs. A model-agnostic approach for co…

4. Very close Same question Not relevant

   Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation

   A. Saporta, X. Gui, A. Agrawal et al. · 2021 · medRxiv · preprint, not peer reviewed · cited 35 times · Not summarized yet

   Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation evaluates Grad-CAM localization across chest X-ray pathologies, as this study does.

   Abstract

   Deep learning has enabled automated medical image interpretation at a level often surpassing that of practicing medical experts. However, many clinical practices have cited a lack of model interpretability as reason to delay the use of "black-box" deep neural networks in clinical workflows. Saliency maps, which "explain" a model's decision by producing heat maps that highlight the areas of the medical image that influence model prediction, are often presented to clinicians as an aid in diagnostic decision-making. In this work, we demonstrate that the most commonly used saliency map generating method, Grad-CAM, results in low performance for 10 pathologies on chest X-rays. We examined under w…

5. Very close Same question Cited by this paper Not relevant

   Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging.

   N. Arun, N. Gaw, Praveer Singh et al. · 2021 · Radiology: Artificial Intelligence · cited 304 times · Not summarized yet

   Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging evaluates saliency localization and reliability on radiographs, while this study evaluates anatomical localization and error detection.

   Abstract

   Purpose To evaluate the trustworthiness of saliency maps for abnormality localization in medical imaging. Materials and Methods Using two large publicly available radiology datasets (Society for Imaging Informatics in Medicine-American College of Radiology Pneumothorax Segmentation dataset and Radiological Society of North America Pneumonia Detection Challenge dataset), the performance of eight commonly used saliency map techniques were quantified in regard to (a) localization utility (segmentation and detection), (b) sensitivity to model weight randomization, (c) repeatability, and (d) reproducibility. Their performances versus baseline methods and localization network architectures were co…

6. Very close Same method, another problem Not relevant

   Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses

   Kesavan Venkatesh, Simukayi Mutasa, Fletcher Moore et al. · 2024 · Journal of Imaging Informatics in Medicine · cited 35 times · Not summarized yet

   Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses evaluates saliency-map localization and reliability for upper-extremity X-ray diagnoses rather than chest-radiograph error detection.

   Abstract

   Saliency maps are popularly used to “explain” decisions made by modern machine learning models, including deep convolutional neural networks (DCNNs). While the resulting heatmaps purportedly indicate important image features, their “trustworthiness,” i.e., utility and robustness, has not been evaluated for musculoskeletal imaging. The purpose of this study was to systematically evaluate the trustworthiness of saliency maps used in disease diagnosis on upper extremity X-ray images. The underlying DCNNs were trained using the Stanford MURA dataset. We studied four trustworthiness criteria—(1) localization accuracy of abnormalities, (2) repeatability, (3) reproducibility, and (4) sensitivity to…

7. Very close Same question Not relevant

   Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

   N. Arun, N. Gaw, P. Singh et al. · 2020 · medRxiv · preprint, not peer reviewed · cited 116 times · Not summarized yet

   Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging evaluates saliency localization and reliability, while this study asks whether anatomical plausibility detects classifier errors.

   Abstract

   Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image. They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the neural network makes. However, the utility and robustness of these visualization maps has not yet been rigorously examined in the context of medical imaging. We posit that trustworthiness in this context requires 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability, and 4) reproducibility. Using the localization informat…

8. Very close Broader context Not relevant

   Mask of Truth: Model Sensitivity to Unexpected Regions of Medical Images

   Théo Sourget, Michelle Hestbek-Moller, Amelia Jiménez-Sánchez et al. · 2024 · Journal of Imaging Informatics in Medicine · cited 9 times · Not summarized yet

   Mask of Truth tests whether chest X-ray and fundus classifiers rely on unexpected image regions, addressing shortcut use related to this study's anatomical plausibility assessment.

   Abstract

   The development of larger models for medical image analysis has led to increased performance. However, it also affected our ability to explain and validate model decisions. Models can use non-relevant parts of images, also called spurious correlations or shortcuts, to obtain high performance on benchmark datasets but fail in real-world scenarios. In this work, we challenge the capacity of convolutional neural networks (CNN) to classify chest X-rays and eye fundus images while masking out clinically relevant parts of the image. We show that all models trained on the PadChest dataset, irrespective of the masking strategy, are able to obtain an area under the curve (AUC) above random. Moreover,…

Prior work it builds on

1. Close Groundwork for this paper Used in depth by this paper Not relevant

   Sanity Checks for Saliency Maps

   Julius Adebayo, J. Gilmer, M. Muelly et al. · 2018 · Neural Information Processing Systems · cited 3,032 times · Not summarized yet

   Sanity Checks for Saliency Maps proposes tests showing some explanations are independent of models or data, grounding later assessments of saliency-map reliability.

   Abstract

   Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, …

2. Close Another approach to the problem Used in depth by this paper Not relevant

   Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed

   Mélanie Bernhardt, Fabio De Sousa Ribeiro, Ben Glocker · 2022 · Trans. Mach. Learn. Res. · cited 18 times · Not summarized yet

   Failure Detection in Medical Image Classification benchmarks confidence scores for classification failures, a detection approach this study compares against saliency-based scores.

   Abstract

   Failure detection in automated image classification is a critical safeguard for clinical deployment. Detected failure cases can be referred to human assessment, ensuring patient safety in computer-aided clinical decision making. Despite its paramount importance, there is insufficient evidence about the ability of state-of-the-art confidence scoring methods to detect test-time failures of classification models in the context of medical imaging. This paper provides a reality check, establishing the performance of in-domain misclassification detection methods, benchmarking 9 widely used confidence scores on 6 medical imaging datasets with different imaging modalities, in multiclass and binary c…

3. Close Same method, another problem Not relevant

   Seeing Isn't Always Believing: Analysis of Grad-CAM Faithfulness and Localization Reliability in Lung Cancer CT Classification

   Teerapong Panboonyuen · 2026 · International Conference on Knowledge and Smart Technology · cited 9 times · Not summarized yet

   Seeing Isn't Always Believing evaluates Grad-CAM faithfulness and localization for lung-cancer CT classification, applying saliency analysis to a different imaging task.

   Abstract

   Explainable Artificial Intelligence (XAI) techniques, such as Gradient-weighted Class Activation Mapping (GradCAM), have become indispensable for visualizing the reasoning process of deep neural networks in medical image analysis. Despite their popularity, the faithfulness and reliability of these heatmap-based explanations remain under scrutiny. This study critically investigates whether Grad-CAM truly represents the internal decision-making of deep models trained for lung cancer image classification. Using the publicly available IQ-OTH/NCCD dataset, we evaluate five representative architectures - ResNet50, ResNet-101, DenseNet-161, EfficientNet-B0, and ViT-Base-Patch16-224 - to explore mod…

4. Close Broader context Not relevant

   Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study

   J. Zech, Marcus A. Badgeley, Man-Way Liu et al. · 2018 · PLoS Medicine · cited 1,890 times · Not summarized yet

   Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs studies cross-hospital model performance, the external-shift setting of this study.

   Abstract

   Background There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task. Methods and findings A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 pati…

5. Close Same method, another problem Not relevant

   Beyond Accuracy: Quantifying Pulmonary Attribution in Anatomy-Guided Chest X-Ray Classification Under Domain Shift

   A. al Mamun, Md. Nasif Khansur, Md. Ashraful Hossen Akash et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Beyond Accuracy quantifies pulmonary attribution containment under domain shift with an anatomy-guided classifier, whereas this study uses saliency enrichment to assess reliability.

   Abstract

   Deep-learning models can achieve strong chest X-ray (CXR) classification performance without establishing whether their predictions predominantly rely on pulmonary image content. This study evaluates pulmonary attribution containment as an anatomy-related reliability property distinct from diagnostic performance. We propose DBCA-SegNet-MGAP, a multi-task anatomy-guided CNN-Transformer framework that combines complementary feature representations through bidirectional cross-backbone attention, predicts a soft lung mask, and incorporates this anatomical prior directly into classification through Mask-Guided Adaptive Global Average Pooling (MGAP). Pulmonary attribution containment is quantified…