Summarize Paper logo SummarizePaper.com

AI summary · arXiv: 2611.05011v1 · cs.CV

Can saliency maps flag chest radiograph classifier errors at a new hospital? An external validation study

Aashish Chapagai, Angad Singh Gurtatta, Erik Cambria
Original paper Submitted to arXiv on 03 Nov. 2026 21 pages, 5 figures, 1 table, 6 Extended Data figures and 4 Extended Data tables. Code: https://github.com/aashish-chapagai/CXR-Saliency-External-Validation License: NONEXCLUSIVE-DISTRIB 1.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Across three hospitals, saliency enrichment did not reliably detect chest-radiograph classifier errors beyond confidence and sometimes harmed transferred error detection.

Question

Can anatomical plausibility scores from saliency maps identify individual chest-radiograph classifier errors at new hospitals, and do they improve on confidence-based detection under external shift?

Method

The authors trained DenseNet-121 and ViT-B/16 models on NIH ChestX-ray14 and evaluated them on 25,554 radiographs from the NIH test set, Stanford CheXpert and Vietnam’s VinDr-CXR, using 12 harmonized findings and radiologist-consensus labels at the external sites. For 303,768 decisions per model, they measured anatomical enrichment of Grad-CAM, with LIME and SHAP on subsets, and compared error-detection performance with margin and test-time-augmentation scores; combinations were calibrated on half the NIH test patients and evaluated on held-out NIH and external sites.

Findings

  • Result Grad-CAM localization was less accurate than radiologists on the CheXpert benchmark, with hit rates of 0.575 for DenseNet-121 and 0.381 for ViT-B/16 versus 0.888 for radiologists.
  • Result Anatomical enrichment generally separated correct from incorrect flags poorly and its relationship with correctness varied across hospitals.
  • Result Adding enrichment to a detector calibrated on NIH did not produce a meaningful gain; for DenseNet-121 it reduced performance at VinDr-CXR.
    See the passage in the paper
    Adding enrichment never improved failure detection meaningfully.
  • Authors’ interpretation The margin was the strongest single error-detection score in five of six model–site combinations, while enrichment could be misleading after transfer.
  • Authors’ interpretation For global-average-pooling classifiers, Grad-CAM is a spatial rearrangement of evidence already summarized by the class score, rather than an independent measurement.

Limitations

  • The NIH reference labels are report-mined and noisy, affecting internal error estimates.
    See the passage in the paper
    NIH labels are report-mined and noisy31, which affects internal failure counts, although both external sites used radiologist-consensus labels.
  • The study examined only one explanation summary, and other explanation statistics might behave differently.
  • The authors did not conduct a reader study, so they cannot establish how radiologists would use heatmaps at a new hospital.
    See the passage in the paper
    Finally, we did not run a reader study, so we cannot say how radiologists would use heatmaps at a new hospital.

Why it matters

The authors argue that anatomical plausibility should not be treated as patient-level assurance or as a dependable error monitor after hospital transfer. Deployment monitoring should be anchored in confidence, with thresholds locally retuned before use.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: Chest radiography; saliency maps; external validation; error detection; domain shift

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
The paper examines whether saliency maps can help identify individual chest-radiograph classification errors at hospitals different from those used for training. The authors studied DenseNet-121 and ViT-B/16 models trained on NIH ChestX-ray14. They evaluated the models on an internal NIH test set and on external radiographs from Stanford Hospital’s CheXpert dataset and two hospitals in Hanoi’s VinDr-CXR dataset. The study covered 12 harmonized findings and 303,768 radiograph–finding decisions per model. Labels were derived from report mining at NIH and radiologist consensus at the external sites. The primary focus was on false-positive flags, though the authors also examined missed findings. For each decision, they compared confidence-related signals with the anatomical concentration of the model’s explanations. They generated Grad-CAM maps and, for subsets, LIME and SHAP maps. Their anatomical-enrichment measure compared the proportion of attribution within a segmented thoracic region with the proportion of the image occupied by that region; a value of 1 indicates uniform attribution. The authors tested whether enrichment varied across hospitals, distinguished correct from incorrect flags, and improved error detection when added to a detector based on decision margin and test-time augmentation agreement. They also assessed localization against radiologist annotations and, for CheXpert, compared it with radiologists’ localization performance. The models retained reasonable ranking performance across sites, but thresholds tuned on NIH transferred poorly. For example, DenseNet-121’s edema sensitivity fell from 0.424 on NIH to 0.008 on CheXpert, despite an external AUROC above 0.82. Many positive flags were incorrect: precision ranged from 0.358 to 0.529 for DenseNet-121 and from 0.343 to 0.514 for ViT-B/16 across the three sites. Grad-CAM localization was weaker than the radiologist benchmark on CheXpert. Hit rates were 0.575 for DenseNet-121 and 0.381 for ViT-B/16, compared with 0.888 for radiologists. Localization also varied substantially by finding and site. Anatomical enrichment did not provide a stable indication of correctness. Among flagged findings at NIH, its error-detection AUROC was 0.484 for DenseNet-121 and 0.524 for ViT-B/16, compared with 0.646 and 0.660, respectively, for the absolute decision margin. At CheXpert, enrichment was near chance for both models. At VinDr-CXR, DenseNet-121’s enrichment was more informative on its own, with an AUROC of 0.618, but still underperformed the decision margin’s 0.685. ViT-B/16’s enrichment remained near chance, at 0.522. Adding enrichment to the confidence-based detector did not meaningfully improve performance. The largest reported upper confidence bound for improvement was 0.0104, and enrichment worsened DenseNet-121’s error detection on VinDr-CXR. In that case, the relationship between enrichment and correctness reversed between NIH and VinDr-CXR, making the NIH-calibrated signal misleading after transfer. LIME and SHAP showed a similar site-dependent pattern for DenseNet-121. The authors conclude that a plausible-looking heatmap should not be taken as reassurance that an individual prediction is correct. Confidence-based signals were more useful for monitoring, although their performance was modest. Local threshold tuning may also be necessary because operating points did not transfer reliably. The study was limited to two architectures, one enrichment-based summary of explanation plausibility, and a specific set of datasets and sites. LIME and SHAP were computed only for subsets, transformer-specific attribution methods were not tested, some findings had few positive cases, and NIH report-derived labels were noisy.
Created on 09 Nov. 2026

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.
AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in
Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero
Citation alerts Citation alerts guide
Who works on it? The people and institutions who publish on its topic
Weekly letter Presented in "Artificial intelligence this week" of 9 November 2026. Artificial intelligence this week: When better scores miss the point
Journal club Premium+ Slides to present it Propose for the journal club
Versions What changed between versions
Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Same question Cited by this paper Used in depth by this paper
    Benchmarking saliency methods for chest X-ray interpretation
    Adriel Saporta, Xiaotong Gui, Ashwin Agrawal et al. · 2022 · Nature Machine Intelligence · cited 325 times · Not summarized yet

    Benchmarking saliency methods for chest X-ray interpretation tests how accurately saliency maps localize radiographic findings, while this study tests localization and error detection across hospitals.

    Abstract

    Saliency methods, which produce heat maps that highlight the areas of the medical image that influence model prediction, are often presented to clinicians as an aid in diagnostic decision-making. However, rigorous investigation of the accuracy and reliability of these strategies is necessary before they are integrated into the clinical setting. In this work, we quantitatively evaluate seven saliency methods, including Grad-CAM, across multiple neural network architectures using two evaluation metrics. We establish the first human benchmark for chest X-ray segmentation in a multilabel classification set-up, and examine under what clinical conditions saliency maps might be more prone to failur…

  2. Very close Same question
    Assessing the validity of saliency maps for abnormality localization in medical imaging
    N. Arun, N. Gaw, Praveer Singh et al. · 2020 · arXiv.org · preprint, not peer reviewed · cited 24 times · Not summarized yet

    Assessing the validity of saliency maps for abnormality localization in medical imaging evaluates Grad-CAM localization on RSNA pneumonia images, while this study measures anatomical localization on chest radiographs.

    Abstract

    Saliency maps have become a widely used method to assess which areas of the input image are most pertinent to the prediction of a trained neural network. However, in the context of medical imaging, there is no study to our knowledge that has examined the efficacy of these techniques and quantified them using overlap with ground truth bounding boxes. In this work, we explored the credibility of the various existing saliency map methods on the RSNA Pneumonia dataset. We found that GradCAM was the most sensitive to model parameter and label randomization, and was highly agnostic to model architecture.

  3. Very close Same question Cited by this paper
    Revisiting the Trustworthiness of Saliency Methods in Radiology AI.
    Jia-Jin Zhang, Hanqing Chao, Giridhar Dasegowda et al. · 2023 · Radiology: Artificial Intelligence · cited 80 times · Not summarized yet

    Revisiting the Trustworthiness of Saliency Methods in Radiology AI tests saliency robustness to input perturbations on chest radiographs, while this study tests anatomical plausibility and error detection under hospital shift.

    Abstract

    Purpose To determine whether saliency maps in radiology artificial intelligence (AI) are vulnerable to subtle perturbations of the input, which could lead to misleading interpretations, using prediction-saliency correlation (PSC) for evaluating the sensitivity and robustness of saliency methods. Materials and Methods In this retrospective study, locally trained deep learning models and a research prototype provided by a commercial vendor were systematically evaluated on 191 229 chest radiographs from the CheXpert dataset and 7022 MR images from a human brain tumor classification dataset. Two radiologists performed a reader study on 270 chest radiograph pairs. A model-agnostic approach for co…

  4. Very close Same question
    Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation
    A. Saporta, X. Gui, A. Agrawal et al. · 2021 · medRxiv · preprint, not peer reviewed · cited 35 times · Not summarized yet

    Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation evaluates Grad-CAM localization across chest X-ray pathologies, as this study does.

    Abstract

    Deep learning has enabled automated medical image interpretation at a level often surpassing that of practicing medical experts. However, many clinical practices have cited a lack of model interpretability as reason to delay the use of "black-box" deep neural networks in clinical workflows. Saliency maps, which "explain" a model's decision by producing heat maps that highlight the areas of the medical image that influence model prediction, are often presented to clinicians as an aid in diagnostic decision-making. In this work, we demonstrate that the most commonly used saliency map generating method, Grad-CAM, results in low performance for 10 pathologies on chest X-rays. We examined under w…

  5. Very close Same question Cited by this paper
    Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging.
    N. Arun, N. Gaw, Praveer Singh et al. · 2021 · Radiology: Artificial Intelligence · cited 304 times · Not summarized yet

    Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging evaluates saliency localization and reliability on radiographs, while this study evaluates anatomical localization and error detection.

    Abstract

    Purpose To evaluate the trustworthiness of saliency maps for abnormality localization in medical imaging. Materials and Methods Using two large publicly available radiology datasets (Society for Imaging Informatics in Medicine-American College of Radiology Pneumothorax Segmentation dataset and Radiological Society of North America Pneumonia Detection Challenge dataset), the performance of eight commonly used saliency map techniques were quantified in regard to (a) localization utility (segmentation and detection), (b) sensitivity to model weight randomization, (c) repeatability, and (d) reproducibility. Their performances versus baseline methods and localization network architectures were co…

  6. Very close Same method, another problem
    Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses
    Kesavan Venkatesh, Simukayi Mutasa, Fletcher Moore et al. · 2024 · Journal of Imaging Informatics in Medicine · cited 35 times · Not summarized yet

    Gradient-Based Saliency Maps Are Not Trustworthy Visual Explanations of Automated AI Musculoskeletal Diagnoses evaluates saliency-map localization and reliability for upper-extremity X-ray diagnoses rather than chest-radiograph error detection.

    Abstract

    Saliency maps are popularly used to “explain” decisions made by modern machine learning models, including deep convolutional neural networks (DCNNs). While the resulting heatmaps purportedly indicate important image features, their “trustworthiness,” i.e., utility and robustness, has not been evaluated for musculoskeletal imaging. The purpose of this study was to systematically evaluate the trustworthiness of saliency maps used in disease diagnosis on upper extremity X-ray images. The underlying DCNNs were trained using the Stanford MURA dataset. We studied four trustworthiness criteria—(1) localization accuracy of abnormalities, (2) repeatability, (3) reproducibility, and (4) sensitivity to…

  7. Very close Same question
    Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging
    N. Arun, N. Gaw, P. Singh et al. · 2020 · medRxiv · preprint, not peer reviewed · cited 116 times · Not summarized yet

    Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging evaluates saliency localization and reliability, while this study asks whether anatomical plausibility detects classifier errors.

    Abstract

    Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image. They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the neural network makes. However, the utility and robustness of these visualization maps has not yet been rigorously examined in the context of medical imaging. We posit that trustworthiness in this context requires 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability, and 4) reproducibility. Using the localization informat…

  8. Very close Broader context
    Mask of Truth: Model Sensitivity to Unexpected Regions of Medical Images
    Théo Sourget, Michelle Hestbek-Moller, Amelia Jiménez-Sánchez et al. · 2024 · Journal of Imaging Informatics in Medicine · cited 9 times · Not summarized yet

    Mask of Truth tests whether chest X-ray and fundus classifiers rely on unexpected image regions, addressing shortcut use related to this study's anatomical plausibility assessment.

    Abstract

    The development of larger models for medical image analysis has led to increased performance. However, it also affected our ability to explain and validate model decisions. Models can use non-relevant parts of images, also called spurious correlations or shortcuts, to obtain high performance on benchmark datasets but fail in real-world scenarios. In this work, we challenge the capacity of convolutional neural networks (CNN) to classify chest X-rays and eye fundus images while masking out clinically relevant parts of the image. We show that all models trained on the PadChest dataset, irrespective of the masking strategy, are able to obtain an area under the curve (AUC) above random. Moreover,…

Prior work it builds on

  1. Close Groundwork for this paper Used in depth by this paper
    Sanity Checks for Saliency Maps
    Julius Adebayo, J. Gilmer, M. Muelly et al. · 2018 · Neural Information Processing Systems · cited 3,032 times · Not summarized yet

    Sanity Checks for Saliency Maps proposes tests showing some explanations are independent of models or data, grounding later assessments of saliency-map reliability.

    Abstract

    Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, …

  2. Close Another approach to the problem Used in depth by this paper
    Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed
    Mélanie Bernhardt, Fabio De Sousa Ribeiro, Ben Glocker · 2022 · Trans. Mach. Learn. Res. · cited 18 times · Not summarized yet

    Failure Detection in Medical Image Classification benchmarks confidence scores for classification failures, a detection approach this study compares against saliency-based scores.

    Abstract

    Failure detection in automated image classification is a critical safeguard for clinical deployment. Detected failure cases can be referred to human assessment, ensuring patient safety in computer-aided clinical decision making. Despite its paramount importance, there is insufficient evidence about the ability of state-of-the-art confidence scoring methods to detect test-time failures of classification models in the context of medical imaging. This paper provides a reality check, establishing the performance of in-domain misclassification detection methods, benchmarking 9 widely used confidence scores on 6 medical imaging datasets with different imaging modalities, in multiclass and binary c…

  3. Close Same method, another problem
    Seeing Isn't Always Believing: Analysis of Grad-CAM Faithfulness and Localization Reliability in Lung Cancer CT Classification
    Teerapong Panboonyuen · 2026 · International Conference on Knowledge and Smart Technology · cited 9 times · Not summarized yet

    Seeing Isn't Always Believing evaluates Grad-CAM faithfulness and localization for lung-cancer CT classification, applying saliency analysis to a different imaging task.

    Abstract

    Explainable Artificial Intelligence (XAI) techniques, such as Gradient-weighted Class Activation Mapping (GradCAM), have become indispensable for visualizing the reasoning process of deep neural networks in medical image analysis. Despite their popularity, the faithfulness and reliability of these heatmap-based explanations remain under scrutiny. This study critically investigates whether Grad-CAM truly represents the internal decision-making of deep models trained for lung cancer image classification. Using the publicly available IQ-OTH/NCCD dataset, we evaluate five representative architectures - ResNet50, ResNet-101, DenseNet-161, EfficientNet-B0, and ViT-Base-Patch16-224 - to explore mod…

  4. Close Broader context
    Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study
    J. Zech, Marcus A. Badgeley, Man-Way Liu et al. · 2018 · PLoS Medicine · cited 1,890 times · Not summarized yet

    Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs studies cross-hospital model performance, the external-shift setting of this study.

    Abstract

    Background There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task. Methods and findings A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 pati…

  5. Close Same method, another problem
    Beyond Accuracy: Quantifying Pulmonary Attribution in Anatomy-Guided Chest X-Ray Classification Under Domain Shift
    A. al Mamun, Md. Nasif Khansur, Md. Ashraful Hossen Akash et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

    Beyond Accuracy quantifies pulmonary attribution containment under domain shift with an anatomy-guided classifier, whereas this study uses saliency enrichment to assess reliability.

    Abstract

    Deep-learning models can achieve strong chest X-ray (CXR) classification performance without establishing whether their predictions predominantly rely on pulmonary image content. This study evaluates pulmonary attribution containment as an anatomy-related reliability property distinct from diagnostic performance. We propose DBCA-SegNet-MGAP, a multi-task anatomy-guided CNN-Transformer framework that combines complementary feature representations through bidirectional cross-backbone attention, predicts a soft lung mask, and incorporates this anatomical prior directly into classification through Mask-Guided Adaptive Global Average Pooling (MGAP). Pulmonary attribution containment is quantified…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 9 November 2026.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.