AI summary · arXiv: 2611.10658v1 · cs.AI
The Delta Gap: Cellular World Models for Perturbation Response Prediction Are Far Harder Than Global Correlation Implies
The essentials
Global transcriptome correlation can greatly overstate perturbation-response prediction; delta-aligned training improves delta-space performance and transfers zero-shot across cell lines.
Question
Does global Pearson correlation accurately measure whether perturbation models predict perturbation-induced changes, or is it inflated by unchanged genes? The paper also investigates whether this gap can be diagnosed and reduced.
Method
The study evaluates GEARS on Replogle K562 data (509 held-out perturbations, 5,000 genes) and Norman A549 data (107 held-out conditions, 5,045 genes), using 30 cells per condition and three seeds for experiments where reported. It compares global and control-subtracted delta metrics, tests co-expression random-walk and predicted-profile k-nearest-neighbor corrections, and trains delta-aligned models; it also evaluates zero-shot transfer from K562 to A549 on 8 conditions and 741 shared genes.
Findings
-
Result
On Replogle, GEARS had global Pearson 0.977 but delta Pearson 0.076, while a mean-training-delta baseline reached 0.267.
See the passage in the paper
On Replogle, global Pearson (0.977) is 13x higher than delta Pearson (0.076), a delta gap of Gap = 13 (Eq. 11).
-
Result
Post-hoc kNN matching raised GEARS delta Pearson to 0.211, while oracle kNN reached 0.592, indicating substantial remaining scope associated with perturbation representations.
See the passage in the paper
kNN matching with k=50 and alpha=0 (pure kNN) achieves 0.211 delta Pearson, a 3.1x improvement over GEARS one-step (0.069) and 2.1x over RWR (0.103).
-
Result
Delta-aligned training increased Replogle delta Pearson to 0.179 and improved direction accuracy and gene recovery beyond the mean-delta baseline.
See the passage in the paper
With lambda = 1.0, delta Pearson on the Replogle holdout rises from 0.076 (GEARS) to 0.132; with lambda = 3.0 it reaches 0.179, a 2.35x improvement over the vanilla model, while top-20 delta Pearson increases from 0.148 to 0.269 (1.8x).
-
Result
The delta-aligned model transferred without adaptation from K562 knockout data to A549 overexpression data, outperforming GEARS and the mean-delta baseline on the reported shared-gene evaluation.
See the passage in the paper
Its zero-shot delta Pearson (0.328) is 4.7x the trivial LinearPerturb baseline (0.070) and 2.1x the standard GEARS model (0.158), even though the target cell line (A549) and perturbation mode (overexpression) were never seen during training.
Limitations
- The study evaluates only GEARS as its deep model, so its findings may not apply to other perturbation-prediction architectures.
See the passage in the paper
First, we only evaluate GEARS as the deep model; we have not tested other 2026 models such as PRiMeFlow [7], scDFM [8], or OCOO-T [9].
- The evaluation covers only two datasets, so the gap may differ across other cell types and perturbation types.
See the passage in the paper
Second, we only test two datasets (Replogle K562 and Norman A549); the delta gap may vary across cell types and perturbation types.
- Experiments use downsampled cells per condition, which could affect absolute performance estimates.
See the passage in the paper
Third, we only use 30 cells per condition (downsampled from full datasets); full-data experiments may yield different absolute numbers, though our downsampling analysis suggests the relative ranking holds.
Why it matters
The authors argue that benchmarks should report perturbation-specific delta metrics alongside global correlations, because global scores can conceal poor prediction of perturbation effects. Their results identify delta-aligned objectives as one remedy and use oracle retrieval to motivate improving perturbation representations.
Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
GEARS: Predicting transcriptional outcomes of novel multi-gene perturbationsYusuf H. Roohani, Kexin Huang, J. Leskovec · 2022 · bioRxiv · preprint, not peer reviewed · cited 35 times · Not summarized yet
GEARS predicts single- and multi-gene perturbation transcriptomes, and the paper evaluates GEARS while measuring its performance on perturbation-induced changes.
Abstract
Cellular response to genetic perturbation is central to numerous biomedical applications from identifying genetic interactions involved in cancer to methods for regenerative medicine. However, the combinatorial explosion in the number of possible multi-gene perturbations severely limits experimental interrogation. Here, we present GEARS, a method that can predict transcriptional response to both single and multi-gene perturbations using single-cell RNA-sequencing data from perturbational screens. GEARS is uniquely able to predict outcomes of perturbing combinations consisting of novel genes that were never experimentally perturbed by leveraging geometric deep learning and a knowledge graph o…
-
Gradient-aware modeling advances AI-driven prediction of genetic perturbation effectsDixian Zhu, Livnat Jerby · 2025 · bioRxiv · preprint, not peer reviewed · cited 1 time · Not summarized yet
GARM uses gradient-aware supervision to predict unseen perturbation effects, whereas the paper evaluates delta-aligned training and corrections to GEARS.
Abstract
Predicting the transcriptional effects of genetic perturbations across diverse contexts is a central challenge in functional genomics. While single-cell perturbational assays such as Perturb-seq have generated valuable datasets, exhaustively profiling all perturbations is infeasible, underscoring the need for predictive models. We present GARM (Gradient Aligned Regression with Multi-decoder), a machine learning (ML) framework that leverages gradient-aware supervision to capture both absolute and relative perturbational effects. Across multiple large-scale datasets, GARM consistently outperforms leading approaches—including GEARS, scGPT, and GenePert—in predicting responses to unseen perturba…
-
COMPASS: Component-Wise Inference of Shared and Gene-Specific Perturbation ResponseHuan Liang, Rohit Singh · 2026 · bioRxiv · preprint, not peer reviewed · cited 2 times · Not summarized yet
COMPASS examines when mean perturbation responses conceal gene-specific effects, while the paper tests whether unchanged genes inflate global correlation.
Abstract
Predicting how a genetic perturbation reshapes a cell’s transcriptome is a central goal of computational biology. Previous studies report that the mean response across training perturbations rivals specialized models on standard accuracy metrics, even though it cannot distinguish which perturbation occurred. Across 2,270 CRISPRi perturbations measured in each of six cell lines, we show that this apparent paradox reflects a conserved organization of perturbation responses. Perturbations span a continuum from responses strongly aligned with the mean to more targeted responses that depart from it. Crucially, a perturbation’s position along this continuum is conserved across cell lines (Kendall’…
-
Diversity by Design: Addressing Mode Collapse Improves scRNA-seq Perturbation Modeling on Well-Calibrated MetricsGabriel M. Mejía, Henry E. Miller, Francis J. A. Leblanc et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 27 times · Not summarized yet
Diversity by Design attributes mean-prediction success to metric artifacts and proposes DEG-aware metrics, while the paper diagnoses inflated global correlation from unchanged genes.
Abstract
Recent benchmarks reveal that models for single-cell perturbation response are often outperformed by simply predicting the dataset mean. We trace this anomaly to a metric artifact: control-referenced deltas and unweighted error metrics reward mode collapse whenever the control is biased or the biological signal is sparse. Large-scale \textit{in silico} simulations and analysis of two real-world perturbation datasets confirm that shared reference shifts, not genuine biological change, drives high performance in these evaluations. We introduce differentially expressed gene (DEG)-aware metrics, weighted mean-squared error (WMSE) and weighted delta $R^{2}$ ($R^{2}_{w}(\Delta)$) with respect to a…
-
Reliable single-cell perturbations explain and improve model performanceXi Wang, J. Kuipers, Florian Hugi et al. · 2026 · bioRxiv · preprint, not peer reviewed · cited 3 times · Not summarized yet
Reliable single-cell perturbations assesses measurement reliability and shared response alignment, addressing another source of uncertainty in perturbation-model performance.
Abstract
Predicting single-cell transcriptional responses to perturbations is central to building the virtual cell, yet recent benchmarks show that simple baseline methods often outperform complex models, and model comparisons depend on the evaluation metric. Most studies assume that preprocessed RNA sequencing data are reliable ground truth for both training and evaluation. Here, we test this assumption by measuring the reliability of perturbations and their alignment with shared perturbation responses, classifying each perturbation as specific, shared, or unreliable. Among 7,170 perturbations from 29 datasets, 65% are unreliable, 11% shared, and 24% specific. Applying these quality labels to publis…
-
Signal, Bounds, and Baselines: Principles for Evaluating Virtual Cell Perturbation ModelsMichael Vollenweider, P. Bühlmann · 2026 · bioRxiv · preprint, not peer reviewed · cited 6 times · Not summarized yet
Signal, Bounds, and Baselines proposes DEG-based evaluation diagnostics to expose masked biological signal, as the paper advocates delta-space evaluation.
Abstract
Foundation models and deep learning systems are increasingly proposed as core components of “virtual cells” capable of forecasting transcriptomic responses to unseen perturbations. Yet rigorous evaluation of perturbation prediction in high-dimensional gene expression space remains challenging, raising concerns of reported performances. Here, we introduce the SBB principles (Signal, Bounds, and Baselines) for evaluating biological perturbation prediction. The Signal pillar introduces diagnostic meta-metrics and promotes differentially expressed gene (DEG)-based weighting or filtering to verify and recover metric sensitivity to biological signal masked by high-dimensional noise. The Bounds pil…
-
Evaluating Single-Cell Perturbation Response Models Is Far from StraightforwardMahshid Heidari, M. Karimpour, Sumana Srivatsa et al. · 2026 · bioRxiv · preprint, not peer reviewed · cited 3 times · Not summarized yet
Evaluating Single-Cell Perturbation Response Models tests metric failure modes in perturbation prediction, including correlation-based measures that the paper finds misleading.
Abstract
Predicting cellular responses to genetic and chemical perturbations remains a central challenge in single-cell biology and a key step toward building in silico virtual cells. The rapid growth of perturbation datasets and advances in deep-learning models have raised expectations for accurate and generalizable prediction. We show that these expectations are overly optimistic, largely due to the failure modes of existing evaluation metrics. In this study, using cross-splitting, controlled noise experiments, and synthetic data, we systematically evaluate both prediction models and evaluation metrics. We demonstrate that widely used metrics, including correlation-based measures and common distrib…
-
Systema: a framework for evaluating genetic perturbation response prediction beyond systematic variationRamon Viñas Torné, Maciej Wiatrak, Zoe Piran et al. · 2025 · Nature Biotechnology · cited 129 times · Not summarized yet
Systema measures systematic variation and metric bias across perturbation datasets, while the paper shows unchanged genes inflate global performance scores.
Abstract
Predicting transcriptional responses to genetic perturbations is challenging in functional genomics. While recent methods aim to infer effects of untested perturbations, their true predictive power remains unclear. Here, we show that current methods struggle to generalize beyond systematic variation, the consistent transcriptional differences between perturbed and control cells arising from selection biases or confounders. We quantify this variation in ten datasets, spanning three technologies and five cell lines, and show that common metrics are susceptible to these biases, leading to overestimated performance. To address this, we introduce Systema, an evaluation framework that emphasizes p…
Prior work it builds on
-
D2R2: Discrete Diffusion with Regulation Reinforcement for Single-Cell Perturbation PredictionNing Fan, Qi Liu, Xunuo Zhu et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
D2R2 generates perturbation responses through regulation-guided discrete diffusion, whereas the paper evaluates and modifies GEARS using delta-focused methods.
Abstract
Predicting single-cell transcriptomic responses to genetic perturbations is central to functional genomics and virtual-cell modeling. Existing approaches, however, typically predict an entire expression profile as a whole, leaving the order in which individual gene responses are generated unmodeled. To address this problem, we introduce \textbf{$D^{2}R^{2}$} (\textbf{D}iscrete \textbf{D}iffusion with \textbf{R}egulation \textbf{R}einforcement), which reformulates perturbation prediction as regulation-guided gene-wise progressive generation. A Masked Discrete Diffusion Model represents expression as ordinal tokens and reconstructs a fully masked profile step by step, allowing generated gene r…
-
Modeling and predicting single-cell multi-gene perturbation responses with scLAMBDAGe-Fei Wang, Tian-Yu Liu, Jia Zhao et al. · 2024 · bioRxiv · preprint, not peer reviewed · cited 34 times · Not summarized yet
scLAMBDA models single- and multi-gene perturbation responses with a deep generative framework, while the paper studies delta-based evaluation and GEARS corrections.
Abstract
Understanding cellular responses to genetic perturbations is essential for understanding gene regulation and phenotype formation. While high-throughput single-cell RNA-sequencing has facilitated detailed profiling of heterogeneous transcriptional responses to perturbations at the single-cell level, there remains a pressing need for computational models that can decode the mechanisms driving these responses and accurately predict outcomes to prioritize target genes for experimental design. Here, we present scLAMBDA, a deep generative learning framework designed to model and predict single-cell transcriptional responses to genetic perturbations, including single-gene and combinatorial multi-ge…
-
Benchmarking virtual cell models for in-the-wild perturbation responseXin-Jie Mao, Song-Ming Zhang, Qi-rong Wen et al. · 2026 · cited 8 times · Not summarized yet
Benchmarking virtual cell models evaluates whether perturbation predictions remain meaningful under realistic conditions, complementing the paper’s critique of standard evaluation.
Abstract
Virtual cell (VC) models aim to predict cellular responses to any perturbations in silico and have emerged as a promising approach for drug discovery and precision medicine. Yet, a clear gap still remains: while models routinely reported impressive results on standard benchmarks, it is unclear whether their predictions are truly meaningful in practice. This is mainly due to limitations in current evaluation setups, which are often overly simplified or inconsistent, and do not reflect the complexity and variability of real biological systems. Here, we introduce a standardized and modular benchmarking framework for virtual cell prediction. Our framework evaluates diverse models under in-the-wi…
-
PRiMeFlow: Capturing Complex Expression Heterogeneity in Perturbation Response ModellingZi-Chao Yan, Yan Wu, M. Ji et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
PRiMeFlow uses flow matching to model heterogeneous perturbation responses, whereas the paper evaluates delta-aligned training and post-hoc corrections to GEARS.
Abstract
Predicting the effects of perturbations in-silico on cell state can identify drivers of cell behavior at scale and accelerate drug discovery. However, modeling challenges remain due to the inherent heterogeneity of single cell gene expression and the complex, latent gene dependencies. Here, we present PRiMeFlow, an end-to-end flow matching based approach to directly model the effects of genetic and small molecule perturbations in the gene expression space. The distribution-fitting approach taken by PRiMeFlow enables it to accurately approximate the empirical distribution of single-cell gene expression, which we demonstrate through extensive benchmarking inside PerturBench. Through ablation s…
-
scDFM: Distributional Flow Matching Model for Robust Single-Cell Perturbation PredictionCheng-Lei Yu, Chuan-Rui Wang, Bangyan Liao et al. · 2026 · International Conference on Learning Representations · cited 27 times · Not summarized yet
scDFM uses conditional flow matching to predict perturbed-cell distributions, rather than the paper’s evaluation and delta-alignment approach for GEARS.
Abstract
A central goal in systems biology and drug discovery is to predict the transcriptional response of cells to perturbations. This task is challenging due to the noisy and sparse nature of single-cell measurements, as well as the fact that perturbations often induce population-level shifts rather than changes in individual cells. Existing deep learning methods typically assume cell-level correspondences, limiting their ability to capture such global effects. We present scDFM, a generative framework based on conditional flow matching that models the full distribution of perturbed cells conditioned on control states. By incorporating a maximum mean discrepancy (MMD) objective, our method aligns p…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 16 November 2026.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.