AI summary · arXiv: 2404.00844v1 · math-ph
Nonlinear ensemble filtering with diffusion models: Application to the surface quasi-geostrophic dynamics
The essentials
- Intersection between classical data assimilation methods and novel machine learning techniques has sparked significant interest
- Researchers developed Ensemble Score Filter (EnSF) using diffusion models for sequential data assimilation
- EnSF does not require training, efficiently generates analysis ensemble members
- EnSF applied to surface quasi-geostrophic model, compared against Local Ensemble Transform Kalman Filter (LETKF)
- EnSF showed stable performance across various settings, competitive with LETKF for linear observations, advantageous for nonlinear observations and unexpected shocks
- EnSF excelled at large scales where LETKF lacked sufficient ensemble spread
- Promising results from initial application of EnSF to geophysical model of intermediate complexity
- Research into nonlinear/non-Gaussian ensemble data assimilation methods expected to gain traction due to growing complexity of numerical models and observing systems
- Advancements in generative artificial intelligence (GenAI) could enhance ensemble DA techniques, such as invertible neural networks showing potential for handling arbitrary distributions effectively
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Feng Bao (10 papers)
10 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
IEnSF: Iterative Ensemble Score Filter for Reducing Error in Posterior Score Estimation in Nonlinear Data AssimilationZe-Zhong Zhang, Feng Bao, Guan-Nan Zhang · 2025 · Journal of Computational Physics · cited 10 times · Not summarized yet
Abstract
The Ensemble Score Filter (EnSF) is a score-based diffusion model approach for solving high-dimensional and nonlinear data assimilation problems. While initial applications of EnSF to the Lorenz-96 model and the quasi-geostrophic system showed potential, the current method employs a heuristic weighted sum to combine the prior and the likelihood score functions. This introduces a structural error into the estimation of the posterior score function in the nonlinear setting. This work addresses this challenge by developing an iterative ensemble score filter (IEnSF) that applies an iterative algorithm as an outer loop around the reverse-time stochastic differential equation solver. When the stat…
-
Ensemble Score Filter with Image Inpainting for Data Assimilation in Tracking Surface Quasi-geostrophic Dynamics with Partial ObservationsSi-Ming Liang, Hoang Tran, Feng Bao et al. · 2026 · SIAM Journal on Applied Mathematics · Not summarized yet
Abstract
Abstract. Data assimilation plays a pivotal role in understanding and predicting turbulent systems within geoscience and weather forecasting, where data assimilation is used to address three fundamental challenges, i.e., high-dimensionality, nonlinearity, and partial observations. Recent advances in machine learning-based data assimilation methods have demonstrated encouraging results. In this work, we develop an ensemble score filter (EnSF) that integrates image inpainting to solve the data assimilation problems with partial observations. The EnSF method, proposed in our previous work [F. Bao, Z. Zhang, and G. Zhang, Comput. Methods Appl. Mech. Engrg., 432 (2024), 117447], exploits an exclu…
-
Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation DataPhillip Si, Peng Chen · 2024 · International Conference on Learning Representations · cited 30 times · Not summarized yet
Abstract
Accurate modeling and prediction of complex physical systems often rely on data assimilation techniques to correct errors inherent in model simulations. Traditional methods like the Ensemble Kalman Filter (EnKF) and its variants as well as the recently developed Ensemble Score Filters (EnSF) face significant challenges when dealing with high-dimensional and nonlinear Bayesian filtering problems with sparse observations, which are ubiquitous in real-world applications. In this paper, we propose a novel data assimilation method, Latent-EnSF, which leverages EnSF with efficient and consistent latent representations of the full states and sparse observations to address the joint challenges of hi…
-
Robustness of the Ensemble Score Filter to the Type of Assimilated Observation NetworksZixiang Xiong, Si-Ming Liang, Feng Bao et al. · 2026 · Atmospheric Science Letters · cited 6 times · Not summarized yet
Abstract
Recent advances in data assimilation (DA) have focused on developing more flexible approaches that can better accommodate nonlinearities in models and observations. However, it remains unclear how the performance of these advanced methods depends on the observation network characteristics. In this study, we present initial experiments with the surface quasi‐geostrophic model, in which we compare a recently developed ensemble filter using score‐based diffusion models with the standard Local Ensemble Transform Kalman Filter (LETKF). Our results show that the analysis solutions respond differently to the number, spatial distribution, and nonlinear fraction of assimilated observations. We also f…
-
A Two-Step Ensemble Score Filter for Data Assimilation in Partially Observed SystemsZixiang Xiong, Feng Bao, Hristo G. Chipilski et al. · 2026 · cited 4 times · Not summarized yet
Abstract
Data assimilation blends model forecasts with observations to estimate the evolving state of complex dynamical systems, but sparse observing networks remain challenging because unobserved state variables are not directly constrained by observations. In this work, we introduce the Ensemble Score Filter with Linear Regression (EnSF-LR), a two-step filtering method for partially observed nonlinear systems. At each analysis time, EnSF-LR first applies the Ensemble Score Filter (EnSF) to update the observed state components using a nonlinear score-based analysis update. It then computes the resulting observed-state analysis increments and maps these corrections to the unobserved components throug…
-
LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse ObservationsPeng Xiao, Phillip Si, Peng Chen · 2024 · arXiv.org · preprint, not peer reviewed · cited 16 times · Not summarized yet
Abstract
Data assimilation techniques are crucial for accurately tracking complex dynamical systems by integrating observational data with numerical forecasts. Recently, score-based data assimilation methods emerged as powerful tools for high-dimensional and nonlinear data assimilation. However, these methods still incur substantial computational costs due to the need for expensive forward simulations. In this work, we propose LD-EnSF, a novel score-based data assimilation method that eliminates the need for full-space simulations by evolving dynamics directly in a compact latent space. Our method incorporates improved Latent Dynamics Networks (LDNets) to learn accurate surrogate dynamics and introdu…
-
Ensemble Score Filter for Data Assimilation of Two-Phase Flow Models in Porous MediaRuo-Yu Hu, Sanjeeb Poudel, Feng Bao et al. · 2025 · Journal of Computational Physics · cited 8 times · Not summarized yet
Abstract
Numerical modeling and simulation of two-phase flow in porous media is challenging due to the uncertainties in key parameters, such as permeability. To address these challenges, we propose a computational framework by utilizing the novel Ensemble Score Filter (EnSF) to enhance the accuracy of state estimation for two-phase flow systems in porous media. The forward simulation of the two-phase flow model is implemented using a mixed finite element method, which ensures accurate approximation of the pressure, the velocity, and the saturation. The EnSF leverages score-based diffusion models to approximate filtering distributions efficiently, avoiding the computational expense of neural network-b…
-
AECSF: Adaptive Ensemble Conditional Score Filtering for High-Dimensional Nonlinear Data AssimilationYang-Wen Zhang, Shi-Wei Ni, Xiao-Ping Zhang et al. · 2026 · Not summarized yet
Abstract
Bayesian state estimation for high-dimensional nonlinear dynamical systems entails a fundamental tension between statistical fidelity and computational tractability, as particle weights can collapse, while Gaussian ensemble updates can miss non-Gaussian posterior structure. Score-based diffusion filters offer a sampling-based alternative, but existing training-free score filters often rely on heuristic likelihood corrections, which can compromise posterior accuracy by neglecting uncertainty about the system state associated with each noisy reverse particle. To address these issues, we propose AECSF, a training-free adaptive ensemble conditional score filter. AECSF constructs an analytically …
Prior work it builds on
-
A Multiscale Local Gain Form Ensemble Transform Kalman Filter (MLGETKF)Xu-Guang Wang, Hristo G. Chipilski, C. Bishop et al. · 2021 · Monthly Weather Review · cited 38 times · Not summarized yet
Abstract
A new multiscale, ensemble-based data assimilation (DA) method, multiscale local gain form ensemble transform Kalman filter (MLGETKF), is introduced. MLGETKF allows simultaneous update of multiple scales for both the ensemble mean and perturbations through assimilating all observations at once. MLGETKF performs DA in independent local volumes, which lends the algorithm a high degree of computational scalability. The multiscale analysis is enabled through the rapid creation of many pseudoensemble perturbations via a multiscale ensemble modulation procedure. The Kalman gain that is used to update the raw background ensemble mean and perturbations is based on this modulated ensemble, which intr…
-
Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filterB. Hunt, E. Kostelich, I. Szunyogh · 2005 · Physica D: Nonlinear Phenomena · cited 1,696 times · Not summarized yet
Abstract
Abstract Data assimilation is an iterative approach to the problem of estimating the state of a dynamical system using both current and past observations of the system together with a model for the system’s time evolution. Rather than solving the problem from scratch each time new observations become available, one uses the model to “forecast” the current state, using a prior state estimate (which incorporates information from past data) as the initial condition, then uses current data to correct the prior forecast to a current state estimate. This Bayesian approach is most effective when the uncertainty in both the observations and in the state estimate, as it evolves over time, are accurat…
-
A Unified Filter Method for Jointly Estimating State and Parameters of Stochastic Dynamical Systems via the Ensemble Score FilterFeng Bao, Guan-Nan Zhang, Ze-Zhong Zhang · 2023 · cited 5 times · Not summarized yet
Abstract
This paper tackles the intricate task of jointly estimating state and parameters in data assimilation for stochastic dynamical systems that are affected by noise and observed only partially. While the concept of ``optimal filtering'' serves as the customary approach to estimate the state of the target dynamical system, traditional methods such as Kalman filters and particle filters encounter significant challenges when dealing with high-dimensional and nonlinear problems. When we also consider the scenario where the model parameters are unknown, the problem transforms into a joint state-parameter estimation endeavor. Presently, the leading-edge technique known as the Augmented Ensemble Kalma…
-
A Quantile-Conserving Ensemble Filter Framework. Part II: Regression of Observation Increments in a Probit and Probability Integral Transformed SpaceJeffrey L. Anderson · 2023 · Monthly Weather Review · cited 17 times · Not summarized yet
Abstract
Traditional ensemble Kalman filter data assimilation methods make implicit assumptions of Gaussianity and linearity that are strongly violated by many important Earth system applications. For instance, bounded quantities like the amount of a tracer and sea ice fractional coverage cannot be accurately represented by a Gaussian which is unbounded by definition. Nonlinear relations between observations and model state variables abound. Examples include the relation between a remotely sensed radiance and the column of atmospheric temperatures, or the relation between cloud amount and water vapor quantity. Part 1 of this paper described a very general data assimilation framework for computing obs…
-
Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statisticsG. Evensen · 1994 · Journal of Geophysical Research · cited 5,444 times · Not summarized yet
Papers that cite it
-
The Ensemble Schr{ö}dinger Bridge filter for Nonlinear Data AssimilationFeng Bao, Hui Sun · 2025 · Foundations of Data Science · cited 3 times · Not summarized yet
Abstract
This work introduces a novel nonlinear optimal filtering method, termed the Ensemble Schr{\"o}dinger Bridge nonlinear filter. The proposed filter combines the standard prediction step with a diffusion-generative-modeling-based analysis step, thereby completing one full filtering update. The resulting approach introduces no structural model error, and is derivative-free, training-free, and highly parallelizable. Numerical experiments demonstrate that the proposed algorithm performs effectively for highly nonlinear dynamics and nonlinear observation processes, including chaotic systems with dimension up to 40 and beyond. The results also show that the method outperforms classical approaches su…
-
EnJoi: Ensemble Joint Score Filter for Generative Data AssimilationJulien Moreau, Marc Lelarge · 2026 · Not summarized yet
Abstract
Data Assimilation (DA) aims to recover the full state of a dynamical system that is only partially observed. A solution is to use Score-based models to generate physically consistent trajectories that agree with the observations. These Autoregressive Diffusion models are trained by conditioning on the previous state; however, they do not take into account the uncertainty of their past predictions. We propose a new diffusion-based assimilation algorithm that dynamically balances the confidence in the current state and the new observations. Crucially, we choose to learn the distribution of the joint state containing both the past and future. This allows us to use a modified version of En4DVar,…
-
Two Localization Strategies for Sequential MCMC Data Assimilation With Applications to Nonlinear Non‐Gaussian Geophysical ModelsHamza M. Ruzayqat, Hristo G. Chipilski, O. Knio · 2026 · Journal of Advances in Modeling Earth Systems · Not summarized yet
Abstract
We present a localized data assimilation (DA) scheme based on sequential Markov Chain Monte Carlo (SMCMC) (Ruzayqat et al., 2024, https://doi.org/10.1002/qj.4710), a provably convergent method for filtering high‐dimensional, nonlinear, and potentially non‐Gaussian state‐space models (SSMs). Unlike particle filters, which are exact for such models, SMCMC does not assign weights to samples, avoiding weight degeneracy in small‐ensemble regimes. We design two localization approaches within the SMCMC framework that exploit observation sparsity to reduce effective degrees of freedom. The first variant collects observed blocks into a single reduced domain and runs parallel MCMC chains over this com…
-
Guided Unconditional and Conditional Generative Models for Super‐Resolution and Inference of Quasi‐Geostrophic TurbulenceAnantha Narayanan Suresh Babu, Akhil Sadam, P. F. J. Lermusiaux · 2025 · Journal of Advances in Modeling Earth Systems · cited 6 times · Not summarized yet
Abstract
Typically, numerical simulations of Earth systems are coarse, and Earth observations are sparse and gappy. We apply four generative diffusion modeling approaches to super‐resolution and inference of forced two‐dimensional quasi‐geostrophic turbulence on the β $\beta $ ‐plane from coarse, sparse, and gappy observations. Two guided approaches minimally adapt a pre‐trained unconditional model: SDEdit modifies the initial condition, and Diffusion Posterior Sampling (DPS) modifies the reverse diffusion process's score. Two conditional approaches, a vanilla variant and classifier‐free guidance, require training with paired high‐resolution and observation data. We consider multiple test cases spann…
-
DAISI: Data Assimilation with Inverse Sampling using Stochastic InterpolantsMartin Andrae, Erik Larsson, S. Takao et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
Abstract
Data assimilation (DA) is a cornerstone of scientific and engineering applications, combining model forecasts with sparse and noisy observations to estimate latent system states. Classical high-dimensional DA methods, such as the ensemble Kalman filter, rely on Gaussian approximations that are violated for complex dynamics or observation operators. To address this limitation, we introduce DAISI, a scalable filtering algorithm built on flow-based generative models that enables flexible probabilistic inference using data-driven priors. The core idea is to use a stationary, pre-trained generative prior that first incorporates forecast information through a novel inverse-sampling step, before as…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 3 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.