Nonlinear ensemble filtering with diffusion models: Application to the surface quasi-geostrophic dynamics

Auteurs : Feng Bao, Hristo G. Chipilski, Siming Liang, Guannan Zhang, Jeffrey S. Whitaker

Soumis à arXiv le 01 Avr. 2024
Licence : CC BY 4.0
Résumé scientifique

The intersection between classical data assimilation methods and novel machine learning techniques has attracted significant interest in recent years. Here we explore another promising solution in which diffusion models are used to formulate a robust nonlinear ensemble filter for sequential data assimilation. Unlike standard machine learning methods, the proposed \textit{Ensemble Score Filter (EnSF)} is completely training-free and can efficiently generate a set of analysis ensemble members. In this study, we apply the EnSF to a surface quasi-geostrophic model and compare its performance against the popular Local Ensemble Transform Kalman Filter (LETKF), which makes Gaussian assumptions on the posterior distribution. Numerical tests demonstrate that EnSF maintains stable performance in the absence of localization and for a variety of experimental settings. We find that EnSF achieves competitive performance relative to LETKF in the case of linear observations, but leads to significant advantages when the state is nonlinearly observed and the numerical model is subject to unexpected shocks. A spectral decomposition of the analysis results shows that the largest improvements over LETKF occur at large scales (small wavenumbers) where LETKF lacks sufficient ensemble spread. Overall, this initial application of EnSF to a geophysical model of intermediate complexity is very encouraging, and motivates further developments of the algorithm for more realistic problems.

Les articles autour de lui, par date et par proximité

Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article

Comment lire cette carte ? Guide illustré

Les articles les plus proches Les travaux antérieurs sur lesquels il s’appuie Les articles qui le citent

Les articles autour de celui-ci

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

  1. Très proche Cite cet article
    IEnSF: Iterative Ensemble Score Filter for Reducing Error in Posterior Score Estimation in Nonlinear Data Assimilation
    Ze-Zhong Zhang, Feng Bao, Guan-Nan Zhang · 2025 · Journal of Computational Physics · cité 10 fois · Pas encore résumé
    Résumé scientifique

    The Ensemble Score Filter (EnSF) is a score-based diffusion model approach for solving high-dimensional and nonlinear data assimilation problems. While initial applications of EnSF to the Lorenz-96 model and the quasi-geostrophic system showed potential, the current method employs a heuristic weighted sum to combine the prior and the likelihood score functions. This introduces a structural error into the estimation of the posterior score function in the nonlinear setting. This work addresses this challenge by developing an iterative ensemble score filter (IEnSF) that applies an iterative algorithm as an outer loop around the reverse-time stochastic differential equation solver. When the stat…

  2. Très proche
    Ensemble Score Filter with Image Inpainting for Data Assimilation in Tracking Surface Quasi-geostrophic Dynamics with Partial Observations
    Si-Ming Liang, Hoang Tran, Feng Bao et al. · 2026 · SIAM Journal on Applied Mathematics · Pas encore résumé
    Résumé scientifique

    Abstract. Data assimilation plays a pivotal role in understanding and predicting turbulent systems within geoscience and weather forecasting, where data assimilation is used to address three fundamental challenges, i.e., high-dimensionality, nonlinearity, and partial observations. Recent advances in machine learning-based data assimilation methods have demonstrated encouraging results. In this work, we develop an ensemble score filter (EnSF) that integrates image inpainting to solve the data assimilation problems with partial observations. The EnSF method, proposed in our previous work [F. Bao, Z. Zhang, and G. Zhang, Comput. Methods Appl. Mech. Engrg., 432 (2024), 117447], exploits an exclu…

  3. Très proche Cite cet article
    Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data
    Phillip Si, Peng Chen · 2024 · International Conference on Learning Representations · cité 30 fois · Pas encore résumé
    Résumé scientifique

    Accurate modeling and prediction of complex physical systems often rely on data assimilation techniques to correct errors inherent in model simulations. Traditional methods like the Ensemble Kalman Filter (EnKF) and its variants as well as the recently developed Ensemble Score Filters (EnSF) face significant challenges when dealing with high-dimensional and nonlinear Bayesian filtering problems with sparse observations, which are ubiquitous in real-world applications. In this paper, we propose a novel data assimilation method, Latent-EnSF, which leverages EnSF with efficient and consistent latent representations of the full states and sparse observations to address the joint challenges of hi…

  4. Très proche Cite cet article
    Robustness of the Ensemble Score Filter to the Type of Assimilated Observation Networks
    Zixiang Xiong, Si-Ming Liang, Feng Bao et al. · 2026 · Atmospheric Science Letters · cité 6 fois · Pas encore résumé
    Résumé scientifique

    Recent advances in data assimilation (DA) have focused on developing more flexible approaches that can better accommodate nonlinearities in models and observations. However, it remains unclear how the performance of these advanced methods depends on the observation network characteristics. In this study, we present initial experiments with the surface quasi‐geostrophic model, in which we compare a recently developed ensemble filter using score‐based diffusion models with the standard Local Ensemble Transform Kalman Filter (LETKF). Our results show that the analysis solutions respond differently to the number, spatial distribution, and nonlinear fraction of assimilated observations. We also f…

  5. Très proche Cite cet article
    A Two-Step Ensemble Score Filter for Data Assimilation in Partially Observed Systems
    Zixiang Xiong, Feng Bao, Hristo G. Chipilski et al. · 2026 · cité 4 fois · Pas encore résumé
    Résumé scientifique

    Data assimilation blends model forecasts with observations to estimate the evolving state of complex dynamical systems, but sparse observing networks remain challenging because unobserved state variables are not directly constrained by observations. In this work, we introduce the Ensemble Score Filter with Linear Regression (EnSF-LR), a two-step filtering method for partially observed nonlinear systems. At each analysis time, EnSF-LR first applies the Ensemble Score Filter (EnSF) to update the observed state components using a nonlinear score-based analysis update. It then computes the resulting observed-state analysis increments and maps these corrections to the unobserved components throug…

  6. Très proche Cite cet article
    LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations
    Peng Xiao, Phillip Si, Peng Chen · 2024 · arXiv.org · preprint, non relu par les pairs · cité 16 fois · Pas encore résumé
    Résumé scientifique

    Data assimilation techniques are crucial for accurately tracking complex dynamical systems by integrating observational data with numerical forecasts. Recently, score-based data assimilation methods emerged as powerful tools for high-dimensional and nonlinear data assimilation. However, these methods still incur substantial computational costs due to the need for expensive forward simulations. In this work, we propose LD-EnSF, a novel score-based data assimilation method that eliminates the need for full-space simulations by evolving dynamics directly in a compact latent space. Our method incorporates improved Latent Dynamics Networks (LDNets) to learn accurate surrogate dynamics and introdu…

  7. Très proche Cite cet article
    Ensemble Score Filter for Data Assimilation of Two-Phase Flow Models in Porous Media
    Ruo-Yu Hu, Sanjeeb Poudel, Feng Bao et al. · 2025 · Journal of Computational Physics · cité 8 fois · Pas encore résumé
    Résumé scientifique

    Numerical modeling and simulation of two-phase flow in porous media is challenging due to the uncertainties in key parameters, such as permeability. To address these challenges, we propose a computational framework by utilizing the novel Ensemble Score Filter (EnSF) to enhance the accuracy of state estimation for two-phase flow systems in porous media. The forward simulation of the two-phase flow model is implemented using a mixed finite element method, which ensures accurate approximation of the pressure, the velocity, and the saturation. The EnSF leverages score-based diffusion models to approximate filtering distributions efficiently, avoiding the computational expense of neural network-b…

  8. Très proche Cite cet article
    AECSF: Adaptive Ensemble Conditional Score Filtering for High-Dimensional Nonlinear Data Assimilation
    Yang-Wen Zhang, Shi-Wei Ni, Xiao-Ping Zhang et al. · 2026 · Pas encore résumé
    Résumé scientifique

    Bayesian state estimation for high-dimensional nonlinear dynamical systems entails a fundamental tension between statistical fidelity and computational tractability, as particle weights can collapse, while Gaussian ensemble updates can miss non-Gaussian posterior structure. Score-based diffusion filters offer a sampling-based alternative, but existing training-free score filters often rely on heuristic likelihood corrections, which can compromise posterior accuracy by neglecting uncertainty about the system state associated with each noisy reverse particle. To address these issues, we propose AECSF, a training-free adaptive ensemble conditional score filter. AECSF constructs an analytically …

Les travaux antérieurs sur lesquels il s’appuie

  1. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise
    A Multiscale Local Gain Form Ensemble Transform Kalman Filter (MLGETKF)
    Xu-Guang Wang, Hristo G. Chipilski, C. Bishop et al. · 2021 · Monthly Weather Review · cité 38 fois · Pas encore résumé
    Résumé scientifique

    A new multiscale, ensemble-based data assimilation (DA) method, multiscale local gain form ensemble transform Kalman filter (MLGETKF), is introduced. MLGETKF allows simultaneous update of multiple scales for both the ensemble mean and perturbations through assimilating all observations at once. MLGETKF performs DA in independent local volumes, which lends the algorithm a high degree of computational scalability. The multiscale analysis is enabled through the rapid creation of many pseudoensemble perturbations via a multiscale ensemble modulation procedure. The Kalman gain that is used to update the raw background ensemble mean and perturbations is based on this modulated ensemble, which intr…

  2. Proche Une méthode qu’il utilise
    Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter
    B. Hunt, E. Kostelich, I. Szunyogh · 2005 · Physica D: Nonlinear Phenomena · cité 1 696 fois · Pas encore résumé
    Résumé scientifique

    Abstract Data assimilation is an iterative approach to the problem of estimating the state of a dynamical system using both current and past observations of the system together with a model for the system’s time evolution. Rather than solving the problem from scratch each time new observations become available, one uses the model to “forecast” the current state, using a prior state estimate (which incorporates information from past data) as the initial condition, then uses current data to correct the prior forecast to a current state estimate. This Bayesian approach is most effective when the uncertainty in both the observations and in the state estimate, as it evolves over time, are accurat…

  3. Proche
    A Unified Filter Method for Jointly Estimating State and Parameters of Stochastic Dynamical Systems via the Ensemble Score Filter
    Feng Bao, Guan-Nan Zhang, Ze-Zhong Zhang · 2023 · cité 5 fois · Pas encore résumé
    Résumé scientifique

    This paper tackles the intricate task of jointly estimating state and parameters in data assimilation for stochastic dynamical systems that are affected by noise and observed only partially. While the concept of ``optimal filtering'' serves as the customary approach to estimate the state of the target dynamical system, traditional methods such as Kalman filters and particle filters encounter significant challenges when dealing with high-dimensional and nonlinear problems. When we also consider the scenario where the model parameters are unknown, the problem transforms into a joint state-parameter estimation endeavor. Presently, the leading-edge technique known as the Augmented Ensemble Kalma…

  4. Proche
    A Quantile-Conserving Ensemble Filter Framework. Part II: Regression of Observation Increments in a Probit and Probability Integral Transformed Space
    Jeffrey L. Anderson · 2023 · Monthly Weather Review · cité 17 fois · Pas encore résumé
    Résumé scientifique

    Traditional ensemble Kalman filter data assimilation methods make implicit assumptions of Gaussianity and linearity that are strongly violated by many important Earth system applications. For instance, bounded quantities like the amount of a tracer and sea ice fractional coverage cannot be accurately represented by a Gaussian which is unbounded by definition. Nonlinear relations between observations and model state variables abound. Examples include the relation between a remotely sensed radiance and the column of atmospheric temperatures, or the relation between cloud amount and water vapor quantity. Part 1 of this paper described a very general data assimilation framework for computing obs…

  5. Proche* Une méthode qu’il utilise
    Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics
    G. Evensen · 1994 · Journal of Geophysical Research · cité 5 444 fois · Pas encore résumé

Les articles qui le citent

  1. Proche
    The Ensemble Schr{ö}dinger Bridge filter for Nonlinear Data Assimilation
    Feng Bao, Hui Sun · 2025 · Foundations of Data Science · cité 3 fois · Pas encore résumé
    Résumé scientifique

    This work introduces a novel nonlinear optimal filtering method, termed the Ensemble Schr{\"o}dinger Bridge nonlinear filter. The proposed filter combines the standard prediction step with a diffusion-generative-modeling-based analysis step, thereby completing one full filtering update. The resulting approach introduces no structural model error, and is derivative-free, training-free, and highly parallelizable. Numerical experiments demonstrate that the proposed algorithm performs effectively for highly nonlinear dynamics and nonlinear observation processes, including chaotic systems with dimension up to 40 and beyond. The results also show that the method outperforms classical approaches su…

  2. Proche
    EnJoi: Ensemble Joint Score Filter for Generative Data Assimilation
    Julien Moreau, Marc Lelarge · 2026 · Pas encore résumé
    Résumé scientifique

    Data Assimilation (DA) aims to recover the full state of a dynamical system that is only partially observed. A solution is to use Score-based models to generate physically consistent trajectories that agree with the observations. These Autoregressive Diffusion models are trained by conditioning on the previous state; however, they do not take into account the uncertainty of their past predictions. We propose a new diffusion-based assimilation algorithm that dynamically balances the confidence in the current state and the new observations. Crucially, we choose to learn the distribution of the joint state containing both the past and future. This allows us to use a modified version of En4DVar,…

  3. Proche Utilise cet article en profondeur
    Two Localization Strategies for Sequential MCMC Data Assimilation With Applications to Nonlinear Non‐Gaussian Geophysical Models
    Hamza M. Ruzayqat, Hristo G. Chipilski, O. Knio · 2026 · Journal of Advances in Modeling Earth Systems · Pas encore résumé
    Résumé scientifique

    We present a localized data assimilation (DA) scheme based on sequential Markov Chain Monte Carlo (SMCMC) (Ruzayqat et al., 2024, https://doi.org/10.1002/qj.4710), a provably convergent method for filtering high‐dimensional, nonlinear, and potentially non‐Gaussian state‐space models (SSMs). Unlike particle filters, which are exact for such models, SMCMC does not assign weights to samples, avoiding weight degeneracy in small‐ensemble regimes. We design two localization approaches within the SMCMC framework that exploit observation sparsity to reduce effective degrees of freedom. The first variant collects observed blocks into a single reduced domain and runs parallel MCMC chains over this com…

  4. Proche
    Guided Unconditional and Conditional Generative Models for Super‐Resolution and Inference of Quasi‐Geostrophic Turbulence
    Anantha Narayanan Suresh Babu, Akhil Sadam, P. F. J. Lermusiaux · 2025 · Journal of Advances in Modeling Earth Systems · cité 6 fois · Pas encore résumé
    Résumé scientifique

    Typically, numerical simulations of Earth systems are coarse, and Earth observations are sparse and gappy. We apply four generative diffusion modeling approaches to super‐resolution and inference of forced two‐dimensional quasi‐geostrophic turbulence on the β $\beta $ ‐plane from coarse, sparse, and gappy observations. Two guided approaches minimally adapt a pre‐trained unconditional model: SDEdit modifies the initial condition, and Diffusion Posterior Sampling (DPS) modifies the reverse diffusion process's score. Two conditional approaches, a vanilla variant and classifier‐free guidance, require training with paired high‐resolution and observation data. We consider multiple test cases spann…

  5. Apparenté Utilise cet article en profondeur
    DAISI: Data Assimilation with Inverse Sampling using Stochastic Interpolants
    Martin Andrae, Erik Larsson, S. Takao et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 6 fois · Pas encore résumé
    Résumé scientifique

    Data assimilation (DA) is a cornerstone of scientific and engineering applications, combining model forecasts with sparse and noisy observations to estimate latent system states. Classical high-dimensional DA methods, such as the ensemble Kalman filter, rely on Gaussian approximations that are violated for complex dynamics or observation operators. To address this limitation, we introduce DAISI, a scalable filtering algorithm built on flow-based generative models that enables flexible probabilistic inference using data-driven priors. The core idea is to use a stationary, pre-trained generative prior that first incorporates forecast information through a novel inverse-sampling step, before as…

Comment ces articles sont-ils trouvés ?

Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.

Mis à jour le 3 novembre 2026.

Le guide illustré des articles autour d’un article