Generating observation guided ensembles for data assimilation with denoising diffusion probabilistic model
This paper presents an ensemble data assimilation method using the pseudo ensembles generated by denoising diffusion probabilistic model.
Tags：Paper and LLMsDenoisingPricing Type
- Pricing Type: Free
- Price Range Start($):
GitHub Link
The GitHub link is https://github.com/yasahi-hpc/generative-enkf
Introduce
The “Generative Ensemble Kalman Filter (Generative EnKF)” is a method designed to improve data assimilation (DA) for numerical simulations. It addresses issues with Ensemble Kalman Filter (EnKF) related to model biases and simulation costs. The approach utilizes pseudo ensembles generated by an observation-guided denoising diffusion probabilistic model (DDPM), which shows better performance than EnKF when dealing with biased simulation models. The method involves training a diffusion model guided by observations and performing simulation experiments with pre-trained models. The code relies on PyTorch, requires packages like numpy, xarray, and netcdf4, and provides scripts for simulation, training, inference, and data conversion. The research is presented in the ICML 2023 Workshop, showcasing its application in data assimilation and deep learning for numerical simulations.
This paper presents an ensemble data assimilation method using the pseudo ensembles generated by denoising diffusion probabilistic model.
Content
Generative EnKF is designed to surrogate data assimilation (DA) for numerical simulations without ensemble runs. Forecasting a real world system is often achieved by combining a scientific model for the time evolution of the system and an estimate of the current state of the system. Basically, DA like Ensemble Kalman Filter (EnKF) blends the simulation states and observation data iteratively to give a reasonable estimate of the current state. Though widely used, EnKF has two critical issues: fragility to model biases (errors) and the ensemble simulation costs. Here, we propose a DA method using the pseudo ensembles generated by observation guided denoising diffusion probabilistic model (DDPM). Thanks to the variance in generated ensembles, our proposed method displays better performance than the well-established ensemble DA method (say, EnKF) when the simulation model is biased. For questions or comments, please find us in the AUTHORS file. This code relies on the following packages. As a deeplearing framework, we use PyTorch. The results would be stored in netcdf files under <base_dir/case_name>. Following table summarizes the major scripts and their inputs.
Related
