Key Advances Needed for Data Assimilation in the Earth's Upper Atmosphere

What is it about?

Improving Our Ability to Predict and Understand Earth's Upper Atmosphere This publication, focused on Data Assimilation (DA) systems for the Ionosphere, Thermosphere, and Mesosphere (ITM), addresses the critical need to improve how we observe and forecast the physical conditions of Earth’s upper atmosphere. The ITM region (roughly 90 km to 500 km above the Earth’s surface) is crucial because changes there directly affect technologies like satellites and communication systems, making its accurate prediction vital for space weather forecasting. What is Data Assimilation (DA)? Simply put, Data Assimilation is a sophisticated method used to create the best possible picture of the atmosphere at any given moment. It works by optimally blending two things: 1. Numerical Model Estimates: Predictions generated by computer models (the "background estimate"). 2. Observations: Real-world measurements and data collected from instruments. This combination removes biases, uses real observations to constrain the computer models, and provides excellent starting points (initial conditions) for short- and medium-range forecasts. This technique has been fundamental to greatly improving lower atmosphere numerical weather prediction (NWP). The Challenge in the Upper Atmosphere (ITM) While significant progress has been made in developing ITM DA systems over the last decade, particularly using physics-based models, the techniques are still in their early stages compared to those used for standard terrestrial weather forecasting. We cannot just copy techniques from the lower atmosphere because the ITM is fundamentally different: 1. External Drivers: The ITM is partly driven by external factors like solar and geomagnetic activity. 2. Unique Physics: It has unique chemistry, dynamics, and physics. 3. Data Gaps: The mesosphere and thermosphere are still relatively poorly observed, meaning we have to be extremely clever about how we use the limited observations available. Key Challenges We Need to Overcome To fully realise the potential of ITM data assimilation, the publication outlines several outstanding challenges that must be tackled in the coming decade: * Handling Waves: The ITM’s variability is strongly driven by waves (e.g., gravity waves). Standard filtering techniques used in lower atmosphere weather forecasting can either remove these important physical waves or accidentally introduce false (artificial) waves that grow large higher up in the atmosphere. We need methods to minimize artificial waves while preserving the real ones. * Maximising Sparse Data: We need better methods to make the best use of the limited observations in the thermosphere and mesosphere, such as by directly assimilating raw radiance observations instead of retrieved parameters. * Developing New Techniques: Current ITM systems mainly use one type of method (the ensemble Kalman filter). It is critical to adopt more advanced, state-of-the-art techniques (like Variational and Hybrid approaches) used successfully in NWP to ensure continued advancement and enable the use of high-resolution models. * Improving Uncertainty Representation: When forecasting, it is vital that the uncertainty represented in the model (the 'ensemble spread') truly reflects the actual forecast uncertainty. Current methods often rely on ad-hoc forcing perturbations, which do not accurately represent the model's true uncertainty. * Standardising Data: Observations are currently sourced from various places, in different file formats, and often lack crucial information about their uncertainties. Developing centralised databases with standard formats and uncertainty information would greatly improve data usage. Scientific and Operational Benefits Advancing ITM data assimilation capabilities will provide major benefits for both the scientific and operational communities: * Better Space Weather Forecasts: Improvements in DA algorithms are highly likely to enhance the skill of space weather forecasts, which depend on an accurate estimate of the current state of the ITM. * Understanding Predictability: DA enables initialized forecast experiments to investigate the ITM’s predictability—something currently poorly understood (how and why it varies with season, solar activity, etc.). * Creating a Global Picture: It will enable the community to create a reliable ITM reanalysis product. This product, which provides the best available, globally consistent estimate of the upper atmosphere's state, would enable significant scientific advances across a broad range of ITM science. * Guiding Model Development: Data assimilation is a powerful tool for model development, helping identify where computer models have large errors compared to observations, thereby guiding efforts to improve model physics. To achieve this potential, the authors strongly recommend prioritising the development of fundamental ITM DA techniques, fostering collaboration between the ITM and NWP communities, and establishing quantifiable metrics to track improvements in forecasts. They emphasize that ITM data assimilation systems must be recognized as a critical component for performing ITM research in the coming decade.

Featured Image

Photo by Kanhaiya Sharma on Unsplash

Photo by Kanhaiya Sharma on Unsplash

Why is it important?

This white paper outlines the outstanding challenges facing ITM DA that need to be addressed in the coming decade, and details how these will benefit the scientific and operational communities. This is a vital area of research because accurate knowledge of the ITM is fundamental for safeguarding essential technologies. Changes in this upper atmospheric region directly affect satellites, navigation, and communication systems, making accurate prediction—or space weather forecasting—a critical operational need. What makes this work particularly unique and timely is that while DA has been foundational for drastically improving standard terrestrial weather forecasts, the DA systems developed for the ITM are still "in their infancy". Unlike the lower atmosphere, the ITM is driven by external solar and geomagnetic activity, requiring entirely customized techniques that cannot simply copy those used in lower atmosphere NWP. The difference these critical advances will make is significant: 1. Improved Forecasting: They are highly likely to enhance the skill of space weather forecasts, providing better initial conditions for prediction models. 2. Scientific Understanding: They will enable scientists to investigate the ITM’s predictability—how and why it varies—which is currently poorly understood. 3. Global Picture: They will allow the creation of a reliable, globally consistent ITM “reanalysis product.” This product does not currently exist but would enable major scientific breakthroughs across a broad range of upper atmospheric science. 4. Model Improvement: DA will act as a powerful tool to guide model development, helping identify exactly where the computer models need physics improvements.

Perspectives