Due to the chaotic nature of the atmosphere, weather forecasts, even with ever improving numerical weather prediction models, eventually lose all skill.
Meteorologists have a strong desire to better understand this process as they try to trace forecast error back to observational gaps and to provide a means for improvement.
Root mean square error (rms, or its square, the variance distance) is often used to measure differences between simulated and observed fields. In this case, scientists measured the distance between a model forecast field within its grid and the verifying analysis field that represents all real-world observations.
However, one must consider that atmospheric features, like fronts and pressure systems are three dimensional weather features in space that computer models displace and also structurally distort as the numerical forecast moves away from initiation. Variance or rms error metrics do not quantify the displacement and distortion of weather systems.
In a recently published paper in Advances in Atmospheric Sciences, a team of scientists with the National Oceanic and Atmospheric Administration (NOAA), the Massachusetts Institute of Technology (MIT), and the University of Connecticut set out to find a general approach to assess the positional and structural components of the total difference between two fields.
Essentially, meteorologists want to assess the accuracy of many different weather features within a model forecast compared to a verifying analysis based on real-world observations.
Sai Ravela from MIT, a co-author of this study, previously developed a Field Alignment method. In this case, his approach aligns the model forecast field with the observationally based analysis in a smooth fashion so their difference is minimized (Step 1 in schematic diagram, see also example map).
Next, small scale errors from uncertain origins are removed from all three fields (the original and aligned forecast as well as the verifying analysis, or proxy for observations) through a process called spatial filtering or smoothing (Step 2).
The total variance distance, or difference, is then partitioned into three unique components (Step 3).
Positional error, which is the variance distance between the smoothed original model forecast and smoothed aligned forecast fields, and structural error that is the variance distance between the smoothed aligned forecast and the smoothed verifying analysis fields, are two sides of the right angle triangle in Fig. 1, and fine scale noise, which are the uncertain small-scale errors removed from the original model forecast and verifying analysis, or observation fields (see smoothing arrows orthogonal to the triangle in Fig. 1).
This method outputs the three orthogonal error components as scalar fields, as well as a vector field (Fig. 2) indicating the large-scale displacement of the forecast compared to the observational analysis field.
Interestingly, throughout all regions and lead times that the team studied, more than half of the total error variance is associated with the misplacement of weather features.
Therefore, displacement is more pronounced than distortion in forecast fields: only about 25% error variance is associated with structural inaccuracies of the partially predictable features, such as fronts and low pressure systems. The rest of the error variance remains unexplained or unpredictable variability, or noise.
“How noise grows in error variance as a function of forecast lead time, and whether a positional-structural-noise decomposition of the spread among an ensemble of perturbed forecasts captures forecast error components is the subject of ongoing studies,” said Dr. Jankov from NOAA, the lead author of the study.