|
The OSSE checklist
(continued...)
a supplement to
Future Observing System Simulation Experiments
by Ross N. Hoffman and Robert Atlas
The Observing System Simulation Experiment or OSSE checklist presented here is meant to be a detailed guide to OSSE practitioners and OSSE consumers to evaluate the design of an OSSE system and OSSE experiments. It provides a step-by-step process to encourage the user to consider all aspects of good OSSE design principles. The current version of the OSSE checklist is directly applicable to weather-related observing, modeling and data assimilation (DA) systems. But very similar issues and concerns apply to OSSEs that would be used across a diverse range of geophysical domains and the user of the OSSE checklist should be able to translate the discussion presented here to these domains fairly directly.
The OSSE checklist based on a recent AMS publication titled "Future Observing System Simulation Experiments."
|
Page 1 |
|
Overall
Your Experiment
This section is intended to record some minimal basic information about your OSSE.
Answers here provide context for the questions that follow.
-
What new instrument(s) and/or new technique(s) are you investigating?
-
What phenomena are of interest for the new instrument(s) and/or new technique(s)?
-
Describe your nature run (NR):
-
Describe your forecast and data assimilation (DA) systems:
-
Describe your experimental design (including your validation approach):
|
Page 1 |
|
|
Page 2 |
|
Nature Run
NR quality
-
Is the NR sufficiently different from the forecast model?
Yes
No
- Discussion: If the NR and forecast model are too similar there may be an identical twin problem, in which the DA system has too easy a time converging on the truth.
- Caution: If not, even with a small amount of data the analyses may be too good, leaving little room for improvement.
-
Is the NR sufficiently high resolution?
Yes
No
- Discussion: The NR should well resolve phenomena of interest. Scales smaller than the observation footprint and/or the model resolution contribute to representativeness error.
- Caution: If not, the phenomena of interest may not be realistic or even present in the NR. The observation simulation may be overly dependent on statistical parameterizations of the representativeness error.
-
Is the NR sufficiently long?
Yes
No
- Discussion: One year or more past spin up is desirable. A large ensemble of forecast cases provides greater statistical significance. Ideally, the OSSE will sample all relevant seasons.
- Caution: If not, sample size may be small. Results may be representative of only a single season.
-
Is the NR generated by an operationally vetted model?
Yes
No
- Discussion: A great deal of effort is devoted to removing biases from operational models. For example, cloud amounts, locations, heights and type should all be realistic, especially for simulating infrared (IR) satellite observations.
- Caution: If not, the climatology of the NR may be different from reality.
-
Is the NR a short- or medium-range forecast (or a collection of such forecasts)?
Yes
No
- Discussion: This is a Quick OSSE, not a full OSSE. Quick OSSEs allow comparison to real and interesting forecast cases.
- Caution: Generally, a single forecast is used as the NR in a Quick OSSE. This does not allow for actionable, statistically significant findings.
-
Is the NR actually a reanalysis?
Yes
No
- Discussion: As with the model identical twin problem, a DA system can quickly converge to a reanalysis. Similarities are too great. New observing systems are handicapped, because in current data voids, the NR is basically a short term forecast and therefore any simulated observations in these areas will not add new information.
- DANGER: If a reanalysis is used as the NR, then this is not really an OSSE; such an experiment cannot be used to evaluate the impact of new observing systems quantitatively.
NR completeness
-
Does the NR database save everything needed at sufficient temporal and spatial resolution?
Yes
No
- Discussion: To accurately simulate observations the highest possible temporal and spatial resolution is required.
- Caution: If not, the simulated observations may not have realistic variability and representativeness errors.
-
Are any auxiliary data needed consistent with the NR?
Yes
No
- Discussion: Examples might be aerosols, from land surfaces and volcanoes, sea ice extent, soil moisture, vegetative health, etc. These should be consistent with the NR. For example, higher aerosol concentrations are expected downwind of source areas (such as the Sahara, Aral Sea, etc.).
- Caution: If not, simulated observations may have inconsistent error characteristics.
-
Is there a capability to convert the NR data to initial conditions (IC) for the forecast model?
Yes
No
- Discussion: Forecast model IC corresponding to the state of the NR are useful in some tests of the OSSE system.
- Caution: If not, assimilation of perfect observations may be needed to generate NR IC for the forecast model.
NR starting time
NR validation
-
Has the NR been properly validated?
Yes
No
- Discussion: It is critical to thoroughly validate the NR.
- Caution: An un-validated NR may result in wasted effort and incorrect conclusions.
-
Are the NR and real climate statistics similar? Are both annual and seasonal or ideally monthly statistics similar.
Yes
No
- Discussion: To obtain similar data coverage and error characteristics, it is necessary to sample a climatology similar to reality. For example, realistic precipitation patterns are needed to obtain realistic patterns of data coverage and errors for microwave sensors. Comparisons should include at least latitude-height cross sections of the zonally and temporally averaged prognostic variables and maps of sea level pressure, integrated water vapor, 250 hPa winds, and 500 hPa heights.
- Caution: If not, some parameters may need to be adjusted when simulating observations to insure realistic patterns of data coverage and errors.
-
Are the NR budgets balanced?
Yes
No
- Discussion: If sea surface temperature (SST) is fixed for example, the heat budget might not balance.
- Caution: Unbalanced budgets may contribute to an unrealistic NR.
-
Are the NR energetics (e.g., eddy kinetic energy or EKE) on different space and time scales for different regions and seasons similar to reality?
Yes
No
- Discussion: EKE is a proxy for variance of the atmospheric variables. If some scales have too little (much) variance in the OSSE, the analysis of these scales in the OSSE may be too good (poor).
- Caution: If not, some phenomena of interest may not be well described in the NR or may not have realistic frequencies of occurrence.
-
Is the NR representative of a typical period in reality?
Yes
No
- Discussion: Consider an ensemble of a metric of interest from the past several years (or decades). Repeatedly replace one year selected randomly with a year from the NR. Is the NR an outlier? Does adding the NR noticeably alter the statistics of the ensemble?
For example, it would not be acceptable if all the cyclone tracks avoid North America. Track statistics should be compared by region and month to the real case. The statistics should include number of cyclones, number of cases of genesis and of cyclolysis, and average central pressure, speed, and direction.
- Caution: The NR does not have to be identical in all statistical respects to reality, but if it cannot pass as a random draw from reality, the OSSE results may not be reliable.
-
Are the phenomena of interest realistically depicted in the NR?
Yes
No
- Discussion: For example, if storm (extratropical or tropical) forecasts are of interest, then storm track and intensity statistics should be similar to reality. Tracks in the OSSE should look reasonable compared to reality. For example, it would not be acceptable if all the cyclone tracks avoid North America. Track statistics should be compared by region and month to the real case. The statistics should include number of cyclones, number of cases of genesis and of cyclolysis, and average central pressure, speed, and direction.
- Caution: If the statistics of a phenomena of interest are skewed in the NR, then quantitative results from the OSSE will likely be skewed as well.
-
Are the key variables needed to simulate the observations of interest realistically depicted in the NR?
Yes
No
- Discussion: These key variables will typically include cloud amount, height, and type as well as sea level pressure, precipitation, and surface winds among others. For these variables plots of monthly and weekly averages, and instantaneous fields should be examined.
- Caution: If the distribution of variables needed to simulate observations are different from reality, then the coverage and errors of the simulated observations will also be different from reality, unless additional measures are taken.
-
Are the NR caveats documented?
Yes
No
- Discussion: No NR is perfect. It is important to document any limitations that might affect the usefulness of the NR for different purposes and to not draw conclusions that go beyond those limitations.
- Caution: Make certain that deficiencies in the NR do not eliminate the NR as applicable for the purposes of the OSSE.
|
Page 2 |
|
|
Page 3 |
|
Forecast model
Forecast model quality
-
Is the forecast model a current operational model?
Yes
No
- Discussion: To draw conclusions from the OSSE applicable to an operational model (or future planned operational model), then that model should be used in the OSSE system.
- Caution: If not, results may not be applicable to other forecast and DA systems using other models.
-
Is the forecast model sufficiently high resolution?
Yes
No
- Discussion: In general, the forecast model resolutions should match the model resolution used in the system for which the OSSE results will be applied. In some situations, a lower resolution version of an operational model may be adequate. A lower resolution version is appropriate for testing and preliminary results and can be useful to narrow the list of interesting experiments that will then be conducted at full resolution.
- Caution: If not, results may not be applicable at higher resolution.
-
Are the phenomena of interest realistically depicted, maintained, and forecast by the model?
Yes
No
- Discussion: There cannot be meaningful forecast impacts for phenomena of interest that are not realistically depicted, maintained, and forecast by the model.
- Caution: If not, useful results cannot be obtained in this case.
Forecast model validation
|
Page 3 |
|
|
Page 4 |
|
Data assimilation (DA) system
DA system quality
DA system completeness
DA system validation
|
Page 4 |
|
|
Page 5 |
|
Observation simulation
Observation coverage
Observation operators
Observation errors
-
Are the radiative transfer (RT) forward models used to generate perfect observations different from those used in the DA system?
Yes
No
- Discussion: Different RT models could be used for simulation and assimilation, or realistic uncertainty could be included by creating an alternative set of coefficients used in the fast RT forward model for simulating observations in the OSSE while retaining the operational coefficients in the DA system. Aspects of the RT calculation used to generate the coefficients that might be perturbed include the line parameters, the treatment of physical processes, the spectral weighting function that corresponds to the sensor response, and the spectral resolution. Or the operational coefficients might simply be perturbed randomly.
- Caution: Identical RT models eliminate an important error source that has the potential to produce biases in reality.
-
Are atmospheric motion vector (AMV) and cloud track wind (CTW) observation locations and height assignment errors realistic?
Yes
No
- Discussion: Two possible approaches are (1) to statistically estimate where trackable features should be found and the size of the height assignment errors and then uses these to simulate AMVs and (2) to simulate imagery and then emulate the entire process of identifying features in the imagery, tracking these, and assigning heights-all following operational practice. Resolutions of 15 minutes and 2-5 km are used operationally and similar resolutions would be required in the NR to generate AMVs and CTWs from simulated imagery in an OSSE.
- Caution: Current operational systems struggle with the complex error characteristics of AMVs and CTWs. If these types of errors are not realistic in the OSSE, then these observations may be "too good".
-
Are the equation of state and gravity model used to simulate GNSS/RO observations different from those used in the DA system?
Yes
No
- Discussion: GNSS/RO observations as a function of altitude may be considered very precise observations traceable to absolute standards. However, the forward model to determine model variables at the observation altitude depends on the determination of altitude hydrostatically through the equation of state and gravity model. As with the discussion of the RT models above, there should be some differences between the observation simulation and the forward model used in the DA system.
- Caution: Identical equations of state and gravity models eliminate an important error source that has the potential to produce biases in reality.
-
Are there appropriate differences between the surface characteristics used to simulate data and used to assimilate those data?
Yes
No
- Discussion: This includes temporal changes of soil moisture, sea ice, and vegetation. As with the discussion of the RT models above, there should be some differences between the observation simulation and the forward model used in the DA system.
- Caution: Identical surface specifications eliminate an important error that has the potential to produce biases in reality.
-
Are added observational errors realistic?
Yes
No
- Discussion: Explicitly added observational errors should have the right characteristics including biases, error-error correlations (vertical for RAOBS, horizontal for CTWs, channel for radiances) and error-geophysical parameter correlations. These geophysical parameters include synoptic situation and land surface properties. For example, errors may be larger in cloudy, precipitating, and/or stormy conditions. The amplitude of error variances and error length scales may also depend on such geophysical parameters.
- Caution: Using the correct added observational errors is critical to a well-calibrated OSSE system and obtaining reliable actionable quantitative results.
|
Page 5 |
|
|
Page 6 |
|
Evaluation strategy
Quantification and interpretation of impacts and results
|
Page 6 |
|
| |