Quantitative Observing System Assessment Program
Smart Instrument Integration for Observing Systems
SCROLL TO LEARN MORE
Optimizing Observing Systems
The QOSAP program evaluates both new and proposed observing systems by conducting experiments to determine the impact of observational data on models (existing or proposed). The QOSAP program has two primary evaluations: Observing System Experiments (OSEs) and Observing System Simulation Experiments (OSSEs). Together, these experiments can help managers make decisions about the effectiveness, impact, and viability of new and proposed observing systems.
Increasing the Length of a Reliable Forecast
The QOSAP program conducted comprehensive experiments with a previously existing global OSSE system to determine the potential value of Radio Occultation constellations for current operational numerical weather prediction.
Radio Occultation (COSMIC-2) observations increased the length of the reliable forecast 28.4% in the tropics with further improvements in both Northern and Southern hemisphere extra-tropics.
Increasing Visibility
The QOSAP program performed experiments to determine the benefits, trade offs, sampling strategies, and ideal configuration for Doppler Wind Lidar instrumentation aboard NOAA’s Hurricane Hunter Aircraft for the purpose of improving Hurricane Forecasts.
A comparison of the assimilated observations (in the Hurricane Weather Research and Forecasting Model) revealed that the Doppler Wind Lidar provided better coverage of the wind field, more accurate wind field analysis (increased speed up to 8 kts in the Boundary Layer, and observational capacity measuring 10x closer to the surface than currently installed instrumentation (an improvement from 250-500 meters to 25 meters above the surface).
Supporting the Weather Research & Forecasting Act of 2017
The Weather Research and Forecasting Innovation Act of 2017 specifically mandates NOAA to perform OSSEs to quantitatively assess the relative value and benefits of observing capabilities and systems.
- AOML conducts Observing System Simulation Experiments (OSSE) to assess the impact of large and expensive observing systems prior to deployment.
- AOML completed OSSE experiments required by the Weather Research and Forecasting Innovation Act of 2017 on Radial Occultation Satellites.
- AOML will conduct OSSEs for all proposed satellite systems in NOAA and optimize the use of NOAA’s reconnaissance aircraft.
The Weather Law H.R. 353 mandates NOAA to perform OSSEs to quantitatively assess the relative value and benefits of observing capabilities and systems. In particular, OSSEs shall be conducted prior to the acquisition of major Government-owned or Government-leased operational observing systems which a lifecycle cost of more than $500,000,000, and prior to the purchase of any major new commercially provided data with a lifecycle cost of more than $500,000,000.
About the QOSAP Program
The QOSAP improves existing atmospheric, oceanic, and coupled models by performing simulation experiments to evaluate the tradeoffs and impacts of different observation types across NOAA Line Offices. These studies aid NOAA management by determining accuracy, cost-efficiency, and viability of observations within various model designs.
Objectives
Improve quantitative and objective assessment capabilities to evaluate operational and future observation system impacts and trade-offs to assess and to prioritize NOAA’s observing system architecture.
- Increase NOAA’s capacity to conduct quantitative observing system assessments.
- Develop and use appropriate quantitative assessment methodologies.
- Inform major decisions on the design and implementation of optimal composite observing systems.
Economic Viability
Observing systems are more cost effective and allow for strategic evolution of emerging technology.
Quantitative Improvements
Effectiveness of new instrumentation is assessed. Integration response is predicted for various instrument configurations, and impacts of data assimilation are diagnosed.
Time Efficiency
Time lags between instrument deployment and operational integration are reduced with OSSE integration.
Key Differences Between OSSEs and OSEs
Which experiment is right for my project? When to use an OSE, an OSSE, or both. Rollover and Click the button on the image for more.
OSE
Observing System Experiments (OSEs) evaluate the impact of real data from deployed instruments on a particular model. OSEs can help scientists determine the most effective way to use instrumentation for a model.

OSSE
Observing System Simulation Experiments (OSSEs) assess the impact of a potential observing system by testing a potential dataset from the proposed instrumentation on a simulated natural environment (called a Nature Run).

Observing System Simulation Experiments (OSSEs)
Fiscal Year 2018 Accomplishments
- Impact of GEOstationary hyperspectral sounders (GeoHSS) on numerical weather prediction
- Radio Occultation observations (COSMIC-2, refractivity and bending angle retrievals)
- Global and regional impacts of Cubesats
- Ocean OSSEs related to the role of ocean observations in hurricane prediction
The OSSE Checklist
The Observing System Simulation Experiment or OSSE checklist presented here is meant to be a detailed guide to OSSE practitioners and OSSE consumers to evaluate the design of an OSSE system and OSSE experiments. It provides a step-by-step process to encourage the user to consider all aspects of good OSSE design principles. The current version of the OSSE checklist is directly applicable to weather-related observing, modeling and data assimilation (DA) systems. But very similar issues and concerns apply to OSSEs that would be used across a diverse range of geophysical domains and the user of the OSSE checklist should be able to translate the discussion presented here to these domains fairly directly.
These are just a few of the questions sampled from the comprehensive checklist for OSSEs. To visit the checklist and take the quiz, click the button below.
Featured Publication
Click to See OSSE Related Refereed Publications
2013
Hoffman, R.N., J.V. Ardizzone, S.M. Leidner, D.K. Smith, and R.M. Atlas. Error estimates for ocean surface winds: Applying Desroziers diagnostics to the cross-calibrated, multiplatform analysis of wind speed. Journal of Oceanic and Atmospheric Technology, 30(11):2596-2603 (doi:10.1175/JTECH-D-13-00018.1) (2013).Nolan, D.S., R. Atlas, K.T. Bhatia, and L.R. Bucci. Development and validation of a hurricane nature run using the Joint OSSE nature run and the WRF model. Journal of Advances in Modeling Earth Systems, 5(2):382-405 (doi:10.1002/jame.20031) (2013).
Privé, N.C., Y. Xie, J.S. Woollen, S.E. Koch, R. Atlas, and R.E. Hood. Evaluation of the Earth Systems Research Laboratory’s global Observing System Simulation Experiment (OSSE) system. Tellus A, 65:19011 (doi:10.3402/ tellusa.v65i0.19011), 22 pp. (2013).
Ralph, F.M., J. Intrieri, D. Andra, R. ATLAS, S. Boukabara, D. Bright, P. Davidson, B. Entwistle, J. Gaynor, S. Goodman, J.-G. Jiing, A. Harless, J. Huang, G. Jedlovec, J. Kain, S. Koch, B. Kuo, J. Levit, S. MURILLO, L.P. Riishojgaard, T. Schneider, R. Schneider, T. Smith, and S. Weiss. The emergence of weather-related testbeds linking research and forecasting operations. Bulletin of the American Meteorological Society, 94(8):1187-1211 (doi:10.1175/BAMS-D-12-00080) (2013).
2014
Baker, W.E., R. Atlas, C. Cardinali, A. Clement, G.D. Emmitt, B.M. Gentry, R.M. Hardesty, E. Kallen, M.J. Kavaya, R. Langland, Z. Ma, M. Masutani, W. McCarty, R.B. Pierce, Z. Pu, L.P. Riishojgaard, J. Ryan, S. Tucker, M. Weissmann, and J.G. Yoe. Lidar-measured wind profiles: The missing link in the global observing system. Bulletin of the American Meteorological Society, 95(4):543-564 (doi:10.1175/BAMS-D-12-00164.1) (2014).Halliwell, G.R., A. Srinivasan, V. Kourafalou, H. Yang, D. Willey, M. Le Henaff, and R. Atlas. Rigorous evaluation of a fraternal twin ocean OSSE system for the open Gulf of Mexico. Journal of Oceanic and Atmospheric Technology, 31(1):105-130 (doi:10.1175/JTECH-D-13-00011.1) (2014).
Nolan, D.S., J.A. Zhang, and E.W. Uhlhorn. On the limits of estimating the maximum wind speeds in hurricanes. Monthly Weather Review, 142(8):2814-2837 (doi:10.1175/MWR-D-13-00337.1) (2014). [first sentence of abstract states: ‟This study uses an observing system simulation experiment (OSSE) approach to test the limitations…”
Privé, N.C., Y. Xie, S. Koch, R. Atlas, S.J. Majumdar, and R.N. Hoffman. An observing system simulation experiment for the unmanned aircraft system data impact on tropical cyclone track forecasts. Monthly Weather Review, 142(11):4357-4363 (doi:10.1175/MWR-D-14-00197.1) (2014).
2015
Atlas, R., V. Tallapragada, and S. Gopalakrishnan. Advances in tropical cyclone intensity forecasts. Marine Technology Society Journal, 49(6):149-160 (doi:10.4031/MTSJ.49.6.2) (2015).Atlas, R., L. Bucci, B. Annane, R. Hoffman, and S. Murillo. Observing System Simulation Experiments to assess the potential impact of new observing systems on hurricane forecasting. Marine Technology Society Journal, 49(6):140-148 (doi:10.4031/MTSJ.49.6.3) (2015).
Atlas, R., R.N. Hoffman, Z. Ma, G.D. Emmitt, S.A. Wood, S. Greco, S. Tucker, L. Bucci, B. Annane, R.M. Hardesty, and S. Murillo. Observing system simulation experiments (OSSEs) to evaluate the potential impact of an optical autocovariance wind lidar (OAWL) on numerical weather prediction. Journal of Atmospheric and Oceanic Technology, 32(9):1593-1613 (doi:10.1175/JTECH-D-15-0038.1) (2015).
Cucurull, L., and R.A. Anthes. Impact of loss of U.S. microwave and radio occultation observations in operational numerical weather prediction in support of the U.S. data gap mitigation activities. Weather and Forecasting, 30(2):255-269 (doi:10.1175/WAF-D-14-00077.1) (2015).
Halliwell, G.R., V. Kourafalou, M. Le Henaff, L.K. Shay, and R. Atlas. OSSE impact analysis of airborne ocean surveys for improving upper-ocean dynamical and thermodynamical forecasts in the Gulf of Mexico. Progress in Oceanography, 130:32-46 (doi:10.1016/j.pocean.2014.09.004) (2015).
Oke, P.R., G. Larnicol, E.M. Jones, V. Kourafalou, A.K. Sperrevik, F. Carse, C.A.S. Tanajura, B. Mourre, M. Tonani, G.B. Brassington, M. Le Hénaff, G.R. Halliwell, R. Atlas, A.M. Moore, C.A. Edwards, M.J. Martin, A.A. Sellar, A. Alvarez, P. De Mey, and M. Iskandarani. Assessing the impact of observations on ocean forecasts and reanalyses: Part 2, Regional applications. Journal of Operational Oceanography, 8(S1):s63-s79 (doi:10.1080/ 1755876X.2015.1022080) (2015).
2016
Androulidakis, Y.S., V.H. Kourafalou, G.R. Halliwell, M. Le Henaff, H.S. Kang, M. Mehari, and R. Atlas. Hurricane interaction with the upper ocean in the Amazon-Orinoco plume region. Ocean Dynamics, 66(12):1559-1588 (doi:10.1007/s10236-016-0997-0) (2016).Boukabara, S.A., I. Moradi, R. Atlas, S.P.F. Casey, L. Cucurull, R.N. Hoffman, K. Ide, V.K. Kumar, R. Li, Z. Li, M. Masutani, N. Shahroudi, J. Woollen, and Y. Zhou. Community global Observing System Simulation Experiment (OSSE) package: CGOP—Description and usage. Journal of Atmospheric and Oceanic Technology, 33(8):1759-1777 (doi:10.1175/JTECH-D-16-0012.1) (2016).
Boukabara, S.A., T. Zhu, H.L. Tolman, S. Lord, S. Goodman, R. Atlas, M. Goldberg, T. Auligne, B. Pierce, L. Cucurull, M. Zupanski, M. Zhang, I. Moradi, J. Otkin, D. Santek, B. Hoover, Z. Pu, X. Zhan, C. Hain, E. Kalnay, D. Hotta, S. Nolin, E. Bayler, A. Mehra, S.P.F. Casey, D. Lindsey, L. Grasso, K. Kumar, A. Powell, J. Xu, T. Greenwald, J. Zajic, J. Li, J. Li, B. Li, J. Liu, L. Fang, P. Wang, and T.-C. Chen. S4: An O2R/R2O infrastructure for optimizing satellite data utilization in NOAA numerical modeling systems: A step toward bridging the gap between research and operations. Bulletin of the American Meteorological Society, 97(12):2359-2378 (doi:10.1175/BAMS-D-14-00188.1) (2016).
Hoffman, R.N., and R. Atlas. Future observing system simulation experiments. Bulletin of the American Meteorological Society, 97(9):1601-1616 (doi:10.1175/BAMS-D-15-00200.1) (2016).
Kourafalou, V.H., Y.S. Androulidakis, G.R. Halliwell, H.-S. Kang, M. Mehari, M. Le Henaff, R. Atlas, and R. Lumpkin. North Atlantic Ocean OSSE system development: Nature Run evaluation and application to hurricane interaction with the Gulf Stream. Progress in Oceanography, 148:1-25 (doi:10.1016/ j.pocean.2016.09.001) (2016).
Lee, P., R. Atlas, G. Carmichael, Y. Tang, B. Pierce, A.P. Biazar, L. Pan, H. Kim, D. Tong, and W. Chen. Observing System Simulation Experiments (OSSEs) using a regional air quality application for evaluation. In Air Pollution Modeling and its Application XXIV, D.G. Steyn and N. Chaumerliac (eds.). Springer International Publishing, 599-605 (doi:10.1007/978-3-319-24478-5_97) (2016).
Ruf, C.S., R. Atlas, P.S. Chang, M.P. Clarizia, J.L. Garrison, S. Gleason, S.J. Katzberg, Z. Jelenak, J.T. Johnson, S.J. Majumdar, A. O’Brien, D.J. Posselt, A.J. Ridley, R.J. Rose, and V.U. Zavorotny. New ocean winds satellite mission to probe hurricanes and tropical convection. Bulletin of the American Meteorological Society, 97(3):385-395 (doi:10.1175/BAMS-D-14-00218.1) (2016).
2017
Cucurull, L., R. Li, and T.R. Peevey. Assessment of radio occultation observations from the COSMIC-2 mission with a simplified Observing System Simulation Experiment configuration. Monthly Weather Review, 145(9):3581-3597 (doi:10.1175/MWR-D-16-0475.1) (2017).Halliwell, G.R., M. Mehari, M. Le Henaff, V. Kourafalou, I. Androulidakis, H. Kang, and R. Atlas. North Atlantic Ocean OSSE system: Evaluation of operational ocean observing system components and supplemental seasonal observations for improving coupled tropical cyclone prediction. Journal of Operational Oceanography, 10(2):154-175 (doi:10.1080/1755876X.2017. 1322770) (2017).
Halliwell, G.R., M. Mehari, L.K. Shay, V.H. Kourafalou, H. Kang, H.-S. Kim, J. Dong, and R. Atlas. OSSE quantitative assessment of rapid-response prestorm ocean surveys to improve coupled tropical cyclone prediction. Journal of Geophysical Research-Oceans, 122(7):5729-5748 (doi:10.1002/ 2017JC012760) (2017).
Hoffman, R.N., N. Prive, and M. Bourassa. Comments on ‟Reanalysis and observations: What’s the difference?” Bulletin of the American Meteorological Society, 98(11):2455-2459 (doi:10.1175/BAMS-D-17-0008.1) (2017).
Hoffman, R.N., S.-A. Boukabara, V.K. Kumar, K. Garrett, S.P.F. Casey, and R. Atlas. An empirical cumulative density function approach to defining summary NWP forecast assessment metrics. Monthly Weather Review, 145(4):1427-1435 (doi:10.1175/MWR-D-16-0271.1) (2017).
Leidner, S.M., T. Nehrkorn, J. Henderson, M. Mountain, T. Yunck, and R.N. Hoffman. A severe weather quick observing system simulation experiment (QuickOSSE) of global navigation satellite system (GNSS) radio occultation (RO) superconstellations. Monthly Weather Review, 145(2):637-651 (doi:10.1175/MWR-D-16-0212.1) (2017).
Li, J., Z. Li, P. Wang, T.J. Schmit, W. Bai, and R. Atlas. An efficient radiative transfer model for hyperspectral IR radiance simulation and applications under cloudy-sky conditions. Journal of Geophysical Research-Atmospheres, 122(14):7600-7613 (doi:10.1002/2016JD026273) (2017).
McNoldy, B., Annane, S. Majumdar, J. Delgado L. Bucci, and R. Atlas. Impact of assimilating CYGNSS data on tropical cyclone analyses and forecasts in a regional OSSE framework. Marine Technology Society Journal, 51(1):7-15 (doi:10.4031/MTSJ.51.1.1) (2017).
Pu, Z., L. Zhang, S. Zhang, B. Gentry, D. Emmitt, B. Demoz, and R. Atlas. The impact of Doppler wind lidar measurements on high-impact weather forecasting: Regional OSSE and data assimilation studies. In Data Assimilation for Atmospheric, Oceanic and Hydrological Applications, Volume 3, S.K. Park and L. Xu (eds.). Springer International (doi:10.1007/978-3-319-43415-5) (2017).
Wentz, F.J., L. Ricciardulli, E. Rodriguez, B.W. Stiles, M.A. Bourassa, D.G. Long, R.N. Hoffman, A. Stoffelen, A. Verhoef, L.W. O’Neill, J.T. Farrar, D. Vandemark, A.G. Fore, S.M. Hristova-Veleva, F.J. Turk, R. Gaston, and D. Tyler. Evaluating and extending the ocean wind climate data record. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(5):2165-2185 (doi:10.1109/JSTARS. 2016.2643641) (2017).
Zhang, S., Z. Pu, D.J. Posselt, and R. Atlas. Impact of CYGNSS ocean surface wind speeds on numerical simulations of a hurricane in observing system simulation experiments. Journal of Atmospheric and Oceanic Technology, 34(2):375-383 (doi:10.1175/JTECH-D-16-0144.1) (2017).
2018
Annane, B., B. McNoldy, S.M. Leidner, R. Hoffman, R. Atlas, and S.J. Majumdar. A study of the HWRF analysis and forecast impact of realistically simulated CYGNSS observations assimilated as scalar wind speeds and as VAM wind vectors. Monthly Weather Review (doi:10.1175/MWR-D-17-0240.1), in press.Blackwell, W.J., S. Braun, R. Bennartz, C. Velden, M. DeMaria, R. Atlas, J. Dunion, F. Marks, R. Rogers, B. Annane, and R.V. Leslie. An overview of the TROPICS NASA Earth Venture mission. Quarterly Journal of the Royal Meteorological Society (doi:10.1002/qj.3290), in press.
Boukabara, S.-A., K. Ide, N. Shahroudi, Y. Zhou, T. Zhu, R. Li, L. Cucurull, R. Atlas, S.P.F. Casey, and R.N. Hoffman. Community global Observing System Simulation Experiment (OSSE) package (CGOP): Perfect observations simulation validation. Journal of Atmospheric and Oceanic Technology, 35(1):207-226 (doi:10.1175/JTECH-D-17-00771) (2018).
Boukabara, S.-A., K. Ide, Y. Zhou, N. Shahroudi, R.N. Hoffman, K. Garrett, V. Krishna Kumar, T. Zhu, and R. Atlas. Community global OSSE package (CGOP). Part III: Calibration assessment and validation using an OSSE/OSE intercomparison of summary assessment metrics. Journal of Atmospheric and Oceanic Technology, submitted.
Christophersen, H., R. Atlas, A. Aksoy, and J. Dunion. Combined use of satellite observations and Global Hawk unmanned aircraft dropwindsondes for improved tropical cyclone analyses and forecasts. Journal of Atmospheric and Oceanic Technology, accepted.
Cucurull, L., R. Atlas, R. Li, M.J. Mueller, and R.N. Hoffman. An observing system simulation experiment with a constellation of radio occultation satellites. Monthly Weather Review, submitted.
Haddad, Z., R. Atlas, G.S. Bhat, D. Bouniol, S. Brown, L. Callahan, P. Chambon, T. Fiolleau, K. Furukawa, C. Goldstein, G. Heymsfield, S. Hristova-Veleva, E. Im, R. Kakar, M.C. Kalapureddy, H. Kim, C. Kishtawal, T. L’Ecuyer, L. Li, Z.J. Luo, G. Mace, P. Mukhopadhyay, T.N. Rao, D. Posselt, A. Protat, R. Roca, G. Skofronick-Jackson, R. Storer, O. Sy, P. Tabary, S. Tanelli, W.-K. Tao, F.J. Turk, S. van den Heever, D. Vane, D. Waliser, D. Wu, and G. Stephens. Distributed satellite microwave observation strategies for cloud and precipitation dynamics. Journal of Earth System Science, submitted.
Hoffman, R.N. The effect of thinning and superobservations in a simple one-dimensional data analysis with mischaracterized error. Monthly Weather Review, 146(4):1181-1195 (doi:10.1175/MWR-D-17-0363.1) (2018).
Leidner, S.M., B. Annane, B. McNoldy, R. Hoffman, and R. Atlas. Variational analysis of simulated ocean surface winds from the Cyclone Global Navigation Satellite System (CYGNSS) and evaluation using a regional OSSE. Journal of Atmospheric and Oceanic Technology, accepted.
Li, Z., W.P. Menzel, J. Li, T. Schmit, and R.M. Atlas. What is an infrared cloud top height? Geophysical Research Letters, submitted.
Peevey, T.R., J.M. English, L. CucurulL, H. Wang, and A.C. Kren. Improving winter storm forecasts with Observing System Simulation Experiment (OSSEs). Part 1: An idealized case study of three US storms. Monthly Weather Review, 146(5):1341-1366 (doi:10.1175/MWR-D-17-0160.1) (2018).
Tratt, D.M., J.A. Hackwell, B.L. Valant-Spaight, R.L. Walterscheid, L.J. Gelinas, J.H. Hecht, C.M. Swenson, C.P. Lampen, M.J. Alexander, L. Hoffman, D.S. Nolan, S.D. Miller, J.L. Hall, R. Atlas, F.D. Marks, and P.T. Partain. GHOST: A satellite mission concept for persistent monitoring of stratospheric gravity waves induced by severe storms. Bulletin of the American Meteorological Society (doi:10.1175/BAMS-D-17-0064.1), in press.
Weatherhead, E.C., B.A. Wielicki, V. Ramaswamy, M. Abbott, T.P. Ackerman, R. Atlas, G. Brasseur, L. Bruhwiler, A.J. Busalacchi, J.H. Butler, C.T.M. Clack, R. Cooke, L. Cucurull, S.M. Davis, J.M. English, D.W. Fahey, S.S. Fine, J.K. Lazo, S. Liang, N.G. Loeb, E. Rignot, B. Soden, D. Stanitski, G. Stephens, B.D. Tapley, A.M. Thompson, K.E. Trenberth, and D. Wuebbles. Designing the climate observing system of the future. Earth’s Future, 6(1):80-102 (doi:10.1002/2017EF000627) (2018).
Zhang, J.A., R. Atlas, G.D. Emmitt, L. Bucci, and K. Ryan. Airborne Doppler wind lidar observations of the tropical cyclone boundary layer. Remote Sensing, 10(6):825 (doi:10.3390/rs10060825) (2018).
Looking for scientific literature? Visit our Publication Database.
Observing System Experiments (OSEs)
Fiscal Year 2018 Accomplishments
- Global experiments to quantify impacts of Radio Occultation, MW, IR and AMV in current operational global NWP/DA
- Regional experiments to evaluate impact of CYGNSS
- Impact of airborne Doppler Wind Lidar on hurricane analysis and prediction
- Impact of current Radio Occultation on hurricane prediction
- Ocean experiments to quantify impact of ocean observations in hurricane prediction
- Testing of new lightning data assimilation algorithms
Contact
| Lidia Cucurull, Ph.D.
Chief Scientist and Deputy Director, QOSAP Program
If you would like more information on the Quantitative Observing System Assessment Program, please contact Lidia Cucurull, Chief Scientist and Deputy Director of the Program.