Christopher W. Landsea
NOAA/AOML/Hurricane Research Division,
Miami, Florida, U.S.A.
John A. Knaff
NOAA/Cooperative Institute for Research in
the Atmosphere, Fort Collins, Colorado, U.S.A.
Bulletin of the American Meteorological
Society, in press.
NOAA/AOML/Hurricane Research Division, Miami, Florida, U.S.A.
John A. Knaff
NOAA/Cooperative Institute for Research in the Atmosphere, Fort Collins, Colorado, U.S.A.
Bulletin of the American Meteorological Society, in press.
7 March, 2000
AMS Copyright Notice
© Copyright 2000 American Meteorological Society (AMS). Permission to use figures, tables, and brief excerpts from this work in scientific and educational works is hereby granted provided that the source is acknowledged. Any use of material in this work that is determined to be "fair use" under Section 107 or that satisfies the conditions specified in Section 108 of the U.S. Copyright Law (17 USC, as revised by P.L. 94-553) does not require the Society's permission. Republication, systematic reproduction, posting in electronic form on servers, or other uses of this material, except as exempted by the above statements, requires written permission or license from the AMS. Additional details are provided in the AMS Copyright Policies, available from the AMS at 617-227-2425 or firstname.lastname@example.org. Permission to place a copy of this work on this server has been provided by the AMS. The AMS does not guarantee that the copy provided here is an accurate copy of the published work.
The very strong 1997-98 El Niņo was the first major event
in which numerous forecasting groups participated in its real-time prediction. A
previously developed simple statistical tool - the El Niņo-Southern Oscillation
CLImatology and PERsistence (ENSO-CLIPER) model - is utilized as a baseline for
determination of skill in forecasting this event. Twelve statistical and
dynamical models were available in real-time for evaluation. Some of the models
were able to outperform ENSO-CLIPER in predicting either the onset or the decay
of the 1997-98 El Niņo, but none were successful at both for a medium-range two
season (6-8 months) lead time. There were no models, including ENSO-CLIPER, able
to anticipate even one-half of the actual amplitude of the El Niņo's peak at
medium-range (6-11 months) lead. In addition, none of the models showed skill
(i.e. lower root mean square error than ENSO-CLIPER) at the zero season (0-2
months) through the two season (6-8 months) lead times. No dynamical model and
only two of the statistical models (the canonical correlation analysis [CCA] and
the constructed analog [ANALOG]) outperformed ENSO-CLIPER by more than 5% of the
root mean square error at the three season (9-11 months) and four season (12-14
months) lead time. El Niņo impacts were correctly anticipated by national
meteorological centers one half year in advance, because of the tendency for El
Niņo events to persist into and peak during the Boreal winter. Despite this, the
zero to two season (0-8 month) forecasts of the El Niņo event itself were no
better ENSO-CLIPER and were, in that sense, not skillful - a conclusion that
remains unclear to the general meteorological and oceanographic
The very strong 1997-98 El Niņo caused dramatic worldwide
changes to weather patterns, such as drought in Indonesia, extreme rains in Peru
and Ecuador and a shutdown of the Atlantic hurricane season (Bell and Halpert
1998). While past El Niņo events have also produced similar though not as large
effects, this was the first El Niņo in which national meteorological centers
made accurate forecasts of its impacts several months in advance (Barnston et
al. 1999a,b). One issue that remains unresolved is how much skill did the
various El Niņo-Southern Oscillation (ENSO) forecast methodologies have for the
event itself at various lead times. This issue was originally addressed in
Barnston et al. (1999a), which analyzed the eight 3-monthly forecast times from
June 1996 to March 1998 for the operationally-available statistical and
dynamical models. They found that some of the models performed worse than a
persistence control forecast, while most of the models performed quite well
relative to persistence for the duration of the 1997-98 El Niņo event. Both
statistical and dynamical models were found in each categorization of
performance. However, three key issues require a re-visitation of this topic.
The first is that an explicit analysis of the models' performance is needed for
the onset and decay of the event. Secondly, stratification of the various
forecasts would allow for analysis of how skill changes with increased lead
time. Finally, the error analysis should be put into the context of evaluating
the available schemes with respect to a common ``no-skill" threshold that is
more challenging than simple persistence alone: the El Niņo-Southern Oscillation
1 and PERsistence (ENSO-CLIPER)
2 model (Knaff and Landsea 1997).
In evaluating ENSO prediction models, two aspects should
be conside red: do the forecasts present useful information and do the forecasts
have skill? The first does not automatically imply the second holds true and
vice versa. The first aspect of ``usefulness" is whether the predictions can
differentiate between the phases (El Niņo, La Niņa and neutral) and, when an El
Niņo or La Niņa is present, to know the approximate magnitude of it. Barnston et
al. (1999a) delineate ENSO phases into the five categories based upon
approximately 1.0oC differences for the
Niņo 3.4 SST anomaly region: strong/very strong El Niņo, weak/moderate El Niņo,
neutral ENSO, weak/moderate La Niņa, and strong/very strong La Niņa. (The same
divisions could be done with approximately 1.35oC for the Niņo 3 SST anomaly region.) We apply this
1.0oC (1.35oC for Niņo 3) threshold as a criterion below which
determines when a prediction is ``useful" from a root mean square error
(RMSE) analysis (Spiegel 1988). This terminology will be utilized for evaluating
onset, decay as well as the whole 1997-98 El Niņo event.
Secondly, to provide a baseline of skill in seasonal ENSO
forecasting, we utilize ENSO-CLIPER (Knaff and Landsea 1997) as a simple,
statistical model that takes best advantage of the climatology of past ENSO
events, persistence of the initial conditions and their current multi-month
trend. The output from ENSO-CLIPER replaces the use of persistence of anomalies
as a skill threshold, although it is recognized that other simple statistical
models could be employed for this test. ``Skill" is then redefined as the
ability to improve upon ENSO-CLIPER - a more difficult task. ENSO-CLIPER was
developed as a multiple least squares regression from a total pool of fourteen
possible predictors which were screened for the best predictors based upon
1950-1994 developmental data. A range of zero to four predictors were chosen in
determining the regression models, developed separately for each initial
calendar month. Most (72%) of the 480 (12 months x 5 predictands x 8 leads)
ENSO-CLIPER regression equations developed contain only zero, one or two
predictors. The predictands to be forecast include Niņo 1+2 (0-10oS, 80-90oW), Niņo 3 (5oN-5oS,
90-150oW), Niņo 3.4 (5oN-5oS,
120-170oW) and Niņo 4 (5oN-5oS,
150oW-160oE) sea surface temperature (SST) anomaly indices for
the equatorial eastern and central Pacific and the Southern Oscillation Index
(SOI) (Figure 1) at lead times ranging from zero seasons (0-2 months) through seven
seasons (21-23 months). The previous one, three and five month observed values
and corresponding time trends of the various Niņo SST indices and the SOI also
serve as possible predictors. The pool contains fourteen potential predictors:
six from the predictand itself and two each from the other four ENSO indices
(Niņo SST regions and SOI). Prospective predictors were retained only if they
correlated in the regression test at a significance level beyond 95% using a
student t test and increased the total variance explained by at least 2.5%. To
reduce the chance of statistical overfitting, only one of the three time
averages for predictors (one, three or five months) and only one of the three
time trends are allowed to be chosen from each of the various Niņo SST indices
and the SOI. Though hindcast ability is strongly seasonally dependent,
substantial improvement is achieved over simple persistence wherein largest
improvements occur for the two to seven season (6 to 23 months) lead
We interpret ENSO-CLIPER as capturing the climatological
aspects of the whole ENSO complex in depicting both mean conditions and
propagation of those features in time. In essence this model, given initial
conditions of ENSO (SST anomalies in Niņo regions 1+2, 3, 3.4, and 4, and the
SOI) and the recent past valid at a particular time, will fit using regression
techniques the best evolution from those initial conditions. The method has been
frozen following its development (over 40 years of dependent data), and yields
the mean climatological evolutions for that period. The ENSO-CLIPER model thus
offers a baseline "no-skill" forecast of ENSO variability and a useful (as
defined earlier) ENSO prediction as well
Operational forecasts from the various ENSO prediction
models were provided through digitizing of results shown in the Experimental
Long-Lead Forecast Bulletin (ELLFB - Barnston 1995, 1996, 1997a; Kirtman 1998)
and the Climate Diagnostics Bulletin (Kousky 1995, 1996, 1997, 1998) with
confirmation of a few cases' values from A. G. Barnston (1999, personal
communication). (Note that while ENSO-CLIPER has been run in real-time from late
1996 onward, June 1997 marked its first appearance in the ELLFB - just before
the publication of Knaff and Landsea (1997).) Only models that provided at least
a two-season lead forecast valid for the entire duration of the El Niņo event
were considered4. The zero and one season forecasts (0-5 months)
will be termed ``short range" predictions. The two and three season forecasts
(6-11 months) will be called ``medium range" predictions. "Long-range"
predictions are those for the remaining four to seven season leads (12-23
months). All forecast SST anomalies were adjusted to the standard 1950-79 base
period climatology, resulting in small (0 to 0.2oC) alterations in some of the model predictions. An
analysis and comparison of the model forecasts for seven non-overlapping seasons
was performed from just before the onset of the El Niņo in early 1997 to just
after the decay of the event in mid 1998.
Models for forecasting the ENSO phenomena can be broadly broken into two categories: statistical and dynamical. The statistical models range from simple linear regression schemes to more sophisticated techniques such as neural networks, time series analysis, multivariate multiple regression and ensemble methods. Dynamical ENSO models range from simplified linear shallow water equations for both the ocean and atmosphere, to the intermediate models with two active layers representing the ocean, to the hybrid coupled models that have a comprehensive ocean system coupled to a statistical atmosphere, to the comprehensive coupled models with multi-level representations of both the ocean and atmosphere. However, even the comprehensive coupled models still require statistical corrections to account for systematic biases in the model output, likely related to the extremely difficult task of ideally specifying the transfer of heat, moisture and momentum from ocean to atmosphere and vice versa.
The models evaluated for prediction of Niņo 3.4 SST
anomalies were the following: ENSO-CLIPER (Knaff and Landsea 1997), the
constructed analog statistical model (ANALOG - Van den Dool 1994), the
consolidation (ensemble) statistical method (CONSOL - Unger et al. 1996), the
National Center for Environmental Prediction comprehensive dynamical model (NCEP
- Ji et al. 1996), the neural network statistical model (NEURAL - Tangang et al.
1997), the Scripps/Max Planck Institute hybrid dynamical model (SCR/MPI -
Barnett et al. 1993), the canonical correlation analysis statistical model (CCA
- Barnston and Ropelewski 1992) and the University of Oxford intermediate
dynamical model (OXFORD
5 - Balmaseda et al. 1994). The models
evaluated for predictions of Niņo 3 SST anomalies were ENSO-CLIPER, the Bureau
of Meteorology Research Centre simplified dynamical model (BMRC - Kleeman et al.
1995), the Center for Ocean-Land-Atmosphere Studies comprehensive dynamical
model (COLA - Kirtman et al. 1997), the Lamont-Doherty simplified dynamical
model (LDEO - Zebiak and Cane 1987), the singular spectrum analysis/maximum
entropy method statistical model (SSA/MEM - Keppenne and Ghil 1992) and the
linear inverse statistical model (LIM - Penland and Magorian 1993). An important
caveat is that some of these models were not specifically designed for only
forecasting ENSO. However, the purpose of this assessment is to analyze
the performance of the models in predicting ENSO, as measured by the Niņo 3 or
Niņo 3.4 SST anomalies. Forecasts based upon the traditional persistence
(PERSIS) of anomalies from the initial season were also utilized for comparison
Figure 2 presents the two-season lead forecasts and the observed SST anomalies,
referred to as verifications, for all twelve schemes as well as ENSO-CLIPER and
persistence. Except for the observations, the forecast values in the figure are
not a time series as the individual points came from separate runs from the
models. It is apparent that none of the models performed extremely well for this
medium range (6-8 months) forecast. Some of the models did well for the onset of
the El Niņo event (e.g. NCEP and CCA), some were able to capture the decay of
the event (e.g. ENSO-CLIPER, SCR/MPI and LIM), but the peak SST anomalies that
occurred in late 1997 were dramatically underestimated (by one half the
amplitude or greater) by all models at this lead. Additionally, none of the
models captured both the onset and the decay of the El Niņo event with success
at this two season lead.
To better quantify how the models performed for the
lifecycle of the El Niņo event, an evaluation of forecast skill was conducted
for the times of the El Niņo's onset and decay, defined similarly to that of
Trenberth (1997). It is not expected that the ENSO prediction models could (or
should) perform better during the onset/decay phases relative to the whole ENSO
lifecycle. However, these phases of El Niņo are of keen climatological and
societal interest because that is when the ENSO teleconnections typically begin
and end. The "onset" stage is defined as the first three month period when the
Niņo 3.4 SST anomalies exceed +0.4oC,
which was Mar.-May 1997. For Niņo 3, the onset occurred during the same months.
The "decay" stage is defined as the first three month period that Niņo 3.4 SST
anomalies averaged lower than +0.4oC once again, which was Apr.-Jun.
1998. For Niņo 3, the decay began slightly later during May-Jul. 1998.
The two three-month forecasts from each scheme that
bracket the onset and decay phase are evaluated for RMSE in
Table 1. For onset, nearly all prediction schemes proved to be useful for the
short and medium range forecasts. The exceptions were LDEO (not useful at either
short or medium ranges) and SCR/MPI (not useful at the medium range). At the
long range forecasts, only ANALOG, BMRC, CCA, ENSO-CLIPER and LIM provided
useful predictions, though several (NCEP, NEURAL, OXFORD, SCR/MPI and SSA/MEM)
are not run this far into the future. However, when one places the additional
constraint of having to outperform ENSO-CLIPER, the number of models that were
useful and showed skill for the El Niņo onset are reduced: short-range - BMRC,
CCA, COLA, LIM and SSA/MEM; medium-range - ANALOG, BMRC, COLA, CONSOL, LIM, NCEP
and SSA/MEM; and long-range - BMRC. Note that it is quite possible to be useful
and show skill at a medium-range forecast, but for these not to hold true at the
short-range (e.g. ANALOG, CONSOL and NCEP). For the medium-range two season (6-8
months) forecasts depicted in Figure 2, NCEP had by far the most useful and skillful predictions of the El Niņo
onset improving upon ENSO-CLIPER by 31% in terms of RMSE.
For the decay of the El Niņo in the boreal spring of
1998, all of the ENSO schemes provided useful short and medium-range predictions
except for CONSOL (not useful at either the short or medium-range), LDEO (not
useful at the medium range), NCEP (not useful at the medium range) and OXFORD
(not useful at either the short or medium-range). For the long-range, only
ANALOG, COLA and ENSO-CLIPER provided useful forecasts; though again several
models (NCEP, NEURAL, OXFORD, SCR/MPI and SSA/MEM) are not run out this far into
the future. When the analysis also includes improving upon ENSO-CLIPER's
forecast of the decay, only a few models are both useful and show skill:
short-range - ANALOG, CCA; medium-range - ANALOG, COLA and LIM; and long-range -
none. For the medium-range two season (6-8 months) predictions depicted in Figure 2, LIM had the most skillful prediction of the El Niņo decay improving upon
ENSO-CLIPER by 22% in terms of RMSE.
To best assess the event-long performance (onset, peak
and decay) of the individual forecasting schemes, the RMSE was calculated for
each lead time in Table 2 6. Note that the RMSE errors for PERSIS - which
has been the traditional standard for determining skill - are quite large and
are only exceeded by LDEO (at a zero and one season lead) and LIM (at a zero
season lead). Additionally, the two season lead forecasts of LDEO had errors
equal to those of PERSIS. However, to state - as has been done traditionally -
that these two season lead forecasts from LDEO in Figure 2 are on the threshold of having "skill" based on persistence as a control
highlights the need for a more stringent standard. For the duration of the El
Niņo event, the models have only limited ability to show usefulness. At the
short-range, ANALOG, BMRC, CCA, COLA, ENSO-CLIPER, NCEP, NEURAL and SSA/MEM
provide useful predictions. None of the models give useful forecasts at the
medium and long-range lead times.
Figure 3 helps to summarize the details found in
Table 2. These diagrams provide a direct comparison of the RMSE of the various
models versus ENSO-CLIPER. Persistence is not plotted because of the extremely
poor predictions utilizing this methodology which would dominate the y-axis in
the figure. It is readily apparent that none of the ENSO models - statistical
and dynamical - were able to provide skillful forecasts at the short-range lead
times. Only CCA, ANALOG and COLA were able to outperform ENSO-CLIPER at the
medium and long-ranges: 23% and 12% lower RMSE for CCA, 6% and 9% for ANALOG,
and 5% and 4% for COLA at the three and four season lead, respectively. The
small number of forecasts (seven per lead time) do not make for very meaningful
significance testing at this time. It is to be noted that, in general, models
improve relative to ENSO-CLIPER as lead time is increased. Thus it is quite
possible that models currently only run out to a two season lead (such as NCEP
and SCR/MPI) might have shown skill if they were integrated further out in time.
Finally, none of the models, however, are both useful and skillful for the
duration of the 1997-98 El Niņo event at any lead time.
Linear correlation coefficients were also calculated for
all models at all forecast periods versus observed anomalies
(Table 3). Results were quite similar qualitatively to those of the RMSE
The analyses here conclude that the best performing model
for forecasting the entirety of the very strong 1997-98 El Niņo at the
short-range lead was the statistical ENSO-CLIPER model (Knaff and Landsea 1997),
while the best at the medium-range lead was the canonical correlation analysis
statistical model (Barnston and Ropelewski 1992). Thus the use of more complex,
physically realistic dynamical models does not automatically provide more
reliable forecasts. Increased complexity can increase by orders of magnitude the
sources for error, which can cause degradation in skill. Despite the lack of
skill in forecasting ENSO itself up to 8 months in advance, national
meteorological centers were able to correctly anticipate the effects of the
1997-98 El Niņo because of the tendency for El Niņo events to persist into and
peak during the boreal winter (Barnston et al. 1999a). Indeed the U.S. Climate
Prediction Center's most skillful tools (measured by the Heidke skill score) for
predicting the U.S. seasonal precipitation anomalies were the statistical ENSO
composites and the statistical optimal climate normals, rather than the NCEP
coupled model (Barnston et al. 1999b). (For seasonal temperature anomalies in
the United States, the two statistical and one dynamical tools were about equal
in skill.) No dynamical models were needed to anticipate a wet and stormy winter
for the southern tier of the United States and a warm winter for the northern
tier of states.
Within this paper we have utilized ENSO-CLIPER as the
baseline methodology against which other prediction schemes can be judged for
skill, depending on whether they outperformed ENSO-CLIPER ("skillful") or not
("no skill"). One may, however, alternatively interpret ENSO-CLIPER to be more
than a strict combination of climatology and persistence, since it does allow
for phase propagation of ENSO within the Niņo regions. As we argue in the
Introduction, this should not invalidate ENSO-CLIPER as a no-skill test since it
is providing the climatological evolution of past ENSO events and is simpler
than many statistical and all numerical models. Even if one does not agree with
this reasoning, two points are still clearly evident: 1) a distinct need exists
for a standard against which ``skill" is to be measured in predicting the ENSO
phenomena. Use of the simple persistence of anomalies is much too easy a
benchmark to exceed. If not ENSO-CLIPER, some other comparative test is
essential for ENSO forecasting. (See Qin and Van den Dool (1996) for a another
creative ``no-skill" benchmark test.); and 2) the multiple regression-based
ENSO-CLIPER outperformed all of the more sophisticated models - both other
statistical schemes as well as numerical techniques - for zero to two season
lead (0 to 8 months). Thus these more complex models may not be doing much more
than carrying out a pattern recognition and extrapolation of their own. National
meteorological centers may wish to consider carefully their resource priorities
(both personnel and computers) when the current best tools are the relatively
cheap statistical systems, relative to the expensive (developmentally and
computationally) dynamical models which have not yet produced skillful and
useful operational ENSO forecasts at any forecast lead times.
The results herein may be surprising given the general
perception that seasonal El Niņo forecasts from dynamical models have been
quite successful and may even be considered a solved problem (e.g. Kerr 1998,
Stern and Easterling 1999, Porter 1999). Kerr's (1998) report - ``Models win big
in forecasting El Niņo" - in Science, in particular, generated widespread
publicity for the success in forecasting the 1997-98 El Niņo's onset by the
comprehensive dynamical models. His report was based upon Barnston's (1997b)
unrefereed and incomplete (since only the onset was considered) analysis. No
followup mention in Science was forthcoming when the Barnston et al. (1999a)
paper was finally published showing that the comprehensive dynamical models did
not ``win big" after all. (It is worth mentioning that the results from Barnston
et al. (1999a) do indeed agree quite well in general with what is shown
here, though the interpretation is very different.)
Also disturbing is that others are using the supposed
success in dynamical El Niņo forecasting to support other agendas. As an
example, an overview paper by Ledley et al. (1999) to support the American
Geophysical Union's ``Position Statement on Climate Change and Greenhouse Gases"
said the following:
"Confidence in [comprehensive coupled] models [for anthropogenic global warming scenarios] is also gained from their emerging predictive capability. An example of this capability is the development of a hierarchy of models to study the El Niņo-Southern Oscillation (ENSO) phenomena.....These models can predict the lower frequency responses of the climate system, such as anomalies in monthly and season averages of the sea surface temperatures in the tropical Pacific."
On the contrary, with this results of this study, one could even have less confidence in anthropogenic global warming studies because of the lack of skill in predicting El Niņo (or, alternatively, the inability of dynamical models to outperform relatively simple statistical schemes). The bottom line is that the successes in ENSO forecasting have been overstated (sometimes drastically) and misapplied in other arenas.
A followup study will assess forecast skill of the strong
1998-2000 La Niņa event, which immediately followed the El Niņo. This
examination of ENSO forecast skill only analyzed seven forecasts per lead time,
so the findings here are rough indications of the relative skills of the various
models and approaches. It may be that with consideration of the most recent,
complete ENSO warm and cold cycle that truly skillful predictions from models
are available. But the current answer to the question posed in this article's
title is that there was essentially no skill in forecasting the very strong
1997-98 El Niņo at lead times ranging from 0 to 8 months when using the
performance of ENSO-CLIPER as the no-skill baseline. Moreover, the lack of skill
at the short to medium-range lead times continues to confirm what was also
observed in independent tests of real-time ENSO prediction models for the period
1993-96 (Knaff and Landsea 1997).
In this study, we have utilized the simple ENSO-CLIPER statistical model as a new baseline standard for determination of "skill" in predicting the very strong 1997-98 El Niņo primarily through analysis of the RMSE.
For the onset of the El Niņo, the following models were both useful and provided skillful forecasts:
For the decay of the El Niņo, the following models were both useful and provided skillful forecasts:
For the overall depiction of the 1997-98 El Niņo event from onset in Spring 1997, to peak in Winter 1997-98, to decay in Spring 1998, the following models provided skillful forecasts:
However, since no models were able to provide useful
predictions at the medium and long-ranges, there were no models that provided
both useful and skillful forecasts for the entirety of the 1997-98 El Niņo. This
is a conclusion that remains unclear to the general meteorological and
We thank Tony Barnston and Dave Enfield fo r fruitful
discussions on the topic of forecasting ENSO and what constitutes measures of
skill. Stan Goldenberg, Dennis Mayer, Huug Van den Dool and two anonymous
reviewers provided very useful comments and suggestions on earlier versions of
Balmaseda, M. A., D. L. T. Anderson and M. K. Davey, 1994: ENSO prediction using a dynamical ocean model coupled to statistical atmospheres. Tellus, 46A, 497-511.
Barnett, T. P., M. Latif, N. Graham, M. Flugel, S. Pazan, and W. White, 1993: ENSO and ENSO-related predictability. Part 1: Prediction of equatorial Pacific sea surface temperature with a hybrid coupled ocean-atmosphere model. J. Climate, 6, 1545-1566.
Barnston, A. G. (Ed.), 1995: Experimental Long-Lead Forecast Bulletin. 4.1, 4.2, 4.3, 4.4, Climate Prediction Center, NOAA, Washington.
Barnston, A. G. (Ed.), 1996: Experimental Long-Lead Forecast Bulletin. 5.1, 5.2, 5.3, 5.4, Climate Prediction Center, NOAA, Washington.
Barnston, A. G. (Ed.), 1997a: Experimental Long-Lead Forecast Bulletin. 6.1, 6.2, 6.3, 6.4, Climate Prediction Center, NOAA, Washington.
Barnston, A. G., 1997b: ENSO forecasts for 1998: Experimental long lead predictions. Proceedings of the Twenty-second Annual Climate Diagnostics and Prediction Workshop, Berkeley, California, NOAA, 6-9.
Barnston, A. G., M. H. Glantz, and Y. He, 1999a: Predictive skill of statistical and dynamical climate models in SST forecasts during the 1997-98 El Niņo episode and the 1998 La Niņa onset. Bull. Amer Meteor. Soc., 80, 217-243.
Barnston, A. G., A. Leetmaa, V. E. Kousky, R. E. Livezey, E. A. O'Lenic, H. Van den Dool, A. J. Wagner, and D. A. Unger, 1999b: NCEP forecasts of the El Niņo of 1997-98 and its U.S. impacts. Bull. Amer. Meteor. Soc., 80, 1829-1852.
Barnston, A. G. and C. F. Ropelewski, 1992: Prediction of ENSO episodes using canonical correlation analysis. J. Climate, 7, 1316-1345.
Bell, G. D., and M. S. Halpert, 1998: Climate assessment for 1997. Bull. Amer. Meteor. Soc., 79, S1-S50.
DeMaria, M., 1996: Hurricane forecasting, 1919-1995. Historical Essays on Meteorology 1919-1995, J. R. Fleming, Ed., American Meteorological Society, Boston, pp. 263-305.
DeMaria, M., M. B. Lawrence, and J. T. Kroll, 1990: An error analysis of Atlantic tropical cyclone track guidance models. Wea. Forecasting, 5, 47-61.
Huschke, R. E., 1959: Glossary of Meteorology. Second Printing, American Meteorological Society, 45 Beacon St., Boston, 638 pp.
Jarvinen, B. R., and C. J. Neumann, 1979: Statistical forecasts of tropical cyclone intensity. NOAA Tech. Memo., NWS NHC-10, 22 pp.
Ji, M., A. Leetmaa, and V. E. Kousky, 1996: Coupled model forecasts of ENSO during the 1980s and 1990s at the National Meteorological Center. J. Climate, 9, 3105-3120.
Keppenne, C. L. and M. Ghil, 1992: Adaptive spectral analysis and prediction of the Southern Oscillation Index. J. Geophys. Res., 97, 20449-20554.
Kerr, R. A., 1998: Models win big in forecasting El Niņo. Science , 280, 522-523.
Kirtman, B. (Ed.), 1998: Experimental Long-Lead Forecast Bulletin. 7.1, 7.2, Center for Ocean-Land-Atmosphere Studies (COLA), Calverton, MD, (http://grads.iges.org/ellfb/).
Kirtman, B. P., J. Shukla, B. Huang, Z. Zhu, and E. K. Schneider, 1997: Multiseasonal predictions with a coupled tropical ocean global atmosphere system. Mon. Wea. Rev., 125, 789-808.
Kleeman, R., A. M. Moore, and N. R. Smith, 1995: Assimilation of sub-surface thermal data into an intermediate tropical coupled ocean-atmosphere model. Mon. Wea. Rev., 123, 3103-3113.
Knaff, J. A, and C. W. Landsea, 1997: An El Niņo-Southern Oscillation CLImatology and PERsistence (CLIPER) Forecasting Scheme. Wea. Forecasting, 12, 633-652.
Kousky, V. E. (Ed.), 1995: Climate Diagnostics Bulletin. 95, Climate Prediction Center, NOAA, Washington.
Kousky, V. E. (Ed.), 1996: Climate Diagnostics Bulletin. 96, Climate Prediction Center, NOAA, Washington.
Kousky, V. E. (Ed.), 1997: Climate Diagnostics Bulletin. 97, Climate Prediction Center, NOAA, Washington.
Kousky, V. E. (Ed.), 1998: Climate Diagnostics Bulletin. 98, Climate Prediction Center, NOAA, Washington.
Ledley, T. S., E. T. Sundquist, S. E. Schwartz, D. K. Hall, J. D. Fellows, and T. L. Killeen, 1999: Climate change and greenhouse gases. Eos, 80, 454-458.
Neumann, C. J., 1972: An alternative to the HURRAN tropical cyclone model system. NOAA Tech. Memo., NWS SR-62, 22 pp.
Penland, C. and T. Magorian, 1993: Prediction of Nino 3 sea-surface temperatures using linear inverse-modeling. J. Climate, 6, 1067-1076.
Porter, H. F., 1999: Forecast: Disaster - The Future of El Niņo. Dell Publishing, New York, 196 pp.
Qin, J., and H. M. Van den Dool, 1996: Simple extensions of an NWP model. Mon. Wea. Rev., 124, 277-287.
Spiegel, M. R., 1988: Statistics-Schaums Outline Series. Second Edition, McGraw-Hill, Inc., New York, NY, 504 pp.
Stern, P. C., and W. E. Easterling (Eds.), 1999: Making Climate Forecasts Matter. Panel on the Human Dimensions of Seasonal-to-Interannual Climate Variability, National Research Council, National Academy Press, Washington, 175 pp.
Tangang, F. T., W. W. Hsieh, and B. Tang, 1997: Forecasting the equatorial Pacific sea surface temperatures by neural network models. Climate Dyn., 13, 135-147.
Trenberth, K. E., 1997: The definition of El Niņo. Bull. Amer. Meteor. Soc., 78, 2771-2777.
Unger, D., A. Barnston, H. Van den Dool, and V. Kousky, 1996: Consolidated forecasts of tropical Pacific SST in Niņo 3.4 using two dynamical models and two statistical models. Experimental Long-Lead Forecast Bulletin, 5.1, 50-52.
Van den Dool, H. M., 1994: Searching for analogues, how long must we wait? Tellus, 46A, 314-324.
WMO, 1992: International Meteorological Vocabulary. Second Edition, Secretariat of the World Meteorological Organization, Geneva, Switzerland, 784pp.
Zebiak, S. E., and M. A. Cane, 1987: A model El
Niņo-Southern Oscillation. Mon. Wea. Rev., 115, 2262-2278.
1 Study of the mean physical state of the atmosphere together with its statistical variations in both space and time as reflected in the weather behavior over a period of many years (WMO 1992). The specific study of Climate. In addition to the presentation of climatic data, it includes the analysis of the causes of differences of climate, and the application of climatic data to the solution of specific design or operational problems (Huschke 1959).
2. The use of "CLIPER" types of models as the standard for comparison has a long history in other forecasting arenas, as for example, in tropical cyclone track and intensity forecasting (Neumann 1972, Jarvinen and Neumann 1979). Such optimal combinations of climatology, persistence and trend have proven to be invaluable tools in validating new tropical cyclone prediction schemes, in both real-time and hindcasts (DeMaria et al. 1990; DeMaria 1996).
3. Details about the ENSO-CLIPER model, including its anticipated forecast performance and its predictor selection rules, can be found in Knaff and Landsea (1997). The program to run ENSO-CLIPER and independent forecasts from ENSO-CLIPER since 1 January 1993 are available at the Web site: http://www.aoml.noaa.gov/hrd/Landsea
4. Here we follow the nomenclature of Barnston and Ropelewski (1992) wherein zero lead indicates predictions for the next immediately upcoming month (their Fig. 5). For example, a forecast issued on 1 February for February through April conditions is termed a zero lead seasonal forecast. A 1 February forecast for May through July is a one season lead forecast and so forth.
5. These forecasts are actually for the region ``Eq. 2" (5oN-5oS, 130-170oW), which is quite similar to Niņo 3.4.
6 .The 1 May 1996 and 1 August 1996 NEURAL forecasts were for the Niņo 3 region. The
remaining NEURAL forecasts were issued for the Nino 3.4 region. These NEURAL
RMSE values should be compared with the following homogeneous ENSO-CLIPER
statistics: zero season lead .40, one season lead .79, two season lead 1.08. The
SCR/MPI forecast proved difficult to score because of a lack in standardization
of predictions presented in the Experimental Long-Lead Forecast Bulletin. Only
six one season lead and seven two season lead forecasts could be verified. The
corresponding verification statistics from ENSO-CLIPER are the ones appropriate
for comparison: one season lead .81 and two season lead 1.06.
|Scheme||Lead 0||Lead 1||Lead 2||Lead 3||
||Lead 5||Lead 6||Lead 7|
|Verification dates: DJF 1996-97 and MAM 1997|
|Scheme||Lead 0||Lead 1||Lead 2||Lead 3||Lead 4||Lead 5||Lead 6||Lead 7|