The EROC Method

An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method Fredrick H. M. Semazzi, and Roberto J. Mera. Journal of Applied Meteorology and Climatology, Semptember 2006. Full text

Share |

The functional relationship between the relative operating characteristic (ROC) and the economic value (EV) graphical methods have been exploited to develop a hybrid procedure called the extended ROC (EROC) method. The EROC retains the appealing simplicity of the traditional ROC method and the ability of the EV method to provide evaluation of the performance of an ensemble climate prediction system (EPS) for a hypothetical end user defined by the cost-loss ratio (mu = C/L). An inequality defining the lower and upper theoretical bounds of mu has been derived. Outside these limits, the EPS yields no added benefits for end user mu relative to the use of climatological persistence as an alternative prediction system. In the traditional ROC graphical method, the ROC skill (ROCS) is often expressed in terms of the area between the ROC graph and the diagonal baseline passing through the origin with slope m = 1. Thus, ROCS = 2A-1, where A is the area under the ROC graph. In the proposed EROC approach, a more general procedure is recommended based on the construction of user-specific baselines that do not necessarily pass through the origin and, in general, have slope m not equal to 1. The skill of a particular EPS computed from the EROC method is proportional to the corresponding estimated value based on the EV graphical method. Therefore, the EROC geometry conveys the same basic information as the EV method. The Semazzi & Mera skill score (SMSS) is proposed as a convenient and compact way of expressing the combined verification based on the ROC and EV methods. The ROCS estimate is a special case of the SMSS. The near-horizontal trail-like geometry sometimes exhibited by EV graphs is also examined. It is shown to occur when either the hit-rate or false-alarm term dominates in the formula for EV, unlike the more typical situation in which both terms are comparable in magnitude.

Presentation at the Sixth Annual Climate Prediction Applications Science Workshop (CPASW), March 2008, Chapel Hill, NC, USA

Sample slides:


Efforts currently underway for the UCAR Africa Initiative will seek to use an ensemble of large scale models to provide operational forecasts for the Africa initiative. The NCSU Climate Modeling Laboratory will also explore the use of ensemble forecasting by downscaling each of the models in operational mode using WRF. Additionally, we will explore the relative skill and economic value of the ensemble prediction forecast using the recently published EROC method. Currently underway is a WRF climate mode ensemble using Final Analyses (FNL) initial conditions and varying physics in the model to simulate multiple model ensemble. Real-time, initial-conditions varying forecasts are also provided in a bi-weekly timescale during the meningitis season. Click on the icon on right to access the forecasts.