How forecast verification scores change when some forecasts are changed in some specific way?

What is it about?

Two sensitivity studies of binary forecast verification scores are examined. The first is the random changes of forecasts based on truly binary systems. The second is the sensitivity to the chosen threshold, when the binary forecast is obtained thresholding a continuous forecasting parameter.

Featured Image

Why is it important?

Do you know that some binary forecast verification scores could be improved even by random changes? When you select a threshold on a continuous forecasting parameter to optimize a given forecast verification score you should know what you are really doing. When you optimize BIAS you want that forecast and observed counts are the same. When you maximize the Pierce Skill Score you want that the forecast posterior probability is equal or larger to the event prior probability (baserate) when you issue a YES. When you optimize Percent Correct you want to forecast an event when the forecast probability exceed 0.5. On the other hand, it is not clear what does it mean to maximize CSI or HSS and how that choice depends on the event prior probability.

Perspectives