Some thoughts on probabilistic forecasting

Probabilistic forecasts can easily be misinterpreted and since they are at the core of everything we do at reask we thought we’d try and lay down some basic concepts from the get go.

Consider the following scenario: you are playing a game where your bank currently sits at 3.5$ and your next payment is about to be set from the outcome of a six-sided die roll. Given what is commonly known about this sort of dice a quick assessment of your risk gives you a 50% chance of bankruptcy after the die is toss (i.e. payments of 4, 5 or 6$). You might be happy with these odds or decide to borrow some money from another player to reduce the risk before the roll (hopefully this scenario does not sound too familiar).

You started the game assuming a traditional die was used, but you can’t be sure. Two dice have been involved until now (one blue and one yellow) and the game is set in a way where you are only shown (1) the colour of the die before the roll and (2) the upper side of the die after a roll (you can’t see the actual die structure). Having seen hundreds of throws since you started playing you start to suspect something is not quite as you first imagined. The blue die seems to come up as 4 more frequently and you are yet to see a 1 from that die. On the other hand the yellow die frequently shows up as a 3 but has never settled with a 6 on top yet.

From this experience you have built the following mental model:

• The blue die does not have a side with number 1, instead it has been replaced by an additional number 4.
• The yellow die does not have a side with number 6, instead it has been replaced with an additional number 3

Fig. 1: Mental model of dice structure

Clearly if you have any trust in this model you should use it to reassess your risk (and potentially how much you want to borrow). If the blue die is about to be thrown your mental model suggests you now face a 66.6% chance of going bust while using the yellow die would reduce the risk to 33.3%.

This assessment is a probabilistic forecast based on your past experience, mental model of the dice and observation of which coloured die is about to be thrown. It can be fully quantified as follows:

Table 1: Outcome probability from the roll of a die, according to the mental model from Fig. 1

There are a few (hopefully obvious) comments I’d like to make at this stage:

1. Once you have realized that all dice are not similar, it would seem ill-advised to keep on assessing risk using the traditional view of how a die is structured. If the colour seems to matter, you want to make sure that information is included in your decision process.
2. When the yellow die is about to be thrown you are happy to take more risk, but also accept that your mental model still considers bankruptcy a likely outcome (i.e. your model is not predicting your solvency).
3. If pushed to make a call on the outcome of a roll from the yellow die your most logical answer is 3 (most likely outcome). Yet this is not a very useful prediction and you would not be surprised to see any other number from 1 to 5 appear (which would certainly not suggest your model was “wrong”).
4. You’d consider your model to be flawed if the yellow die shows a 6 (i.e. a number to which the model had attached a 0% probability of occurrence) or if a very large number of new rolls suggest number 3 does not actually occur more frequently.

Seasonal hurricane forecasting models follow a very similar logic. Mother nature has thrown many hurricane seasons at us but, as in the game above, we are mostly limited to seeing the outcome of a season’s activity without a full understanding of the mechanisms responsible. However, like for dice colours, there are signs we can pay attention to (e.g. the state of known climate teleconnections) and from past experience we have built some mental models of the climate mechanisms responsible for hurricane activity.

The main take home message from this analogy is that comments 1-4 above, which I am sure seemed obvious when thinking about dice, are just as valid when it comes to probabilistic forecasts of hurricane activity. In particular:

1. All seasons are not equal in terms of risk and, like with the coloured dice, the signs are here to help us improve our assessments. Knowledge of the state of known climate teleconnections such as the Atlantic Multidecadal Oscillations (AMO) or El Niño–Southern Oscillation (ENSO) provide valuable insight that should inform any seasonal activity forecast.
2. A forecast for a “below normal activity” does not imply that a high activity season is impossible, simply that it is less likely. In other words, a model that predicts a season with below normal activity is not “wrong” if the season turns out to be active. Only repeated misalignment between model predictions and observed outcomes would indicate a flawed model.
3. It is our view that providing a single number as a prediction is neither helpful nor honest. Instead we aim to quantify the probability associated with a range of outcomes in a similar fashion to Table 1 above. Our models simulate a distribution of outcomes and we will always communicate our predictions using the full distribution (there is nothing special about the mean forecast!).
4. As a consequence, we will not be judging model performance based on common error metrics such as RMSE or MAE. Instead, probabilistic prediction scores will be our guide to assess model performance over the coming seasons.

Stay tuned for reask’s first northern hemisphere tropical cyclone forecast to be released early June 2018, with other southern hemisphere basins to follow in October.