Archive for September 2014
A great example of a historical forecasting methodology is PECOTA. PECOTA forecasts future performance for a player by first going back in time and finding historical players that are “similar”. It then uses information on how those historical players evolved over their careers to project how the current player will evolve. So far, this approach is as good as it can get for forecasting player performance.
Not all forecasts are this obviously historical, but all forecasts really are about intelligent selection of historical comparators.
This key relationship indicates why forecasts will always need both quantitative and qualitative components. Quantitative components — from numerical data that describes the past — are key to anchoring estimates of magnitude in an objective way.
Qualitative components are necessary to adjust for limitations in the data and to accommodate the possibility that this time “really is different”. Frequently, historical data exists because of convenience or some other business purpose; rarely is the historical data directly applicable to the current problem. A significant degree of wisdom is required to judge when “this time really is different”, as is perhaps obvious.
This post may appear to be a truism, but I've found that model interpretation and forecasting errors frequently stem from a lack of appreciation regarding the relationship between history and prediction. Curiosity, energy, and time are all required to investigate the past in a comprehensive way. It is difficult — even in retrospect — to identify key causes for historical events. It is exponentially more difficult to select and measure which of those causal relationships will be the key drivers in the future.
Companies would do well to keep in mind that forecasts are as much about the past as they are about the future. The better you know where you've been, and why, the better you will be able to navigate where you will be.
 Even quantitative measures are susceptible to subjective interpretation and biases that influence the selection of the data. Nevertheless, quantitative evaluation helps provide a degree of dispassion, if used wisely.
[Technical Post] On Friday, a Vice President in our company asked me what our definition of Economic Capital was. I responded that we defined “economic capital” as the amount of capital necessary to cover unexpected losses at the 99% confidence level. That is total and complete gibberish. I have no idea what it means, I just mirror the sentence structure used by others. For examples, see Investopedia, and other sources.
Investopedia also provides a standard graphical representation, produced below:
Below I will describe why my definition is gibberish, I will contrast this to what we are really trying to say, and I’ll close by saying that this is more than a semantics problem.