80 Fundamental Models for Business Analysts: 2. SUPERFORECASTING

The only true wisdom is in knowing you know nothing.

(Socrates)

Even though this is not a technique, I decided to start this toolbox with superforecasting, inspired by the book by Philip E. Tetlock and Dan Gardner,[1] which I consider a must for any analyst. The authors try to explain why some experts are good at making predictions and others are not based on several forecasting experiments.

The first reason for poor forecasting is overconfidence in our knowledge (“illusion of knowledge”); the example is given of how medicine improved when experimentation was introduced. Our brain works using two systems, 1 and 2.[2] System 1 is good for fast, effortless reasoning that happens unconsciously and is useful in pattern recognition (for example, when driving on a route that we follow often, we use System 1). However, System 1 can create biases or illusions, since the reasoning behind it is based on simplifications to make fast and effortless responses. In the case of dealing with problems with possible biases, we have to use System 2, the system that uses reasoning and logic and, despite requiring more time and effort, allows us to overcome the “illusion of knowledge.” Usually, for complex problems we need to use System 2, but sometimes, when there are many patterns or much information, intuition can be a better solution, since it can simplify all the information and analyze unconsciously a quantity of patterns that we are not able to analyze logically.

Another reason for unfortunate forecasts is the lack of precision and/or measurement. Vague forecasts cannot be measured, and therefore we cannot improve our forecasting methods. A good prediction has to include:

- A measurable outcome: “the USSR will dissolve”;
- A more precise forecast with a possible date frame: “within one year from now”;
- Probability: “80% probability that it will happen.”

Probability should not be a measure of our confidence in the results but the result of our reasoning (based on several pieces of information) about the probability of an event occurring. To measure the results of our forecasts with probabilities, it is necessary to make several similar predictions to calculate their accuracy and calibration.

The author also proves that average group predictions are better than individual predictions, because they use different perspectives, knowledge, and information. If we have to perform forecasts individually, we can behave like a group by aggregating different sources of information, asking for other people’s perspectives and opinions, and always questioning the results. We should also remember to update our results based on any new, relevant information that we obtain.

Finally, a wonderful technique that will help in making complex forecasts is “Fermi estimation.” It consists of breaking down a problem into several probability questions to make more accurate guesses and then adding up those questions that will produce a more realistic final outcome. This technique usually starts by taking into consideration an outer perspective. For example, we are presented with a picture of a family and we have to predict whether they have a pet or not. Instinctively we may start by focusing on the family’s characteristics that we can identify. However, we should start with a broader perspective, for example the number of families in a certain geographical space (country, state, city, etc.) who own pets, namely 30%. This is our starting point, so we know without any other additional information that our best guess would be that there is a 30% probability that this family owns a pet. We can then adjust our prediction with an inside view focusing on the family’s characteristics, for example age, ethnicity, children, and so on. A famous example is the estimation of the number of piano tuners in Chicago. Instead of guessing the number directly, the problem is broken down into several questions, starting with defining the total population of the city, then determining the number of households, guessing the number of households with a piano, gauging how often a piano needs to be tuned, and so on.[3]