The only true wisdom is in knowing you know
nothing.
(Socrates)
Even though
this is not a technique, I decided to start this toolbox with superforecasting, inspired by the book by Philip E.
Tetlock and Dan Gardner,[1]
which I consider a must for any analyst. The authors try to explain why some experts
are good at making predictions and others are not based on several forecasting experiments.
The first
reason for poor forecasting is overconfidence in our knowledge (“illusion of knowledge”);
the example is given of how medicine improved when experimentation was
introduced. Our brain works using two systems, 1 and 2.[2]
System 1 is good for fast, effortless reasoning that happens unconsciously and
is useful in pattern recognition (for example, when driving on a route that we follow
often, we use System 1). However, System 1 can create biases or illusions,
since the reasoning behind it is based on simplifications to make fast and
effortless responses. In the case of dealing with problems with possible biases,
we have to use System 2, the system that uses reasoning and logic and, despite
requiring more time and effort, allows us to overcome the “illusion of
knowledge.” Usually, for complex problems we need to use System 2, but
sometimes, when there are many patterns or much information, intuition can be a
better solution, since it can simplify all the information and analyze
unconsciously a quantity of patterns that we are not able to analyze logically.
Another
reason for unfortunate forecasts is the lack of precision and/or measurement.
Vague forecasts cannot be measured, and therefore we cannot improve our
forecasting methods. A good prediction has to include:
- - A measurable outcome: “the USSR will dissolve”;
- - A more precise forecast with a possible date frame: “within one year from now”;
- - Probability: “80% probability that it will happen.”
Probability
should not be a measure of our confidence in the results but the result of our
reasoning (based on several pieces of information) about the probability of an
event occurring. To measure the results of our forecasts with probabilities, it
is necessary to make several similar predictions to calculate their accuracy
and calibration.
The author
also proves that average group predictions are better than individual
predictions, because they use different perspectives, knowledge, and
information. If we have to perform forecasts individually, we can behave like a
group by aggregating different sources of information, asking for other
people’s perspectives and opinions, and always questioning the results. We
should also remember to update our results based on any new, relevant
information that we obtain.
Finally, a
wonderful technique that will help in making complex forecasts is “Fermi estimation.”
It consists of breaking down a problem into several probability questions to
make more accurate guesses and then adding up those questions that will produce
a more realistic final outcome. This technique usually starts by taking into
consideration an outer perspective. For example, we are presented with a
picture of a family and we have to predict whether they have a pet or not.
Instinctively we may start by focusing on the family’s characteristics that we can
identify. However, we should start with a broader perspective, for example the
number of families in a certain geographical space (country, state, city, etc.)
who own pets, namely 30%. This is our starting point, so we know without any
other additional information that our best guess would be that there is a 30% probability
that this family owns a pet. We can then adjust our prediction with an inside
view focusing on the family’s characteristics, for example age, ethnicity,
children, and so on. A famous example is the estimation of the number of piano
tuners in Chicago. Instead of guessing the number directly, the problem is
broken down into several questions, starting with defining the total population
of the city, then determining the number of households, guessing the number of
households with a piano, gauging how often a piano needs to be tuned, and so on.[3]
