There’s a wonderful paper by Richard Zeckhauser (most of his stuff is pretty good) called investing in the unknown and unknowable. I could write about it but it’s not too long and worth a read without me spoiling it. One thing that I do want to put up though is a little quiz from the end of it. The quiz (below) works as follows; you are given a selection of questions you have no reason to know the answers to and are asked to create confidence intervals of varying degrees. For example the number you put as your 1st %ile should be a number you don’t think you have more than 1 in 100 chance of the answer being lower. The range from your 1st %ile to the 99th %ile should be wide enough that you should only get 1 in 50 of these ranges incorrect. Sounds simple enough so give it ago. I recommend taking a screenshot of the grid and either printing it, or putting it in a word document and overlaying a spreadsheet.
The actual answers were in the paper so I’ll put them below. And yes this means you could look at them but then you’d only learn less about yourself in the process.
.
.
.
.
.
.
.
.
.
.
Answers below:
- 173,710
- 2,716
- 2,007,901
- 130,119
- 13
- 12,212,000
- $259B
- 13.45%
- 853,000
- 173
- 12,466
- 2,444
What I think will become, at least for many people, clear is that we seem to have a large degree of overconfidence even when we know that is what we are being tested on. In this number of questions we should expect nothing outside our 1%-99% range and 6 answers at most outside your 25%-75% range. It would also be useful to look at if you tend to overestimate or underestimate answers. Two observations. Firstly just from my own attempt (which I’ll put below) I saw that knowledge definitely helps, for example I have enough of an understanding of bonds to know the 30 year yield is not likely to be negative or above 100%. This may seem a very broad bracket but when I tested this on some other people who would be happy to admit little knowledge of bonds they had to have a much much larger band of answers. Two, in the answers I was more unsure about a logarithmic scale seems to work quite well. For example the population of Cambodia or length of the Congo river. It seems like this maybe a good rule of thumb to use when one is in a highly uncertain environment to avoid falling foul to overconfidence. For my attempt red means the actual answer was outside my most extreme bounds the rest should hopefully be obvious.
The link below has an excel sheet that will mark everything for you and then show answers. You can also save different peoples responses to another sheet. It has macros so you need them enabled to use it.
https://www.dropbox.com/s/npzlnck5yr233ix/confidence%20level%20quiz%20copy.xlsm?dl=0
Neil Hacker 