When we see an economic study or scientific test how do we know if the results are reliable? How do we know if we should pay attention to what we see labelled as scientific? We can look at the methodology or the ways the data was analysed but there is one thing that can act as a quick indicator as to whether a study is reliable. Sample size. Would you be wise to take the findings of a study that got data on 3 subjects? How about 50, or 200? To find the appropriate sample we need a few things.
- Confidence level (Z-score) e.g for a 95% confidence level we would use a Z-score of 1.96
- Standard deviation (P), for a distribution of 50% we will use 0.5
- Margin of error/confidence interval, this is how much higher/lower than the mean you are happy with e.g +/- 3%
We then can use a two step process for finding the true sample size.
For the numbers we used above this would give us a sample size of around 1067. We then take this value and adjust it for the population size we are trying to find statistics on. For example if we wanted to find a relevant population size for all of London, or around 10,000,000 people we would adjust using the formula below:
This would then give us a sample size of 10,660 for the population of London. The alarming things about this number is how few studies come anywhere near this threshold.
One side note to this however is that even with smaller samples if the results skew massively one way then they can still be useful indicators. For example if you randomly survey 50 people and 49 of them say they are pro choice then this does not seem like the kind of evidence you can disregard based on a small sample size.
Neil Hacker 