Margin of Error -- What Talking Heads Don't Know

This could apply in the benefits or compensation world. Why? Because we look at lots of data. And, when we look at data, we often do sampling, whether we realize it or not.

But, this is my turn to rant. Why? Because it's been a while since I last ranted.

It is mind-boggling to me that most of the talking heads (newscasters if you prefer) on TV have college degrees, or for that matter, high school diplomas. They don't understand basic math.

From one of them today: "Scott Walker is ahead in all the polls we are seeing, but they are all within the margin of error, so the race is too close to call."

Think about it. If you take one poll and the margin of error is, say, plus or minus 5% and Walker leads by 51.8 to 48.2, then that poll is within the margin of error.

Note that the margin of error is based on the sample size not the whim of the pollster. 

Now, suppose that you have 20 polls and Walker leads in each one and in each one, his lead is near the outer limits of the margin of error. What does that tell you?

Well, the sample size has grown. If the sample size in poll #1 is n1 and the sample size in poll 2 is n2, etc, then the total sample size assuming that no single person was surveyed twice among the twenty polls is n1 + n2 + n3 + ... + n20. Ostensibly the margin of error is inversely proportional to the square root of the sample size. Or, in lay terms, every time the sample size gets multiplied by 4, the margin of error gets cut in half.

So, for simplicity, if each of n1 through n20 is 192, then the margin of error is roughly 7%. But, if there are no duplicates in the 20 populations, then the total sample size of the 20 is 3840 leading to a margin of error in the vicinity of 1.5%. 

So, if Walker is ahead by 3% in each of the 20 small polls, then each poll shows that the race is within the margin of error. However, taken together, the race is well outside the margin of error and they are predicting that Walker will win.

Duh!