First Day of School

Kenton White | November 10, 2020  | Methodology, Politics | 2 Comments

“You’re being too hard on Polly!”

I’ve heard this a lot since the election.  Some think that forecasting Biden winning makes Polly right.  Others say that Polly got all the states right except for Texas and Florida.  Some argue that the final results were within her confidence interval and that other pollsters are using that same argument as justification that they were right.  Still Polly got the election wrong and here is why.

Confidence interval isn’t meant to be a hedge, a statistical tool that includes many outcomes.  Confidence is meant to exclude possibilities.  Sir Ronald Aylmer Fisher, the father of modern statistics, asserted that confidence could never prove an outcome but could possibly disprove an outcome.  In Physics, the discipline I trained in, we try to make our confidence interval small enough that it excludes other possibilities.  If our confidence interval includes a competing outcome, then there is no forecast.

Monday Pfizer announced a COVID-19 vaccine that was effective with a 90% confidence.  This doesn’t prove that the vaccine is 90% effective, which is why Pfizer added the note “as the study continues, the final vaccine efficacy percentage may vary.”  They have proved that they can exclude with a 90% confidence other reasons that their study group is showing immune response to the virus.

I’m very happy Polly’s confidence interval eliminated Trump winning the election.  She called a Biden win with 98% confidence.  That is a forecast with no wriggle room.  With that confidence, she was excluding a Trump win while not proving that Biden would win, just that the data she was seeing was consistent with a Biden victory.  Had Trump won, that would be strong evidence that Polly’s data was incorrect.  

And Polly’s forecast went further than calling a Biden win.  That confidence excluded a squeaker.  If Polly was right, the election would have been called late Tuesday night, not 3 days later.  This is the specificity we hold ourselves to.  If we are being too hard on Polly — too hard on ourselves — so be it.  Innovation doesn’t come easy.

So what is next?

When I woke Wednesday morning, all of the swing states were red.  Not just Florida and Texas.  Michigan, Wisconsin, Pennsylvania and Georgia.  Even Arizona and Nevada looked like they could turn red.  This was not a good day for me or for Polly.  Tuesday evening I could posit that we missed the Cuban vote in Florida.  Wednesday morning I couldn’t blame the Wisconsin and Michigan Cubans for Polly’s miss!

As the election played out, all of those swing states went how Polly said they would go.  This didn’t make Polly less wrong.  It did give me confidence that Polly’s miss was confined to Florida and Texas.  And it wasn’t that Polly somehow missed the Hispanic vote — she accurately gauged the Hispanic vote in Arizona, Nevada and New Mexico.  Here is what happened.

It was a red sock in white laundry.

We’ve all missed our child’s red sock when sorting our clothes.  That one red sock gets in with the other white socks and white t-shirts.  Now that one little sock shouldn’t pose a big problem.  If it got in with my black socks and shirts it wouldn’t.  But it got in with the whites and turns all my clothes pink.  That one red sock has a larger impact than it should and over time I’ve learned to be hypervigilant about red socks in my white laundry. 

As Polly becomes more accurate, we can see in greater detail where things go wrong.  She is very good now at gauging the majority of people.  And the better she gets at assessing the majority, the more we’ve seen challenges arising from small groups with outsized impact.  In Canada, this is the Green Party.  In the US election, it was minority voters in Florida and Texas.  

Over the next few days Polly is looking for similarities between her misses in Canadian elections and her misses in the US election.  She is learning to find red socks in white laundry. 

This quote from T.S. Eliot’s “Murder in the Cathedral”, which has guided my scientific life, applies here:  “The last act is the greatest treason.  To do the right deed for the wrong reason.”|

Photo by Debby Hudson on Unsplash

About these results

Submit a comment

Your email address will not be published. Required fields are marked *

2 thoughts on “First Day of School

    1. Polly won’t be forecasting the Georgie runoff election. Here is the answer from Shainen Davidson, Senior AI Scientist:

      “The quality of available polling isn’t sufficient to train on directly. I looked a bit into using transfer learning from other elections as training, but because this one is a special election it doesn’t map well onto other election data.”

Comments are closed.

How reliable is your market research?

asiadmin | August 26, 2019


Where did the change happen? or, The “P” in Polly stands for Probability

shainen | October 27, 2020


Share This