Hats off to Nate Silver: The New Hari Seldon

The American people and our future were the big winners last night.  But in terms of individuals, the biggest winner of last night's election after Barack Obama was Nate Silver.

His New York Times blog and its statistical analysis predicted the winner, to a tee.   Understandably, Republicans attacked his work and his objectivity with increasing intensity as we approached the election.  Understandable and not surprising, not only because Republicans wanted their man to win, but because this is the Party that has denied evolution and global warming.  When you deny such scientific realities as those, denying statistics is small potatoes.

There have great historical errors in statics, from the Literary Digest poll that predicted an FDR loss in the 1936 to polls that predicted Obama over Hillary Clinton in the 2008 New Hampshire Primary.  But polling has learned from those errors, and has gotten the results right the vast majority of times.

Could Silver's analysis have turned out wrong last night?  Of course it could have - polls are always especially vulnerable when they're based on public testimony - how did you vote, what did you watch - rather than measurements of real actions.   But Silver got it right last night.

The result is not only a memorable victory for the American people, but for the practice of polling.   We should never blindly accept it or any scientific theory or procedure, but Nate Silver has ensured that there will be a little less hooting against polling when it's not going a given party's way next time around.

My favorite science fiction writer, as many of you know, is Isaac Asimov.  And my favorite work of his is the Foundation trilogy - which tells the story of Hari Seldom, and his psychohistory which, through statistical analysis,  could mathematically predict  the future - more or less.   Nate Silver may be the Hari Seldon of our time.

The Polls and Surveys Won Big Last Night, Too

Not only Barack Obama and the American people won big last night - so did the polls that predicted Obama would win the Presidency.

They weren't perfect - some of the polls saw Obama winning the popular vote by double digits - but they got the big victory right. Obama won by impressive numbers in Pennsylvania, just as the polls predicted. And he won in Florida, Ohio, Colorado, Iowa, Nevada - so-called "red states" - just as the polls predicted. I suspect these states won't be red again for a long time. Indeed, I wonder how much of a future the Republican Party, as it currently is, will have in any case...

I wonder what the polls will say in the weeks, months, and years ahead about that.

They were wrong, big time, in their prediction that FDR would lose in 1936, and Truman would lose in 1948. They got it wrong in the Democratic Primary in New Hampshire earlier this year.

But the polls were right, big time, about the revolution that has brought America up to par with its greatest dreams and highest ideals last night.

Making Sure Polls on Election Day = Polls Where Obama Is Today

As we move into the last 10 days before the most important election of our time, it is well to bear in mind the world of difference between two processes that have the same name: polls (as in surveys about how people will be voting) and polls (the places in which people actually cast their votes.

Surveys are useful tools, and have been generally accurate over the years. But they are not infallible, indeed have been wrong a few crucial times in our history, and in plain fact elect no one. The reasons poll-surveys have been in error almost don't matter. But they range from deliberately or accidentally biased samples - the people questioned are not representative of the general population - to people just not telling the truth when asked about their voting intentions. Infamous examples of polls being wrong range from the 1936 Literary Digest poll predicting Alf Landon's victory over FDR in that election to polls for the New Hampshire Democratic primary earlier this year which showed Obama winning comfortably. Pollsters learn from their mistakes. But there is no way to learn from a mistake you do not realize you are making - until the actual vote at the polls proves your poll-survey wrong.

Just to be clear - most polls, the vast majority of them, have been correct in their prediction. But the stakes are so high in this election, that those of us who support Obama can't gamble, even with the odds so heavily in our favor, that the current polls showing Obama leading in so many states are right.

Fortunately, we do not have to gamble. We can make sure we get to the polls - the place where we cast our votes - on Election Day. We can encourage, cajole, pressure our friends and family to go to the polls on Tuesday, November 4 - or, any day before the election, if your state has early voting. You can get an account on mybarackobama.com, and get a list of phone numbers of voters that you can start calling any time, when you have a few minutes, with encouragement to vote for Obama.

We shouldn't think for a minute that we're voting on an equal playing field. Not only is it possible that the current polls may not be accurate, that voters might change their minds between now and Election Day, but Republicans are doing their best to discourage and suppress Democratic voters. In Alabama, for example, citizens with parking tickets and no criminal record whatsoever were sent notices that they would not be able to vote. Robert F. Kennedy, Jr. and Greg Palast have also written an important article in Rolling Stone about the Republican attempt to stifle democracy and freedom in this election.

We all have it in our power to not let this happen. It's as easy voting and getting everyone we know to polls. Let's prove the pollsters right this time, and make the current polls showing Obama handily winning to be in complete 100% agreement with what happens at the polls on Election Day.

The Problems with Polls: A Brief Primer

In view of the inaccuracy of the poll projections in the Democratic Presidential Primary in New Hampshire on Tuesday - the polls showed Obama winning by 10 or more percentage points, in contrast to his actually losing by three - I thought I'd don my hat as a professor (I just finished teaching a graduate course in Media Research Methods at Fordham University this Fall), and offer a little primer on why poll projections can and do go wrong.

To begin with, New Hampshire this Tuesday is hardly the first time that a poll's predictions have been wrong.

The most famous example is the Literary Digest Poll of 1936, which predicted Alf Landon beating President Franklin Delano Roosevelt, 57 to 43 percent, a near landslide. In fact, FDR went on to win 62 percent of the vote.

What was wrong with the poll?

The people polled were drawn from lists of Americans who owned automobiles and had telephones in their homes - and, in 1936, cars and phones were much more likely to be in the hands of the rich than the poor or even the middle class. The sample, in other words, was biased towards upper income respondents, who had no love for Roosevelt.

That was the last time that kind of error was made in polling. But the prediction of human behavior, or attempts to measure it based on what people say they did, is still vulnerable to all kinds of errors.

Here are some perennials pitfalls of polling - with solutions or precautions, if possible:

1. People polled before voting can either be lying, or change their mind. Remedy: none.

2. People polled after voting can be lying (since they already voted, they can't change their mind). Remedy: none.

Strategies, however, which might or might not help with lying include interviewing people separately from their spouses. Of course, that creates a new problem of people not willing to participate, or answer questions:

3. People refuse to answer the pollster. Remedy: none.

Now a courteous pollster might have more success than an abrasive pollster, but no one can force an uncooperative person to participate in a poll. The best that surveyers can do is report the number of people who refuse to participate. If the percentage gets too high, this can invalidate the results. For example, if 10 people are polled, and 5 refuse to answer, 4 answer "a" and 1 answers "b" - what does this tell you? Not much, because who knows how the 5 who refused would have answered.

This problem can be aggravated by pollsters who under-report non-participants - because they think (perhaps correctly) that a high number of non-participants makes them (the pollsters) look bad.

And, to make matters even worse, in addition to the above problems, every poll faces the problem of getting a genuinely random sample to answer the questions. The 1936 poll wasn't random - it was biased towards the rich. We see that now. But what kind of biases are afoot that we are not aware of in our current poll?

In sum: polls do succeed, most of the time.

But the above problems are formidable and in principle intractable. Which means that, forever and anon, we are wise to take all poll results with a nice, big, sparkling grain of salt.