September 24th, 2007

Why “Scientific” Polling isn’t Always Accurate

I came across the following post on Democratic Underground, where one user relates his experience with a research organization conducting a (no doubt “scientific”) poll:

My New Hampshire phone rings (caller ID 000-000-0000 WTF?) and I pick it up out of curiosity. “This is (talking like a magpie so I haven’t got a clue what the name of the company is)Research. Would you like to be part of a poll for the Democratic presidential primary candidate?” I agree and the woman rapidly reads the list of candidates. “Clinton, Obama, Edwards, Dodd, Biden and Richardson.” I say, “Kucinich.” The woman on the phone says “Who?” I say, “Kucinich. Dennis Kucinich.” I hear coaching in the background – apparently the call is being monitored. “I’m sorry, but Dennis Kucinich is not on our list. Is there anyone on the list I gave you who you would vote for?”

Media-anointed mainstream politicians are given the limelight time and again, while so-called “sideline” candidates, including Ron Paul, are unfairly treated. And then the “official”, “scientific” polling shows them with small numbers. If this user’s experience is reflective of what occurs with other research companies and polling inquiries, one wonders why the polls are to be trusted at all!

I think many people will be surprised come election time to see how disparate the results are from the predictions shown by polling.

15 Responses to “Why “Scientific” Polling isn’t Always Accurate”

  1. Brian D.
    September 24, 2007 at 3:56 pm #

    Nothing in politics is scientific. Especially not the polling!

  2. Dan
    September 24, 2007 at 4:56 pm #

    Connor,

    *Shakes head*

    The poll still ends up being “scientific” especially in comparison to the Internet polls you like to quote. The question in this particular incident may not have been a good question, seeing that they left out several candidates, but the methodology applied is still quite accurate.

    Take for example the latest straw poll done in Michigan. Based on what you would like for us to think, Connor, Ron Paul won the poll. But reality is that Ron Paul got only 10% of the votes in the straw poll. That’s a fairly good number for one so extreme as Mr. Paul, but gauging by this field of Republican candidates, anybody would do fairly well.

  3. Connor
    September 24, 2007 at 5:44 pm #

    Dan,

    How is it good science to completely disregard the stated opinion of the person you’re including in your survey? Forcing a person to choose between a list of provided choices to the exclusion of other viable ones is hardly the process of true scientific polling.

    Your second paragraph is amusing to me. In it, you presume to know “what [I] would like [you] to think”. How foolhardy to put words in my mouth as you have! And besides, why would I supposedly claim that he won the poll when he came in third…?

    And please stop shaking your head. It reeks of condescension.

  4. John
    September 24, 2007 at 6:05 pm #

    Dan,

    You’re kiddding, right?

    Let me ask you: Iraq war: Great war, or Greatest war?

  5. Russell Page
    September 24, 2007 at 6:47 pm #

    I don’t smell anything?

    Polling is not designed to seek opinion in political races. It is specifically designed to be a marketing tool. They are often and frequently designed to favor a certain candidate, and this tactic is not unique to party lines or certain candidates.

    Take a look at all the polls collected at PollingReport.com. They post results from polls all around the country. How is it that two “scientific” polls can elicit results that are as much as a 10-point difference from one poll to the next?

    The problem is that people have the impression (true or not) that the word science means trusted or exact when no two polls are using the same “science” or guidelines for measurement.

  6. Dan
    September 24, 2007 at 8:09 pm #

    Connor,

    The reason I shake my head is because you really know very little about polling. You cite one example of a bad poll question and say, see, scientific polling ain’t that scientific. As with everything else, however, there is a science to polling. It must actually be done properly. In regards to this particular question, indeed the question was poor because it did not account for all of the declared (and even undeclared) candidates. This particular incident does not in any way refute fairly accurate and consistent polling which places Mr. Paul fairly low on the Republican totem pole, as much as you would like it to be different.

    Please, I implore you, before discussing polling again, please take PoliSci 300 at BYU. :)

  7. Connor
    September 24, 2007 at 8:22 pm #

    The reason I shake my head is because you really know very little about polling. You cite one example of a bad poll question and say, see, scientific polling ain’t that scientific.

    Did I ever say that “scientific” polling in general isn’t scientific? No. Please read the title once again. I blogged about this instance to show why it’s not always accurate. Again, you’re misreading my statement and putting words in my mouth.

    This particular incident does not in any way refute fairly accurate and consistent polling which places Mr. Paul fairly low on the Republican totem pole, as much as you would like it to be different.

    There are other factors which influence the “scientific” polls you love to cling to. Russell cited a great example of their varying results showing how inaccurate they can be. They also rely on increasingly antiquated methods to poll the citizenry: land-line telephones. Given the rising number of people who don’t use one, one might automatically assume that the polls don’t reflect the portion of the citizenry that opts to use cell phones instead of landlines.

    While certain polls may be “scientific” (and that’s debatable), they aren’t always reflective of the American people as a whole, and as such should not be heavily relied upon—especially when other poll results exist (whether “scientific” or not) that heavily contradict what they say.

  8. Brandon
    September 25, 2007 at 12:25 am #

    Dan,

    One thing I can’t understand about the “scientific” polls is that they have Rudy Giulliani leading. I know a lot of Republicans and I have not met a single one that said they would vote for Rudy. I know that isn’t scientific, but to me it seems wierd that a guy could be leading the polls and have virtually no support on the ground that I can see. The same holds true for the democrats. Obama, Kuchinich and Gravel are the only candidates I have heard anyone talk about and yet Billary has a commanding lead. I don’t buy it.

  9. Dan
    September 25, 2007 at 4:25 am #

    Brandon,

    The really good polls try to get as random of a sampling as possible from all across the nation. You can’t for example do a poll in just Texas to see what all of America thinks about George Bush.

    You don’t need to ask every single person in a survey group to gauge what they think. Say you do a survey at your local college of 1000 students and you want to know what they think of a particular policy. You don’t need to get all their views to know what the general view is. You ask about 5 or 10% of the group randomly and you really do get a good idea of just what the whole campus thinks, plus or minus 3-5% (and that number there tells you that the poll is not meant to be exact, but rather an idea).

    In regards to national polls, one poll is not enough to tell you how Americans really think, because we can never truly get 5% or 10% of the whole American population surveyed at once. So you get multiple polls, and generally speaking they tend to be fairly accurate. Nearly 70% of Americans really do think that Bush is doing a bad job as president.

    In regards to those you know, Brandon, who don’t support Giuliani or Obama, or Kuchinich, you have to remember that you are generally speaking in a particular type of group of Americans. I’m surrounded by Mormons a lot (online and in person), and they tend to talk about Mitt Romney far more than the rest of America. If you can do an actual 10% random sampling of Americans, you’ll find that a very large number of them have never heard of Mitt Romney. It is one reason why he suffers greatly at many polls in various states against Democrats (some polls have Hilary winning against Romney by nearly 60% to 30% for example—where if you put Giuliani’s name in there, because he is more well known, Hilary only wins by 52% to 45%). A lot of the polling and election actions really are like a high school student body president election, where name recognition means more than actual substance of policy. This is why sloganeering is so popular, creating the sound byte that one could associate with that particular candidate. So for example Mitt Romney says he wants to “double Guantanamo”, well that phrase is now tied to him. Remember Bush in 2000? Remember “compassionate conservatism?” Sure it didn’t actually reflect any real compassionate conservatism (for one it is a contradiction in terms) policy, but it sounds great and everybody tied it to Bush. You just ain’t gonna have the Douglas-Lincoln debates anymore. Those are relics of an era long past.

  10. Dan
    September 25, 2007 at 4:28 am #

    Connor,

    I blogged about this instance to show why it’s not always accurate.

    I guess it was assumed that we all knew that not all polls were scientific. Heck, FauxNews puts polls all the time with the basic gist being, “Do you want America to win in Iraq?” Who would answer “no” to that?!?!?!

    They also rely on increasingly antiquated methods to poll the citizenry: land-line telephones. Given the rising number of people who don’t use one, one might automatically assume that the polls don’t reflect the portion of the citizenry that opts to use cell phones instead of landlines.

    The landline issue is a good one to ponder on for pollsters. However, given that Ron Paul only got 9% in the Iowa straw poll and now only 10% in the Michigan straw poll, well, it says that the issue of landline doesn’t really affect the overall number that greatly.

  11. Connor
    September 25, 2007 at 7:40 am #

    However, given that Ron Paul only got 9% in the Iowa straw poll and now only 10% in the Michigan straw poll, well, it says that the issue of landline doesn’t really affect the overall number that greatly.

    You might want to view the results of all the straw polls, where Rep. Paul ffeatures such percentages as 33, 65, 45, 81, 72, among the nines and tens you love to feature.

  12. Travis
    September 27, 2007 at 1:57 am #

    I have to agree with Dan. He knows what he is talking about. Sure semantically Connor is correct. However, you are making another classic reporters mistake and saying something to make it look like it is more than it is.

    You should really take at least a Stats 100 level class and you will get the big pictures. I agree the question wasn’t a great question. But that doesn’t make all statistics bad (political or not).

  13. Connor
    January 3, 2008 at 1:54 pm #

    Here’s another good example of the fallacy of “scientific” polling.

  14. Connor
    March 10, 2008 at 12:32 pm #

    I thought that this comic related well to this post:

  15. Dan
    March 11, 2008 at 3:11 am #

    Ah, let’s revisit polling. It sure seems that polling was accurate about Ron Paul’s actual support level.

Leave a Reply

Leave your opinion here. Please be nice. Your Email address will be kept private.