How to Find Reliable Opinion Polls

Avoiding Unreliable Polls

Stay away from polls sponsored by partisan organizations. Whether it be a party you identify with or a party that you oppose, partisan polls should be met with more skepticism than their nonpartisan counterparts. Partisan polls include those done by PACs, Super PACs, and Hybrid PACs, although some polling organizations have different guidelines regarding what they consider partisan. The ulterior motive of partisan groups and media outlets is to make their side look good. This may be via avoiding covering poll results that are bad for their candidate or withholding details (such as the dates of the poll). On average, "internal polls" often show results that are 4 to 5 percentage points more favorable to their sponsor than is actually the case. Major news outlets may also exaggerate results of their own polls for views. Consider if the "shock poll" in the headline is actually shocking, or if it's simply trying to elicit an emotional response. Resources like ABC's FiveThirtyEight, which covers polling for many major elections within the US, specify on their lists of polls which are partisan-affiliated.

Be critical of the poll's content. Analyze the specifics. Was the poll opt-in or did it randomly select its participants? Did the poll sample participants from all major parties? Is it transparent regarding how it was conducted? If any information concerning methodology, sponsorship, sample size, or poll questions seems dubious, it's best to take it with a grain of salt. It's important to look at the margin of error for the given poll. Keeping the margin of error in mind can help weed out sampling errors or leads that may be too close to call. As a general rule, a national survey should include around 1000-1500 participants. Any number significantly fewer than that will not provide a sufficiently representative sample of the population. Alongside random sampling, quota sampling is another representative method of sampling. Here, researchers set specific quotas for certain characteristics and then seek respondents who fit these quotas to participate in the survey.

Analyze the poll's questionnaire. Oftentimes, low-quality pollsters will skew their questionnaires to be more favorable to a certain response. Avoid polls that seem like their questions are biased or otherwise may be influencing the respondent to give a specific answer, as they may not be reflective of the actual public opinion. Be aware of when the poll was conducted, as the results may be taken after a major event has swayed public opinion on a candidate. If you'd like, you can compare the questionnaire in a poll to a poll that has been relatively accurate in the past. This way, you have a baseline for what kinds of questions are better suited for opinion polls. Stay away from polls that don't release their questionnaires. Not only is the pollster not being clear, but they may also be insincere about other aspects of the poll. Transparency is key.

Avoid polls with extreme sample sizes. Polls with fewer than 400 participants may not be representative, but bigger sample sizes aren't necessarily better. Oftentimes, huge polls are actually using unreliable polling methods, and wind up oversampling certain demographics, which introduces error. The "sweet spot" for polls is often about 1,000 to 1,200. It's just enough people to be representative of a population, but not enough people to be overzealous and risk sampling error.

Be conscious of different types of bias. Aside from sampling and questionnaire bias, there are other types, such as nonresponse bias, survivorship bias, acquiescence bias, primacy bias, and conformity bias. These are all different types of bias common in surveys, and opinion polls are not exempt from these. It is important to pay attention to the poll's methodology statement and consider these types of bias. Nonresponse bias is a type of bias where the response rate is abnormally low, thus skewing the results. Survivorship bias involves only receiving responses from people loyal to the company conducting the survey, which can cause the poll to shift in the direction of the types of people who are more likely to participate in polls. Think: if a poll is conducted by a news organization, the poll may be skewed in the direction that the news tends to be biased toward. Acquiescence bias is a type of bias that results from the desire to say "yes" to seem polite. People are more likely to vote in a manner that reflects their beliefs when they are voting than when responding to a poll. Primacy bias is a type of bias that, much like questionnaire bias, stems from the content being asked in the poll's questionnaire. However, it results in a tendency towards choosing the last answer choice due to it being fresher in the respondent's mind. Conformity bias is a type of bias where the respondent chooses their answers in a manner that they deem to be more "socially acceptable," affecting the outcome of the survey as a result. Regardless of confidentiality (meaning this type of bias can also affect elections), people may feel pressured to misrepresent their opinions out of fear of social repercussions.

Be wary of opt-in polls. Polls can only represent the opinions of a population when everyone in that population has an equal chance to take it. Opt-in polls usually attract people who feel passionately about the subject of the poll, rather than a representative sample. Extra information (such as gender and age) that would allow a judgment to be made about the nature of the sample is rarely collected by such polls, and so they should never be interpreted as a representative survey.

Vet exit polls carefully. While they are an excellent source when properly conducted, being the only chance to survey only voters, they can be skewed by three main sources of error: Differential non-response, where voters for a particular party or candidate are more likely to disclose their support than others. If the sample of polling stations is not representative. Even with a large sample, the overall results could be skewed if the political balance of those stations is not representative of the entire electorate. People accidentally placing the wrong vote, spoiling their ballot paper, or not filling out the paper correctly. Although it sounds unlikely, it is thought to be responsible for historically skewed exit polls.

Finding High-quality Polls

Be mindful of what kind of polls you're reading. Polls are sorted into 3 general categories— General Population (GP), Registered Voters (RV), and Likely Voters (LV); from most to least broad. Excluding non-voters from polls prior to an election tends to draw more accurate conclusions, as those excluded will have no impact on the final election result. However, GP polls are more common during non-election years. LV polls are generally considered the most reliable. This is because they exclude voters who are registered, but may be conflicted or otherwise against voting in the upcoming election cycle.

Check pollster ratings. Sites like ABC's FiveThirtyEight and Nate Silver's Silver Bulletin have ratings of different pollsters based on their performance in past election cycles. If a certain pollster has a higher rating, it is more likely to be of a higher quality. Pay attention to the last time pollster ratings were updated, as their reliability can shift over time depending on what is happening in the race.

Take note of trends in data over time. If a poll has done multiple surveys for a specific election, taking note of the given trend line can help see who is ahead in the race. For example, if a candidate is polling ahead but by a less significant margin than seen in previous polls, that may have more to say about the outcome of the election. If the trend line doesn't change at all over time for one poll compared to others, it may have more to say about the pollster's methodology than the actual outcomes of the election.

Look for polls sponsored by major news organizations. Some of the most reliable pollsters in the US are those sponsored by major news networks, like CNN, Fox, NBC, and CBS. They run all kinds of polls, including favorability, election results, and key issues surrounding elections. News organizations also frequently partner with research organizations to help produce better-quality polls; for example, Reuters/Ipsos and YouGov/The Economist. It is also important to check the reliability of both the news organization's polls and the research organization's polls to ensure that one of the pollsters isn't forking the overall results in a certain direction.

Pay attention to how weighting works in polls. Weighting is the act of adjusting the results to give less weight to certain respondents and more to others. While this may sound like an act of censorship, it's quite the opposite! This process corrects for demographic imbalances and makes sure the sample more accurately reflects the broader population. For example, non-college-educated voters are known to be harder to reach for polling, so their voices are given a boost in polls to put them on-par with the rest of the respondents. For example, if 48% of Americans have some sort of college degree, the remaining 52% should be non-college-educated. If only 26% of the respondents are non-college-educated, their responses need to hold twice the weight of a college-educated person's. Making sure responses are actually weighed correctly may take a bit of math. It helps to have a calculator on hand if you really want to be precise.