Friends

How to Understand Polling of Underrepresented Communities (June 2020 Update)

Eva Galanes-Rosenbaum
June 17, 2020

Following the murders of George Floyd, Ahmaud Arbery, Breonna Taylor, Tony McDade, and other Black Americans, hundreds of thousands have been agitating for overdue change in the United States and around the world. Unlike prior moments of protest around police brutality and racial injustice in recent memory, public opinion appears to be shifting very quickly in favor of equality and the Movement for Black Lives.

We have noticed a real irony in all of this: many of the recent polls about racial justice—even those about personal experiences with police and discrimination—do not include representative samples of Americans of color. This isn’t new. People of color and other minorities (including Muslim, Black, Latinx, Asian American/Pacific Islander, Native American/American Indian, young adults, and many others) are routinely underrepresented in polling because these demographic groups are less likely to participate in research and, thus, more expensive to include.

So when you’re reading a poll, how can you tell if a poll that makes claims about underrepresented communities is accurate and trustworthy? Today, we have updated our post from October 2018 about what to look for.

THREE THINGS TO PAY ATTENTION TO

1. Sample size

As with any poll, the bigger the sample the better. But to make claims about underrepresented communities, pollsters must oversample all racial and linguistic groups that will be analyzed. That is, they must recruit enough members from populations of interest so that each group of underrepresented respondents makes up a larger share of the sample than they do in the overall population. For example, if Latinx make up approximately 20% of the US population, a survey of 1,000 American adults would recruit not 200 but 400-500 Latinx respondents (an oversample).      

2.  Sample composition

When pollsters oversample underrepresented communities, they can recruit enough respondents to build subsamples that are actually representative of the diversity of particular groups.

The best way to do this is by establishing sample quotas for demographics like age, race, and language. If a pollster wants to make claims about the Latinx population in the US, they must make sure their oversample includes respondents from all ages, racial backgrounds, different regions, different countries of origin, different immigrant generations, and speakers of both English and Spanish. Otherwise, they’re likely to overrepresent those who are easier to reach or more likely to answer surveys, which undermines the accuracy of the findings.

Election polls

In an effort to predict election outcomes, pollsters try to survey only the subset of adults who are  most likely to vote (i.e., “likely voters”). But how do we know who will actually vote in a particular election? The answer is: we don’t. We can make very educated guesses, but they are always based on assumptions that could prove false. 

Pollsters usually define “likely voters” by weighting a number of predictive characteristics like thought given to the election, partisan enthusiasm in a particular campaign, demographic characteristics, and especially past voting behavior. This makes sense since having voted in the past is one of the strongest predictors of whether someone will vote in the future. The problem is that sampling only “likely voters” leaves out all newly eligible voters (young adults, new citizens with no voting history, or people whose voting rights have been restored) and often misses the mark with racial and ethnic minorities. 

For example, according to the Current Population Survey, the percentage of Latinx Americans who voted in midterm elections hovered around 33% from 1998 to 2010 and then dropped to 27% in 2014. If a pollster is surveying only “likely voters” in 2020, they may leave out a substantial share of Latinx eligible voters who may very well turn out to vote. So, to poll underrepresented populations accurately, election polls must include more than likely voters.

3.  Translation and Cultural Competence

One of the biggest challenges of polling underrepresented populations is that many people in immigrant communities do not speak English. Therefore, high-quality translation is fundamental for accurate findings. Pollsters must use bilingual callers and professionally translate survey questions and answers. 

But even when speaking their own language, many people in underrepresented communities are not comfortable identifying themselves and giving honest answers over the phone. To overcome this, callers must establish trust and rapport with respondents, which requires cultural competence and asking a series of non-sensitive questions first.

DEMANDING BETTER POLLING   

Since conventional methods and data systems were not designed with diverse populations in mind and the above best practices are more expensive to implement, polling of underrepresented communities is too often subpar. As advocates, we must hold researchers accountable for adjusting their practices. 

  When hiring a pollster

  • Ask for concrete plans and justifications for choices around data collection.
  • Ask for concrete experience, especially with polling underrepresented communities.
  • Ask for innovation in outreach and analysis.
  • Ask for oversampling and high-quality translation.  

 Additional Resources