How to Understand Polling of Underrepresented Communities
With five days before the midterms, many of us are preparing to peruse exit poll data hoping to understand how or why Americans voted one way or another. But we’ll do so with caution since recent elections have reminded us of the limitations of exit polling—primarily problems with representative sampling (because we won’t know what the electorate’s demographics will be while people are voting) and non-response (the number of people who will simply decline to be interviewed). These issues are often more salient for underrepresented communities, as many of these voters are likely to feel uncomfortable speaking to an exit poll interviewer as they leave their voting station, which can skew the survey sample and the results.
After covering the key things to pay attention to when evaluating a new poll, we focus today’s post on polling of underrepresented communities (e.g., Muslims, Black, Latinx, Asian/Pacific Islander, etc.) because they pose particular challenges for pollsters. So how can you tell if a poll that makes claims about underrepresented communities is accurate and trustworthy?
THREE THINGS TO PAY ATTENTION TO
1. Sample size
As with any poll, the bigger the sample the better. But to make claims about underrepresented communities, pollsters must oversample all racial and linguistic groups that will be analyzed. That is, they must recruit enough members from populations of interest so that each group of underrepresented respondents makes up a larger share of the sample than they do in the overall population. For example, if Latinx make up approximately 20% of the US population, a survey of 1,000 American adults would recruit not 200 but 400-500 Latinx respondents (an oversample).
2. Sample composition
When pollsters oversample underrepresented communities, they can recruit enough respondents to build subsamples that are actually representative of the diversity of particular groups.
The best way to do this is by establishing sample quotas for demographics like age, race, and language. If a pollster wants to make claims about the Latinx population in the US, they must make sure their oversample includes respondents from all ages, racial backgrounds, different regions, different countries of origin, different immigrant generations, and speakers of both English and Spanish. Otherwise, they’re likely to overrepresent those who are easier to reach or more likely to answer surveys, which undermines the accuracy of the findings.
Election polls
In an effort to predict election outcomes, pollsters try to survey only the subset of adults who are most likely to vote (i.e., “likely voters”). But how do we know who will actually vote in a particular election? The answer is: we don’t. We can make very educated guesses, but they are always based on assumptions that could prove false.
Pollsters usually define “likely voters” by weighting a number of predictive characteristics like thought given to the election, partisan enthusiasm in a particular campaign, demographic characteristics, and especially past voting behavior. This makes sense since having voted in the past is one of the strongest predictors of whether someone will vote in the future. The problem is that sampling only “likely voters” leaves out all newly eligible voters (young adults or new citizens with no voting history) and often misses the mark with racial and ethnic minorities.
For example, according to the Current Population Survey, the percentage of Latinx Americans who voted in midterm elections hovered around 33% from 1998 to 2010 and then dropped to 27% in 2014. If a pollster is surveying only “likely voters” in 2018, they may leave out a substantial share of Latinx eligible voters who may very well turn out to vote. So, to poll underrepresented populations accurately, election polls must include more than likely voters.
3. Translation and Cultural Competence
One of the biggest challenges of polling underrepresented populations is that many people in immigrant communities do not speak English. Therefore, high-quality translation is fundamental for accurate findings. Pollsters must use bilingual callers and professionally translate survey questions and answers.
But even when speaking their own language, many people in underrepresented communities are not comfortable identifying themselves and giving honest answers over the phone. To overcome this, callers must establish trust and rapport with respondents, which requires cultural competence and asking a series of non-sensitive questions first.
DEMANDING BETTER POLLING
Since conventional methods and data systems were not designed with diverse populations in mind and the above best practices are more expensive to implement, polling of underrepresented communities is too often subpar. As advocates, we must hold researchers accountable for adjusting their practices.
When hiring a pollster:
- Ask for concrete plans and justifications for choices around data collection
- Ask for concrete experience, especially with polling underrepresented communities
- Ask for innovation in outreach and analysis
- Ask for oversampling and high-quality translation
