We all know it’s true – or at least we suspected it.
A person who sits at home, answering lengthy skip-logic surveys for $5 (or less) is a different cat. Plus, there are the folks who sign up for a bunch of different panels – the survey gig workers – who look even different still.
The most respected panel companies account for the demographic skews of these paid survey takers through basic reweighting or quota sampling. That’s easy.
But how do you account for the less visible systemic biases, potentially true of every gig panelist? What’s the census weighting factor for “Really, really loves taking long surveys” or “Has ample time and motivation to answer questions ad nauseum for store rewards?”
CivicScience may not be able to answer those questions – but we work diligently to understand how these systemic biases impact other psychographic attributes of respondents and, thereby, the sometimes-wonky signals garnered from panel-based research.
Survey Panelists and the Real World
Among the over 350,000 questions in our database is a profile question asking our unpaid respondents whether they participate in one or more survey panels for compensation. In total, over 1 million U.S. adults have answered this question, with 9% saying they belong to one survey panel, 5% who say they belong to more than one, and 12% saying they used to belong to a panel but no longer do. Seventy-four percent of our respondents say they never have.
While we can’t be certain, it’s likely that the 14% incidence rate of current paid survey panelists in our sample is higher than reality. One can presume that a person who’s inclined to answer a voluntary, intrinsically-rewarding CivicScience poll would over-index as a habitual survey taker. In other words, the percentage of U.S. adults who engage in a paid survey panel is likely somewhat – if not considerably – lower than 14%.
Although our methodology may not be perfect (but less imperfect than all others), it uniquely allows us to study the differences between the small segment of Americans who regularly answer paid surveys and the 74% who never have. CivicScience clients can cross-tabulate billions of responses in our database against our survey panel question. To say the least, the differences are substantial.
At a high level, we know that paid panelists are significantly more brand-aware and brand-opinionated across the board. They have different media consumption habits, are more likely to use coupons, and more likely to eat from a value menu at a QSR.
The Rosy Economic Views of Gig Survey Takers
Perhaps nowhere, however, are the systemic biases of paid survey panelists more evident and more permeating than when we look at their economic attitudes and financial outlook.
One of our more powerful and popular products at CivicScience is our Consumer Financial Health Index, which tracks a variety of crucial consumer attitudes about the economy, their financial health, spending behavior, and lots of other factors on a daily basis. When comparing these indicators through the lens of survey panelists versus non-panelists, an appreciable and consistent effect is clear.
Across nearly all the questions in our index, gig survey takers are significantly more financially positive and optimistic than the average bear. I guess if you’re not too busy to answer lengthy surveys for a couple bucks, perhaps your life is pretty chill.
On some level, these biases are manageable for survey panel companies, so long as they’re consistent. Insurmountable, however, is how these psychographic skews of paid panelists, meaningfully affect other factors, like their brand preferences, shopping behavior, media habits, or causes they care about. It affects how price sensitive they might (or might not) be, whether they’re attracted to certain promotions, or what retailers they frequent.
When I wrote about these economic sentiment disparities in my weekly “What We’re Seeing” email recently, one of our readers replied with the following:
“Hi John! The data re: compensated panelists’ different opinions is eye-opening!
When I ran [a well-known market research company’s] panel business this data would have killed us! Glad I didn't know about your charts then and didn't get asked about it
Even though we knew it was true - we never talked about it and the other deep dark secrets of online panels.”
The Case FOR Survey Panels
To be clear, paid survey panels can still serve very valuable purposes that CivicScience cannot. If your research objective requires a longer qualitative conversation with consumers or viewing of rich media, you will need to pay participants for their time. Or, if you have a custom panel of your most loyal and motivated customers, there’s significant benefit in studying them as deeply as you can.
Just understand the tradeoff you’re making and the potential blind-spots you might have about the larger, more representative universe of consumers. Because they matter.
Popular Posts
