{"id":17115,"date":"2022-12-12T09:40:25","date_gmt":"2022-12-12T08:40:25","guid":{"rendered":"https:\/\/surveyinsights.org\/?p=17115"},"modified":"2022-12-15T08:27:00","modified_gmt":"2022-12-15T07:27:00","slug":"cost-efficiency-of-incentives-in-mature-probability-based-online-panels","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=17115","title":{"rendered":"Cost Efficiency of Incentives in Mature Probability-based Online Panels"},"content":{"rendered":"

Introduction<\/h1>\n

To keep response rates high in panel surveys, survey designers often rely on incentives. While there is a lot of evidence about effects of different incentives on response rates in cross-sectional surveys or in the first wave of a panel survey, we know surprisingly little about effects on response rates and costs of incentives in mature panel surveys. This is particularly the case in probability-based online panels on the general population. Trading-off the costs and the effects on response of incentives is even more relevant in longitudinal than in cross-sectional surveys: cumulative attrition leads to small and increasingly selective samples which make incentives necessary and \u2013 due to repeated use \u2013 expensive (Lynn 2009). In online surveys, incentives increase sample members\u2019 response in the first wave and also the likelihood to becoming panel members; see Witte et al. (2021). However, the cost efficiency of incentives in subsequent waves should decrease, because as time passes, factors explaining respondents’ likelihood of participation other than incentives such as intrinsic motivation should play an ever-greater role than incentives.<\/p>\n

In this paper, we analyze effects from an incentive experiment on response rates and costs of a probability-based online panel of Swiss adult citizens in wave 5 of an ongoing Panel Survey of the Swiss Election Study (Selects). Specifically, we use a conditional cash incentive (CHF 20) for those with a low response probability and randomize the remaining sample into a conditional cash incentive group (CHF 10) and a conditional lottery (5×300 CHF). Costs of the CHF 10 cash-incentive are about six times higher than those of the lottery-incentive. Given the maturity of our panel, we expect that while cost differences are substantial, the difference in attrition between the two groups is small and that the two designs do not lead to different sample compositions.<\/p>\n

Incentives, response rates, and cost efficiency in longitudinal surveys<\/h1>\n

The incentive used in the first wave of \u201ca particular longitudinal survey has long-term consequences for the survey both financially in terms of cost and the expectations of respondents\u201d (Laurie & Lynn 2009: 209). For example, incentives offered in the first wave may lead to expecting incentives in subsequent waves. While most of the current literature (see e.g., McGonagle 2020) indicates that positive effects of incentives offered at the start of a panel persist across waves, others do not find such carryover effects (Becker & Glauser 2018; Wong 2020). Because nonresponse is especially high (Lepkowski & Couper 2002; Watson 2003; Uhrig 2008) and selective (Lipps 2007; J\u00e4ckle & Lynn 2008) in the first waves and because prepaid cash often performs best (Warriner et al. 1996; Pforr et al. 2015; Pforr 2016; Lipps et al. 2019), a good strategy may be to use an (expensive) unconditional cash incentive rather than a (inexpensive) lottery incentive in the first wave (Witte et al. 2021).<\/p>\n

Regarding costs, Gajic and colleagues (2012) found in a cross-sectional survey of a general community population that while the prepaid cash incentive generated the highest response rates and no incentive generated the lowest, a lottery with few prizes was the most cost-effective incentive for obtaining completed interviews. The question is about which incentive is most efficient in subsequent waves: while the most efficient incentive with respect to response rates, prepaid cash, is also the most expensive alternative, the survey budget exerts constraints on what researchers can do in terms of survey designs (Olson et al. 2021). In mature panels, cost efficiency of expensive incentives may be lower since respondents who participate in several waves are more likely to be driven by other factors than incentives themselves (Pforr et al. 2015; Sacchi et al. 2018). For example, Pekari (2017) found that already after the first wave with an unconditional cash incentive, adding a cost-efficient lottery of iPads was just as effective as an expensive conditional cash-prize at reducing attrition for wave 2 and 3. Factors that are likely to become increasingly important for participation include a higher interest in the survey and a higher intrinsic motivation. For such motivated respondents using (inexpensive) conditional incentives in the following waves may be a sufficient motivator to continue to respond. While lotteries may work in later waves for respondents who did not need extra treatment (such as reminders) during the first waves, for more difficult respondents, it may be worth using more expensive incentives (Becker et al. 2019).<\/p>\n

Data and experimental design<\/h1>\n

The Swiss Election Study \u201cSelects\u201d investigates the determinants of turnout and vote choice in national parliamentary elections. We use a self-administered online Panel Survey (see Selects 2022) that was part of Selects 2019. The Panel Survey investigates the dynamics of opinion formation and vote intention during the election year and annually between two successive national elections. Six waves have been conducted so far, three in 2019 (one before the 2019 election campaign, one during the election campaign and one after the federal elections), and three subsequent yearly waves in 2020, 2021 and 2022.<\/p>\n

The initial sample was drawn by the Swiss Federal Statistical Office at random from the Swiss population register and comprised 25,575 Swiss citizens aged 18 and above. Sample members received a pre-notification letter one week before the beginning of the online survey together with an unconditional incentive of CHF 10 in the form of a postal check. In addition, respondents who participated in the waves 1, 2 and 3 were told that they could win one of five iPads. 7,939 valid interviews (>50% completed) were conducted in wave 1. All respondents from wave 1 were invited to take part in wave 2 (5,577 valid interviews; AAPOR2 response rate=68.4%) and in wave 3 (5,125 valid interviews; AAPOR2 response rate=64.9%). At the end of wave 3, 3,030 respondents gave consent to participate in yearly follow-up surveys until the next national elections (in 2023). Consenters were informed that they would receive CHF 10 (cash) upon completion of wave 4 of the online survey. In total, 2,499 valid interviews were conducted in wave 4 (out of 2,955 contacted persons; AAPOR2 response rate=84.6%).<\/p>\n

Consenters from wave 3 were again contacted in wave 5 of the study (2,878 contacted persons, excluding respondents no longer in the population (e.g., deceased) or who asked to be withdrawn from the sample. Those with a low political interest in wave 1 (not at all or rather not interested, N=536, versus rather or very interested, N=2,342) and those who systematically responded only after the second reminder were deemed low probability respondents (N=592). This group was offered a CHF 20 incentive and excluded from the experiment. The remaining sample (2,286 sample members) was randomized into two equally sized subsamples. The first subsample was informed that they would receive CHF 10 (cash) upon completion of the questionnaire. The second subsample was informed that they would participate in a cash lottery upon completion of the questionnaire (5×300 CHF). Note that a proper \u201ccontrol\u201d group receiving no treatment is not included in the experiment. To calculate valid responses, speeders (N=2), respondents reporting a sex or a birthyear inconsistent with information from the sampling frame (N=41), and drop-offs (N=20) counted as non-respondents.<\/p>\n

Using a logistic regression, we checked the randomization into the two experimental groups by regressing the two incentives on the variables political interest, having voted in national elections, education level and whether the questionnaire of wave 3 was interesting to answer. We imputed 4 missing values for political interest, 18 for voted, 2 for education level and 17 for interest in the wave 3 questionnaire using chained equations. Previous research shows that these variables, which are good proxies for social inclusion and participation, are generally negatively correlated with attrition (e.g., Voorpostel 2010). This is important for the test of selection effects in the next chapter. We also include gender, language region (Swiss-German, French, Italian), and age group (18-30, 31-43, 44-56, 57-69, 70+) in the regression model. The model turned out to be insignificant (Prob > chi2<\/sup> = 0.719), suggesting that the variables are not jointly different in the two experimental designs.<\/p>\n

Results<\/h1>\n

While the response rate of the full sample (N=2,878) amounts to 80.7%, this is 78.0% in the CHF 20 group (N=592), 82.9% in the CHF 10 group (N=1,143), and 80.0% in the lottery incentive group (N=1,143). Based on two-sided t-tests, the response rates are different on the 5% significance level between the CHF 20 group and the CHF 10 group (Pr(|T| > |t|) = 0.015; df=1733), on the 10% level between the CHF 10 group and the lottery group (Pr(|T| > |t|) = 0.076; df=2284), and on the 35% level between the CHF 20 group and the lottery group (Pr(|T| > |t|) = 0.348; df=1733). This means that there is no significant effect between the two experimental groups (CHF 10 and lottery) on a conventional significance level.<\/p>\n

To test respondent selection effects of the incentives in the two experimental groups, we regress completion in the two experimental groups on the variables used for the randomization check above. Using a logit model, these variables do not jointly interact with the experimental group on the 10% significance level (Prob > chi2<\/sup> = 0.271). We conclude that the respondent sample composition is the same in the two experimental groups. Only two variables interact significantly with incentive: gender (2% level) and age groups from 44 years on (5% level; baselevel 18-30 years old). We illustrate the interaction of these variables with incentive using marginal effects in Figure 1:<\/p>\n

Figure 1: Predictive Marginal Effects of completion by gender and age group interacted with incentive.<\/strong><\/p>\n

\"\"<\/a><\/p>\n

Note: Data: Wave 5 (Selects 2019 Panel Survey), N=2,286 sample members in the two experimental groups.<\/em><\/p>\n

Men (red) are more responsive to conditional cash than women (blue) and younger people (blue: 18-30 years old; red: 31-43 years old) are more sensitive to incentives, with conditional cash having positive effects on participation.<\/p>\n

While neither response rates nor respondent sample composition differ much between the two experimental groups, incentive costs do, however: the per respondent incentive costs excluding administrative costs (letters, etc.) amount to CHF 10 in the first group and to CHF 1.6 in the lottery group, making the latter incentive design much cheaper.<\/p>\n

Discussion<\/h1>\n

Comparing a conditional CHF 10 cash incentive with a lottery of 5×300 CHF in wave 5 of a probability-based online panel survey among Swiss adult citizens, we find that neither response rates (82.9% vs. 80.0%) nor sample composition are statistically significantly different from each other. Interestingly, the only variables for which different categories have different response rates according to the incentives used are age and gender. These socio-demographic variables are typically not prone to cause large differences in nonresponse, see Lipps and Pekari (2021), who find little effects in the 2015 edition of the Selects survey except for older people who are underrepresented in the web only design. Social inclusion and participation related variables, however, do not interact with the incentives. While it may be an idea to use conditional cash for men or younger sample members and a lottery for women or older sample members, overall, the sample composition does not differ much across the two experimental groups.<\/p>\n

The largest difference between the two incentives are costs: in our setup, conditional cash incentives cost about six times as much as the lottery. This suggests that \u2013 at least in a mature panel \u2013 it is possible to reduce incentive costs drastically without too much damaging the size and composition of the sample. Sample members invited (participants of wave 3) are rather driven by interest in the survey: while only 71% of the wave 3 respondents, who strongly or rather disagree with the statement that the questionnaire was interesting (N=112), participate in wave 5, this percentage increases to 79% for those who neither agree nor disagree (N=332), to 80% for those who rather agree (N=1594), and to 84% for those who strongly agree (N=823). This of course reflects an increasing selection of sample members to those who are interested in the survey.<\/p>\n

One limitation of this study is that we had to exclude low propensity respondents. These people \u2013 selected by means of their little political interest and late participation in previous waves \u2013 have different properties and were not part of the incentive experiment because not offering them an especially high incentive (CHF 20) would have been too high a risk of losing them. Since this led to a further (positive) sample selection in addition to already restricting the sample to the mature respondents of wave 3, the resulting sample was probably less sensitive to different incentives. Our results are therefore likely to be conservative. Nevertheless, we think that we took a pragmatic choice which didn\u2019t involve too high risks to lose low probability respondents. To sum up, in a mature panel of adult citizens, it is possible to save costs on expensive incentives without sacrificing response rates and sample composition too much.<\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Introduction To keep response rates high in panel surveys, survey designers often rely on incentives. While there is a lot of evidence about effects of different incentives on response rates in cross-sectional surveys or in the first wave of a panel survey, we know surprisingly little about effects on response rates and costs of incentives […]<\/p>\n","protected":false},"author":989,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[902,660,904,905,550,903],"class_list":["post-17115","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-costs","tag-incentives","tag-mature-panel","tag-online","tag-probability-sample","tag-response"],"acf":[],"_links":{"self":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/17115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/users\/989"}],"replies":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=17115"}],"version-history":[{"count":50,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/17115\/revisions"}],"predecessor-version":[{"id":17812,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/17115\/revisions\/17812"}],"wp:attachment":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=17115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=17115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=17115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}