State of research on incentives in surveys<\/h1>\n
In general, incentives have been found to increase response rates in a wide range of surveys (Church, 1993; Kephart & Bressler, 1958; Lipps et al., 2022; Ryu et al., 2006; Shuttleworth, 1931; Singer & Ye, 2013; Smith et al., 2019; Wolf et al., 2021). Incentives are given in various forms, such as a prize draw (Porter & Whitcomb, 2003; Singer & Ye, 2013; Warriner et al., 1996), contributions to charities (Warriner et al., 1996), vouchers, lottery tickets (Budowski & Scherpenzeel, 2005; Church, 1993), or small gifts such as pens, stamps, or postcards (Wetzels et al., 2008; White et al., 2005). What has proven most effective in increasing response rates, however, is cash (Kephart & Bressler, 1958; Lipps et al., 2023; Lipps et al., 2022). Another differentiation concerns pre-and postpaid incentives (Church, 1993; Wolf et al., 2021): It has been established that unconditional, prepaid incentives are more effective in raising response rates than promised incentives or lotteries (Pforr et al., 2015; Singer & Ye, 2013). Overall, the payment of cash before the interview has the strongest positive effect on participation, even though the amounts of money are typically fairly small (Hsu et al., 2017). Occasionally, prepaid cash incentives may cause problems, as some respondents are mistrustful concerning the seriousness of an agency that sends unsolicited cash (e.g. B\u00f6rsch-Supan et al., 2013), but the positive effects far outweigh the negative ones.<\/p>\n
There are several theoretical accounts that provide explanations for why incentives motivate respondents to take part in surveys. One prominent explanation is reciprocity: respondents who receive money or a gift feel obliged to return the favor by responding to the survey (Gouldner, 1960). In a similar vein, Dillman (1978) argues \u2013 with recourse to social exchange theory \u2013 that by offering a (prepaid) incentive, survey organizations demonstrate trust that a potential respondent will comply with the request, thus stimulating cooperation. The leverage-saliency theory of survey participation (Groves et al., 2000) states that different aspects of a survey (such as the topic, organization, or incentive) may appeal to different respondents. Therefore, incentives may be particularly effective in low-income households, or to motivate survey respondents who are not interested in the topic of the survey, lack altruistic motives, or have alternative obligations (Singer, Groves, & Corning, 1999).<\/p>\n
In contrast to these theoretical expectations, Eyerman et al. (2005) and Zagorsky and Rhoton (2008) found that a monetary incentive did not affect sample composition with regard to different demographic subgroups. Similarly, an experiment on postpaid incentives in the German General Social Survey (ALLBUS) 2010 showed that, while the incentives improved the overall response, they did not change the sample composition in a significant way regarding indicators of socio-economic status (Blohm & Koch, 2013). Messer and Dillman (2010) found no effect for gender, age, income, and race, but a significant effect of incentives for education, as less educated respondents had higher levels of cooperation in the incentive condition. Similar results are reported by Petrolia and Bhattacharjee (2009) and Ryu et al. (2006). A study on the use of incentives in the German Socio-Economic Panel (GSOEP) demonstrated that monetary incentives lead to a higher contact rate for immigrant households, also having a positive effect on participation in subsequent waves (Schr\u00f6der et al., 2013). However, a study by Feskens et al. (2008) suggests this positive effect only for immigrants from western countries, while the cooperation rate of participants of non-western background does not improve with a prepaid incentive (in the form of stamps).<\/p>\n
In summary, existing research shows that incentives are an effective means of boosting response rates, but the calculation of costs and benefits may vary with the population of interest and the potential error associated with (selective) unit nonresponse, as well as the general survey costs. Kephart and Bressler (1958, p.\u00a0123) tested different incentive values (between one penny and a quarter) as well as different methods of mail delivery, to ultimately conclude that \u201ca follow-up letter was just as effective as more expensive and complicated arrangements\u201d in increasing the response rate. Haas et al. (2023) state that incentives should mainly be used when the sample is limited, thus justifying higher per-complete costs. Saunders et al. (2006, p.\u00a01228) argue that in order to minimize overall survey costs, \u201cthe optimum monetary incentive was found to be a function of the variable costs of mailing a questionnaire excluding the incentive, the survey response rate with no incentive, and the coefficient for the mail incentive\u201d. In other words, if survey costs are low, the value of the incentive should also be low to prevent a disproportionate increase in costs.<\/p>\n
Regarding the cost efficiency of incentives, it is particularly relevant to assess whether the effects of incentives differ between survey modes in order to find the most efficient strategy to increase response rates and avoid systematic nonresponse. Alongside differing cost structures, survey modes also differ in sampling strategies, response rates, and potential errors. While web surveys are usually the cheapest survey option, the drawing of a representative online sample is challenging (Baker et al., 2010). Therefore, many web surveys that aim for representativeness use offline recruitment (via mail or telephone), despite higher costs. The sending of invitations by mail has also been shown to improve the response rate for web surveys (Daikeler et al., 2020; Millar & Dillman, 2011) and enables the use of methods to increase response rates, e.g. the inclusion of cash incentives in the envelope (Manfreda et al., 2008). In addition, studies showed that a concurrent design (i.e., giving respondents the choice between modes, typically paper and web) increases response rates compared to a web-first design or a paper-only design (Cornesse et al., 2022; Holmberg et al., 2010; Wolf et al., 2021). In a study by Campbell et al. (2018), a more expensive mixed-mode (mail\/web) approach yielded a response rate of 39%, while the online-only option reached only 4.5%, but seemed to be equally effective with regard to the sample composition.<\/p>\n
Wolf et al. (2021) showed that older age groups are likely to choose the paper questionnaire over a web survey. Stadtm\u00fcller et al. (2023) found that small prepaid incentives (1 or 2 euro coins) worked better in a concurrent design of a self-administered mixed-mode survey compared to a sequential design. Previous research on survey costs by Biemer et al. (2018) shows that while sending participants both a paper questionnaire and a web link does increase the cost per complete, it also has a positive effect on response rates of about 5 percentage points (from 35% in web only to 40% in concurrent mode). In contrast, Medway and Fulton (2012) argue that incorporating a concurrent web option in mail surveys may actually lower response rates.<\/p>\n
Drawing on the state of research, in this paper we want to investigate the effects of a prepaid monetary incentive on participation rates in a sample of a special population: people who have moved away from (or within; that is, they have changed their place of residence) two inner-city neighborhoods in Cologne. The experimental design regarding incentives is complemented by a variation in the mode-choice design of the invitation: concurrent (paper\/web) vs. push-to-web. Our sample of movers includes information on the age and gender of all sample members, thus allowing for a test of the effects of incentive and invitation group on the probability of participation in a cross-sectional study (RQ1 & RQ2). Further, we compare the incentivized and non-incentivized sub-samples with regard to the socio-demographic characteristics income, education and migration background (RQ3) and calculate costs for the different experimental conditions (RQ4). As such, we contribute to research on the interplay of incentive schemes and mode-choice design of invitations, and the influence of those factors on response rates and survey costs (Biemer et al., 2018; Stadtm\u00fcller et al., 2023) in a young, urban population of movers, using a prepaid monetary incentive of a five euro banknote, which is the smallest incentive value that is commonly used in large-scale surveys in Germany (Blohm & Koch, 2021; Pforr et al., 2015).<\/p>\n
In line with previous research, we expect that the use of the incentive of a five-euro banknote sent with the initial invitation will have a positive effect on participation. Furthermore, we expect a higher probability of participation for respondents in the concurrent invitation group, as the paper questionnaire is likely to act as a reminder for participation (Dillman et al., 2014, p.\u00a0382; Manfreda et al., 2008) and giving people the choice between participating on paper or online yields higher response rates than web only (Biemer et al., 2018).\u00a0 Drawing on findings of incentives being particularly effective in a concurrent design (Stadtm\u00fcller et al., 2023), we also test for an interaction between invitation group and incentive (RQ2).<\/p>\n
In terms of age and gender (RQ1), the current state of research does not consistently suggest a direction of effects on participation. We assume, however, that incentives influence sample composition by attracting hard-to-reach groups such as respondents with lower educational levels and income, as well as migrants (RQ3). Finally, we review the costs per complete associated with each experimental condition (RQ4) and reflect on the cost-efficiency of incentives and other design features.<\/p>\n
Data and methods<\/h1>\n
The present study is part of a larger project investigating the process of gentrification in two residential areas close to Cologne\u2019s city center, whose main feature is a panel study where dwellings constitute the units of research. In this dwelling panel, the same dwellings are approached in each wave, and one current resident of each dwelling is interviewed (for the sample design and main research interest, see Friedrichs & Blasius 2015, 2020). While this design provides information on staying residents as well as new in-movers in the sampled dwellings, information on out-movers was hitherto missing. For this reason, we conducted a study focusing on residents who moved out of their dwellings. To provide an adequate sample size, we used a random sample of all out-movers from the two neighborhoods in question between 2018 and 2022, irrespective of whether they had been part of the dwelling panel before.<\/p>\n
The current addresses of out-movers, including those who moved within the neighborhoods, along with information on their age and gender, were provided by the city of Cologne\u2019s Office for Urban Development and Statistics. As the overall focus of the research project is on issues of gentrification and displacement, we excluded residents who moved further away than the city\u2019s own commuter belt (Stadt K\u00f6ln, 2021), as gentrification-induced displacement is expected to happen within local boundaries (Janssen et al., 2023). From a total population of 14,026 out-movers between 2018 and 2022, we drew a random sample comprising n=3,000 persons, who were then randomly assigned into the invitation groups concurrent (n=1,000) and push-to-web (n=2,000). Within each invitation group, n=500 participants were randomly selected to receive a prepaid monetary incentive in the form of a five-euro banknote enclosed with the invitation letter. All invitations were sent to the participants\u2019 postal addresses in June 2023. The letters addressed the topic of the study (life satisfaction in general, satisfaction with participants\u2019 old and new neighborhoods) and informed prospective respondents about their inclusion in the sample based on their recent move from their former neighborhood. The questionnaire comprised about 30 questions \u2013 mostly standardized \u2013 over 11 pages (print version); it took approximately 12.5 minutes to complete (web version). The administration of the survey, as well as response monitoring and data cleaning, was not outsourced to a field agency, but carried out by the research team itself.<\/p>\n
Target persons in the concurrent group received an envelope with the invitation to participate in the interview, a printed questionnaire, and a prepaid return envelope, as well as a link and QR code to the online survey. Target persons in the push-to-web mode received the postal invitation with a link and QR code to the online survey, and were informed that they could order a paper questionnaire and return envelope by contacting our office via phone or email. One reminder with a web link only was sent by mail after four weeks to all persons who had not yet participated, regardless of their initial invitation mode. Neutral losses comprised n=258 persons who were excluded from the random sample due to undeliverable letters (n=214), persons reporting never to have lived in the neighborhoods (n=30), and other neutral drop-out reasons (n=13). In addition, 74 sample members from the push-to-web\/no incentive group were excluded from the present study as they received an incentive with the reminder, deviating from the experimental design. In total, 91 five-euro banknotes were sent back, either because of undeliverable mail or (few) active refusals. Data cleaning based on item nonresponse (>50% of sociodemographic and total questions) led to the exclusion of 63 cases, which were nearly equally distributed between incentivized and non-incentivized participants. In total, 928 valid interviews (34.8% cooperation rate, 31.7% response rate, see AAPOR 2023, version 5.1) constitute the basis of our study.<\/p>\n
A comparison of the age and gender of participants to official registration data exhibits high reliability, with n=910 (98.1%) of answers on gender matching the registered gender (the survey offered a third gender option, while registration does not); survey answers and registration data on age also correlated strongly, with r=.98.<\/p>\n
Looking at the socio-demographic composition of the gross sample (n=2,668) and respondents (RQ1), table 1 shows that there is a gender difference between the gross sample (54.5% males) and respondents (45.7% males), suggesting that women had a higher propensity to participate in the study. This gender effect is slightly mitigated in the incentivized group. Regarding age, it is visible that our sample of out-movers is disproportionately young, with less than 10% of persons aged 60 years and older. This reflects the fact that younger persons are more likely to change accommodation (Dieleman, 2001), while there are no substantial age differences between the gross sample and respondents, as well as between incentivized and non-incentivized groups.<\/p>\n
Table 1:<\/strong> Socio-demographic composition of gross sample and participants, stratified by incentives<\/p>\n We investigate our research questions by first assessing the individual response rates of each experimental condition (RQ2). The probability of participation by socio-demographic characteristics and their interaction with the use of incentives are then examined in a logistic regression (RQ1 and RQ2). In a further step, we compare incentivized and non-incentivized participants in the sample of respondents only, to assess the effect of incentives on sample composition in terms of educational level, migration background, and household income (RQ3). Finally, a calculation of survey costs is presented to investigate the cost efficiency of the experimental conditions (RQ4).<\/p>\n A first insight into the returns by invitation group, use of incentives, and mode of participation (see Fig. 1) shows that the incentives had a strong positive effect on return, almost doubling the return rate in both invitation groups. Respondents in the concurrent group who received an incentive were slightly more likely to return a paper questionnaire (54.1%) than non-incentivized respondents (39.8%).<\/p>\n Figure 1<\/strong>: Cooperation by invitation group, use of incentive, and mode of participation<\/p>\n In the next step, we run a logistic regression on the gross sample in order to assess the probability of participation (table 2). Explanatory variables are gender and age according to registration data, as well as the experimental conditions incentive and invitation group (RQ1 and RQ2). We also check interaction effects of incentive with age, gender and invitation group. First, the model confirms the strong effect of incentives on the probability of participation. With an odds ratio of 2.32, incentives more than doubled the probability of participation. Gender has a significant main effect (men are less likely to participate than women) but shows no interaction with incentives. The main effects of age groups do not reach a significant level, neither do their interaction with incentives. The invitation group has, in contrast to our expectations, no effect on participation probability, the interaction with incentive is not significant either.<\/p>\n Table 2<\/strong>: Logistic model on probability of participation<\/p>\n In the next analysis, we restrict our sample to participants only, as further information on socio-demographics is only available from those who took part in the survey. Here, we test the expectation that incentives are effective in increasing the participation of hard-to-reach groups (RQ3). As shown in table 3, in the incentivized group, there is a higher share of participants without secondary education (24.8% in the incentivized group vs. 16.3% in the non-incentivized group). This effect is significant (p<.01) and of small to moderate strength (CV=0.10). The same trend can be seen for the categories of professional-level education, with a higher share of respondents from the lowest category of vocational education in the incentivized group. In addition, the share of low-income households shows a significant (p<.05) relation to incentive. Both the relative share of households in the lowest (up to 2,000 euro per month) and the second-lowest income group (2,001\u20133,000 euro per month) is higher in the incentivized group. Finally, respondents born outside Germany are also more strongly represented in the incentive group (20.7% vs. 15.3%; p<.05).<\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2024\/04\/Table_1_new.png\")
<\/a><\/p>\nResults<\/h1>\n
<\/a><\/p>\n<\/a><\/p>\n