An Experiment Comparing Concurrent and Sequential Choice+ Mixed-Mode Data Collection Protocols in a Self-Administered Health Survey

Research note

---

Abstract

In self-administered web and paper mode surveys, the Choice+ protocol, which offers both modes concurrently but with a differentially higher incentive for responding by web, has been shown to increase the share of responses obtained by web and to increase response rates overall relative to a concurrent design with equal incentive amounts. This protocol was initially adopted for the 2020 Healthy Chicago Survey. After observing fewer than 20% of the 2020 completes coming via paper, however, an experiment was fielded in the 2021 administration to evaluate a sequential variant of the Choice+ protocol in which the paper mode was only offered to nonrespondents after an initial web-only invitation. This article reports on the key findings from this experiment. Although the sequential Choice+ protocol sacrifices 1.4 percentage points in the yield rate, it comes with a 15.2% lower cost-per-complete and produces a respondent set that looks very similar to the concurrent Choice+ protocol with respect to both demographics variables and key health outcomes measured by the survey. As a result of these findings, the sequential Choice+ variant has supplanted the concurrent Choice+ design as the primary data collection protocol in the Healthy Chicago Survey.

Keywords

Choice+, mixed-mode design, mode experiment, paper survey, web survey

The conclusions in the article are those of the authors and do not necessarily represent the views of the Chicago Department of Public Health.

Background

A common concurrent mixed-mode design (de Leeuw, 2018) for mail contact surveys is to offer respondents two methods for completing the survey: (1) return an enclosed paper questionnaire or (2) log into a web-based version of the instrument. When provided the choice, research has shown respondents are more likely to choose paper over web (e.g., Biemer et al., 2016), even though web is the more cost-effective, environmentally friendly method, and produces more timely, higher quality data, since proper branching and data consistency checks can be programmed directly into the electronic instrument. Respondents’ preference of paper over web assumes, however, equivalent incentive amounts across the two modes. Biemer et al. (2018) coined the term Choice+ for a concurrent mixed-mode data collection protocol that assigns a higher incentive for the respondent to complete the survey via the web mode relative to the paper mode. Biemer and colleagues evaluated the Choice+ method on the 2015 Residential Energy Consumption Survey (RECS) National Pilot with a $10 promised incentive for completing by paper but a $20 promised incentive for completing by web, finding that the majority of respondents chose the web mode over paper. An added benefit was that the Choice+ protocol resulted in a nearly 3 percentage point response rate increase over an analogous concurrent mixed-mode protocol without the differential incentives. Similar findings have been obtained in other surveys—for an example, see the discussion by Medway and Megra (2022) of mode experiments conducted during the 2019 National Household Education Survey.

In the wake of these findings, the Choice+ protocol has been adopted in numerous mixed-mode mail contact surveys, including the 2018 National Recreational Boating Safety Survey (Duffy et al., 2020), the 2020 New York State Problem Gambling Prevalence Survey (New York State Office of Alcoholism and Substance Abuse Services, 2020), and the 2020 Healthy Chicago Survey (HCS) (RTI International, 2020). As reported in RTI International (2020), the Choice+ protocol implemented in the 2020 HCS resulted in a mix of 81% web respondents and 19% paper respondents. Given the low proportion of paper respondents and the non-trivial printing and postage costs of the 20-page survey booklet, the HCS administration team experimentally evaluated a sequential version of the Choice+ protocol in the 2021 HCS. In the first contact attempt, respondents were offered the web mode exclusively, with the Choice+ protocol offered to nonresponding households three weeks later. The purpose of this article is to report results from this experiment, which we found to produce a nearly equivalent yield rate, but at a significantly lower cost-per-complete and without introducing substantive differences with respect to demographic variables or key health outcome variables.

Data and Methods

Data from this article are drawn from the HCS, an annual health survey of the adult population of the City of Chicago sponsored by the Chicago Department of Public Health. HCS data are used to develop and evaluate numerous public health interventions and policies to promote health equity. First launched in 2014 as a dual-frame, random-digit dial (DFRDD) telephone survey, HCS transitioned to a self-administered format in 2020 using an address-based sampling (ABS) frame (Unangst et al., 2022). The households in the target population are stratified into 77 community areas, and those sampled are asked to have the adult next to celebrate a birthday complete the survey (Olson et al., 2014). The survey sample is fielded in two subsequent releases to target between 4,000 – 4,500 completes overall with a minimum of 35 completes from each of the 77 distinct community areas.

The Choice+ design utilized in the 2020 HCS began by sending households a survey packet containing a $2 pre-incentive, an invitation letter with log-in credentials to complete the survey via web, and a paper copy of the survey for respondents who wish to complete it using that mode. While the survey only takes about 20 minutes to complete, a 20-page booklet is required to print all of its questions. The invitation letter informed the respondent that a $10 incentive would be paid for completing via paper but a $20 incentive would be paid for completing via web. A reminder postcard was sent one week later, and two weeks thereafter another full survey packet was sent to nonresponding households. The final contact attempt was a reminder postcard sent one week after the second full survey packet.

In the 2020 HCS administration, the concurrent Choice+ protocol resulted in a yield rate of 30.5%, but with only 19% of respondents completing the survey by paper. By comparison, Biemer and colleagues (2018) reported that approximately 35% of respondents completed the 2015 RECS National Pilot by paper in the Choice+ condition. The relatively lower share of respondents completing the 2020 HCS via paper led the survey administration team to consider whether the paper booklet could be excluded from the first mailing to save on data collection costs and improve data quality without having a noticeable impact on respondent demographic distributions or distributions of key health outcomes. To test this hypothesis, an experiment was fielded during the first sample release of the 2021 HCS administration whereby a randomly designated 25% of the sampled addresses (n = 1,750) was designated to receive the same Choice+ protocol as was administered in the 2020 HCS, while the complementary 75% (n = 5,250) received the sequential variant. Specifically, the sequential data collection protocol involved offering web response exclusively in the first mailing for a $20 incentive, and then introducing the paper mode as an alternative avenue of response three weeks later for a $10 incentive. Table 1 summarizes the four contact attempts for this sequential Choice+ design alongside the contact attempts for the concurrent Choice+ design.

 

Table 1: Timeline of Contact Attempts for the Concurrent Choice+ and Sequential Choice+ Data Collection Protocols Experimentally Compared during the 2021 Healthy Chicago Survey.

 

	Concurrent Choice+		Sequential Choice+
Day	Mailing Description		Mailing Description
0	Survey Packet		Web Invitation Letter
7	Reminder Postcard		Reminder Postcard
21	Survey Packet		Survey Packet
28	Reminder Postcard		Reminder Postcard

Results

Table 2 presents disposition codes and percentages of the 7,000 sampled addresses allocated to the two Choice+ experimental conditions during the first sample release of the 2021 HCS. The yield rate for the sequential Choice+ protocol, at 24.7%, is only 1.4 percentage points lower than the 26.1% yield rate for the concurrent Choice+ protocol. As would be expected, the sequential Choice+ protocol also yielded fewer paper responses: only 10.1% of completes were obtained by paper, whereas 20.6% of the completes came via paper for the concurrent Choice+ protocol. Although the higher portion of web responses resulted in a higher average incentive payment, the sequential Choice+ variant still resulted in a cost-per-complete that was 15.2% lower. In essence, the printing and postage costs saved by excluding the 20-page survey booklet from the first mailing outweighed the increased average incentive amount.

 

Table 2: Dispositions of Sample Cases Allocated to the Two Mixed-Mode Data Collection Protocols – Concurrent Choice+ and Sequential Choice+.

 

 

Cost savings aside, another key analysis of interest to the HCS administration team was to compare household- and individual-level demographic variable distributions across the two respondent sets as well as distributions for certain key health outcomes measured by the survey. Results of these comparisons are provided in Tables 3 and 4, respectively. The specific variables chosen are many of the same variables investigated as part of the DFRDD-ABS mode transition analyses conducted in Unangst et al. (2022). Statistical significance tests were carried out for the variable as whole using a chi-square test, and for specific categories of a variable using a two-sample t-test.

Table 3 shows how the demographic distributions at both the household and individual level are very similar across the two Choice+ protocols. There are only two categories – 25-29-year-olds and Hispanic/Latino individuals – where the percentage of respondents differs significantly. As judged by the chi-square test p-values, none of the ten demographic variables, on the whole, has a distribution that differs significantly. We viewed this as a positive finding considering that the sequential Choice+ protocol generated one-half the rate of paper completes than the concurrent Choice+ protocol.

With respect to the key health outcomes, Table 4 conveys a similar story of highly compatible distributions across the two Choice+ approaches. However, we must acknowledge that distributions were significantly different at conventional levels for the following two health outcomes: (1) the indicator for whether the respondent currently has asthma and (2) whether the respondent had experienced psychological distress in the past 30 days. Respondents from the sequential Choice+ protocol were more likely to currently have asthma (40.0% vs. 25.8%) and less likely to have experienced mild or moderate psychological distress in the past 30 days (77.3% vs. 71.4%). It is certainly plausible these differences occurred by chance, as there were no statistically significant differences observed across any of the other eleven health outcomes, and neither protocol appears to result in a respondent set that looks systematically different.

 

Table 3: Comparison of Household- and Individual-Level Demographic Distributions of Respondents from the Two Mixed-Mode Data Collection Protocols – Concurrent Choice+ and Sequential Choice+.

 

 

Table 4: Comparison of Select Health Outcome Distributions of Respondents from the Two Mixed-Mode Data Collection Protocols – Concurrent Choice+ and Sequential Choice+.

 

 

Conclusion

The Choice+ survey data collection protocol described in Biemer et al. (2018), which offers both web and paper response modes concurrently, but with a differentially higher promised incentive for those completing by web, has been adopted by several surveys in recent years as a proven way to increase web participation—thereby increasing data quality and timeliness—and to increase response rates. The original design calls for sending a paper questionnaire to households at the same time as the web invitation. While maintaining the ability for individuals to complete the survey by paper is important, especially if the propensity to not have internet readily accessible is correlated with key survey outcomes, the protocol may not be cost-efficient in settings where a low proportion of respondents opt for the paper mode, as was the case in the 2020 HCS where fewer than 20% of completes were obtained by paper. This article presented results from an experiment fielded during the 2021 HCS evaluating a sequential Choice+ variant, one offering a web mode exclusively at first, then introducing the paper mode (alongside another web invitation) later in the data collection period.

Results from the experiment are as follows. The sequential Choice+ protocol had a 1.4 percentage point lower yield rate relative to the concurrent Choice+ protocol, but it resulted in a cost-per-complete metric that was 15.2% lower. Moreover, the sequential Choice+ protocol produced a respondent set that mirrors the respondent set produced by the concurrent Choice+ protocol with respect to both demographic variables and key health outcomes from the survey. In light of these findings, the HCS administration team has adopted the sequential Choice+ design as the survey’s sole data collection protocol moving forward.

We hope results from this experiment prompt analogous experiments in surveys such as RECS and the New York State Problem Gambling Prevalence Survey, where the most recent administrations utilizing the concurrent Choice+ design resulted in 27% and 15% of completes, respectively, coming by paper. These may be additional settings where cost savings could be realized without detriment to key dimensions of the surveys’ respondent distributions. Further research could also aim to gain insights into other mitigating factors such as the length and complexity of the questionnaire or the magnitude of the incentive differential, perhaps relative to a concurrent mixed-mode condition with both modes offering the same lower incentive amount. That equal incentive protocol would likely have a lower response rate and lower cost-per-complete metric, yet it may be defensible if the respondent profile it produces does not differ dramatically from that of the differential incentive protocol.

References

1. Biemer, P., Murphy, J., Zimmer, S., Berry, C., Lewis, K., and Deng, G. (2016). A Test of Web/PAPI Protocols and Incentives for the Residential Energy Consumption Survey. Presentation given at the annual meeting of the American Association for Public Opinion Research. Austin, TX.
2. Biemer, P., Murphy, J., Zimmer, S., Berry, C., Lewis, K., and Deng, G. (2018). Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys. Journal of Survey Statistics and Methodology, 6(2), 240 – 261.
3. Duffy, T., Kilpatrick, G., Krotki, K., Lewis, T., McMichael, J., Palmer, D., Ridenhour, J., Ryder-Burge, A., Thomas, I., Willis, S., Mahoney, E., and Herbowicz, T. (2020). National Recreational Boating Safety Survey: Exposure Survey Final Report. Available at: https://uscgboating.org/library/recreational-boating-servey/NRBSS-Exposure-Survey-Final-Report-11302020.pdf.
4. de Leeuw, E. (2018). Mixed-Mode: Past, Present, and Future. Survey Research Methods, 12(2), 75 – 89.
5. Medway, R., and Megra, M. (2022). Pick Me! Incentivizing Web Response in a Concurrent Mixed-Mode Design. Presentation given at the annual meeting of the American Association for Public Opinion Research. Chicago, IL.
6. New York State Office of Alcoholism and Substance Abuse Services. (2021). 2020 New York State Problem Gambling Prevalence Survey: Final Report. Available at: https://nyproblemgambling.org/resource-lib-item/2020-new-york-state-problem-gambling-prevalence-survey/.
7. Olson, K., Stange, M., and Smyth, J. (2014). Assessing Within-Household Selection Methods in Household Mail Surveys. Public Opinion Quarterly, 78(3), 656 – 678.
8. RTI International (2020). 2020 Healthy Chicago Survey (HCS) Methodology Report. Available online at: https://www.chicago.gov/content/dam/city/depts/cdph/statistics_and_reports/2020_HCS_Methodology_Report.pdf.
9. Unangst, J., Lewis, T., Laflamme, E., Prachand, N., and Weaver, K. (2022). Transitioning the Healthy Chicago Survey from a Telephone Mode to Self-Administered by Mail Mode. Journal of Public Health Management & Practice, 28(3), pp. 309 – 316.