In social science research, there is a continuing trend to switch from interviewer-administered to self-administered or mixed-mode surveys using web- or paper-based questionnaires. This trend is primarily driven by declining response rates, rising survey costs (e.g., Olson et al., 2021; Wolf, Christmann, Gummer, Schnaudt, & Verhoeven, 2021), and increasing internet penetration rates (Eurostat, 2022). Although a switch from single- to mixed-mode designs has been made before by some studies (e.g., J\u00e4ckle, Lynn, & Burton, 2015; Luijkx et al., 2021; Martin, 2011; Wolf et al., 2021), this trend was accelerated by the COVID-19 pandemic, when many renowned surveys switched from face-to-face interviews to self-administered mixed-mode or web surveys (e.g., Burton, Lynn, & Benzeval, 2020; Kohler, 2020).<\/p>\n
However, moving from face-to-face to a self-administered mixed-mode survey design presents challenges, primarily when recruiting a panel study. In this field report, the main methodological challenges are illustrated using the recruitment of the FReDA panel as an example. We summarize the research design and questionnaire content of FReDA and describe how we adapted the questionnaire to self-administered web- and paper-based modes. We discuss whether the adaptations can serve as good practice for population surveys.<\/p>\n
\u2018FReDA \u2013 The German Family Demography Panel Study\u2019 addresses issues related to family life and development, fertility, and family well-being (Schneider et al., 2021). FReDA is designed as a multi-actor panel study surveying target respondents and their partners. FReDA panel recruitment was initially scheduled for 2020 based on the 60-minute Generations and Gender Survey (GGS) questionnaire designed for face-to-face interviews (Gauthier et al., 2020). Due to the COVID-19 pandemic and related restrictions on public and private life beginning in early 2020, several adjustments were made to the FReDA survey design, including the postponement of the field start to early 2021 and the switch to a self-administered mixed-mode design using web- and paper-based questionnaires (Gummer et al., 2020). Different mode choice designs (i.e., concurrent, web-first, and web-only) were implemented (Christmann et al., 2022).<\/p>\n
Mixed-mode survey designs may have several advantages, such as reducing survey costs, improving coverage by accounting for respondents\u2019 preferences, and reducing nonresponse. Yet, they also have certain difficulties, especially regarding questionnaire design and differential measurement across modes (de Leeuw, 2005, 2018). Particularly when adapting a face-to-face interview to the self-administered mode, the length and complexity of the questionnaire are critical factors in determining respondent burden and willingness to participate (Dillman, Smyth, & Christian, 2014, p. 32ff). With no interviewer present to assist and motivate respondents as needed, a clear questionnaire structure and simple question layout are essential for complete and accurate survey data (Ganassali, 2008). For mixed-mode designs, a unified stimulus for all modes is generally recommended, including the structure, wording, and visual presentation of the questions (Dillman & Edwards, 2016). However, de Leeuw (2018) points out that \u201cthe purpose of equivalent questionnaire design is to maximize data quality in a specific mode and minimize differences in data across modes\u201d (p. 81), so questions may differ between modes if there are good reasons to do so. Especially in the context of self-administered mixed-mode panel studies, the length and complexity of the questionnaire are of particular importance for recruitment rates, panel consent rates, and panel attrition in follow-up waves (Gummer & Daikeler, 2020).<\/p>\n
In the FReDA panel recruitment, two measures were taken to adapt the face-to-face interview to a self-administered mixed-mode survey design: (1) splitting up the questionnaire into three shorter questionnaires and (2) adapting the question wording and layout to the web- and paper-based mode.<\/p>\n
It should be noted that FReDA is a panel survey, so some of the methodological decisions presented in the following might be different for cross-sectional surveys.<\/p>\n
We split the German version of the 60-minute GGS questionnaire into three parts: a short recruitment wave (W1R) and two further subwaves (W1A three months after W1R and W1B four months after W1A). Each subwave should be close to the ideal length of a (web) survey of 10 to 15 minutes and below the maximum length of 30 minutes (Revilla & H\u00f6hne, 2020). There are a few studies on within-respondent modularization of questionnaires by splitting them into shorter parts and offering them to the respondents at several points in time. Findings showed that participants are more willing to respond to shorter questionnaires than to a long undivided questionnaire, but that they are also more likely to drop out, meaning that they do not complete all modularized parts. Overall, this leads to cumulative response rates based on the modularized parts that are similar (Toepoel & Lugtig, 2022) or lower (Andreadis & Kartsounidou, 2020) than the response rate of a long undivided questionnaire. Similarly, some recent studies hint that a self-administered survey longer than 30 minutes may not be as problematic as expected. For example, although the Norwegian GGP 2021 survey had a breakoff rate of 29%, it still achieved a response rate of 33.5% (Dommermuth & Lappeg\u00e5rd, 2021; for similar findings for the GGP countries Germany, Croatia, and Portugal, see Emery et al., 2022, and for ISSP countries, see Sapin, Joye, Nisple, Reveilhac, and Steinmetz, 2022). However, these data refer to cross-sectional studies. As there is little empirical evidence yet on how the length of the first wave of a panel study affects panel consent and participation in subsequent waves, FReDA opted for a shorter recruitment survey, accepting the potentially higher risk of dropout between waves.<\/p>\n
W1R is a 10-minute questionnaire designed to arouse respondents\u2019 interest in the survey topic and create an enjoyable survey experience, ultimately achieving high recruitment and panel consent rates. W1R is therefore short, contains questions that all respondents have something to say about (e.g., life satisfaction), and are easy to answer (e.g., no retrospective questions). In addition, W1R includes basic socio-demographic questions relevant for nonresponse analyses (e.g., gender, date of birth, education).<\/p>\n
W1A and W1B are based on a questionnaire of 25-30 minutes each and includes several modules (see Table 1). Guiding principles for the questionnaire split were:<\/p>\n
(1)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 maintaining the question modules so that questionnaires remain meaningful to respondents in terms of content,<\/p>\n
(2)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 maintaining the order of questions within each module to ensure international comparability with the GGS questionnaire.<\/p>\n
<\/p>\n
Table 1: Overview of GGS questionnaire modules in FReDA W1A and W1B<\/strong><\/p>\n <\/p>\n Notes: X \u2013 Module is completely included in the respective subwave; (X) \u2013 Module is almost completely included in the respective subwave. Exceptions are:<\/em> <\/p>\n Some questions are asked repeatedly in two or all three subwaves, including time-varying information with high thematic relevance (e.g., relationship satisfaction, number of children), information to identify respondents (e.g., gender, date of birth), and information to filter follow-up questions (e.g., employment status).<\/p>\n In general, the main differences between survey modes are (a) the presence or absence of an interviewer, (b) aural versus visual communication, and (c) computerization (Dillman et al., 2014, p. 99ff). Primarily due to the absence of an interviewer and the visual stimulus presentation in web- and paper-based modes, the design of self-administered questionnaires differs from face-to-face questionnaires (e.g., grid versus item-by-item format). Because of computerization, complex question filtering, dynamic question formats (e.g., drop-downs), and other interactive design features (e.g., dynamic loops) can be used in web-based mode but not in paper-based mode (Couper & Bosnjak, 2010).<\/p>\n In the following, we illustrate some examples of key differences in questionnaire design between the face-to-face and self-administered web- and paper-based modes in the FReDA panel recruitment.<\/p>\n In interviewer-administered surveys, an explicit \u2018don\u2019t know\u2019 (DK) or \u2018refuse to answer\u2019 category is usually omitted, as interviewers record such responses with a pre-coded but unread response option. In self-administered surveys, researchers must decide in advance whether to offer a \u2018no opinion\u2019 category. Because selecting DK is also considered a satisficing response strategy and a simple way to reduce cognitive effort, an explicit DK category is often omitted in self-administered surveys (DeRouvray & Couper, 2002; Krosnick et al., 2002). For example, de Leeuw, Hox, and Boev\u00e9 (2016) showed that omitting the DK category leads to the lowest amount of missing information and the highest reliability. Similarly, Kmetty and Stefkovics (2022) consider a skipping-allowed design (i.e., no explicit DK category is offered, but respondents are allowed to skip a question) to be the best choice, as it reduces missing information without harming data quality (e.g., reliability, midpoint responding). Looking at established studies that use self-administered web- or paper-based questionnaires, it seems also common practice to offer a DK category only in exceptional cases (e.g., the German Internet Panel (GIP), German ESS 2020).<\/p>\n In FReDA, there is no explicit DK category, except for a few questions where it can be assumed that respondents really do not know the answer. Exceptions are factual questions that presuppose respondents\u2019 knowledge about themselves or others (e.g., date of immigration), questions that ask for proxy information about others (e.g., partner\u2019s reasons for living apart), or prospective questions (e.g., expected number of children). There is also no explicit \u2018refuse to answer\u2019 category, as skipping questions is allowed in the web-based questionnaire.<\/p>\n In interviewer-administered surveys, item batteries comprising several items using the same answer categories are usually presented in an item-by-item format (i.e., items are asked one-by-one). In self-administered web- and paper-based surveys, item batteries are commonly presented in a grid format because they are time- and space-saving (Couper, Traugott, & Lamias, 2001; de Leeuw, 2018; Toepoel, Das, & van Soest, 2009). However, on devices with small screens such as smartphones, a mobile optimized item-by-item format is preferred (Mavletova, Couper, & Lebedev, 2018; Revilla & Couper, 2018; Revilla, Toninelli, & Ochoa, 2017).<\/p>\n Deciding how many items to display on the screen at once is a tradeoff. Multiple items on a single screen shortens the survey duration but may also increase item nonresponse and respondent dissatisfaction (Couper, Tourangeau, Conrad, & Zhang, 2013). Toepoel et al. (2009) recommend \u201cplacing four to ten items on a single screen, avoiding the necessity to scroll\u201d (p. 210). Similarly, Hofstein Grady, Greenspan, and Liu (2019) concluded that \u201chaving around five rows or potentially fewer per page, and around five columns for answer options, gives the optimal survey experience, with equal or better data quality, when using matrix-style questions in an online survey\u201d (p. 435). Liu and Cernat (2018) recommend an item-by-item format rather than grid format for nine or more answer categories.<\/p>\n In FReDA, item batteries in the web-based questionnaire are generally presented in a grid format, except for devices with small screens where an item-by-item format is used. A maximum of 4 items are shown per screen; longer batteries are split across multiple screens. In the paper-based questionnaire, item batteries are usually presented in a grid format regardless of the number of items.<\/p>\n Loop questions allow dynamic iteration of a series of follow-up questions based on responses to a preceding multiple-choice or frequency question (Eckman & Kreuter, 2018). They are used to request identical information for several people or events. For example, the first question, \u2018How many children do you have?\u2019 is followed by the same (set of) questions multiple times for each child. Loop questions allow a lot of detailed information about several persons or events to be collected one after the other without overwhelming the respondents with too much information at once. In a paper-based questionnaire, however, it is not feasible to present sets of identical questions one after the other due to space constraints. Instead, loop questions are presented in a table format.<\/p>\n In FReDA, we use loop questions for previous partnerships, children, and household members in the web-based questionnaire.<\/p>\n Web-based questionnaires enable the use of soft prompts, which means \u201cnot offering DK, but allowing respondents to skip questions, followed by a polite probe when skips occurred\u201d (de Leeuw et al., 2016, p. 116). Soft prompts may successfully reduce the amount of missing information without causing adverse reactance effects such as dropout (e.g., de Leeuw et al., 2016).<\/p>\n In FReDA, we use soft prompts in the web-based questionnaire primarily for \u2018core\u2019 filter questions that determine whether a subsequent block of multiple follow-up questions is asked (e.g., respondent has a partner, paid work last week), or for filter questions that prevent inappropriate follow-up questions (e.g., questions about child\u2019s residence in the case of deceased child).<\/p>\n In general, it is assumed that well-designed questionnaires reduce the response burden and motivate respondents to participate in the survey. In the context of recruiting a panel study, this is of particular importance, as a high panel consent rate after the first wave and high re-participation rates in the follow-up waves are crucial for the success of a panel study. To this end, we look at established response indicators (see Table 2).<\/p>\n Table 2: Response indicators for FReDA W1R, W1A, and W1B (in %)<\/strong><\/p>\n <\/p>\n Notes: Recruitment and response rates are calculated according to AAPOR RR2, where the number of complete and partial interviews is divided by the sum of all interviews, non-interviews, and all cases of unknown eligibility (AAPOR, 2016).<\/em> <\/p>\n W1R achieved a recruitment rate of 34.9% (RR2; AAPOR, 2016), which is comparable to other surveys in Germany. For example, the face-to-face cross-sectional ALLBUS 2018 had a response rate of 32% (Wolf et al., 2021) and the self-administered mixed-mode recruitment survey of the German Internet Panel (GIP) 2018 had a recruitment rate of 38% (Cornesse, Felderer, Fikel, Krieger, & Blom, 2022).<\/p>\n In W1R, 71.7% of the respondents, including complete and partial cases, consented to be reinterviewed. Of particular note is that the panel consent rate was considerably higher among respondents who completed the web-based questionnaire (74.8%) than among those who completed the paper-based questionnaire (58.7%). The FReDA panel consent rate is comparable to other studies. For example, the online access DEZiM.panel, which was self-administered recruited in 2021, had a panel consent rate of 73.3%, and the rate was also higher in the web-based mode (84.1%) than in the paper-based mode (63.5%) (Dollmann et al., 2022).<\/p>\n In W1R, most respondents participated via the web-based questionnaire with 79.9%, of whom 51.4% reported completing the questionnaire via their smartphone. Both the proportion of respondents who completed the web-based questionnaire and, of these, the proportion of smartphone respondents increased even further in W1A and W1B. In comparison, participation via the web-based questionnaire was lower at 65% for the GESIS Panel recruitment in 2014 (Pforr & Dannwolf, 2017). Participation via smartphones in FReDA is also quite high and was not reached in any age group in the GESIS Panel in 2020 (65 and above around 5%, 18-25 years 45%) (Wei\u00df, Silber, Struminskaya, & Durrant, 2022).<\/p>\n Response rates in W1a and W1B were 81.8% and 75.7%, respectively. Considering non-consenters and dropouts between waves, the cumulative response rates were 28.5% and 21.6%, respectively.<\/p>\n In response to the COVID-19 pandemic, the survey design of the FReDA panel recruitment was changed from a 60-minute face-to-face interview to three shorter, self-administered web- and paper-based questionnaires in 2021. In this field report, we have summarized the key challenges associated with the transition to self-administered mixed-mode survey designs and describe how FReDA responded to these challenges.<\/p>\n While we do not have experimental data to evaluate how well our measures of questionnaire splitting and mode adaptations worked, common response indicators suggest that the FReDA panel recruitment was successful. Our recruitment rate of 34.9% and consent rate of 71.7% in W1R are satisfactory and comparable to those of other self-administered panel studies. The re-participation in the follow-up waves W1A and W1B can also be considered good with response rates of 81.8% and 75.7%, respectively.<\/p>\n The high proportion of respondents participating via the web-based questionnaire shows that the web-based mode is clearly preferred over the paper-based mode in FReDA. In addition, the high proportion of smartphone respondents underscores the importance of a mobile optimized, responsive questionnaire design.<\/p>\n In future research, nonresponse analysis and further data quality analysis have to show whether the quality of respondents\u2019 answers differ by survey mode. Also, the high participation rate in the web-based mode and the large number of respondents using their smartphone to complete the web-based questionnaire highlight the necessity to evaluate device-specific response behavior and probably further optimizations of the web-based questionnaires\u2019 responsive design in case of increasing smartphone participation.<\/p>\n","protected":false},"excerpt":{"rendered":" Introduction In social science research, there is a continuing trend to switch from interviewer-administered to self-administered or mixed-mode surveys using web- or paper-based questionnaires. This trend is primarily driven by declining response rates, rising survey costs (e.g., Olson et al., 2021; Wolf, Christmann, Gummer, Schnaudt, & Verhoeven, 2021), and increasing internet penetration rates (Eurostat, 2022). […]<\/p>\n","protected":false},"author":4956,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[41],"tags":[947,847,934,933,932],"class_list":["post-17948","post","type-post","status-publish","format-standard","hentry","category-questionnaire_design","tag-freda-ggs-survey","tag-mixed-mode-design","tag-mode-adaptation","tag-questionnaire-splitting","tag-web-and-paper-based-questionnaires"],"acf":[],"_links":{"self":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/17948","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/users\/4956"}],"replies":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=17948"}],"version-history":[{"count":25,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/17948\/revisions"}],"predecessor-version":[{"id":19469,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/17948\/revisions\/19469"}],"wp:attachment":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=17948"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=17948"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=17948"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n\n
\n <\/td>\n Questionnaire modules (GGS)<\/strong><\/td>\n W1A<\/strong><\/td>\n W1B<\/strong><\/td>\n<\/tr>\n \n DEM<\/td>\n Demographics<\/td>\n (X)1<\/sup><\/td>\n <\/td>\n<\/tr>\n \n LHI<\/td>\n Life Histories<\/td>\n X<\/td>\n <\/td>\n<\/tr>\n \n FER<\/td>\n Fertility<\/td>\n X<\/td>\n <\/td>\n<\/tr>\n \n HHD<\/td>\n Household Decisions<\/td>\n X<\/td>\n <\/td>\n<\/tr>\n \n GEN<\/td>\n Generations<\/td>\n <\/td>\n X<\/td>\n<\/tr>\n \n WEL<\/td>\n Well-Being<\/td>\n <\/td>\n (X)2<\/sup><\/td>\n<\/tr>\n \n WRK<\/td>\n Work<\/td>\n <\/td>\n (X)3<\/sup><\/td>\n<\/tr>\n \n INC<\/td>\n Income<\/td>\n <\/td>\n X<\/td>\n<\/tr>\n \n ATT<\/td>\n Attitudes<\/td>\n <\/td>\n X<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\n1<\/sup> Questions included in W1R: Country of birth, Place of birth, Date of immigration, Citizenship German\/Country, Highest school leaving certificate, Date school leaving certificate reached, Highest vocational education, Date vocational education reached, Education: Type of academic institution, Internet connection, Internet use, Language at home.<\/em>
\n2<\/sup> Questions included in W1A: 5-item battery on depression, question on subjective health.<\/em>
\n3 <\/sup>Questions included in W1A: 5-item battery on work balance.<\/em><\/p>\nMode adaptations<\/h1>\n
\u2018No opinion\u2019 category<\/h2>\n
Item batteries<\/h2>\n
Loop questions<\/h2>\n
Soft prompts<\/h2>\n
Results<\/h1>\n
\n\n
\n Subwave<\/td>\n W1R<\/strong>1<\/sup><\/td>\n W1A<\/strong>2<\/sup><\/td>\n W1B<\/strong>2<\/sup><\/td>\n<\/tr>\n \n Field period start<\/td>\n 07 Apr 2021<\/td>\n 07 Jul 2021<\/td>\n 04 Nov 2021<\/td>\n<\/tr>\n \n Field period end<\/td>\n 31 Jun 2021<\/td>\n 22 Sep 2021<\/td>\n 31 Jan 2022<\/td>\n<\/tr>\n \n Recruitment rate<\/td>\n 34.9<\/td>\n \u00a0–<\/td>\n \u00a0–<\/td>\n<\/tr>\n \n Panel consent rate<\/td>\n 71.7<\/td>\n \u00a0–<\/td>\n \u00a0–<\/td>\n<\/tr>\n \n Response rate<\/td>\n \u00a0–<\/td>\n 81.8<\/td>\n 75.7<\/td>\n<\/tr>\n \n Cumulative response rate<\/td>\n \u00a0–<\/td>\n 28.5<\/td>\n 21.6<\/td>\n<\/tr>\n \n Participation in web-based mode<\/td>\n 79.9<\/td>\n 85.4<\/td>\n 85.5<\/td>\n<\/tr>\n \n Smartphone participation in web-based mode<\/td>\n 51.4<\/td>\n 52.9<\/td>\n 54.3<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\n1 <\/sup>Results for W1R are based on FReDA panel data from the release v.1.0.0 (DOI: 10.4232\/1.13745), Bujard et al. (2022).<\/em>
\n2 <\/sup>Results for W1A and W1B are based on preliminary prerelease data.<\/em><\/p>\nConclusion<\/h1>\n