Most studies on the quality of responses in web surveys have compared the web mode to an interviewer-administered mode (e.g., Braunsberger, Wybenga, and Gates 2007; Cernat and Revilla 2021; Felderer, Kirchner, and Kreuter 2019; Fricker et al. 2005; Greene, Speizer, and Wiitala 2008; Roster et al. 2004). This comparison is somewhat problematic since the presence of an interviewer, either face-to-face or by telephone, is likely to cause the respondent to answer more in line with the presumed expected answer (cf. social desirability bias; see Kreuter, Presser and Tourangeau 2008) or to make a greater effort. Studies also find that it is possible for an interviewer to motivate the respondent to complete the questionnaire, if he or she considers skipping either single items or half the questionnaire, or even to answer the questions more carefully (for an overview see West and Blom 2017). Thus, it is difficult to determine whether previous findings result from interviewer effects.<\/p>\n
As to self-administered survey modes, several studies have analysed differences in coverage, response rates etc. between web and paper surveys (e.g., Bj\u00f6rkan 2010; Denscombe 2008, 2009; Bech and Kristensen 2009; see also Shin, Johnson and Rao 2012 for an overview), but fewer studies have focused on differences in response behaviour. Magraw-Mickelsen, Wang and Gollwitzer (2022) examined the effect of survey mode on careless responding with data from three different samples, finding only minimal and ambiguous differences between modes across cultures. Likewise, a meta-analysis of 51 articles analysing mode effects on social desirability (Dodou and de Winter 2014) concludes that: \u201c\u2026social desirability in offline, online, and paper surveys is practically the same.\u201d (p.494). A panel study by Shin, Johnson, and Rao (2012) found that web surveys produced fewer \u201cdon\u2019t know\u201d responses than mail surveys but concluded that the characteristics of the panelists might affect the findings, as most of the mode effect disappeared after controlling for sociodemographic characteristics of respondents (pp. 222-223). Furthermore, lower item non-response on web and more detailed answers to open questions was found in a direct comparison, if respondents were allowed to choose between web or paper mode themselves (de Rada and Dom\u00ednguez-\u00c1lvares 2014).<\/p>\n
To contribute to the question of whether self-administered web-based surveys encourage satisficing behaviour, we need to compare it to another self-administered mode. The aim of the present study is to gain firmer knowledge on satisficing in web-based surveys. By comparing survey response behaviour on experimental data from representative population samples in four high internet coverage (above 92 percent)[1]<\/a> countries (Denmark, Iceland, Finland, and Norway), where respondents are randomly assigned to survey completion on either web (\u201cweb group\u201d) or paper (\u201cpaper group\u201d), it is possible to determine whether web mode induces poorer survey responses among respondents. Comparisons of data from high-coverage countries eliminate the impact of differences in coverage across countries, the experimental design enables the elimination of self-selection effects, and comparing web mode with another self-administered mode of data collection eliminates potential interviewer effects.<\/p>\n The study utilised data from national survey experiments in four high coverage countries: Denmark, Norway, Iceland, and Finland. The experiments were conducted in 2014\/15 as part of the ISSP module on \u201cCitizenship\u201d and were based on nationally representative samples. In all four countries, respondents were randomly assigned to one of two mode groups: \u201cweb group\u201d or \u201cpaper group.\u201d In Norway, Finland, and Iceland, the main samples were asked to complete the questionnaire on paper, and a sub-sample was asked to complete a web-based version. If respondents from the sub-sample refused to answer the web-based questionnaire, they were offered a paper questionnaire. In Denmark, the main sample was web-based, and the sub-sample was paper. Furthermore, the Danish experiment was conducted with \u201cpure\u201d modes, meaning that respondents in the main sample were only allowed to complete the web-based questionnaire, and respondents in the sub-sample were only allowed to complete a paper questionnaire. In this study, web respondents were instructed to complete the questionnaire on a computer, as the questionnaire was not designed for smartphones or tablets[2]<\/a>.<\/p>\n A total of 6,219 respondents in the four countries completed the questionnaire (Denmark n=1,758; Norway n=1,459; Finland n=1,505; Iceland n=1,497)[3]<\/a>. See Table 1 below for an overview and for the response rates for the different samples in the four countries:<\/a><\/p>\n In both Denmark, Iceland, and Finland, the response rates were higher for web respondents than for paper respondents, but in Norway the response rate is higher on paper than on web. The highest response rate was found among the web mode sample in Denmark, where 51.3 percent of the sample completed the questionnaire.<\/p>\n Thus, as demonstrated in Table 2, Danes constitute 40.7 percent of the \u201cweb group\u201d, compared to only 17.3 percent Norwegians. Conversely, Danes constitute only 16.5 percent of the \u201cpaper group\u201d compared to 30.4 percent Norwegians.<\/p>\n Randomly assigning respondents to mode groups is supposed to ensure identical gross samples. However, due to differences in the countries\u2019 representation in the two groups, and since Norway, Iceland, and Finland did not implement pure modes in the sampling, self-selection biases may occur. Thus, it is necessary to compare the web and paper groups on core demographic variables before turning to the analysis of differences in satisficing behaviour.\u00a0<\/strong><\/p>\n As summarized in Table 3, statistically significant differences were found between web and paper respondents. Respondents in the web group were 3.4 to 5 years younger than the respondents in the paper group (T-test for difference, p value<\/em> < 0.001), and the proportion of females was significantly lower in the web group (Z-test for difference, p value<\/em> < 0.001). Only smaller differences were found regarding years of education (95% CI: 0.3 to 0.9 years higher in the web group, p value<\/em> < 0.001). This necessitate controlling for these variables in the analyses (see Data Analysis). <\/em><\/p>\n The level of satisficing has been operationalised and analysed using several indicators of the respondent not investing the necessary amount of cognitive effort to answer the questions in an optimal manner (Ro\u00dfmann and Silber 2020). For the operationalisation of the measure of satisficing we used indicators based on satisficing theory (e.g., Krosnick 1991), and as listed by Roberts et al. (2019). As the questionnaire originally was not designed to test differences in satisficing but was a part of the general ISSP data collection, our operationalisation options were limited. This, of course, needs to be taken into consideration when concluding on the results. Inspired by previous empirical studies (see e.g., Keusch and Yan 2017; Lugtig and Toepoel 2016; Mavletova, 2013; Clement, Severin, and Shamshiri-Petersen 2020) and based on the data available, this study adopted three measures of satisficing, i.e., a measure of primacy effects, acquiescence, and no-opinion filter effects:<\/p>\n Table 4 provides an overview of respondent mean scores on the constructed measures of satisficing across the four countries. As evident from the table, the mean scores vary across the measures, with the highest mean scores on the acquiescence index and the lowest mean scores on the no-opinion filter effects index. Country and mode differences will be discussed further in the results section.<\/p>\n The data analysis is carried out in three steps, i.e., one for each satisficing measure included. In each of the steps, multiple linear regression analyses are conducted in IBM SPSS Statistics 27 with one of the satisficing measures as the dependent variable and survey mode (0: paper group, 1: web group) as the main independent variable. This is conducted for each of the four countries separately to detect potential cross-country differences.<\/p>\n The regression analyses are built hierarchically, and each consist of two blocks. In the first block, the uncontrolled association between web-based survey response and the inclination to satisfice is examined (measured as either acquiescence, primacy effects, or no-opinion filter effects). In the second block, core demographic variables (gender, age, and years of education) are added to the model to account for the differences between the web and paper group, as mentioned previously. The regression analyses examine main effects and include Adjusted R2<\/sup>, p-values, and unstandardized parameter estimates to assess whether web-based survey response is associated with higher inclination to satisfice than paper-based survey response.<\/p>\n In the first regression analyses for each of the four countries, the measure of primacy effects is treated as the dependent variable in the models. As displayed in Table 5, in three out of the four countries, there is no statistically significant effect from web-based survey completion on respondents\u2019 inclination to select the first response option compared to completions on paper. In Finland and Norway, the unstandardised coefficient for web mode in Model I is positive, indicating that compared to paper respondents, the score on the primacy index is respectively 0.31 and 0.41 points higher in the web group. However, both coefficients are insignificant and decrease in the controlled models (Model II). In Denmark, the coefficient for web mode is positive (0.05) in Model I and increases when controlling for gender, age, and years of education (Model II). Still, in both models the unstandardised coefficient is inconsiderable and statistically insignificant.<\/p>\n In the Icelandic sample, we find that respondents in the web group are more prone to choose the first option compared to the paper group. A significant coefficient indicates that when going from paper to web, the score on the primacy effect variable increases with 0.79 points. This holds in the controlled model, where the coefficient decreases, but remains significant.<\/p>\n Replacing the measure of primacy effects with acquiescence as dependent variable, we do not find any consistent support for the assumption that web mode increases satisficing behaviour. As demonstrated in Table 6, none of the coefficients for the web group are significant.<\/p>\n When controlling for gender, age, and years of education in the second models (Model II), the negative (insignificant) coefficients in Finland and Norway becomes positive, and the positive estimates in Denmark and Iceland increase a bit. However, none of them is even close to being statistically significant.<\/p>\n As evident from Table 7, the third regression analyses, where the measure of no-opinion filter effects is treated as the dependent variable, reveals evidence of a web mode effect on satisficing behaviour. In Finland, Iceland, and Norway the unstandardised coefficients for the web group are negative in Model I (-0.57, -0.30 and -0.30), indicating that the paper group is more prone to select the \u201cCan\u2019t choose\u201d response option compared to the web group. In Denmark, a positive coefficient of 0.22 indicates the opposite. However, in all four countries coefficients are insignificant, and in the controlled models (Model II) all coefficients decrease and remain insignificant.<\/p>\n In the analyses above, the four countries are analysed separately. However, in order to examine differences in satisficing behaviour between web and paper questionnaire completions, data are tentatively pooled (see Appendix B). In the pooled sample, significant effects are found. However, no clear conclusion is to be drawn here either. As to the inclination to agree (p value <\/em>< 0.01) and to select the first possible response option irrespective of the question content (p value <\/em>< 0.001) there is a statistically significant positive<\/em> effect from web-based survey response \u2013 which in both cases remains when controlling for core demographic variables. However, with regard to respondents\u2019 inclination to select the \u201cCan\u2019t choose\u201d response option, there is at significant negative<\/em> effect from web, which remains significant at 0.05 level in the controlled model, indicating that the paper group is more prone to satisficing. Thus, whereas the un-pooled data quite consistently provided insignificant effects, the pooled data provide quite inconsistent, i.e., both negative and positive, but significant effects. Together it fosters the conclusion as regards web- and paper modes, that one does not promote satisficing behaviour more than the other.<\/p>\n Results from analyses of three different measures of satisficing behaviour in survey responses do not give support to an assumption that self-administered web-based surveys cause respondents to put lesser effort into answering the questionnaire in general. As evident from above, there is not statistically significant indications of the web group in the present experiment being more prone to neither acquiescence, primacy nor choosing the \u201cCan\u2019t choose\u201d response option than the web group. Effect estimates tend to go in different directions in the four countries \u2013 and controlling for core demographic variables also impacts the effect from mode differently, both across countries and across satisficing measures. Only in one case a statistically significant result was found: In Iceland primacy effects was significantly more present in the web group than in the paper group.<\/p>\n Thus, results do not substantiate greater satisficing behaviour among the web group. Rather, and in line with parts of previous research on the field, differences found in the present study are both minor and ambiguous \u2013 not revealing any systematic mode differences regarding satisficing.<\/p>\n To examine whether web-based surveys encourage satisficing behaviour, this study has compared completion on web to paper. Most previous studies are based on comparisons of web- and telephone-based surveys, but since interviewers can foster and maintain survey respondents\u2019 motivation to make an effort during completion of the questionnaire, it was crucial to eliminate the potential interviewer effect. However, based on our results, we do not find indications of web-based surveys being more cognitively burdensome or distracting to respondents.<\/p>\n Regarding limitations, the present study would have benefitted from more measures of satisficing behaviour, including attitude questions with the positive response option presented last in order to separate acquiescence and primacy effects. Unfortunately, data did not enable further operationalisation. Additional measures would have allowed for analysing more dimensions of satisficing behaviour among respondents in the two groups, presumably substantiating the conclusion that web and paper respondents do not differ significantly as regards to satisficing behaviour.<\/p>\n Further, as we argue that many differences in data quality between web mode and paper mode most likely can be ascribed to differences in coverage, the present study is based on four high-coverage countries; Denmark, Iceland, Finland, and Norway, with general populations characterized by a high level of technological literacy. This in order to minimise potential effects from some respondents\u2019 unfamiliarity with the Internet and new technologies. That means, however, that we are not able to determine whether results are applicable to low-coverage countries. Clearly, in countries with lower levels of technological literacy, web-based surveys could potentially increase the cognitive burden, ultimately leading to poorer response quality. However, in that regard it can be considered an advantage that data dates back to 2014\/2015. Whereas the four high-coverage countries could be regarded as \u201ccritical cases\u201d (cf. Flyvbjerg 2006) at the time, development \u2013 i.e., more and more countries reaching similar level of coverage – has led them and thus the present results to be more representative. That said, it is a limitation of the present study that the questionnaire was not designed for mobile devices but only for PC. Today, an increasing number of respondents will use smartphones or tablets to answer, and that has raised questions of data quality across completion devices (e.g., Couper, Antoun and Mavletova 2017), as well as survey designs need to adjust accordingly.<\/p>\n For further research into quality of web-based surveys, it would be beneficial to examine the potential they hold to motivate respondents and to decrease the cognitive burden – potentially improving data quality. The interactive and dynamic nature of web media allows for survey designs that are more visually appealing and interesting for respondents to interact with, e.g., by interactive feedback, motivational messages etc., just as it holds the potential to assist respondents during the process, e.g., by providing different language options, read-aloud of instructions, interactive voice response, etc. Studies have to some degree assessed these advantages, but the potential for cognitively engaging and maintaining respondents in web-based surveys deserve to be subjected to exhaustive examination.<\/p>\n \u00a0<\/em> [1]<\/a> Access from household: Denmark: 94%, Norway: 97%, Iceland: 96%, Finland: 92% (Eurostat, 2016 figures).<\/p>\nMethods<\/h1>\n
Data<\/h2>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-1-1.jpg\")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-2.jpg\")
<\/a><\/p>\nSample Characteristics<\/h1>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-3.jpg\")
<\/a><\/p>\nMeasurement of Satisficing<\/h2>\n
\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-4.png\")
<\/a><\/p>\nData Analysis<\/h2>\n
Results<\/h1>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table_5-837x1024.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table_6-832x1024.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table_7-810x1024.png\")
<\/a><\/p>\nConclusion\/Discussion<\/h1>\n
Appendix A. Overview of items used to construct satisficing measures<\/h2>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Appendix-A-2.jpg\")
<\/a><\/p>\nAppendix B.\u00a0 Regression analyses on pooled data<\/h2>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-8-NEW.jpg\")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-9-NEW.jpg\")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2021\/12\/Table-10-NEW.jpg\")
<\/a><\/p>\n