{"id":15753,"date":"2023-04-04T11:49:37","date_gmt":"2023-04-04T10:49:37","guid":{"rendered":"https:\/\/surveyinsights.org\/?p=15753"},"modified":"2024-01-09T17:47:04","modified_gmt":"2024-01-09T16:47:04","slug":"the-randomized-overclaiming-method-rome-a-paradigm-for-assessing-the-validity-of-social-desirability-reduction-techniques-using-objective-measures","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=15753","title":{"rendered":"The Randomized Overclaiming Method (ROME): A paradigm for assessing the validity of social desirability reduction techniques using objective measures"},"content":{"rendered":"

1. Introduction<\/h1>\n

When answering survey questions, respondents often exhibit socially desirable responding (SDR), \u201cthe tendency to give overly positive self-descriptions\u201d (Paulhus, 2002, p. 50). This phenomenon threatens the validity of both univariate statistics and measures of association (Wolter & Preisend\u00f6rfer, 2013, p. 322). Consequently, researchers have long used questions that employ SDR reduction techniques such as suggesting that the characteristic in question is exhibited by many people (Barton, 1958). So far, these attempts have been met with limited success, and SDR reduction remains an active area of research (e.g., Harling et al., 2021; Hibben et al., 2022; Wolter & Preisend\u00f6rfer, 2020).<\/p>\n

Part of the reason for this limited success may be that the effectiveness of SDR reduction techniques is hard to study. Ideally, researchers would be able to compare a respondent\u2019s answer regarding a characteristic to a criterion measure displaying the true value of this characteristic (H\u00f6glinger & Jann, 2018; Preisend\u00f6rfer & Wolter, 2014, p. 127). While studies of this type exist, their number is limited, as true values for the characteristic in question are often unavailable. When they are available, the criterion measures themselves may be measured with error (e.g., Boon et al., 2010, pp. 744-745; Cassel, 2004; Lu et al., 2008, p. 86), which makes the comparison with questionnaire-derived values less informative.<\/p>\n

Hence researchers studying social desirability bias have usually relied on the more-is-better\/less-is-better assumption. It states that when two approaches to asking about an undesirable (desirable) characteristic are compared, the one that results in higher (lower) estimates for the characteristic is to be preferred (Krumpal, 2013, p. 2033). While this assumption is \u201coften plausible, it is still just an assumption\u201d (Tourangeau & Yan, 2007, p. 863) and any conclusions drawn on its basis hence come with considerable uncertainty attached (H\u00f6glinger & Jann, 2018, p. 3). Even if we were to believe the assumption and prefer one estimate of a characteristic over another, we still would not know how far the preferred estimate is from the truth. Moreover, if an SDR reduction technique is successful in moving carriers of an undesirable characteristic from not reporting this characteristic to reporting it, the technique may also move some non-carriers from not reporting to reporting (Diekmann, 1996, pp. 384-385). This is not implausible given that some studies found that noteworthy proportions of respondents underreported desirable or overreported undesirable characteristics (Kreuter et al., 2008; Wyner, 1980).<\/p>\n

Recognizing this, H\u00f6glinger & Diekmann (2017) recently argued that researchers \u201cmust stop relying blindly on the more-is-better assumption and explicitly consider the possibility of false positives\u201d (p. 136). To this end, they suggested the use of \u201czero-prevalence items\u201d in the study of SDR. These are items that no truthful respondent can answer in the affirmative and which hence can be used to detect overreporting without the need for the researcher to obtain measures of the true values. However, finding feasible zero-prevalence items is not trivial. Items used so far include having received a donated organ and suffering from rare diseases (H\u00f6glinger & Diekmann, 2017; Schnapp, 2019; Wolter & Diekmann, 2021). The true prevalence of these items is not exactly zero and they are not obviously socially undesirable or desirable.<\/p>\n

This article hence presents a method which allows researchers to assess the effectiveness of SDR reduction techniques by including a large number of credible but non-existent zero-prevalence items in a survey. The Randomized Overclaiming Method (ROME) uses a questionnaire that tests respondents\u2019 knowledge of a number of items and mixes existing items, called reals<\/em>, with non-existing items, called foils<\/em>. Respondents are randomly assigned either to a control condition or a treatment condition which uses some SDR reduction technique. The number of foils claimed serves as a measure of SDR. If the treatment condition yields significantly fewer foils claimed than the control condition, it may be concluded that the SDR reduction technique was effective. Also, in contrast to studies using the more-is-better\/less-is-better assumption, the amount of SDR in all conditions can be quantified precisely and without the need to collect additional data.<\/p>\n

The number of foils claimed can serve as a valid measure of SDR under two assumptions: First, the respondents cannot truly know the foils; second, it is desirable for respondents to claim knowledge of the foils. The second assumption may often be debatable. In a discussion of overclaiming, Paulhus (2011) argued that \u201cpeople do not invest their egos in knowledge about topics that are irrelevant (or in opposition) to their identities\u201d (p. 156). This jibes well with a perspective on desirable responding which interprets \u201cnormative behavior overreporting […] as the enactment of a salient identity\u201d (Brenner, 2012, p. 422). If these views are right, we should expect desirability effects on overclaiming primarily when being knowledgeable about the topic of the questionnaire is important to a respondent\u2019s identity (or the respondent has an extrinsic incentive to overclaim). Hence, the ROME attempts to ascertain that respondents are selected such that they deem claiming knowledge of the items in question desirable.<\/p>\n

The method builds on previous research. The use of overclaiming questionnaires as measures of socially desirable responding was popularized in psychology by Paulhus et al. (2003), who drew on earlier work (Phillips & Clancy, 1972; Randall & Fernandes, 1991; Stanovich & Cunningham, 1992). This line of research studies the association of overclaiming measures with measures of personality. However, following Paulhus et al. (2003), the majority of such studies has measured \u201coverclaiming\u201d as the general tendency to claim knowledge of any<\/em> items (reals and foils), while only a minority has used measures of genuine over<\/em>claiming based on foils only. The latter body of research has found that genuine overclaiming is often positively associated with extraversion and openness to experience (Dunlop, Bourdage, de Vries, Hilbig et al., 2017; Dunlop, Bourdage, de Vries, McNeill et al., 2020, Study 1). Evidence on associations with other dimensions of personality is more scant, less consistent, or shows null results (Barber et al., 2013; Bertsch, 2009; Bing et al., 2011; Calsyn et al., 2001; Dunlop et al., 2017; Dunlop et al., 2020; Franzen & Mader, 2019; Ludeke & Makransky, 2016; Mesmer & Magnus, 2006; Schoderbek & Deschpande, 2011).<\/p>\n

A previous study by Calsyn et al. (2001) used a design quite similar to the ROME. These researchers studied respondents\u2019 tendencies to claim knowledge of social services agencies that did not exist and randomly assigned participants, drawn from the general population, to conditions. It was found that warning respondents that the list of agencies presented would contain foils significantly reduced overclaiming, while providing them with a memory retrieval strategy did not.<\/p>\n

The Calsyn et al. (2001) design differs from the ROME in that it was not ensured that respondents were matched to the topic such that they would deem claiming knowledge of the items desirable. Had this been done, one might have expected larger treatment effects. Perhaps more importantly, judging from citations, this paper has made no impact on survey methodology research \u2013 which is unsurprising given that it was published in a journal not typically read by survey methodologists (Evaluation Review<\/em>) at a time well before the recent increase in interest in objective criterion measures of undesirable behaviours.<\/p>\n

The present paper hence demonstrates the use of the ROME by means of a small study of an SDR reduction technique. An overclaiming questionnaire was used which mixed real and fake English words and asked respondents whether they knew them. Respondents were selected such that they were not native speakers of English but proficiency in English was important to them. They were randomized into a control or a treatment group. Measures of traits that have been shown or seemed likely to correlate with overclaiming were also collected to control for remaining imbalances between the treatment and the control group in multiple regressions.<\/p>\n

The SDR reduction technique used for this demonstration is a combination of forgiving wording and appeals to honesty. Most previous studies of appeals to honesty and forgiving wording using the more-is-better-assumption have found mostly null or mixed results (Brener et al., 2004; Catania et al., 1996; Contzen et al., 2015; Holtgraves et al., 1997; N\u00e4her & Krumpal, 2012; Ong & Weiss, 2000; Persson & Solevid, 2014). However, in the single randomized study of the effect of appeals to honesty (Brener et al., 2004), the appeal may have been overlooked by many respondents (p. 99) as it was placed on the cover page of a paper questionnaire (p. 94). If not overlooked, it may have caused reactance in some of the adolescent respondents, given that it was printed in italics and the wording was bordering on the impolite (\u201cIt is extremely important that you tell the truth when answering the questions\u201d, Brener et al., 2004, p. 94). Hence, it may have caused some respondents to report more honestly (as intended) and others to report less honestly (due to reactance), with the effects cancelling each other out. As for forgiving wording, there are also studies reporting the expected effect (Acquisti et al., 2012, Study 1; Raghubir & Menon, 1996, Study 1). It hence seems premature to declare either technique ineffective.<\/p>\n

Moreover, it was reasoned that the combination<\/em> of the two techniques might be particularly effective. Appeals to honesty aim to instil in respondents a positive motivation to tell the truth. Forgiving wording aims to remove a barrier to telling the truth. It hence seemed probable that the combination of the two techniques would lead to more honest answers. Note that the study\u2019s design does not allow for distinguishing between main and interaction effects of forgiving wording and appeals to honesty, but only allows for assessing the combination of the two. Readers are also asked to keep in mind that the main motivation for the present article was not the study of this combination of techniques but rather to demonstrate how the ROME may be used to study the efficacy of a given SDR reduction technique. The value of this demonstration is independent of the specific results regarding the effectiveness of the specific SDR reduction technique studied.<\/p>\n

2 Data and methods<\/h1>\n

2.1 Preregistration<\/h2>\n

Data collection stopping rules, inclusion\/exclusion criteria, hypotheses and data analyses were preregistered. Deviations from the preregistration are noted in section 2.6 below.<\/p>\n

2.2 Participants and Procedure<\/h2>\n

A German-language online survey was programmed in SoSci Survey (Lehner, 2019a). All questions had to be answered to proceed to the next page of the questionnaire. Data were collected via SurveyCircle, a German language online platform where participants can collect points by filling in surveys. The more points a participant collects, the more points others will collect when filling in a survey this participant posts. The sample thus drawn is not representative of an identifiable population. As no invitations were sent, no response rate can be calculated.<\/p>\n

The survey was posted under the description \u201cRecognition of English words\u201d and users were invited to participate if they had at least \u201cgood\u201d knowledge of English, were native speakers of German but not English and were aged 18-49.[1]<\/a> Respondents could continue from the first page only if they confirmed that they fulfilled the inclusion criteria and agreed to the use of their data for the stated research purposes.<\/p>\n

The first pages of the questionnaire asked respondents about sociodemographics and proficiency in German and English. Respondents who did not meet the eligibility criteria, according to their answers, were redirected to an exclusion page which stated they did not fulfil these criteria and thanked them for their interest.<\/p>\n

The next five pages comprised of an introductory page and 4 pages with 5 items each measuring how important it was for respondents to speak English well and 5 personality constructs.<\/p>\n

The following page introduced the section on English vocabulary in general and the practice section in particular. It informed respondents about the task and the fact that there would be 3 practice items. Invisible to respondents, this page also contained the randomization device.<\/p>\n

After the three practice items, participants were taken to another text page introducing the \u201cactual questions on word recognition\u201d. It was followed by 30 items (reals and foils), a text page informing participants they had reached the midpoint of the word list, and 30 more items. One item was presented per page. Respondents were taken to the next page upon indicating whether they knew the word by clicking \u201cyes\u201d or \u201cno\u201d. All pages contained a \u201cback\u201d button.<\/p>\n

Another text page informed respondents they had reached the end of the word list and there would be one more question. It was displayed on the next page and asked whether respondents had filled in the questionnaire before. A final page debriefed respondents and contained the code participants needed to collect points.<\/p>\n

2.3 Treatment<\/h2>\n

Respondents were randomly assigned to either the control or the treatmet condition. The treatment consisted of a combination of appeals to honesty and forgiving wording. Four kinds of pages differed between the treatment and control conditions.<\/p>\n

 <\/p>\n

Figure 1<\/strong><\/p>\n

Introductory page to the main part of the word recognition task<\/em><\/p>\n

\"\"<\/a><\/strong><\/p>\n

Note.<\/em> Translated from the original German. Text within the grey box was included only in the treatment condition and missing in the control condition; the screens displayed to respondents did not contain any such boxes.<\/p>\n

 <\/p>\n

On the introductory page to the main part of the word recognition task (figure 1), respondents in the treatment condition were informed that they were answering a version of the questionnaire that exhibited the \u201clevel of difficulty: very high\u201d (forgiving wording), meaning that practically nobody can identify all words correctly (forgiving wording). They were also asked to be an honest survey participant (appeal to honesty) and \u201cremember: not everyone can know everything\u201d (forgiving wording).<\/p>\n

 <\/p>\n

Figure 2<\/strong><\/p>\n

Page displayed at the midpoint of the main part of the word recognition task<\/em><\/p>\n

\"\"<\/a><\/strong><\/p>\n

Note.<\/em> Translated from the original German. Text within the grey box was included only in the treatment condition and missing in the control condition; the screens displayed to respondents did not contain any such boxes.<\/p>\n

 <\/p>\n

A short version of this part of the treatment was repeated on the treatment version of the text page displayed after half the items had been answered (figure 2).<\/p>\n

 <\/p>\n

Figure 3<\/strong><\/p>\n

Example of a word recognition page from the main part of the questionnaire<\/em><\/p>\n

\"\"<\/a><\/p>\n

Note.<\/em> The figure shows a translation; in the version displayed to respondents, all text except for the stimulus word (\u201cmensible\u201d) was German. Text within the grey box was included only in the treatment condition and missing in the control condition; the screens displayed to respondents did not contain any such boxes.<\/p>\n

 <\/p>\n

Treatment and control versions of the questionnaire also differed in terms of the texts included on the word recognition pages. In the treatment condition, each page repeated the texts \u201cLevel of difficulty: very high\u201d and \u201cPlease be an honest survey respondent and remember: not everyone can know everything\u201d (figure 3). The latter phrase was also displayed in the practice section of the treatment version (figure 4). In the control version, these texts were not displayed.<\/p>\n

 <\/p>\n

Figure 4<\/strong><\/p>\n

Example of a word recognition page from the practice part of the questionnaire<\/em><\/p>\n

\"\"<\/a><\/p>\n

Note.<\/em> The figure shows a translation; in the version displayed to respondents, all text except for the stimulus word (\u201cplatery\u201d) was German. Text within the grey box was included only in the treatment condition and missing in the control condition; the screens displayed to respondents did not contain any such boxes.<\/p>\n

 <\/p>\n

2.4 Measures<\/h2>\n

2.4.1 Dependent variable<\/h3>\n

The dependent variable is the sum of foils claimed as known, excluding practice items. The list of 40 reals (e.g., \u201cmuddy\u201d, \u201clistless\u201d, \u201cwrought\u201d), 20 foils (e.g., \u201cpulsh\u201d, \u201cquirty\u201d, \u201cpudour\u201d) and their order were taken from LexTALE, a validated test of proficiency in English (Lemh\u00f6fer & Broersma, 2012; \u201cWord list and instructions for use with other software\u201d, n. d.). Note that the instructions were not taken from the LexTALE and the present study should hence not be considered an application of this test.<\/p>\n

2.4.2 Independent variables<\/h3>\n

Personality and subjective importance of English<\/em>. Narcissism was measured using the 9-item German scale by Malesza et al. (2019), with punctuation corrected as in Bude (2019). Openness, conscientiousness and extraversion were measured by 2 items each, taken from the German BFI-10 (Rammstedt et al., 2013). A tendency to respond in a socially desirable manner was measured by Schnapp et al.\u2019s (2017) variant of Winkler et al.\u2019s (2006) 3-item German version of Paulhus\u2019s (1991) SDE scale.<\/p>\n

Subjective importance of English was measured by an ad-hoc scale comprised of the items \u201cI don\u2019t really care whether my English is good or bad\u201d (reverse scored) and \u201cBeing good at speaking English is important to me\u201d. The preregistration stated that the mean of these two variables would be used if and only if they correlated at r<\/em> = .3 or above; otherwise, two single variables would be used.<\/p>\n

Items were rated on a 5-point scale ranging from \u201cis not accurate at all\u201d to \u201cis completely accurate\u201d and the mean of the items was calculated. Higher values indicate higher narcissism, openness, conscientiousness, extraversion, self-enhancement, and importance of English, respectively.<\/p>\n

Self-rated knowledge of German and English<\/em>. Two questions asked participants to rate their knowledge of German and English, respectively. Available categories were \u201cbasic\u201d, \u201cadvanced basic\u201d, \u201cgood\u201d, \u201cfluent\u201d, \u201cbusiness fluent\u201d, and \u201cmother tongue\u201d.<\/p>\n

Other items<\/em>. The question about gender contained the options \u201cmale\u201d, \u201cfemale\u201d and \u201cdiverse\u201d. Age was measured in categories, \u201cless than 18 years\u201d, \u201c18-19 years\u201d, \u201c20-24 years\u201d, \u201c25-29 years\u201d etc., up to \u201c45-49 years\u201d and \u201c50 or more years\u201d. Country of residence was measured in the four categories \u201cGermany\u201d, \u201cAustria\u201d, Switzerland\u201d and \u201cother country\u201d. The last question, taken from Schnapp (2019), asked respondents whether they had completed this questionnaire before while informing them their answer would have no effect on their obtaining the code on the last page; response options were \u201cyes\u201d, \u201cno\u201d, and \u201cI am not sure\u201d.<\/p>\n

2.5 Exclusion criteria<\/h2>\n

The remaining respondents were excluded if they (i) did not reach the last but one page of the survey, (ii) filled in any of the pages displaying 5 personality items in fewer than 10 seconds; (iii) filled in these 4 pages at least twice as fast as the median respondent (see Leiner, 2019b); (iv) reported that they had filled in the same survey before or (v) were not sure, or (vi) indicated low subjective importance of English by scoring below 2 on the subjective importance of English scale (possible range: 1-5).<\/p>\n

2.6 Hypotheses and statistical analyses<\/h2>\n

It was expected that treated participants would score significantly lower than respondents in the control condition when counts of foils claimed were compared by a test of mean differences (Hypothesis 1a), a bivariate regression (Hypothesis 1b) or a multiple regression using the full set of controls (Hypothesis 1c). Tests of mean differences do not assume equal variances. The full set of control variables in the multiple regression was preregistered. Given that the choice of the appropriate regression technique depends on the distribution of the data, which could not be known in advance, the preregistration did not specify which regression technique would be used. Given the distribution of the data, bivariate and multiple negative binomial regressions were run. In reaction to a comment by the associate editor, another (not preregistered) regression model was added that contained a limited number of control variables. It was chosen on the basis of minimizing the AICc<\/sub><\/em> (a variant of Akaike\u2019s Information Criterion which includes a correction for small sample size; Portet, 2020, p. 123), under the restriction that the treatment indicator must remain in the model.<\/p>\n

The preregistration specified that tests of significance would be one-sided unless noted otherwise; it also noted that, in the multiple regressions only, tests would be two-sided for gender and age, but one-sided for all other variables. This combination of tests of significance would introduce inconsistencies between analyses and even within multiple regressions. Moreover, use of one-sided tests may be misleading to cursory readers (who expect the more common two-sided tests) and some authors argue that \u201c[e]xpectation of a difference in a particular direction is not adequate justification\u201d for using one-sided tests (Bland & Altman, 1994, p. 248). Hence, p<\/em>-values based on the customary two-sided tests are displayed throughout. The original values of the t<\/em>– and \u03c72<\/sup><\/em>-tests are also presented.<\/p>\n

SPSS (versions 20 and 24) was used for all analyses.<\/p>\n

3 Results<\/h1>\n

Data were collected between October 3 and October 16, 2020. In apparent accordance with data collection stopping rules laid down in the preregistration, the survey was closed when it had been taken off the list of surveys at SurveyCircle \u2013 which happens when the code has been used 100 times \u2013 and no new answers had been collected for four days. Analysis of the data revealed that only 99 respondents had actually completed the questionnaire, presumably because the code had been passed on to 1 person who used it without taking the survey. The survey had been started 170 times, for a completion rate of 58%. Of the 71 respondents that started but did not finish the survey, 59 (83%) were redirected to the exclusion page after having rated their knowledge of German as something other than \u201cmother tongue\u201d. 3 (4%) were excluded after reporting an age out of the admissible range, and 1 (1%) after reporting knowledge of English less than \u201cgood\u201d. That is, in 63 of the 170 times the questionnaire was started (37%), participants were excluded because, according to their responses, they did not fulfil the inclusion criteria. In the remaining 8 cases (11%), the questionnaire was simply not finished (drop-out). Of the remaining 99 respondents, 5 were excluded because they stated they had filled in the questionnaire before. Another 6 were excluded because they had filled in at least one of the 4 pages containing personality items in less than 10 seconds; of these, 5 had also filled in the 4 pages at least twice as quickly as the median respondent. The final sample size is n = 88.<\/p>\n

The median response time (from starting the questionnaire to finishing it, measured in full seconds) was 284.50 seconds (mean = 287.09, sd<\/em> = 53.77, min = 182.00, max = 434.00). The two variables measuring subjective importance of English correlated at r<\/em>(86) = .58 (p<\/em> < .001) and were hence combined into a single measure by taking their mean.<\/p>\n

Table 1: Descriptive statistics for full sample (n = 88)<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u00a0<\/strong><\/td>\nMean<\/td>\nSD<\/em><\/td>\nMin<\/td>\nMax<\/td>\n<\/tr>\n
Sum foils claimed (0-20)<\/td>\n1.66<\/td>\n2.02<\/td>\n0.00<\/td>\n8.00<\/td>\n<\/tr>\n
Narcissism (1-5)<\/td>\n2.56<\/td>\n0.62<\/td>\n1.33<\/td>\n4.11<\/td>\n<\/tr>\n
Extraversion (1-5)<\/td>\n3.16<\/td>\n1.10<\/td>\n1.00<\/td>\n5.00<\/td>\n<\/tr>\n
Conscientiousness (1-5)<\/td>\n3.82<\/td>\n0.77<\/td>\n2.50<\/td>\n5.00<\/td>\n<\/tr>\n
Openness (1-5)<\/td>\n3.61<\/td>\n0.90<\/td>\n1.50<\/td>\n5.00<\/td>\n<\/tr>\n
SDE (1-5)<\/td>\n3.38<\/td>\n0.59<\/td>\n2.00<\/td>\n4.67<\/td>\n<\/tr>\n
Subjective importance of English (1-5)<\/td>\n4.18<\/td>\n0.66<\/td>\n2.00<\/td>\n5.00<\/td>\n<\/tr>\n
English \u201cgood\u201d (0-1)<\/td>\n0.30<\/td>\n0.46<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
English \u201cfluent\u201d (0-1)<\/td>\n0.41<\/td>\n0.49<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Age 18-24 (0-1)<\/td>\n0.41<\/td>\n0.49<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Age 30-49 (0-1)<\/td>\n0.15<\/td>\n0.36<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Male (0-1)<\/td>\n0.34<\/td>\n0.48<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Residence not Germany (0-1)<\/td>\n0.06<\/td>\n0.23<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Not Student (0-1)<\/td>\n0.10<\/td>\n0.30<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Abbreviations. SD<\/em>: standard deviation; Min: Minimum; Max: Maximum; SDE: Self-deceptive enhancement<\/p>\n

Theoretical (maximum possible) range in parentheses<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Table 1 shows descriptives, including for dummy variables on the basis of categorical measures, for the full sample (n = 88). No respondent chose the gender category \u201cdiverse\u201d, so that all non-male participants are female. In the cases of the age and country variables, cells with values below 10% were combined.<\/p>\n

Table 2: Tests of mean differences between control (n = 44) and treatment group (n = 44)<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/td>\nMeans<\/td>\nMean diff.<\/td>\nt<\/em><\/td>\np<\/em><\/td>\n<\/tr>\n
<\/td>\ncontrol<\/td>\ntreatment<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Sum foils claimed (0-20)<\/td>\n1.82<\/td>\n1.50<\/td>\n0.32<\/td>\n0.74<\/td>\n0.46<\/td>\n<\/tr>\n
Narcissism (1-5)<\/td>\n2.50<\/td>\n2.61<\/td>\n-0.11<\/td>\n-0.83<\/td>\n0.41<\/td>\n<\/tr>\n
Extraversion (1-5)<\/td>\n3.08<\/td>\n3.24<\/td>\n-0.16<\/td>\n-0.68<\/td>\n0.50<\/td>\n<\/tr>\n
Conscientiousness (1-5)<\/td>\n3.66<\/td>\n3.99<\/td>\n-0.33<\/td>\n-2.05*<\/td>\n0.04<\/td>\n<\/tr>\n
Openness (1-5)<\/td>\n3.59<\/td>\n3.64<\/td>\n-0.05<\/td>\n-0.24<\/td>\n0.81<\/td>\n<\/tr>\n
SDE (1-5)<\/td>\n3.30<\/td>\n3.45<\/td>\n-0.14<\/td>\n-1.15<\/td>\n0.25<\/td>\n<\/tr>\n
Subjective importance of English (1-5)<\/td>\n4.08<\/td>\n4.27<\/td>\n-0.19<\/td>\n-1.39<\/td>\n0.17<\/td>\n<\/tr>\n
English \u201cgood\u201d (0-1)<\/td>\n0.27<\/td>\n0.32<\/td>\n-0.05<\/td>\n-0.46<\/td>\n0.64<\/td>\n<\/tr>\n
English \u201cfluent\u201d (0-1)<\/td>\n0.41<\/td>\n0.41<\/td>\n0.00<\/td>\n0.00<\/td>\n1.00<\/td>\n<\/tr>\n
Age 18-24 (0-1)<\/td>\n0.50<\/td>\n0.32<\/td>\n0.18<\/td>\n1.74<\/td>\n0.08<\/td>\n<\/tr>\n
Age 30-49 (0-1)<\/td>\n0.09<\/td>\n0.20<\/td>\n-0.11<\/td>\n-1.50<\/td>\n0.14<\/td>\n<\/tr>\n
Male (0-1)<\/td>\n0.20<\/td>\n0.48<\/td>\n-0.27<\/td>\n-2.79**<\/td>\n0.01<\/td>\n<\/tr>\n
Residence not Germany (0-1)<\/td>\n0.07<\/td>\n0.05<\/td>\n0.02<\/td>\n0.46<\/td>\n0.65<\/td>\n<\/tr>\n
Dummy not Student (0-1)<\/td>\n0.11<\/td>\n0.09<\/td>\n0.02<\/td>\n0.35<\/td>\n0.73<\/td>\n<\/tr>\n
Abbreviations. Mean diff.: mean differences; SDE: Self-deceptive enhancement<\/td>\n<\/tr>\n
*: p < .05; ** p < .01, two-sided tests<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Theoretical (maximum possible) range in parentheses<\/td>\n<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Table 2 shows comparisons of means for the two groups. Participants in the control group claimed a mean of 1.82 foils, taking a little less than 10% of the opportunities to show SDR. The value in the treatment group is lower, but this difference is not statistically significant and corresponds to an effect size of d<\/em> = .15, typically considered \u201csmall\u201d (Hyde, 2005, p. 581). Hypothesis 1a is hence not confirmed.<\/p>\n

Table 2 also shows that randomization did not lead to balancing on all variables in this small sample. It is hence possible that these differences mask a treatment effect. This suspicion is not confirmed in negative binomial regressions. Neither is a significant effect found in a bivariate model (B<\/em> = -.19, p<\/em> = .47) nor in a model controlling for all variables displayed in Table 1 (B<\/em> = -.13, p<\/em> = .61) nor in a model containing a reduced set of controls chosen on the basis of minimizing the AICc <\/sub><\/em>(B<\/em> = -.05, p<\/em> = .84). Hypotheses 1b and 1c are hence not confirmed. Full tables of regression results and a table of correlations are provided in the appendix.<\/p>\n

4 Discussion and Conclusion<\/h1>\n

4.1 Exclusion of careless respondents<\/h2>\n

In 37% of the cases in which the questionnaire was started, respondents were excluded because their responses indicated they did not fulfil the inclusion criteria. The vast majority of these were excluded after choosing an answer other than \u201cmother tongue\u201d to the question on their knowledge of German. The inclusion criteria were clearly stated on the SurveyCircle main page from which the survey was linked and respondents had confirmed they fulfilled the inclusion criteria, which were explicitly stated, on the questionnaire\u2019s first page. In fairness to respondents, they may have expected a question on their knowledge of English<\/em> (which was presented only on the next page) rather than German<\/em>; however, this explanation presupposes that respondents did not read the question carefully. Clearly, the high exclusion rate indicates a noteworthy level of carelessness on part of the respondents when selecting into the survey and\/or when reading the question. This result of the present study is not out of the ordinary; Nichols and Edlund (2020, Study 3) found that about two thirds of participants recruited via MTurk did not meet the advertised eligibility criteria, as measured by participants\u2019 answers to questions on sociodemography. While both the present and the Nichols and Edlund (2020) studies recruited participants via online platforms, presumably the mode of administration is not the decisive commonality. Rather, it seems likely that results of the two studies were broadly comparable because on both platforms, participants have an extrinsic incentive to finish many surveys quickly in order to collect money (MTurk) or points (SurveyCircle). Researchers collecting data on such platforms may want to consider including extensive checks for eligibility and careless responding.<\/p>\n

4.2 Forced answering and its potential consequences<\/h2>\n

As noted, the questionnaire used in the present study forced respondents to give substantial answers to all items if they were to continue to the next page. On the one hand, this ensures complete data for all respondents that finished the questionnaire. On the other, one may be concerned that this feature increases drop-out of respondents and reduces the quality of the answers. However, these concerns are not clearly supported by the results of randomized experiments on these potential negative effects of forced answering (relative to allowing leaving items unanswered and\/or including a \u201cprefer not to answer\u201d category).<\/p>\n

Concerning drop-out, Sischka et al. (2022) found that forcing answers lead to higher drop-out rates, while other studies found mixed (Roster et al., 2014) or null results (Albaum et al., 2010; Mergener et al., 2015). For the present study, possible drop-out caused by this feature is not a major concern, as only 8 out of 170 (5%) of respondents who started the questionnaire voluntarily dropped out and this study did not aim to describe a specific population.<\/p>\n

Concerning the effect of forced answering on answer quality, Albaum, Roster, Wiley et al. (2010) and Albaum, Roster, and Smith (2014) found no effect on the means of a number of scales, suggesting no effect of the feature on answers in this respect. In contrast, Mergener et al. (2015, p. 8) report significantly lower self-reported participation in crime measured by a single item when answers were forced, but the publication leaves unclear what the results regarding other items were. To the extent that reduced answer quality is a problem in the present study and may be conceptualized as classical measurement error, the main concern would be that the control variables used in the multiple regressions are measured with inflated error. Such error would lower the expectation of the association of control variables with the dependent variable (though this effect is the less pronounced the smaller the sample; Loken & Gelman, 2017, p. 585). This, in turn, would mean that they are less useful in terms of controlling for remaining imbalances between the treatment and control group. This would be a concern had a significant effect of the treatment indicator on the outcome been found in the multiple regressions. One might then be concerned that such a treatment effect would not have been found had control variables been measured with less error. However, in the present study, no significant effect of the treatment was found in the first place. This renders the possible inflation of measurement error in the control variables due to forcing answers a minor worry at most.<\/p>\n

4.3 Socially desirable responding and its reduction via the example treatment<\/h2>\n

Across conditions, some 8% of foils from the LexTALE word list were claimed in the present study. This result was surprising given that the percentages of foils claimed as known reported for online tests using the same word list with other samples were much higher: 42% in a sample of 87 Korean university students pre-screened for proficiency in English (Lemh\u00f6fer & Broersma, 2012), 45% in a sample of 111 Japanese first-year university students of English (Nakata et al., 2020), and 17% in a sample of 72 Dutch respondents drawn from a participant pool that consisted of university students \u201cfor the greatest part\u201d (Lemh\u00f6fer & Broersma, 2012, p. 328). These differences might be due to a host of factors including actual differences in proficiency in English, different motivations for participation and differences in procedures to exclude careless respondents. However, in hindsight it seems apparent that the results are not directly comparable in the first place due to a crucial methodological difference. The present study took the list of stimuli from the LexTALE, but deliberately did not copy its instructions. In the present study, participants were asked to indicate whether they \u201cknow\u201d a word. The studies by Lemh\u00f6fer & Broersma (2012) and Nakata et al. (2020) used the LexTALE instructions (K. M. Lemh\u00f6fer, personal communication, January 4, 2022; T. Nakata, personal communication, January 3, 2022), which inform participants that they will be presented with \u201ca string of letters\u201d and asked to click \u201cyes\u201d if \u201cyou think it is an existing English word\u201d and \u201cno\u201d \u201cif you think it is not an existing English word\u201d (\u201cWord list and instructions for use with other software\u201d, n.d., p. 1). It seems probable that these instructions encourage guessing, and hence indicating \u201cyes\u201d, more than the instructions used in the present study, which asked whether participants \u201cknow\u201d a word. The instructions used here seem right for the present purposes as they unequivocally allow for the interpretation of the foils as zero-prevalence items, which would be doubtful if the instructions had asked respondents whether they merely \u201cthink\u201d the foils presented are English words. It does, however, help explain the fact that the sum of foils claimed is so low in the present study compared to extant results.<\/p>\n

The demonstration used a combination of appeals to honesty and forgiving wording as an example treatment aimed at reducing SDR. It found that the effects of this treatment were in the expected direction, but neither of substantial size nor statistically significant. This result should be interpreted in the light of the study\u2019s small sample size. A related potential limitation is the use of control variables in some of the regression models. These were included to control for any imbalances that might remain between the treatment and control groups despite randomization. However, it is possible that variables which should have been measured and included as controls were not. In particular, one might argue that measures of formal education are crucial in the present context. They were not included due to the assumption that the range of such variables would be restricted in the present sample and that the self-reported proficiency in English would be more relevant. The former assumption is confirmed by the finding that 90% of respondents in the final sample were students (Table 1). Nonetheless, it is possible that the inclusion of fine-grained measures of education would have made a difference in terms of the associations between treatment status and the outcome in the multiple regressions.<\/p>\n

While these limitations should be kept in mind, it is worth noting that the findings presented herein are broadly consistent with extant null and mixed results in studies employing the more-is-better\/less-is-better assumption which test the effects of appeals to honesty (Brener et al., 2004) and forgiving wording (Catania et al., 1996; Contzen et al., 2015; Holtgraves et al., 1997; N\u00e4her & Krumpal, 2012; Ong & Weiss, 2000; Persson & Solevid, 2014; but see Acquisti et al., 2012, Study 1; Raghubir & Menon, 1996, Study 1).<\/p>\n

4.4 The ROME: Strengths, limitations, perspectives<\/h2>\n

More importantly, the study demonstrates how the ROME can be used to assess the impact of SDR reduction techniques. It also shows the advantages of the ROME over other methods for assessing such techniques. In contrast to validations against external measures of constructs, no additional data had to be collected and there is no uncertainty regarding the validity of such data. In contrast to studies based on the more-is-better\/less-is-better assumption, the extent to which respondents engaged in desirable responding could be quantified objectively for both groups, making unnecessary any assumptions about the true values and relations between them.<\/p>\n

Research on SDR reduction techniques is typically conducted and reported under the assumption that the results found using a specific instrument on a specific sample under specific conditions will generalize to other instruments, samples, and conditions. The ROME can, by design, only be used with knowledge questions. Its utility is bounded by the extent to which results found for such knowledge questions generalize to other types of information, such as attitudes or self-reported behaviour. As pointed out by a reviewer, decisions about whether or not to use the ROME to test the efficacy of SDR reduction techniques will hence depend on the research context. Researchers who are primarily interested in the validity of self-reports in a specific applied context (e.g., voting behaviour) may prefer to spend their limited resources on efforts to improve the validity of their instruments which do not involve the ROME (e.g., obtaining official records of voting behaviour as measures of true values). In contrast, survey methodologists interested in testing SDR reduction techniques that are believed to work generally might want to first test them using the ROME, as it offers an easy way to compare survey results to true values. Findings on the efficacy of SDR reduction techniques obtained using the ROME could then be supplemented with results using other topics and validation methods. If the results are similar, confidence in their generality increases.<\/p>\n

Readers wishing to employ this method may wonder how to increase statistical power relative to the present study. The most obvious choice is increasing the sample size. Choice of mode may also have an effect. The present study used a web survey, i.e., a mode without interviewer presence. The weight of the evidence suggests that interviewer presence increases SDR (Krumpal, 2013, p. 2034; Yan & Cantor, 2019, p. 49S). This suggests that modes with interviewer presence have more baseline SDR to work with; it should hence be easier to find effects of SDR reduction techniques in such modes.<\/p>\n

A different approach would change the overclaiming instrument. There are multiple aspects to this. First, even respondents in the control condition did not exhibit a lot of overclaiming in the present study. While this may in part be ascribed to the mode, other instruments or other topics may be preferable. Second, the number of foils could be increased, increasing precision of measurement. The present study included 20 foils and 40 reals. The median response time for these items was merely 2.05 seconds each. Such sections could hence be lengthened without increasing respondent burden by much. These short response times were enabled in the present study by programming the pages in question such that the next page displaying the next item automatically loaded once an answer had been given. This probably shortens response times considerably compared to questionnaires in which respondents have to click a \u201cforward\u201d button after giving their answer.<\/p>\n

These are minutiae, however. More generally, it is worth pointing out that researchers continue to propose new methods for reducing SDR, yet these are either not validated at all or tested using the more-is-better\/less-is-better assumption. As H\u00f6glinger & Diekmann (2017, p. 136) have argued convincingly, using zero-prevalence items is a valid and simple method for validating SDR reduction techniques. In contrast to extant studies using zero-prevalence items, the ROME allows for the application of this idea using a large number of items that may be assumed to be sensitive to socially desirable responding. It hence seems to be a valuable addition to the survey methodologist\u2019s toolbox.<\/p>\n

 <\/p>\n

Appendix<\/h1>\n

Appendix Table 1: Bivariate correlations (r<\/em>) between variables (n = 88)<\/strong><\/p>\n

 <\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/td>\n<\/td>\n1<\/td>\n2<\/td>\n3<\/td>\n4<\/td>\n5<\/td>\n6<\/td>\n7<\/td>\n8<\/td>\n9<\/td>\n10<\/td>\n11<\/td>\n12<\/td>\n13<\/td>\n14<\/td>\n<\/tr>\n
1<\/td>\nSum foils claimed<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u00a0<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
2<\/td>\nTreatment<\/td>\n-0.08<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
3<\/td>\nNarcissism<\/td>\n0.05<\/td>\n0.09<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
4<\/td>\nExtraversion<\/td>\n0.30<\/em><\/strong><\/td>\n0.07<\/td>\n0.62<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
5<\/td>\nConscientiousness<\/td>\n0.25<\/strong><\/td>\n0.22<\/strong><\/td>\n-0.04<\/td>\n0.17<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
6<\/td>\nOpenness<\/td>\n-0.10<\/td>\n0.03<\/td>\n0.07<\/td>\n-0.04<\/td>\n-0.02<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
7<\/td>\nSDE<\/td>\n0.22<\/strong><\/td>\n0.12<\/td>\n0.41<\/em><\/strong><\/td>\n0.38<\/em><\/strong><\/td>\n0.10<\/td>\n0.01<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
8<\/td>\nSubjective importance of English<\/td>\n-0.16<\/td>\n0.15<\/td>\n0.12<\/td>\n0.07<\/td>\n0.12<\/td>\n0.16<\/td>\n0.13<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
9<\/td>\nEnglish “good”<\/td>\n0.01<\/td>\n0.05<\/td>\n-0.10<\/td>\n-0.05<\/td>\n-0.11<\/td>\n0.02<\/td>\n-0.06<\/td>\n-0.33<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
10<\/td>\nEnglish \u201cfluent\u201d<\/td>\n0.04<\/td>\n0.00<\/td>\n0.20<\/td>\n-0.03<\/td>\n-0.05<\/td>\n0.04<\/td>\n0.10<\/td>\n0.11<\/td>\n-0.54<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
11<\/td>\nAge 18-24<\/td>\n0.10<\/td>\n-0.18<\/td>\n-0.13<\/td>\n-0.21<\/td>\n-0.04<\/td>\n0.31<\/em><\/strong><\/td>\n-0.15<\/td>\n0.24<\/strong><\/td>\n-0.03<\/td>\n-0.03<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
12<\/td>\nAge 30-49<\/td>\n-0.04<\/td>\n0.16<\/td>\n0.02<\/td>\n-0.03<\/td>\n0.01<\/td>\n0.00<\/td>\n-0.03<\/td>\n-0.19<\/td>\n0.15<\/td>\n-0.09<\/td>\n-0.35<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n\u00a0<\/em><\/strong><\/td>\n<\/td>\n<\/tr>\n
13<\/td>\nMale<\/td>\n-0.10<\/td>\n0.29<\/em><\/strong><\/td>\n0.29<\/em><\/strong><\/td>\n0.11<\/td>\n-0.10<\/td>\n-0.06<\/td>\n0.18<\/td>\n-0.18<\/td>\n0.06<\/td>\n-0.01<\/td>\n-0.16<\/td>\n0.24<\/strong><\/td>\n\u00a0<\/strong><\/td>\n\u00a0<\/strong><\/td>\n<\/tr>\n
14<\/td>\nResidence not Germany<\/td>\n-0.10<\/td>\n-0.05<\/td>\n-0.25<\/strong><\/td>\n-0.26<\/strong><\/td>\n-0.14<\/td>\n-0.03<\/td>\n-0.10<\/td>\n0.08<\/td>\n-0.05<\/td>\n0.10<\/td>\n0.00<\/td>\n-0.10<\/td>\n-0.18<\/td>\n<\/td>\n<\/tr>\n
15<\/td>\nNot student<\/td>\n-0.10<\/td>\n-0.05<\/td>\n-0.25<\/strong><\/td>\n-0.26<\/strong><\/td>\n-0.14<\/td>\n-0.03<\/td>\n-0.10<\/td>\n0.08<\/td>\n-0.05<\/td>\n0.10<\/td>\n0.00<\/td>\n-0.10<\/td>\n-0.18<\/td>\n-0.10<\/td>\n<\/tr>\n
Correlations which are significant (two-sided tests) at p<\/em> < .05 in bold type; at p<\/em> < .01 in bold type and italics.<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Appendix Table 2: Negative binomial regression on the sum of foils claimed \u2013 bivariate model<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u00a0<\/strong><\/td>\nB<\/em><\/td>\nSE<\/em> (B<\/em>)<\/td>\n95% Wald CI<\/td>\n<\/td>\nSignificance<\/td>\n<\/tr>\n
Upper bound<\/td>\nLower bound<\/td>\n<\/td>\nWald-\u03c72<\/em><\/td>\np<\/em><\/td>\n<\/tr>\n
Constant<\/td>\n.60<\/td>\n.19<\/td>\n.23<\/td>\n.96<\/td>\n<\/td>\n10.26**<\/td>\n.00<\/td>\n<\/tr>\n
Treatment<\/td>\n-.19<\/td>\n.27<\/td>\n-.72<\/td>\n.33<\/td>\n<\/td>\n.51<\/td>\n.47<\/td>\n<\/tr>\n
Dispersion<\/td>\n.98<\/td>\n.28<\/td>\n.56<\/td>\n1.73<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Pearson’s \u03c72<\/em><\/td>\n79.97<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
LL<\/em><\/td>\n-154.68<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
AICc<\/sub><\/em><\/td>\n315.64<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
BIC<\/em><\/td>\n322.79<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
n<\/td>\n88<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
** p<\/em> < .01; * p<\/em> < .05 (two-sided tests)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Appendix Table 3: Negative binomial regression on the sum of foils claimed \u2013 full model<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u00a0<\/strong><\/td>\nB<\/em><\/td>\nSE <\/em>(B<\/em>)<\/td>\n95% Wald CI<\/td>\n<\/td>\nSignificance<\/td>\n<\/tr>\n
Lower bound<\/td>\nUpper bound<\/td>\n<\/td>\nWald-\u03c72<\/em><\/td>\np<\/em><\/td>\n<\/tr>\n
Constant<\/td>\n-1.75<\/td>\n1.40<\/td>\n-4.49<\/td>\n0.99<\/td>\n<\/td>\n1.57<\/td>\n0.21<\/td>\n<\/tr>\n
Treatment<\/td>\n-0.13<\/td>\n0.26<\/td>\n-0.65<\/td>\n0.38<\/td>\n<\/td>\n0.26<\/td>\n0.61<\/td>\n<\/tr>\n
Narcissism<\/td>\n-0.35<\/td>\n0.27<\/td>\n-0.88<\/td>\n0.19<\/td>\n<\/td>\n1.61<\/td>\n0.20<\/td>\n<\/tr>\n
Extraversion<\/td>\n0.47<\/td>\n0.16<\/td>\n0.16<\/td>\n0.78<\/td>\n<\/td>\n8.89**<\/td>\n0.00<\/td>\n<\/tr>\n
Conscientiousness<\/td>\n0.49<\/td>\n0.18<\/td>\n0.14<\/td>\n0.84<\/td>\n<\/td>\n7.61**<\/td>\n0.01<\/td>\n<\/tr>\n
Openness<\/td>\n-0.10<\/td>\n0.15<\/td>\n-0.38<\/td>\n0.19<\/td>\n<\/td>\n0.43<\/td>\n0.51<\/td>\n<\/tr>\n
SDE<\/td>\n0.39<\/td>\n0.23<\/td>\n-0.05<\/td>\n0.84<\/td>\n<\/td>\n2.97<\/td>\n0.08<\/td>\n<\/tr>\n
Subjective importance of English<\/td>\n-0.46<\/td>\n0.22<\/td>\n-0.89<\/td>\n-0.03<\/td>\n<\/td>\n4.46*<\/td>\n0.03<\/td>\n<\/tr>\n
English “good”<\/td>\n0.33<\/td>\n0.34<\/td>\n-0.34<\/td>\n1.00<\/td>\n<\/td>\n0.94<\/td>\n0.33<\/td>\n<\/tr>\n
English “fluent”<\/td>\n0.53<\/td>\n0.32<\/td>\n-0.10<\/td>\n1.17<\/td>\n<\/td>\n2.74<\/td>\n0.10<\/td>\n<\/tr>\n
Age 18-24<\/td>\n0.74<\/td>\n0.28<\/td>\n0.20<\/td>\n1.29<\/td>\n<\/td>\n7.25**<\/td>\n0.01<\/td>\n<\/tr>\n
Age 30-49<\/td>\n0.13<\/td>\n0.38<\/td>\n-0.62<\/td>\n0.88<\/td>\n<\/td>\n0.12<\/td>\n0.73<\/td>\n<\/tr>\n
Male<\/td>\n-0.36<\/td>\n0.31<\/td>\n-0.96<\/td>\n0.24<\/td>\n<\/td>\n1.41<\/td>\n0.24<\/td>\n<\/tr>\n
Residence not Germany<\/td>\n-0.10<\/td>\n0.64<\/td>\n-1.37<\/td>\n1.16<\/td>\n<\/td>\n0.03<\/td>\n0.87<\/td>\n<\/tr>\n
Not student<\/td>\n0.50<\/td>\n0.42<\/td>\n-0.33<\/td>\n1.34<\/td>\n<\/td>\n1.42<\/td>\n0.23<\/td>\n<\/tr>\n
Dispersion<\/td>\n0.40<\/td>\n0.16<\/td>\n0.18<\/td>\n0.88<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Pearson’s \u03c72<\/em><\/td>\n73.73<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
LL<\/em><\/td>\n-137.83<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
AICc<\/sub><\/em><\/td>\n315.32<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
BIC<\/em><\/td>\n347.29<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
n<\/td>\n88<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
** p<\/em> < .01; * p<\/em> < .05 (two-sided tests)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Appendix Table 4: Negative binomial regression on the sum of foils claimed \u2013 reduced model (best fit)<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u00a0<\/strong><\/td>\nB<\/em><\/td>\nSE <\/em>(B<\/em>)<\/td>\n95% Wald CI<\/td>\n<\/td>\nSignificance<\/td>\n<\/tr>\n
Lower bound<\/td>\nUpper bound<\/td>\n<\/td>\nWald-\u03c72<\/em><\/td>\np<\/em><\/td>\n<\/tr>\n
Constant<\/td>\n-1.26<\/td>\n1.12<\/td>\n-3.46<\/td>\n0.94<\/td>\n<\/td>\n1.26<\/td>\n0.26<\/td>\n<\/tr>\n
Treatment<\/td>\n-0.05<\/td>\n0.26<\/td>\n-0.56<\/td>\n0.46<\/td>\n<\/td>\n0.04<\/td>\n0.84<\/td>\n<\/tr>\n
Extraversion<\/td>\n0.32<\/td>\n0.12<\/td>\n0.09<\/td>\n0.55<\/td>\n<\/td>\n7.21**<\/td>\n0.01<\/td>\n<\/tr>\n
Conscientiousness<\/td>\n0.46<\/td>\n0.17<\/td>\n0.13<\/td>\n0.79<\/td>\n<\/td>\n7.64**<\/td>\n0.01<\/td>\n<\/tr>\n
SDE<\/td>\n0.36<\/td>\n0.22<\/td>\n-0.07<\/td>\n0.79<\/td>\n<\/td>\n2.63<\/td>\n0.10<\/td>\n<\/tr>\n
Subjective importance of English<\/td>\n-0.59<\/td>\n0.20<\/td>\n-0.98<\/td>\n-0.20<\/td>\n<\/td>\n8.72**<\/td>\n0.00<\/td>\n<\/tr>\n
Age 18-24<\/td>\n0.57<\/td>\n0.26<\/td>\n0.06<\/td>\n1.08<\/td>\n<\/td>\n4.84*<\/td>\n0.03<\/td>\n<\/tr>\n
Male<\/td>\n-0.51<\/td>\n0.29<\/td>\n-1.07<\/td>\n0.05<\/td>\n<\/td>\n3.24<\/td>\n0.07<\/td>\n<\/tr>\n
Dispersion<\/td>\n0.46<\/td>\n0.18<\/td>\n0.21<\/td>\n1.00<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
Pearson’s \u03c72<\/em><\/td>\n80.26<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
LL<\/em><\/td>\n-140.79<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
AICc<\/sub><\/em><\/td>\n301.89<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
BIC<\/em><\/td>\n321.88<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
n<\/td>\n88<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
** p<\/em> < .01; * p<\/em> < .05 (two-sided tests)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[1]<\/a> The age restriction was introduced to maintain some homogeneity in terms of age without restricting the population of potential participants too much. Given that SurveyCircle appears to be used mostly for data collection for theses, including PhD theses, it was reasoned that participants aged 50 and above would be unusual in ways that might influence their patterns of answers. Although this potential problem is reduced due to randomization, in small samples in particular, imbalances between the treatment and the control group can remain despite randomization.<\/p>\n","protected":false},"excerpt":{"rendered":"

1. Introduction When answering survey questions, respondents often exhibit socially desirable responding (SDR), \u201cthe tendency to give overly positive self-descriptions\u201d (Paulhus, 2002, p. 50). This phenomenon threatens the validity of both univariate statistics and measures of association (Wolter & Preisend\u00f6rfer, 2013, p. 322). Consequently, researchers have long used questions that employ SDR reduction techniques such […]<\/p>\n","protected":false},"author":1800,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[41],"tags":[942,943,941,940,83,399],"class_list":["post-15753","post","type-post","status-publish","format-standard","hentry","category-questionnaire_design","tag-appeals-to-honesty","tag-counterbiasing","tag-forgiving-wording","tag-randomized-overclaiming-method","tag-social-desirability","tag-socially-desirable-responding"],"acf":[],"_links":{"self":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/15753","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/users\/1800"}],"replies":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=15753"}],"version-history":[{"count":30,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/15753\/revisions"}],"predecessor-version":[{"id":19474,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/15753\/revisions\/19474"}],"wp:attachment":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=15753"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=15753"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=15753"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}