{"id":8887,"date":"2017-05-16T18:00:46","date_gmt":"2017-05-16T17:00:46","guid":{"rendered":"http:\/\/surveyinsights.org\/?p=8887"},"modified":"2020-04-14T14:22:06","modified_gmt":"2020-04-14T13:22:06","slug":"testing-the-validity-of-the-crosswise-model-a-study-on-attitudes-towards-muslims%e2%80%a0-2","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=8887","title":{"rendered":"Testing the Validity of the Crosswise Model: A Study on Attitudes Towards Muslims"},"content":{"rendered":"

Introduction<\/strong><\/h1>\n

Social Desirability Bias (SDB) induced by other-deception \u2013 i.e., respondents purposely misrepresenting the truth as a form of impression management motivated by the desire to avoid evaluation by a third party, such as survey interviewers (Goffman, 1959; Nederhof, 1985) \u2013 poses a threat to the quality of survey responses. Early research on socially desirable response patterns has indicated that, even when factual questions are concerned, researchers need to be cautious about the validity of respondents\u2019 answers. It is suggested that the range of invalidity of responses to more sensitive items is \u201csufficient to cause worry, and indicates a great need for further research on the truthfulness of respondents’ statements\u201d \u00a0(Parry, & Crossley, 1950; see also Bradburn, & Sudman, 1979). In addition to this, Cannell, Oksenberg, and Converse (1977) and Cannell, Miller, and Oksenberg (1981) further distinguished the effects of positive and negative distortion in survey responses, linking positive bias to socially desirable and negative bias to socially undesriable behaviour. As such, there has been demand for the improvement of old and development of new techniques to enable survey respondents\u2019 to more truthfully answer survey questions and, thus, to enhance the overall quality of survey data.<\/p>\n

So-called\u00a0 Randomized Response Techniques (RRTs) have been developed to improve the way in which respondents\u2019 anonymity and confidentiality in reporting sensitive attitudes or behaviour in surveys is preserved and to reduce bias due to socially desirable response behaviour (Warner, 1965; Horvitz et al., 1968; Greenberg et al., 1969; Boruch, 1971; Bradburn, & Sudman, 1979; Locander, Sudman, & Bradburn, 1976;\u00a0 Kuk, 1990; Daniel, 1993; see also Lensvelt-Mulders, Hox, van der Heijden, & Maas, 2005; Krumpal, Jann, Auspurg, & von Hermanni, 2015; Fox, 2015; Chaudhuri, 2016). Typically, RRTs rely on randomization devices, which obscure the meaning of an answer communicated to the interviewer and then allow estimating the prevalence of the sensitive characteristic using elementary probability theory (Krumpal et al., 2015). However, while successfully reducing SDB, RRTs may induce bias by design, for example, by providing complex or unusual instructions which immediately alert respondents (Ulrich, Schr\u00f6ter, Striegel, & Simon, 2012) and may lead to instant self-protective \u2018no\u2019-answers (Krumpal et al., 2015).<\/p>\n

The Crosswise Model (CM) was especially designed to overcome these issues (Yu, Tian, & Tang, 2008). It relies on a simple design that only requires respondents to provide one simple yes\/no-answer to a set of two different questions without burdening them with complex instructions or activities (Ulrich et al., 2012, H\u00f6glinger, Jann, & Diekmann, 2014, Kornd\u00f6rfer, Krumpal, & Schmukle, 2014). While one question directly asks about the sensitive attitude, the other one directly enquires about an unrelated non-sensitive topic. However, respondents are instructed to provide only one<\/em> answer to both questions. Response A, if the answers to both questions are the same (either both \u2018yes\u2019 or both \u2018no\u2019); Response B, if the answers to the two questions differ (either \u2018yes\u2019 and \u2018no\u2019 or \u2018no\u2019 and \u2018yes\u2019).<\/p>\n

Simple probability methods then allow estimating the prevalence of the sensitive item, if (1) both behaviours are captured as binary responses; (2) the non-sensitive behaviour is unrelated to the sensitive one; and (3) the non-sensitive behaviour has a known probability distribution p <\/em>(Krumpal et al., 2015): for example, assuming a uniform distribution of birthdays, p<\/em>is equal to 0.25 if the unobtrusive item asks whether a person was born in October, November, or December<\/p>\n

The prevalence of the sensitive item \u00a0\"\"<\/a> can then be estimated (Krumpal et al., 2015; see also Yu et al., 2008; Warner, 1965) by<\/p>\n

\"\"<\/a><\/p>\n

where p<\/em> is the known population prevalence of the non-sensitive item \u2013 e.g., p<\/em> = 0.25 in the birthday example \u2013 and\u00a0\"\"<\/a> is proportion of respondents giving the same answer to both questions in the CM.<\/p>\n

The sampling variance is then obtained as follows:<\/p>\n

\"\"<\/a><\/p>\n

Individual respondent\u2019s responses to the sensitive item are covered up by the design of the question, so that interviewers and researchers are unable to identify whether or not the respondent provided a yes-answer to the sensitive item. By design, the question should thus encourage respondents to more honest self-reports. In addition to this, the CM is also an attractive way of asking about sensitive behaviour across different survey modes (Yu et al., 2008; Krumpal, 2015), given that sufficient statistical power is provided by the survey design to ensure similar precision of the estimates as direct question formats (e.g., Jann, Jerke, and Krumpal, 2012, Ulrich et al., 2012, Kornd\u00f6rfer, Krumpal, & Schmukle, 2014).<\/p>\n

Previous studies have indicated that the CM successfully reduces under-reporting of socially undesirable behaviour, such as plagiarism (Coutts, Jann, Krumpal, & N\u00e4her, 2011, Jann, Jerke, & Krumpal, 2012, H\u00f6glinger, Jann, & Diekmann, 2014), cheating in games (Hoffmann, Diedenhofen, Verschure, & Musch, 2015; H\u00f6glinger, & Jann, 2016), substance abuse (Nakhaee, Pakravan, & Nakhaee, 2013, Shamsipour, Yunesian, Fotouhi, Jann, Rahimi-Movaghar, Saghari, & Asghar Aklhlaghi, 2014), donation of organs (H\u00f6glinger, & Diekmann, 2017), sexual behaviour (Vakilian, Mousavi, & Keramat, 2014; Vakilian, Mousavi, Keramat, & Chaman, 2016), tax evasion (Kornd\u00f6rfer, Krumpal, & Schmukle, 2014; \u00a0Kundt, 2014; H\u00f6glinger, & Jann, 2016), undeclared employment (Schnell, Thomas, & Noack, 2017), shoplifting (H\u00f6glinger, & Jann, 2016), non-voting (H\u00f6glinger, & Jann, 2016), and Xenophobia and Islamophobia (Hoffmann, & Musch, 2016).<\/p>\n

However, little research has attempted to more critically evaluate the CM’s underlying mechanisms and their consequences. For instance, only two studies indicate that the CM is prone to producing false positive results that should not be ignored (H\u00f6glinger, & Diekmann, 2017; H\u00f6glinger, & Jann, 2016). It remains open how the technique performs when (1) high incidence behaviour, i.e. behaviour that occurs more frequently (e.g., Wolter, & Laier, 2014), (2) representative samples (with the exception of Schnell, Thomas, & Noack, 2017), or (3) real-world settings (Krumpal et al., 2015) are concerned. Hence, validation studies exploring how well and in which situations the CM works are required (Jann et al., 2012).<\/p>\n

Our core research aim is to investigate the CM\u2019s validity when high incidence behaviour is concerned looking at prejudice against Muslims in the Austrian context. More precisely, we test the concurrent validity of the CM focussing on self-reported attitudes towards Muslims. We proceed as follows: we begin by discussing our approach in further detail. Next, we present the results of our research. We close with a discussion of our findings and their implications for future research<\/p>\n

 <\/p>\n

Testing the Concurrent Validity of the CM<\/strong><\/h1>\n

Our research relies on data collected by the Austrian National Election Study (AUTNES) (Kritzinger, Johann, Thomas, Glantschnigg, Aichholzer, Glinitzer, Gr\u00fcndl, Oberluggauer, & Wagner, 2016a; \u00a0Kritzinger, Johann, Thomas, Glantschnigg, Aichholzer, Glinitzer, Gr\u00fcndl, Oberluggauer, & Wagner, 2016b). Respondents were sampled from an online access panel provided by Lightspeed GMI\/TNS Option Brussels. The non-probability panel consistsed of approximately 15,000 panelists recruited on an opt-in basis using a combination of indirect approximation methods (Vehovar, Toepoel, & Steinmetz, 2016), such as recruitment by email, co-registration, e-newsletter campaigns, banner placements on webpages and distribution through internal and external networks. In order to participate in the study, respondents had to be at least 16 years old and eligible to vote in the Austrian National Election on Election Day 2013. Eligible panellist were invited to participate in the AUTNES sample according to quotas for gender, age, region, and household size. The panel closely represented the Austrian target populations as described by Statistik Austria (www.statistik.at<\/a>) with over-representation observed for females (\u0394=+1.2%), 20 to 29 year olds (\u0394 =+1.1%) and the cohort of 60 to 69 year olds (\u0394=+5.4%), two person households (\u0394=+10.3%) and the regions Vienna and Lower Austria (both \u0394=+2.6%) as well as the Burgenland (\u0394=+1.1%). Singles (\u0394=-17.7%) as well as citizens aged 70 and above (\u0394=-11.9%) are under-represented (see Kritzinger et al., 2016b).<\/p>\n

We relied on a repeated design: respondents were asked the direct question (DQ) in Wave 1of the panel study, which was fielded between 16 and 28 August 2013. The same respondents were asked the CM question in Wave 4 of the study, which went live on 30 September 2013 until 7 October 2013. The data were collected in Computer Assisted Web Interviews (CAWI). The DQ asked respondents whether or not they would prefer a neighbourhood where no Muslims live<\/em> and provided a simple binary response code (yes\/no). The CM presented respondents with an instruction to provide only one answer (either both answers are the same or they are different) to the following two questions:<\/p>\n

Question A: \u00a0\u00a0 Was your mother\u2019s birthday in January, February, or March?<\/em><\/p>\n

Question B: \u00a0\u00a0\u00a0 If you think of your living arrangements, would you prefer a neighbourhood where no Muslims live?<\/em><\/p>\n

The direct question was coded as 1, if respondents stated that they preferred a neighbourhood without Muslims and 0 otherwise. The CM was coded as 1, if respondents repondented to the two questions differently, and 0, if respondents gave the same answer to both questions.<\/p>\n

Religious beliefs may be prone to SDB driven by prejudiced survey respondents, who do not want to reveal their prejudice (e.g., Cook and Selltiz 1964; Abronson, Wilson, and Brewer 1998). In theory, prejudice is caused by tensions between in-groups (i.e., a large group of citizens within a society sharing a particular characistic) and out-groups (i.e., people who do not share the in-group characteristic), resulting in more frequent negative evaluations of the out-group members (e.g., McConahay, & Hough 1976, Tajfel 1979, Sears 1988, Hogg 2006). Indeed, prior research indicates that estimates of respondents\u2019 holding negative atttiudes towards religious out-groups, such as Muslims, may be affected by SDB, which is visible in under-reporting of the negative attitudes in DQ formats (e.g., Rowatt, Franklin, & Cotton 2005, Velasco Gonz\u00e1lez et al. 2008, Beyer & Krumpal 2010, Krumpal 2012). Indeed, two studies have tested different RRT formats including a CM question in capturing Xenophobia and Islamophobia suggesting that all RRTs perform better than the DQ format, but both have fail to more concisely explore the concurrent validity of the RRT answers in relation (Hoffmann, & Musch, 2016; Ostapczuk, Musch, & Moshagen, 2009). Religious prejudice thus seems to constitute an interesting and relevant field of research to further test the validity of the CM.<\/p>\n

Typically, research on the properties of the CM begins by estimating the prevalence of the sensitive behaviour or attitudes descriptively, i.e. the descriptive results of the CM are compared to the results of the DQ format. Following the \u201cmore is better\u201d-assumption (e.g., Umesh, & Peterson, 1991, Lensvelt-Mulders et al., 2005, Krumpal, 2012, Thomas, Johann, Kritzinger, Plescia, & Zeglovits, 2016),[1]<\/a> higher descriptive estimates of the CM indicate more accurate results, meaning the CM is better able to capture the actual share of people behaving in the respective way or holding the respective attitudes. However, as the DQ format and the CM are employed in different waves and attitudes towards Muslims may have changed during the electoral campaign, the study design does not allow us to assess whether the CM indeed reveals higher and more accurate estimates of respondents reporting negative attitudes towards Muslims.[2]<\/a> Yet, the study design provides important insights to the validity of the CM looking at the concurrent validity of the CM, which we investigate by following a stepwise modelling strategy comparing models predicting negative attitudes towards Muslims as estimated by the DQ format and the CM, respectively. We estimate logistic regressions for binary dependent variables for the DQ and Maximum Likelihood (ML) Regression models for the CM (see Jann, Jerke, and Krumpal, 2012, for the derivation and a discussion of the ML estimator for the CM). Our stepwise procedure is appropriate because we expect that some independent variables indirectly influence respondents\u2019 attitudes towards Muslims (e.g. citizens\u2019 implicit preference for Orientals over Occidentals). Other predictors should have a direct impact (e.g., respondents\u2019 explicit attitudes towards Muslims).<\/p>\n

We include social demographic predictors that allow us to control for a potential bias caused by discrepancies between sample and population: individuals\u2019 gender, age and education as well as political interest. Age is measured continuously in years. We create binary indicators for gender (1: females, 0: males) and education (1: qualification for higher education, 0: no qualification for higher education). Political interest is measured on a 4-point scale, where higher values indicate increased interest.<\/p>\n

Models 1a and 1b also predict respondents\u2019 preferences for a neighbourhood without Muslims looking at their left-right stance. People classifying themselves as more rightist should hold negative attitudes about Muslims (e.g., Rowatt, Franklin, & Cotton, 2005). Respondents\u2019 left-right stance is measured on an 11-point scale, where higher values indicate more rightist views.<\/p>\n

In Models 2a and 2b, we introduce respondents\u2019 self-reported attitudes towards Muslims measured on 5-point agreement scales about the following five statements: (1) \u201cDue to the many Muslims living in Austria, I sometimes feel like a stranger in my own country\u201d, (2) \u201cThe European lifestyle and the Muslims\u2019 lifestyle are easily compatible\u201d, (3) \u201cMuslims should adapt in order to have fewer problems\u201d, (4) \u201cThe Muslims are largely to blame for occasional tensions between Muslims and non-Muslims in Austria\u201d, and (5) \u201cIf I had a child, I would feel uncomfortable if my they married a Muslim\u201d. Higher values indicate agreement with the respective statements. We include all attitudinal items in our models at the same time; the Variance Inflation Factor (VIF), which is throughout smaller than 3, indicates that there is no multicollinearity problem including the five statements measuring respondents\u2019 self-reported attitudes towards Muslims separately.<\/p>\n

Moreover, we consider respondents\u2019 implicit preference for Occidentals over Orientals (Models 3a and 3b). It is measured by an Affective Misattribution Procedure (AMP) (Appendix O2; see Kritzinger et al. 2016b for a detailed description) with values ranging from -1 (=implicit preference for Orientals over Occidentals) to 1 (=implicit preference Occidentals over Orientals) (see Aichholzer and Zeglovits, 2015, for an example how far the AMP can serve as a validation criterion).<\/p>\n

Finally and for completeness, we present full models incorporating all variables at the same time (Models 4a and 4b). Once again, we treat all attitudinal statements as individual explanatory factors to disentangle each individual statements\u2019 influence.<\/p>\n

Following Baker et al. (2013), we would like to emphasise that we consider our results to be indications rather than hard estimates, as they are based on data collected on a non-probability panel.<\/p>\n

 <\/p>\n

Results <\/strong><\/h1>\n

Table 1 summarizes our models predicting negative attitudes towards Muslims. Looking at Models 1a and 1b, it is noteworthy that we find a statistically significant negative influence of responsents with high educational attainment on negative attitudes towards Muslims in the DQ, which substantively implies that well educated respondents are less likely to hold negative attitudes towards Muslims. We do not find a similar effect looking at the CM. A Wald Test indicates that the coefficients of both models significantly differ (p < 0.05). Morever, our results indicate positive influences of respondents\u2019 left-right self-placement on negative attitudes towards Muslims in both models: more rightist citizens appear to hold more negative attitudes towards Muslims in their neighbourhood.<\/p>\n

In addition to the socio-demographic controls, Models 2a and 2b include self-reported attitudes towards Muslims. At the first glance, the results reveal relatively consistent results regarding reports about the influence of negative views about Muslims on respondents\u2019 preferences to live in a neighbourhood without Muslims across the CM and DQ model: the coefficients of four critical attitudes point in the same positive direction in the DQ and CM condition indicating that negative attitudes towards Muslims and foreigners correspond with a lower likelihood of wanting to live in a neighbourhood with Muslims. However, \u00a0the liberal statement suggesting that the European and Muslim lifestyles are compatible stands out: the results reveal a negative influence in the DQ format suggesting that more liberal views correspond with a lower likelihood to live in a neighbourhood with Muslims (which we would expect). However, the coefficient indicates a positive direction for the CM question (which we did not expect). A Wald Test indicates that the coefficients differ significantly (p < 0.001).<\/p>\n

Looking at implicit preferences for Occidentals over Orientals in Models 3a and 3b, the results imply that those respondents also hold more negative attitudes towards Muslims, as suggested by the positive direction of the AMP coefficient.<\/p>\n

For completeness, we also present full results in Models 4a and 4b, which reinforce some of the inconsistencies regarding explicitly stated attitudes towards Muslims and which additionally indicate a statsically different coefficient for the statement \u201cMuslims are largely to blame for occasional tensions between Muslims and Non-Muslims in Austria\u201d \u00a0(p < 0.05).<\/p>\n

Table 1: Predicting Attitudes towards Muslims using the CM (a) and the DQ (b)<\/strong><\/p>\n

\"\"<\/a><\/p>\n

 <\/p>\n

Discussion and Conclusion<\/strong><\/h1>\n

The Crosswise Model has repeatedly been found to reliably reduce Social Desriabiliy Bias when estimating low incidence behaviour. However, little is known about its validity when high incidence behaviour or a real-world setting is concerned. By investigating the concurrent validity of the Crosswise Model on reported negative attitudes towards Muslims in Austria, we contribute to a study that may help us to further evaluate the properties of the Crosswise Model.<\/p>\n

We would have expected to find that our predictors of negative attitudes towards Muslims point in the same direction across the Direct Question format and the Crosswise Model. However, our results raise further questions about the validity of the Crosswise Model with regard to the concurrent validity in predicting negative attitudes towards Muslims. While we identify some factors that appear to predict the estimates obtained on the basis of the Direct Question format and Crosswise Model well (i.e., regression coefficients point in the same direction and reach conventional levels of statistical significance), other results are rather puzzling.<\/p>\n

Traditional indicators of liberalism, such as respondents\u2019 self-reported placement on the left-right continuum, as well as other attitudes towards Muslims seem to produce fairly consistent results indicating that both question formats are able to estimate respondents\u2019 rejection of Muslims as neighbours quite well. However, we are particularly puzzled by some insignificant coefficients of self-reported attitudes towards Muslims in the Crosswise Model, which even point into the opposite direction than in the DQ model. As we rely on strong measures that should indicate an influence in the same direction, such as other negative attitudes towards Muslim and implicit associations, we believe that our results point in the direction that the Crosswise Model may come with problems that have not been fully uncovered yet. As Schnell, Thomas, and Noack (2017) indicate the success of the Crosswise Model in capturing sensitive behaviour or attitudes may be related to respondents cognitive abilities, especially their educational background: well educated respondents may be better able to understand the core principle of the Crosswise Model protecting their anonymity and confidentiality, while less well educated respondents are more concerned about the unusual question format. Our rather peculiar finding regarding respondents education in this study, may be interpreted as support for their findings pointing in a similar direction. Furthermore, we may observe that more sophisticated respondents are not genuinely less negative about Muslims as studies based on Direct Question formats suggests, but that they are simply more aware or more prone to give socially acceptable answers per se, which some research on this matter proposes (e.g., Jackman, 1973, Wagner, & Zick, 1995, Ostapczuk, Musch, & Moshagen, 2009).<\/p>\n

To conclude, although the Crosswise Model has previously been commended to be a promising method to reduce Social Desirablity Bias, our research suggests that we yet do not fully understand under what circumstances the method is applicable. Without completely questioning the validity of the Crosswise Model, we encourage future research to more rigorously test questions using the question format in larger population studies, ideally based on probability samples; to investigate high and low incidence behaviour or attitudes; and to study the performance of the method in different countries. In order to fully understand under what conditions the Crosswise Model works and to further develop the technique, it indispensable to address these issues.<\/p>\n

 <\/p>\n

Online Appendix<\/a><\/strong><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

\n
\n
\n

[1]<\/a> Note that Cannell, Oksenberg, and Coverse (1981) were one of the first scholars to theorize and link socially undesirable behaviour to under-reporting and socially desriable behaviour to over-reporting of attitudes providing the basic idea of the so-called \u201emore is better\u201c assumption in randomize response techniques (see Umesh, & Peterson, 1991, for a critical review).<\/p>\n<\/div>\n

\n

[2]<\/a> We would like to report the prevalence of negative attitudes towards Muslims anyway. Both measures estimate almost identical proportions of negative attitudes towards Muslims (DQ: 59.8%, SE=1.4, n=1205; CM: 60.2%, SE=2.9, n=1205).<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Introduction Social Desirability Bias (SDB) induced by other-deception \u2013 i.e., respondents purposely misrepresenting the truth as a form of impression management motivated by the desire to avoid evaluation by a third party, such as survey interviewers (Goffman, 1959; Nederhof, 1985) \u2013 poses a threat to the quality of survey responses. Early research on socially desirable […]<\/p>\n","protected":false},"author":489,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[41],"tags":[395,394,409],"class_list":["post-8887","post","type-post","status-publish","format-standard","hentry","category-questionnaire_design","tag-attitudes-towards-muslims","tag-crosswise-model","tag-validity"],"acf":[],"_links":{"self":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/8887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/users\/489"}],"replies":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8887"}],"version-history":[{"count":30,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/8887\/revisions"}],"predecessor-version":[{"id":13899,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/8887\/revisions\/13899"}],"wp:attachment":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8887"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}