{"id":16693,"date":"2022-12-22T14:59:23","date_gmt":"2022-12-22T13:59:23","guid":{"rendered":"https:\/\/surveyinsights.org\/?p=16693"},"modified":"2023-01-05T12:38:19","modified_gmt":"2023-01-05T11:38:19","slug":"an-overview-of-the-scales-characteristics-for-10-well-established-face-to-face-social-science-surveys","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=16693","title":{"rendered":"An overview of the scales’ characteristics for 10 well-established face-to-face social science surveys"},"content":{"rendered":"

1. Introduction<\/h1>\n

Since decades, surveys have been the main source of data in many studies (Saris and Gallhofer, 2014). How surveys are designed matters 1) for respondents, a bad experience participating in one survey can lead to break-off and\/or discourage respondents to participate in future surveys; and 2) for researchers, since it affects the representativeness (who participates, who breaks-off) and data quality (item non-response, non-differentiation, etc.), and may therefore affect the substantive conclusions reached (Saris and Gallhofer, 2014).<\/p>\n

Therefore, a lot of literature has been produced to help researchers designing high quality surveys (e.g. Schuman and Presser, 1981; Sudman and Bradburn, 1982; Alwin and Krosnick, 1991; Alwin, 2007; Dillman, 2011; Saris and Gallhofer, 2014). This literature discussed the numerous choices that have to be made when designing a questionnaire (number of answer categories, use of labels, etc.), and used empirical data (mainly survey experiments) to study the impact of some of these decisions on data quality, measured in different ways.<\/p>\n

Previous literature also focused on providing practical recommendations and specifying best practices on how to design questionnaires to maximize data quality, representativeness, and respondents’ satisfaction with their survey participation.<\/p>\n

Even when these recommendations are clear and there is large agreement about what the best practices are, we can find discordance between what is done in practice and what the literature recommends. For instance, many studies recommend to avoid the use of agree\/disagree (from now on A\/D) items, in which a statement is presented, and respondents are asked to what extent they agree\/disagree with this statement (e.g. Krosnick, 1991; Alwin, 2007; Saris et al., 2010; Revilla and Ochoa, 2015; H\u00f6hne, Revilla and Lenzer, 2018). However, Revilla (2017) found that 39.2% of the surveys implemented in an opt-in online panel in Spain included A\/D items. Thus, there is a gap between the academic literature recommendations and the way questions are designed in practice, which might be due to the necessity to do trade-offs between data quality, comparability (across waves or countries), and costs.<\/p>\n

In this paper, our main goal is to provide information about the response scales used in practice by 10 well-established social science surveys conducted by face-to-face, mainly in Europe and America. We focus on five aspects: the response scales’ evaluative dimension (A\/D or item specific), the number of answer categories, the use of grids, of check-all-that-apply format (CATA), and of fixed-reference points. We selected these aspects because 1) the agreement in the literature about what is preferable is high and 2) their impact on data quality seems to be important (Smyth et al. 2006; Saris and Gallhofer, 2007; DeCastellarnau, 2018). Section 2 presents the main recommendations that the literature provides for each of these five aspects. Then, Section 3 presents the data used in this study, Section 4 explains how the analyses were done, and Section 5 reports the main results. Finally, Section 6 concludes.<\/p>\n

2. Recommendations from the literature<\/h1>\n

2.1 Avoid A\/D scales<\/h2>\n

The first aspect is the scales\u2019 evaluative dimension. The literature differentiates A\/D and item-specific (IS) questions. The format of A\/D questions is always the same: first a statement is presented (e.g. “I am satisfied with my life”) accompanied by a dis\/agree scale, commonly featuring different levels of intensity. In contrast, IS questions are usually formed by a request for an answer (e.g. “Are you satisfied with your life?”) and an answer scale matching the concept under evaluation (e.g. dissatisfied to satisfied).<\/p>\n

A\/D questions are quicker to design, since all questions use a similar scale, whatever the concept to be measured. Only the statement needs to be formulated. A\/D questions often present the request for an answer only once, followed by several items for which the respondents need to say to what extent they agree\/disagree.<\/p>\n

However, the literature recommends to avoid A\/D questions and use instead IS ones, for different reasons. First, the cognitive process is more complex for A\/D than IS questions (Saris et al., 2010; Revilla, Saris and Krosnick., 2014; Kunz, 2017). Second, the use of A\/D questions leads to weariness (H\u00f6hne, Schlosser and Krebs, 2017) because the format remains the same except for the statement to be judged. Furthermore, acquiescence bias (a tendency to agree with any statement) is expected with A\/D scales (Schuman and Presser, 1981). Moreover, results differ depending on the statement used (e.g. “I am satisfied with my life”, versus “I am dissatisfied with my life”). Finally, A\/D questions have lower measurement quality (Saris et al., 2010; Revilla and Ochoa, 2015).<\/p>\n

2.2 Use 6 to 11 answer categories for IS scales and maximum 5 for A\/D scales<\/h2>\n

The decision about the number of scale points depends on the type of variables and the scales\u2019 evaluative dimension (A\/D or IS).
\nConcerning the type of variables, following Dillman (1978), we distinguish between questions measuring behaviors, attitudes, beliefs and attributes: behavioral questions are about people\u2019s actions (what they have done, currently do, or plan to do). Attitude questions are about what people (dis)like requiring them to indicate whether they have positive or negative feelings. Belief questions are about what people think is true or false eliciting their perceptions of past, present, or future reality. Finally, attributes questions are related with personal or demographic characteristics.
\nOne of the most important rules to design an effective answer scale is to make it complete (Krosnic and Presser, 2010): the answer scale has to cover all possible options. When measuring attributes or behaviors, the adequate number of scale points depends on the object measured. For example, if the interest is in which region the respondent lives, all regions in a given country have to be offered as response categories. When it comes to attitude or belief questions, we have to differentiate between A\/D and IS questions to decide about the scale length (Revilla et al., 2014).<\/p>\n

Literature concerning attitude or belief IS questions suggests using six to 11 points (Alwin and Krosnick, 1991; Alwin, 1997; Asensio and Revilla, 2021; Revilla et al. 2014; Saris and Gallhofer, 2014; Revilla and Ochoa, 2015). On one hand, the theory of information (Garner, 1960) states that for bipolar concepts, while 2-point scales measure the direction of an attitude or belief, 3-point scales (and longer scales with odd numbers) also measure the neutral middle point, and scales with more points additionally measure the intensity of the attitude or belief. Moreover, Alwin and Krosnick (1991) suggest that enough answer categories are needed to allow scale differentiation. With too few categories, people with different true values may have to answer the same as there are no more option to choose from. On the other hand, the literature does not recommend more than 11 points because more categories can lead to ambiguity and non-discrimination between categories (Schaeffer and Presser, 2003; Krosnick and Presser, 2010).<\/p>\n

For A\/D scales, even though such scales are not recommended (see section 2.1), Revilla et al. (2014) found that five points is better than seven and 11 points. Furthermore, for A\/D scales, having five points instead of more, reduces the extreme response style bias, understood as the tendency to disproportionally use the extreme response categories in a rating scale (Weitjers, Cabooter and Schillewaert, 2010).<\/p>\n

2.3 Avoid grid format<\/h2>\n

Most questions are presented using an item-by-item format. However, when a set of questions shares the same response scale, it is possible to use a grid format. Then, the main request for an answer (e.g. \u201cHow much do you trust the following institutions?\u201d) and the response scale (e.g. 0 \u201cNo trust\u201d to 5 \u201cComplete trust\u201d) are usually repeated only once for a series of items (e.g. the parliament, the legal system, the police).<\/p>\n

In visual modes of data collection (e.g. web or face-to-face using showcards), the items are usually presented in rows, while the common set of response options is usually presented in columns (Couper et al., 2013, p. 322).
\nGrid format allows the information (Revilla, Toninelli and Ochoa, 2016) to be condensed, which can be timesaving for respondents. Indeed, completion times are lower for grids than item-by-item formats (Couper, Traugott and Lamias, 2001; Bell, Mangione and Kahn, 2001; Tourangeau, Couper and Conrad, 2004). Furthermore, higher inter-item correlation has been found when using grids (Tourangeau et al., 2004). This is often seen as an indicator of higher quality, as it improves the reliability (Ferketich, 1991).<\/p>\n

However, many studies recommend avoiding grids for different reasons (Poynter, 2001; Wojtowicz, 2001; Dillman, 2011). First, shorter completion times seem to be due to respondents not putting enough efforts into answering the questions, and not to the task being easier. For instance, Couper et al. (2013) suggested that shorter completion times may be consequence of choosing the same answer category for all items which is a suboptimal responding strategy known as non-differentiation or straightlining. Other studies showed that grid formats tend to increase the number of missing items (Iglesias, Birks and Torgerson, 200; Manfreda, Batageli and Vehovar, 2002), which in turn, can also reduce completion times. Toepoel et al. (2009) showed that this item-missing trend systematically increased with the number of items inside the same grid. Second, the higher inter-item correlations in grids, which was sometimes considered as an indicator of higher quality, seems to be due to higher non-differentiation and systematic measurement errors (Peytchev, 2005). Third, respondents\u2019 satisfaction with their survey experience decreases when grids are used (Thorndike et al., 2009; Toepoel et al., 2009). This, in turn, can lead those respondents to refuse to participate in future surveys. Finally, the use of grids increases respondents\u2019 break-off rates (Puelston and Sleep, 2008).<\/p>\n

2.4 Avoid check-all-that-apply formats<\/h2>\n

Check-all-that-apply (CATA) is a question format where respondents are asked to select from a list all the options applying to them (Smyth et al., 2006). For instance, \u201cPlease, indicate all the devices you use to go online: PC, tablet, smartphone\u201d. The alternative to this format is usually called \u201cforced-Choice\u201d: in this case, the respondents are asked to provide an answer (typically yes or no) for each item in the list (Smyth et al., 2006). For instance, \u201cDo you use the following devices to go online: PC? Yes\/No; Tablet? Yes\/No; Smartphone? Yes\/No\u201d. However, it is possible to add an option \u201cprefer not to answer\u201d or even to let respondents continue without providing an answer when using a \u201cforced-Choice\u201d format.<\/p>\n

In oral modes like face-to-face it might be difficult to distinguish between CATA and forced-choice. However, when the face-to-face interviews use showcards, then the distinction is clearer.<\/p>\n

CATA items are common in surveys mainly for two reasons: completion times are shorter (Sudman and Bradburn, 1982) and the design is efficient, in the sense that it allows respondents to select several options (Smyth et al., 2006). However, the literature recommends avoiding CATA format because, first, its nature encourages weak satisficing of the respondents, who tend to report less items than in forced-choice formats (Smyth et al., 2006; Jaeger et al., 2014). Even if the higher reporting in forced-choice formats could be partly due to acquiescence bias, forced-choice has a higher external validity (Revilla, 2015). Lau and Kennedy (2019) results based on undesirable but common events (lost job or being arrested) also suggest that respondents report less items than they should in the CATA format. Second, the longer response times for forced-choice formats can be positive, if due to a more careful answer process. Finally, the meaning of not selecting an option in the CATA format is unclear: it could be that the option effectively does not apply to the respondents, that respondents are neutral, or that they overlooked it (Sudman and Bradburn, 1982). In contrast, forced-choice formats allow better differentiation because options are marked negatively (Smyth et al., 2006).<\/p>\n

2.5 Use fixed reference points<\/h2>\n

Fixed reference points are labels that \u201cset no doubt about the position of the reference point on the subjective scale in the mind of the respondent\u201d (Saris and Gallhofer, 2014, p.110). For example, words as \u201ccompletely\u201d or \u201cextremely\u201d define an abstract object to its maximum value, whereas other words such as \u201cslightly\u201d or \u201csomewhat\u201d can be interpreted in different ways by the respondents.
\nIn behavioral questions, it is often possible to use fixed reference points for each answer category, for instance when measuring a frequency (e.g., \u201conce a week\u201d, \u201ctwice a week\u201d, etc), or a duration (e.g. \u201c0h00 to 5h00\u201d, \u201c5h01 to 10h00\u201d, etc.), instead of using vague quantifiers (e.g. \u201coften\u201d or \u201ca lot\u201d). For attitude and belief questions, fixed reference points are usually possible for the end points and, for bipolar concepts, the midpoint. Attribute questions usually use fixed reference points for all answer categories (e.g. regions, education, income).<\/p>\n

The literature recommends the use of fixed reference points whenever possible. First, when fixed reference points are not provided respondents might interpret the answer categories differently. Then, the assumption of equality of the response function does not hold, i.e. respondents with different true values can chose the same answer category, and respondents with similar true values can chose different answer categories (Saris and De Rooij, 1988). These variations in the response function can be prevented to a large extent by using at least fixed reference points for the end points of the scale (Saris et al., 1988; Batista-Foguet and Saris, 1988).<\/p>\n

Second, when the response function has been equalized due to the use of fixed reference points, the measurement error that may arise from its variation is reduced (Scherpenzeel, 2003). In line with this, Revilla and Ochoa (2015) found that the use of two fixed reference points in the end points slightly increases measurement quality.<\/p>\n

3. Data<\/h1>\n

We selected 10 well-established surveys, recognized for their quality and contribution to the production of knowledge across the world. All surveys cover broad social sciences topics and are implemented as face-to-face interviews with showcards across different countries, mainly in Europe, Latin America, and the USA.<\/p>\n

We focus on face-to-face surveys because they are usually considered the ones with highest data quality and because most of the studies that studied the impact of changing the response scales and allowed to formulate recommendations were conducted face-to-face. Moreover, we focus on Europe and America because literature concerning survey methods has been mostly produced using data from these areas. It is not clear whether the current recommendations from the survey methodology literature apply in a similar way for other areas using different languages, alphabets, and linguistic rules (e.g. Arab or Asiatic countries).
\nThe selected surveys are the following: Las Am\u00e9ricas y el Mundo (AMERICAS), European Social Survey (ESS), Eurobarometer (EURO), European Values Study (EVS), General Social Survey (GSS), Proyecto de Opini\u00f3n P\u00fablica de Am\u00e9rica Latina (LAPOP), Latinobar\u00f3metro (LATINOBAR), Quality of life in European Cities \u2013 Flash Eurobarometer (QEC), Survey of Health, Ageing and Retirement in Europe (SHARE), and World Values Survey (WVS). Table 1 provides information about each of them.<\/p>\n

Table 1. Overview of the key characteristics of the selected surveys and their questionnaires<\/strong><\/p>\n

To the table<\/a><\/p>\n

Most of these surveys exist for many years. However, we are interested in the current state of their questionnaires, to compare it with the current recommendations of the literature. Thus, we analyzed the last questionnaire available in each case when this research was conducted (2019).<\/p>\n

4. Analyses<\/h1>\n

First, we retrieved the last questionnaires available from the surveys’ websites. From those questionnaires, we coded the following characteristics for each question:<\/p>\n