{"id":20179,"date":"2025-11-18T15:45:39","date_gmt":"2025-11-18T14:45:39","guid":{"rendered":"https:\/\/surveyinsights.org\/?p=20179"},"modified":"2025-12-22T16:27:32","modified_gmt":"2025-12-22T15:27:32","slug":"the-relationship-between-measurement-error-representation-bias-language-and-country-a-comparative-analysis-using-the-european-social-survey-rounds-5-to-7","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=20179","title":{"rendered":"The Relationship Between Measurement Error, Representation Bias, Language, and Country: A Comparative Analysis Using the European Social Survey (Rounds 5 to 7)"},"content":{"rendered":"

Introduction<\/h1>\n

In a world where public opinion, often captured through surveys, informs policy-making decisions, the accuracy of the measurement instruments (i.e., survey items) used to elicit respondents’ opinions is crucial (Groves, 2005). Measurement quality is the basis of credible research (Alwin, 2007; Repke et al., 2024; Saris & Gallhofer, 2007). It determines to what extent the collected data reflect the realities of the population under study (Biemer, 2010). High-quality instruments with minimal measurement error ensure that the nuanced opinions, attitudes, and behaviours of individuals are captured well, ultimately allowing researchers and policymakers to draw meaningful conclusions. However, designing such high-quality measurement instruments is a complex task, especially in cross-national research projects that must account for many languages, cultures, and institutional settings (Boer et al., 2018; Davidov et al., 2014; Harkness et al., 2003; Saris, 1998). Even small differences in the wording of the translations, for instance, can alter the meaning of an item or change the properties of an answer scale (e.g., turning a bipolar scale into a unipolar scale), introducing additional error at the measurement level (Repke & Dorer, 2021).<\/p>\n

Beyond measurement challenges, large cross-national surveys\u2014such as the European Social Survey (ESS)\u2014must also address factors that can introduce bias at the level of representation (see, e.g., Rybak, 2023). Even in surveys based on probability samples, like the ESS, not all sampled individuals respond to the survey, most commonly because they either refuse to participate or cannot be contacted (Groves & Couper, 1998; Stoop, 2005). If survey nonresponse is more prevalent among certain population groups\u2014such as individuals with lower levels of education\u2014these groups may be underrepresented in the survey, potentially leading to nonresponse bias (Groves & Peytcheva, 2008), a key source of representation bias. Ideally, all participating countries would utilize the same sampling frame and apply identical sampling procedures, resulting in national samples that represent their populations in a comparable way. In practice, however, national coordination teams often need to implement different sampling designs and work with different data collection agencies (Jowell et al., 2007). This reduces control over the sampling process and results in samples that vary in how well they represent each country\u2019s population, further adding to representation bias (Menold, 2014). In the following, we use the term representation bias<\/em> to refer to differences in the socio-demographic composition between the survey respondents and the target population.<\/p>\n

Error Sources and Quality Estimation<\/h2>\n

In survey methodology, the two primary sources of error and bias\u2014those related to measurement and those related to representation\u2014are typically treated as two independent dimensions within the Total Survey Error (TSE) framework. This conceptual model links each step of the survey process\u2014design, data collection, and estimation\u2014to these two error sources (for a graphical illustration, see Groves & Lyberg, 2010; Groves et al., 2011). Measurement errors affect the accuracy of individual responses, while representation biases arise when the achieved sample of survey respondents does not adequately reflect the target population.<\/p>\n

Although this distinction is widely accepted, these two error sources can also interact in meaningful ways, especially in cross-cultural surveys, where both measurement quality and representation can vary across countries and languages. For example, a translation\u2019s degree of adaptiveness versus functional equivalence may influence measurement error differently depending on respondents\u2019 cultural backgrounds (Repke & Dorer, 2021). In contrast, different sampling strategies may over- or underrepresent certain linguistic or demographic groups.<\/p>\n

Conceptually, measurement error<\/em> refers to the discrepancy between the observed responses to a survey item and the true value of the measured construct. Within the framework of classical test theory, this is expressed as the observed score being the sum of the true score and an error component (Alwin, 2007; Gulliksen, 2013; Lord & Novick, 2008; Saris & Gallhofer, 2007). This error is typically assumed to be random, resulting from factors such as respondents accidentally providing incorrect answers or interviewers making recording errors. However, measurement error can also contain a systematic component, as in the true score model proposed by Saris and Andrews (1991). This type of error\u2014often referred to as method effect\u2014stems from features of the survey item\u2019s design (e.g., question wording or the format of the answer scale), which can influence how respondents interpret and answer survey questions (Saris & Gallhofer, 2007).<\/p>\n

Researchers typically estimate the size of the average measurement error empirically, using data collected from actual survey respondents. However, these estimates are not immune to representation issues. If the sample does not adequately represent the target population, the resulting measurement quality estimates may be biased, reflecting the characteristics of the sampled respondents rather than the population under study (Biemer, 2010). For instance, if the sample disproportionally includes women or younger individuals, the estimated measurement error may reflect how these specific groups interpret and react to survey items\u2014potentially diverging from how men or older respondents would respond. These group-specific response tendencies are often referred to as response functions (see Saris, 1988). In this way, measurement error estimates are inherently tied to the composition of the sample used for analysis.<\/p>\n

One established method for estimating measurement quality is the multitrait-multimethod (MTMM) approach, originally proposed by Campbell and Fiske (1959) and further developed by Saris and Andrews (1991). In an MTMM design, several related constructs (known as traits) are repeatedly measured using different methods\u2014commonly different response scales or question formats. The goal is to disentangle the variance in survey responses attributable to the construct of interest from the variance introduced by the measurement method.<\/p>\n

The typical MTMM experiment uses a 3×3 design in which three traits are each measured with three different methods. This setup allows researchers to estimate reliability (the consistency of the measure) and validity (the degree to which the measure reflects the intended construct) of each measurement and to isolate systematic method effects. To analyze MTMM data, researchers can use different statistical models. The classical MTMM model applies confirmatory factor analysis (CFA) to account for trait and method variance in the observed correlations (e.g., J\u00f6reskog, 1970; Althauser et al., 1971; Alwin, 1973). An alternative is the true score model, which decomposes measurement quality into reliability and validity and allows for the estimation of the method effect (for detailed information, see Saris and Andrews, 1991; Saris et al., 2022).<\/p>\n

Although measurement error may consist of multiple components (see, e.g., Backstr\u00f6m et al., 2025), this paper focuses on the systematic component of measurement error (i.e., the method effect), which we define as:<\/p>\n

measurement error = 1 \u2013 validity<\/p>\n

While the discussed models provide valuable estimates of measurement quality, their assumptions do not always hold in practice. A notable example is the assumption of independent repeated measurements in MTMM designs. Schwarz et al. (2020) demonstrate that memory effects\u2014respondents recalling their previous answers and replicating them\u2014can violate this assumption, leading to biased estimates of measurement quality. This underscores the need for caution when interpreting MTMM-based estimates.<\/p>\n

Comparability in Practice: The ESS<\/h2>\n

The ESS is a biennial, large-scale cross-national survey that collects data on social attitudes, political opinions, preferences, and behaviours across many European countries. Established in 2001, the ESS employs random probability sampling to generate representative samples of individuals aged 15 and above residing in private households in participating countries. To date, 39 countries have taken part in at least one ESS round (European Social Survey, 2024; see also www.europeansocialsurvey.org<\/a>).<\/p>\n

Widely regarded for its rigorous methodological standards, the ESS has been recognized as a benchmark for high-quality cross-national survey research (Jowell et al., 2007; Stoop et al., 2010). As part of its commitment to data quality, the ESS systemically evaluates measurement error using MTMM experiments based on the true score model by Saris and Andrews (1991; for an example, see Poses et al., 2021). So far, MTMM experiments have been conducted in ESS rounds 1 to 9, covering variables such as political efficacy, media use, social trust, and attitudes toward immigration.<\/p>\n

These experiments have also laid the foundation for the Survey Quality Predictor (SQP), an open-access software designed to predict the quality of survey items. The tool is based on a meta-analysis of the formal and linguistic characteristics of thousands of survey questions and their quality estimates, primarily obtained from MTMM experiments conducted within the ESS (Saris et al., 2000; Saris, 2001; Saris et al., 2004, 2011; Felderer et al., 2024). It is available online at sqp.gesis.org<\/a>. Not only was SQP developed within the ESS context, but it has also been applied actively in the survey\u2019s operational processes, including the development of the source questionnaire and during translation checks (see, e.g., European Social Survey, 2018).<\/p>\n

Additionally, the ESS monitors the socio-demographic composition of its samples to ensure that they adequately reflect the target population. In rounds 5 to 7, sample compositions (e.g., gender, age, work status, and nationality) were compared with external benchmark data from the European Labour Force Survey (EU-LFS; see https:\/\/ec.europa.eu\/eurostat\/web\/microdata\/european-union-labour-force-survey<\/a>). Note that, because these data are available only at the country level, they cannot be used to assess the socio-demographic representation of linguistic subgroups within countries.<\/p>\n

The Current Study<\/h2>\n

This paper investigates the interplay between measurement error, representation bias, language, and country in the ESS. By exploring how measurement errors in survey items vary across different European countries and linguistic groups, this study seeks to uncover factors influencing data quality in comparative social research. It addresses two key research questions (RQ):<\/p>\n

    \n
  1. To what extent are the measurement errors of survey items associated with representation bias?<\/li>\n
  2. Do measurement errors of survey items vary across different country-language groups? And if yes, how?<\/li>\n<\/ol>\n

    <\/h2>\n

    Data and Method<\/h1>\n

    MTMM Experiments in the ESS<\/h2>\n

    We use data from ESS rounds 5 to 7, encompassing 1,452 measurement estimates for 93 items fielded across 24 countries and 21 languages. These rounds were selected because they are the only ones for which both measurement quality estimates from MTMM experiments and detailed sample composition information are available in a consistent and comparable format. Earlier rounds were excluded due to missing sample composition data. Round 8 was not included because the MTMM estimates were based on a different analytical procedure, which renders its results not directly comparable to those from rounds 5 to 7. Although round 9 returned to the original analytical approach, it was excluded to preserve a continuous and methodologically consistent time series.<\/p>\n

    Table 1 presents the number of items, along with the mean and standard deviation of measurement error for all items within each country-language group, broken down by ESS round. The observed variation in mean measurement error across these groups indicates that, on average, the questionnaire performs better in some country-language contexts than in others. For instance, in round 5, the Swedish items in Sweden show the lowest mean measurement error, whereas the Hebrew items in Israel exhibit the highest. Notably, the Arabic items in Israel display considerably lower measurement error, highlighting the potential influence of language context within the same country. Examining the distribution of measurement errors within country-language groups reveals that the Slovakian items in Slovakia, in round 5, have the highest standard deviation. This suggests greater variability in measurement error across items compared to other country-language groups, indicating that some items perform much better\u2014or worse\u2014than others within that context.<\/p>\n

     <\/p>\n

    Table 1<\/strong><\/p>\n

    Mean Measurement Error and Standard Deviation Over All Items by Country-Language Group and ESS Round<\/em><\/p>\n

    \"\"<\/a><\/p>\n

    Note.<\/em> In rounds 6 and 7, some items were not fielded in certain country-language groups.<\/p>\n

     <\/p>\n

    An examination of the distribution of measurement error\u2014characterized by mean and standard deviation for each item across the different country-language groups\u2014reveals substantial variation between groups (see Table A1 in the Appendix). This suggests that the same item may exhibit low measurement error in one group but high error in another. To quantify this variability in a standardized way across the different items, we computed the coefficient of variation (CV) for each item, defined as the ratio of the standard deviation to the mean measurement error across the country-language groups:<\/p>\n

    CV =
    \nstandard deviation <\/span>mean<\/span><\/span><\/p>\n

    Figure 1 illustrates the distribution of CVs for all items in each of the three ESS rounds. The results show that the CVs are centered around 0.5 in all rounds but show considerable spread. A low CV indicates that measurement error is relatively consistent across country-language groups for a given item, whereas a high CV signals pronounced differences in measurement error between country-language groups. For reference, a CV of 0.5 implies that the standard deviation is half the size of the mean measurement error.<\/p>\n

     <\/p>\n

    \"\"<\/a>

    Figure 1.<\/strong> Distributions of the Coefficients of Variation of Measurement Error Across Country-Language Groups for All Items in ESS Rounds 5 to 7<\/p><\/div>\n

     <\/p>\n

    Representation Bias in the ESS Compared to Benchmarks from the EU-LFS<\/h2>\n

    The EU-LFS is an annual, large-scale probability-based survey of residents in private households across Europe, covering many of the same countries as the ESS. Following the argumentation of Koch et al. (2014) and Koch (2016, 2018), we consider the EU-LFS a useful data set for evaluating representation bias in the ESS. Building on their approach, our analysis relies on the dissimilarity scores reported by these authors to compare ESS samples to corresponding EU-LFS data.<\/p>\n

    Specifically, Koch and colleagues use Duncan\u2019s index of dissimilarity (Duncan & Duncan, 1955) to assess representation bias for the variables gender, age, marital status, work status, nationality, and household size\u2014provided these characteristics are available from the EU-LFS. The index ranges from 0 to 100, where 0 indicates identical distributions (i.e., no differences) between the ESS and the EU-LFS, and 100 represents complete dissimilarity. Duncan\u2019s dissimilarity index reflects the percentage of respondents that would need to change categories in the ESS to achieve perfect alignment with the EU-LFS distribution (Koch et al., 2014).<\/p>\n

    Due to missing EU-LFS data in some countries, household size is excluded from our analysis for all rounds, and marital status is excluded for round 5. For all available variables, we calculate the average of the reported dissimilarity indices across all socio-demographic characteristics for each country, producing a country-level aggregate measure of representation bias (see Table 2).<\/p>\n

     <\/p>\n

    Table 2<\/strong><\/p>\n

    Mean Index of Dissimilarity<\/em><\/p>\n

    \"\"<\/a><\/p>\n

    Note.<\/em> The table is generated based on the information presented in Koch et al. (2014) and Koch (2016, 2018). The index averages the dissimilarity indices of gender, age, work status, nationality, and marital status, the latter only being available for rounds 6 and 7.<\/p>\n

    Our analysis of representation bias is restricted to the country level, as the EU-LFS does not allow for differentiation by language groups within countries. As a result, we cannot assess how well specific language populations (e.g., German-speaking Swiss respondents) are represented relative to their respective subpopulations. Further, given that measurement error estimates from the MTMM experiments are available at the country-language level, we cannot assess representation bias in countries with more than one common language. Consequently, this limitation precludes us from including multilingual countries in the representation bias analysis.<\/p>\n

    To study the relationship between measurement error and representation bias, we regressed measurement error on the average dissimilarity index per country. The analysis combines data from rounds 5 to 7 and uses multilevel models to account for the hierarchical structure of the data, including random intercepts for items and countries. We ran beta regression to allow the measurement error to only take values between 0 and 1, using the package glmmTMB (Brooks et al., 2017) in R.<\/p>\n

     <\/p>\n

    Results<\/h1>\n

    This section presents our findings, structured according to the two guiding research questions.<\/p>\n

    RQ1. Association Between Measurement Error and Representation Bias<\/h2>\n

    To address our first research question\u2014examining the extent to which measurement errors are associated with representation bias\u2014we performed a multilevel beta regression analysis. Measurement error was regressed on representation bias, as measured by Duncan\u2019s dissimilarity index. To account for potential non-linear effects, we included both a linear and a squared term for representation bias in the model. The model incorporates random intercepts to account for the nested structure of the data. Table 3 summarizes the results for the random effects and fixed effects.<\/p>\n

     <\/p>\n

    Table 3<\/strong><\/p>\n

    Multi-Level Beta Regression Results: Measurement Error as a Function of Representation Bias Measured Using Duncan\u2019s Dissimilarity Index<\/em><\/p>\n

    \"\"<\/a><\/p>\n

    First, the estimated variance components of the random intercepts show that most of the unexplained variation of measurement error can be attributed to the item level rather than the country:item level. Second, the fixed effects of both the linear and squared terms of representation bias are statistically significant, suggesting a curvilinear relationship between representation bias and measurement error. However, including only representation bias as an independent variable explains little variation in measurement error (R\u00b2<\/em> = 0.007).<\/p>\n

    To illustrate the relationship between measurement error and representation bias, we present the predicted measurement errors for three example items, one for each ESS round under study:<\/p>\n