{"id":14864,"date":"2021-10-12T11:43:19","date_gmt":"2021-10-12T10:43:19","guid":{"rendered":"https:\/\/surveyinsights.org\/?p=14864"},"modified":"2021-10-12T11:43:19","modified_gmt":"2021-10-12T10:43:19","slug":"revisiting-the-ess-r8-sample-a-year-after-lessons-from-a-re-contact-survey-to-test-patterns-of-unit-non-response-in-hungary","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=14864","title":{"rendered":"Revisiting the ESS R8 sample a year after – Lessons from a re-contact survey to test patterns of unit non-response in Hungary"},"content":{"rendered":"

1. Introduction and research questions<\/h1>\n

Unit non-response \u2013 when a household\/individual in a sample is not interviewed at all \u2013 has been rising worldwide in most surveys. Unit non-response generally arises from non-contact (the household\/individual cannot be reached), from refusal (the household\/individual is reached, but refuses to participate in the survey) or from incapacity to answer (the household\/individual would participate, but cannot \u2013 for example, because of health problems) (Osier, 2016). Unit non-response only leads to bias if it is non-random. But if non-response relates to the variables under investigation, then the survey distributions will not accurately reflect the true distribution of the variables of interest in the population, and this results in systematically biased sample-based inferences, even in large samples (Korinek et al., 2007). However, it can be difficult to explore whether unit non-response is random or is dependent on any variables under investigation, as only limited information is available on the characteristics of non-respondents. Even if non-respondents are similar to respondents in terms of a limited number of characteristics \u2013 gender, age and geography \u2013 that does not mean that these groups are similar along other dimensions, such as attitudes, trust or beliefs (Meyer et al., 2015).<\/p>\n

The social background of unit non-response has attracted great research interest (Goyder et al., 2002). The most commonly used hypotheses are linked to the theory of general and social trust and to the theory of social integration (Amaya and Presser, 2016). Other studies have examined basic proxies for a more complex background (Abraham et al., 2006). Another reason for non-response could be \u201cinability\u201d: some people are mentally or physically unable to participate, while others may be left out for reasons of language. In ageing societies and countries (and also in societies and countries with increasing levels of immigration), these reasons for non-response may become more important (Vandenplas et al., 2018). Several analyses have identified response characteristics within population subgroups; however, further research is needed on how to identify stable and universal non-response patterns and on response probability estimates. Research has found that single-person households, tenants and individuals who are out of the labour force are less likely to participate in surveys than are other social groups (Abraham et al., 2006; Meyer et al., 2015). Social changes, such as shifts in the work\/life balance, falling trust in surveys or an increased survey burden, may also affect individual response propensities (Vandenplas et al., 2018).<\/p>\n

Given that unit non-response often seems to be non-random, it may lead to bias (termed \u201cnon-response error\u201d). There are two ways of reducing non-response error: the first is by conducting interviews with as many individuals in the sample as possible and then analysing the interview data to detect whether those individuals who have chosen to respond are somehow different from the entire population (Klofstad, 2005). This is a good (albeit expensive) way of increasing the response rate in a survey, though analysing non-responders is more complex. The nature of the bias is most often measured by comparing the data collected with administrative sources, in order to determine key deviations from population parameters (Klofstad, 2005). Often, the administrative data are difficult to access and outdated, and the scope for combining parameters (e.g. age and schooling) is limited. The only other solution is the second way: by collecting data from non-responders, though this actually sounds like a contradiction in itself. However, although there is a group of people in the population that can never<\/em> be reached, there is another group whose inaccessibility is only temporary<\/em> \u2013 and with special methodological research, we can find out more about them.<\/p>\n

The extent of the problem in the field is pressing. Unit non-response rates rose by some 3\u201312 percentage points over the 1990s for six US Census Bureau surveys, and by 2013 the rates had reached almost 35% (Meyer et al., 2015). Non-response rates and the rates of non-contact have risen over the past 20 years in Hungary as well: in 1996, 16% of the households selected refused to answer an optional survey block during the microcensus (Havasi, 1997); by 2016, a survey conducted by the Hungarian Central Statistical Office experienced a response rate of just 59% (with a 3% rate of non-contact) (European Central Bank, 2016). In one of our own previous survey results in 2015, 40% of the households selected refused to answer, while 2% of the addresses selected were ineligible (T\u00e1rki, 2016). Since this is a problem that primarily occurs in the field, stepping up fieldwork efforts would seem to be a promising strategy. Previous research has found that these efforts do have an effect on non-contact rates, but they are less effective at reducing refusal rates (Vandenplas et al., 2018). In longitudinal surveys, occasional methodological changes have coincided with changes in the contact rates: some of these changes \u2013 such as the provision of incentives \u2013 may have been effective; however, increases in the response rate thanks to incentives are almost always temporary and are often followed by a sharp decline (Williams and Brick, 2018). Therefore, fieldwork characteristics, methodological changes and the characteristics of the interviewer may all play an important role in any analysis of non-response patterns.<\/p>\n

This paper presents findings from a methodological study of survey non-response, based on a European Social Survey (ESS) sample (Round 8, fielded originally between May and September 2017). We recontacted a random selection of the original sample after roughly a year and a half (between December 2018 and February 2019), with the aim of finding out how far the original reach could be extended by a second attempt, and how previously unsuccessful cases (no-contact and refusals) can be characterized on the basis of their second responses (if available).[i]<\/a><\/p>\n

We seek to answers to the following research questions:<\/p>\n

Q1: How can the two surveys and the joint sample be characterized regarding outcome rates? What is the scale of non-response in the two samples and to what extent can non-response and no-contact be considered stable in the joint sample?<\/p>\n

Q2: What factors affect success in contacting those respondents who were unreachable (refusal or no-contact) during the first fieldwork (ESS R8 sample)? Does reach depend on the characteristics of the locality and neighbourhood? On the characteristics and operation of the fieldwork apparatus? Or does it depend on the demographic characteristics of the target individuals?<\/p>\n

Q3What factors affect interview response (once contact is made)? Which of the three broad categories of potential drivers (i.e. variables related to locality and neighbourhood; fieldwork characteristics (including the characteristics of interviewers, such as gender, age and level of education); and the demographic characteristics of target persons) is most important in determining interview success?<\/p>\n

The remainder of the paper is organized as follows: first, we briefly introduce the basis of the analysis in chapter 2.1 (Round 8 of the European Social Survey) and chapter 2.2 (the recontact fieldwork). In chapter 2.3 we introduce the definitions and terminology used and present the analysis methods. Chapter 3 reports our findings. Chapter 3.1 presents the rates of response, non-response and contact in both surveys; this is followed in chapter 3.2 by a short presentation of the distribution of the responses to some target questions that were part of both questionnaires. Chapter 3.3 then presents the results of multivariate analysis for the determinants of contact, refusal and success rates. Finally, we summarize our conclusions (chapter 4).<\/p>\n

[i]<\/a> Details of the survey are available in the original Hungarian report (Szeitl and T\u00f3th, 2020) and data from the survey have been deposited in the openly accessible T\u00e1rki Social Science Data Archive (https:\/\/adatbank.tarki.hu\/en\/<\/a>).<\/p>\n

 <\/p>\n

2. Data and methods<\/h1>\n

2.1 Round 8 of ESS in Hungary<\/h2>\n

For Round 8 of the ESS, a stratified sample of individuals was drawn from the Hungarian population register. ESS targets all persons aged 15 and over: no upper age limit is set for the responders, and nor are there other restrictions on eligibility (e.g. based on ethnicity, locality, etc.). The only exception is that people living in institutions are excluded. Hence, the aim was to achieve equal coverage of all residents within private households in Hungary. As the ESS R8 quality matrix states (Wuyts-Loosveldt, 2019: 20, Table 2.8), the sampling frame (purchased from the Hungarian population registry) was somewhat outdated, compared to other participating countries: 8.5% of the gross sample (of 4,006 cases) had moved by the time of first contact. That figure is relatively high: of the 13 countries using similar population registers of individuals for sampling, it is only exceeded by Poland (14.7%) and Spain (11.1%).<\/p>\n

For the selection of the field unit, a multistage cluster design was applied in the domain that represented rural areas (mostly villages), while an unclustered design was employed in a second domain representing urban centres with high population density (see the quality matrix, as above). The realized gross sample was 3,954 individuals; the realized net sample size was 1,625; and the realized effective net sample size was 1,330. The fieldwork lasted from May to December 2017. Round 8 of the ESS in Hungary involved a computer-assisted personal interview (CAPI) survey. During the data collection, ESS standards were followed.[i]<\/a><\/p>\n

2.2 The recontact sample<\/h2>\n

The aim of the recontact sample was to investigate how non-responders and ineligible households could be characterized. For reasons of cost, a randomly generated subsample of 3,000 addresses from the original ESS R8 gross sample of 4,006 was recontacted, irrespective of their availability and previous interview take-ups. The fieldwork lasted from December 2018 to February 2019.[ii]<\/a> The fieldwork protocols of the ESS[iii]<\/a> were closely followed during the recontact survey, too. All the original fieldwork conditions and standards regarding data collection were applied: addresses were randomly distributed across the same group of interviewers as during the initial fieldwork. For the selection of the recontact sample, we conducted simple random sampling. This approach was chosen to produce an additional random sample, and also to provide an opportunity to specify different types of non-responder (participants in both surveys; only in the ESS; only in the recontact survey). The wording of the questions remained exactly the same, but a shortened version of the original questionnaire was used, with some specific additions related to the recontact methodology. In the recontact survey (just as in the baseline one), a very detailed record of contact attempts was made, including data on the circumstances of the attempts, the characteristics of the neighbourhoods, etc. Later, administrative data on settlements were added to the respondents\u2019 records; and, in addition to the questionnaire files, the demographic and work experience characteristics of the interviewers were added and analysed.<\/p>\n

2.3 Some definitions and terminology used<\/h2>\n

In what follows, for the sake of simplicity, we call the ESS R8 sample ESS and the follow-up survey RECONTACT. In this section, we describe general outcome rates calculated from the final disposition codes in both surveys and in the joint sample. Outcome rates are outlined, based on the official definitions of the American Association for Public Opinion Research (AAPOR, 2016). We use the following final disposition codes:<\/p>\n

I = Complete interview<\/p>\n

P = Partial interview<\/p>\n

R = Refusal and break-off<\/p>\n

NC = Non-contact<\/p>\n

O = Other<\/p>\n

UH = Unknown whether household\/occupied<\/p>\n

UO = Unknown, other<\/p>\n

We define the response rate as the number of successful interviews, divided by the number of eligible units in the sample (AAPOR, 2016):<\/p>\n

Response Rate (RR1)=<\/p>\n

(1) <\/span>   <\/span>\"\begin{equation*}\frac{I}{(I+P)+(R+NC+O)+(UH+OH)}\end{equation*}\"<\/p>\n

 <\/p>\n

We define the refusal rate as the proportion of all cases in which a housing unit or respondent refuses to give an interview\/to comply (AAPOR, 2016):<\/p>\n

Refusal Rate (REF1)=<\/p>\n

(2) <\/span>   <\/span>\"\begin{equation*}\frac{R}{(I+P)+(R+NC+O)+(UH+OH)}\end{equation*}\"<\/p>\n

 <\/p>\n

We define the no-contact rate based on the AAPOR definition of contact rates (CON1): the no-contact rate is the proportion of all cases in which no responsible member of the housing unit could be reached by the interviewer:<\/p>\n

No-contact Rate (NOCON1)=<\/p>\n

(3) <\/span>   <\/span>\"\begin{equation*}\frac{NC}{(I+P)+(R+NC+O)+(UH+OH)}\end{equation*}\"<\/p>\n

 <\/p>\n

To answer Q1, one- and two-dimensional tables are presented. For Q2, distributions of respondents and non-respondents in ESS are compared. For Q3, logit analysis with specified subsamples is used.<\/p>\n

\u00a0<\/strong><\/h1>\n

3. Results<\/h1>\n

The response rate (i.e. the rate of successful interviews within the gross sample) was higher in the RECONTACT survey (48%), than in the original ESS (41%). All in all, the no-contact rate fell from 11% to 5% in the second survey. The share in other categories remained roughly the same (Table 1).<\/p>\n

Table 1 Results in the ESS and the RECONTACT sample<\/strong><\/p>\n

\"\"<\/a>
\nSource: ESS and RECONTACT survey.<\/em><\/p>\n

The percentages for various success\/non-response combinations are shown in Table 2 for the combined ESS\/RECONTACT sample. Numbers in brackets refer to the internal distribution of all cases, net of those not included in the RECONTACT sample. When taking the total of the two surveys combined, we see that still in over a third of cases (36.4% \u2013 1,091 individuals) where an attempt at contact was made, that attempt failed (Table 2, upper-left cell, figure in brackets); meanwhile, in rather more than a quarter of cases (25.7% \u2013 772 individuals), a person was available for both surveys (Table 2). Over the two surveys, there were 461 cases (15.4%) where only the ESS questionnaire was filled in, and 676 cases (22.5%) where only the RECONTACT questionnaire was completed.[iv]<\/a> The combined reach of the two surveys rose to 63.6% (Table 2, the sum of cells showing success in either survey, 63.6=22.5+15.4+25.7) from the initial 41.1% response rate of Round 8 of the ESS (Table 1).<\/p>\n

Table 2 Distribution of all contact attempts, % (in brackets: % distribution of the joint ESS\u2013RECONTACT sample)<\/strong><\/p>\n

\"\"<\/a>
\nNote: Percentages (not in brackets) are inclusive for all attempts (included are those not selected for the RECONTACT sample), percentages in brackets are inclusive only for the joint sample (selected for the RECONTACT sample).<\/p>\n

Source: ESS and RECONTACT survey, N of joint sample=3,954.<\/em><\/p>\n

3.2 Sample distributions of some core variables of the survey<\/strong><\/h2>\n

As we show, the RECONTACT survey was able to bring in a group of previous ESS non-responders. Here we compare the sample distributions of their answers to those of the answers of the original ESS respondents for four variables. Two of these are subjective target variables of the ESS (generalized trust and satisfaction with their own lives), while the other two serve to indicate sample composition by education and working hours in employment (Figure 1). Some caution in interpretation is warranted, but nevertheless these preliminary results do perhaps provide a sufficient basis for more detailed further research, despite the small number of cases.<\/p>\n

In general, the recontacted sample seems to show a somewhat wider distribution of these test variables.<\/p>\n