Aside from a few minor exceptions, the FEVS sampling frame is derived from the Statistical Data Mart of the Enterprise Human Resources Integration (EHRI-SDM), a large-scale database of U.S. Federal government personnel maintained by OPM.\u00a0 The FEVS target population is full- or part-time, permanently employed (i.e., non-seasonal and non-political) civilian personnel who have been employed by their agency for at least six months prior to the start of the survey.\u00a0 As described in U.S. Office of Personnel Management (2017), a stratified sample of 1,139,882 employees from 85 agencies was selected for the 2017 FEVS that was fielded between May 2 and June 22, 2017.\u00a0 Agencies had staggered start and end dates, but each agency had a field period lasting exactly six weeks.\u00a0 On launch day, sampled employees were sent an email invitation to participate that contained a unique URL.\u00a0 Five weekly reminder emails were sent to nonrespondents thereafter, with the final reminder including wording to indicate that the survey would close at the end of the business day.<\/p>\n
A total of 112,576 employees, roughly 10% of those sampled for the 2017 FEVS, were randomly selected to participate in the opt out experiment\u2014small and independent agencies, often with fewer than 200 employees, were excluded.\u00a0 Figure 1 demonstrates the prototypical email body in which the opportunity to opt out was offered via a link labeled \u201cClick here if you are considering not participating in the FEVS.\u201d\u00a0 For those designated to be part of the opt out experiment, the link appeared in the initial invitation and all subsequent reminders.\u00a0 The link was absent in emails to employees not designated for the opt out experiment.<\/p>\n
<\/p>\n
Figure 1<\/strong>: Example Email Invitation Body for the 2017 Federal Employee Viewpoint Survey Containing a Link to Opt Out.<\/p>\n <\/p>\n A short survey was launched when the link to opt out was clicked.\u00a0 The first question posed was \u201cWould you say that you are unsure about participating in the FEVS, or that you do not wish to participate?\u201d\u00a0 The intent of this question was to gauge the individual\u2019s conviction level about not participating\u2014essentially, to enable a proxy distinction between a soft and hard initial refusal. \u00a0The second question in the opt out survey ascertained the single most influential reason for not wanting to participate in the 2017 FEVS.\u00a0 A screen shot of this question is given in Figure 2.\u00a0 Six options were offered (e.g., being too busy, data confidentiality concerns), which were drawn from results to an open-ended question included in a nonrespondent follow-up study fielded at the conclusion of the 2004 administration of the survey (U.S. Office of Personnel Management, 2006).\u00a0 Also included was a write-in option for someone to specify an unlisted reason.\u00a0 Two research team members independently recoded 176 write-in responses into new or existing categories, with 28 initially discordant recodes requiring reconciliation.\u00a0 Some of the reasons unaccounted for by the original set of response options include a recent or pending employment status change, technical issues accessing the survey, and a belief that one had already taken the survey.\u00a0 In all, 126 of the write-in responses were grouped into new or existing categories, while the other 50 remained classified as \u201cOther.\u201d<\/p>\n <\/p>\n Figure 2<\/strong>: Screenshot of Opt Out Survey Question Regarding One\u2019s Primary Reason for Refusing to Participate in the 2017 Federal Employee Viewpoint Survey.<\/p>\n <\/p>\n Upon answering the question shown in Figure 2, a randomly predetermined 75% of individuals received a last-moment appeal in the form of a succinct, bulleted list of 3-4 assurances about which the individual may not have been aware.\u00a0 These assurances were tailored to the response given (e.g., brief itemization of protections in place for those expressing confidentiality concerns).\u00a0 The complementary 25% of individuals were routed to a page with short message stating no further email reminders to participate would be sent from the FEVS administration team, but that the survey link would remain active in case the individual changed his or her mind before the end of the field period.\u00a0 The purpose of this randomization\u2014and the first opt out survey question regarding nonresponse conviction level\u2014was to assess the effectiveness of the last-moment appeal, which we viewed as an automated refusal conversion procedure of sorts.\u00a0 Results pertinent to that research objective can be found in Lewis et al. (2019).\u00a0 The express purpose of this article is to report on the other research objective underpinning this experiment: to assess whether those who opt out are representative of the larger pool of 2017 FEVS nonrespondents.<\/p>\n The flowchart in Figure 3 summarizes core dispositions and associated counts for individuals randomly selected for the 2017 FEVS opt out experiment.\u00a0 Of the original count of 112,576 individuals, 105,319 were deemed eligible to be included in the present analyses.\u00a0 We excluded from consideration individuals who had no chance to participate, which includes those for whom a valid email address could not be obtained, indication in EHRI-SDM of departing one\u2019s position between the time of sample selection and survey administration, or being on an approved absence encompassing the full field period.<\/p>\n <\/p>\n Figure 3<\/strong>: Flowchart of Key Dispositions and Counts for Individuals Randomly Selected for the 2017 FEVS Opt Out Experiment.<\/p>\n <\/p>\n Considering the overall response rate to the 2017 FEVS was 45.5%, we were surprised to find that only 1,533 individuals (roughly 1.5% of those eligible) clicked on the link to launch the opt out survey.\u00a0 A similarly designed pilot study conducted during the 2016 FEVS resulted in a comparably low click rate, but was attributed at the time to placing the link too far towards the bottom of the email body and introducing the opt out link at the midpoint of the field period.\u00a0 Evidently, a more prominent placement and including from the onset of the field period fails to entice a greater proportion of individuals, at least for this survey and target population.<\/p>\n Of the 1,533 individuals clicking on the opt out survey link, 485, or 31.6%, opted out of the 2017 FEVS.\u00a0 On the other hand, 831 individuals, or 54.2%, decided to take the survey.\u00a0 This was notably larger than the response rate of those designated for the experiment but who never clicked on the link: 46,897\/103,786 = 45.2%.\u00a0 The increase observed is a credit to the automated refusal conversion strategy targeting approximately 75% of these individuals, as discussed in Lewis et al. (2019).\u00a0 The remaining 217 individuals who viewed the opt out survey neither opted out nor completed the survey.<\/p>\n Table 1 summarizes the distribution of the primary reason the 485 opters out cited for declining to participate in 2017 FEVS.\u00a0 At 29.2%, the most frequently cited reason was the belief that results would not be used to enact any substantive workplace changes.\u00a0 This is understandable, considering the survey is administered on an annual basis and it takes several months following field period closeout for data processing, weighting for unit nonresponse, and report generation to occur.\u00a0 Results must then be interpreted, action plans created, and interventions approved by management prior to being implemented.\u00a0 Logistically, it can be difficult to make noticeable cultural changes in response to one year\u2019s survey results prior to the commencement of the subsequent year\u2019s FEVS administration. \u00a0The second most frequently cited reason was concern about the confidentiality of one\u2019s responses.\u00a0 This, too, is understandable, because the survey instrument includes sensitive items regarding one\u2019s perception of an immediate supervisor\u2019s performance, and reports are generated for work units with as few as 10 respondents.\u00a0 A public-use analysis file with individual-level responses is also released.\u00a0 For both formats, a rigorous sequence of confidentiality checks and protections are applied prior to release, yet the perceived risk of disclosure apparently remains high.\u00a0 Lastly, it is interesting to note how the distribution of reasons to not participate has changed relative to the nonrespondent follow-up study conducted after the 2004 administration.\u00a0 As reported in U.S. OPM (2006), the number one reason cited at that time was being too busy, at 46.9%.\u00a0 The belief that results would not be used to change anything was a distant second, at 9.5%.\u00a0 Confidentiality concerns were only cited 6.7% of the time.<\/p>\n <\/p>\n Table 1<\/strong>: Distribution of Primary Reason for Opting Out of the 2017 Federal Employee Viewpoint Survey.<\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2020\/01\/Lewis_fig1-300x257.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2020\/01\/Lewis_fig2-300x81.png\")
<\/a><\/p>\nResults<\/h1>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2020\/01\/Lewis_fig3-251x300.png\")
<\/a><\/p>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2020\/01\/Lewis_fig4_rev-215x300.jpg\")
<\/a><\/p>\n