In this paper, we focus on the challenges of mitigating nonresponse bias effects in business surveys, using empirical examples from one survey to illustrate challenges common to many programs. The terms \u201cestablishment survey\u201d and \u201cbusiness survey\u201d are often used interchangeably. We use the latter term since many business surveys select companies or firms, which comprise establishments. Most business surveys publish totals such as revenue, expenditures, and employees.\u00a0 Consequently, complete-case analyses are always biased. \u00a0We identify two separate but highly related estimation challenges with nonresponse in business surveys: (1) the difficulty in developing adjustment cells for nonresponse treatment\u00a0 that use auxiliary variables that are predictive of both unit response and outcome and (2) the difficulty in developing appropriate nonresponse treatments for surveys that collect a large number of data items, many of which are not strongly related to key data items or to the available auxiliary data.<\/p>\n
The General Setting: Business Populations and Business Data<\/h2>\n
Economic data generally have very different characteristics from their household counterparts. First, business populations are highly skewed, i.e. the majority of a tabulated total in a given industry comes from a small number of large units. Consequently, business surveys often employ single stage samples with highly stratified designs that include the \u201clargest\u201d cases with certainty and sample the remaining cases. \u00a0Thus sampled cases with large design weights may often contribute very little to the overall tabulated totals.<\/p>\n
An efficient highly stratified design requires that within-strata means are the same, and the between-strata means are different (Lohr, 2010, Ch.3).\u00a0 For this to happen, the unit measure of size (MOS) variable used for stratification must be highly positively correlated with the survey\u2019s characteristic(s) of interest. However, it is possible for a given characteristic to have no statistical relationship with unit size.\u00a0 For example, the frame MOS could be total receipts for the business, but an important characteristic of interest could be electrical consumption.\u00a0 Furthermore, although business populations are highly positively skewed, not all business characteristics are strictly positive (e.g. \u00a0income, profit\/loss).<\/p>\n
Not all sampled units respond.\u00a0 To account for this nonresponse, the survey designers partition the population into P<\/em> disjoint adjustment cells using xp<\/sub><\/span><\/em><\/strong>, a vector of auxiliary categorical variables available for all units. Each adjustment cell contains np<\/sub><\/em> units, of which rp<\/sub><\/em> respond. Nonresponse adjustment procedures are performed within adjustment cell, with the assumption that the respondents comprise a random subsample within the nonresponse adjustment cells.<\/p>\n \u00a0<\/strong>Many business programs collect detail items \u2013 groups of items that sum up to their respective totals.\u00a0 The total and associated details items are referred to together as \u201cbalance complexes\u201d (Sigman and Wagner, 1997). All <\/em>survey participants are asked to provide values for the key items (hereafter referred to as \u201ctotals\u201d), whereas the type of requested details requested can vary. For example, Figure 1 presents the balance complex included on the Service Annual Survey (SAS) questionnaire mailed to companies that operate in the airline industry.\u00a0 [The SAS population comprises several industries]. The information requested in lines 1a through 2 are details that are only<\/em> requested from sampled units that operate in the airline industry (and are referred to herafter as ‘detail items’), and the information requested in line 3 is collected from all units that are sampled in the SAS.<\/p>\n Figure 1: Sample Balance Complex from the Services Annual Survey (Transportation Sector)<\/p>\n <\/p>\n Data collection and nonresponse adjustment for the total items are much less problematic than for the detail items because companies are usually able to proportion out their \u201cbottom line\u201d total items. Moreover, alternative data are often available for substitution or validation of these items.\u00a0 In contrast, with smaller units, the requested detail level data may not be available from all respondents, and auxiliary data are generally not available (Willimack and Nichols, 2010).<\/p>\n Furthermore, the larger units are more likely to provide response data than are the smaller units. First, the smaller units may not keep track of all of the requested data items (Willimack and Nichols, 2010) or may perceive the response burden as being quite high (Bavda\u017e 2010).\u00a0 Second, operational procedures increase the likelihood of obtaining valid response from large units.\u00a0 Analyst procedures in business surveys are designed to improve the quality of published totals. This is best accomplished by unit nonresponse follow-up of the large cases expected to contribute substantially to the estimate, followed by intensive analyst research for auxiliary data sources such as publicly available financial reports to replace imputed values with equivalent data (Thompson and Oliver, 2012). This approach works well for the key survey totals items, where alternative data are available for substitution or validation, but not for the detail items.<\/p>\n Frequently Used Adjustment Procedures for Unit and Item Nonresponse<\/strong><\/p>\n There are two treatments for unit and item nonresponse:\u00a0 adjustment cell weighting and imputation.\u00a0 In household surveys, where there is generally little or no information corresponding to the missing data from the sampled units, adjustment weighting \u2013 which increases the sampling weights of the respondents to represent the nonrespondents \u2013 is the only legitimate option (Kalton and Kaspryzk, 1986).\u00a0 In business surveys, imputation can be as appealing as weight adjustment for treating unit nonresponse, especially when valid data from the same sample unit are often available for direct substitution. Indeed, Beaumont et al (2011) prove that such auxiliary variable imputation can yield identical variances as those obtained from the full response data. In contrast to weighting, imputation is performed by item, using a hierarchy that imputes items in a pre-specified sequence determined by the expected reliability of available imputation models [Note: hot deck imputation and certain Bayesian models are exceptions to this univariate procedure but are not further discussed in this paper as their usage is fairly rare with business surveys]. This approach allows great flexibility and preserves the expected cell totals, but does not preserve multivariate relationships between items.<\/p>\n In our setting, the business survey has a random sample of size s<\/em> that has been partitioned into P disjoint unit nonresponse adjustment cells, indexed by p.<\/em> In each imputation cell p,<\/em> sp,r<\/sub><\/em> units respond and sp,nr<\/sub><\/em> units do not.\u00a0 Thus, survey data are available for the variable of interest y<\/em> from the sp,r <\/sub><\/em>responding units. A vector of auxiliary variables x<\/em><\/strong> exists for all the sampled units (respondents and nonrespondents). Under complete response, the population total Y would be estimated as <\/span> <\/span> where Our case study considers three commonly used imputation models. Each model can be re-expressed as an adjustment-to-sample weighting estimators, as described in Kalton and Flores-Cervantes (2003). Here, the weighted estimator of the population total for characteristic y<\/em> is given by<\/p>\n <\/span> <\/span> where Table 1 presents the three imputation\/weighting procedures. The count_u<\/em> procedure imputes the weighted average value in the imputation cell for the missing value.\u00a0 This is equivalent to adjusting the respondent units\u2019 final weights by the weighted inverse response rate (Oh and Scheuren, 1983).<\/p>\n The count <\/em>procedure uses an unweighted mean for imputation, which is equivalent to multiplying the respondent units\u2019 final weights by an unweighted inverse response rate (see S\u00e4rndal and Lundstr\u00f6m, 2005, Chapter 7.3, and Little and Vartivarian, 2005).<\/p>\n If the probability of unit nonresponse does not depend on the values of the observed characteristic y<\/em>, then the data are missing at random (MAR) as defined in Rubin (1976). Under this assumption, the probability of response in each adjustment cell p<\/em> is a constant, and the \u201cinverse response rate\u201d adjustment to the design weights produces an \u201cunbiased\u201d total from the respondent data. These adjustments are simple to compute, but the additional stage of weighting increases the variance (Kish, 1992); Kalton and Flores-Cervantez, 2003; Little and Vartivarian, 2005).<\/p>\n With business survey data, the probability of response is often related to unit size, and the uniform response assumption (i.e., MAR) is not realistic.\u00a0 Shao and Thompson (2009) describe the more general covariate-dependent response mechanism, which allows the probability of response to depend on a strictly positive auxiliary variable x<\/em> such as the MOS. Under this response model, the count adjustments described in the paragraph above do not mitigate the nonresponse bias and can only increase the sampling variance (Little and Vartivarian, 2005).<\/p>\n The ratio<\/em> procedure predicts a value for the missing y<\/em> with the no-intercept linear regression regression model described by Table 1: Nonresponse Adjusted Estimators Considered in the Case Study<\/p>\n Hereafter, as in Table 1, we use the term ‘imputation model’ to describe the formula used to obtain an imputed (replacement) value for the missing y<\/em> and the term ‘imputation parameter’ to describe data-driven estimates obtained from respondent values to compute these replacement values (the weighted or unweighted sample mean or ).<\/p>\n The Service Annual Survey (SAS)<\/strong><\/p>\n For the remainder of the report, we discuss the analysis of the nonresponse adjustment procedures for the Service Annual Survey (SAS).\u00a0 The SAS is a mandatory survey of approximately 70,000 employer businesses having one or more establishments located in the U.S. that provide services to individuals, businesses, and governments, identified by North American Industry Classification Series (NAICS) system code on the sampling frame. We examine the SAS sections covering the transportation and health industries (SAS-T and SAS-H, respectively). Information on the SAS design and methodology is available at http:\/\/www.census.gov\/services\/sas\/about_the_surveys.html<\/a>.<\/em><\/p>\n The SAS uses a stratified random sample. Companies are stratified by their major kind of business (industry), then are further sub-stratified by estimated annual receipts or revenue. All companies with total receipts above applicable size cutoffs for each kind of business are included in the survey as part of the certainty stratum. Within each noncertainty size stratum, a simple random sample of employer identification numbers (EINs) is selected without replacement. Thus, the sampling units are either companies or EINs.\u00a0 The initial sample is updated quarterly to reflect births and deaths.<\/p>\n The key items collected by SAS are total revenue and total expenses, both of which are totals in balances complexes. The revenue detail items vary by industry within sector. Expense detail items are primarily the same for all sectors, with an occasional additional expense detail or two collected for select industries. Total payroll is collected in all sectors as a detail item associated with expenses. For editing and imputation, payroll is treated as a total item, as auxiliary administrative data are available.\u00a0 Imputation is used to account for both unit and item nonresponse. Auxiliary variable and historic trend imputation (which uses survey data from the same unit in a prior collection period) are preferred for revenue, expenses, and payroll.\u00a0 Otherwise, SAS-H and SAS-T utilize the trend and auxiliary ratio imputation models, where the trend<\/strong> module predicts a current period value of y<\/em> from a prior period value and the auxiliary<\/strong> model uses a different auxiliary variable obtained from the same unit and collection period.<\/p>\n The imputation cells for SAS are six-digit industry (NAICS) code cross-classified by tax-exempt status. Unlike the sampling strata definitions, the imputation cells do not account for unit size, and imputation parameters use certainty and (weighted) noncertainty units within the same cell. The imputation base for the ratio imputation parameters is restricted to complete respondent data, subject to outlier detection and treatment.<\/p>\n Response Propensity Analysis<\/strong><\/p>\n Response propensity modeling uses logistic regression analysis to determine sets of explanatory covariates related to unit response. Separately examining the SAS-T and SAS-H data, we used the SAS SURVEYLOGISTIC procedure[i]<\/a> to fit two logistic regression models: (1) a simple model that used only the existing imputation cells as independent variables; and (2) a nested model that also included the continuous MOS variable as a covariate. The logistic regression analysis therefore examines whether the categorical variables used to form adjustment cells are predictive of unit nonresponse and to check if other variables are missing in the construction of the adjustment cell.<\/p>\n We tested the goodness-of-fit hypothesis of each fitted model. All were significant, so we examined the marginal test results for individual imputation cells for cells with good fits. Rejecting the goodness-of-fit null hypothesis provides evidence that at least one<\/em> of the variables used to construct adjustment cells are <\/em><\/strong>related to response propensity. Examining the marginal results highlights individual imputation cells where there may be a missing predictor.<\/p>\n Figure 2 presents side-by-side bubble plots summarizing the logistic regression results for SAS-H.\u00a0 Figure 3 presents the corresponding counts for SAS-T.<\/p>\n Figure 2:\u00a0 Logistic Regression Results for SAS-H. Each dot represents number of significant marginal test results for an individual imputation cell, with the number of significant tests indicated on the y-axis. A strongly predictive model should have significant results in at least four of the six studied years.<\/p>\n <\/p>\n Figure 3: Logistic Regression Results for SAS-T<\/p>\n For both programs, the logistic regression analysis provides evidence that the industry\/tax status categories used to form adjustment cells are not strongly related to response propensity. Including the continuous nested MOS covariate in the SAS-T model improves the predictions, although there is no evidence that this is the case with SAS-H.<\/p>\n Clearly, the existing sets of categorical variables used to form imputation cells for SAS-T are inadequate for mitigating unit nonresponse. Initially, we considered using the sampling strata as adjustment cells.\u00a0 However, a high proportion of strata contained fewer than five units because of the highly stratified design and the limited number of large companies and large tax-entities in the sampling universe.<\/p>\n Unit Response Rate Comparisons<\/strong><\/p>\n With SAS, certainty status is directly related to response propensity through the analyst follow-up procedures. S\u00e4rndal and Lundstr\u00f6m (2005) recommend exploring whether there is a sytematic difference in response propensities on a single category by comparing their unit response rates (URR) in the same imputation cell. In the Economic Directorate of the U.S. Census Bureau, the URR is the ratio of units that reported valid data to the total number of eligible units, computed without survey weights (Thompson and Oliver 2012).<\/p>\n Figure 4 presents the average URR (across the six years) for each SAS-H imputation cell, with blue squares presenting the certainty-unit URR, and the red squares presenting the noncertainty-unit URR in the same imputation cells. In the majority of cases, the certainty and noncertainty URRs within the same cell are dissimilar, although the direction of the difference is not consistent.<\/p>\n Figure 5 presents the corresponding measures for SAS-T. As with SAS-H, the URRs within the same imputation cell clearly differ by certainty status. In contrast to the SAS-H results, there is a very clear pattern within the SAS-T cells, where the unit response rates for certainty units are generally higher than the corresponding noncertainty measures.<\/p>\n Figure 4: Average URR by Certainty Status within Imputation Cell (SAS-H)<\/p>\n Figure 5: Average URR by Certainty Status within Imputation Cell (SAS-T)<\/p>\n The transportation sector tends to have a few very large units (businesses) in each industry, with the remaining units being fairly homogeneous in size, and the analysts attempt to obtain complete data from all certainty cases. In contrast, the unit size within the health sector is much more variable, and the SAS-H sample is much more highly stratified. Analysts must obtain valid responses from certainty and \u201clarge\u201d noncertainty units, so the response rate pattern is not as consistent.<\/p>\n \u00a0Alternative Weighting Comparisons<\/strong><\/p>\n The earlier analysis indicates that the studied programs\u2019 imputation cells fail to satisfy the MAR assumption. That said, if the degree of nonresponse bias in the studied estimates is small, then this might not be of strong concern. Groves and Brick (2005) propose evaluating the magnitude of the nonresponse bias by altering the estimation weights and using the various weights to construct different estimates. If the difference between the estimates is trivial, there is evidence that the nonresponse bias may not be large.<\/p>\n To vary the weights, we re-express the ratio imputation models as ratio reweighting models as shown in Table 1, and likewise re-express the presented alternative mean imputation models as the reweighted count<\/em> and count_u<\/em> estimators. We computed these three alternatively weighted estimates for each item by publication industry in our six years of data.\u00a0 For each item, we obtain the ratio of the count and count_u weighted estimates to the ratio estimates (the current imputation method).\u00a0 Figures 6 and 7 presents the \u201cdouble-averaged\u201d estimate ratios[ii]<\/a> for the SAS-H and SAS-T items.<\/p>\n Figure 6: SAS-H Reweighted Estimates (Averaged Within Statistical Period and Across Industry). In these plots, the total items (receipts, expenditures, and payroll) are represented by squares, and the various detail items are represented by circles. Each graph includes a horizontal asymptote at y<\/em> = 1 to indicate the estimate ratios that are essentially unaffected by reweighting.<\/p>\n Figure 7: SAS-T Reweighted Estimates (Averaged Within Statistical Period and Across Industry)<\/p>\n With SAS-H, the count and ratio estimates are very close, regardless of whether the collected item is a total or a detail.\u00a0 However, the differences between the count_u and ratio estimates are more pronounced. The SAS-H results are \u201cdifferent enough\u201d to merit some concern about unmitigated nonresponse bias, whereas the SAS-T results are much more conclusive. The differences among the three sets of SAS-T estimates are very pronounced, indicating estimation effects caused entirely by changes in adjustment methodology.<\/p>\n These analyses highlight issues with unit nonresponse in business data and challenges in remediating these issues. First, the URR is not necessarily a good measure of representativeness of the sample (Peytcheva and Groves, 2009).\u00a0 In our case study, the majority of the URRs are at an acceptable level, but the other analyses show that the larger units respond at a higher rate than the smaller units. By partitioning the existing imputation cells by size categories, we can likely reduce the nonresponse bias. However, there are insufficient numbers of sampled units in the sampling strata to use them as adjustment cells, and the small number of \u201clarge\u201d units makes it challenging to subdivide the existing cells. In the future, it may be possible to develop strata collapsing procedures during the survey design stage.<\/p>\n Evaluation of the Prediction (Imputation) Models<\/strong> <\/strong><\/p>\n The SAS uses the ratio imputation model from Table 1 when auxiliary data or historic data from the same unit are not available: Matthews (2011) and Nelson (2011) provides information on each item\u2019s imputation model.\u00a0 To assess the imputation models\u2019 predictive properties, we fit each regression imputation model within the currently used imputation cells with the SAS SURVEYREG procedure, again excluding certainty cases.\u00a0 Figures 8 and 9 summarize the regression analysis results for SAS-H and SAS-T, respectively. These figures plot the average R2<\/sup> value from each model. We consider any R2<\/sup> value above y <\/em>= 0.75 would to be strongly predictive. The total items (receipts, expenses, and payroll) and detail items are separated by a vertical asymptote and are annotated as such in the Figures.<\/p>\n Figure 8: Regression Analysis Results for Item Imputation Models (SAS-H). A blue diamond indicates a consistently significant model (at \u03b1 = 0.10) and a red square indicates the reverse. Figure 9:\u00a0 Regression Analysis Results for Item Imputation Models (SAS-T)<\/p>\n In both programs, the models that predict the totals items are strongly predictive. However, the models used to impute the detail items are generally not.\u00a0 Hence model-imputation for totals is appropriate, but rarely used due to the availability of alternative data sources such as administrative or historic data, and model-imputation for details is not necessarily appropriate, but is frequently employed.<\/p>\n Of course, the effectiveness of a ratio imputation model for correcting nonresponse bias is highly dependent on the availability of data for parameter estimation. For SAS, the respondent units must provide valid values for either revenue or<\/em> expenses, not necessarily both. On the average, the item response rates for SAS are quite low – generally between 50 to 60 percent for totals and between 40 to 60 percent for detail items, regardless of the sector.\u00a0 Furthermore, the unit size does not appear to be a factor in item nonresponse: item response rates computed separately for certainty and noncertainty units in the same industry tend to be very close.<\/p>\n The earlier analyses provided indications that the SAS imputation cells should be further subdivided to account for unit size. If the imputation parameters are approximately the same for each unit size category within an imputation cell, then the \u201cdominance\u201d of the large cases would not influence the predictions. On the other hand, if imputation parameters did differ by unit size within industry, then the adjustment strategy being used is inducing systematic bias.<\/p>\n To investigate this, we obtained the ratio imputation parameters in the current imputation cells, then refit the same regression models with more refined industry cells (splitting the industry data into certainty and and noncertainty components). Figure 10 presents stacked imputation parameters from the ratio model that uses expenditures to predict revenue using 2010 SAS-H data. Each bar represents a set of regression imputation parameters from the original imputation cell.<\/p>\n Figure 10: Ratio Imputation Parameters for Revenue\/Expenses (SAS-H 2010). The blue bar is the regression parameter obtained using all units in the industry, the red bar is the regression parameter obtained using only the certainty units, and the green bar is the regression parameter obtained from the noncertainty units.<\/p>\n In Figure 10, all of the imputation parameters are approximately the same, with a few exceptions. This pattern repeats in the SAS-H and SAS-T data for all data collection years.\u00a0 However, this is a ratio of two well-reported totals items that are generally imputed with auxiliary data. When we examine a similar plot for a typical SAS-H detail item, the situation is quite different, as shown in Figure 11.<\/p>\n Figure 11: Ratio Parameters for a Typical SAS-H Detail Item Ratio<\/p>\n Here, the imputation parameters computed from the certainty cases have almost exactly the same value as the parameters computed from the complete data in the imputation cell, and the imputation parameters for the noncertainty units are each quite different. In short, the ratio imputation model causes all<\/em> imputed units to resemble the certainty units. Similar plots are available for all ratio imputation parameters upon request, but are not included for brevity. However, the vast majority of imputation model analyses for the detail items demonstrated similar patterns.<\/p>\n Finally, we examine the effect of choice of imputation cell by item, given a nonresponse adjustment method.\u00a0 Figures 12 and 13 shows \u201cdouble-average\u201d ratios of estimates computed using the same weights with different adjustment cells, comparing estimates obtained using the existing cells subdivided by certainty status (more refined parameters) to those obtained from the currently used imputation cells. For SAS-T, the totals do not vary much, regardless of adjustment method, and many of the detail items that were imputed with the ratio model maintain similar levels as well. With SAS-H, the choice of adjustment cell has a very large impact on the estimate levels, regardless of whether the item is a total or a detail.<\/p>\n Recall that the SAS-T sampled<\/em> unit population is fairly homogeneous in size, in contrast to the SAS-H sampled<\/em> unit population. For SAS-T, the choice of adjustment cell is the most important factor in nonresponse bias mitigation. In this population, the ratio imputation models (which incorporate unit size in the parameter estimation) are quite good for totals, but not so for details.\u00a0 With SAS-H, it is not immediately clear which factor (adjustment cell or adjustment method) is more important in nonresponse bias mitigation. Although it appears that unit size is not strongly related to response propensity for this population, it is also apparent that unit size is very related to prediction for the key totals. Unfortunately, this strong relationship is not true for the SAS-H details.<\/p>\n Figure 12: Comparison of Alternatively Weighted Estimates by Imputation Cell (SAS-H)<\/p>\n <\/p>\n Figure 13: Comparison of Alternatively Weighted Estimates by Imputation Cell (SAS-T)<\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2013\/08\/Business-figure1.png\" "\\"Business")
<\/a><\/p>\n where wj<\/sub><\/em> is a weight associated with unit j<\/em> (usually the inverse probability of selection). \u00a0<\/em>The imputed estimator of the population total for characteristic y<\/em> is given by<\/p>\n<\/p>\n is the imputed value obtained for nonrespondent unit in adjustment cell . <\/p>\n<\/p>\n is a weight adjustment factor to account for unit nonresponse, is a unit response indicator, and is the nonresponse adjusted weight for unit in adjustment cell under procedure . The adjusted weight will be a positive value for respondents and will equal zero otherwise.<\/p>\n. The weighted model incorporates unequal sampling and unit size in the parameter estimation, and the weighted least squares estimate is the best linear unbiased estimator of \u03b2<\/em> under this model. Note that S\u00e4rndal and Lundstr\u00f6m (2010) recommend the inclusion of an intercept, but we have found that the intercept is non-significant in many business data sets (ex., businesses with no employees have no payroll).\u00a0 If the covariate-dependent response mechanism is appropriate and the auxiliary variable x<\/em> is used in the ratio model or is highly correlated with the ratio model, then the ratio adjusted estimates described in Table 1 will have improved precision over the correponding count adjusted estimates. If the prediction model is not valid or if the strength of association between x<\/em> and y <\/em>is weak, then the bias induced by the ratio estimator increases the MSE over the other reweighted estimates.\u00a0 This is more likely to occur with the detail items than with the total items.<\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n