{"id":16588,"date":"2022-12-06T17:03:01","date_gmt":"2022-12-06T16:03:01","guid":{"rendered":"https:\/\/surveyinsights.org\/?p=16588"},"modified":"2022-12-06T17:03:01","modified_gmt":"2022-12-06T16:03:01","slug":"binary-response-format-or-11-point-scale-measuring-justice-evaluations-of-earnings-in-the-soep","status":"publish","type":"post","link":"https:\/\/surveyinsights.org\/?p=16588","title":{"rendered":"Binary response format or 11-point scale? Measuring justice evaluations of earnings in the SOEP"},"content":{"rendered":"

Introduction<\/h1>\n

Empirical justice research has gained momentum in recent decades. Questions about fair treatment at work, fair chances to excel in life, and the fairness of income are investigated in various social science disciplines (Liebig and Sauer 2016). A vital branch of empirical justice research investigates distributive justice and, in particular, the justice of one\u2019s own earnings (Jasso, T\u00f6rnblom, & Sabbagh 2016; Jasso 1978). Questions on the perceived justice of earnings are included in several international comparative surveys\u2014such as the International Social Survey Project (ISSP), the European Social Survey (ESS), and the International Social Justice Project (ISJP)\u2014as well as in many national surveys\u2014including Australia, Canada, Germany, and the U.S.\u2014and employ a variety of empirical strategies (Jasso 2007).<\/p>\n

For example, the justice evaluation of earnings can be assessed by asking respondents about the earnings they would consider just along with querying information on their actual earnings. The justice evaluation is then assessed as the natural logarithm of the ratio of both amounts (e.g., Jasso 1978; Sauer & Valet 2013; Schunck, Sauer, & Valet 2015). Some authors simply draw on the information on the amount of reported just earnings (e.g., Jasso and Wegener 2021; Schr\u00f6der 2017; Schneck 2014). Another approach asks respondents to evaluate the justice of their own earnings on a scale. Across surveys, a variety of response scales is used to assess respondents\u2019 justice evaluations of their own earnings. While some use a binary response format that simply differentiates if respondents evaluate their earnings as just or unjust (e.g., Falk et al. 2011; Pfeifer & Stephan 2019; H. Schneider 2018), others rely on response scales that capture the degree of perceived injustice (e.g., Adriaans & Targa 2022; Narisada 2020; S.M. Schneider & Valet 2017; Tao 2015).<\/p>\n

While different approaches are used, we know little about how these affect respondents\u2019 justice evaluations. This general problem of missing insights into the role of measurement choices on the sense of justice is illustrated by the case of the German Socio-Economic Panel Study (SOEP). SOEP uses a mixture of measurement strategies: First, asking respondents to directly evaluate their earnings, and then asking follow-up questions on actual and just earnings. From 2005 to 2015, SOEP respondents were asked: \u201cIs the [gross] income that you earn at your current job just, from your point of view?\u201d Response options were \u201cyes\u201d and \u201cno\u201d. While this response format allows to assess whether a respondent expresses to receive unjust pay, it lacks important information that is of interest to empirical justice research. First, it ignores the bipolar nature of injustice in favour of a unipolar conception (e.g., Menold & Bogner 2016): Are the received earnings unjustly low or unjustly high? Second, the magnitude of perceived injustice is overlooked: Do respondents perceive extreme, moderate, or slight injustice? Third, the question wording may be leading as it only asks whether income is \u201cjust\u201d but does not mention injustice. This may lead respondents to evaluate their earnings as just.<\/p>\n

To address the shortcomings of the binary measure and to increase comparability with established measures in empirical justice research (Auspurg et al. 2017; Sauer & May 2017; S.M. Schneider & Valet 2017), SOEP introduced an 11-point response scale in 2017. This new response scale enables respondents to express both the direction and intensity of perceived injustice and thus should allow researchers to analyse more subtle distinctions in respondents\u2019 earnings injustice evaluations (e.g., Krosnick & Fabrigar 1997, Menold & Bogner 2016). The new question reads: \u201cIf you think about your current gross income, would you say that it is just, considering the type of work you do, or that it is unjustly high or low?\u201d Respondents can answer on an 11-point rating scale running from -5 (labelled \u201cunjustly low\u201d) through 0 (labelled \u201cjust\u201d) to +5 (labelled \u201cunjustly high\u201d). Building on the assumption that, in the case of justice, actual earnings perfectly align with just earnings (Jasso 1978; Jasso, T\u00f6rnblom, & Sabbagh 2016), both the binary and the 11-point scale filter respondents rating their earnings as unjust into a follow-up question on the amount of just earnings. Thus, respondents choosing \u201cno\u201d in the binary response format or any value except for 0 (= \u201cjust\u201d) on the 11-point scale are asked: \u201cHow high would your gross income have to be in order to be just?\u201d with an open answer format asking for Euros per month.<\/p>\n

With the introduction of an 11-point response format, the share of SOEP respondents who evaluated their gross earnings as just dropped to 49 percent in 2017 compared to 62 percent who evaluated their gross earnings as just in 2015 using the binary response scale. This considerable change in average justice evaluations may point to the relevance of measurement choices when assessing justice evaluations of earnings. Evidence on measurement equivalence across different rating scales generally suggests caution in comparing evaluations from different rating scales (Menold & Tausch 2016). In the applied case here, the pattern of more respondents evaluating their gross earnings as just on the binary scale compared to the 11-point scale may be due to stronger acquiescence or agreement biases on binary yes\/no scales (e.g., Bentler, Douglas, & Messick 1971; Billiet & Davidov 2008; Krosnick & Presser 2010). Moreover, a binary yes\/no response option may convey to respondents that researchers are only interested in severe instances of injustice (Kn\u00e4uper 1998). Accordingly, the binary response format may pick up only strong injustice perceptions, while the 11-point response scale should pick up small injustice perceptions as well. However, systematic insights into how such measurement choices affect justice evaluations are missing. We address this gap by means of a survey experiment that was fielded as part of the Innovation Sample of the Socio-Economic Panel Study (SOEP-IS). Comparing justice evaluations from a split-ballot experiment, we asked if the choice of response scale\u2014yes\/no compared to an 11-point scale\u2014affected the share of respondents who reported to be justly paid and follow-up questions on just earnings. Additionally, we asked if the 11-point scale succeeded in capturing more subtle deviations from the state of perfect justice compared to the binary format.<\/p>\n

Data and methods<\/h1>\n

Data<\/h2>\n

We drew on a survey experiment that was fielded as part of the Innovation Sample of the Socio-Economic Panel Study (Richter & Schupp 2015) in 2016 (10.5684\/soep.is.2019). SOEP-IS is a longitudinal general population survey fielded annually since 2011, with its starting cohort sampled from two subsamples of the SOEP-Core study. Additional random samples of private households in Germany were added to the SOEP-IS study in 2012, 2013, and 2014 (Richter & Schupp 2015; Zweck & Glemser 2018). All interviews in the 2016 SOEP-IS were conducted as computer-assisted personal interviews (CAPI) using showcards to display response options to respondents. Using a split-ballot design, 2562 respondents of SOEP-IS who reported to receive earnings from work were randomly allocated to one of two experimental groups: The first group received the binary response option and the second group received the 11-point scale response option. By comparing the share of those who evaluated their earnings as just, the just reward, and the intensity of reported injustice across conditions, we gained insights into the role of response format for measuring respondents\u2019 sense of justice. Appendix <\/a>Table A1 shows exact wording in English and German. Both groups first evaluated their gross earnings and subsequently their net earnings. We only report findings on gross earnings but substantive conclusions on the effect of response format were identical for net earnings (see Appendix<\/a> Table A2).<\/p>\n

\"\"<\/a>

Table 1: Summary statistics, by experimental condition.<\/b>
Data: SOEP-IS 2016. a<\/sup> Two-tailed t-test. b<\/sup>\u03c72<\/sup>-test. +<\/sup> p < .10, * p < .05, ** p < .01, *** p < .001<\/em><\/p><\/div>\n

Comparing basic socio-demographic and employment-related characteristics across experimental conditions revealed that randomization was successful (see Table 1). On average respondents in the binary scale condition did not differ from those in the 11-point scale condition with respect to gender, living in East Germany, years spent in education, occupational prestige, or gross earnings. Yet, respondents in the binary condition, on average, were about one year older compared to respondents in the 11-point scale condition. Additional analyses corroborated that the effects of response format were unaffected by systematic differences in age across conditions.<\/p>\n

Methods<\/h2>\n

Our main variables of interest were four measures of earnings justice commonly studied in empirical justice research. First, we investigated the effect of response format on the share of respondents who evaluated their earnings as just (the share of respondents that answered \u201cyes\u201d in the binary condition and those who answered 0 in the 11-point scale condition). Second, we investigated the effect of response format on the just earnings expressed in form of monthly gross earnings in Euro that would be considered just. All those respondents who evaluated their earnings as unjust were asked to state the level of earnings they would perceive as just. This is the observed just reward. Third, we calculated the hybrid just earnings, which included all respondents and used information on the actual earnings for those respondents who evaluated their earnings as just. This is common practice in empirical justice research and rests on the assumption that individuals perceive perfect justice if just earnings and actual earnings are perfectly matched (Jasso 1978). Information on actual earnings (A) and just earnings (C) were queried in an open response format and, therefore, prone to extreme outliers (e.g., accidently adding or dropping zeros). To avoid outliers driving our results, we trimmed responses at the 1st<\/sup> and 99th<\/sup> percentile. Fourth, following the justice evaluation function (Jasso 1978; Jasso, T\u00f6rnblom, & Sabbagh 2016), we calculated the experienced justice evaluation J*<\/em> = ln(A\/C<\/em>), using information on the hybrid just reward. Due to the natural logarithm, the experienced justice evaluation J*<\/em> was zero for respondents who evaluated their earnings as just. Finally, we investigated the association between the justice of earnings and related earnings evaluations (i.e., appropriateness of earnings and earnings satisfaction) across experimental conditions. For this, we ran logistic regressions in which a dummy variable (indicating whether gross earnings were evaluated as just) was regressed on a measure of the appropriateness of earnings (or earnings satisfaction) and its interaction with the experimental condition. This allowed us to investigate if the 11-point scale captured more fine-grained evaluations compared to the binary format.<\/p>\n

Results<\/h1>\n

Figure 1 shows the share of respondents who evaluated their gross earnings as just across experimental conditions. Among those who answered the question on the binary yes\/no scale, 57.99 percent perceived their earnings as just. Among those who answered the question on the 11-point scale, only 49.76 percent perceived their earnings as just. This difference of 8.23 percentage points was statistically significant (\u03c72<\/sup>(1) = 17.46, p<\/em> < .001). Therefore, the response-format systematically influenced the share of respondents evaluating their earnings as just.<\/p>\n

\"Answer<\/a>

Figure 1: Share of respondents who evaluated their gross earnings as just, by response scale.<\/b>
Data: SOEP-IS 2016. N = 2562. The share of missing values was 1.94 percent for the binary response format (left-hand panel) and .87 percent for the 11-point scale (right-hand panel).<\/em><\/p><\/div>\n

As described above, in addition to the share of respondents who evaluated their earnings as just, we calculated two measures of the just reward\u2014the observed and the hybrid just earnings\u2014as well as the experienced justice evaluation J*<\/em> (see Appendix <\/a>Table A3 for summary statistics). The average observed just earnings did not differ significantly between the yes\/no scale (M <\/em>= 2908.77, SD <\/em>= 1662.70) and the 11-point scale condition (M <\/em>= 2971.14, SD <\/em>= 1629.11), t<\/em>(1089) = -.63, p<\/em> = .53. The average hybrid just earnings were 2883.03 (SD<\/em> = 1885.95) in the yes\/no condition and 2869.66 (SD<\/em> = 1804.15) in the 11-point condition. These hybrid just earnings did not differ significantly across experimental conditions (t<\/em>(2319) = .17, p<\/em> = .86), suggesting the reported just earnings were unaffected by the format of the filter question.<\/p>\n

Figure 2 plots the distribution of the observed and hybrid monthly just gross earnings by experimental condition, showing largely similar patterns across the entire range of just earnings for both experimental conditions. Moreover, the mean justice evaluations J*<\/em> in the yes\/no scale condition (M <\/em>= -.12, SD <\/em>= .22) and the 11-point scale condition (M <\/em>= -.14, SD <\/em>= .25) did not differ significantly (t<\/em>(2281) = 1.23, p<\/em> = .22).<\/p>\n

\"Observed<\/a>

Figure 2: Observed and hybrid just gross earnings, by response scale.<\/b>
Data: SOEP-IS 2016. N (observed) = 1091, N (hybrid) = 2321. Observed just gross earnings: Just gross earnings of respondents who evaluated their earnings as unjust. Hybrid just earnings include all respondents and used information on the actual earnings for those respondents who evaluated their earnings as just.<\/em><\/p><\/div>\n

Association with related evaluations of earnings<\/h2>\n

Finally, we investigated if the 11-point scale succeeded in capturing more subtle deviations from the state of perfect justice compared to the binary format. Figure 3 shows predicted probabilities of evaluating one\u2019s own gross earnings as just along with respondents\u2019 agreement to the statement \u201cWhen I consider all my accomplishments and efforts, my salary seems to be appropriate\u201d for the two response formats. The predicted probabilities barely differed in the extreme ends of the appropriateness-spectrum (also see Appendix<\/a> Table A4). This means, respondents, who strongly disagreed that their earnings were appropriate, almost never rated their earnings as just. This was the case in both response formats (\u03c72<\/sup>(1) = 1.76, p<\/em> = .18). Similarly, respondents, who strongly agreed that their earnings were appropriate, predominantly rated their earnings as just in both response formats (\u03c72<\/sup>(1) = 1.14, p<\/em> = .29). However, for the middle categories \u201cdisagree\u201d (\u03c72<\/sup>(1) = 8.55, p<\/em> < .01) and \u201cagree\u201d (\u03c72<\/sup>(1) = 12.52, p<\/em> < .001), predicted probabilities to rate earnings as just were significantly lower in the 11-point format than in the binary format. Consequently, respondents with ambiguous feelings regarding the justice of their earnings were more likely to rate their earnings as just if confronted with a binary choice. Similar respondents were more likely to differentiate their injustice ratings on the 11-point scale.<\/p>\n

\"Justice<\/a>

Figure 3: Association of likelihood to evaluate earnings as just with appropriateness of earnings, by response scale<\/b>
Data: SOEP-IS 2016. N = 790. Displayed are predicted probabilities (with 95-% confidence intervals) based on a logistic regression, where the likelihood to evaluate one\u2019s own earnings as just was regressed on the perception of appropriateness of earnings and its interaction with the experimental condition. As the single-item measure on the appropriateness of earnings was only fielded in a subsample of the SOEP-IS, the number of observations for this analysis decreased compared to the full sample. Predicted probabilities are presented in Appendix<\/a> Table A4.<\/em><\/p><\/div>\n

We further studied the association of respondents\u2019 justice evaluation and their earnings satisfaction measured on an 11-point scale ranging from \u201ccompletely dissatisfied\u201d (0) to \u201ccompletely satisfied\u201d (10). We found that respondents in the yes\/no condition who were satisfied with their earnings\u2014scoring 6 or higher\u2014were more likely to evaluate their earnings as just compared to respondents in the 11-point condition (see Figure 4 and Appendix <\/a>Table A4). For respondents who were dissatisfied with their earnings, no such differences between scale conditions was observed. Our findings therefore indicated that the 11-point scale captured more subtle nuances of injustice.<\/p>\n

\"\"<\/a>

Figure 4: Association of likelihood to evaluate earnings as just with earnings satisfaction, by response scale.<\/b>
Data: SOEP-IS 2016. N = 2557. Displayed are predicted probabilities (with 95-% confidence intervals) based on a logit regression, where the likelihood to evaluate one\u2019s own earnings as just was regressed on earnings satisfaction and its interaction with the experimental condition. Predicted probabilities are presented in Appendix <\/a>Table A4.<\/em><\/p><\/div>\n

Conclusion and Discussion<\/h1>\n

Addressing the need for clarification on the role of measurement choices in assessing the perceived justice of earnings, we analysed a split-ballot experiment in a random sample of German adults. Findings from the SOEP-IS underscored that the share of respondents who evaluated their own gross earnings as just varied systematically by response format. Fewer respondents evaluated their earnings as just in the 11-point scale condition compared to the binary condition. Question wording in the yes\/no condition that emphasized justice but not injustice, may have led respondents to provide a satisficing \u201cyes\u201d response. Moreover, we found supportive evidence that the observed differences were related to respondents with ambiguous feelings regarding their earnings being more likely to report their earnings to be just in the binary response format. This method effect also suggests that the observed drop in individuals who evaluated their earnings as just between 2015 and 2017 in the SOEP must\u2014at least partially\u2014be attributed to changes in measurement. Follow-up questions on just earnings seemed, however, more robust to changes in the response scale as neither just earnings nor the derived justice evaluation J*<\/em> varied systematically by experimental condition. This suggests that time-series information on just earnings in the SOEP are robust to the implemented changes in response scale while the share of respondents who consider themselves justly paid is sensitive to such changes.<\/p>\n

Survey methodologists warn that respondents may understand varying rating scales differently and that researchers therefore cannot assume that the same latent variable is measured (Menold & Tausch 2016). From this perspective, it might be advisable to refrain from comparing results based on different scales entirely. However, for practitioners in the field, such comparisons within and between surveys are often the only possibility to approach important research questions. Based on our experimental results, we advise researchers to refrain from comparing aggregate measures on the justice of earnings between and within surveys if there are differences in response scales. Moreover, they should also consider that binary response formats may not pick up small deviations from the state of perfect justice. More elaborate measures of earnings justice that use information on just earnings seem more robust to changes in measurement. This suggests that the SOEP time-series on just earnings spanning 2005 to 2019 is intact. Survey organizers in the field should carefully consider the effect of changes in response scale, including the effect on follow-up questions (also see Menold & Tausch 2016). In any case, they should clearly flag such changes in their survey documentation and, if possible, use survey experiments to understand the effects of the implemented changes.<\/p>\n","protected":false},"excerpt":{"rendered":"

Introduction Empirical justice research has gained momentum in recent decades. Questions about fair treatment at work, fair chances to excel in life, and the fairness of income are investigated in various social science disciplines (Liebig and Sauer 2016). A vital branch of empirical justice research investigates distributive justice and, in particular, the justice of one\u2019s […]<\/p>\n","protected":false},"author":2666,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[41],"tags":[804,802,803,228,373],"class_list":["post-16588","post","type-post","status-publish","format-standard","hentry","category-questionnaire_design","tag-germany","tag-justice-of-earnings","tag-response-format","tag-soep","tag-survey-experiment"],"acf":[],"_links":{"self":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/16588","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/users\/2666"}],"replies":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16588"}],"version-history":[{"count":63,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/16588\/revisions"}],"predecessor-version":[{"id":17718,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/16588\/revisions\/17718"}],"wp:attachment":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16588"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=16588"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=16588"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}