![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2015\/06\/Figure-1.png\")
<\/a><\/p>\nSource: \u00a0\u00a0\u00a0\u00a0\u00a0 http:\/\/www.paywizard.co.uk\/main\/pay\/salarysurvey\/salary-survey-employees<\/a>, accessed 8 AUG 2014<\/p>\n This paper uses the data of the LISS (Longitudinal Internet Studies for the Social Sciences) panel. LISS is a probability-based online panel in the Netherlands and consists of 5,000 households, comprising of 9,219 individuals aged 16 and over (October 2009). The LISS panel is part of the MESS project (Measurement and Experimentation in the Social Sciences) and it is administered by CentERdata at Tilburg University. The panel was drawn from the population register in collaboration with Statistics Netherlands. Even though the questionnaire is completed online, all people in the sample were recruited in traditional ways by letter, followed by telephone call and\/or house visit with an invitation to participate in the panel (for details about the recruitment: Scherpenzeel & Das 2010; Scherpenzeel & Bethlehem, 2011). Households that could not otherwise participate have been provided with a computer and Internet connection.<\/p>\n Each month the panel members are asked to complete a questionnaire. In October 2009 the LISS panel was used for a study to further insight into bias in volunteer samples and to develop methods to adjust for survey bias. The Dutch questionnaire of the WageIndicator web survey was completed by the LISS panel members. Full details of the results of the comparison between the LISS and the WageIndicator data can be found in Steinmetz et al. (2014). Appendix 1 holds the Dutch and the English Codebook of the survey. The current article uses the LISS data regarding the survey question \u2018What is your occupation?\u2019 It does not compare the two datasets, but focusses solely on the data from the LISS panel.<\/p>\n In total 5,577 persons responded to this particular LISS survey, reflecting a response rate of 60.5% (Hootsen, 2010). Note that the monthly response of participants varies between 50 and 80%. For our study, only respondents in paid employment were asked about their occupation (3,444 respondents). The occupation question was not asked to students, retired persons and other individuals not active in the labour market. Note that in the LISS panel respondents hardly break off during survey completion because they are instructed not to do so, whereas in the volunteer WageIndicator web survey they do break off to a considerable degree.<\/p>\n The LISS respondents could self-identify their occupation by using a compulsory three-level search tree with a look-up table of 1,603 unique occupational titles, all coded according to the most recent ISCO-08 classification. The occupation search tree used in the LISS panel was similar to the one used in the WageIndicator web survey in the Netherlands. Appendix 2 presents the search tree and its look-up table in Dutch and its translations in English. Note that today the semantic matching technique is widely used for searching an occupation look-up table, particularly by job boards and employment agencies, but in 2009 this technique was not yet in use for the LISS survey. The search tree consisted of 23 entries in level 1 (for example \u2018Guards, army, police\u2019), 207 entries in level 2 (for example \u2018Guard\u2019), and 1,603 occupational titles in level 3 (for example \u2018Bodyguard\u2019 or \u2018Doorkeeper\u2019). Hence, the entries in the 3rd level jointly make up the look-up table. In this level, some occupational titles are inserted on more than one place if the search paths were ambiguous, making in total 2,456 entries. To explore the quality of the search tree and its look-up table, the LISS search tree was in one respect different from the WageIndicator tree. On request of the author one extra feature was added. At the bottom of each 3rd level in the search tree an option \u2018other\u2019 and a subsequent text box was included, allowing to study to what extent and for which respondents the search tree and its look-up table were not sufficiently detailed.<\/p>\n The research questions in our study are threefold. First, what proportion of respondents ticked \u2018other\u2019 and reported their occupation via the text box in the search tree? Of these, what proportion could have identified their occupation in the search tree and what proportion had an occupation which was absent in the look-up table?<\/p>\n Second, is the use of \u2018other\u2019 and the text box related to the design of the levels in the search tree, to the look-up table or possibly to respondents\u2019 personal characteristics? Here we specify for those who could have identified their occupation and for those whose occupation was absent in the table.<\/p>\n Third, how many of the 1,603 occupational titles were used by the respondents and how often were they ticked? Were the highly skewed distribution and the very long tail reflected in the response and how could this distribution best be described?<\/p>\n For the purpose of this study, the author coded all text box responses manually and identified whether the coded occupation actually was available in the look-up table or not. Descriptive statistics were used for the first research objective. For the second objective, the likelihood of ticking the text box was modeled for the 23 first level entries and for the personal characteristics age, gender and waged employment. For the third objective the distributional characteristics of the occupation look-up table were used. Note that time stamps or other para-data have not been used in the analyses here.<\/p>\n The first objective aims to present descriptive statistics about the use of the search tree and the text box. Table 1 shows that in the LISS survey drop out during search tree completion is 0.5%, which is much lower compared to the 10 to 20% drop-out rates in the search tree in the volunteer WageIndicator web surveys in Great Britain, Belgium and the Netherlands (Tijdens 2014b). Of the LISS respondents who completed the 3rd level in the search tree, 67% selected an occupation from the look-up table and 32% ticked \u2018other\u2019 and entered their job title in the text box. In a next step the author coded these job titles, using the look-up database with 1,603 job titles. It turned out that 14% could have identified their occupation in the 3rd level of the search tree, but obviously had not found it, implying that they had used different search paths which did not result in their occupation at the 3rd<\/sup> level. Note that the coding process was solely based on the matching of the occupations keyed in with the look-up database, thereby only controlling for typing errors. Job descriptions were not asked in the survey.\u00a0 The remaining 17% expressed an occupation which was indeed absent in the look-up table. If job descriptions had been asked, probably more occupations could have been coded according to the look-up table and had therefore not been classified as absent. Another 0.5% keyed in unidentifiable text.<\/p>\n The 23 first level entries in the search tree are the largest hurdle for respondents, because the number of characters to be read is large and might therefore be difficult to comprehend. Per entry Table 2 shows the distribution over the three groups \u2013 the respondents who selected an occupation in the search tree, the ones who ticked the text box but could have found their occupation, and the ones who ticked the text box and the occupation was absent. Table 2 points to the most problematic entries. In the first level entry \u2018Oil, gas, mining, utilities\u2019 only 18% of respondents could identify their occupation in the look-up table, whereas 56% keyed in an occupational title that was absent in the table, pointing to shortcomings in the look-up table. This shortcoming also appears to be the case for the entry \u2018Cars, mechanics, technicians, engineers\u2019 where the share of absent occupations is high with 30%. For the entry \u2018Food manufacturing\u2019 only 44% of respondents could identify their occupation. Here the problem seems to be related to shortcomings in the search paths, because 28% respondents could not find their occupations although these were included.<\/p>\n The last column in Table 2 presents the ratio between the columns C and D. Higher ratios point to relative difficulties in the look-up table, whereas lower ratios do so for the search paths. For the entry \u2018Management, direction\u2019 the look-up table obviously has many missing occupations, whereas for the entry \u2018Clerks, secretaries, post, telephone\u2019 the search paths need to be improved.<\/p>\n Source:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 WageIndicator Questionnaire administered to the LISS panel, October 2009, excluding respondents who did not complete all three levels (18 observations), who entered unidentifiable text (16 observations) and who had missing values for gender (3 observations)<\/span><\/p>\n The second objective is to explore whether the use of the text box is related to the design of the search tree or to respondents\u2019 personal characteristics. Model 1 in Table 3 explores this for those respondents who could have identified their occupation versus those who selected an occupation in the search tree, hence identifying problematic search paths (496 versus 2,312 respondents). In Model 2 we do so for those whose occupation is absent in the database versus those who selected an occupation in the search tree, hence pointing to problems in the look-up table (599 versus 2,312 respondents).<\/p>\n For respondents selecting the first level entry \u2018Oil, gas, mining, utilities\u2019 the odds ratios in Model 1 increases approximately 9 times compared to the reference entry. For respondents who selected the first level entry \u2018Food manufacturing\u2019 the odds ratios increases 4 times. The first level entries \u2018Media, graphic, printing, culture, design\u2019, \u2018Legal, administration, inspection, policy adviser\u2019, and \u2018Construction, fittings, housing\u2019 reveal increases of more than 2 times. The effects of the search tree entries hardly change once personal characteristics are included in Model 1b.<\/p>\n In Model 2 – entering a job title that is absent in the look-up table -, the odds ratios for the first level entry \u2018Oil, gas, mining, utilities\u2019 increase even 15 times compared to the reference entry. For two entries the odds ratio increase 3 times, namely \u2018Food manufacturing\u2019 and \u2018Cars, mechanics, technicians, engineers\u2019. For another two entries they increase more than 2 times, namely \u2018Industrial production, manufacture, metal\u2019 and \u2018Agriculture, nature, animals, environment\u2019. Here too the effects hardly change when personal characteristics are entered into Model 2b.<\/p>\n Concerning the personal characteristics Table 3 shows that for respondents in waged employment the odds ratio in Model 1b decreases with 26%, whereas the odds ratio in Model 2b is not affected significantly. For women, the odds ratio increases with 40% in Model 1b whereas the odds ratio in Model 2b is not affected significantly. In both models, the odds ratios increase with age.<\/p>\n In conclusion, the use of the text box is highly affected by the design of the search tree. In particular, five of the 23 first level entries point to ambiguous search paths, requiring that more occupations from the look-up table are to be inserted in these entries. Another five first level entries point to absent occupations and these have to be added to the look-up table.<\/p>\n The third objective is to explore how many of the 1,603 occupational titles in the look-up table are used by the respondents and how often each title is selected. In other words: how is the highly skewed occupational distribution in the labour force reflected in the survey response and how could this distribution best be described?<\/p>\n The 2,313 respondents who did select an occupation in the search tree used 585 of the 1,603 titles in the look-up table. The 497 respondents who completed the text box but could have identified their occupational title used 207 titles from the table, of which 139 were also selected by the group of 2,313 respondents. Jointly these two groups of 2,810 respondents ticked 653 titles.<\/p>\n Figure 2 reveals that very few occupations are selected by 30 respondents or more. This applies to 4 occupations, selected by 10.8% of the 2,313 respondents, who selected an occupation in the search tree. It applies to 6 occupations, selected by 13.0% of the 2,810 respondents, after the text box answers were coded. Frequently mentioned occupations are \u2018Office clerk\u2019, \u2018Primary school teacher\u2019, \u2018Health associate professional\u2019 and \u2018Elderly aide\u2019. Only 48 respectively 60 titles were selected by 10 to 29 respondents and another 187 respectively 210 titles by 3 to 9 respondents, totaling to 69% respectively 70% of respondents. Another 114 respectively 126 titles were selected by only 2 respondents and 236 respectively 257 occupations only once. In total 1,018 respectively 950 of the 1,603 titles in the look-up table were not selected in our survey.<\/p>\n Source:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 WageIndicator Questionnaire administered to the LISS panel, October 2009<\/p>\n In total 600 respondents completed the text box and could not have identified their job titles, because they were absent in the look-up table. After cleaning for misspellings and harmonization of gendered job titles, they keyed in 555 different job titles.<\/p>\n In summary, the look-up table included 1,603 occupational titles, of which 653 were selected by respondents (41%). An additional 555 occupational titles should have been listed, but were absent. Assuming that the unlisted occupational titles hold the same rate of 41%, the look-up table should have been extended with 1,362 titles to meet the demands of the LISS sample. Hence, a sample size of 3,444 respondents requires a database of at least 1,603 + 1,362 = 2,965 titles. Larger sample sizes require larger databases.<\/p>\n Web surveys allow for respondents\u2019 self-coding by using a search tree with a look-up table for the survey question \u2018What is your occupation?\u2019. This is in contrast to other survey modes which can apply mainly an open format question with office-coding. Using a representative sample of 3,444 web survey respondents in the Netherlands and a three-level search tree with a look-up table with 1,603 occupational titles, 67% of respondents selected an occupation from this table and 32% used the text box, which was included as the last entry at each 3rd<\/sup> level of the search tree. \u00a0After coding these responses, it turned out that almost half of them could have identified their occupation but had not found and thus pointing to poor search paths, whereas slightly over half expressed an occupation which was absent in the look-up table. Using multivariate analyses, we identified that for five of the 23 search tree\u2019s first level entries the likelihood of text box use due to poor search paths was substantially higher than in the remaining entries. For another five entries, of which two were overlapping, the likelihood of an absent occupation was substantially higher. For this reason particularly respondents with an occupation in the entry \u2018Oil, gas, mining, and utilities\u2019 encountered difficulties in identifying their occupation. Older respondents had more difficulties in identifying their occupation in the look-up table, but we have no evidence whether older respondents have more cognitive difficulties in doing so or that the table included fewer occupations associated with older workers.<\/p>\n Given the 10,000s of occupational titles in any national labour force and the long tail of the distribution of workers over occupations, it is not surprising that the skewed distribution was noticed in our sample too. 11% of respondents selected only 4 occupations, 69% selected 235 occupations, and the remaining 20% selected 350 titles. We computed that our look-up table would have needed at least 2,965 instead of 1,603 occupational titles to allow all respondents to select an occupation. Larger samples will need larger tables. Of course, a major challenge relates to identifying which occupational titles should be included in the look-up table, because this has to be determined before a web survey starts. Otherwise, manual coding remains necessary, and this is particularly expensive for the many occupations with relative few jobholders in the long tail. As explained, the number of entries in a search tree is maximized. Therefore, a semantic matching tool should be preferred over a search tree when large numbers of entries are included in the look-up table.<\/p>\n Although beyond the scope of this article, we want to make a comment concerning semantic matching used for respondents\u2019 self-identification compared to auto-coders used for office-coding. Although both require a list of coded occupational titles, a look-up table for self-identification is different from a training set for auto coders facilitating machine learning algorithms. During survey completion a semantic matching tool provides respondents with a list of matched occupations while they type a few characters and the match list is adapted when respondents enter more characters. Auto-coders are used after survey completion and they also need to correct text strings for typing errors, for highly aggregated occupational titles, for synonyms, for female\/male expressions of occupations, and alike, leading to a set of so-called hard codes. Semantic matching lists will not include hard codes, because typing errors will not lead to matches and respondents will understand instantly that they need to correct for errors for the purpose of a match.\u00a0 Look-up tables for semantic matching should not include the aggregate occupational title \u2018clerk\u2019, but a list of specified clerk titles, inviting respondents to tick one of them. In many industrialised countries, auto-coding has gradually developed, as for example publications by Hoffmeyer-Zlotnik, Hess, Geis (2006) and Bethmann et al (2014) show for Germany.<\/p>\n Survey holders can of course use the search tree and look-up table used in this study (see Appendix 2 for the table in Dutch and in English). The first four digits in the first column reflect the ISCO-08 code. On request the author can supply translations in other languages. As of mid-2015 an extension of the look-up table is scheduled. The table will be made available by means of an API (Application Program Interface), such that any survey holder can include a link in his\/her web survey that calls for this API for the survey question \u2018What is your occupation?\u2019.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Many surveys have one or more questions with thousands of response categories, the so-called long-list variables such as occupation, industry, car brand, medical drugs, company name and alike. This paper focusses on the measurement of occupations, addressed in most socio-economic and health surveys with a question \u2018What is your occupation?\u2019 or similar (see for […]<\/p>\n","protected":false},"author":1035,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[314,312,326,311,315],"class_list":["post-6967","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-closed-survey-question","tag-isco-classification","tag-look-up-table","tag-occupations","tag-search-tree"],"acf":[],"_links":{"self":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/6967","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/users\/1035"}],"replies":[{"embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6967"}],"version-history":[{"count":34,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/6967\/revisions"}],"predecessor-version":[{"id":7162,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=\/wp\/v2\/posts\/6967\/revisions\/7162"}],"wp:attachment":[{"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6967"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6967"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/surveyinsights.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6967"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Data and methods<\/h1>\n
Data<\/h2>\n
Research questions<\/h2>\n
Methods<\/h2>\n
Results<\/h1>\n
Use of the search tree and the text box<\/h2>\n
<\/a><\/p>\n<\/a><\/p>\nWho uses the text box?<\/h2>\n
<\/a><\/p>\nWhich occupations are ticked?<\/h2>\n
Figure 2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 The number of occupations selected by the 2,313 respondents who selected an occupation and by the 2,810 respondents including those with coded occupation<\/h4>\n
<\/a><\/p>\nConclusion\/Discussion<\/h1>\n
<\/h1>\n
Appendix 1: The WageIndicator Questionnaire administered to the LISS panel<\/h2>\n
Dutch version<\/a><\/h3>\n
English version<\/a><\/h3>\n
Appendix 2: The look-up table and its search tree.<\/h2>\n
WageIndicator occupation database<\/a><\/h3>\n
<\/h3>\n