From a survey methodology point of view, mobile CATI surveys are still at an infancy stage. Researchers have begun the process of optimizing mobile calling methods but there is much work to be done. A logical starting point is to use the methods developed for fixed CATI surveys and explore which adaptations should be made for the specificity of mobile phones (Reimer et al. (and , respectively)2012). Our study contributes to this topic by evaluating call outcomes and call patterns in the context of a mobile CATI survey. We use the term mobile CATI survey to denote an interviewer-administered telephone interview conducted in a CATI facility where the respondents answer using a mobile phone.<\/p>\n
In fixed CATI surveys, it is recognized that call scheduling, i.e., days and times of the calls, number of callbacks, days and times of callbacks, time lag between calls, has a great influence on the probability of making contact and of gaining cooperation (Hansen 2008). Although this is to some extent applicable to mobile CATI surveys, previous work has shown that calling strategies that maximize the likelihood of contact and of interview for fixed CATI surveys may not be the most effective for mobile CATI surveys. For example, Montgomery et al. (2011) and Reimer et al. (2012) identified significant differences in best times to make first calls for fixed and mobile CATI surveys.<\/p>\n
As previous research on other survey modes suggests (e.g. Massey et al 1996, Groves and Couper 1998, Stec et al. 2004), when investigating call pattern effects, it is pertinent to distinguish between: (a) different call outcomes (contact, interview, refusal, non-contact), and (b) first calls and subsequent calls. In our study, we are interested in identifying calling protocol features of a mobile CATI survey that increase the likelihood of making contact and of obtaining an interview. We explore which timing periods are best when making first call attempts, and which call history factors most affect the probability of making contact and obtaining an interview in calls after the first dial. Given the small body of research about mobile CATI surveys, our study does not have strong theoretical predictions about the expected effects of call scheduling. Whenever pertinent, we draw a parallel with fixed CATI surveys.<\/p>\n
Hypotheses<\/strong><\/h1>\nConsiderable research has been conducted on the best times to make contact and obtain peoples\u2019 cooperation in surveys. In household interview surveys, the relative success of a call is dependent on the at-home times of households. Specifically, the likelihood of finding someone at home is greater in the evening hours and on weekends (Weeks et al. 1980) and therefore these tend to be the best periods to make contact. Groves and Couper (1998) show that approximately half of the first calls yield a contact in household surveys but this figure rises to nearly 60% if the call is made in the evening (after 6 pm) or during the day on weekends. For fixed CATI surveys, contact rates tend to be higher on week nights (after 5 pm) than they are during weekday mornings and afternoons when many respondents are at work (Brick et al. 1996, Massey et al. 1996, Groves and Couper 1998, Stec et al. 2004, Montgomery et al. 2011). Weekend calling often results in higher contact rates than weekday daytime calling (Massey et al. 1996, Stec et al. 2004). In mobile CATI surveys, there is evidence that contact rates are more consistent across timing periods than they are for fixed CATI surveys (Montgomery et al. 2011, Brick et al. 2007, Yuan et al. 2005). Even so, the Code of the American Association for Public Opinion Research<\/a> (AAPOR) suggests that mobile phone \u201ccalls should be attempted on different days of the week and times of the day\u201d to increase the likelihood of contact and of interview (AAPOR\u2013Article on Calling Protocols 2008). Similarly, the Code for Mobile Research of the European Society for Opinion and Market Research (ESOMAR) states that \u201cresearchers should observe the same calling hours as for fixed-line phone surveys\u201d (ESOMAR 2011, 2012). Additionally, family\u00a0 time occupation is not uniform across days and hours of the day which is likely to influence people’s availability to take mobile phone calls and to agree to an interview. For example, Tuesdays and Wednesdays tend to be the busiest days for the working population and Sundays and Saturdays the least occupied. In terms of timing, workers are usually on the home-work-home commute between 7.30 and 8.30 am and between 5.30 and 7 pm, they are in their work place between 10.30 am and 5.30 pm, and are usually at home after 7 pm (European Commission 2004, p.32). Thus, for mobile CATI surveys we hypothesize:<\/p>\nHypothesis 1A:<\/strong> The timing period of first calls affects the likelihood of contact<\/p>\nHypothesis 1B:<\/strong> The timing period of first calls affects the likelihood of interview<\/p>\nAlthough good survey practice involves attempting to complete the survey based on the sample initially drawn, it is well known that a 100% response rate is never achieved with a single call attempt to sample units. The number of callbacks to non-contacted numbers varies depending on the research design, budget and the length of the fieldwork period, but it is generally accepted that the probability of making contact and obtaining an interview increases with more call attempts both in fixed CATI surveys (Schmich and Jentsch 2012, Holbrook et al. 2008) and mobile CATI surveys (Vicente and Reis 2009, Steeh et al. 2007). Literature suggests that around 12 to 15 call attempts is the optimal number for fixed CATI surveys (Srinath et al. 2001, Hansen 2008, Schmich and Jentsch 2012) but the Code of AAPOR recommends 6 to 10 calls for mobile CATI surveys because the mobile phone is a personal and private instrument and overburdening should be avoided (AAPOR\u2013Article on Calling Protocols 2008, p.26-27); the Code of ESOMAR suggests \u201climiting the number and pattern of callbacks when contacting a known mobile number\u201d (ESOMAR 2011).<\/p>\n
It is acknowledged that it is difficult in fixed CATI surveys to contact or interview sample units with more than five prior \u201cbusy\u201d or \u201crings but no answer\u201d outcomes (Stec et al. 2004). Mobile phones have call-blocking features that impede or postpone accessing the mobile phone user: the voice-mail can be activated and calls diverted without answering them; the mobile phone can be set on silent mode or even turned off, blocking the contact at the moment of the call. Thus, we hypothesize:<\/p>\n
Hypothesis 2A:<\/strong> Previous call outcomes affect the likelihood of contact<\/p>\nHypothesis 2B:<\/strong> Previous call outcomes affect the likelihood of interview<\/p>\nFor fixed CATI surveys, some previous work has addressed the optimal amount of time to allow between call attempts to the same telephone number. Results are somewhat mixed, and this may be because of differences in the length of the field period and of the time lag in surveys, and in the frequency of attempts to call the same number. Stokes and Greenberg (1990) found that longer delays between first and second calls were associated with higher contact rates, while Reimer et al. (2012) found that longer time lags (six or more days) between first and second calls were less likely to yield a cooperative contact. In addition to shorter (or greater) time lag between calls, callback efficiency can rise when the timing period (day and time) of the calls varies (Kulka and Weeks 1988, Stokes and Greenberg 1990). In mobile CATI surveys, Reimer et al. (2012) found that shorter lag times result in more cooperation, although this effect was evaluated considering only the time lag between first and second calls. Thus:<\/p>\n
Hypothesis 3A:<\/strong> Time pattern of callbacks affects the likelihood of contact<\/p>\nHypothesis 3B:<\/strong> Time pattern of callbacks affects the likelihood of interview<\/p>\nData and methods<\/h1>\nData collection<\/strong><\/h2>\nThe data used for this study come from a nationwide mobile CATI survey conducted in Portugal by a market research company in its CATI facilities during May 2012. The selection of the mobile phone numbers to be dialed followed the survey company’s standard procedures in studies of this size. In Portugal, there are no official lists of mobile phone numbers (or mobile phone subscribers) that can be used as a sampling frame, and mobile operators provide no information on the attribution of numbers. The sample of mobile numbers was therefore randomly generated. Mobile phones have nine-digit numbers and the first two digits \u2013 91, 93 or 96 \u2013 identify the operator. The Portuguese Telecommunications Regulation Authority provides information about the market share of each of the three mobile phone operators in Portugal and this was used to stratify the sample into three subgroups. Within each two-digit prefix, mobile phone numbers were created by a generator of 7-digit random numbers. A total of 11,472 mobile phone numbers were dialed. Calls were made between 5pm and 10pm on weekdays and between 10am and 2pm on weekends following the ESOMAR Code that recommends survey companies avoid anti-social times in mobile CATI surveys (ESOMAR 2011, 2012). The CATI system automatically managed the dialing of numbers and callbacks. Mobile phone numbers were randomly assigned to interviewers.<\/p>\n
One of the following disposition codes was attributed to numbers after each call: (a) completed interview; (b) partial interview, when the interview did not reach the end; (c) refusal, when the mobile phone user answered the call but refused to cooperate; (d) busy; (e) rings but no answer, when the call was not answered after 8 ring tones; (e) voice mail (message from the operator directing the caller to voice mail either immediately or after 7 ring tones); (f) temporarily unavailable (message from the operator saying \u201cthe mobile number you dialed is not available at the moment, please try later\u201d); (g) reject the call (hang up without answering the call); (h) out of the scope (if the mobile user was less than 15 years old); (i) non-working or disabled (when an almost continuous busy signal was heard); (j) non attributed (message from the operator). In addition to the information about each call outcome, the date and time of calls are recorded automatically by the CATI system which is used in our study to describe the call history for a given number (number of calls, time elapsed between successive calls, and number of calls resulting in a specific outcome).<\/p>\n
For the analysis, \u201ccontact\u201d is defined as any call in which a mobile phone user was reached and includes those coded as completed interview, partial interview, refusal, and out of the scope. A \u201cnoncontact\u201d includes cases coded as busy signal, rings but no answer, voice mail, temporarily unavailable and reject the call. An \u201cinterview\u201d corresponds to the cases coded as \u201ccompleted interview\u201d. Cases coded as non-attributed, non-working or disabled are considered \u201cineligible\u201d for the purposes of this study. If a call outcome is a completed or partial interview, that number is not called back. If a call outcome is a \u201cnon-contact\u201d, that mobile number stays in the \u201csampling frame\u201d and the scheduling algorithm may select it again to be called back. The survey company makes three call backs to cases coded as \u201ctemporarily unavailable\u201d and the mobile number is then coded as \u201crings but no answer\u201d if the \u201ctemporarily unavailable\u201d status continues after the third attempt. A \u201crefusal\u201d is only called back with the agreement of the mobile phone user and is otherwise excluded from the \u201csampling frame\u201d. Numbers with a non-functional, disabled or non-attributed call outcome are immediately excluded from the \u201csampling frame\u201d. Ineligible numbers were considered irrelevant to our analysis and therefore excluded as our goal is to explore the determinants of \u201ccontact\u201d and \u201cinterview\u201d. Among the 11,472 mobile phone numbers dialed, 8,083 were eligible (final disposition). The eligible numbers were called 19,890 times which means an average of almost 3 call attempts per number. The number of call attempts ranged from 1 to 15 which is the survey company\u2019s standard procedure for mobile CATI surveys.<\/p>\n
<\/em><\/strong>Modeling<\/strong> likelihood of contact and of interview<\/strong><\/h2>\nRegression models are commonly used to estimate the likelihood of contact and likelihood of interview in surveys, more specifically, binary logistic regression (Groves and Couper 1998, Brick et al. 1996), OLS regression (Holbrook et al. 2008) or multinomial logistic regression (Steele and Durrant 2011). Multi-level models are an alternative approach when information is available about sample design features with clustering effects (e.g. O\u2019Muircheartaigh and Campanelli 1999, Durrant and Steele 2009, Wagner 2009).<\/p>\n
We adopt binary logistic regression to estimate the probability of making contact \u2013 dependent variable assuming the codes 1-\u201ccontact yes\u201d and 0-\u201ccontact no\u201d \u2013 and the probability of interview \u2013 dependent variable assuming the codes 1-\u201cinterview yes\u201d and 0-\u201cinterview no\u201d. \u201cNo interview\u201d includes cases that were contacted but refused cooperation, and cases that could not be contacted at all. When modelling the likelihood of contact and the likelihood of interview, we distinguish between first calls and subsequent calls.<\/p>\n
As the time and day of the call is the only information available for first calls, the model of first call includes the following explanatory categorical variables: (1) DAY (seven categories) which represents the day of week of the call, and (2) TIME (five categories) which represents the time slot of the call (10am-12noon; 12noon-2pm; 5 pm-7 pm; 7 pm-9 pm; 9 pm-10 pm). The categories of DAY and TIME are included in the models as dummy variables. Tuesday and 7 pm-9 pm were chosen as the reference categories of the DAY and TIME, respectively, as the biggest percentage of eligible mobile numbers were called then (n=2743 and n=3111). In addition to main effects, the model includes interaction effects as there is evidence from fixed CATI surveys that specific combinations of day*time affect the likelihood of contact and of interview (e.g. Brick et al. 1996, Massey et al. 1996, Stec et al. 2004).<\/p>\n
We model the likelihood of contact and of interview in subsequent calls (two to fifteen attempts) by taking data related to the call history of the numbers. The models have the following explanatory variables: (1) CALL WEEKEND which is the ratio between the number of calls made on weekends and the total number of call attempts for a given mobile phone number, (2) CALL MONDAY, CALL TUESDAY, CALL WEDNESDAY, CALL THURSDAY, CALL FRIDAY which is the proportion of calls to a mobile phone number on each week day, (3) CALL 5 pm-7 pm, CALL 7 pm-9 pm, CALL 9 pm-10 pm which is the proportion of calls to a mobile phone number on each time slot (the 10 am-12 noon and 12 noon-2 pm shifts are not included because they are implicit in the CALL WEEKEND factor), (4) TIME LAG which is the mean time elapsed between two consecutive calls to a mobile phone number, (5) BUSY which is the proportion of a mobile phone number’s call outcomes that was coded as \u201dbusy\u201d, (6) VOICE MAIL which is the proportion of a mobile phone number’s call outcomes that was coded as \u201cvoice mail\u201d, (7) NO ANSWER which is the proportion of a mobile phone number’s call outcomes that was coded as \u201crings but no answer\u201d and (8) TEMP_UNAVAILABLE which is the proportion of a mobile phone number’s call outcomes that was coded as \u201ctemporarily unavailable\u201d. All factors are metric and standardized variables. Binary logistic regression coefficients were estimated using the Maximum Likelihood Method and a Stepwise approach was adopted using 0.05 for statistical significance level.<\/p>\n
Results<\/h1>\n
The results for the likelihood of contact and of interview on first calls are shown in Table 1. Only the variables with a significant effect are presented (p<0.05<\/em> for Stepwise entering).<\/p>\n
Hypothesis 1A:<\/strong> The timing period of first calls affects the likelihood of contact<\/p>\n Hypothesis 1B:<\/strong> The timing period of first calls affects the likelihood of interview<\/p>\n Although good survey practice involves attempting to complete the survey based on the sample initially drawn, it is well known that a 100% response rate is never achieved with a single call attempt to sample units. The number of callbacks to non-contacted numbers varies depending on the research design, budget and the length of the fieldwork period, but it is generally accepted that the probability of making contact and obtaining an interview increases with more call attempts both in fixed CATI surveys (Schmich and Jentsch 2012, Holbrook et al. 2008) and mobile CATI surveys (Vicente and Reis 2009, Steeh et al. 2007). Literature suggests that around 12 to 15 call attempts is the optimal number for fixed CATI surveys (Srinath et al. 2001, Hansen 2008, Schmich and Jentsch 2012) but the Code of AAPOR recommends 6 to 10 calls for mobile CATI surveys because the mobile phone is a personal and private instrument and overburdening should be avoided (AAPOR\u2013Article on Calling Protocols 2008, p.26-27); the Code of ESOMAR suggests \u201climiting the number and pattern of callbacks when contacting a known mobile number\u201d (ESOMAR 2011).<\/p>\n It is acknowledged that it is difficult in fixed CATI surveys to contact or interview sample units with more than five prior \u201cbusy\u201d or \u201crings but no answer\u201d outcomes (Stec et al. 2004). Mobile phones have call-blocking features that impede or postpone accessing the mobile phone user: the voice-mail can be activated and calls diverted without answering them; the mobile phone can be set on silent mode or even turned off, blocking the contact at the moment of the call. Thus, we hypothesize:<\/p>\n Hypothesis 2A:<\/strong> Previous call outcomes affect the likelihood of contact<\/p>\n Hypothesis 2B:<\/strong> Previous call outcomes affect the likelihood of interview<\/p>\n For fixed CATI surveys, some previous work has addressed the optimal amount of time to allow between call attempts to the same telephone number. Results are somewhat mixed, and this may be because of differences in the length of the field period and of the time lag in surveys, and in the frequency of attempts to call the same number. Stokes and Greenberg (1990) found that longer delays between first and second calls were associated with higher contact rates, while Reimer et al. (2012) found that longer time lags (six or more days) between first and second calls were less likely to yield a cooperative contact. In addition to shorter (or greater) time lag between calls, callback efficiency can rise when the timing period (day and time) of the calls varies (Kulka and Weeks 1988, Stokes and Greenberg 1990). In mobile CATI surveys, Reimer et al. (2012) found that shorter lag times result in more cooperation, although this effect was evaluated considering only the time lag between first and second calls. Thus:<\/p>\n Hypothesis 3A:<\/strong> Time pattern of callbacks affects the likelihood of contact<\/p>\n Hypothesis 3B:<\/strong> Time pattern of callbacks affects the likelihood of interview<\/p>\n The data used for this study come from a nationwide mobile CATI survey conducted in Portugal by a market research company in its CATI facilities during May 2012. The selection of the mobile phone numbers to be dialed followed the survey company’s standard procedures in studies of this size. In Portugal, there are no official lists of mobile phone numbers (or mobile phone subscribers) that can be used as a sampling frame, and mobile operators provide no information on the attribution of numbers. The sample of mobile numbers was therefore randomly generated. Mobile phones have nine-digit numbers and the first two digits \u2013 91, 93 or 96 \u2013 identify the operator. The Portuguese Telecommunications Regulation Authority provides information about the market share of each of the three mobile phone operators in Portugal and this was used to stratify the sample into three subgroups. Within each two-digit prefix, mobile phone numbers were created by a generator of 7-digit random numbers. A total of 11,472 mobile phone numbers were dialed. Calls were made between 5pm and 10pm on weekdays and between 10am and 2pm on weekends following the ESOMAR Code that recommends survey companies avoid anti-social times in mobile CATI surveys (ESOMAR 2011, 2012). The CATI system automatically managed the dialing of numbers and callbacks. Mobile phone numbers were randomly assigned to interviewers.<\/p>\n One of the following disposition codes was attributed to numbers after each call: (a) completed interview; (b) partial interview, when the interview did not reach the end; (c) refusal, when the mobile phone user answered the call but refused to cooperate; (d) busy; (e) rings but no answer, when the call was not answered after 8 ring tones; (e) voice mail (message from the operator directing the caller to voice mail either immediately or after 7 ring tones); (f) temporarily unavailable (message from the operator saying \u201cthe mobile number you dialed is not available at the moment, please try later\u201d); (g) reject the call (hang up without answering the call); (h) out of the scope (if the mobile user was less than 15 years old); (i) non-working or disabled (when an almost continuous busy signal was heard); (j) non attributed (message from the operator). In addition to the information about each call outcome, the date and time of calls are recorded automatically by the CATI system which is used in our study to describe the call history for a given number (number of calls, time elapsed between successive calls, and number of calls resulting in a specific outcome).<\/p>\n For the analysis, \u201ccontact\u201d is defined as any call in which a mobile phone user was reached and includes those coded as completed interview, partial interview, refusal, and out of the scope. A \u201cnoncontact\u201d includes cases coded as busy signal, rings but no answer, voice mail, temporarily unavailable and reject the call. An \u201cinterview\u201d corresponds to the cases coded as \u201ccompleted interview\u201d. Cases coded as non-attributed, non-working or disabled are considered \u201cineligible\u201d for the purposes of this study. If a call outcome is a completed or partial interview, that number is not called back. If a call outcome is a \u201cnon-contact\u201d, that mobile number stays in the \u201csampling frame\u201d and the scheduling algorithm may select it again to be called back. The survey company makes three call backs to cases coded as \u201ctemporarily unavailable\u201d and the mobile number is then coded as \u201crings but no answer\u201d if the \u201ctemporarily unavailable\u201d status continues after the third attempt. A \u201crefusal\u201d is only called back with the agreement of the mobile phone user and is otherwise excluded from the \u201csampling frame\u201d. Numbers with a non-functional, disabled or non-attributed call outcome are immediately excluded from the \u201csampling frame\u201d. Ineligible numbers were considered irrelevant to our analysis and therefore excluded as our goal is to explore the determinants of \u201ccontact\u201d and \u201cinterview\u201d. Among the 11,472 mobile phone numbers dialed, 8,083 were eligible (final disposition). The eligible numbers were called 19,890 times which means an average of almost 3 call attempts per number. The number of call attempts ranged from 1 to 15 which is the survey company\u2019s standard procedure for mobile CATI surveys.<\/p>\n Regression models are commonly used to estimate the likelihood of contact and likelihood of interview in surveys, more specifically, binary logistic regression (Groves and Couper 1998, Brick et al. 1996), OLS regression (Holbrook et al. 2008) or multinomial logistic regression (Steele and Durrant 2011). Multi-level models are an alternative approach when information is available about sample design features with clustering effects (e.g. O\u2019Muircheartaigh and Campanelli 1999, Durrant and Steele 2009, Wagner 2009).<\/p>\n We adopt binary logistic regression to estimate the probability of making contact \u2013 dependent variable assuming the codes 1-\u201ccontact yes\u201d and 0-\u201ccontact no\u201d \u2013 and the probability of interview \u2013 dependent variable assuming the codes 1-\u201cinterview yes\u201d and 0-\u201cinterview no\u201d. \u201cNo interview\u201d includes cases that were contacted but refused cooperation, and cases that could not be contacted at all. When modelling the likelihood of contact and the likelihood of interview, we distinguish between first calls and subsequent calls.<\/p>\n As the time and day of the call is the only information available for first calls, the model of first call includes the following explanatory categorical variables: (1) DAY (seven categories) which represents the day of week of the call, and (2) TIME (five categories) which represents the time slot of the call (10am-12noon; 12noon-2pm; 5 pm-7 pm; 7 pm-9 pm; 9 pm-10 pm). The categories of DAY and TIME are included in the models as dummy variables. Tuesday and 7 pm-9 pm were chosen as the reference categories of the DAY and TIME, respectively, as the biggest percentage of eligible mobile numbers were called then (n=2743 and n=3111). In addition to main effects, the model includes interaction effects as there is evidence from fixed CATI surveys that specific combinations of day*time affect the likelihood of contact and of interview (e.g. Brick et al. 1996, Massey et al. 1996, Stec et al. 2004).<\/p>\n We model the likelihood of contact and of interview in subsequent calls (two to fifteen attempts) by taking data related to the call history of the numbers. The models have the following explanatory variables: (1) CALL WEEKEND which is the ratio between the number of calls made on weekends and the total number of call attempts for a given mobile phone number, (2) CALL MONDAY, CALL TUESDAY, CALL WEDNESDAY, CALL THURSDAY, CALL FRIDAY which is the proportion of calls to a mobile phone number on each week day, (3) CALL 5 pm-7 pm, CALL 7 pm-9 pm, CALL 9 pm-10 pm which is the proportion of calls to a mobile phone number on each time slot (the 10 am-12 noon and 12 noon-2 pm shifts are not included because they are implicit in the CALL WEEKEND factor), (4) TIME LAG which is the mean time elapsed between two consecutive calls to a mobile phone number, (5) BUSY which is the proportion of a mobile phone number’s call outcomes that was coded as \u201dbusy\u201d, (6) VOICE MAIL which is the proportion of a mobile phone number’s call outcomes that was coded as \u201cvoice mail\u201d, (7) NO ANSWER which is the proportion of a mobile phone number’s call outcomes that was coded as \u201crings but no answer\u201d and (8) TEMP_UNAVAILABLE which is the proportion of a mobile phone number’s call outcomes that was coded as \u201ctemporarily unavailable\u201d. All factors are metric and standardized variables. Binary logistic regression coefficients were estimated using the Maximum Likelihood Method and a Stepwise approach was adopted using 0.05 for statistical significance level.<\/p>\n The results for the likelihood of contact and of interview on first calls are shown in Table 1. Only the variables with a significant effect are presented (p<0.05<\/em> for Stepwise entering).<\/p>\nData and methods<\/h1>\n
Data collection<\/strong><\/h2>\n
<\/em><\/strong>Modeling<\/strong> likelihood of contact and of interview<\/strong><\/h2>\n
Results<\/h1>\n
![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2017\/07\/Table_1-11.30.39.png\" "\\"Table_1")
<\/a><\/p>\n![\"\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/2017\/07\/Table_21.png\" "\\"Table_2\\"")
<\/a><\/p>\n![\"\hat{\beta}\"](\"https:\/\/surveyinsights.org\/wp-content\/uploads\/ql-cache\/quicklatex.com-9386035890ef07fdb34d8143b454afb8_l3.png\" "\\"Rendered")
=-0.544) or on Thursdays between 9pm and 10pm (