Data collection mode differences between national face-to-face and ...

Women's Studies International Forum 60 (2017) 11–16

Contents lists available at ScienceDirect

Women's Studies International Forum journal homepage: www.elsevier.com/locate/wsif

Data collection mode differences between national face-to-face and web surveys on gender inequality and discrimination questions Mingnan Liu SurveyMonkey, 101 Lytton Avenue, Palo Alto, CA 94301, USA

a r t i c l e

i n f o

Article history: Received 18 June 2015 Received in revised form 11 November 2016 Accepted 13 November 2016 Available online xxxx Keywords: Face-to-face survey Web survey Data collection mode effect Gender inequality Item nonresponse

a b s t r a c t The topic of gender inequality and discrimination has received constant attention from social scientists. Given that the majority of the research is based on survey data, a solid understanding of the impact of data collection mode on survey responses to this type of questions is important. This study utilizes the 2012 American National Election Studies to examine the response difference between face-to-face and Web surveys. The analyses reveal that mode effects exist both for substantial responses to the survey questions, but also item nonresponses. As expected, face-to-face surveys elicit more socially desirable responses than Web surveys. Also, the item nonresponse rate is higher in face-to-face surveys than Web surveys. In addition, this study demonstrates that the mode effect is not uniform across all respondents. Rather, the mode effect is larger for male respondents than female respondents. This is evidenced by the larger mode effect among male than female respondents in terms of both substantial responses and item nonresponses. Direction for future research is discussed. © 2016 Elsevier Ltd. All rights reserved.

1. Introduction

2. Literature review

Gender inequality and discrimination have always been an important line of research in many social science fields (Fenstermaker, West, & Zimmerman, 2002). Discrimination and inequality are frequently studied in both the public domain, such as employee segregation (Kmec, 2005), earning inequality (Huffman & Velasco, 1997), and access to authority (Gorman & Kmec, 2009; Stainback & Tomaskovic-Devey, 2009), as well as the private domain, such as inequality at home and personal life (Calasanti & Bailey, 1991; Kane & Sanchez, 1994; Shaw & others, 1985). Survey data is one of the data sources for studies on gender inequality and discrimination. Therefore, a sound understanding of the measurement of survey questions on this topic is of great importance to not only survey researchers, but also social scientists in general. This study examines the data collection mode effects on attitudinal questions related to gender inequality issues. Specifically, this study focuses on comparing substance survey responses and item nonresponses between two national probability surveys, one done face-to-face and the other on the Web. These two modes differ on several dimensions (Couper, 2011), and these differences can influence the way respondents answer sensitive questions, like the ones examined in this study. As many flagship national surveys are moving toward Web data collection, a solid understanding of the mode effect on survey responses and data quality will provide important insights into survey design and social science research in general.

2.1. Mode effect between face-to-face and web surveys

E-mail address: [email protected].

http://dx.doi.org/10.1016/j.wsif.2016.11.007 0277-5395/© 2016 Elsevier Ltd. All rights reserved.

Couper (2011) describes a framework for comparing data collection mode, which can be used to explain potential differences between faceto-face and Web surveys in terms of responses to attitudinal questions. The first dimension differentiates face-to-face and Web surveys by the level of interviewer involvement: face-to-face surveys have the highest level of interviewer involvement while Web surveys have the least interviewer involvement. A related dimension of the framework for mode difference is the level of contact from the survey organization with the respondents. Specifically, in face-to-face surveys, interviewers make direct contact with the selected respondents, seek their participation, answer their questions and address their concerns, administer the survey, and record the answers. In contrast, Web surveys usually lack direct interpersonal interaction between the survey organization and the respondents. Instead, survey participation is often solicited by email or mail. These differences between face-to-face and Web surveys can result in two major consequences, namely differential response rate and data quality. Studies on responses rates between these two modes show consistent findings, that is face-to-face surveys tend to achieve higher response rates than Web surveys (Christensen, Ekholm, Glümer, & Juel, 2014; Heerwegh & Loosveldt, 2008). Studies on data quality between these two modes reveal mixed results. Some studies report that faceto-face surveys produce superior data quality, as measured by lower item nonresponse rate and less non-differentiation (Goldenbeld & de Craen, 2013; Heerwegh, 2009; Heerwegh & Loosveldt, 2008). They

12

M. Liu / Women's Studies International Forum 60 (2017) 11–16

attribute better data quality to interviewer involvement as they can provide guidance to the respondents and motivate them to finish the survey more carefully and thoroughly. However, there are also studies that report Web surveys provide better or no worse data quality than face-to-face surveys. For example, Liu and Wang found that face-toface respondents provide more rounded answers, lower accuracy to political knowledge questions, and no difference on item nonresponse rates (Liu & Wang, 2014, 2015). The speculated reason is that face-toface respondents are under higher time pressure, which poses a challenge to the respondent's cognitive capacity. It may have resulted in superficial comprehension and retrieval of information, and all of these are reflected in the survey responses, such as rounding. A third distinction between face-to-face and Web surveys is the degree of privacy. The face-to-face survey has a lower level of privacy compared to the self-administered Web survey because of interviewer involvement, which can cause measurement error, often known as the interviewer effect (e.g., Liu & Stainback, 2013). The higher level of privacy of the Web survey is typically seen as valuable when collecting sensitive information, as a Web survey is able to elicit more selfdisclosure than a face-to-face survey (Tourangeau, Conrad, & Couper, 2013). The locus of control and channel of communication are other dimensions that differentiate face-to-face and Web surveys. For a face-to-face survey, interviewers administer the survey and hence they control the flow and speed of the survey. Web survey respondents, in comparison, have more autonomy over the survey taking process, as they can decide when and where to take the survey, at what pace, and through which device. This distinction can result in a mode difference between faceto-face and Web surveys. For example, Goldenbeld and de Craen (2013) found that face-to-face surveys elicit more extreme responses on ordinal rating scales than do Web surveys. The reason, as the authors suggested, is that the lack of control from the respondent's perspective in face-to-face surveys, combined with higher time pressure, have posed a challenge to respondent's cognitive capacity and resulted in superficial comprehension and information retrieval process during the interview. As a result, respondents are likely to simplify the cognitive process of interpreting ordinal scales by treating them in a dichotomous manner. In terms of the communication channel of these two modes, face-to-face surveys are primarily oral, or a combination of oral and visual if visual materials, such as show cards, are used during the interview. Web surveys use only visual presentation without an oral component. The different channels of communication are associated with different levels of cognitive burdens for responding to survey questions, which in turn can influence the responses provided in different modes. 2.2. Social desirability The topic of gender inequality and discrimination is a sensitive one, and social desirability theory is the most relevant in predicting and explaining the mode difference between responses in face-to-face and Web surveys. Social desirability refers to the tendency of over-reporting socially acceptable attitudes and behaviors while under-reporting socially less acceptable ones (Callegaro, 2008). Apparently, respondents in face-to-face surveys are more susceptible to social desirability bias than Web surveys. Face-to-face respondents have a higher tendency to edit their undesirable answers and provide desirable ones in order to portray themselves in a more positive manner so as to avoid any potential tensions between them and the interviewers. In a self-administered Web survey, the absence of an interviewer increases the perceived level of privacy, which in turn can result in more self-disclosure of undesirable answers (Goldenbeld & de Craen, 2013; Heerwegh, 2009). For example, a higher proportion of abortion was reported in a self-administered mode than an interviewer-administered mode in the National Survey of Family Growth (Fu, Darroch, Henshaw, & Kolb, 1998). In another study, researchers found that the level of

reporting of the sensitive information and the response accuracy were higher in self-administered mode (web and interactive voice recognition) than telephone interview (Kreuter, Presser, & Tourangeau, 2008). Also, research has shown that not all questions are subject to social desirability in the same fashion (Christensen et al., 2014; Duffy, Smith, Terhanian, & Bremer, 2005). Rather, questions that seek sensitive information or questions that can potentially result in socially unacceptable responses are more susceptible to social desirability bias. Logically, these questions are more likely to show differential mode effect between face-to-face and Web surveys. In contrast, when responding to non-sensitive or intrusive questions, respondents tend to give candid and consistent responses, regardless of the survey mode. In addition, not everyone is susceptible to the mode effect that is rooted in social desirability bias. If one's true answer, whether attitude or behavior, is consistent with socially acceptable norms, then that person will provide an answer regardless of the survey mode he/she is interviewed in. By contrast, people with undesirable responses are more likely to suppress their true attitudes and give untruthful albeit more desirable answers in face-to-face than Web survey. For example, a study examining mode effect on attitudes toward homosexual rights found a significant mode effect only exists among heterosexual respondents where homosexual respondents provide similar responses to either face-to-face surveys or Web surveys (Liu & Wang, 2016). The questions examined in this study are sensitive and are likely to be susceptible to social desirability bias. As shown in the next section, the questions focus on the general public opinion toward women's rights, particularly gender inequality and discrimination. Apparently, being supportive of gender equality and eliminating gender-based discrimination is more congruent with the social norms in current society and I expect face-to-face respondents to provide more such answers than Web respondents. Furthermore, I expect that the mode effect will be nonuniform: male respondents are more subject to the mode effect than female respondents. Male respondents are probably more likely to edit their responses based on the interview mode and provide more socially acceptable answers in either the less private or anonymous face-to-face survey than the self-administered Web survey. Female respondents are likely to be more consistent with their answers to questions on this topic, since this is centered around an issue that is more relevant to themselves. An earlier study examined mode effect on homosexual issues and found a significant effect among heterosexual response (Liu & Wang, 2016). For homosexual respondents, their answers were similar and not significant between face-to-face and web surveys. The authors argued that when the issue under study was more relevant to a subset of the population, that subset was likely to hold a well-formed opinion and less subject to the impact of mode. Consider this, I expect a larger mode effect on gender inequality and discrimination among male than female respondents. Item nonresponse is another reflection of social desirability bias. When one possesses undesirable attitudes, not providing an answer to the survey question is another choice of hiding one's unacceptable opinion. Since questions in face-to-face surveys are more vulnerable to social desirability bias, I expect the item nonresponse rate in face-to-face surveys to be higher than Web surveys on questions regarding gender inequality and discrimination. Likewise, I expect that the mode effect on item nonresponse to be more salient among male than female respondents. 3. Data and measures This study utilizes the 2012 American National Election Studies (ANES), a national survey on electoral participation, voting behavior, public opinion, as well as media exposure, cognitive style, and values and predispositions. The ANES contained both a pre-election study and a post-election study, and the same respondents were interviewed twice for these two studies. In 2012, for the first time, ANES conducted two surveys, one face-to-face and the other on the Web, using two


independent national probability samples and one identical questionnaire. The face-to-face interviews were conducted using an addressbased, stratified, multi-stage cluster sample in 125 census tracts. The sample contained a main sample and two over samples for African Americans and Hispanic Americans respectively. In the analysis, I included both the main sample and the two oversamples for the faceto-face survey. The Web survey was conducted using the GfK KnowledgePanel probability Web panel. The Web panelists were recruited through address-based sampling or random-digit dialing. For each selected households, household members were enumerated and basic demographic information was collected. For households without computers and/or Internet services, equipment and/or service were provided in order to remove the coverage bias of the probability panel. As such, these two samples, one face-to-face and the other for the Web, have equivalent coverage of the same target population. This study examines eight questions on gender inequality and discrimination. All questions were administered in the post-election study. The question wordings and response options are presented in the Appendix. As we can see, almost all questions are sensitive to some extent and are likely to suffer from social desirability bias. Six questions, including Q3 (discrimination), Q4 (media coverage), Q5 (special favor), Q6 (discrimination at work), Q7 (causing more problem), and Q8 (equal opportunity), are coded in a way that a lower number indicates pro-women attitudes, against gender-based discrimination and inequality, admitting the existence and serious of gender discrimination, and advocating for more attention from the media on gender discrimination issues. For Q1 (working mother) and Q2 (women work), neither ends of the scale reflects a more socially acceptable answer. The middle category, however, seems to reflect a more gender equality attitude. Therefore, for these two questions, responses close to the middle category are likely to be more socially desirable and perceived to be acceptable based on social norms. In the analysis, I first conducted an independent t-test of the eight questions to test whether substantial responses differ by the data collection mode for the whole sample combined, and for male and female respondents separately. Similarly, I tested the mode effect on item nonresponse (don't know and refusal combined) using an independent t-test for the whole sample and by respondent's gender. All analyses are weighted using the weighting variable provided by the survey organization. Based on the documentation published by the survey organization, the post-stratification weights account for the probability of household selection, the probability of respondent selection within the household, nonresponse, and random sampling error. The weights are post-stratified to produce estimates that match known population proportions for selected characteristics, including sex, age, race and ethnicity, marital status, education attainment, household income, home ownership, household internet access, country of birth, census region, and metropolitan status, for face-to-face and Web samples separately. The weighted analyses in each sample therefore should produce unbiased estimates for the same population. Consequently, the differences in the estimates between modes are most likely due to the mode difference, rather than coverage or differential nonresponse bias. Similar approach has been used to analyze the ANES data for survey measurement in previous studies (Liu, Conrad, & Lee, 2016; Liu & Wang, 2014, 2015). 4. Results I start by comparing the key demographic variables between the two modes. As Table 1 shows, the unweighted demographic distributions differ substantially between face-to-face and Web surveys. Once the weight is applied, the mode differences disappear and the respondent demographics become comparable. This indicates that the weighted results are unlikely to be a reflection of differential nonresponse bias between face-to-face and Web surveys. The responses by data collection mode are presented in Table 2. Among the eight questions I examined, four of them show significant

13

mode effect. Compared to Web respondents, face-to-face respondents are more likely to say that discrimination against women in the U.S. is a more serious problem (p b 0.0001), news media should pay more attention to discrimination against women (p b 0.0001), and women are not seeking special favors when demanding equality (p b 0.05). For the first question, more face-to-face respondents admit that it is harder for working mothers to establish a warm and secure relationship with their children (p b 0.0001). For the other four questions, there are no reliable differences between face-to-face and Web surveys. I then conducted the same analysis separately for male and female respondents. As I stated above, the effects of gender discrimination and inequality is more pertinent to women than men. I hypothesize that female respondents would show less mode effect than male respondents as female respondents are more likely to voice their opinions and defend their rights, regardless of the specific survey situation, such as the mode. This is indeed the case. As Table 2 shows, for male respondents, the same four questions, including working mother (p b 0.0001), women discrimination (p b 0.0001), media coverage of discrimination (p b 0.0001), and special favor (p b 0.001), show significant mode effect and they are in the same direction as the results from the whole sample. For female respondents, the mode effect only exists in the working mother question (p b 0.001). For the rest of the seven questions, no reliable mode effect is detected from female respondents. Regression models were also conducted with gender, age, race, education, and income as the control variables (results in the supplement document.) The main effect of mode and the interaction effect between mode and gender in the regression models were consistent with the bivariate results in Table 2. In addition to the mean score comparison, Cronbach's alpha was also calculated for the eight items by mode and by respondent's gender. For the whole sample, the alpha for face-to-face survey was 0.34, for web was 0.50, and the difference was significant (p b 0.001). For male respondents, the alpha for face-to-face and web was 0.34 and 0.51 respectively (p b 0.001). For female respondents, the alpha for face-to-face and web was 0.32 and 0.45 respectively (p b 0.001). Consistently, the Cronbach's alpha for web was higher than fact-to-face surveys for the whole sample and by respondent's gender. Also, the difference between face-to-face and web was smaller for female than male. Table 3 presents a comparison of item nonresponse to the eight questions on gender inequality and discrimination by data collection mode. Overall, the item nonresponse rates are relatively low but they are consistently higher in face-to-face surveys, in which interviewers are involved, than Web surveys, which are self-administered. The difference in the whole sample ranges from 1% to 4.7%. I then broke down the analysis of item nonresponses by respondent's gender and found that the responses of both male and female respondents conform to the general trend. Specifically, as Table 3 shows, for seven out of eight questions, male respondents in face-toface surveys provide significantly more non-substantial (“don't know” or “refusal”) answers than male respondents in Web surveys. Similarly, for female respondents, the item nonresponse rates are higher in faceto-face than Web surveys for six items. One pattern is that the mode difference on item nonresponse is smaller for female than male respondents. This once again suggests that the mode effect tends to be smaller for female than male respondents. As a sensitivity check, six questions that were arguably less likely to invoke social desirability bias, were analyzed and compared between the two modes (see results in online supplement). All but one showed nonsignificant mode effect on the survey responses and item nonresponse. This suggested that the results reported below is due to the measurement bias caused by the data collection mode, rather than nonresponse bias between the two samples. 5. Discussion The results from this mode effect study on responses to gender inequality and discrimination questions show some evidence of social

14


Table 1 Demographic variables for face-to-face (FTF) and Web surveys, 2012 American National Election Studies. Unweighted

Weighted

FTF (%)

Web (%)

χ2

p-value

FTF (%)

Web (%)

χ2

p-value

Female Male

56.9 43.1

49.2 50.8

30.84

b0.0001

52.0 48.0

52.1 47.9

0.00

0.96

Age b30 30–39 40–49 50–59 60–69 70+

22.4 19.8 16.2 19.6 12.7 9.3

12.7 12.1 16.2 23.9 22.1 13.1

218.68

b0.0001

21.2 15.6 17.3 18.9 14.3 12.8

21.0 15.0 16.7 20.1 16.3 10.9

8.96

0.44

Race/ethnicity White (non-Hispanic) Black (non-Hispanic) Hispanic Other (non-Hispanic)

44.9 25.0 23.1 7.0

67.1 13.2 13.9 5.9

288.34

b0.0001

70.9 11.9 10.9 6.3

70.9 11.9 11.2 5.9

0.52

0.95

Education High school or less Some post high school College or more

42.0 34.1 23.9

31.6 33.4 35.0

93.05

b0.0001

40.6 30.1 29.3

40.1 30.4 29.5

0.13

0.96

Marital Married Widowed/Divorced/Separated Never married

40.2 27.8 32.0

54.9 23.4 21.7

124.93

b0.0001

53.2 21.2 25.5

53.3 21.1 25.7

0.04

0.99

Income $49,999 or less $50,000–$99,999 $100,000–$149,999 $150,000 or more

64.9 24.4 6.3 4.4

49.2 30.0 12.4 8.4

147.68

b0.0001

50.1 29.9 11.7 8.3

49.5 31.9 11.7 6.9

5.52

0.46

Table 2 Means for gender inequality and discrimination questions by mode of data collection, 2012 American National Election Studies (weighted results). Face-to-face Web Different Whole sample Working mother Women work Discrimination Media coverage Special favor Discrimination at work Causing more problem Equal opportunity Male Working mother Women work Discrimination Media coverage Special favor Discrimination at work Causing more problem Equal opportunity Female Working mother Women work Discrimination Media coverage Special favor Discrimination at work Causing more problem Equal opportunity

Table 3 Percentages (%) of item nonresponse by mode of data collection, 2012 American National Election Studies (weighted results).

Effect size (Cohen's d)

5.49 2.94 3.22 3.40 2.19 3.63

5.18 2.97 3.35 3.64 2.26 3.63

0.31 −0.03 −0.13 −0.24 −0.08 0.00

⁎⁎⁎ 0.17 0.01 ⁎⁎⁎ 0.10 ⁎⁎⁎ 0.11 ⁎ 0.06 0.00

2.46

2.49

−0.02

0.02

2.86

2.89

−0.03

0.02

5.70 2.93 3.30 3.48 2.22 3.71

5.28 2.89 3.54 3.88 2.41 3.76

0.42 0.04 −0.24 −0.40 −0.19 −0.06

⁎⁎⁎ 0.25 0.02 ⁎⁎⁎ 0.19 ⁎⁎⁎ 0.20 ⁎⁎ 0.14 0.05

2.40

2.52

−0.11

0.08

3.07

3.18

−0.11

0.06

5.29 2.95 3.14 3.33 2.15 3.56

5.08 3.04 3.18 3.41 2.13 3.51

0.21 −0.09 −0.04 −0.08 0.03 0.05

2.52

2.46

0.06

0.04

2.67

2.62

0.05

0.03

⁎⁎

0.11 0.04 0.03 0.04 0.02 0.04

Face-to-face Web Difference Whole sample Working mother Women work Discrimination Media coverage Special favor Discrimination at work Causing more problem Equal opportunity Male Working mother Women work Discrimination Media coverage Special favor Discrimination at work Causing more problem Equal opportunity Female Working mother Women work Discrimination Media coverage Special favor Discrimination at work Causing more problem Equal opportunity

Effect size (Cohen's d)

1.3 2.6 1.6 2.9 5.0 5.3

0.3 0.5 0.5 0.4 0.5 0.6

1.0 2.1 1.1 2.5 4.6 4.7

⁎⁎ ⁎⁎⁎ ⁎⁎ ⁎⁎⁎ ⁎⁎⁎ ⁎⁎⁎

4.7

0.5

4.2

⁎⁎⁎ 0.17

1.8

0.4

1.4

⁎⁎

2.2 3.2 1.7 3.0 6.3 6.1

0.2 0.2 0.6 0.2 0.7 0.6

2.0 3.0 1.0 2.8 5.6 5.5

⁎⁎ 0.13 ⁎⁎⁎ 0.15 0.07 ⁎⁎ 0.12 ⁎⁎⁎ 0.20 ⁎⁎⁎ 0.20

5.8

0.7

5.1

⁎⁎⁎ 0.18

1.9

0.3

1.6

⁎⁎

0.5 2.0 1.5 2.8 3.9 4.6

0.5 0.8 0.3 0.5 0.3 0.6

0.1 1.2 1.1 2.3 3.6 4.0

0.01 0.06 ⁎ 0.09 ⁎⁎ 0.13 ⁎⁎⁎ 0.19 ⁎⁎⁎ 0.18

3.7

0.4

3.3

⁎⁎⁎ 0.17

1.7

0.5

1.2

⁎

0.08 0.11 0.08 0.12 0.19 0.19

0.09

0.11

0.08


desirability bias. First, when all respondents are taken together, face-toface respondents are more likely to respond in a way that is supportive of gender equality and less likely to give gender discrimination answers. In particular, compared to Web respondents, face-to-face respondents are more likely to report that discrimination against women in the U.S. is still a serious issue, news media should pay more attention to discrimination against women, and women are not seeking special favor when demanding equality. Second, when analyzed separately by respondent's gender, male respondents reveal more salient mode effect than female respondents: four items show significant mode effect among male respondents while only one significant item among female respondents. This finding lends support to our hypothesis that the mode effect on gender inequality and discrimination questions is not uniform. Rather, it varies based on the respondent's characteristics and how pertinent the questions are to the respondents. Female respondents are more easily related to the question asked in this study. Thus, they are more likely to express their opinions and defend their rights regardless of the survey mode. Male respondents, by contrast, are apparently more susceptible to social desirability bias when responding to these questions. Under social normal pressure, they have a higher tendency to edit their answers before providing them to a human interviewer in face-to-face surveys. In Web surveys, they feel less pressured to give a socially desirable response because of the absence of an interviewer and the enhanced level of anonymity. In addition, the reliability, as measured by the Cronbach's alpha, was higher for web than face-to-face surveys. However, the difference between the two modes was smaller for female than male respondents. Lastly, as expected, item nonresponse is higher in face-to-face surveys than in Web surveys for all eight questions when all respondents are combined. Fourth, the patterns of item nonresponse between faceto-face and Web surveys for both male and female respondents are in the same direction as the whole sample. That is, face-to-face surveys elicit fewer substantial responses than Web surveys. However, the difference is smaller for female than male respondents, suggesting that the mode effect is not as striking for female as it is for male respondents. Apparently, in addition to editing one's answer and providing a socially acceptable response, one can also choose not to give an answer of substance when he/she feels that his/her true attitudes are against the general social norm. This finding is in line with the previous study where item nonresponse rates are higher in face-to-face surveys than Web surveys when asking about attitudes toward homosexual rights and abortion (Liu & Wang, 2016). Taken together, this study shows that not only mode effect exists for sensitive questions, particularly for topics about gender inequality and women's rights, the difference is larger for male than female respondents. For the first question that asks whether working mothers can establish a warm and secure relationship like stay-at-home mothers, I treat the middle category as an indication of gender equality. Consequently, Web respondents are more likely to express attitudes that are against gender inequality and discrimination. This is true for the whole sample, as well as for male and female respondents separately. One possible explanation for this finding is that, in general, it is more difficult for parents to bond with their children, regardless of the gender of the parent. Face-to-face respondents tend to admit this because to deny such an apparent fact will make them look untruthful and tarnish their credibility. However, this is after all speculation and there is no data to demonstrate it. The mixed findings from this study call for research on mode effect on measurement errors of sensitive questions, particularly between face-to-face and Web surveys. The questions under study are all attitudinal questions. Unlike factual questions, attitudinal questions are designed to measure an unobservable underlying construct. Therefore, it's not possible to link the survey estimate to a true value or record, as we sometimes can do for factual questions, to benchmark how accurate the survey estimates are. What we do know is that people often have a tendency to reveal

15

the positive side of themselves while hiding the negative side. Providing socially acceptable answers to sensitive questions in survey interviews is one of the ways to avoid unnecessary judgments and tensions resulting from socially unacceptable answers. In that sense, Web surveys provide respondents a more comfortable survey environment for respondents to more freely and honestly express their real attitudes and opinions toward sensitive issues. At the same time, face-to-face surveys have advantages over Web surveys, such as a higher response rate. Future research should explore ways of combining these two modes to substitute the cons of one mode with the pros of the other. For example, testing ways of embedding self-administer modules in a face-to-face survey for sensitive questions can potentially enhance the privacy of survey response while maintaining high response rates. Also worth exploring is other ways to design questions that seek sensitive information, such as an item count technique. The 2012 ANES face-to-face survey has a computer-administered self-interview (CASI) module for asking several racial-related attitudinal questions. Future study should explore a similar approach of embedding a self-administered module in an interviewer-administered survey. A mixed-mode design for asking sensitive questions in surveys is a promising research plan. Future research should also examine whether all types of questions are equally subject to the mode effect or whether certain questions are more susceptible to the mode effect than other questions. This includes both the question content and the question type. For example, whether factual questions are less like to be affected by mode than attitudinal questions? Whether open-ended questions are more likely to show a mode difference than closed-ended questions? Answering those questions will bring huge value to the field of survey research and benefit researchers in other fields as well. Last but not least, this study used two independent national probability sample to examine the mode effect. Future study should replicate this study through a fully randomized mode study to identify the causal effect. Appendix A. Appendix Question wordings and response options. 1. Do you think it is easier, harder, or neither easier nor harder for mothers who work outside the home to establish a warm and secure relationship with their children than it is for mothers who stay at home? 1. A great deal easier 2. Somewhat easier 3. Slightly easier 4. Neither easier nor harder 5. Slightly harder 6. Somewhat harder 7. A great deal harder 2. Do you think it is better, worse, or makes no difference for the family as a whole if the man works outside the home and the woman takes care of the home and family? 1. Much better 2. Somewhat better 3. Slightly better 4. Makes no difference 5. Slightly worse 6. Somewhat worse 7. Much worse 3. How serious a problem is discrimination against women in the United States? 1. An extremely serious problem 2. A very serious problem 3. A moderately serious problem 4. A minor problem 5. Not a problem at all 4. Should the news media pay more attention to discrimination against women, less attention, or the same amount of attention they have been paying lately?

16


1. A great deal more 2. Somewhat more 3. A little more 4. Same amount of attention 5. A little less 6. Somewhat less 7. A great deal less 5. When women demand equality these days, how often are they actually seeking special favors? 1. Never 2. Some of the time 3. About half the time 4. Most of the time 5. Always 6. When employers make decisions about hiring and promotion, how often do they discriminate against women? 1. Always 2. Most of the time 3. About half the time 4. Some of the time 5. Never 7. When women complain about discrimination, how often do they cause more problems than they solve? 1. Never 2. Some of the time 3. About half the time 4. Most of the time 5. Always 8. In the U.S. today, do men have more opportunities for achievement than women have, do women have more opportunities than men, or do they have equal opportunities? 1. Men have many more 2. Men have moderately more 3. Men have slightly more 4. Equal opportunities 5. Women have slightly more 6. Women have moderately more 7. Women have many more Appendix B. Supplementary data Supplementary data to this article can be found online at http://dx. doi.org/10.1016/j.wsif.2016.11.007. References Calasanti, T. M., & Bailey, C. A. (1991). Gender inequality and the division of household labor in the United States and Sweden: A socialist-feminist approach. Social Problems, 38(1), 34–53.

Callegaro, M. (2008). Social desirability. Encyclopedia of Survey Research Methods, 825–826. Christensen, A. I., Ekholm, O., Glümer, C., & Juel, K. (2014). Effect of survey mode on response patterns: Comparison of face-to-face and self-administered modes in health surveys. The European Journal of Public Health, 24(2), 327–332. http://dx.doi.org/10. 1093/eurpub/ckt067. Couper, M. P. (2011). The future of modes of data collection. Public Opinion Quarterly, 75(5), 889–908. Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face-to-face surveys. International Journal of Market Research, 47(6), 615–639. Fenstermaker, S., West, C., & Zimmerman, D. (2002). Gender inequality: New conceptual terrain. Doing gender, doing difference (pp. 25–39). Fu, H., Darroch, J. E., Henshaw, S. K., & Kolb, E. (1998). Measuring the extent of abortion underreporting in the 1995 National Survey of family growth. Family Planning Perspectives, 128–138. Goldenbeld, C., & de Craen, S. (2013). The comparison of road safety survey answers between web-panel and face-to-face; Dutch results of SARTRE-4 survey. Journal of Safety Research, 46, 13–20. http://dx.doi.org/10.1016/j.jsr.2013.03.004. Gorman, E. H., & Kmec, J. A. (2009). Hierarchical rank and Women's organizational mobility: Glass ceilings in corporate law firms. American Journal of Sociology, 114(5), 1428–1474. http://dx.doi.org/10.1086/595950. Heerwegh, D. (2009). Mode differences between face-to-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21(1), 111–121. http://dx.doi.org/10.1093/ijpor/ edn054. Heerwegh, D., & Loosveldt, G. (2008). Face-to-face versus web surveying in a high-internet-coverage population differences in response quality. Public Opinion Quarterly, 72(5), 836–846. http://dx.doi.org/10.1093/poq/nfn045. Huffman, M. L., & Velasco, S. C. (1997). When more is less sex composition, organizations, and earnings in US firms. Work and Occupations, 24(2), 214–244. Kane, E. W., & Sanchez, L. (1994). Family status and criticism of gender inequality at home and at work. Social Forces, 72(4), 1079–1102. http://dx.doi.org/10.2307/2580293. Kmec, J. A. (2005). Setting occupational sex segregation in motion demand-side explanations of sex traditional employment. Work and Occupations, 32(3), 322–354. Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72(5), 847–865. http://dx.doi.org/10.1093/poq/nfn063. Liu, M., & Stainback, K. (2013). Interviewer gender effects on survey responses to marriage-related questions. Public Opinion Quarterly, 77(2), 606–618. Liu, M., & Wang, Y. (2014). Data collection mode effects on political knowledge. Survey methods: Insights from the field (SMIF) (Retrieved from http://surveyinsights.org/? p=5317). Liu, M., & Wang, Y. (2015). Data collection mode effect on feeling thermometer questions: A comparison of face-to-face and web surveys. Computers in Human Behavior, 48(7), 212–218. http://dx.doi.org/10.1016/j.chb.2015.01.057. Liu, M., & Wang, Y. (2016). Comparison of face-to-face and web surveys on the topic of homosexual rights. Journal of Homosexuality, 63(6), 838–854. http://dx.doi.org/10. 1080/00918369.2015.1112587. Liu, M., Conrad, F. G., & Lee, S. (2016). Comparing acquiescent and extreme response styles in face-to-face and web surveys. Quality & quantity. http://dx.doi.org/10. 1007/s11135-016-0320-7. Shaw, S. M., et al. (1985). Gender and leisure: Inequality in the distribution of leisure time. Journal of Leisure Research, 17(4), 266–282. Stainback, K., & Tomaskovic-Devey, D. (2009). Intersections of power and privilege longterm trends in managerial representation. American Sociological Review, 74(5), 800–820. Tourangeau, R., Conrad, F., & Couper, M. (2013). The science of web surveys. Oxford University Press.