Jun 13, 2018 - Page 1 ... MMDC. Problem. Data. Methods. Results. Discussion. 3 / 20 ... Methods. Effect calcullation. Inferences. Results. Discussion. 12 / 20 ...
ELABORATED EVALUATION OF MODE EFFECTS IN MIXED MODE DATA Jorre Vannieuwenhuyze & Geert Loosveldt ITSEW 2010, june 13–16 1 / 20
Introduction
Introduction Data
Data
Methods Results
Methods
Discussion
Results Discussion
2 / 20
Introduction MMDC Problem Data Methods Results Discussion
Introduction
3 / 20
Mixed-Mode Data Collection (MMDC) Introduction MMDC Problem
■
different sample members choose different modes
Data Methods Results
■
mode effects:
Discussion
◆ selection effect: different mode, different respondents makes MM attractive
◆ measurement effect: different mode, different measurement e.g. social desirability, primacy and recency effects, acquiescence, . . .
4 / 20
Problem Introduction MMDC Problem Data Methods
■
differences between groups
Results Discussion
◆ = different respondents? ◆ = different measurement?
5 / 20
Problem Introduction MMDC Problem Data Methods
■
differences between groups
Results Discussion
◆ = different respondents? ◆ = different measurement? ■
solution: compare MM data with comparable single-mode data
5 / 20
Introduction Data ESS Comparability Variables Methods Results Discussion
Data
6 / 20
ESS Introduction Data ESS
■
European Social Survey (ESS), 2008, 4th wave
Comparability Variables Methods Results
■
FTF = expensive
Discussion
7 / 20
ESS Introduction Data ESS
■
European Social Survey (ESS), 2008, 4th wave
Comparability Variables Methods Results
■
FTF = expensive
■
⇒ MMDC experiment
Discussion
◆ the Netherlands ◆ costs ↓? ◆ equal data quality?
7 / 20
ESS Introduction Data ESS Comparability Variables Methods Results Discussion
8 / 20
Comparability assumption Introduction Data ESS Comparability Variables Methods Results Discussion
■
comparability? ◆ response rates ◆ socio-demographic comparison
9 / 20
Variables Introduction
■
political Interest
Data ESS Comparability
◆ 1=not at all interested 2=hardly interested 3=quite interested 4=very interested
Variables Methods Results Discussion
◆ expectations: ■
social desirability bias
■
nonrespondents less interested
◆ two versions: ■
FTF: Ip ∼ Multin(π p )
■
Tel./Web: Iwt ∼ Multin(π wt ) 10 / 20
Variables Introduction Data ESS Comparability Variables Methods Results
■
further we define M
Discussion
◆ the mode a respondent ‘chooses’ ◆ 0= CATI or CAWI, 1=CAPI ◆
M ∼ b(τ )
11 / 20
Introduction Data Methods Effect calcullation Inferences Results Discussion
Methods
12 / 20
Effect calcullation 0.4
Introduction Data
0.0
sample proportion 0.1 0.2 0.3
Methods Effect calcullation Inferences Results Discussion
1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
0.0
sample proportion 0.1 0.2 0.3
Methods Effect calcullation Inferences Results Discussion
1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences Results Discussion
1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences Results Discussion
1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences Results Discussion
P(IP = 2|M=1) 1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences Results
P(IWT = 2|M=0)
Discussion
P(IP = 2|M=1) 1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences
P(IP = 2|M=0)
Results
P(IWT = 2|M=0)
Discussion
P(IP = 2|M=1) 1
2
3 IP
4
1
2
3
4
IWT
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences
P(IP = 2|M=0)
Results
P(IWT = 2|M=0)
Discussion
P(IP = 2|M=1) 1
2
3
4
1
2
IP
3
4
IWT
selection effect = P (Ip |M = 1) −P (Ip |M = 0)
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences
P(IP = 2|M=0)
Results
P(IWT = 2|M=0)
Discussion
P(IP = 2|M=1) 1
2
3
4
1
2
IP
3
4
IWT
selection effect = P (Ip |M = 1) −P (Ip |M = 0) measurement effect = P (Iwt |M = 0) −P (Ip |M = 0)
13 / 20
Effect calcullation 0.4
Introduction Data
P(IP = 2)
sample proportion 0.1 0.2 0.3 0.0
Methods Effect calcullation Inferences
P(IP = 2|M=0)
Results
P(IWT = 2|M=0)
Discussion
P(IP = 2|M=1) 1
2
3
4
1
2
IP
3
4
IWT
selection effect = P (Ip |M = 1) −P (Ip |M = 0) measurement effect = P (Iwt |M = 0) −P (Ip |M = 0)
P (Ip |M = 0) = P (Ip )
1 P (M =0)
− P (Ip |M = 1)
P (M =1) P (M =0) 13 / 20
Inferences Introduction Data Methods Effect calcullation
selection effect = P (Ip |M = 1) − P (Ip |M = 0) measurement effect = P (Iwt |M = 0) − P (Ip |M = 0)
Inferences Results Discussion
(M =1) P (Ip |M = 0) = P (Ip ) P (M1=0) − P (Ip |M = 1) PP (M =0)
■
bayesian approach
■
p(θ|y) =
p(y|θ)p(θ) p(y)
■
simulation of 10.000 values
14 / 20
Introduction Data Methods Results Graphs Results Discussion
Results
15 / 20
Graphs of simulations measur. effect
selection effect Introduction Data Methods
µ
Results Graphs Results Discussion
−0.3
−0.2
−0.1
0.0
−0.6
−0.4
−0.2
0.0
0.2
σ2
−0.4
−0.3
−0.2
−0.1
0.0
0.1
0.2
−0.4
−0.2
0.0
0.2
0.4
16 / 20
Results 95% confidence interval median 2,5 percentile 97,5 percentile MEASUREMENT EFFECT P(not at all interested) -0,029 -0,068 0,014 P(hardly interested) 0,179 0,106 0,253 P(quite interested) -0,119 -0,198 -0,040 P(very interested) -0,032 -0,074 0,015 mean -0,153 -0,273 -0,038 variance -0,068 -0,187 0,063 SELECTION EFFECT P(not at all interested) 0,032 -0,041 0,131 P(hardly interested) 0,083 -0,033 0,218 P(quite interested) -0,070 -0,205 0,068 P(very interested) -0,054 -0,111 0,027 mean -0,206 -0,425 0,001 variance 0,023 -0,185 0,271
p∗
Introduction Data
0,087 0,000 0,002 0,081 0,005 0,150
Methods Results Graphs Results Discussion
0,219 0,086 0,158 0,086 0,026 0,425
based on 10.000 simulations [∗] The probability p refers to the number of simulations with a measurement effect/selection effect smaller than zero (P(sim0)) for negative median values.
17 / 20
Introduction Data Methods Results Discussion Limitations Future
Discussion
18 / 20
Limitations Introduction Data
■
measurement effects
(πwt |M = 0) − (πp |M = 0)
Methods Results Discussion Limitations
■
comparability assumption?
Future
◆ bias? ◆ mode-preference, mode-acceptance?
19 / 20
Limitations Introduction Data
■
measurement effects
(πwt |M = 0) − (πp |M = 0)
Methods Results Discussion Limitations
■
comparability assumption?
Future
◆ bias? ◆ mode-preference, mode-acceptance?
■
interpretation? Web and telephone confounded → use maximum 2 modes 19 / 20
Future Introduction Data Methods Results Discussion Limitations Future
■
MMDC: include small comparative group
■
simulate respondents’ values ≈ multiple imputation
■
power?
20 / 20
