Basic Principles of Survey Design

Basic Principles of Survey Design Thomas Lindsay Andrew Sell

Introductions • Name, Institution, why are you here?

Survey Review – 5 minutes • Political Participation and Attitudes Survey – What is the research question being investigated? – What is the audience? – Is questionnaire appropriate length? – Are any questions especially good? – Are any questions especially problematic?

RESEARCH PURPOSE

Step 1: Define Research Purpose • DO NOT start writing questions! • What are you trying to learn, and why? • If it can’t be reduced to a single sentence, it’s probably not focused enough

Source: http://ddeubel.edublogs.org/2011/01/22/cart-before-the-horse/

Step 1: Define Research Purpose • Considering Purpose Statement: – What specific goals or objectives are there? – What questions need to be answered in order to reach actionable outcomes?

Step 1: Define Research Purpose • Is there a well-defined population? – If so, what is the universe of this population? – If not, can a segment be identified?

• What distinguishing characteristics do you expect in your population? – Age, gender, attitudes, behaviors – Are any of these relevant for research purpose?

Step 2: Determine Data Sources • Is it necessary to collect new data? • What data sources already exist?

Step 2: Determine Data Sources • Do not assume a survey is the best way to answer the research question • Consider strengths and weaknesses of all potential data sources

Small Group Exercise – 5 min. • Consider scenario – What is the overarching research question? – Develop objectives and desired outcomes – What population will this effect? – What data sources might be applicable?

Small Group Exercise – 5 min. Imagine you are a librarian at a large university library. Your library serves faculty, students and staff throughout the university. In any given year approximately 500 faculty, 10,000 students, and 1,000 staff work at or attend the university. Recently your supervisor asked you to create a plan to introduce social media into the mix of services your library provides. She mentioned that she expects you to survey library users to gather their opinions about social media and incorporate these insights into your plan.

Discussion – Purpose Definition • • • • •

What is the overarching research question? What are objectives of project? What are desired outcomes? What population will this affect? What data sources might be applicable?

Data Sources, continued • Secondary Research – Previous research – Census or other public data

Data Sources, continued • Non-self-report Primary Research – Observation – Usage Data • Website Traffic • Physical Traffic

Data Sources – Self Report • Self-Report Primary Research – Interviews – Focus Groups – Surveys

Data Sources – Self Report • May be qualitative or quantitative, or both • Is self-report data compatible with objectives? • Is it actionable?

Population - Surveys • Participant Pool: – Census – attempt to reach entire population • Difficult to reach entire population • Non-respondents are likely different from respondents

– Sample – Contact small portion of population • Difficult to define sample • Matching sampling approach to research needs can be challenging

Probability Sampling • Generally requires that you can define your population very well • People are excluded randomly, not categorically • Examples: – Simple Random Sample • Must be truly randomly drawn from entire population – Stratified Sample • Requires knowledge of key segmenting characteristics of population

Nonprobability Sampling • Does not require knowledge of population • Sampling Error is Unknowable • Examples: – Convenience Sample • Subgroups that are easier to reach than others

– Quota Sample • Participants are selected to meet quotas based on researcher interest rather than known population characteristics

Survey Modes • Paper – mail, lab, face-to-face • Online – e-mail, lab, web • Telephone

BREAK

QUESTIONNAIRE DESIGN

Paper and Online Surveys Paper • Universal Access • Allows RICH responses • Limited control over respondent • Space is extremely limited • Branching is awkward, space-consuming, and visible

Online • Digital Divide • No chance to provide unexpected data • Allows greater control over respondent • Much more space available • Branching can be complex and time-saving

Paper Survey

Online Survey

Question Types • Open-Ended – Blank space or box for free response in participant’s own words

• Closed-Ended / Scalar – List of predefined answer choices from which participants must choose to answer

Question Types Open-Ended • Response is minimally influenced by researcher • Provides rich information • Especially good when researcher knows little about question ahead of time • For numerical data, allows great precision

Closed-Ended • Structures and shapes answer choices • Reduces error from interpretation • Easy for participant to answer • Lower incidence of nonresponse • Easier to analyze

Question Types Open-Ended • Fill-in-the-blank – Free text – Numeric

• (Other, please specify) • Essay

Closed-Ended • Choose Many • Choose One • Scale

• Rank Order

Question Types

Data Types • • • •

Ratio Interval Ordinal Nominal

Data Types - Ratio • • • •

Data points are ordered Interval between items is constant Natural zero-point exists Examples – Height – Age – Temperature - Kelvin

Data Types - Interval • • • •

Data points are ordered Interval between items is constant No natural zero-point Examples – Date – Temperature – Celsius, Fahrenheit

Data Types - Ordinal • Data points are ordered • Interval between items is unknown or not constant • Examples – Rankings (e.g., school rankings, Olympic medals) – Academic Degree Attained

Data Types – Nominal • Data points have no order (are equivalent) • Categories are mutually exclusive • Examples – Gender – Species – Musical genre - ? – Color - ???

Small Group Exercise – 5 min. • Develop and write questions for survey • Consider desired data type for answers • Consider desired question types • Do not write answer choices (that’s later!)

SCALE ITEMS

Scale Items • Should you use a scale? – If question is clearly dichotomous don’t use scale • True/false, yes/no

– Nominal items are not compatible with scales – Scales are usually interval or ordinal, can be ratio

Scale Items • Two-sided Scales (Bipolar) – Anchors have equal valence at opposite sides of a neutral mid-point

• One-sided Scales (Unipolar) – Low anchor represents absence of property, zero point, or least valence – High anchor represents high point or greatest valence

Scale Items – Two-Sided Scales • Positive – neutral – negative • Typically, sides should be balanced in number of items and degree of labels – Rarely, research interests may dictate otherwise

• Scale should have a neutral midpoint – Absence of reasonable midpoint may indicate that scale is single-sided

Scale Items – Two-Sided Scales • Labeled neutral midpoint improves reliability • Two or three levels of differentiation on either side of neutral • In all, five or seven answer choices

Scale Items – Two-Sided Scales

Scale Items – Two-Sided Scales

Scale Items – One-Sided Scales • • • •

Zero – Maximum Measure levels or gradation, not direction Scale does not have a midpoint (“neutral”) Make sure bottom point represents absence or very lowest presence of measured trait • Optimal length is generally four, or sometimes five, categories

Scale Items – One-Sided Scales

Scale Items – One-Sided Scales

Scale Items – One or Two Sides?

Scale Items – Validity • Don’t make your participants lie! – If acceptable choice is not present, participant must either close survey, skip question, or lie

• Scales should include full range of choices • Anchors should reasonably segment range • Labels should be easy for participants to interpret and evaluate

Scale Items - Validity • Two-sided scales should include neutral midpoint • One-sided scales should start at zero point, e.g., “none”, “not at all”, “never” – If “never” is truly unexpected, it can be combined with next-lowest choice, e.g., “never or rarely”

SHAPING RESPONSE

Response – Construct-Specific Labels • Choice labels that directly describe the choice (“very anxious”) are more reliable than indirect labels (“strongly agree”) – Before deciding how to answer agreement scales, participants must analyze original statement – Direct statements do not require this extra step

• Direct statements are not always feasible

Response – Construct-Specific Labels

Response – Anchoring Effects • Two Types: – Extremity and/or wording of question labels (especially endpoints) influences choice selection

– Early questions affect perception of later questions

Response – Anchoring Effects




Response – Acquiescence Bias • Social Desirability – Participants acquiesce toward choices they view as more socially acceptable – People also tend away from extremes

• Leading Questions – Question text may indicate researcher’s desired or expected answers

Response – Acquiescence Bias

Response – Acquiescence Bias

Response – Acquiescence Bias • Leading Questions: – If using two-sided scale, make sure both options are in the question • “How much do you agree…” should be “How much do you agree or disagree…”

– Avoid words that pass judgment when possible • “Do you prefer high art or popular music?” could be reframed as “Do you prefer opera or rock and roll music?”

Response – Recency Bias • Respondents heavily weight recent events and discount less recent events • Questions should scope timing whenever possible • Avoid “Typically, how often do you…” or “In the last five years, how often have you…”

Response – Free Response • Free response questions need to be specific and direct • Respondents are less likely to respond to vague, overarching questions • Participants who respond will say more if question is specific • Space given implies expected answer length

Response – Free Response

Small Group Exercise – 10 min. • Refine and edit questions • Create answer choices

BREAK

Demographics • Only ask demographics that are needed – If you don’t need to know income, don’t ask it – If your population is uniform in level of education, you already know it – don’t ask it

• Know the context of your audience – Gender / sex and available choices – Race / Ethnicity and available choices

PARTICIPANT EXPERIENCE

Participant Experience • Participants are people too! • If burdens of time, comprehension, or confidentiality are too much, participant may: – Exit the survey – Answer dishonestly – Lose focus – Become distracted by other things

Participant Experience • Survey Flow – Survey is an interview, conversation – Avoid placing probing questions early – Build and transition smoothly and logically – Avoid “tricking” respondents – Consider effects question order may have

Participant Experience • Time – How much of your participant’s time are you expecting to take? BE REALISTIC!

• Temporality – Consider what is going on for your population at the time you’re collecting data • Holidays, work loads, semesters, other similar surveys, etc.

Survey Review – 5 minutes • Small Group Surveys – How does it look? – How long will it take? – Is it asking the right questions? – Are any topics missing? – Will the results address the research question? – Can actions be taken based on responses?

Survey Review – 5 minutes • Political Participation and Attitudes Survey – What is the research question being investigated? – What is the audience? – Is questionnaire appropriate length? – Are any questions especially good? – Are any questions especially problematic?

Resources • Dillman, Don, Jolene D. Smyth, and Leah Melani Christian. Internet, Mail, and Mixed Mode Surveys: the Tailored Design Method. Hoboken, NJ: John Wiley & Sons, Inc., 2009. • Weisberg, Herbert F., Jon A. Krosnick, Bruce D. Bowen. An introduction to survey research, polling, and data analysis. Thousand Oaks, Calif. : Sage Publications,1996. • McDaniel, Carl, Jr., Roger Gates. Marketing Research. Hoboken, NJ: John Wiley & Sons, Inc., 2010.

Discussion • Thank you!