How True Is True? Assessing Socially Desirable ... - Springer Link

5 downloads 0 Views 67KB Size Report
Abstract. Health care professionals' (HCPs) opinions and perspectives are highly valued because these individuals often play a major role in developing and ...
Quality & Quantity 37: 327–335, 2003. © 2003 Kluwer Academic Publishers. Printed in the Netherlands.

327

Research Note

How True Is True? Assessing Socially Desirable Response Bias B. ALEX MATTHEWS1 , FRANK BAKER2 and RACHEL L. SPILLERS2 Center for Patient Care and Outcomes Research, Medical College of Wisconsin, Milwaukee, Wisconsin, U.S.A.; 2 Behavioral Research Center, National Home Office, American Cancer Society, 1599 Clifton Road, N. E., Atlanta, GA 30329-4251, U.S.A.

Abstract. Health care professionals’ (HCPs) opinions and perspectives are highly valued because these individuals often play a major role in developing and implementing support and education services aimed at cancer patients and their families. The purpose of this study was to examine the efficacy of adding a substantive measure that would be useful for determining socially desirable responses (SDRs) without adding unnecessary length to the questionnaire design. A total of 1180 nurses, physicians, and social workers specializing in oncology returned fully completed mailed questionnaires (61% response rate) originally intended to measure HCPs’ awareness (i.e., knowledge, helpfulness, and recommendations) of cancer support services. SDRs were assessed by the inclusion of a bogus program that was compared to actual support programs. Results indicated that relative to the bogus program, HCPs were significantly more likely to endorse programs that they knew about, thought helpful, and recommended. Evidence of SDR bias was lacking. These findings provide support for the inclusion of measures that can be used on brief questionnaires to strengthen research methodology. Key words: socially desirable responses, health care professionals, cancer, research methodology.

1. Introduction It is important that researchers investigate sources of bias that could affect conclusions (Asch et al., 1997). It is particularly important for surveys of physicians and other health care professionals (HCPs) who are difficult to recruit (Asch et al., 1997; Fridinger et al., 1992; Mayheux et al., 1989) and in the domain of health care services because positive adaptation to illness often depends on effective interventions (Fawzy et al., 1995; Gray et al., 1999). Ironically, although members of busy professional health care teams are more likely to be besieged with surveys, they also may have less time to complete surveys and do not wish to spend the limited time they have contemplating subject matter of little substantive interest  Author for correspondence.

328

B. ALEX MATTHEWS ET AL.

such as that contained on standard measures of social desirability (Asch et al., 1997; MacPherson and Bisset, 1995; Maheux et al., 1989). Conventional methods of assessing socially desirable responses (SDRs) or response-set bias consume considerably more space that lengthen questionnaires and involve greater participant time. Therefore, questionnaires that are salient, tailored, and short also have the potential for the highest return (Dillman, 2000; Salant and Dillman, 1993). The purpose of this study was to examine the efficacy of adding a substantive measure that would be useful for determining bias due to response style without adding unnecessary length to the questionnaire design. Bias is defined as anything that produces systematic error in a research finding (i.e., the effects of any factor that the researcher did not expect to influence the dependent variable (Paulhus, 1991; Vogt, 1993). Mailed questionnaires are often used as a preferred method for data collection because they can be completed at the convenience of the respondent and they offer perceived anonymity (Dillman, 2000; Lohr, 1999; Paulhus, 1991). However, the validity of the data obtained by this method may be hampered by the tendency to present oneself in a more favorable light than is actually the case (Dijkstra et al., 2001). Social desirability bias is generally defined as providing responses that are perceived as more acceptable than the response that the participant would have made under neutral conditions. This type of bias results from respondents trying to answer questions as a good person should rather than in a way that reveals what they actually believe or feel (Vogt, 1993). Social desirability is one of the most common sources of bias and can seriously affect the validity of the survey findings (Nederhof, 1985). In the case at hand, HCPs could conceivably want to show that they were more (or less) aware of programs that were sponsored by a favored (or disfavored) organization. Or, respondents may wish to show that they knew about more cancer support services than they actually did, simply because of social expectations (Hagedoorn et al., 2000). Although there are several standard scales available for assessing SDR, they typically add bulk to the questionnaire while not contributing substantively to the study. Additionally, social desirability scales that use Likert-style response formats require more thought than simple dichotomous response formats and place greater burden on respondents. Reverse coded items also increase completion time because participants must stop and consider the item more carefully (Buss and Perry, 1992; Lohr, 1999). However, simple yes or no response items may increase the risk of response set bias (i.e., repeatedly answering in one direction only). Therefore, the addition of a dichotomously scored program might also serve as a check against response set bias. Given the constraints of a difficult, but important population to assess, together with limited study time and questionnaire space, we devised a method of investigating item response bias by including a non-existent or bogus cancer support program together with a list of very real programs. We expected that if the respondents were answering all items positively, regardless of whether the respondent actually

HOW TRUE IS TRUE?

329

knew about, recommended, or thought the program was helpful, the bogus program would be rated as highly across the dependent variables as the actual programs. Additionally, if social desirability in the form of appearing to know more than what one actually knows was present, or if the respondent simply wanted to endorse programs that were aligned (or not aligned) with the sponsoring organization, then again, the bogus program would be as likely to be endorsed as other programs. We hypothesized that this would not be the case. 2. Method 2.1. SAMPLE AND PROCEDURES This study was conducted as an adjunct to another study designed to investigate awareness and recommendations to cancer support services among oncologic health care professionals (Matthews et al., 2002). A brief (M = 5 minute, 4 page) questionnaire was mailed to a randomly selected sample of 2000 HCPs (1000 physicians, 500 nurses and social workers) who also were active members of their respective oncology associations. The study was completed within a 20week period over three mailing waves (initial, follow-up studies at 30 days and 90 days). Each of the three mailings included a letter of invitation explaining the purpose of the study, the questionnaires, a pre-stamped self-addressed envelope, and a contingent thank-you packet containing cancer educational materials. Incentive information packets were contingent to decrease the potential for positive bias toward the sponsoring agency prior to completion of the questionnaires. Of the 1241 questionnaires returned (62% return rate), 47 were excluded because the HCP did not meet inclusion criteria (i.e., adult-only practice, active, US-based members of the American Society of Clinical Oncology, the Oncology Nursing Society, or the Association of Oncology Social Work), 8 were received after the close of the study, and 6 were grossly incomplete, resulting in a total sample of 1180 (61% response rate). 2.2. MEASURES Twenty-four of the 63 items contained on the questionnaire asked HCPs whether they had heard of, recommended, or thought a specific list of randomly ordered cancer support services were helpful to patients. Of the 24 listed, 22 were existing services. These services were selected from a variety of cancer resources (online information, brochures, and other printed materials) representing a variety of program types (e.g., education, emotional, or tangible support such as home nursing or transportation services). The remaining options included 1) an open-ended other category to allow HCPs to name unlisted services and 2) a bogus non-existent program. The bogus program, Links to Life, was believably described and chosen because it was very close in name to an existing program Life Links. We hoped that such a fine discrimination would make it even more difficult to detect the

330

B. ALEX MATTHEWS ET AL.

bogus program and ensure that the questionnaire was actually measuring what we intended. Sample demographic and vocational characteristics also were included on the questionnaire. 2.3. DATA ANALYSIS Frequencies, measures of central tendency, and measure of variability were used to examine the data. Nonparametric procedures (McNemar and Sign Test) for related samples were used to test the hypothesis that the two programs had the same distribution. According to Hurlburt (1998), the null hypothesis for the McNemar Test is that the proportion of individuals who said yes under one condition (e.g., an existing cancer support/education program) is the same as the proportion of those same individuals who said yes under another condition (e.g., a non-existent program). The differences between the two variables for all cases are computed and classified as either positive, negative, or tied. If the two variables are similarly distributed, non-tied cases should be split about evenly between positives and negatives. The Sign Test (which is the square of the McNemar coefficient) will not obtain significance if the numbers of positive and negative differences do not differ (Norusis, 1997). The alternative hypothesis is that the proportions differ. 3. Results On average, HCPs reported knowing about 9 of the listed services, almost 7 of the programs were regarded as helpful, and 6 were recommended. As shown in Table I, greater percentages of HCPs knew about, regarded as helpful, or recommended well-established cancer programs, especially those offering educational and emotional support, than those offering tangible support such as transportation. Additionally, programs that were tertiary in nature (i.e., generally aimed at patients after a cancer diagnosis) were more likely to be endorsed than were primary prevention programs such as breast cancer screening or solar protection programs. Only one service was less likely than Links to Life, the bogus program, to be recognized, thought helpful, or recommended. In general, knowledge tended to be greater than ratings of program helpfulness or recommendation, but all three constructs were highly significant (r = 0.65 for knowledge and helpfulness, r = 0.73 for knowledge and recommendations, and r = 0.76 for helpfulness and recommendations, ps< 0.001). Table II displays the proportion of discordant responses reported by the same respondent under two conditions. In Column 1, the percentage of affirmative responses (i.e., yes) to the reference program (e.g., I Can Cope) was compared to negative responses (i.e., no) for the bogus program as reported by the same individual. Column 2 shows the percentage of cases in which a respondent endorsed the bogus program to a greater extent than an actual program. Column 3 shows the results of the sign test (given in Z-scores for comparability between programs). For

Education and support through seminars 78.6 Support/visitation program for women with breast cancer 1-800-ACS-2345, cancer information and database Hair and make-up for women with cancer 1-800-4 CANCER, information and local resources Magazine/catalogue with products for women undergoing cancer treatment Educational material for health professionals, patients, and the public Web Page: www.cancer.org, education and resources Support groups, counseling, and wig bank for breast cancer survivors Support group for men with prostate cancer Beds, wheelchairs, and equipment Transportation to/from appointments Visitation program from other survivors Information for cancer survivors and caregivers on cassette tapes Referrals to local hospices for cancer survivors and family caregivers Living accommodations for patient and family Needs-based air transport-special criteria apply Mammogram screening Exercise and peer-group support program for breast cancer survivors Information and sun protection products Education and support throughout the colorectal cancer experience Referral program for survivors and their families Online information and documentation on health and related topics

I Can Cope Reach to Recovery ACS CID Look Good . . . Feel Better NCI Hotline “tlc” CancerNet ACS Homepage Y-Me Man to Man Home Care Equipment Road to Recovery CanSurmount NCCS Tool Kit Hospice Link Hope Lodge Airlifeline Tell a Friend Encore Slip! Slop! Slap! Frankly Speaking ∗ Links to Life WHO.ch

54.6 76.9 76.5 72.6 72.4 52.1 49.2 45.5 46.3 44.7 44.2 43.9 39.5 30.1 28.9 23.2 21.5 20.2 16.6 10.3 8.9 4.3 2.3

K% 49.1 63.1 57.5 59.8 57.5 41.1 33.6 29.8 26.0 27.8 38.0 35.0 22.5 21.8 24.6 17.6 14.7 12.8 9.9 7.5 6.8 4.7 3.3

H%

59.2 55.5 57.6 56.7 40.2 29.7 28.5 21.9 23.0 36.4 34.2 17.5 19.9 22.5 14.0 12.5 10.0 6.6 4.5 3.9 2.4 0.9

R%

Note. American Cancer Society’s Cancer Information Database (ACS CID), National Cancer Institute (NCI), Tender Loving Care (tlc), National Coalition for Cancer Survivorship (NCCS), and World Health Organization (WHO). ∗ Links to Life was the bogus program.

Description

Program/Service

Table I. Health care professionals’ ratings of knowledge (K), helpfulness (H), and recommendation (R) of cancer support programs/services

HOW TRUE IS TRUE?

331

74.6 73.1 72.6 68.6 68.6 49.2 45.9 43.2 42.1 41.3 40.9 40.3 36.4 27.9 25.3 20.1 19.7 17.5 14.3 8.9 6.5 1.3

I Can Cope Reach to Recovery ACS Cancer Info Database Look Good . . . Feel Better NCI Hotline tlc Cancernet Y-Me ACS Homepage Man to Man Home Care Equipment Road to Recovery CanSurmount NCCS Tool Kit Hospice Link Hope Lodge Airlifeline Tell A Friend Encore Slip! Slop! Slap! Frankly Speaking World Health Organization

0.3 0.6 0.4 0.3 0.5 1.4 1.0 1.3 0.9 0.9 1.0 0.8 1.3 2.1 0.7 1.2 2.5 1.6 2.0 2.9 1.9 3.3

% Links

Helpfulness % Program 50.3 58.7 53.1 55.4 53.3 37.1 30.1 21.9 25.9 23.9 34.5 30.9 19.0 18.2 20.3 13.9 11.5 9.2 6.7 4.6 3.1 0.5

z −29.43 −28.99 −28.99 −28.22 −28.09 −23.02 −22.48 −21.56 −21.52 −21.29 −21.13 −21.16 −19.63 −16.10 −16.52 −14.01 −12.46 −12.40 −10.37 −5.94 −5.30† −3.13†∗

Note. N = 1180. See Table I for acronyms. All ps significant at 0.001, except ∗ p < 0.01. † Limited expected n in one or more cells.

Knowledge % Program

Program/Service

0.4 0.3 0.3 0.3 0.4 0.7 1.1 0.6 0.8 0.8 1.2 0.6 1.2 1.1 0.4 0.9 1.5 1.0 1.4 1.8 1.0 1.9

% Links −24.03 −26.06 −24.76 −25.36 −24.74 −20.31 −17.78 −15.39 −16.60 −15.95 −19.11 −18.51 −13.55 −13.31 −14.95 −11.49 −9.43 −8.67 −6.23 −3.70† −3.43† −2.84†∗

z 47.1 57.1 53.6 55.3 54.7 38.4 28.1 20.3 26.8 21.4 34.7 32.4 16.4 18.6 20.5 12.5 11.4 8.4 5.8 3.6 2.6 0.3

0.4 0.3 0.4 0.1 0.3 0.6 0.8 0.8 0.7 0.8 0.7 0.5 1.2 1.1 0.4 0.9 1.3 0.8 1.5 1.5 1.1 1.7

Recommendation % Program % Links

Table II. Percentage of health care professionals’ discordant responses to existing versus bogus cancer support program/service

−23.22 −25.69 −24.80 −25.46 −25.12 −20.75 −17.41 −14.48 −17.06 −14.98 −19.61 −19.04 −12.37 −13.50 −15.02 −10.79† −9.67† −8.56† −5.28† −3.07† −2.56† –†

z

332 B. ALEX MATTHEWS ET AL.

HOW TRUE IS TRUE?

333

virtually all programs, the percentage of negative cases was significantly greater than positive cases, ps < 0.001, indicating that the referenced program was significantly more likely to be endorsed on any of the measures than was the bogus program. Only the last program (WHO.ch) showed a positive difference (bogus program endorsed more than the reference program), p < 0.01. Z-scores decreased as discordance decreased and the respondent was less apt to report knowing about a true program, thinking it helpful, or recommending it, compared to the bogus program. As shown in the final rows, a few of the coefficients were unstable because of limited expected n in one or more cells.

4. Discussion The purpose of this study was to examine the efficacy of a measure to assess SDRs among HCPs responding to a brief mailed questionnaire regarding cancer support and education services. These results suggest that the inclusion of substantive items is a viable method for assessing potential response bias. Despite the fact that the bogus program resembled an existing program and was believably described, HCPs were significantly more likely to know about, recommend, and think that true programs were helpful compared to a bogus program. Conversely, very few HCPs who endorsed the bogus program also endorsed extant programs or services, indicating that error was more random than systematic. Evidence of SDR bias was lacking. Programs that were expected to be less well known, such as primary preventive type services, also were less likely to be endorsed on any measure. Because most oncology professionals initially see patients after they are diagnosed with cancer (Eakin and Strycker, 2001), it should be expected that primary prevention programs would not be as well known as tertiary programs. Additionally, very few participants endorsed the WHO.ch perhaps because the site for this organization appeared on the questionnaire without spelling out the acronym. Of course, had the full name of the organization been provided, endorsements no doubt would have increased, but the object was to list the services as they would have been used by the respondent. For example, to use a web site, the address, rather than the full name, would typically be specified. However, because the information for the lowest ranked program was not specific to cancer, a lower endorsement rate would still be expected compared to cancer-specific services. Comparisons of negative responses among well known cancer programs showed little difference within services; that is, participants were as likely to know about, recommend, and think one highly visible program as efficacious as another. Greater discrepancies were shown between negative and positive responses for programs that were informational or educational in nature compared to tangible services. These results corroborate those reported by Fawzy et al. (1995) meta-analysis of psychosocial interventions in cancer care (i.e., information and education programs are better known than other cancer services). However, as the listed program be-

334

B. ALEX MATTHEWS ET AL.

came increasingly unfamiliar, respondents were less likely to endorse it on any measure and the gap between discordant responses decreased. One limitation of the study is that there were insufficient responses in some cells to assess SDRs when the programs were not as well known. Although these coefficients are reported as unstable, they are in the expected direction and make intuitive sense. In short, previous research has shown that although standard measures of social desirability and response-set bias have been tested, they are often not feasible for short questionnaires. To our knowledge, this is the first study of its kind to test response style bias by the inclusion of a substantive variable such as endorsement of a bogus program. In summary, these findings suggest that self-report data resulting from mailed surveys appear to be valid and that methods to assess socially desirable responses can be introduced even in short surveys. Methods to assess and reduce bias are important because interventions such as education programs and support services depend on bias-free data that can be trusted to measure the intended, rather than unintended. Any potential source of measurement error, including that introduced by the respondent, should be investigated because error in measurement has implications not only for research findings, where it might lead to erroneous results, but also for the development of public health policies and program intervention development, implementation, and monitoring (Ramirez et al., 2000). Failure to account for systematic bias may lead to ineffective social service policies. References Asch, D. A., Jedrziewski, M. K. & Christakis, N. A. (1997). Response rates to mail surveys published in medical journals. Journal of Clinical Epidemiology 50: 1129-1136. Buss, A. H. & Perry, M. (1992). The aggression questionnaire. Journal of Personality and Social Psychology 63: 452–459. Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method (2nd edn). New York: John Wiley & Sons. Dijkstra, W., Smit, J. H. & Comijs, H. C. (2001). Using social desirability scales in research among the elderly. Quality & Quantity 35: 107–115. Eakin, E. G. & Strycker, L. A. (2001). Awareness and barriers to use of cancer support and information resources by HMO patients with breast, prostate, or colon cancer: Patient and provider perspectives. Psycho-Oncology 10: 103–113. Fawzy, F. I., Fawzy, N. W., Arndt, L. A. & Pasnau, R. O. (1995). Critical review of psychosocial interventions in cancer care. Archives of General Psychiatry 52: 100-113. Fridinger, F., Goodwin, G. & Chng, C. L. (1992). Physician and consumer attitudes and behaviors regarding self-help health support groups as an adjunct to traditional medical care. Journal of Health & Social Policy 3: 47–71. Gray, R. E., Carroll, J. C., Fitch, M., Greenberg, M., Chart, P. & Orr, V. (1999). Cancer self-help groups and family physicians. Cancer Practice 7: 10–15. Hagedoorn, M., Buunk, B. P., Kuijer, R. G., Wobbes, T. & Sanderman, R. (2000). Couples dealing with cancer: Role and gender differences regarding psychological distress and quality of life. Psycho-Oncology 9: 232–242. Hurlburt, R. T. (1998). Comprehending Behavioral Statistics (2nd ed). Pacific Grove, CA: Brooks/Cole.

HOW TRUE IS TRUE?

335

Lohr, S. L. (1999). Nonresponse. In Sampling: Design and Analysis. Pacific & Grove, CA: Duxbury Press, pp. 255–287. MacPherson, I. & Bisset, A. (1995). Not another questionnaire!: Eliciting the views of general practitioners. Family Practice 12: 335–338. Maheux, B., Legault, C. & Lambert, J. (1989). Increasing response rates in physicians mail surveys: An experimental study. American Journal of Public Health 79: 638–639. Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology 15: 263–280. Norusis, M. J. (1997). SPSS: Guide to Data Analysis. Upper Saddle River, NJ: Prentice Hall. Paulhus, D. L. (1991). Measurement and control of response bias. In: J. P. Robinson, P. R., Shaver, P. R. & L. S. Wrightsman (eds), Measures of Personality and Social Psychological Attitudes. San Diego, CA: Academic Press. Ramirez, M., Ford, M. & Stewart, A. L. (2000). Measurement and methodological issues in minority aging research. The Behavioral Measurement Letter 7: 13–18. Salant, P. & Dillman, D. A. (1994). How to Conduct Your Own Survey. New York: John Wiley & Sons. Vogt, W. P. (1993). Dictionary of Statistics and Methodology. Newbury Park, CA: Sage.