EMPIRICAL REPORT
Evaluating Scientific Research in the Context of Prior Belief: Hindsight Bias or Confirmation Bias? Amy M. Masnick, PhD Hofstra University Corinne Zimmerman, PhD Illinois State University When there is a mismatch between new evidence and prior beliefs, do people reject the conclusions from this evidence because of confirmation bias or do they support them because of hindsight bias? Ninety-four participants expressed a belief about a study’s outcome before reading a research report. When belief was confirmed, the study’s methodology was subsequently rated more positively and findings (whether presented with or without an explanation) were rated as more obvious, important, and interesting than when beliefs were disconfirmed. However, the presence of an explanation for the reported findings affected ratings of obviousness and interestingness but not of methodology. These results indicate that judging a research finding to be obvious involves more than a simple hindsight bias.
Keywords: confirmation bias; hindsight bias; evidence evaluation; reasoning; obviousness
M
ost people have conceptions about how the natural and social world works. These beliefs may be held with varying degrees of conviction. At the same time, people encounter new information every day that may call into question some of these pre-existing beliefs. When new knowledge and prior beliefs conflict, how do individuals evaluate and reconcile the two? Reconciling existing beliefs with new evidence is one of the key components of scientific reasoning (across the lifespan), but it is also necessary for reasoning about the scientific findings individuals are exposed to regularly through a variety of media sources. Therefore, the answer to how individuals reconcile conflicting beliefs and evidence is important not only for understanding the cognitive processes involved in scientific reasoning, but it also has practical implications for understanding scientific literacy and reasoning about science.
Personal, professional, and public policy decisions may be informed by evaluating research findings (Miller, 2004; Shapin, 1992), and these evaluations may include assessing empirical findings, theoretical explanations, or both. However, past research in cognitive science and social cognition leads to different conclusions about how people evaluate new information. On one hand, there is evidence that people are reluctant to consider evidence that disagrees with their beliefs and have a bias to look for information that confirms prior beliefs. (In this article, we consider confirmation bias as a bias toward one’s initial beliefs rather than as a hypothesis-testing strategy [e.g., Klayman & Ha, 1987].) On the other hand, there is evidence that people are susceptible to hindsight bias; that is, they agree with newly learned conclusions and claim to have held these beliefs all along.
Journal of Psychology of Science and Technology, Volume 2, Number 1, 2009 © Springer Publishing Company DOI: 10.1891/1939-7054.2.1.29
29
30
Numerous studies of scientific reasoning have shown that it is difficult to integrate evidence disconfirming prior belief (see Zimmerman, 2000, 2007, for reviews). In particular, when new evidence challenges an existing causal belief, that belief is more resistant to change than when a noncausal belief is challenged (e.g., Kanari & Millar, 2004). Additionally, people are particularly likely to disbelieve new evidence when a theoretical reason or plausible causal mechanism to maintain the current belief is present, and are more likely to take seriously an implausible covariation between two factors when there is a plausible causal explanation (Ahn, Kalish, Medin, & Gelman, 1995; Koslowski, 1996). Similar types of studies on attitude change have also looked at how individuals evaluate evidence in the context of prior belief. Participants in these studies evaluate evidence about controversial topics for which they often have strong prior beliefs (e.g., gun control, capital punishment). For example, Munro, Leary, and Lasane (2004) found that participants whose beliefs were disconfirmed by a fictitious study about the relationship between homosexuality and mental illness were more critical of the study’s methodology than those whose beliefs were confirmed. In general, biased assimilation and attitude polarization are common when evaluating evidence (e.g., Lord, Ross, & Lepper, 1979). That is, research is evaluated more favorably if it supports initial attitudes (biased assimilation), and rather than becoming more moderate in the face of disconfirming evidence, attitudes often become more extreme (attitude polarization; see MacCoun, 1998, for a review). In contrast to research showing that prior beliefs are resistant to change, other research evaluation studies have focused on the perception that, independent of prior belief, social science and educational research findings are considered “obvious” or “common sense” (e.g., Gage, 1991; Yates, 2005). In these studies, participants are shown either true findings or their foils (i.e., false findings). When presented with research findings from the areas of personality (Barnett, 1986), developmental psychology (Barnett, Knust, McMillan, Kaufman, & Sinisi, 1988), and social psychology (Richard, Bond, & Stokes-Zoota, 2001), accuracy in distinguishing true findings from foils ranged from 66%–75% (chance is 50%). Using a similar methodology, Wong (1995) found that
MASNICK AND ZIMMERMAN
when presented with both a true finding and a foil about educational research, participants were equally likely to select either version as the actual finding. Moreover, ratings of obviousness were equivalent for findings and nonfindings. Results presented with an explanation for the finding, however, were rated as significantly more obvious than those without an explanation, for both actual findings and foils. These findings are consistent with research on hindsight bias (e.g., Guilbault, Bryant, Brockway, & Posavac, 2004). That is, when individuals have knowledge of an outcome, they tend to overestimate their knowledge in advance of knowing that outcome (i.e., the “I knew it all along” phenomenon). Descriptions of hindsight typically include two components: (a) overestimation of one’s ability to predict an outcome after the fact, and (b) the belief that one is not in fact influenced by the knowledge of the outcome (Hawkins & Hastie, 1990). More recent discussions, however, distinguish several subtypes of hindsight bias (Blank, Nestler, von Collani, & Fischer, 2008). Here we concentrate on a hindsight bias, focusing on one’s ability to assess the likelihood of an outcome after knowing the outcome, as measured by ratings of how obvious research findings are after one has read them. Wong’s findings suggest that this type of hindsight bias may be even stronger in the presence of a plausible explanation for the reported finding. Most previous research examining the obviousness of findings has not specifically examined the effects of prior expectations about the findings. Past research on scientific reasoning and social cognition typically uses topics for which individuals have prior expectations, due either to ingrained or politicized beliefs or to fairly robust science misconceptions. The current study explored whether findings from newly reported research would be subject to a “knew-itall-along” hindsight bias when a prior belief is expressed. We also explored whether the presence of an explanation for a finding influences evaluations of obviousness of a study after having one’s belief either confirmed or disconfirmed. We created a task that includes characteristics used by researchers studying reasoning, attitude change, and perceptions of the obviousness, importance, and interestingness of research findings. We explored the process of evaluating educational research findings because individuals have been shown
EVALUATING RESEARCH
to be less accurate in discriminating true findings and foils in this domain (Wong, 1995). If evaluating research after expressing a belief about the outcome leads to the confirmation bias found when evaluating evidence that challenges or confirms political attitudes (e.g., Lord et al., 1979; Munro et al., 2004), then evaluations should differ as a function of prior belief. Specifically, evaluations about the methodology used to conduct the study should be more positive after learning that the result confirms prior belief (and negative when results disconfirm prior belief ). Overall evaluations of the research would be more positive by those whose beliefs are confirmed by a study’s findings because participants would be allowing prior beliefs to bias interpretation of new information. Moreover, the presence of an explanation may lead to different cognitive processing based on prior belief. Explanations for confirmed beliefs may result in positive evaluations, whereas explanations for disconfirmed beliefs may result in negative evaluations, as predicted by confirmation bias. Alternatively, if simply reading about a research result can make a finding seem obvious in hindsight, then participants’ evaluations about study quality and the obviousness of results should be similar regardless of prior expectations about the findings (e.g., Richard et al., 2001; Wong, 1995). In addition, the presence of an explanation for the findings may affect evaluations even when a belief is challenged. Investigations of scientific reasoning have shown that the ability to reconcile beliefs and evidence is often mediated by the presence of (or ability to generate) plausible causal mechanisms for the pattern of evidence/data (e.g., Ahn et al., 1995; Koslowski, 1996; Wong, 1995). When reasoning about scientific research, it is reasonable to assume that individuals are also considering theoretical explanations or causal mechanisms (e.g., Chinn & Brewer, 2001). That is, hindsight bias may be more evident when explanations are present, such that an explained finding may seem even more obvious regardless of prior belief. In sum, the present study explored two factors that could influence the evaluation of a research study: (a) having one’s belief about an educational research question confirmed or disconfirmed by evidence, and (b) the presence or absence of a causal explanation for the finding.
31
METHOD
Participants Ninety-four undergraduate students (82 women, 12 men) participated for course credit. The average age was 22.1 (SD = 4.1). They had an average of 3.8 years of college-level instruction (SD = .74), with an average of 1.8 (SD = 1.1) methods/statistics courses. The racial composition of the sample was 73% White, 12% African American, and 7% each Hispanic and Asian. Most participants indicated they were majoring in psychology (37%), education (30%), or psychology and education (13%).
Materials and Procedure A questionnaire was administered in small groups. The first page briefly described an issue in early science education. Participants were told that researchers have been studying the relative efficacy of direct instruction (being taught information explicitly) as compared to discovery learning (being given a goal and some materials and told to explore on one’s own). Participants were asked to indicate which method they believed was the most effective (i.e., a forcedchoice question). The procedure of requiring a forced choice between the two sides of this issue allows participants to be later classified as having their belief confirmed or disconfirmed by the result of a study (Munro et al., 2004).1 All participants then read the same one-page description of an experimental study, including a brief introduction (background and research questions) and method section. Participants then evaluated the appropriateness of the methods, design, participants, and measures on a 7-point Likert scale (from strongly disagree to strongly agree that each was appropriate). After evaluating the study’s methodology, participants were randomly assigned to read one of four versions reporting the study’s findings. In two versions, direct instruction was described as the more effective intervention. One version asserted this finding without an explanation. The other version included the explanation that direct instruction was more effective because learning is facilitated by the teacher’s organized presentation and focus on relevant concepts and procedures. In the other two versions, discovery learning was described as the more effective
32
intervention. One version included the explanation that discovery learning is more effective because of students’ active involvement in the learning process, which makes the information more meaningful and likely to be remembered. After reading the findings (with or without an explanation) participants rated characteristics of the conclusions of the study using a 7-point Likert scale. Participants rated how strongly they disagreed or agreed with statements about the conclusions with respect to whether they were (a) obvious, (b) important, and (c) interesting. Participants then completed a second evaluation of the methods, design, participants, and measures. Participants were debriefed at the end of the study about the study’s purpose.
RESULTS Based on the initial forced-choice question, one-third of the students believed that direct instruction was more effective and two-thirds believed that discovery learning was more effective. Because of random assignment to which results were seen, approximately equivalent groups had initial belief confirmed (n = 46) and disconfirmed (n = 48). To assess the equivalence of versions, ratings were initially analyzed disregarding prior belief. Ratings of the obviousness of the reported findings were similar for those who read the discovery-learning conclusion (M = 4.9, SD = 1.4; n = 46) and the direct-instruction conclusion (M = 4.6, SD = 1.4; n = 48). In addition, no differences were found for the presence of an explanation or in initial or postfinding evaluations of methodology based on which version participants read (all Fs ≈ 1, ps > 0.10). Thus, the content of the conclusion participants read (discovery learning or direct instruction) did not influence participants’ reasoning about the study. We therefore focused on whether participants’ beliefs were confirmed or disconfirmed, disregarding prior belief. The seven ratings participants gave (method, design, participants, measures, obviousness, importance, and interestingness) were entered into a multivariate 2 (belief: confirmed vs. disconfirmed) × 2 (explanation: present vs. absent) analysis of variance (ANOVA).
MASNICK AND ZIMMERMAN
Judgments About Methodology No differences in initial ratings of appropriateness of the method, design, participants, or measures were found as a function of either belief or explanation (Fs ≈ 1, ps > 0.10) so analyses were conducted on change scores (second minus first assessment). Ratings of the appropriateness of the methods increased for those whose belief was confirmed (M = .50; SD = 1.1) but decreased for those whose belief was disconfirmed (M = −.56; SD = 1.6), F(1, 90) = 13.3, p < .001, partial η2 = .13 (Figure 1). No differences were found for the presence of an explanation, and there was no interaction between explanation and belief confirmation (Fs ≈ 1, ps > 0.10). Participants’ ratings of the design used by researchers increased when beliefs were confirmed (M = .74, SD = 1.1) but decreased (M = −.35, SD = 1.1) when beliefs were disconfirmed, F(1,90) = 13.9, p < .001, partial η2 = .13. No differences were found for the presence of an explanation, and there was no interaction (Fs ≈ 1, ps > 0.10). The pattern was similar to that illustrated in Figure 1. There were small positive changes in ratings about the appropriateness of the participants and measures but no differences among groups (all Fs ≈ 1, ps > 0.10).
Judgments About the Findings and Conclusions Participants’ ratings of the obviousness of the conclusion were greater when initial belief was confirmed (M = 5.1, SD = 1.4) than when initial belief was disconfirmed (M = 4.4, SD = 1.4), F(1,90) = 5.78, p = .018, partial η2 = .06. Although there was no main effect of explanation, there was an interaction between belief confirmation and explanation, F(1,90) = 4.26, p = .042, partial η2 = .05 (Figure 2). When there was no explanation for the finding, ratings were similar for those whose beliefs were confirmed or disconfirmed. Ratings of obviousness were much higher for those whose belief was confirmed compared to those whose belief was disconfirmed, F(1,90) = 9.70, p = .002, partial η2 = .10. That is, when findings were presented with an explanation, there was a large difference in perceived obviousness. Participants’ ratings of the importance of the research were also greater when beliefs were confirmed
EVALUATING RESEARCH
FIGURE 1. Mean change in appropriateness of methods rating (before and after reading the results of the study) as a function of whether one’s belief was confirmed or disconfirmed by the findings, and whether or not an explanation for the findings was presented.
FIGURE 2. Mean ratings of the obviousness of the findings/conclusions as a function of whether one’s belief was confirmed or disconfirmed by the findings, and whether or not an explanation for the findings was presented.
33
34
(M = 6.1, SD = 1.3) than when beliefs were disconfirmed (M = 5.4, SD = 1.4), F(1,90) = 7.94, p = .006, partial η2 = .08. The presence of an explanation resulted in higher ratings of the findings as important, F(1,90) = 4.06, p = .047, partial η2 = .04, but the presence of an explanation did not interact with belief. The same pattern was found for ratings of interestingness, with higher ratings from those whose belief was confirmed, F(1,90) = 12.09, p = .001, partial η2 = .19, and for those who read an explanation for the finding, F(1,90) = 7.80, p = .006, partial η2 = .08, with no interaction.
DISCUSSION The results demonstrate that when evaluating reports of social science research, the tendency toward hindsight bias is trumped by beliefs expressed before evaluating the findings. Evidence that contradicts initial belief is assessed differently from evidence that supports it. Consistent with predictions from research on attitude change and scientific reasoning, there was a strong effect of confirmation on ratings of methodology, design, and judgments of several qualities of the research, such as its obviousness. This finding indicates that when stating an expectation about the finding prior to reading the outcome, participants are more likely to indicate a confirmation bias than a hindsight bias. Both the methods and the findings were rated more positively after a belief was confirmed and more negatively after a belief was disconfirmed. Recall that both findings were judged to be equivalent, independent of prior belief (i.e., the discovery learning and direct instruction versions were rated equally obvious), and methodology ratings were equivalent prior to reading the findings. Thus, it was not the specific content that affected evaluation but rather the confirmation or disconfirmation of prior beliefs. Hindsight bias was not in evidence: people remained tethered to their initial beliefs more than the newly presented conclusions, as indicated by the effect of confirmation on the change in ratings of the appropriateness of the methodology. This finding suggests that working to change someone’s view simply based on content may be unsuccessful without considering initial views on the topic. Evaluating the methodological soundness
MASNICK AND ZIMMERMAN
of a study may also be influenced by pre-existing beliefs. One difference between the current study and many other studies of hindsight bias (e.g., Hawkins & Hastie, 1990) is that research findings can be assessed both on the nature of the conclusions and on methods by which the conclusions were drawn (unlike, for example, the assessment of a historical event). This distinction may be one reason we found little evidence of hindsight bias. Assessments of the appropriateness of participants (i.e., generalizabililty to other populations) and measures (i.e., construct validity) were not influenced by prior belief. Participants may have been able to evaluate these features of the study more objectively, or they may not have believed these characteristics were as important in evaluating a study’s validity. Having one’s belief confirmed also resulted in ratings of the reported findings as more obvious, interesting, and important. In contrast to previous research showing a type of hindsight bias in the evaluation of research findings, when prior belief was taken into consideration, ratings of obviousness varied as a function of that prior belief. If hindsight bias were at work, we would expect equivalent (and positive) obviousness ratings regardless of whether a prior belief was confirmed or challenged. Although lower ratings of obviousness do not necessarily indicate participants are discounting or discrediting the study, the differences in ratings are still changing based on confirmation. Two possible explanations seem plausible for this effect. First, although we presume participants’ initial opinions on the topic (i.e., educational interventions) were formed quickly and on the spot, most participants were psychology or education majors (80% of the sample), and so they may have had occasion to consider such topics prior to participation. If so, then challenging or confirming those beliefs would be expected to result in behavior similar to that observed when other strongly held attitudes are challenged (e.g., Munro et al., 2004). Second, it is also possible that being forced to state an opinion early, even if chosen somewhat arbitrarily, leads to a commitment to the position and consistency within the task. To explore this possibility further, it will be necessary to replicate this finding with a group that does not express an opinion prior to reading the methods and findings.
EVALUATING RESEARCH
The presence of an explanation for a finding led to higher ratings of how important and interesting that finding was. These results are consistent with past work indicating that explanations are a critical part of reasoning about science (e.g., Ahn et al., 1995; Koslowski, 1996; Wong, 1995). Individuals are more likely to believe an empirical finding if there is a theory or explanation for that finding. Thus, it is unsurprising that the presence of explanatory information would increase perceptions of how important and interesting a topic is. Yet the explanation effect was not completely straightforward. An unexpected finding was the interaction of explanation and belief confirmation on ratings of obviousness. Wong (1995) found that the presence of an explanation increased feelings of obviousness for both true findings and foils. We found similar results, but only under certain conditions. In the absence of an explanation, findings were rated as equally obvious regardless of prior belief. However, when an explanation was provided, a confirmed belief was rated as much more obvious than a disconfirmed belief. Thus, the absence of an explanation actually eliminated the effect of prior belief. It is possible that participants generate their own explanations for the results and therefore rate the findings as equally obvious. In addition, with disconfirming evidence, participants may become more critical and discredit both the finding and the explanation for that finding to maintain their belief, leading to a lower rating of obviousness (Klaczynski & Narasimham, 1998). If a belief has just been disconfirmed, then the accompanying explanation is presumably one that has either not been considered earlier or has been considered and dismissed. In summary, the methodology developed in the current research can be productive for examining the process of evidence evaluation and for making links among several research literatures. The findings add to the literature on scientific reasoning by showing that the presence of an explanation does not always affect evaluation of scientific reports in a uniform way. Consistent with previous research on confirmation bias, we found that prior belief exerts a powerful influence, but our results suggest this effect is not diminished by the presence of a plausible explanation. The current findings also contribute to the hindsight bias literature by demonstrating that although partici-
35
pants often consider research findings to be obvious, there are some boundary conditions for such judgments (e.g., belief confirmation or disconfirmation in conjunction with the presence of an explanation). The judgment of a research finding to be obvious clearly involves more than a simple hindsight bias. Additional research is needed to further explore the complexities of evaluating the kinds of research evidence commonly reported in the media that may be used to inform personal and policy decisions.
NOTE 1. Munro et al. (2004) allowed individuals to indicate ambivalence by selecting a midpoint on a Likert scale when expressing views about homosexuality. This procedure resulted in removing only a small minority of individuals from their analyses (4 of 92).
REFERENCES Ahn, W., Kalish, C. W., Medin, D. L., & Gelman, S. A. (1995). The role of covariation versus mechanism information in causal attribution. Cognition, 54, 299–352. Barnett, M. A. (1986). Commonsense and research findings in personality. Teaching of Psychology, 13, 62–64. Barnett, M. A., Knust, J., McMillan, T., Kaufman, J., & Sinisi, C. (1988). Research findings in developmental psychology: Common sense revisited. Teaching of Psychology, 15, 195–197. Blank, H., Nestler, S., von Collani, G., & Fischer, V. (2008). How many hindsight biases are there? Cognition, 106, 1408–1440. Gage, N. L. (1991). The obviousness of social and educational research results. Educational Researcher, 20, 10–16. Guilbault, R. L., Bryant, F. B., Brockway, J. H., & Posavac, E. J. (2004). A meta-analysis of research on hindsight bias. Basic and Applied Social Psychology, 26, 103–117. Hawkins, S. A., & Hastie, R. (1990). Hindsight: Biased judgments of past events after the outcomes are known. Psychological Bulletin, 107, 311–332. Kanari, Z., & Millar, R. (2004). Reasoning from data: How students collect and interpret data in science investigations. Journal of Research in Science Teaching, 41, 748–769. Klaczynski, P. A., & Narasimham, G. (1998). Development of scientific reasoning biases: Cognitive versus egoprotective explanations. Developmental Psychology, 34, 175–187. Klayman, J., & Ha, Y. (1987). Confirmation, disconfirmation, and information in hypothesis-testing. Psychological Review, 94, 211–228.
36 Koslowski, B. (1996). Theory and evidence: The development of scientific reasoning. Cambridge, MA: MIT Press. Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37, 2098–2109. MacCoun, R. J. (1998). Biases in the interpretation and use of research results. Annual Review of Psychology, 49, 259–287. Miller, J. D. (2004). Public understanding of, and attitudes toward, scientific research: What we know and what we need to know. Public Understanding of Science, 13, 273–294. Munro, G. D., Leary, S. P., & Lasane, T. P. (2004). Between a rock and a hard place: Biased assimilation of scientific information in the face of commitment. North American Journal of Psychology, 6, 431–444. Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2001). “That’s completely obvious . . . and important”: Lay judgments of social psychological findings. Personality and Social Psychology Bulletin, 27, 497–505.
MASNICK AND ZIMMERMAN Shapin, S. (1992). Why the public ought to understand science-in-the-making. Public Understanding of Science, 1, 21–30. Wong, L. Y. (1995). Research on teaching: Process-product research findings and the feeling of obviousness. Journal of Educational Psychology, 87, 504–511. Yates, G. C. R. (2005). “How obvious”: Personal reflections on the database of educational psychology and effective teaching research. Educational Psychology, 25, 681–700. Zimmerman, C. (2000). The development of scientific reasoning skills. Developmental Review, 20, 99–149. Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle school. Developmental Review, 27, 172–223.
Correspondence regarding this article should be directed to Amy M. Masnick, Hofstra University, Hauser Hall, Psychology Department, Hofstra University, Hempstead, NY, 11549. E-mail:
[email protected]
