establishing reliability and validity in qualitative ...

Jharkhand Journal of Development and Management Studies XISS, Ranchi, Vol. 12, No.1, March 2014, pp. 5743'5753 .

ESTABLISHING RELIABILITY AND VALIDITY IN QUALITATIVE INQUIRY: A CRITICAL EXAMINATION Satyendra C. Pandey* a n d Srilata Patnaik** Both qualitative and quantitative paradigms try to find same result; the truth. However, both are often thought as competing paradigms where they take different approaches to establish truth. Reliability and validity are generally considered to be a concern for quantitative research; advocates of quantitative research often describe these issues in great detail in their research. However, in case of qualitative research the role of these dimensions are found to be blurred. Some researchers argue that the tools of reliability and validity are not applicable to qualitative research and they must be geared in a different way to measJre rigor and responsibility of qualitative research. This paper makes an attempt to clarify the meaning and use of reliability and validity in the qualitative research paradigm. I t uses the widely accepted constructs proposed by Lincoln and Guba in their seminal work on qualitative paradigm. The paper also explains the techniques that can be used by the investigators to ensure the reliability and validity in a qualitative research.

Introduction Debate of worth and rigor between qualitativeand quantitative research has a long history. Advocates of qualitative researchers often argue that human behavior cannot be transformed and measured into numbers, opposing this argument quantitative researchers posit that qualitative research is anecdotal, unscientific and opinionatedwith the lens of the investigator. Lacking the certainty of hard numbers is often expressed as a crisis of confidence in qualitativeresearch. Research as defined by many authors (Gay, 1996;Patton, 2001;Creswell, 2003)is the systematic application of scientific method to the problem under consideration. Therefore, without rigor research becomes fiction and loses its worth. The rigor can be ensured only by considering validity and reliability in all kind of research methods.

!

!

1

This paper discusses the issues and meaning of reliability and validity in qualitative research. It presents a recapitulation on the meaning of quantitative and qualitative research in section two. In the third section it discusses the issues arising in qualitative research. Section four discusses the concept of reliability and validity

/*

FPM Scholar, Xavier Institute of Management, Bhubaneswar, INDIA **FPM Scholar, Xavier Institute of Management, Bhubaneswar, INDIA

5744

Qualitative Inquiry

5745

Understanding Quantitative and Qualitative Research Paradigms

the researcher does not attempt to manipulate the phenomenon st and only try to unveil the ultimate truth. The researcher plex, holistic picture, analyzes words, reports detailed s of informants, and conducts the study in a natural setting: swell (1998) defines it as an inquiry process of understanding d on distinct methodological traditions of inquiry that explore a a1 or human problem.

Quantitative Research Paradigm

ues in Qualitative Research

for facilitating readers and examining what these two terms mean. ~t also discusses how reliability and validity can be tested in the qualitative research paradigm. Finally, the paper concludes with the discussion on the findings.

Quantitative research is based upon o b s e ~ a t i o n that s are converted into discrete units that can be compared to other units by using statistical analysis. Quantitative research utilizes experime quantitative measures to test hypotheses; and ge the outcomes of this test. They also emphasize the analysis of causal relationships between variables (Creswell, 20 McMillan and Schumacher, 2006). Researchers have argued that in case of quantitative research the emphasis mainly rests upon the facts and causes of behavior (Golafashani, 2003); information in quantitative research is in the form of numbers that can be quantified and summarized. For analyzing the numeric data mathematical processes are utilized and the final result is reported and expressed in statistical terminologies.

Qualitative Research Paradigm According to Denzin and Lincoln (1994) qualitative research is a field of inquiry in its own right. It crosscuts disciplines, fields, and subject matters. They examine complex, interconnected family of terms, concepts, and assumptions surrounding the qualitative ~aradigm in research. In their words qualitative research is multi-method in focus, involving an interpretive, naturalistic approach to its subject matter. This means that qualitative researchers study things in their natural settings, attempting to make sense of or interpret phenomena in terms of the meanings people bring to them. Qualitative research involves the use and collection of a variety of empirical materials, case studies, ~ e r s o n a lexperiences, introspectives, life story interviews, observational, historical, interactional, and visual texts-that describe routine and ~roblematicmoments and meaning in individuals' lives. In his study Golafashani (2003) described that qualitative research uses a naturalistic approach that seeks to understand phenomena in context-specific settings, such as real world setting

criticized regarding the assessment of Qualitative research is often cited as being too specific rticular social setting to be generalizablrb to a wider world. As quantitative research, qualitative research lacks any statistical ample size calculation. Qualitative research should be stood as an effort to seek depth rather than breadth. Since ative researchers are very close to their research settings and ects, issues of biasness and subjectivity in interpretation of the Its are often raised. ve and qualitative researchers need o attain this, a great deal of attention is ity and validity in all research methods. But, the major n the debate of reliability and validity in the ple Merriam, 1995; Morse et al., 2002). one of the major challenges in qualitative research.

alidity in.qualitativeresearch is generally assessed using the tests ofconstruct ty, external validity and reliability. Yin ( 2009,

iConstruct validity: identifying correct operational measures for the concepts being studied. tablish a causal relationship, whereby d to lead to other conditions

&ernal Validity: defining the domain to which the study's findings can be generalized Reliability: demonstrating that the operations of a study -such as the data collection procedures- can be repeated, with the same results

Establishing Reliability and Validity in Qualitative Inquiry

5747

Pandey and Patnaik

5746

In the context of qualitative research, many perspectives, terms and procedures have been developed to establish reliability and validity like internal validity, external validity, reliability, objectivity ( LeCom~te & Goetz,1982);credibility, transferability,dependability,conformability ( Lincoln & Guba, 1985); and credibility, authenticity, integrity, explicitness, vividness, creativity, thoroughness, congruence and sensitivity ( Whitternore, Chase & Mandle, 2001 as cited in Creswell, 2012). In fact, ( Morse et al., 2002) argue that reliability and validity have been subtly replaced by criteria and standards for evaluation of the overall significance, relevance, impact and utility of completed research in qualitative inquiry. The credibility of a qualitative research paradigm depends upon the ability and effort of the researcher. So far, whenever the researcher addressedthe concepts of reliability and validity in qualitative research they referred credibility. The question arises here that how to enhance credibility of the research. Lincoln & Guba (1985) in their seminal work posit that trustworthiness of a research study is important to evaluating its worth. Golafshani (2003)suggests that if the trustworthinesscan be maximized thenmore credible and defensible result may lead to generalizability. This concept was suggested by Stenbacka (2001)as the structure for both doing and documenting high quality qualitativeresearch. Therefore, the q u d t y of a research is related to generalizability of the result and thereby to the testing and increasing the trustworthiness of the research.

Ensuring Trustworthiness in Qualitative Research According to Lincoln and Guba (1985) ~rustworthinesiinvolves establishing: (i)Credibility Credibilityis similar to internal validity in positivist research, confidence in the.'truth' of the findings.

-

(ii)Transferability- showing that the findings have applicability in other contexts. Transferability is in preference to external validity/ generalizability in the positivist paradigm.

(iii) Dependability - Dependability is in preference to reliability in positivist approach, showing that the findings are consistent and could be repeated.

(iv) Conformability - Conformability is in preference to objectivity. Conformability can be seen as a degree of neutrality or the extent to which the findings of a study are shaped by the respondents and not r ~ c m r r h e hias. r motivation, or interest.

One of the key criteria addressed by positivist researchers is that of internal validity, in which they seek to ensure that their study measures or tests what is actually intended. Accordingto Merriam (1998),the qualitative investigator's equivalent concept, 1.e.credibility, deals with the question, "HOWcongruent are the findings with reality?" Lincoln and Guba ( 1985)argue that ensuring credibility is one of most important factors in establishing trustwokhiness. According to Tracy (2010), credibility refers to the trustworthiness, verisimilitude and plausibility of the research findings.

Techniques for Establishing Credibility Following techniques may be used by the researchers to ensure credibility in qualitative research:

Prolonged Engagement:Lincoln and Guba (1985),Erlandson and his colleagues (Erlandson, Harris, Skipper, & Allen, 1993) and Merriam (1995) are among the many other who recommend "prolonged engagement" between the investigator and the participants in order to gain an adequate understanding of an organization and to establish a relationship of trust between the investigator and the participants. This involves spendingadequate time observing various aspects of a setting, speaking with a range of people, and developing relationships and rapport with members of the organization or community. Persistent Observation:In words of Lincoln and Guba (1985) "If the purpose of prolonged engagement is to render the investigator open to the multiple influences - the mutual shapers and contextual factors - that impinge upon the phenomenon being studied, the purpose of persistent observation is to identlfythose characteristics and elements in the situation that are most relevant to the problem or issue being pursued and focusingon them in detail. Ifprolonged engagement provides scope, persistent observation provides depth Triangulation:Qualitative researchers generally use this technique to ensure that an account is rich, robust, comprehensive and welldeveloped. Triangulation involves using multiple data sources in an investigation to produce greater understanding (Merriam, 1995). Triangulation rests upon the belief that a single method can never adequately explain a phenomenon. Using multiple methods can help to facilitate a deeper understanding. The concept has been aptly captured by Denzin (1978)who argues that triangulation in qualitative research

5748

Pandey and Patnaik

assumes that if two or more sources of data, theoretical frameworks, types of data collected, or researchers converge on the same conclusion, then the conclusion is more credible ( as cited in Tracy, 2010). Four types of triangulation are identified by Denzin (1978)and Patton ( 1999):

-

Methods triangulation checking out the consistency of findings generated by different data collection methods. It is common to have qualitative and quantitative data in a study; these elucidate complementary aspects of the same phenomenon. ORen the points where these data diverge are of great interest to the qualitative researcher and provide the most insights.

-

Triangulation of sources examining the consistency of different data sources from within the same method. For example:At different points in time; in public and private settings; and comparing people with different viewpoints.

-

Analyst Triangulation using multiple analysts to review findings or using multiple observers and analysts. This can provide a check on selective perception and illuminate blind spots in an interpretive analysis. The goal is not to seek consensus, but to understand multiple ways of seeing the data.

-

Theorylperspective triangulation using multiple theoretical perspectives to examine and interpret the data.

Peer debriefing: "It is a process of exposing oneself to a disinterested peer in a manner paralleling an analytical session and for the purpose of exploring aspects of the inquiry that might otherwise remain only implicit within the inquirer's m i n d (Lincoln & Guba, 1985, p. 308) Purpose of debriefingis thorough analytical probing; a debriefer can help to uncover those biases which are often taken for granted, perspectives and assumptions on the researcher's part are also subject to challenge in peer debriefing. This process helps the researcher to become aware of hisher posture toward data and analysis

Negative or Deviant Case Analysis: Negative or Deviant case analysis is recommended by several qualitative investigators including Miles and Huberman (1994) and Creswell (1998). This involves searching and discussing elements of the data that do not support or appear to contradict patterns or explanations that are emerging from data analysis. Analysis of deviant cases may help in revising, broadening and confirming the patterns emerging from data analysis.


5749

Referential Adequacy: This involves identifying a portion of data to be archived, but not analyzing. The researcher then conducts the data analysis on the remaining data and develops preliminary findings. Finally, the researcher returns to this archived data and analyzes it as a way to test the validity of his or her findings. Member Checks: Lincoln and Guba (1985)consider member checks as the single most important provision that can be made to strengthen a study's credibility. This can be done both formally and informally as opportunities for member checks may arise during the normal course of observation and conversation. Participants may also be asked to read any transcripts of dialogues in which they have participated. Here the emphasis should be on whether the informants consider that their words match what they actually intended. Merriam (1995) further elaborates that it is not only about taking data collected from the study participants, but also the tentative interpretations of these data, back to the people from whom they were derived and asking if the interpretations are plausible, if they "ring true." Thick description: Tracy (2010)argues that one ofthe most important means for achieving credibility in qualitative research is thick description. This involves providing enough details to the readers. It requires that the researcher accounts for the complex specificity and circumstantiality of their data.

(ii) Transferability According to Merriam (1998) in quantitative inquiry, there is a concern of demonstrating that the findings of the work a t hand can be applied to a wider population. Qualitative inquiries are often speciiic and the findings are applicable for a particular environment or a small group ofindividuals, it is extremely difficult to demonstrate that the findings and conclusions are applicable to other situations and populations. The works of researchers like Cole and Gardner (1979) and Marchionini and Teague ( 1987)highlights the importance ofthe researcher's conveying to the reader the boundaries of the study. Lincoln and Guba suggest that it is the responsibility of the investigator to ensure that sufficient contextual information about the field work sites is provided to enable the transferability of such research inquiries.

Techniques for establishing transferability Following techniques may be used by the researchers to ensure transferability in qualitative research:

5750

Pandey and Patnaik


Thick Description: Thick description is described by Lincoln and Guba (1985) as a way of achieving a type of external validity. By describing a phenonienon in sufficient detail one can begin to evaluate the extent to which the conclusions drawn are transferable to other times, settings, situations, and people. The term thick descriptions was first used by Ryle (1949) and later by Geertz (1973)who applied it in ethnography. Thick description refers to the detailed account of field experiences in which the researcher makes explicit the patterns of cultural and social relationships and puts them in context (Holloway, 1997).In order to assess the extent to which'fhdings may be true of people in other settings, similar projects employing the same methods but conducted in different environments could well be of great value.

(iii) Dependability In addressing the issue of reliability, quantitative researchers employ techniques to show that, if the work were repeated, in the same context, with the same methods and with the same participants, similar results would be obtained. However the nature of the phenomena scrutinized by qualitative researchers renders such provisions probledatic in their work.

Techniques for establishing dependability Following techniques may be used by the researchers to ensure dependability in qualitative research:

Inquiry Audit: 1;incoln and Guba emphasize "inquiry audit" as one measure which may enhance the dependability of qualitative research. It involves having a researcher not involved in the research process to examine both the process and product of the research study (Hoepfl, 1997).Merriam (1995)uses the term 'audit tiial' as one of the techniques to establish dependability.According to Meniam (1995),in order for an audit to take place, the investigator must describe in detail how data were colle'cted, how categories were derived, and how decisions were made throughout the inquiry. The purpose is to evaluate the accuracy and evaluate whether or not the findings, interpretations and conclusions are supported by the data. It provides an opportunity for an outsider to challenge the process and findings of a study. Important feedback from an external researcher or expert can lead to additional data gathering and the development of stronger and better articulated findings.

5751

(iv) Conformability

'

The concept of conformability in the qualitative paradigm is comparable to the concern of objectivity in quantitative research. Steps must be taken by the investigator to ensure as far as possible that the findings are the result of the experiences and ideas of the informants and the participants, rather than the characteristics and preferences of the researcher. This will eliminate biases and favors from the findings obtained.

Techniques for establishing conformability Following techniques may be used by the researchers-to ensure conformability in qualitative research:

Audit Trails:Audit trails are performed to keep a record of what was done in an investigation. It is a transparent description of the research steps taken from the start of a-research project to the development and reporting of findings. Halpern (1983) provides a categorization for reporting information while developing an audit trail in the following order: Raw data -.including all raw data, written field notes, unobtrusive measures (documents) Data reduction and analysisproducts -including summaries such as condensed notes, unitized information and quantitative summaries and theoretical notes Data reconstruction and synthesis products - including structure of categories (themes, definitions, and relationships), findings and conclusions and a final report including connections to existing literatures and an integration of concepts, relationships, and interpretations Process notes - including methodological notes (procedures, designs, strategies, and rationales),trustworthiness notes (rdating to credibility, dependability and confirmability)and audit trail notes Materials relating to intentions and dispositions - including inquiry proposal, personal notes (reflexivenotes and motivations) and expectations (predictions and intentions) Instrument development information preliminary schedules, observation formats

- including pilot

forms,

5752

Pandey and Patnaik

Reflexivity: Reflexivity is an attitude of attending systematically to the context of knowledge construction, especially to the effect of the researcher, at every step of the research process.

"A researcher's background and position will affect what they choose to investigate, the angle of investigation, the methods judged most adequate for this purpose, the findings consideredmost appropriate, and the framing and communication of conclusions"(Malterud, 2001). The perspective or position of the researcher shapes all research quantitative, qualitative, even laboratory science. To foster reflexivity and reflexive research design followingthings can be taken into consideration:

Designing research that includes multiple investigators Including multiple investigators can help in fostering dialogue this can lead to the development of complementary as well as divergent understandings of a study situation and provide a context in which researchers' beliefs, values, perspectives and assumptions can be revealed and contested. Develop a reflexive journal (Lincoln and Guba) - Type of diary where researcher can make regular entries during the research process. Methodological decisions, the logistics of the study and reflection upon what is happening in terms of one's own values and interests should be entered in this diary.

Establishing Reliability and Validity i n Qualitative Inquiry

5753

these techniques could also be used to test more than one criteria of trustworthiness. A combination of these techniques can be used t~ serve the purpose of establishing rigor and acceptance of qualitative work among the investigators and critique of qualitative paradigm.,

REFERENCES Creswell,