SHOULD WE USE STUDENT SUBJECTS IN ...

CAN STUDENTS PASS THE TEST: SHOULD WE USE STUDENT SUBJECTS IN OPERATIONS MANAGEMENT RESEARCH Xiaowen Huang Department of Management Richard T. Farmer School of Business Miami University Oxford, Ohio 45056, USA [email protected] Thomas F. Gattiker Department of Information Technology and Supply Chain Management College of Business and Economics Boise State University 1910 University Dr. Boise ID 83725-1615 [email protected] Joshua L. Schwarz Department of Management Richard T. Farmer School of Business Miami University Oxford, Ohio 45056, USA [email protected]

ABSTRACT Student subject research is rarely used and published in the OM field. This study investigates whether there is a place for student subjects-based research in our discipline and, if so, to define when such research is appropriate. Three major issues are discussed: (i) validity trade-offs and methodological choices, (ii) the nature and relevance of dissimilarity between students and practitioners, and (iii) the empirical evidence of generalizability. Keywords: empirical research methodologies, student subjects INTRODUCTION Most empirical operations management (OM) scholarship is based on field research—either surveys or case studies using data collected from practitioners in the context of the organization in which they work. The OM discipline has relatively few extant studies utilizing students as research subjects. By student subjects research, we refer to research that places students in contrived settings, such as “simulation games” played in lab settings, or that exposes students to hypothetical scenarios and then collects data from those students, either by directly observing their behavior or by questioning them (with a post treatment survey or interview) about their experience. In this paper, the term student subjects research does not include research that

- 4931 -

collects data from students about their actual work experiences. For example, our definition would exclude a study using questionnaire data obtained from part-time MBAs on their experiences in their current or previous actual workplace. The absence of student subjects research in OM can be partially attributed to the fact that the majority of OM studies focus on investigating organizational-level practices. The individuals investigated in these studies serve as key informants, providing data about an organization (i.e., a company, SBU, or plant) or supply chain. Recently, however, there is a growing interest in behavioral research in the OM field [1]. Here the unit of analysis is usually the individual. In other words, such research seeks to explain variation from individual to individual in outcome variables such as wait perception bias [13], individual productivity [12], individual adoption of a technology [2], an individual’s trust in another person [6], or individual decision-making performance [5]. When the unit of analysis shifts from the organization to the individual, researchers are faced with the choice of whether to collect data from practitioners in situ or to utilize student subjects, and hence call for discussions on the appropriateness of using student subjects in OM research. In general OM reviewers and editors are less familiar with student subjects research. They tend to consider student subjects-based studies less rigorous or less valid than practitioner-based studies. Their concerns about the credibility of using student subjects often lead to routine rejection of student-based studies, which perhaps is another important reason why student subjects research is rarely published in OM journals. The student subjects issue is an important one. In OM, student subjects research largely goes hand in hand with, experimental research. Since experiments are an important element of the theory building process, de facto policies that have the practical effect of drastically reducing the number of experiments being conducted and published also have the effect of stunting theory development in our field. The aim of this paper is to explore whether there is a place for student subjects-based research in our discipline and, if so, to define the circumstances under which such research is appropriate. To better understand the power and limitations of student subjects research, we start with the criticism and the underlying assumptions commonly associates with the use of student subjects, and then discuss three major issues—i.e., i) validity trade-offs and methodological choices, ii) the nature and relevance of dissimilarity between students and practitioners, and iii) the empirical evidence of generalizability. Based on our discussion we conclude that there are limited conditions under which the use of student subjects is a desirable research strategy and indeed conditions under which student subjects are a better choice than practitioner subjects. CRITICISM OF THE USE OF STUDENT SUBJECTS A central question regarding the legitimacy of using student subjects is: Can the research findings obtained from student subjects be generalized to actual organizational settings? In other words, whether the findings obtained from student subjects have external validity? Critics of student subjects research argue that students are different from practitioners; and hence the dissimilarity between them precludes the validity of students subject research. This reasoning

- 4932 -

implies three assumptions. First, it assumes that external validity is the most important determinant of the value of all research. Second, it assumes that practitioners and students are indeed different. Third, it implies that any differences between students and practitioners are important to the research problems, and hence will influence the findings of the research. Below we scrutinize each of these assumptions by discussing three key issues related to the use of student subjects research. Validity Trade-offs and Methodological Choices Handfield and Melnyk [8] state that the scientific theory-building process contains multiple stages, each involving different activities and objectives. For instance, in the theory validation stage, causal and confirmatory research are required to verify the relationships suggested by earlier work. It is imperative to match the theory-building objective to the research methodology [8] because all methods are well suited to some objectives but not a good fit to other objectives [15]. According to Cook and Campbell [4], when the research objective is testing causality, internal validity is the most important consideration. Internal validity refers to the ability to infer that an observed relationship between two variables is causal—e.g. that an observed increase in Y is caused by a concomitant increase in X [4]. Internal validity can be contrasted with external validity. External validity refers to the extent to which research findings from one context or population generalize to other contexts or populations. External validity is maximized by research designs that utilizing subjects and settings which resemble the subjects and settings to which the results should generalize [4]. In OM, this typically means collecting data from practitioners by questioning them about phenomena that exist in their actual workplace or (less frequently) by directly observing actual events or by using archival data. On the other hand, internal validity is maximized by (1) Deliberate treatments, rather than reliance on naturally occurring differences across people or organizations; (2) Random assignment of subjects to treatments; (3) Pre-treatment and posttreatment measures; (4) Short elapsed time between the administration of treatments and measures of the (potential) effects of those treatments; and (5) holding confounding factors constant [4] . Elements 1 through 5 above are best executed via the experiment methodology, as opposed to surveys and case studies. For example, Carter and Stevens [3] list numerous threats to internal validity and enumerate how these threats manifest themselves in the survey and case study literature on Internet reverse auctions. Using one experiment [6] as an example, Table 1 shows how these threats can be eliminated by experiments that carefully incorporate elements 1 through 5 discussed earlier. Hence, it is clear that experiments in a controlled setting can eliminate most threats to internal validity. Unfortunately, under certain circumstances the associated costs and barriers can render this methodology infeasible. In particular, if practitioners are the desired research participants, barriers to recruitment are often too great to be overcome, especially with the resources allowed by university research budgets. For example, one significant barrier for conducting the Gattiker, Huang & Schwarz study with practitioners would be identifying a situation, such as a conference that attracts large numbers of both professional buyers and sellers. Travel and facility rental

- 4933 -

would be important costs. However, these would be dwarfed by costs of providing incentives sufficient to entice practitioners to give up several hours of time to participate. TABLE 1: ILLUSTRATION OF ELEMENTS OF EXPERIMENTAL METHDOLOGY Elements of Experimental Methodology

1. Deliberate treatments

2. Random assignment

3. Pre & Post treatment measures 4. Short time between treatment and post-treatment measure 5. Confounding factors held constant

Example (from Gattiker, T., Huang, X. & Schwarz, J. Negotiation, Email, and Internet Reverse Auctions: How Sourcing Mechanisms Deployed by Buyers Affect Supplier's Trust. Journal of Operations Management, 2007, 25(1), 184-202.) Subjects placed in either a bargaining channel (reverse auction, email negotiation, face-to-face negotiation) and a high or low level of commodity complexity– a 2x3 design, a total of 6 experimental conditions. Subjects randomly assigned role (buyer or seller); Buyer-seller dyads randomly assigned to a bargaining channel and a level of commodity complexity Dependent variable measured before the auction/negotiation treatment and after Less than one week All subjects were provided with written scenario describing the commodity to be bought/sold, the competitive environment, etc. All details in the scenarios were parallel across subjects except for the descriptions of the treatments (the commodity complexity and the bargaining channel to be used). Other factors such as incentives also held constant.

One way to avoid these costs is to use student subjects. Of course the advantages of student subjects must be weighed against any disadvantages. We evaluate the magnitude of potential disadvantages in the next sections. In particular we discuss whether differences between practitioners and students as research subjects matter, assuming they exist. We then “back up a step” and examine the extant evidence on whether such differences indeed exist. Do Differences Between Practitioner and Student Subjects Matter? An important assumption behind the criticism of student subjects research is based on the dissimilarity between practitioners and students. Many researchers contend that students are not representative of practitioners. However, to defend this argument, one must specify: i) in what respects students and practitioners are different; and ii) how relevant these dissimilarities are to the investigation of the research. This statement also ignores the possibility that there might be sufficient similarity between students and practitioners which allows students to be a representative sample of the intended population. Therefore, before accepting that students are

- 4934 -

not representative of the population, it is critical to discuss 1) what criteria should be used to decide whether students are representative of the population to which the research is intended to generalize; and 2) how the nature and relevance of dissimilarity between practitioners and students will affect the research findings. The answers to these questions come from the underlying theory investigated in the specific research. Good theory should clearly describe the nature of research phenomena under investigation and specify the domain of where the theory applies [5]. Bendoly et al. [1] argue that not all research phenomena rely on subject’s work experience (a major dissimilarity between students and practitioners). There are times when the research phenomena of interest deals with the “universalistic” theory intended to explain general behavioral issues among human beings [7][1]. In these cases, the contextual content of such research is minimal and, therefore, the unique social and culture differences among subjects are less relevant. After all, students and practitioners are both human beings, and, for these studies, students are also representative of the population under investigation—human beings. In other cases, the theories being investigated are most likely contextually-based. In such research, whether the findings obtained from students subject can be generalized to the field depends on the boundary conditions of the theory. These boundary conditions identify the essential conditions under which the theory being investigated applies. Students and practitioners could be dissimilar in various aspects, but as long as they have similarity in terms of essentials stated in the boundary conditions, the findings from student subjects research can be generalized to the practitioners [9]. In other words, the differences between students and practitioners don’t matter unless these differences are associated with the essential attributes identified by boundary conditions. In addition, the potential impact of these nonessential attributes can be minimized by a well designed and executed experiment through randomization. Do Differences Between Practitioner and Student Subjects Exist? Above we argue that when internal validity, as opposed to external validity, is the goal, using student subjects is a valid research approach –i.e., the researcher may trade off external validity for internal validity. We also discuss that not all dissimilarity between students and practitioners are relevant. The differences in nonessential attributes should not preclude the generalization of the results from students subjects to practitioners. But, ultimately, the question of generalization is an empirical one that can only be answered by comparing the results from different samples— i.e. students and practitioners. Looking at the empirical results in the OM field, all student-practitioner comparisons in our literature review have found results obtained from the two sources to be equivalent. Sterman [14, pp. 328, 338] collected beer game data from undergraduate, MBA and Ph.D. students at MIT's Sloan School, executives from a variety of firms participating in short courses on computer simulation and senior executives of a major computer firm, and then concluded that the results were “strikingly similar.” Min, Latour & Jones [11, p. 21] used an undergraduate student sample and an "industrial" sample consisting of members of several NAPM chapters. The researchers found that "The experiments did produce some minor differences between the student group and the industrial counterpart." However, the results are "essentially the same." Machuca & Barajas [10] collected decision making data from university students and executives.

- 4935 -

They conclude "The experiments did produce some minor differences between the student group and the industrial counterpart." But again, these authors argued that the results are "essentially the same." Croson & Donohue [5] compared students from an undergraduate core business course with professionals from a CSCMP chapter. They too found them to be substantively equivalent. Our literature review has turned up no studies finding significant differences between students and practitioners. CONCLUSION Wacker [15] found that less than one percent of studies in OM utilize the experiment methodology. Experiments were outnumbered by both case studies and surveys by more than ten to one. Noting the lack of experiments in the literature, Wacker [15, p. 380] concludes "In short, the evidence presented here suggests that operations management is not fully utilizing all research methods to verify old relationships and to build integrated theory." We would not argue that experiments or any other methodology are "best" or "most rigorous." Rather all methods have advantages and disadvantages. This includes trade-offs between external and internal validity. When testing causality is needed as a part of the theory building process, true experiments (those with random assignment) with high levels of control are the method of choice because of the high level of internal validity that they provide. Using student subjects is an important and appropriate component of many experimental designs. As we describe in this paper, there are limited conditions under which the use of student subjects is a desirable research strategy and indeed conditions under which student subjects are a better choice than practitioner subjects. The justification of students subject research can be made based on the nature of the research phenomena and the degree of similarity between students and practitioners in essential attributes identified by the boundary conditions of the underlying theory. REFERENCES References available upon request from Xiaowen Huang (email address: [email protected]).

- 4936 -