differential effects of learner effort and goal orientation

PERSONNEL PSYCHOLOGY 1998.51

DIFFERENTIAL EFFECTS OF LEARNER EFFORT AND GOAL ORIENTATION ON TWO LEARNING OUTCOMES

SANDRA L. FISHER Personnel Decisions Research Institutes, Inc. J. KEVIN FORD Michigan State University

Ti-aining research is beginning to examine the trainee as an active participant in the learning process. One component of this process involves the choices learners make about the effort they will apply to the learning task. Learner effort was examined in two ways: amount of effort and type of effort. Both amount and type of effort were measured in multiple ways. The results indicated that mastery orientation and time on task were the strongest predictors of performance on the knowledge learning outcome, while perceived mental workload and the use of an example during learning predicted performance on the application learning outcome. Implications for training practice and research are discussed. Given the substantial investment in time and resources devoted to training and education every year, psychologists in many disciplines have investigated influences on individual learning. Learning is typically defined as "a relatively permanent change in knowledge or skill produced by experience" (Weiss, 1990, p. 172). In organizational settings, we are often interested in knowledge and skill acquisition and retention as a function of some training experience (Baldwin & Ford, 1988; Goldstein, 1993). Much of training research has focused on the learner as a passive recipient rather than an active participant to various interventions or designs (Ford & Kraiger, 1995). The training literature has begun to examine the impact of increasing the level of participation of trainees in their own training. Most of this research has examined pretraining activities such as increasing trainee participation in the identification of training needs, the determination of the training content, the design of This article is based on the first author's master's thesis, conducted under the supervision of Kevin Ford at Michigan State University. An earlier version of this article was presented at the 11th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, CA. Correspondence and requests for reprints should be addressed to Sandra L. Fisher, Personnel Decisions Research Institutes, 1300 Wilson Blvd., Suite 1000, Arlington, VA, 22209 or E-mail: [email protected]. COPYRIGHT © 1998 PERSONNEL PSYCHOLOGY. INC.

397

398

PERSO^fNEL PSYCHOLOGY

the training program, and the choice as to whether to attend a training program (Baldwin & Magjuka, 1997; Noe & Wilk, 1993; Tannenbaum & Yukl, 1992). Instructional psychologists have been particularly interested in the active learner in the learning process. This literature focuses on the choices one makes within the learning process itself (Mattoon & Klein, 1993; Milheim & Martin, 1991; Steinberg, 1989). Two key choices that learners can make within a learning situation are how much effort to apply to learning new knowledge and skills, and the personal strategies that are most effective in acquiring knowledge and skills (Steinberg, 1989). These choices revolve around the amount and type of effort to apply in a learning situation. A strength of the instructional psychology literature is its emphasis on the learner within the learning situation. The research is limited, though, in its generalizability to training research in three ways. First, most of the research has focused on investigating the amount of effort or the type of effort independently of one another. Thus, we do not know much about the relationship between various dimensions of amount and type of effort. Second, the amount of effort has typically been operationalized as time on task. This measure of effort is deficient as it does not take into account whether the learner is actually processing training information while working on the task. Third, the research has tended to examine the impact of learner effort on a single type of learning outcome. Therefore, it is unclear whether the findings from a study examining declarative knowledge will generalize to studies examining a different learning outcome such as skill compilation. The purpose of this study is to more closely examine the learner within the learning process. TTie focus is on the amount and type of effort that learners may apply to acquiring two learning outcomes—declarative knowledge and skill application. The impact ofthe motivational orientation of the learner is also examined for its relationship to the amount and type of effort expended on the learning task. The next section provides an overview of the effort construct. This is followed by a discussion of the linkage of effort and learning and how the motivational orientation of the learner might impact effort and learning. The research hypotheses for the study are then presented. Leamer Effort The construct of learner effort has been examined across a wide range of literatures including human factors (Hancock & Caird, 1993; Hart & Staveland, 1988), education (Paas, 1992), cognitive science (Tulving & Thomson, 1973), instructional psychology (Schmeck, 1988), and

FISHER AND FORD

399

industrial-organizational psychology (Kanfer & Ackerman, 1989). An examination of these literatures reveals two conceptualizations of effort. First, many authors have used measures tapping the amount of effort used by a learner. This category includes time on task, attentional and perceived effort measures. The second category of effort variables can be characterized as type of effort. This category includes the more qualitatively oriented measures examining the choices or strategies used by learners within a learning context. Amount ofeffort. Amount of effort has typically been operationalized as time on task (e.g., Dweck, 1986; Steinberg, 1989). Although this measure of effort does not depend on self-report perceptions, time on task is a contaminated measure of learner effort. An individual may appear to be working on a task, or thinking about a task, but his or her attention may be focused elsewhere. Paas and colleagues (Paas, 1992; Paas & Van Merrienboer, 1994) have operationalized mental effort separately from time on task. According to Paas (1992), mental effort is the amount of capacity that is allocated to instructional demands, and is unrelated to time on task. Similarly, Hart and Staveland (1988) defined mental workload as the cost incurred by a person to achieve a given performance level. Mental workload is typically measured through learner reports of perceived effort. Kanfer and Ackerman (1989) address the attentional focus aspect of effort. They used a self-report measure of thought content which tapped into an individual's mental activity such as the extent to which individuals set goals for themselves, compared their performance to others, or daydreamed. The off-task attention measure was intended to capture mental resources that were not devoted to learning. Each measure of amount of effort discussed above, time on task, mental workload, and off-task attention, captures part of the construct space. The relationships among these measures should be explored, as well as their relationship with learning outcomes. It is expected that although these measures will be related, they will not be identical and that they will have differential impacts on learning outcomes. Type of effort. Type of effort is conceptualized as the cognitive processes involved in learning: encoding, organization, and retrieval. Each of these processes is a part of on-task effort. The accuracy of retrieval processes is dependent upon what information is stored, how the information is stored, and how that information matches the subsequent cues for retrieval (Lord & Maher, 1991). It is not just the information itself that is important, but also the organization of the information in memory.

400

PERSONNEL PSYCHOLOGY

Tulving and colleagues contend that the manner in which information is encoded "determines what retrieval cues are effective in providing access to what is stored" (Tulving & Thomson, 1973, p. 369). According to the encoding specificity principle, performance is facilitated to the degree that the test stimulus conditions match the presentation, or learning conditions (e.g., Tulving & Thomson, 1973; Weldon, Roediger, & Challis, 1989). Morris, Bransford, and Franks (1977) extended the encoding specificity principle with their notion of "transfer appropriate processing." They found that subjects performed better on a semantic meaning test when they had used a semantic acquisition task, and performed better on a rhyming test when they had used a rhyme-focused acquisition task. Transfer appropriate processing has also been supported in a classroom setting (e.g., Barnett, DiVesta, & Rogozinski, 1981; McKelvey & Lord, 1986). Encoding during learning can be examined through learning strategies. Gagn6, Briggs, and Wager (1992) define a learning strategy as "an internal process by which learners select and modify their ways of attending, learning, remembering, and thinking" (p. 66). Three categories of strategies directly related to learning are rehearsal, organizing, and elaboration. Rehearsal is a forced learning, repetitious procedure. Organizing strategies require the learner to find similarities and themes within the new material. Elaboration requires the learner to associate the new material with other, familiar material. This strategy differs from organizing in that elaborative learning links the new material to already familiar material, while in organizing, links are found within the new material. These strategies can be placed along a continuum of complexity, with rehearsal as the simplest strategy, and elaboration as the most complex. An individual can use any combination of these strategies in a given learning situation. Gagn6, (1984) suggests that these categories of learning strategies are differentially useful in the acquisition of various learning outcomes. The best strategy for a given situation depends on the learning objectives (Gagn6, 1984; Levin, 1986). For a knowledge outcome such as verbatim reproduction of text, the learner does not need to go through the stages of learning that are typical for a more complex, or application outcome (i.e., composition and automaticity). The learner could use the rehearsal strategy to acquire the declarative knowledge (Gagn6,1984). Learning outcomes such as skill application or procedural knowledge typically require the use of more complex strategies such as elaboration. Learners must make the linkages between individual pieces of information through the effortful process of proceduralization. Anderson and Fincham (1994) suggested that in particular learning contexts, this step may be unnecessary. Instead of first learning declarative information.

FISHER AND FORD

401

and then proceduralizing that information, learners may use specific examples to learn complex skills. Based on research in math and science, Anderson and Fincham contend that learners often use examples to solve structurally similar but novel problems. Thus, the use of examples is a fourth potential learning strategy. The Linkage of Effort to Learning Outcomes To understand learning, the amount of effort should be studied in conjunction with the type of effort used. Knowing that a trainee uses a certain learning strategy has a greater impact when we also know how much effort is put forth with that strategy. In addition, the choices made by a learner regarding the effort put forth during learning will be differentially effective depending on the learning outcomes. Kraiger, Ford, and Salas (1993) presented a framework which divides learning outcomes into three categories. Affective learning outcomes consist of attitudes and motivation which might be desired outcomes of training. Behavioral, or skill-based, outcomes consist of psychomotor skills one could learn in training. Cognitive outcomes consist of verbal knowledge, as well as higher-order knowledge organization and cognitive strategies. Cognitive outcomes have been further divided into categories. Anderson's ACT* theory (Anderson, 1982) presents the acquisition of knowledge and skills as a stage model. This theory posits that the dimensions of learning are arranged hierarchically. Declarative knowledge is knowledge about facts. Procedural knowledge is knowledge of how to do things. According to the ACT* theory, one cannot learn procedural skills without first having acquired the declarative basis for that skill. Bloom (1956) presented a similar hierarchical model of learning outcomes. Basic information is needed to develop higher-order knowledge structures. Instead of procedural knowledge. Bloom referred to application, where learners use a complex concept in a new situation. Application involves higher-level knowledge structures which require the use of relevant factual knowledge. However, one important quality of true procedural knowledge is that the learner need not consciously access the declarative knowledge to perform the behavior. Application is not an automatic process. Application is more relevant to initial learning processes, as it is more likely to be achieved than proceduralization in the brief time span of one class period or one training session. In the present study, declarative knowledge and application learning outcomes were examined. We expect that different patterns of learner motivation and effort will be associated with successful performance on the two types of learning outcomes.

402

PERSOhfNEL PSYCHOLOGY

Successful performance on the different learning outcomes depends on how the learner chooses to distribute cognitive resources during learning. According to Kanfer and Ackerman (1989), individuals possess a limited amount of cognitive resources which can be distributed among three types of behaviors: task related, self-regulation, and off task. In the early stages of learning, allocation of effort to self-regulation and off-task behaviors can deter learning, as learners may have insufficient resources allocated to task-related behaviors to learn effectively. A multidimensional approach to learner effort may allow greater understanding of this critical concept. Kanfer and Ackerman focused primarily on goal manipulations and ability as determinants of training performance. They referred to variations of task-related effort as increases or decreases, rather than changes in learning strategy or type of effort. In addition, learner resources must be distributed within the category of task-related behaviors. Learners must decide which specific strategies they will use. Certainly the amount of effort is important for learning. However, training researchers must also consider that qualitatively different learning strategies, such as rehearsal, organizing, elaboration, or use of examples, may be differentially effective depending on the type of learning outcome. Thus, the amount of effort and type of effort variables are suggested to be differentially effective in learning, and consequently lead to differential performance on learning outcomes. Kanfer and Ackerman (1989) suggest that the allocation of effort toward learning activities is driven by individual motivational processes. The motivational processes that affect resource allocation include goals, incentives, individual personality differences, and metacognitive knowledge. One motivational variable that has been found to affect how individuals approach learning tasks is goal orientation. The Impact of Goal Orientation on Learner Effort Goal orientation, defined as the broad goals held by an individual as he or she faces a learning task, has been demonstrated to affect how individuals learn (Dweck, 1986; Dweck & Leggett, 1988). Goal orientation approaches can be summarized into two categories: task or mastery and ego/social or performance orientation (Farr, Hofmann, & Ringenbach, 1993; Meece, Blumenfeld, & Hoyle, 1988). Mastery orientation is a dedication to increasing one's competence on a task. These learners seek to improve their ability on the task. With a performance orientation, learners focus on task performance and comparisons with others. These learners seek to prove their ability on the task to others (Dweck).

FISHER AhfD FORD

403

Kanfer and Ackerman (1989) suggest that motivational variables drive the allocation of attentional effort in skill acquisition, and direct the allocation of effort within the learning task. Goal orientation may also serve these functions. Learners with a high mastery orientation will direct attention to the task, and learn for the sake of learning, and thus will devote greater effort to learning (Dweck, 1986; Button, Mathieu, & Zajac, 1996). Learners with a high performance orientation will direct attention toward performing well on learning indicators and thus devote less effort to the task, because they also devote resources to ego management. To summarize, goal orientation is a motivational variable expected to affect the allocation of effort during learning. Amount of effort is defined as the amount of cognitive, or attentional, resources devoted to the learning task. Type of effort is defined as the form of cognitive processing used during the encoding stage of the learning task. It is suggested that different types of effort will lead to successful performance on different types of learning outcome measures. Hypotheses

This study examines the impact of goal orientation on the amount and type of effort displayed in a learning context and the impact of various effort variables on two types of learning outcomes. Goal orientation can be inastery or performance based. The amount of effort extended can be measured as time on task, mental workload, and off-task attention. The type of effort expended can include rehearsal, organizing, and elaboration strategies, as well as the use of examples. The examination of various components of effort allows for greater complexity in the hypothesized relationships. For example, both of the on-task effort measures, time and mental workload, should have a positive relationship with mastery orientation. However, because time is a contaminated measure of effort, that relationship should be smaller. It is expected that performance orientation will have negative relationships with both time and mental workload. However, performance orientation should have a stronger relationship with time than with mental workload, because time on task is an observable variable. Others cannot tell how hard a person is working mentally; they can only observe how long that person works. To protect the ego, the performance-oriented individual should minimize time on task (Dweck & Leggett, 1988). Regarding off-task attention, high mastery individuals will focus their attention on the learning task at hand, while performance-oriented individuals will be concerned with other issues. Thus, the following hypotheses were proposed:

404


Hypothesis la: Mastery orientation will have a positive effect on mental workload while performance orientation will have a negative effect on mental workload. Hypothesis Ib: Performance orientation will have a negative effect on time spent learning. Hypothesis la Mastery orientation will have a negative effect on off-task attention. Performance orientation will have a positive effect on off-task attention. Meece et al., (1988) have suggested that learning motivation can directly influence the selection of learning strategies. The learner who is motivated primarily by mastery will process deeply because of the overriding motivation to really learn the material. Schmeck (1988) suggests that mastery learners are more likely to use conceptualizing and elaboration strategies. Performance orientation is associated with a belief that effort and ability are inversely related. That is, learners with a high performance orientation tend to believe that if one must work hard at a task, then one must not have high ability (Dweck & Leggett, 1988). Because performance-oriented learners strive to prove their ability to others, they will use rehearsal strategies that minimize the amount of effort required. Hypothesis 2a: Mastery orientation will be positively related to the use of organizing, elaboration, and examples strategies, but not related to the use of rehearsal strategies. Hypothesis 2b: Performance orientation will be positively related to the use of rehearsal strategies, but not related to the use of the other three strategies. The strategies used by the learner should affect performance on the learning outcomes. According to the principle of encoding specificity (Tulving & Thomson, 1973), learners will be able to retrieve information that was encoded in a manner consistent with the retrieval method required by the testing situation. Information that was stored through memorization should be easier to retrieve in a situation that requires only recall of individual facts. Similarly, encoding problem-solving information in a complex, interrelated manner will help the learner apply that information to solve future problems. This type of practice is expected to be irrelevant to performance on the knowledge outcome. Thus, the following hypotheses were proposed: Hypothesis 3a: The use of rehearsal strategies will be positively associated with performance on the declarative knowledge outcome, and unrelated to performance on the application outcome.

FISHER AND FORD

405

Hypothesis 3b: The use of organizing and elaboration strategies will be positively associated with performance on the application outcome, and unrelated to performance on the knowledge outcome. Hypothesis 3c: The use of examples learning strategy will be positively associated with performance on the application outcome, and not associated with performance on the knowledge outcome. Regarding amount of effort, research indicates that more time on the task and greater attention to the task lead to higher performance on a learning outcome (e.g., d'Ydewalle & Swerts, 1980). However, as time on task is suggested to be a contaminated measure of effort, it should impact performance less than mental workload and off-task attention. Hypothesis 4a: Performance on both learning outcomes will be more positively associated with mental workload than with time on task. Hypothesis 4b: Performance on both learning outcomes will be more strongly associated with off-task attention than with time on task. In summary, we suggest that the individual difference variable,, goal orientation, will affect both the type and amount of effort used in learning. Performance on two different learning outcomes will then be affected by the amount and type of effort. Cognitive ability has not been specifically addressed in the above hypotheses. However, it could affect several different relationships in the model. Thus, cognitive ability was covaried out of the regression analyses. We also examined, where applicable, the extent to which effort mediates the relationship between goal orientation and learning outcome performance. In addition to testing the hypotheses, we were also interested in examining the amount of variance accounted for in the two learning outcome measures. Methods Participants The participants in this study were 121 undergraduate students recruited from the Psychology Department subject pool at Michigan State University. Students earned partial course credit for participating in the study. Leaming Task The learning materials were based on the multiple cue probability learning (MCPL) task used by Earley, Connoly, and Ekegren (1989).

406


Participants were asked to learn the prediction method (multiple regression) prior to the learning outcome tests. The task required participants to read a one page fictional description of how investment counselors make stock price predictions for their clients. The second page of learning materials detailed how multiple regression could be used to predict stock prices. This task has desirable characteristics for testing the hypotheses in this study. First, the individuals who participated in the study were unlikely to be familiar with the task and therefore be at relatively early stages of learning. Second, the task allowed for free variation in the amount and type of effort used in the learning portion of the study. Participants were run individually to avoid contamination of the time spent learning. Independent Variables

Cognitive ability was measured with the 50-item Wonderlic Personnel Test. The Wonderlic is an individually administered pencil-andpaper test with a 12-minute time limit. Test-retest reliability estimates for the Wonderlic range from .82 - .94, and the internal consistency reliability (KR-20) is estimated at .88 (User's Manual, 1992). Goal orientation was measured with two 8-item Likert-type measures of mastery and performance goal orientations developed by Button et al. (1996). Similar to Button et al., we treated goal orientation as a stable trait that varies between individuals. The mastery scale had an internal consistency reliability estimate of .73. A sample item is: The opportunity to learn new things is important to me. The performance goal scale had an internal consistency reliability of .82. A sample item for this scale is: I like to work on tasks that I have done well on in the past. The amount of effort devoted to the learning task by the participants was measured in three ways. First, a measure of time spent learning (in minutes) was taken. Second, a 13-item measure adapted from Kanfer and Ackerman (1989) and Kanfer, Ackerman, Murtha, Dugdale, and Nelson (1994) was used to measure the amount of off-task mental effort. A sample item from this scale is: I took "mental breaks" while learning. Using a 5-point Likert-type scale {strongly agree to strongly disagree), the internal consistency reliability was .87. Mental workload was measured with a 6-item scale adapted from the NASA-TLX scale (Hart & Staveland, 1988). Items regarding physical effort, and items which were conceptually very similar to the off-task attention scale were discarded from the original 10 items. A 5-point Likert scale was used, which had an internal consistency reliability of

FISHER AND FORD

407

.87. A sample item from this scale is: I had to work very hard to learn the stock price prediction materials. The type of effort used in the learning task by participants was measured in two ways. First, participants responded to a 17-item questionnaire concerning the learning strategies (rehearsal, organizing, and elaboration) they used during the experiment. This scale was adapted from the Inventory of Learning Processes (ILP) created by Schmeck (1983). The rehearsal subscale contained 5 items representing processes such as verbal or mental repetition, with a focus on specific details (e.g., I tried to remember exact words or phrases used in the materials). The organizing subscale contained 6 items measuring activities such as organizing material into a chart or diagram, or making lists (e.g., I made lists of associated ideas). The elaboration subscale contained 6 items representing mental activities such as paraphrasing and generating questions with answers (e.g., I created my own examples). Coefficient alpha for these scales was .73, .80, and .71, respectively. A 5-point Likert scale was used. To capture the use of examples learning strategy, we coded the work done on the sample problem included in the learning materials. This measure captured active practice of the learning materials, combining and using the information, and rules needed for successful performance on the application test. Therefore, working the sample problem is easily distinguished from the more conceptual strategies of rehearsal, elaboration, and organizing. Correct completion of the sample problem on the last page of the learning materials was scored as a 2. Attempted but incorrect problems were scored as a 1, and unattempted problems were scored 0. Dependent Variables

The Knowledge Learning Outcome was an 18-item multiple-choice test, with five options per item. The items were developed from the learning materials, and focus on facts found in the text of those materials. A sample item from this measure is: The companies list their stocks on the following stock exchange: (a) NYSE, (b) OTC, (c) NASDAQ, (d) CBOT, and (e) AMEX. Coefficient alpha for this test was .65. The Application Learning Outcome required participants to predict stock prices of 10 fictional companies, using performance data for three divisions of each company. There were three rules for determining the values for beta weights for each term. These rules required participants to choose between three possible beta weights for each term, depending on the value of the performance term. Participants were provided with data for the quarterly performance of three or four divisions of the

408

PERSO^fNEL PSYCHOLOGY

company. Participants then applied multiple regression procedures to estimate the future price of the stock. The problems varied in difficulty, requiring use of one to three rules. Coefficient alpha was .89. The application outcome was scored as the number of correct responses. Procedure

Participants first completed the Wonderlic Personnel Test, and the Learning Orientation questionnaire. Participants were then placed in rooms individually, and were instructed to take as much time as they needed to learn the Stock Prediction Task materials. Participants were also instructed that they were free to choose whatever study strategies they desired. One sample problem was provided, but the instructions provided no direction for approaching the learning task. Finally, participants were instructed that when they were done learning the materials, they would be asked to demonstrate their learning on a short test. These instructions were intended to increase participants' motivation to learn without significantly affecting performance orientation. Upon completion ofthe learning portion, subjects completed the effort measures. Completion of the effort measures at this time allowed the participants to most easily and accurately report their thoughts and mental processes during the learning of the stock prediction materials. In addition, participants' reactions to the knowledge and application outcome measures could not affect their motivation to respond accurately to the effort measures. Finally, participants completed the learning outcome measures. Analytic Strategy

Hierarchical regression analyses were conducted to test the impact of goal orientation on the amount and type of effort. Cognitive ability was entered first as a covariate and the two goal orientation variables were entered on the second step. The mediator relationships were also tested with hierarchical regression. As the learning orientations and application outcome were not correlated, no mediation model was tested for that outcome variable. For the knowledge outcome, cognitive ability was entered on the first step, followed by the effort variables, followed by the goal orientation variables. The effort variables were entered in blocks, one block for amount and one block for type. In each hierarchical regression, the impact of the variables entered at each step was determined by examining the significance of the increase in I^. The final beta weights are presented in each case to investigate the relative impact of each variable. There were 121 subjects in all analyses.

FISHER AND FORD

409

Results Scales and Intercorrelations

A common factor analysis of the goal orientation measure (principal factors extraction, oblimin rotation) supported the 2-factor structure (mastery and performance) reported by Button et al. (1996). An additional factor analysis using the same methods supported the separation ofthe two amount of effort variables—off-task attention and mental workload. Correlations among all independent and dependent variables are presented in Table 1. All scales are coded such that a high score reflects a greater amount of that construct. Internal consistency reliabilities (coefficient alpha) for all scales appear on the diagonal of the matrix. The covariate, cognitive ability, was negatively correlated with time spent learning, and positively related to mental workload, but was unrelated to off-task attention. It was positively correlated with performance on both the knowledge test (r = .27) and the application test (r = .23). Cognitive ability was not related to mastery or performance orientation, nor related to the use of rehearsal, organization or elaboration learning strategies, but was positively related to working the sample problem ('• = •21). The amount of effort variables showed the expected pattern of correlations. Time on task was uncorrelated with mental workload and offtask attention, but mental workload and off-task attention were negatively correlated (r = -.48). The learning strategies were hierarchically related by complexity. Elaboration, the most complex strategy, was unrelated to rehearsal, the least complex strategy. Organizing, the strategy associated with a moderate degree of complexity, was positively related to both rehearsal (r = .20) and elaboration (r = .40). Working the sample problem was unrelated to the other three strategies. Three of the four learning strategies were associated with greater amounts of effort. Rehearsal, the least complex strategy, was not related to any of the amount of effort measures. Organizing was positively correlated with time on task and negatively related with off-task attention. Elaboration was also negatively correlated with off-task attention, and positively correlated with mental workload. Working the sample problem was positively correlated with time on task and mental workload. The results indicate that the use of all but the simplest learning strategies required increased learner effort.


410

I -- 9 n

•S

s a.

3 I

i

11

II Is £ a.

II a >

Ii

FISHER AND FORD

411

Regression Analyses

Regression analyses with the amount of effort measures as dependent variables were performed (see Table 2). Hypothesis la was partially supported as mastery orientation was a significant predictor of mental workload {b = .19, p < .05) while performance orientation was not related. Hypothesis lb was not supported as performance orientation and time were unrelated. Mastery orientation was negatively correlated with off-task attention (r = -.17), although the final beta weight was significant only at p < .10. In addition, performance orientation was a significant predictor of off-task attention (b = .33, p < .01). Thus, Hypothesis lc was generally supported. The regression analyses with the type of effort measures as outcome variables (see Table 3) indicated that mastery orientation (b =.28, p < .01) and performance orientation (b = -.18, p < .01) significantly predicted use of the elaboration strategy, with the beta weights in opposite directions. Performance orientation positively predicted the use of rehearsal strategies (b = .24, p < .01). Neither of the goal orientations was related to the use of the organizing strategy, or working the sample problem. Thesefindingsprovide partial support for Hypothesis 2. Table 4 presents the analyses which examined the effects of type of effort on the learning outcomes. The use of the rehearsal strategy was not associated with performance on either learning outcome test. Thus, Hypothesis 3a was partially supported. The use of elaboration and organizing strategies did not predict performance on either learning outcome; Hypothesis 3b was not supported. Working the sample problem significantly predicted performance on the application outcome (6 = .29, p < .01), and was unrelated to performance on the knowledge outcome. Thisfindingsupports Hypothesis 3c. Regarding the fourth hypothesis, amount of effort was generally predictive of test performance. Mental workload significantly predicted performance on both learning outcomes. Off-task attention was negatively related to performance on both learning outcomes. However, time on task was also a significant predictor of performance on the knowledge outcome {b = .30, p < .01). To investigate the relative strength of the relationships of the effort variable with performance on learning outcomes, the significance of the difference between dependent correlations was examined (Cohen & Cohen, 1983). The pairs of correlations were significantly different for the application test; mental workload was significantly more related to performance than was time on task, f(118) = 2.39, p = .05, and the correlation between off-task attention and performance was stronger than that between time and performance,

412

PERSONNEL PSYCHOLOGY TABLE 2 Regression Analysis Results on Amount of Effort Variable

Off-task attention Cognitive ability Performance orientation Mastery orientation Time Cognitive ability Performance orientation Mastery orientation Mental workload Cognitive ability Performance orientation Mastery orientation

Beta

R

.01

.02 .34 .37

.00 .12 .14

.00

.05 .06 .07

.05*'

.12

.22 .24 .27

.16' -.10 .18**

.17 .22 .29

.03 .05 .08

.03*'

.33"* -.15 -.24" -.07

Note: Final beta weights are presented. AT = 121

fl' Change

* p< .10

.12"' .02*

.01 .02

** p< .05

.01 .03** *** p< .01

= 2.69, p = .05. There were no significant differences between the correlations for the knowledge test. These results support Hypothesis 4a but do not support Hypothesis 4b. The mediation effects were tested only for the knowledge outcome, as the relationship between goal orientation and the application outcome was non-significant. A separate mediation test was conducted for each effort variable. Cognitive ability was entered in the first step in all equations to control for individual differences. The effort variable was entered on the second step, and the goal orientation measures were entered on the third step. In each case, the goal orientation variables added significant additional variance to the regression equation. However, the increase in i?^ for goal orientation was less in each case than the change in R'^ when goal orientation was entered before the effort variable. When goal orientation was entered in the second step, after cognitive ability but before the effort variable, the change in R^ was .137. When goal orientation was entered in the third step, after an effort variable, the change in R^ associated with goal orientation ranged from .08 - .118. Thus, some of the variance in the knowledge outcome associated with goal orientation is mediated by the effort variables. To further investigate the relationships among the variables of interest, analyses were performed examining the direct effects of cognitive ability, goal orientation, and effort on the learning outcomes. Performance on the learning outcomes was affected by the amount and type of effort used by learners. Cognitive ability, goal orientation, and learner effort explained 32% of the variance in the application outcome, and

FISHER AND FORD

413

TABLE 3

Regression Anaiysis Results on Type of Effort Variable Rehearsal Cognitive ability Performance orientation Mastery orientation Organizing Cognitive ability Performance orientation Mastery orientation Elaboration Cognitive ability Performance orientation Mastery orientation Worked sample Cognitive ability Performance orientation Mastery orientation Note: N = 121

Beta

R

-.14 .24"*

.11 .26 .30

.01 .07 .07

.01

.05 .15

.08 .09 .18

.01 .01 .03

.01 .00

-.01 -.18" .28"*

.00 .20 .34

.00 .04 .11

.00

.21"

.21 .21 .22

.04 .04 .05

.04"

.13 -.11

.01 -.08

Final beta weights are presented. * p< .10

R^ Change

" p< .05

.06"'

.02

.02*

.04" .07"*

.00 .01 * " p< .01

40% of the variance in the knowledge outcome (see Table 4). Moreover, the variance in learning outcomes was explained by different patterns of predictor variables. Cognitive ability, mastery orientation, and time on task are the most important predictors of performance on the knowledge outcome. Working the sample problem and mental workload are the best predictors of performance on the application outcome. Thus, although the mediated regression was not wholly supported, the data provide support for the effects of goal orientation and effort on the learning outcomes. Discussion

The purpose of the present study was to examine the role of both amount of effort and type of effort in learning, and individual motivational influences on effort. The study provided partial support for the direct effects of goal orientation on effort. Mastery orientation was associated with greater effort, and the use of more complex learning strategies. Performance orientation was associated with less on-task effort, and less frequent use of complex learning strategies. The results also point to the amount of effort associated with different learning strategies. The use of organizing strategies was associated with greater time spent learning, and less off-task attention. Elaboration

414

PERSONNEL PSYCHOLOGY TABLE 4 Hierarchical Regression Analysis Results on Learning Outcomes

Dependent variable Step 1: Cognitive ability Step 2: Mastery orientation Performance orientation Step 3: Organizing Elaboration Rehearsal Sample Step 4: Workload Time Off-task attention Note: N = 121

Application outcome Beta R^ SR^ .13

.05 .06

.05* .01

-.03 .03

.33"

.07 .21

.07" .14"

.27

.08

.40

.13"

.18* -.07 .23

.16"

-.07 .13 -.09 .30"

.06 .05 .06 -.08 .32

.21* .06 -.17

Knowledge outcome Beta R^ SR'

.09" .16 .30" -.18*

Final beta weights are presented. ' p< .05 " p< .01

was associated with less off-task attention, and greater perceived workload. Rehearsal, or simple memorization, appeared to be the strategy requiring the least amount of effort. The most frequently used strategy in the study was rehearsal. Ironically, rehearsal was not related to performance on either learning outcome. Participants may not have known how to use strategies other than rehearsal (Steinberg, 1989). Rehearsal was expected to positively affect the knowledge outcome. However, because rehearsal was consistently high across participants, amount of effort accounted for the variance in the knowledge learning outcome. This effect suggests that performance oriented learners, who tended to apply less effort to the learning task, would perform less well on traditional, knowledge-based learning outcomes. In practice, these learners might be encouraged to devote more effort to the learning task. Further, there is some evidence that goal orientation could be modified (Boyle & Klimoski, 1995). Consequently, inducing a mastery orientation is another intervention which could assist performance oriented individuals in learning situations which call for increased effort. The relationships among the measures of amount of effort reinforce the need to measure this construct in multiple ways. As suggested by Paas (1992), time on task was unrelated (r = .02) to the self-report measure of mental workload. Off-task attention and perceived mental workload were inversely related (r= -.48). Participants who were thinking of

FISHER AND FORD

415

things other than the task while learning felt the learning required less effort. Alternatively, those who felt the task required less effort were more likely to think of other things. Off-task attention and time on task were minimally related (r = -.15). However, all three measures of amount of effort were directly related to performance on the outcome measures. Regardless of the learning outcome, on-task effort is important, and focusing on external issues detracts from learning. Participants who felt they had devoted a great deal of mental effort to the task tended to perform well on the outcome tests. This result is consistent with the position of Kanfer and Ackerman (1989), that mental effort, or the devotion of attentional resources, is related to learning and task performance. The current study also highlighted the difference between two different types of learning outcomes. The knowledge and application learning outcomes were based on the same materials. However, performance on the two types of learning outcomes was explained by different variables. Performance on the knowledge test was primarily a function of cognitive ability and time on task. Lesser, but still significant predictors of knowledge test performance were mastery orientation and off-task attention. None ofthe learning strategies predicted performance on the knowledge test. Conversely, the primary predictor of performance on the application test was practicing the work sample problem. Working the sample problem had no effect on knowledge test performance. The application outcome was not directly affected by goal orientation. Active practice on the required behavior (i.e., the use of examples) while learning was the most vital factor associated with improved post-learning performance. This result concurs with Anderson's recent data on the proceduralization of skills (Anderson & Fincham, 1994), which suggests that examples can provide a direct linkage to proceduralization, without necessarily learning a declarative representation of the concept. Limitations This study has several potential issues that could limit the generalizability of results. First, the sample consisted of undergraduate psychology students. We considered this an appropriate sample for our specific research questions which focused on individual learning processes. As with other recent studies on motivational processes and learning (e.g., Kanfer & Ackerman, 1989), the control provided by the laboratory environment allowed us to examined more carefully the effort construct. For example, the student sample and novel task lead to variations in effort that allowed us to test the study hypotheses. However, there may be important differences between these students and trainees in a work

416


organization, including differences in preferred learning strategies, that would lead to further understanding of the relationships between the amount and type of effort and their impacts on learning that need to be explored. In addition, the data collection method calls for consideration of common method variance and other response biases. Although the measurement was carefully sequenced to minimize these concerns, the fact remains that nearly all of the data were collected from participants using paper-and-pencil instruments. The accuracy of these types of measures is a constant concern in psychological research. Further, the use of selfreport measures may have allowed for possible response distortion. For example, individuals high in performance orientation may have intentionally distorted their effort ratings in order to appear more competent to the researcher. The effects of such possible distortion appear minimal. For example, the correlation between mental workload and performance orientation was non-significant. Had high performance oriented participants been systematically distorting their responses in the manner suggested, this correlation would have been stronger. Regardless of goal orientation, how self-reflective a person is may impact how accurately he or she can report on the application of mental resources. This issue of ability to accurately self-report was not studied. Research Directions and Practical Implications

Although many of the expected results were found, some findings require additional research attention. For example, although it was expected that the two learning outcome measures, knowledge and application, were distinct learning outcomes, the two measures were positively correlated. The two tests did cover the same general content area, and both were paper and pencil tests. In support of the distinction between the tests, the patterns of correlations between the tests and other variables were quite different. The knowledge test was positively correlated with task/mastery orientation, and negatively correlated with performance orientation. The application test was not significantly correlated with either orientation. The nested nature of types of learning outcomes makes it difficult to separate the effects of the knowledge and application measure. Qearly distinct testing in future research may help to separate the effects. Another measurement issue that warrants further study is the validity of the effort measures which were developed for this study. The Learning Strategy measures, for example, were based on Schmeck's (1983) Inventory of Learning Processes. Future work should continue to study the construct validity of these measures.

FISHER AND FORD

417

The research design also did not allow for clearly understanding why people chose the learning strategies that they did, or whether learners change strategies over time. Such research would require a focus on metacognitive issues. Metacognition is the awareness and control of one's own cognition and learning strategies (Nelson & Narens, 1990). Participants in this study tended to use a rehearsal strategy. We need to examine why individuals choose that strategy over others. Probed verbal protocols have been found to be an appropriate methodology for eliciting underlying strategies, for investigating the choices that individuals make, and for analyzing changes in strategies over time (e.g., Ford, Schmitt, Schectman, Hults, & Doherty, 1989). From a practical perspective, the results of this study suggest interventions one could use to improve learner performance on a range of learning outcomes. First, one could provide training on metacognition to improve learners' ability to select appropriate learning strategies for a given task. Consistent with our results, Steinberg (1989) suggested that college students may have already fixated on rehearsal as the default learning strategy for most learning tasks. College-educated employees in training courses may display the same tendency. Training in metacognitive skills may increase learner sophistication in the selection of strategies, as well as improving learners' allocation of effort to learning (Ford & Kraiger, 1995). This intervention assumes that learners know how to use a variety of strategies, but have difficulty choosing a relevant strategy. Learners may not know how to use strategies other than rehearsal. Thus, a second intervention is to teach trainees or students a number of different learning strategies (Schmeck, 1988). Finally, the long-term effects of the amount and type of effort used during learning must be considered. It has been suggested that the same cognitive processes are not involved in successful training performance and successful transfer of training (Schmidt & Bjork, 1992). For example, trainees who learn feedback giving skills by memorizing a list of guidelines may perform well on a multiple-choice test at the end of the training session, but may not be able to apply those guidelines successfully in a real work situation. An extension of the current study could consider the difference between the learning outcome at the end of training, and the required performance conditions present in the transfer environment. It follows that cognitive processes required for a test at the end of training should be as similar as possible to those processes required at transfer. Future research on learning motivation and learner effort must address transfer of learning as well as initial learning. REFERENCES Anderson JR. (1982). Acquisition of cognitive skill. Psychological Review, 89,369-406.

418


Anderson JR, Fincham JM, (1994), Acquisition of procedural skills from examples, Journai of Experimentai Psychology: Learning, Memory and Cognition, 20,1322-1340, Baldwin TT, Ford JK, (1988), TVansfer of training: A review and directions for future research, PERSONNEL PSYCHOLOGY, 47, 63-105,

Baldwin TT, Magjuka RJ, (1997), Organizational context and training effectiveness. In Ford JK & Associates (Eds,), Improving training effectiveness in work organizations (pp. 99-128). Mahwah, NJ: Lawrence Earlbaum Associates, Barnett JE, DiVesta FJ, Rogozinski JT. (1981). What is learned in note-taking? Joumai of Educational Psychoiogy, 73, 181-192. Bloom BS. (1956). Taxonomy of educationai objectives: The classification of educational goals. London: Longmans, Green and Co, Boyle KA, Klimoski RJ. (1995, May). The role of goal orientation in a training context. Paper presented at the Tenth Annual Conferences of the Society for Industrial and Organizational Psychology, Inc., Orlando. Button SB, Mathieu JE, Zajac DM. (1996). Goal orientation in organizational research: A conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67 (1), 26-48. Cohen J, Cohen P. (1983). Appiied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates. Dweck CS. (1986). Motivational processes affecting learning, American Psychologist, 41, 1040-1048. Dweck CS, Leggett EL. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95,256-273. d'Ydewalle G, Swerts A. (1980). Motivational variables in knowledge acquisition: Test expectancies. In Glaser R, Lompscher J (Eds.), Cognitive and motivational aspects of instruction (pp. 166-187). New York: North-Holland. Earley PC, Connoly T, Ekegren G. (1989). Goals, strategy development, and task performance: Some limits on the efficacy of goal setting. Joumai of Applied Psychology, 74, 24-33. Farr JL, Hofmann DA, Ringenbach KL. (1993). Goal orientation and action control theory: Implications for industrial and organizational psychology. In Cooper CI, KobeTtsonlT (Eds.), Intemational review of industrial and organizational psychology (Vol. 8, pp. 193-232), New York: Wiley, Ford JK, Schmitt N, Schectman SL, Hults BM, Doherty M, (1989), Process tracing methods: Contributions, problems, and neglected research questions. Organizational Behavior and Human Decision Processes, 43, 75-117, Ford JK, Kraiger K, (1995), The application of cognitive constructs and principles to the instructional systems model of training: Implications for needs assessment, design, and transfer. In Cooper CI, Robertson IT (Eds.), Intemational review of industrial and organizational psychology (Vol. 10, pp. 1-48). New York: Wiley. Gagni RM. (1984). Learning outcomes and their effects. American Psychologist, 39, 371385. Gagn6 RM, Briggs U , Wager WW. (1992). Principles of instructional design. Philadelphia: Harcourt Brace Jovanovich. Goldstein IL. (1993). Training in work organizations. In Dunnette MD, Hough LM (Eds.), Handbook of industrial and organizationai psychology (Vol. 2, pp. 507-620). Palo Alto, CA: Consulting Psychologists Press, Hancock PA, Caird JK, (1993), Experimental evaluation of a model of mental workload. Human Factors, 35, 413-429, Hart SG, Staveland LE, (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Hancock PA, Meshkati N (Eds.), Human

FISHER A N D F O R D

419

mental workload (pp. 77-106). New York: Elsevier Science Publishers B.V (NorthHolland). Kanfer R, Ackerman PL. (1989). Motivation and cognitive abilities: An integrative/ aptitude-treatment interaction approach to skill acquisition. Joumal ofApplied Psychology, 74, 657-«90. Kanfer R, Ackerman PL, Murtha TC, Dugdale B, Nelson L (1994). Goal setting, conditions of practice, and task performance: A resource allocation perspective. Joumal ofAppUed Psychology, 79,826-835. Kraiger K, Ford JK, Salas E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Joumal of Applied Psychology, 78, 311-328. Levin JR. (1986). Four cognitive principles of learning-strategy instruction. Educational Psychologist, 21 (1-2), 3-17. Lord RG, Maher KJ. (1991). Cognitive theory in Industrial and Organizational Psychology. In Dunnette MD, Hough LM (Eds.), Handbook of industrial and organizational psychology (Vol. 2, pp. 1-62). Palo Alto, CA: Consulting Psychologists Press. McKelvey JD, Lord RG. (1986, April). The effects of automatic and controlled processing on rating accuracy. Paper presented at the First Annual Conference of the Society for Industrial and Organizational Psychology, Inc., Chicago. Mattoon JS, Klein JD. (1993). Controlling challenge in instructional simulation. Joumal of Educational Computing Research, 9, 219-235. Meece JL, Blumenfeld PC, Hoyle RH. (1988). Students' goal orientations and cognitive engagement in classroom activities. Joumal of Educational Psychology, 80(4), 514523. Milheim WD, Martin BL. (1991). Theoretical bases for the use of learner control: Three different perspectives. Joumal of Computer-based Instruction, 18, 99-105. Morris CD, Bransford JD, Franks JJ. (1977). Level of processing versus transfer appropriate processing. Joumal of Verbal Leaming and Verbal Behavior, 16, 519-533. Nelson TO, Narens L. (1990). Metamemory: A theoretical framework and new findings. The Psychology of Leaming and Motivation, 26, 125-141. Noe RA, Wilk SL. (1993). Investigation of the factors that influence employees' participation in development activities. Joumal ofApplied Psychology, 78, 291-302. Paas FGWC. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Joumal of Educational Psychology, 84 (4), 429-434. Paas FGWC, Van Merrienboer JJG. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive load approach. Joumal of Educational Psychology, 86 (1), 122-133. Schmeck RR. (1983). Learning styles of college students. In Dillon RF, Schmeck RR (Eds.), Individual differences in cognition (Vol. 1, pp. 233-279). New York: Academic Press. Schmeck RR. (1988). Individual differences and learning strategies. In Weinstein CE, Goetz ET, Alexander PA (Eds.), Leaming and study strategies: Issues in assessment, instruction and evaluation (pp. 171-191). San Diego: Academic Press. Schmidt RA, Bjork RA. (1992). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science, 3, 207217. Steinberg E. (1989). Cognition and learner control: A literature review, 1977-1988. Journal of Computer-based Instruction, 16, 117-121. l^innenbaum SI, Yukl G. (1992). Training and development in work organizations. Annual Review of Psychology, 43, 399-441.

420


Tulving E, Thomson DM. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352-373. ifser's Manual for the Wbnderiic Personnel Test and the Scholastic Level Exam. (1992). Libertyville, IL: Wonderlic Personnel Test, Inc. Weiss HM. (1990). Learning theory and industrial and organizational psychology. In Dunnette MD, Hough LM (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 171-221). Palo Alto, CA: Consulting Psychologists Press. Weldon MS, Roediger HL, Challis BH. (1989). The properties of retrieval cues constrain the picture superiority effect. Memory and Cognition, 17, 95-105.