The Reliability and Validity of the Goal Orientation and ... - EBSCOhost

0 downloads 0 Views 1MB Size Report
The Goal Orientation and Leaming Strategies Survey (GOALS-S; Dowson & Melnemey, 2004) is an instrument designed to assess four sets of eonstruets: (I) ...
The Asia-Pacific Education Researcher 20:3 (2011), pp. 579-594

The Reliability and Validity of the Goal Orientation and Learning Strategies Survey (GOALS-S): A Filipino Investigation Ronnel B. King,* David A. Watkins

The University of Hong Kong *[email protected]

The Goal Orientation and Leaming Strategies Survey (GOALS-S; Dowson & Melnemey, 2004) is an instrument designed to assess four sets of eonstruets: (I) aeademie goals, (2) soeial goals, (3) cognitive strategies, and (4) metaeognitive strategies of high school students. This instrument was initially developed and validated among students in Australia. The applicability of this instrument to the Philippine setting was tested in a study involving 1,147 Filipino students from Metro Manila. Responses to this questionnaire are shown to have good intemal eonsisteney reliability. Support is provided for its within-network eonstruet validity in terms of its faetorial structure and evidence of its between-network eonstruet validity is shown through its correlations with other valued edueational outeomes. Different multigroup eonfirmatory factor analysis likewise indieated that the instrument is invariant aeross genders, aeross year levels, and aeross type of sehool. Suggestions for frirther researeh using the GOALS-S are provided. Keywords: GOALS-S, validation, eonfirmatory factor analyses, Filipino students

Students' academic goals (Covington, 2000; Elliot, 2005; Kaplan & Maehr, 2007; Maehr & Zusho, 2009), soeial goals (Urdan & Maehr, 1995; Wentzel, 2000), eognitive strategies (Fredricks, Blumenfeld, & Paris, 2004; Lau, Liem, & Nie, 2008; Marton, Dall'Alba, & Kun, 1996; Pressly & Harris, 2006), and metaeognitive strategies (Hacker, 1998; Sehunk, 2001; Wright & Jacobs, 2003) have all been implicated as crucial for sueeessful leaming in sehool. Thus the accurate measurement of these constructs is important for both educational researchers and practitioners. However, there has been a dearth of instmments that measured all these constructs in one seale. Most of the existing seales only measured one or two of these constructs together (e.g., Elliot & MeGregor, 2001; Midgley et al., 2000; Pintrieh & DeGroot, 1990). As such, most edueational researchers are forced to use different instruments to assess constructs relevant to their research. The use of different instruments, however, may cause some problems because the psychometric

properties of these instruments might not be fully known before data colleetion. Thus Dowson and Melnemey (2004) developed the Goal Orientation and Leaming Strategies Survey (GOALS-S) to remedy this absence of a coherent set of measures that assesses all these sets of constructs in a single instrument. GOALS-S is an 84-item self-report questionnaire that assesses four sets of constmets that are further divided into different subseales: (1) academic goals (mastery goals, performanee goals, and work avoidance goals), (2) soeial goals (social affiliation goals, soeial approval goals, social concem goals, soeial responsibility goals, and social status goals), (3) cognitive strategies (elaboration, organization, and rehearsal), and (4) metacognitive strategies (monitoring, planning, and regulating). An advantage of GOALS-S is that it measures a multiplicity of students' goals. It incorporates three kinds of academic goals (mastery, performance, and work avoidance) and five kinds of soeial goals (soeial

Copyright© 2011 De La Salle University, Philippines

580

THE ASIA-PACIFIC EDUCATION RESEARCHER

affiliation, social approval, social concern, social responsibility, and social status goals). This is one of the few instruments that measure a wide range of goals, both social and academic in a single instrument (see Mclnemey & AH, 2006 for an exception). Most of the existing instruments only focused on academic goals usually mastery and performance goals (e.g., Midgley et al., 2000, Elliot & McGregor, 2001; Elliot & Murayama, 2008). This narrow focus has been criticized for being overly restrictive since it neglects social goals which have also been shown to be salient in motivating students (Ford, 1996; Martin & Dowson, 2009; Urdan & Maehr, 1995; Wentzel, 1996, 2000). In addition, the measurement of social goals becomes even more relevant when studying students from collectivist cultures where the social dimension of motivation is more salient (Kumar & Maehr, 2007; Salili, 1996; Mclnemey, 2008; Yu & Yang, 1994). Although there have been some previous instmments that did measure social goals, they usually included only one (e.g., Chang & Wong, 2008; Cheng, 2005; Tao, 2003; Yu & Yang, 1994) or two types of social goals (e.g., Mclnemey & Ali, 2006; Miller, Greene, Montalvo, Ravindran, & Nichols, 1996). This can be considered a disadvantage given that a lot of qualitative studies have indicated that students pursue a multiplicity of social goals (Dowson & Mclnemey, 2001, 2003; Lemos, 1996; Mansfield, 2009). In addition, these existing instruments usually fail to make distinctions between various types of social goals. For example, Chang and Wong's (2008) Social Achievement Goal Scale combines items such as "I study in order to avoid being looked down upon by my peers" and "I study so my teachers praise me." Whereas thefirstitem pertains more to peers or lriends, the latter one focuses more on teachers. This can be considered a weakness of the instrument because social goals pertaining to peers may have differential effects lrom those involving teachers. As Urdan and Maehr (1995) cogently argued, "There is a critical need to untangle the many constmcts represented by the term social goals" (p. 232). Therefore the capacity of GOALS-S to distinguish amongfivedifferent types of social goals is a considerable advantage when compared to other existing instruments that only measure one or two kinds of goals or those that confound different types of goals into an omnibus measure. It is also important to note that the social goals measured by GOALS-S refer to the social reasons for trying to achieve academically (Urdan & Maehr, 1995). This conceptualization is

VOL 20 NO. 3

distinct from other social goal measures which try to assess how often students try to achieve specific social outcomes. For example, Wentzers( 1991,1993,1994, 1996) Social Goals Scale is designed to measure how often students want to be prosocial in sehool or how often they try to be responsible in school. GOALS-S, on the other hand, measures the social reasons behind studying and not social outcomes that students want to achieve per se. GOALS-S also measures different kinds of cognitive and metacognitive strategies. The cognitive component comprises three subscales: elaboration, organization, and rehearsal. The metacognitive strategies section focuses on students' use of strategies to monitor and regulate their cognition, and it includes three subscales: planning, monitoring, and regulating. The inclusion of various types of cognitive and metacognitive strategies enables researchers using GOALS-S to have a more nuanced view of the different strategies that students use in school. Other widely used instruments mostly focus on larger units of analysis which fail to take into account finegrained distinctions in terms of the strategies that students adopt in the school setting (e.g.. Biggs, 1987; Biggs, Kember, & Leung, 2001 ; Pintrich & DeGroot, 1990). For example, a popular instmment to measure strategies is the Motivated Strategies for Leaming Questionnaire (MSLQ; Pintrich & DeGroot, 1990). The MSLQ aggregates all metacognitive strategies into one metacognitive self-regulation subscale. For example, items such as "When I study for this class I set goals for myself in order to direct my activities for each study period" and "I ask myself questions to make sure I understand the material I have been studying in this class" are aggregated into one subscale. The first item refers to planning activities which involves goalsetting while the latter pertains more to monitoring which involves self-questioning and self-checking. In contrast, GOALS-S posits distinctions among different types of metacognitive strategies. Thus GOALS-S offers a more detailed view of the various kinds of strategies which can be beneficial for teachers and practitioners who may want to know more about the type of strategies students are adept at using and the kinds of strategies they need more help with. Within-network and Between-network Approaches to Construct Validation Thus far, GOALS-S has only been validated in the Australian setting (Dowson & Mclnemey, 2004)

THE RELIABILITY AND VALIDITY OF THE GOAL ORIENTATION

and has only been used in studies with Australian students (e.g., Dowson, Mclnemey, & Nelson, 2006). Therefore, it is important that we investigate whether GOALS-S is also applicable to other non-Westem cultures such as the Philippines. The present study adopts a constmct validation approach to the empirical assessment of GOALS-S (Marsh, 1997). Studies that adopt this approach can be classified as withinnetwork or between-network studies. Within-network construct validation, also called intemal construct validation refers to the examination of the factor structure and factor correlation matrix. It typically involves statistical techniques such as confirmatory factor analysis (CFA) and reliability analysis. On the other hand, between-network or extemal construct validation approach entails examining pattems of relationships between the scales and other theoretically related constmcts utilizing statistical techniques such as correlational analysis (Marsh, 1997). Few studies adopt this dual approach to validity thus providing relatively limited input into understanding the constmcts being investigated. In contrast, the present study uses both approaches. First, we conduct a within-network study using confirmatory factor analysis to test the psychometric properties of GOALS-S. Consistent with the constmct validation approach, it is not only important to address validity within an instmment (within-network validity) but it is also imperative to explore the possible differential relationships between the four sets of constructs in GOALS-S and a set of theoretically relevant measures (between-network validity). In our study, we looked at how the four sets of constructs in GOALS-S (academic goals, social goals, cognitive strategies, and metacognitive strategies) are related to behavioral and emotional engagement. Behavioral engagement has traditionally been defined as involvement in learning tasks which includes behaviors such as effort, persistence, concentration, asking questions, and contributing to class discussions (Birch & Ladd, 1997; Finn, Pannozzo, & Voelkl, 1995). Emotional engagement has been defined as students' feelings about peers, teachers, schoolwork, or their affective reactions to the classroom (Fredericks et al., 2004). The four sets of constructs (academic goals, social goals, cognitive strategies, and metacognitive strategies) in GOALS-S are posited to have different kinds of relationships with behavioral and emotional engagement. For the academic goals scale, mastery' and performance

KING, R.B., & WATKINS, D.A.

581

goals^ are assumed to have a positive relationship with both behavioral and emotional engagement. Work avoidance goals which refer to preference for easy work and the avoidance of effort exertion in school would be negatively related to both behavioral and emotional engagement (Elliot & Harackiewicz, 1996; Nicholls, Cobb, Wood, Yackel, & Patashnick, 1990). The relationship between the various kinds of social goals to both behavioral and emotional engagement is not yet that well-established, hence we did not advance any hypotheses a priori with regard to the relationship between these two sets of constmcts. The various kinds of cognitive and metacognitive strategies are also assumed to be positively correlated with both behavioral and emotional engagement. Invariance testing Aside from testing for construct validity, it is also useful to look at whether different kinds of students respond to GOALS-S in a similar manner It is common practice in educational research to pool together data from different kinds of students (e.g., different genders, year level, and types of school). However, combining datasets together would only be warranted if invariance in terms of factor stmcture can be shown. Previous studies have given inadequate attention to the investigation of the equivalence of educational constructs to students of different genders, year levels, and school types thus we wanted to investigate whether GOALS-S has invariant factor structure across different kinds of students. THE PRESENT STUDY The aim of this research was to examine crosscultural applicability of GOALS-S in the Philippine setting. This study adopted both within-network and between-network approaches to constmct validation. In testing for within-network constmct validity, we hypothesize that the model will refiect a sound firstorder factor structure in regard to the four sets of constructs measured in GOALS-S. For the academic goals scale, the first-order stmcture is composed of three factors: mastery goals, performance goals, and work avoidance goals. For the social goals scale, the first-order structure consists of five factors: social affiliation goals, social approval goals, social coneem goals, social responsibility goals, and social status goals. For the cognitive strategies scale, there are three factors: elaboration, organization, and rehearsal. For the metacognitive strategies scale, there are also

582

THE ASIA-PACIFIC EDUCATIQN RESEARCHER

three factors: monitoring, planning, and regulating; In addition to doing separate CFAs, we also tested for the invariance ofthe GOALS-S across students of different genders, year levels, and school types. For between-network construct validity, we tested for the relationship of the various scales of GOALS-S to behavioral and emotional engagement. It is hypothesized that academic goals such as mastery and performance goals will be positively related to these two between-network constructs while work avoidance goals will be negatively related to them. It is also hypothesized that all the cognitive and metacognitive strategies will be positively related to behavioral and emotional engagement. However, no specific hypotheses are advanced with regard to the relationship between social goals and these two between-network constructs, because the nomological network of social goals is still under-researched. METHOD Participants The participants of this study were 1,147 high school students from five schools (four private schools and one public school) in Metro Manila, Philippines. Studentsfromthe four private schools can be classified as middle class and their SES is comparatively higher than the SES of the students from the public school. The medium of instruction in all these schools was English. The students were distributed from first year to fourth year high school. There were 622 males (54.2%) and 524 (45.7%) females, with one student failing to identify the gender. Six hundred ninety-six students were drawn from the four private schools and 451 were from the one public school. At the time of testing, 603 students were in first or second year high school, while 544 students were in third or fourth year high school. The mean age ofthe students was 14.20 years old (SD = 1.39). All these five schools cater to secondary school students from first year to fourth year high school. A dministration The first author administered the GOALS-S to participants in class groups with the assistance of teachers at each school. The teachers were instructed not to interpret any of the GOALS-S items for students but to tell students to interpret the items as best as they can or leave it blank if they do not understand it. An earlier pilot study conducted on a

VQL 20 NQ. 3 smaller scale has indicated that the level of difficulty was appropriate for Filipino high school students thus we decided to administer the English version ofthe questionnaire. Measures The main instrument was the 84-item GOALS-S which was designed to measure four sets of constructs: (1) three academic goals, (2) five social goals, (3) three cognitive strategies, and (4) three meta-cognitive strategies. Each of these factors was measured through six items. A six-point Likert-type scale was used for each item ranging from 1 (lowest) to 6 (highest) with 1 refiecting lowest level of endorsement and 6 reflecting the highest level of endorsement for a particular item.^ Academic goals Factor 1. Mastery goals. This measures the degree to which students want to achieve in school to develop their competence relative to self-set standards (e.g., "I work hard at school because I am interested in what I am leaming.") Factor 2. Performance goals. This measures the degree to which students want to achieve in school to outperform other students (e.g., "I want to leam things so that I can come near the top ofthe class.") Factor 3. Work avoidance goals. This measures the degree to which students want to achieve with as little effort as possible (e.g., "I choose easy options in school so that I don't have to work too hard.") Social goals Factor 4. Social affiliation goals. This measures the degree to which students want to achieve in school to enhance a sense of belonging to the group (e.g., "I want to do well at school so that I can feel close to my group of friends.") Factor 5. Social approval goals. This measures the degree to which students want to achieve to gain the approval of parents and teachers (e.g., "I want to do well at school so that I can get praise from my teachers.") Factor 6. Social concern goals. This measures the degree to whieh students want to achieve to be able to assist others in their academic work when they need it. (e.g., "I try to do well at school so that I can I help my friends with their schoolwork.")

Factor 7. Social responsibility goals. This measures the degree to which students want to achieve

THE RELIABILITY AND VALIDITY OF THE GOAL ORIENTATION

to meet social role obligations or meet interpersonal commitments (e.g., "I do good schoolwork so that I don't have any trouble with my parents or teaehers.") Factor 8. Social status goals. This measures the degree to which students want to attain wealth or position in school and later life (e.g.,"I do good schoolwork so that I have a good future.") Cognitive strategies Factor 9. Elaboration. This strategy refers to making connections between present and previously leamed information which my involve strategies like paraphrasing, generating analogies, etc. (e.g., "When leaming things for school, I try to see how they fit together with other things.") Factor 10. Organization. This strategy refers to selecting, sequencing, outlining or summarizing important information in class (e.g., "I try to organize my school notes when I want to leam things for school.") Factor 11. Rehearsal. This strategy refers to listing, memorizing, and reciting facts to be leamed (e.g., "I try to memorize things I want to leam for school") Meta-cognitive strategies Factor 12. Monitoring. This strategy involves selfchecking for understanding and involves a systematic aftempt to evaluate the assimilation and organization of material that has been leamed in class (e.g., "I often ask myself questions to see if I understand what I am leaming.") Factor 13. Planning. This strategy involves prioritizing, managing time, and setting realistic goals in order to complete the work needed (e.g., "When I want to leam things for school I pick out the most important parts first.") Factor 14. Regulating. This strategy involves rectifying deficits in leaming that were identified by the student and includes aftempts to seek explanations from teachers and identify mistakes in reasoning (e.g., prioritizing, managing time, and setting realistie goals in order to complete the work needed (e.g., "If I am having trouble leaming something at school, I ask for help.") To assess behavioral engagement, we used the ongoing engagement subscale (e.g., "I work very hard on my schoolwork.") of the Student Engagement and Disaffection in School—Student Report from the Roehester Assessment Package

KING, R.B., & WATKINS, D.A.

583

(RAPS) for Schools developed by Wellborn and Connell (1987). This subscale had four items and exhibited good psychometric properties in the current study {M = 4.41, SD= .91, Cronbach's alpha = .70). To measure emotional engagement, we used the affeet to school subscale (e.g., "I like working at school.") of the Facilitating Conditions Questionnaire (FCQ; Mclnemey, Dowson, & Yeung, 2005), which also consists of four items {M = 4.27, SD = .86, Cronbach's alpha = .58). Both the RAPS and FCQ from which the behavioral and emotional engagement indices were drawn have been shown to have good psychometric properties in a lot of previous studies conducted in various cultural contexts including the Philippines (e.g., Ganotice, 2010; Gonida, Voulala, & Kiosseoglou, 2009; Mclnemey, 2008; Mclnemey et al, 2005; Skinner & Belmont, 1993; Wellbom & Connell, 1987). Statistical Analysis To examine the within-network validity of GOALS-S among Filipino high school students, we first computed for the descriptive statistics and reliability alpha coefficients. Four separate confirmatory factor analysis were then conducted to test the factor stmcture of the four sets of constructs in GOALS-S. As reported earlier, the GOALS-S has 84 items. Bagozzi and Heatherton (1994) argued that when there are more than four or five indicators per factor in a large sample, it is quite likely to lead to an unsatisfactory fit in the measurement model. To address this, we aggregated the items to form "parcels" as indicators in the CFA (see Little, Cunningham, Shara, & Widaman, 2002 for a review of issues surrounding parcelling). The parcels were formed by randomly combining two items in each factor. Since each factor was originally measured with six items, this resulted in each factor in GOALS-S being measured by three parcels. All the parcels were specified as indicators of only one factor, and the uniqueness of each parcel was constrained to be independent. For example, mastery goals were measured through three parcels and this was the case for all other latent factors. This resulted in a total of 42 parcels (14 latent factors each measured by three parcels) We conducted four separate CFAs on conceptually distinct sets of scales relating to students' (1) academic goals, (2) social goals, (3) cognitive strategies, and (4) metacognitive strategies. We allowed the factor correlations in the four separate CFAs to befi-eelyestimated. All analyses were conducted using Amos 12 (Arbunckle, 2007),

584

VOL 20 NO. 3

THE ASIA-PACIFIC EDUCATION RESEARCHER

and all parameters were estimated using maximum likelihood procedure. To test for between-network construct validity, we computed for the correlations of the four sets of constructs in GOALS-S with measures of behavioral and emotional engagement. We also conducted multigroup eonfirmatory factor analysis to assess the faetorial invariance of GOALS-S. Invariance analysis is done to provide information about the equivalence of the data across multiple groups (Marsh, 1993, 1994). In the present study, we did three invariance tests related to testing the equivalence of GOALS-S (1) aeross genders (male and female), (2) across year levels (junior and senior high school) and (3) across academic institutions (private and public schools). In testing for invariance aeross year levels, students who were in their first and second year were put into the junior high school group, while participants on their

third and fourth year were put into the senior high school group. We then eonducted invariance analysis on the junior and senior high sehool groups to check whether the students in their early years (P' and 2"'') and students who were more advanced (3"' and 4* year) answered the questionnaire in the same way. In testing for invariance across the private and public schools, students from the four private schools were put into the private school category, while students from the public school were put into the public school eategory. We then conducted an invariance analysis for students from these two groups. RESULTS Preliminary Analyses The means, standard deviations, and reliability alpha coefficients of GOALS-S are shown in Table 1.

Table 1 Descriptive Statistics, Cronbach s Alphas, and CFA Loadings for the Different Subseales of GOALS-S Cronbach's CFA loadings alpha in Cronbach's range ior Dowson and alpha in the SD Mean Subseales the different current study Melnerney's parcels (2004) study Academic goals .649 to .748 .78 .81 0.76 4.76 Mastery goals .745 to .815 .87 .81 1.01 4.16 Performanee goals .450 to .855 .72 .67 1.00 3.00 Work avoidance goals Social goals Soeial affiliation goals Soeial approval goals Soeial eoneem goals Soeial responsibility goals Soeial status goals

CFA loadings mean

.712 .773 .682

4.05 4.37 4.59 5.21

1.16 1.07 1.06 0.88 0.74

.86 .83 .92 .79 .82

.83 .84 .74 .82 .84

.771 to .855 .484 to .877 .887 to .929 .683 to .797 .758 to .888

.803 .736 .910 .737 .815

Cognitive strategies Elaboration Organization Rehearsal

4.56 4.58 4.53

0.87 0.93 0.91

.87 .87 .85

.73 .82 .76

.797 to .823 .794 to .847 .815 to .837

.812 .823 .824

Metacognitive strategies Planning Monitoring Regulating

4.48 4.60 4.47

0.88 0.83 0.80

.82 .83 .72

.81 .83 .79

.740 to .816 .770 to .821 .579 to .726

.789 .799 .650

3.74

THE RELIABILITY AND VALIDITY OF THE GOAL ORIENTATION

KING, R.B., & WATKINS, D.A.

585

Table 2 Bivariate Correlations Among the Academic and Social Goals in GOALS-S 1. Mastery

2

3

4

5

6

7

8

.429***

-.252*"

.254***

.330*"

.479***

.550*"

.414*"



.007

.469***

.620***

.344*"

.542***

.326*"

.192*"

.055-1-

-.172*"

-.136*"

-.155*"



.535*"

.392"*

.423***

.114*"



.369*"

.528***

.306***



.451*"

.280***



.495***

2. Performance 3. Work avoidance 4. Affiliation 5. Approval 6. Concem 7. Responsibility 8. Status Note: *p