Running head: MULTIDIMENSIONAL ASSESSMENT

Running head: MULTIDIMENSIONAL ASSESSMENT OF TEAMWORK IN SPORT 1 2

The Development and Psychometric Properties of the Multidimensional Assessment of

3

Teamwork in Sport (MATS)

4 5

Authors: Desmond McEwan*, Bruno D. Zumbo, Mark A. Eys, & Mark R. Beauchamp

6 7

*Corresponding author ([email protected])

8 9

Institutional Affiliations:

10

Desmond McEwan & Mark R. Beauchamp: School of Kinesiology, The University of British

11

Columbia, Vancouver, British Columbia, Canada

12

Bruno D. Zumbo: Department of Educational and Counseling Psychology, Faculty of Education,

13

University of British Columbia, Vancouver, British Columbia, Canada

14

Mark A. Eys: Departments of Kinesiology/Physical Education and Psychology, Wilfrid Laurier

15

University, Waterloo, Ontario, Canada

16 17

As accepted for publication in Journal of Sport & Exercise Psychology, ©Human Kinetics

18

Date of acceptance: March 15, 2018

19 20

MULTIDIMENSIONAL ASSESSMENT OF TEAMWORK IN SPORT 1 2

2

Abstract The purpose of this research was to develop a questionnaire to assess the multidimensional

3

construct of teamwork in sport and examine various aspects of validity related to that instrument. A

4

preliminary questionnaire was first created, and feedback on this instrument was then obtained from

5

a sample of team sport athletes (n = 30) and experts in sport psychology (n = 8). A modified version

6

of the questionnaire was then completed by 607 athletes from 48 teams, and five multilevel

7

confirmatory factor analyses were conducted to examine the structural properties of data derived

8

from this instrument. Evidence of adequate model-data fit along with measurement reliability was

9

obtained for each of the five models. Taken together, the results from this research provide support

10

for the content, substantive, and structural aspects of construct validity (cf. Messick, 1995) for data

11

derived from the 66-item Multidimensional Assessment of Teamwork in Sport.

12

Keywords: Team; Group; Processes; Effectiveness.

MULTIDIMENSIONAL ASSESSMENT OF TEAMWORK IN SPORT 1

The Development and Psychometric Properties of the Multidimensional Assessment of

2

Teamwork in Sport (MATS)

3

3

Teamwork is often noted as an important variable within the vernacular of sport. Coaches

4

frequently emphasize the importance of players working together, with athletes similarly attributing

5

team outcomes to the extent to which they work well with their teammates. Despite this seeming

6

importance of teamwork, there has been surprisingly limited research on this construct within sport

7

settings (Carron, Martin, & Loughead, 2012; McEwan & Beauchamp, 2014). Indeed, although

8

decades of research has accumulated within other team contexts such as health care, aviation,

9

business, and military settings, the evidence related to teamwork in sport has been sparse,

10 11

fragmented, and mostly anecdotal (McEwan & Beauchamp, 2014). In an effort to begin to fill this considerable hole in the field of sport psychology, McEwan

12

and Beauchamp (2014) conducted a theoretical and integrative review in order to provide a working

13

definition and conceptual framework of teamwork in sport. They defined teamwork as “a dynamic

14

process involving a collaborative effort by team members to effectively carry out the independent

15

and interdependent behaviors that are required to maximize a team’s likelihood of achieving its

16

purposes” (p. 233). Their conceptual framework—which includes five overarching aspects and 14

17

lower-order dimensions of teamwork—was informed by Rousseau, Aubé, and Savoie’s (2006)

18

teamwork framework. The framework by Rousseau et al. (2006) was itself based on a

19

comprehensive analysis of 29 frameworks used to study teamwork behaviors in a range of group

20

settings (e.g., health care, business, military) and, as such, can be considered a ‘meta-framework’ of

21

teamwork. Twelve of the dimensions within McEwan and Beauchamp’s (2014) framework focus on

22

behaviors associated with team task performance (i.e., regulation of team performance), while two

23

dimensions focus on behaviors associated with the management of team maintenance (i.e.,

24

interpersonal dynamics of a team).


4

With regard to regulating team performance (RTP), teamwork behaviors were

2

conceptualized by McEwan and Beauchamp (2014) to be enacted over four separate phases: (a)

3

preparation, (b) execution, (c) evaluation, and (d) adjustments. Three behavioral dimensions

4

comprise the preparation phase, which take place in advance of a team task (e.g., at the start of a

5

team’s season, before a game or competition). First, mission analysis involves defining a team’s

6

overall purpose or reasons for being together (e.g., to win a league championship, to have fun).

7

Second, goal specification involves setting team goals in order to achieve those team purposes (e.g.,

8

to obtain a certain race time in team kayaking, to score a specific number of runs per game in

9

baseball). Third, action planning involves establishing strategies for obtaining team goals (e.g.,

10

identifying defensive matchups in a basketball game, running drills in a practice).

11

Three dimensions comprise the execution phase of RTP, which occur during a team task

12

(i.e., while a team is competing). First, communication involves the information sharing that occurs

13

between teammates (e.g., field hockey players calling for a pass, a coxswain relaying stroke

14

commands in crew rowing). Second, coordination involves teammates being synchronized with

15

each other in terms of the sequencing and timing of their actions (e.g., quarterbacks passing the ball

16

once their receivers have run their route in football, a track team exchanging the baton at the ideal

17

time). Third, cooperation involves teammates working together in a unified manner (e.g., hockey

18

players blocking opponents’ shots for the benefit of their team, rugby players working together as

19

one strong entity in order to advance the ball in a scrum in a joint manner). Although the latter two

20

dimensions are similar in many regards, it is important to note that they are conceptually distinct.

21

With regard to coordination, the execution of a player’s task is dependent upon, and/or subsequently

22

links to, other players’ task executions. With regard to cooperation, team success is based on team

23

members helping one another and working collaboratively as a single, interconnected unit on a team

24

task.


5

The third and fourth phases of RTP—evaluation and adjustments—occur after a team has

2

completed its task (e.g., during the halftime break in soccer, following the conclusion of a

3

tournament). The evaluation phase is comprised of two dimensions, which act as a means of self-

4

regulation. These dimensions include performance monitoring, whereby teams monitor how well

5

they have performed (e.g., whether the team won a recent game, how effectively the team worked

6

together) as well as systems monitoring, wherein teams monitor various conditions that may have

7

affected their performance (e.g., situations during a competition, changes in team personnel).

8

Following task execution, four teamwork behaviors comprise adjustments, the fourth phase

9

of RTP, whereby team members might diagnose issues in team functioning and devise solutions as

10

a result (i.e., team self-correction). First, problem solving involves identifying why a team has been

11

unsuccessful (e.g., poor coordination) and how they can perform better (e.g., carrying out certain

12

drills in practice focused on improving coordination). Second, innovation involves enacting novel

13

approaches in order to enhance team task execution (e.g., altering game strategies, trying players

14

out at new positions). Third, backing up involves teammates helping one another enhance their

15

performance (e.g., showing a teammate how to perform a certain manoeuvre, redistributing the task

16

responsibilities of players who are feeling overworked). The fourth dimension, intrateam coaching,

17

is similar to backing up behaviours but focuses on the verbal feedback that teammates provide to

18

each other following task performance (e.g., an experienced veteran player providing helpful advice

19

to a more inexperienced player, an injured player observing team competitions and making

20

suggestions to teammates on how they can be successful). The main distinction between these latter

21

two dimensions is that intrateam coaching involves verbal assistance between teammates whereas

22

backing up entails non-verbal assistance.

23 24

Rather than directly focused on team performance, the fifth aspect of teamwork—known as the management of team maintenance (MTM)—is focused on behaviors associated with keeping the

MULTIDIMENSIONAL ASSESSMENT OF TEAMWORK IN SPORT

6

1

team together, and ensuring that personal and/or interpersonal non-performance-related issues do

2

not preclude a team from functioning effectively. Two dimensions comprise this aspect of

3

teamwork. The first dimension, psychological support, involves teammates providing assistance to

4

one another in the form of emotional (e.g., sharing and listening to each other about events going on

5

in one’s life outside of sport), esteem (e.g., helping teammates feel confident about themselves),

6

informational (e.g., teammates providing advice to one another), and/or tangible (e.g., providing

7

concrete, instrumental assistance such as rides to and from practice) support. The second dimension,

8

integrative conflict management, involves managing interpersonal conflicts between team members

9

if/when they arise (e.g., resolving disagreements between teammates).

10

As research on teamwork in sport is still in its infancy (cf. Carron et al., 2012; McEwan &

11

Beauchamp, 2014), it is perhaps unsurprising that there have not yet been any attempts to measure

12

this multidimensional construct. The availability of a psychometrically sound measure of teamwork

13

in sport is critical in order for this potentially important new line of research within sport

14

psychology to progress. Moreover, such an instrument has the potential to be useful to coaches and

15

sport psychology consultants. Specifically, coaches/consultants could have athletes complete the

16

teamwork measure, determine areas of strength and weakness with regard to teamwork, and target

17

those components that require improvement. With these potential uses in mind, the purpose of this

18

research was to develop a questionnaire designed to assess teamwork in sport and examine various

19

aspects of validity associated with measures derived from this instrument. This research was guided

20

by Messick’s (1995) unified view of validity, namely by the process of construct validation, which

21

comprises the collection of evidence and rationale that support the trustworthiness of data derived

22

from a focal measure (Messick, 1995). From this perspective, construct validation is considered an

23

ongoing process that occurs over the course of several studies, with six aspects of validity being

24

examined throughout (Hubley & Zumbo, 1996; Messick, 1995). These aspects include content (the


7

1

relevance, representativeness, and technical quality of a questionnaire and its items), substantive

2

(respondents’ interpretations and judgments of a questionnaire and its items), structural (the extent

3

to which the model-data structure of measures from a questionnaire aligns with the conceptual

4

framework underpinning that construct), generalizability (the extent to which score properties and

5

interpretations generalize across populations, contexts, and tasks), external (convergence with

6

and/or discrimination between a focal construct and other variables), and consequential (the

7

implications that may result from using a test, such as providing a ‘basis for action’) validity

8

(Messick, 1995). Hence, validity needs to be considered at the outset of a research program as

9

opposed to being viewed as a ‘one-off’ procedure in a single study of test development (Anastasi,

10 11

1950; Hubley & Zumbo, 1996; Smith, 2005). The first phase of this construct validation process involves fully articulating a conceptual

12

framework and definition of the construct being studied (Clark & Watson, 1995; Cronbach &

13

Meehl, 1955; Smith, 2005). In the context of the current research, this was carried out via the

14

theoretical and integrative review by McEwan and Beauchamp (2014). While this framework

15

appears to provide a comprehensive and viable basis for examining teamwork, its utility within the

16

context of sport has yet to be tested. The next phase involves developing a psychometrically sound

17

questionnaire that accurately measures the focal variable, namely teamwork in sport. This phase

18

represented the overall focus of the research presented in this paper, which took place over two

19

studies. In Study 1, we sought to develop/collect a comprehensive pool of items, which was

20

informed by McEwan and Beauchamp’s (2014) framework and definition of teamwork. Once a

21

preliminary questionnaire was created, we sought to examine the content and substantive aspects of

22

validity by consulting with, and obtaining feedback from, the target population (i.e., team sport

23

coaches and athletes) as well as experts within the specific field of study (i.e., those with a PhD

24

focused on Sport Psychology; cf. Messick, 1995). The objective of Study 2 was to obtain evidence


8

1

for the structural aspect of validity by examining data from a heterogenous sample of team sport

2

athletes who completed the questionnaire (cf. Clark & Watson, 1995; Flora & Flake, 2017;

3

Messick, 1995). Developing this questionnaire and obtaining support for the content, substantive,

4

and structural aspects of validity will enable researchers to examine the external, generalizability,

5

and consequential aspects of validity related to this instrument in future studies.

6

Study 1

7

The initial study of this research consisted of three parts. First, following procedures

8

outlined by Clark and Watson (1995) and DeVellis (2012), part A of the study involved creating an

9

initial pool of potential items that measured the 14 dimensions of teamwork. To do so, we referred

10

back to the conceptual framework and definition of teamwork (cf. McEwan & Beauchamp, 2014) in

11

order to create a pool of items that reflected those teamwork behaviors. In addition, we reviewed the

12

literature for previous attempts to measure teamwork in some way (in both sport settings and other

13

team contexts). A total of 566 items were identified as a result of this procedure (mean = 40 items

14

per dimension, range = 8 [systems monitoring] to 112 [conflict management]). Following

15

recommendations by Clark and Watson (1995), a rigorous and iterative process was then

16

undertaken to modify, amalgamate, or delete items where necessary. Specifically, the first and last

17

author examined items multiple times to ensure that: (a) the wording of the items was simple and

18

straightforward; (b) overly colloquial expressions were not used; (c) ‘double-barrelled’ items that

19

measure more than one component of a dimension (e.g., “we resolve conflict respectfully and

20

efficiently”) were avoided; and (d) items accurately reflected a teamwork behaviour (e.g., items

21

assessing members’ knowledge or affect with regard to their team were excluded or modified). This

22

process ultimately resulted in a preliminary 74-item measure of teamwork that we titled the

23

Multidimensional Assessment of Teamwork in Sport (MATS). A variety of response scales were then

24

considered including a dichotomous scale (i.e., where participants choose one of two response


9

1

options), an ipsative scale (i.e., forced-choice scale that does not include a middle option), or a

2

Likert-type scale with various ranges (e.g., five-point, seven-point) and response formats (e.g., level

3

of agreement, frequency; Clark & Watson, 1995; DeVellis, 2012). Ultimately, a five-point, Likert-

4

type response scale (1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree;

5

and 5 = strongly disagree) was deemed to be most appropriate in assessing the initial set of items

6

and was, therefore, selected in this preliminary measure.

7

A sample of 25 team sport athletes and five coaches (23 males, 7 females) then participated

8

in part B of the study. These participants competed in a variety of team sports, including volleyball

9

(n = 15), curling (n = 4), soccer (n = 3), rowing (n = 2), Canadian football (n = 1), baseball (n = 1),

10

ice-hockey (n =1), and multiple sports (n = 3). The participants competed at a range of levels,

11

including university (n = 18), recreational (n = 5), national (n = 4), and rep/select teams competing

12

against other teams within their district/region (n = 3). In this part of the study, participants

13

completed, and provided feedback on, the preliminary questionnaire in a focus group format

14

through use of a ‘retrospective think-aloud’ protocol (Willis, 2005). This was done in order to better

15

understand how members of the target population interpret and respond to items on the

16

questionnaire (i.e., examine response processes). Specifically, consulting with a sample of the target

17

population in this way helps to establish evidence of the content (i.e., the relevance,

18

representativeness, and technical quality of the items) and substantive (i.e., understanding how

19

respondents interpret and make sense of the items) aspects of validity (Messick, 1995).

20

After providing consent, participants were asked to complete (individually) a subset of items

21

from the preliminary MATS (approximately 18-19 items per session). The reason for having

22

participants only complete a subset of items (as opposed to the entire MATS) was to reduce

23

participant burden and ensure quality within their responses and the subsequent group discussion.

24

Participants were able to participate in multiple focus group sessions (i.e., provide feedback on


10

1

separate sections of the questionnaire); on average, participants took part in two study sessions. A

2

total of 21 focus group sessions were ultimately carried out, each of which lasted approximately 45-

3

60 minutes and consisted of 2-5 participants. While going through the questionnaire individually,

4

participants were encouraged to make notes of any aspect of the instrument that they felt lacked

5

clarity or could otherwise be improved.

6

Once all participants completed the questionnaire, the researcher facilitated the group

7

discussion about the MATS. Participants were encouraged to provide feedback on the questionnaire

8

at any point throughout the discussion, however minor or substantial (i.e., the “think-aloud”

9

component of the discussion). To begin the protocol, the researcher asked participants for their

10

feedback on the questionnaire’s five-point scale. Specifically, participants were asked if they felt

11

that they could respond to each item accurately with this scale, or if an amended scale—such as an

12

ipsative scale, dichotomous scale, or other Likert-type scale with fewer or additional response

13

choices—would be more useful. The researcher then went through each item on the questionnaire

14

with the participants and asked for their feedback with regard to an item's relevance to the context

15

of team sport, representativeness of the construct of teamwork, wording clarity, and potential item

16

redundancy. Specific probes for each item included the following: (a) “what, in your words, do you

17

feel is being asked in this item?”; (b) “did the answer choices include your response?”; (c) “did you

18

understand how to answer this question?”; and (d) “how would you modify this item?”. At the end

19

of each dimension of the questionnaire, participants were asked “was there anything in this section

20

that was left out that you feel is important with regard to [name of teamwork dimension]?”. Once all

21

sections were discussed, participants were given a final opportunity to provide any open-ended

22

feedback that they felt was relevant to the questionnaire. After going through each item within the

23

respective subsection of the questionnaire, participants were thanked for their participation and

24

provided with a $10 (CAD) honorarium. As an ongoing and iterative process, changes were made to


11

1

the MATS throughout the course of this part of the instrument development (i.e., between focus

2

group sessions), until no further changes were identified (each subset of items was reviewed a

3

minimum of five times).

4

Finally, in part C of the study, a sample of eight experts from the field of sport psychology

5

(i.e., with a PhD focused on sport psychology or closely related subject) reviewed, and also

6

provided feedback on, the questionnaire. The purpose of this expert review was to further maximize

7

the content and substantive aspects of validity associated with measures derived from the

8

questionnaire (cf. Clark & Watson, 1995; DeVellis, 2012). After agreeing to take part in the study,

9

participants were sent a Word document version of the MATS. They were first given the definition

10

of teamwork and each of its dimensions (taken from McEwan & Beauchamp, 2014). Participants

11

then rated—on a seven-point scale from ‘-3: Not At All’ to ‘0: Uncertain’ to ‘3 Very Much’—each

12

item's relevance to and representativeness of the definition of teamwork, clarity of wording, and

13

item redundancy. There was also space for participants to provide comments on each item as well as

14

any general feedback about the questionnaire. Any items whereby the expert reviewers did not

15

provide a +3 rating for relevance, representativeness, and clarity, or -3 for redundancy were flagged

16

for potential revision or deletion. As with part B of this study, changes to the MATS were made

17

with each iteration of feedback provided by the experts. Hence, we continued to recruit experts who

18

would complete an updated version (i.e., based on feedback provided by previous participants) of

19

the MATS. This process concluded once no further substantive changes were identified.

20

Several modifications to the questionnaire were made as a result of parts B and C of this

21

study. Some of these changes were minor (e.g., improving the look/presentation of the

22

questionnaire), while others were more substantive. For one, the response scale of the questionnaire

23

was changed to a 7-point Likert-type scale (1 = completely disagree, 2 = mostly disagree, 3 =

24

slightly disagree, 4 = neither agree nor disagree, 5 = slightly agree, 6 = mostly agree, 7 = completely


12

1

agree), as the majority of participants indicated that this expanded scale allowed them to provide

2

responses that more accurately reflected their thinking and appraisals compared to the original 5-

3

point scale. In particular, several participants in the early focus groups of part B of the study

4

remarked that having more than one response option between the neutral and completely agree

5

options would allow them to distinguish between items that were just slightly above the ‘neither

6

agree nor disagree’ mark with those that were more highly above this response but not quite

7

reaching the ‘completely agree’ mark. After adjusting this scale, all participants in subsequent focus

8

groups felt that this scaling was appropriate—no participants in either part B or C of the study

9

indicated that an alternate scale (including the original five-point scale) would be more useful

10 11

compared to this seven-point scale. Second, preambles that described the dimension at the beginning of each section of the

12

questionnaire were expanded and included examples to help foster understanding of each

13

dimension. Third, two new items were created and 32 existing items were deleted, modified, or

14

combined in some way; these amendments were made in order to avoid item redundancy and

15

eliminate items that were (a) unsuitable/inappropriate to sports or (b) included the use of potentially

16

confusing or colloquial terms. For example, an item “Teammates’ actions are executed in harmony

17

with each other” from the original version of the MATS was deleted as several participants

18

remarked that (a) this item was subsumed within the other items of the coordination section, and (b)

19

the phrase ‘in harmony’ was overly colloquial. These modifications ultimately resulted in a revised

20

70-item version of the MATS (consisting of 14 teamwork dimensions and four to seven items per

21

dimension), with support for the content and substantive aspects of validity related to this

22

instrument. The MATS displays a Grade 7-8 (aged 13+ years) reading ability score (Flesch-Kincaid

23

Grade Level = 7.7, Flesch Reading Ease score = 55.1; cf. Flesch, 1948; Kincaid, Fishburne, Rogers,

24

& Chissom, 1975).

MULTIDIMENSIONAL ASSESSMENT OF TEAMWORK IN SPORT 1 2

13

Study 2 The next stage in the construct validation process involved examining the structural aspect

3

of validity of data derived from the MATS. Specifically, we sought to examine the extent to which

4

the items from the MATS aligned with/loaded onto their respective teamwork dimension, as well as

5

whether the factor structure of those dimensions coincided with the conceptual framework of

6

teamwork in sport (cf. McEwan & Beauchamp, 2014). We also sought to examine the reliability of

7

each of the 14 subscales of the MATS. In order to accomplish these objectives, a heterogeneous

8

sample (in terms of sport type, age, sex, and competitive level) of teams and athletes completed the

9

MATS. Five multilevel, multidimensional confirmatory factor analytic models were then carried out

10

in order to examine the structural properties of data derived from the MATS. The multilevel

11

component of the analyses stems from the fact that the athletes were clustered/nested within the

12

teams from which they were sampled.

13

Methods

14

Following institutional ethics approval, a heterogeneous sample (cf. Clark & Watson, 1995)

15

of 607 athletes (65.3% males; mean age = 17.7 years, range = 13-73) from 48 teams were recruited

16

to participate in the study. These teams competed in a range of sports (both interdependent and

17

coactive) including ice-hockey (n = 8), basketball (n = 7), soccer (n = 6); baseball (n = 5), curling (n

18

= 5), water polo (n = 4), rugby (n = 3), volleyball (n = 2), track and field (n = 2), lacrosse (n = 1),

19

cycling (n = 1), field hockey (n = 1), crew rowing (n = 1), swimming (n = 1), and synchronized

20

swimming (n = 1). The majority (n = 36) of these teams were rep/select teams competing against

21

other teams in their region/district; the competitive levels of the remaining teams included seven

22

university teams, two national teams, two provincial teams, and one professional team. The

23

researcher met with the teams before or after practices at the teams’ practice facilities (between


January 2016 – June 2017) where participants (i.e., athletes within those teams) were asked to

2

complete a demographic form as well as the updated (i.e., based on study 1) 70-item MATS.

3

Data Analysis

14

4

Drawing directly from the a priori conceptual framework by McEwan and Beauchamp

5

(2014), five separate multilevel confirmatory factor analyses (MLCFA) of the data from the 70-item

6

MATS were conducted corresponding to each of the five aspects of teamwork—preparation,

7

execution, evaluation, adjustments, and management of team maintenance (see below for model

8

specification and rationale). Factors and indicators were standardized separately at the athlete and

9

team levels. Within each model, intraclass correlations (ICCs) were calculated to estimate how

10

much variance in each item was observed at the group level. If item-level ICCs are greater than .20,

11

then the clustering of the data should not be ignored (Muthén & Satorra, 1995). We also computed

12

both within-team (i.e., level 1) and between-team (i.e., level 2) ordinal composite reliability (CR)

13

values to examine the internal consistency of each of the 14 subscales of the MATS (Bollen, 1989;

14

Geldhof, Preacher, & Zyphur, 2014; Zumbo, Gadermann, & Zeisser, 2007). Computing these within

15

and between ordinal CR coefficients accounts for the clustering of the data (i.e., athletes within

16

teams) and is, therefore, a more accurate estimator of reliability compared to other estimators, such

17

as Cronbach’s alpha which assumes a single-level factor structure (Geldhof et al., 2014).1 An

18

acceptable indication of reliability is evident if CR values exceed .70 (DeVellis, 2012; Hair, Black,

19

Babin, Anderson, & Tatham, 2006). Finally, we computed Average Variance Extracted (AVE)

20

scores, which measure the convergence among the items in each model. It is recommended that the

21

AVE values for each factor not exceed the squared correlations between that factor and any other

22

variable (Fornell & Larcker, 1981).

23

In accordance with guidelines for conducting MLCFA (Brown, 2006; Muthén, & Muthén,

24

2017), the individual items (i.e., observed variables or indicators) were first specified to load onto


15

1

their theoretically-aligned teamwork dimension (i.e., latent factors). Within the models examining

2

the preparation, execution, and adjustments phases of teamwork, those lower-order latent factors

3

were then specified to load onto a second-order latent factor designed to reflect an omnibus/global

4

representation of each of those phases. Specifically, preparation was modelled (see Figure 1) to be

5

comprised of three lower-order latent variables measuring mission analysis (5 items), goal

6

specification (6 items), and planning (7 items). Execution was also modelled (see Figure 2) to

7

consist of three lower-order latent variables measuring coordination (4 items), cooperation (5

8

items), and communication (5 items). Adjustments was modelled to be comprised of problem

9

solving (5 items), innovation (4 items), intrateam coaching (4 items), and backing up (5 items).

10

Since it is not mathematically possible to test a higher-order structure from two lower-order factors

11

(a minimum of three lower-order factors is required, otherwise the model would be under-

12

identified; Brown, 2006), we specified a first-order measurement model for the evaluation phase

13

(see Figure 3) that included two correlated latent factors related to performance monitoring (6

14

items) and systems monitoring (4 items). Similarly, we specified a first-order measurement model

15

for the management of team maintenance (see Figure 5) that included two correlated latent factors

16

related to conflict management (5 items) and psychological support (5 items).2

17

The factor analyses were carried out in Mplus Version 8 (Muthén & Muthén, 2017). A

18

variety of estimation methods were considered in conducting these analyses, including Weighted

19

Least Squares Means and Variance adjusted (WLSMV), Maximum Likelihood (ML), and Bayesian

20

estimation. While each method has strengths and limitations, we ultimately chose to use the

21

WLSMV method of estimation, as this method performs best when conducting CFA with ordinal

22

data and is also suitable when dealing with a large number of participants (e.g., over 200) and

23

multilevel data structures (Brown, 2006; Muthén, Muthén, & Asparouhov, 2015). Following

24

recommendations provided by Brown (2006) and Jackson and Gillaspy (2009), global model fit was


16

1

evaluated with a variety of fit indices. First, we considered the chi-square test to examine absolute

2

fit. However, several authors have suggested that obtaining a non-significant 𝜒2 value is highly

3

unrealistic (Barrett, 2007; Hu & Bentler, 1995) and, although this value is commonly reported,

4

supplemental indices should be relied on more heavily to examine fit (Brown, 2006). Thus, we

5

computed two indices of absolute fit—Root Mean Square Error of Approximation (RMSEA) and

6

Standardized Root Mean Square Residual (SRMR)—and two indices of comparative fit—

7

Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI). Adequate fit between the hypothesized

8

model and the data occurs when CFI and TLI values are close to or greater than .90 (ideally >.95),

9

SRMR values are close to or below .08, and RMSEA values are less than .08 (ideally