3 Factor Structure and Incremental Utility of the ...

Running head: DISTRESS TOLERANCE SCALE Tables: 3

Factor Structure and Incremental Utility of the Distress Tolerance Scale: A Bifactor Analysis Travis A. Rogers1, Joseph R. Bardeen1, Thomas A. Fergus2, & Natasha Benfer1 1

2

Department of Psychology, Auburn University, United States

Department of Psychology and Neuroscience, Baylor University, United States

NOTICE: this is the author’s version of a work that was accepted for publication in Assessment. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Assessment.

Author Note Correspondence concerning this article should be addressed to Joseph R. Bardeen, Department of Psychology, Auburn University, Auburn, AL 36832. Voice: 334-844-6647; Email: [email protected].

1

DISTRESS TOLERANCE SCALE

2 Abstract

The Distress Tolerance Scale (DTS) is a self-report measure of perceived capacity to withstand aversive emotions. Initial factor analysis of this measure suggested a structure comprising one higher-order factor and four lower-order domain-specific factors. However, there is limited evidence in support of the DTS’s purported multidimensionality, and despite use of the DTS subscales, research has yet to assess their incremental utility. The current investigation sought to rectify the paucity of evidence in support of the DTS’s factor structure and independent use of DTS subscales via bifactor analysis. In the present study (N = 826 community adults), a bifactor model of the DTS provided the best fit to the data. However, an examination of statistical indices associated with bifactor modeling, as well as results from an examination of incremental utility, suggest that the domain-specific factors are largely redundant with the general factor and do not provide incremental utility in predicting relevant clinical constructs beyond the general factor. Measurement invariance between sexes was confirmed. Taken together, results support use of a DTS total score, but not subscale scores. Keywords: emotional distress intolerance, distress tolerance scale, bifactor analysis, psychometrics


3

Factor Structure and Incremental Utility of the Distress Tolerance Scale: A Bifactor Analysis Distress tolerance is broadly defined as one’s capacity to withstand aversive physical and psychological states (Leyro, Zvolesnky, & Bernstein, 2010). Although there is not yet consensus regarding the operational definition of distress tolerance, compelling conceptualizations have been put forward that propose that distress tolerance is made up of several distinct, but related, facets (e.g., uncertainty, physical discomfort; Zvolensky, Vujanovic, Bernstein, & Leyro, 2010). One facet of the broader construct of distress tolerance is emotional distress intolerance (EDI), which reflects the capacity to withstand the distress associated with aversive emotions (Simons & Gaher, 2005). Conceptual models of psychopathology position EDI as a central explanatory factor in the pathogenesis and maintenance of a wide variety of pathological presentations (Leyro, Zvolensky, & Bernstein, 2010; Zvolensky & Otto, 2010), including substance use disorders, disordered eating, and posttraumatic stress (Bardeen & Fergus, 2016; Bernstein, Vujanovic, Leyro, & Zvolensky, 2011; Vujanovc, Bernstein, & Litz, 2011). Moreover, EDI has been identified as a potentially important mechanism of change in psychological interventions for a wide variety of symptom presentations (see Zvolensky, Bernstein, & Vujanovic, 2011), thus resulting in the development of clinical interventions that directly target EDI in the service of symptom relief (e.g., Dialectal Behavior Therapy [DBT]; Linehan, 1993). Given EDI’s transdiagnostic status and clinical utility, it is important to develop psychometrically sound measures of this construct. The assessment of EDI has garnered substantive interest in the extant literature, including through the development of self-report and behavioral methods. As reviewed by Leyro et al. (2010), Simons and Gaher's (2005) self-report measure, known as the Distress Tolerance Scale (DTS), is the most commonly used index of EDI.1 Since Leyro et al.'s review, McHugh and Otto


4

(2012) developed a self-report measure that includes items from the DTS and other measures in an attempt to refine the assessment of the construct. A limitation surrounding McHugh and Otto's (2012) measure is that theoretically (e.g., Leyro et al., 2010) and empirically (e.g., Bardeen, Fergus, & Orcutt, 2013b) distinct components of the distress tolerance construct are lumped together to produce a composite score, thereby limiting the use of this measure as an index of EDI per se. Veilleux, Pollert, Zielinski, Shaver, and Hill (in press) developed a behavioral measure of EDI. Because behavioral methods often assess more "state-like" tendencies and selfreport measures often assess more "trait-like" tendencies, Veilleux et al.'s (in press) behavioral measure holds promise for offering a complementary approach to the DTS for assessing EDI. Overall, additional research is needed to identify optimal ways to assess EDI and, because of its widespread use, further examination of the DTS is warranted, particularly related to the factor structure and resulting operationalization of the DTS item scores. The DTS was originally validated in two large, independent samples of university students. Specifically, Simons and Gaher (2005) generated a pool of 16 items to be consistent with their conceptualization of EDI. These items were submitted to an exploratory factor analysis (EFA). Fifteen items were retained for further testing, and the EFA supported a four-factor solution. Confirmatory factor analysis (CFA) supported a hierarchical structure, in which four lower-order factors loaded onto a higher-order factor. The lower-order factors of the DTS purportedly reflect (1) the perceived capacity to tolerate or endure negative emotions (i.e., Tolerance; “Feeling distressed or upset is unbearable to me”), (2) the extent to which attention is occupied by emotional distress (i.e., Absorption; “When I feel distressed or upset, all I can think about is how bad I feel”), (3) the subjective appraisal of negative emotions (i.e., Appraisal; “My


5

feelings of distress or being upset are not acceptable”), and (4) efforts aimed at managing emotional distress (i.e., Regulation; “I’ll do anything to stop feeling distressed or upset”). Psychometric support for scores on the 15-item measure has been provided, including evidence of internal consistency for the total score (α range = .82 – .91) and subscale scores (α range = .72 – .82; Leyro, Bernstein, Vujanovic, McLeish, & Zvolensky, 2011; Simons & Gaher, 2005), as well as retest reliability for the total score over a six-month interval (r = .61; Simons & Gaher 2005). In addition, the DTS has been used across multiple domains of psychological research, and evidence has accumulated in favor of (a) its concurrent validity (Bernstein, Marshall, & Zvolensky, 2011; Simons & Gaher 2005), and (b) incremental utility in accounting for variance within relevant clinical constructs (e.g., symptoms of bulimia, impulsivity) beyond that explained by anxiety, depression, and other self-report and behavioral measures related to EDI (e.g., Anestis et al., 2012). However, some evidence calls the validity of the subscales into question. Cougle, Bernstein, Zvolensky, Vujanovic, and Mcatee (2013) found that the DTS total score provided a significant incremental contribution in predicting tolerance and perceived threat of anger, fear, sadness, and disgust in response to emotionally evocative film clips, even after accounting for sex, emotional intensity, and anxiety sensitivity. In contrast, DTS subscales exhibited inconsistent associations with constructs to which they should be theoretically related. For example, the DTS Tolerance subscale was incrementally associated with perceived threat and tolerance for the anger film clip, but not for three other film clips used to elicit sadness, fear, and disgust. Such results run counter to the generally supported validity of these sub-domains of emotional distress intolerance. Cougle et al.’s (2013) results notwithstanding, the extant literature has been largely supportive of the utility of the DTS total and subscale scores. However, the structure of this


6

measure has yet to be adequately examined (see Table 1). Leyro et al. (2011) provided the only known CFA of the DTS after its initial publication and found several areas of possible model strain that raise concerns about (a) the conceptual distinctions between the DTS factors (i.e., multiple potential cross loadings, strong inter-factor correlations [rs > .80]), and (b) the underperformance of some DTS items relative to others (i.e., standardized item-factor loading range = .29 – .86). Furthermore, Leyro et al. used a 14-item version of the DTS that was developed prior to Simons and Gaher’s (2005) publication. Specifically, item 14 of the Regulation subscale (“When I feed distressed or upset, I must do something about it immediately”) was omitted. Leyro et al. also found that correlations between three of the DTS subscales (Tolerance, Absorption, and Appraisal) were strong enough (rs > .80) to raise concerns about the conceptual distinctions between them. Following Leyro et al.’s (2011) CFA, Hsu, Collins, and Marlatt (2013) examined the factor structure of the DTS via principal component analysis (PCA). Contrary to Simons and Gaher’s (2005) original publication, Hsu et al.’s results yielded one unitary factor, suggesting that the DTS might be better modeled as a unidimensional measure of EDI. It should also be noted that PCA is primarily a data reduction technique (see Abdi & Williams, 2010). Thus, Hsu et al.’s results are less likely to generalize to other factor analytic work (see Suhr, 2005). Put succinctly, these limited investigations leave a number of questions about the structure of the DTS unanswered. The paucity of empirical evidence in support of the factor structure of the DTS remains a significant limitation to future work with this measure. Studies using the DTS have tended either to use the DTS total score as the sole index of EDI (e.g., Anestis, Bagge, Tull, & Joiner, 2011; Anestis et al., 2012; Bernstein, Marshall, et al., 2011; Vujanovic, Bonn-Miller, Potter, Marshall,


7

& Zvolensky, 2011) or to interpret both the total and subscale scores (e.g., Ameral, Palm Reed, Cameron, & Armstrong, 2014; Leyro et al., 2011; Simons & Gaher, 2005; Shorey et al., 2017; Timpano, Buckner, Richey, Murphy, & Schmidt, 2009). These approaches make at least two assumptions about the nature of the DTS. First, use of a total score assumes that each DTS domain-specific factor represents an aspect of the same overarching general construct. Second, use of subscale scores assumes that the lower-order factors capture unique information beyond the total score. However, the existing literature base for the DTS has yet to adequately address the tenability of these two approaches. While investigation of the purported hierarchical structure of the DTS is useful for examining the first of the two assumptions, it cannot directly test the assumption that items of domain-specific factors provide unique information beyond a higherorder or general factor (Reise, 2012). Such analyses are necessary because, as noted, a number of investigations have used DTS subscales and found differential relations with constructs of interest. Thus, it is important to use an analytic approach that can simultaneously address both of these assumptions (i.e., bifactor analysis). Bifactor models permit investigation of the presence of a general factor and the degree to which each domain-specific factor is meaningfully distinct from the general factor (Reise, Bonifay, & Haviland, 2013; Reise, Moore, & Haviland, 2010; Rodriguez, Reise, & Haviland, 2016). Moreover, bifactor models have demonstrated good fit to transdiagnostic self-report measures in prior research (e.g., Ebesutani, McLeish, Luberto, Young, & Maack, 2014; Ebesutani et al., 2011). To our knowledge, the DTS has yet to be examined via a bifactor modeling approach. Such an analysis is important and would likely yield new insights into the performance of the DTS. For example, redundancy between a general factor and domain-specific factors may suggest that a total score should be used in lieu of subscale scores (Reise, 2012).


8

Therefore, the primary aim of the current investigation was to examine the fit of a bifactor model of the DTS in a large adult sample. Consistent with standard methodology for comparing model fit (Brown, 2015; Kline, 2016), we compared the fit of a bifactor model to competing models (i.e., a correlated four-factor model, a one-factor model, and a hierarchical model). Some have suggested that the utility of comparing bifactor models to alternative models is limited because bifactor models often provide better fit to the data due to their inherent qualities (e.g., high flexibility; Bonifay, Lane, & Reise, 2017; Reise, Kim, Mansolf, & Widaman, 2016). As such, a number of additional statistical indices, developed for use with the bifactor approach, were examined (Rodriguez, Reise, & Haviland, 2016). These statistics provide information central to the aims of the present study (e.g., determining the degree to which domain-specific factors have value beyond the general factor, determining the stability and replicability of the factors). An additional benefit of bifactor modeling is that the relation between domain-specific factors and criterion variables can be examined while holding the general factor constant (Brown, 2015). Thus, we also examined the incremental utility of the DTS domain-specific factors in predicting theoretically relevant constructs (i.e., depression, anxiety, and stress) after accounting for a general EDI factor. Depression, anxiety, and stress were selected as outcome variables because research suggests that emotional distress intolerance plays an important role in the etiology and maintenance of emotional disorders (e.g., Bernstein, Vujanovic, et al., 2011; McHugh et al., 2014). Another unaddressed question regarding the structure of the DTS relates to whether it is similar between males and females. In the original validation study, females reported significantly poorer tolerance for emotional distress than males (Cohen’s d = 0.32; Simons & Gaher, 2005). This effect remained significant in a regression analysis partialling out the shared


9

variance between sex and negative affectivity, indicating that mean differences in DTS total scores could not be attributed to differences in negative affect. This difference may reflect either a true difference in emotional distress tolerance between males and females or may be a function of item content that promotes differential responding. Significant mean differences across sexes have been reported in other investigations (e.g., Cougle et al., 2013; Dennhardt & Murphy, 2011) and have indicated that females tend to report poorer emotional distress tolerance than males. An investigation of sex invariance within the DTS can speak to the potential need for differential scoring and modeling of DTS items among females and males. For example, if the loadings of DTS items on a general factor and domain-specific factors differ as a function of sex, it may be possible that certain DTS domain-specific factors are independent and incrementally useful in one sex but not the other. Therefore, in addition to an examination of (a) the purported multidimensionality of the DTS and (b) the viability of using the DTS domain-specific factors in predicting clinically relevant constructs, the present study also sought to investigate (c) measurement invariance of DTS items across males and females. Methods Participants and Procedure One thousand and nine participants were recruited using Amazon’s Mechanical Turk (MTurk), an online labor market where individuals can participate in research in exchange for financial compensation. Previous research has supported the use of MTurk samples, citing their reliability and diverse demographics relative to other online and undergraduate samples (e.g., Behrend, Sharek, Meade, & Wiebe, 2011; Buhrmester, Kwang, & Gosling, 2011; Chandler & Shapiro, 2016). To minimize the effect of random responding or inattentiveness, three catch questions (e.g., “Select ‘Much’ if you are paying attention right now”) were imbedded in the


10

online survey (Oppenheimer, Meyvis, & Davidenko, 2009; Paolacci, Chandler & Ipeirotis, 2010). Participants who answered fewer than two of the three catch questions correctly (n = 144; 14.27% of original sample) were excluded from analyses (for precedence, see Bardeen, Fergus, Hannan, & Orcutt, 2016; Bardeen, Fergus, & Orcutt, 2013a, 2013b; Bardeen & Michel, 2017). Additionally, 39 participants (3.87% of the original sample) who failed to provide responses for the majority of DTS items (eight out of fifteen [at least 50%; see Graham & Schafer, 1999]) were removed from the sample. The proportion of missing cases within all variables of interest was ≤ 1%, and covariance coverage was high across all variables, ranging from .985 to 1.00. The final sample (N = 826) was 60.5% female. Five participants elected not to report their sex. The average age of the sample was 33.68 years (SD = 12.54, range = 18 to 73). Concerning race, 81.6% of the final sample identified as White, 6.5% as Black, 6.3% as Asian, 1.0% as American Indian or Alaskan Native, 0.1% as Native Hawaiian or Pacific Islander, 2.8% as “Other,” and 1.7% preferred not to respond. Additionally, 89.1% of the final sample identified as NonHispanic, 6.8% as Hispanic, and 4.1% preferred not to report ethnicity. All study procedures were approved by the local institutional review board. Informed consent and self-report measures could be completed from any computer with internet access. The median time to complete the online battery was 29.19 minutes. Upon completion, participants were compensated with $0.50, an amount consistent with compensation provided to MTurk workers who completed studies of similar length (e.g., Buhrmester et al., 2011). Measures Distress Tolerance Scale (DTS). As described, the DTS (Simons & Gaher 2005) is a 15item measure designed to assess individual differences in EDI. Items are rated on a 5-point scale from 1 (Strongly agree) to 5 (Strongly disagree). Higher total scores are indicative of greater


11

tolerance for emotional distress, while lower scores reflect a relative intolerance of emotional distress. The Depression Anxiety Stress Scales – 21-item version (DASS-21). The DASS-21 (Lovibond & Lovibond, 1995) is a 21-item measure with subscales assessing symptoms of depression, anxiety, and stress during the past week. Each subscale comprises seven items rated on a 4-point Likert scale ranging from 0 (Did not apply to me at all) to 3 (Applied to me very much, or most of the time); thus, higher scores suggest greater symptomatology. The DASS-21 has evidenced good psychometric properties in a number of studies (Bardeen, Fergus, & Orcutt, 2014; Henry & Crawford, 2005; Lovibond & Lovibond, 1995) and has evidenced strong convergent validity with other measures of depression and anxiety (Antony, Bieling, Cox, Enns, & Swinson, 1998). The DASS-21 subscales evidenced good internal consistency in the current sample, with Cronbach’s αs of .93, .90, and .86 for depression, stress, and anxiety, respectively. Data Analytic Strategy Confirmatory Factor Analysis. The following four models were examined via CFA. The first model was a correlated four-factor model proposed by Simons and Gaher (2005), with three items loading onto the Tolerance factor, three items loading onto the Absorption factor, six items loading onto the Appraisal factor, and three items loading onto the Regulation factor. Correlations between all four factors were modeled, and no item had a secondary loading. Standard procedures for conducting model comparisons, especially for measures that produce a total score, requires examination of a one-factor model to ensure that a unidimensional model does not provide better fit to the data (Brown, 2015; Reise et al., 2010). As such, the second model was a one-factor model, within which all items of the DTS loaded onto one factor. The third model was a hierarchical model, in which correlations between factors were omitted and


12

direct pathways were modeled from a higher-order factor onto each domain-specific factor (i.e., identical to that proposed in the original DTS publication). The fourth model was a bifactor model, in which all 15 items loaded onto a general factor as well as onto their respective domainspecific factor, and covariance terms between all factors were held to zero (Brown, 2015). Model Estimation and Comparison. All models were tested using MPlus 7.4 (Muthén & Muthén, 2015). Missing data were handled using full information maximum likelihood, which is the default method for imputing missing data in MPlus. Robust maximum likelihood (MLR) estimation was used to test all models, as MLR is robust to violations of the assumption of normality (Brown, 2015; Kaplan, 2009). Therefore, all reported chi-square (χ2) values represent the Satorra-Bentler scaled χ2 (Satorra & Bentler, 1994). Four commonly recommended fit statistics were used to evaluate the fit of each model to the data: the comparative fit index (CFI), the Tucker-Lewis fit index (TLI), standardized root mean square residual (SRMR), and root mean square error of approximation (RMSEA; Brown, 2015; Kline, 2016). The following guidelines were used to evaluate model fit: CFI and TLI should be near .95; SRMR should be less than .08 (Hu & Bentler, 1999); RMSEA should be near .06, and the upper limit of the 90% RMSEA confidence interval (CI) should not exceed .10 (Kline, 2016). In addition to an evaluation of model fit, model comparisons were evaluated by way of scaled χ2 difference testing. However, due to the sensitivity of χ2 distributions to large samples, χ2 difference testing might suggest significant differences between models when differences in the magnitude of parameter estimates are trivial (Brown, 2015; Kline, 2016). Thus, alternative tests for comparing model fit were used, including an examination of differences in sample-size corrected Akaike Information Criterion (AICc; Akaiki, 1974; Sugiura, 1978), which provides a more quantitative rather than qualitative (e.g., better/worse) estimate of model comparison. Additionally, RMSEA


13

90% CIs were examined. Relative differences in model fit were considered meaningful if models differed in AICc by 10 or more (see Burnham, Anderson, & Huyvaert, 2011, for a discussion), and had non-overlapping RMSEA 90% CIs (e.g., Wang & Russell, 2005). Bifactor Model Evaluation. In addition to a basic examination of fit statistics between models, the following statistical indices were examined to further evaluate the bifactor model (see Rodriquez et al., 2016). All additional bifactor model indices were calculated using the bifactor indices calculator provided by Dueber (2016). OmegaH (ωH) reflects the proportion of variance in DTS scores attributable to a general factor, while OmegaHS (ωHS) reflects the proportion of variance in scores explained by each domain-specific factor after removing variance explained by the general factor. Both ωH and ωHS are best understood as indices of factor reliability. Explained common variance (ECV) is calculated by dividing variance attributable to the general factor by variance attributable to both general and specific factors; thus, ECV serves as a better index of unidimensionality within a measure than ωH. Percentage of uncontaminated correlations (PUC) is interpreted along with ECV and serves as an indicator of the percentage of DTS item correlations contaminated by variance attributed to the general and domain-specific factors. Item-level ECV (I-ECV) indicates the amount of common variance from each item that is attributable to the general factor (Stucky, Thissen, & Edelen, 2013), thus serving as an index of unidimensionality at the item level. Construct replicability (H) reflects the degree to which a factor is well defined by its indicators. Factor determinacy (FD) is the correlation between factor scores and the factors and serves as an indication of whether factor scores are valid for independent use. Finally, average relative parameter bias (ARPB) serves as an indication of bias across parameters when items are forced into a unidimensional structure.


14

Structural Regression Model. A structural regression model was used to examine whether the domain-specific factors of the DTS relate to clinically relevant constructs when simultaneously accounting for a general emotional distress intolerance factor. Specifically, the general factor and domain-specific factors from the bifactor model were simultaneously regressed onto the three uncorrelated domains of the DASS-21. Domain-specific factors from the bifactor model that exhibited clear redundancy with the general factor (e.g., non-significant residual variance) were removed from the model. Path coefficients from the general factor and remaining domain-specific factor(s) to each of the DASS scales were freely estimated. Measurement Invariance. A multiple-group CFA was used to test measurement invariance, with restrictive models testing for (1) configural invariance (i.e., equal structural form), (b) metric invariance (i.e., equal factor loadings), and (c) scalar invariance (i.e., equal indicator intercepts) between males (n = 321) and females (n = 500). To test configural invariance, the adequacy of the final DTS bifactor structure was examined independently and simultaneously in both sexes. The test of metric invariance constrained the factor loadings within the final DTS bifactor model to equality across both sexes. Scalar invariance was examined by constraining indicator intercepts to equality across sexes. Because χ2 is sensitive to large samples and interpretation of multiple-group CFA results is complicated by unequal group sizes (Brown, 2015), RMSEA 90% CIs and CFI were also examined between invariance models. Decrements in model fit were considered meaningful when parameter constraints resulted in either (a) nonoverlapping RMSEA 90% CIs or (b) changes in CFI (∆CFI) greater than .01 (Cheung & Rensvold, 2002). Results


15

Model Estimation and Comparison. Goodness-of-fit statistics from the measurement models are presented in Table 2. The correlated four-factor model evidenced adequate fit to the data. CFI and TLI approximated the specified criteria of adequate model fit, and RMSEA, RMSEA 90% CIs, and SRMR of the four-factor model indicated good fit. As the one-factor model is nested within the correlated four-factor model (Brown, 2015), the one-factor model was examined next for comparative purposes. Relative to the four-factor model, the one-factor model provided significantly worse fit to the data, as evidenced by a significant change in χ2, ∆χ2(6) = 296.66, p < .001, differences in AICc of 528.70, and non-overlapping RMSEA 90% CIs. The hierarchical model was fitted to the data next. CFI and TLI approximated the specified criteria of adequate model fit, and RMSEA, RMSEA 90% CIs, and SRMR indicated good fit. In comparison to the correlated four-factor model, there was a significant change in χ2, ∆χ2(2) = 13.15, p < .01, and meaningful change in AICc of 15.23. In contrast, RMSEA 90% CIs of the correlated four-factor and hierarchical models were overlapping. The bifactor model provided adequate to good fit to the data; all fit indices met or approximated specified criteria. Hierarchical models are nested within bifactor models and therefore can be directly compared (Brown, 2015). χ2, AICc values, and RMSEA 90% CIs were compared across models. χ2 difference testing indicated a significant difference between the fit of the (a) correlated four-factor and bifactor models, ∆χ2(9) = 42.78, p < .001, (b) one-factor and bifactor models, ∆χ2(15) = 347.00, p < .001, and (c) hierarchical and bifactor models, ∆χ2(11) = 55.46, p < .001. Differences in AICc values between the correlated four-factor and bifactor models, ∆AICc = 53.91, between the one-factor and bifactor models, ∆AICc = 582.61, and between the hierarchical and bifactor models, ∆AICc = 69.14, also favored the bifactor model. However, RMSEA 90% CIs were overlapping between all models. As two of three comparative


16

indices suggested better fit in the bifactor model, it was retained for further examination. The standardized factor loadings from the bifactor model are presented in Table 3. While nine of 15 items exhibited significant factor loadings (p < .05) onto their domain-specific factors, factor loadings of all items of the Absorption factor and items 6, 9, and 10 of the Appraisal factor were rendered non-significant in the bifactor model. In contrast, all items loaded significantly onto the general factor. Redundancy between the Tolerance and Absorption factors and the general factor was evidenced by non-significant residual variances within the bifactor model. The Appraisal and Regulation factors were the only domain-specific factors to exhibit significant residual variance. Bifactor Model Evaluation. Additional indices derived from the bifactor model are presented in Table 3. According to Stucky and Edelen (2015), I-ECV values greater than .80 or .85 indicate unidimensionality at the item level. The majority of DTS items (nine out of 15) demonstrated I-ECV values greater than .80. Within the domain-specific factors, two out of three Tolerance items, two out of three Absorption items, and five out of six Appraisal items, evidenced content redundancy with the general factor. In contrast, the Regulation factor was the only domain-specific factor whose items demonstrated domain-specific content, as evidenced by I-ECV values ≤ .72. As reviewed by Rodriguez et al. (2016), ω and ωS serve as indices of the reliability of the general and domain-specific factors, respectively. Adequate reliability was demonstrated by both the general factor (ω = .95) and domain-specific factors (ωS range = .83 – .88). The general factor explained a substantial amount of variance in DTS scores (ωH = .91). However, after partitioning out variance explained by the general factor, the domain-specific factors accounted for a small proportion of variance within their respective items (ωHS range = .07 – .35).


17

According to Rodriguez et al. (2016), ECV and PUC reflect the degree to which parameter estimates are biased when multidimensional constructs are forced into a unidimensional model. When ECV and PUC are greater than .70, common variance within a model can be regarded as essentially unidimensional. Within the bifactor model, ECV of the general factor was .80, and PUC was .77, supporting a unidimensional rather than multidimensional conceptualization of the DTS. When H is high (H > .70; see Hancock & Mueller, 2001), a latent variable may be considered well defined by its items, and thus, will demonstrate greater stability across studies. Within the bifactor model, the general factor evidenced acceptable construct replicability, H = .95. However, no domain-specific factor demonstrated adequate construct replicability, with H values ranging from .25 to .55. Moreover, Gorsuch (1983) recommends that factor score estimates should be used only when FD is greater than .90. Per this criterion, the general factor estimate may be interpreted, with FD = .96. However, no domain-specific factor met Gorsuch’s guideline, with FD values from domainspecific factors ranging from .67 to .84. Lastly, ARPB of the bifactor model was .03, suggesting minimal differences between parameter estimates in a unidimensional and bifactor solution. ARPB less than 0.10 or 0.15 suggests that multidimensionality within a measure is not substantial enough to negate the tenability of a unidimensional solution (Muthén, Kaplan, & Hollis, 1987; Rodriquez et al., 2016). Modification indices derived from the bifactor model yielded 12 potential item crossloadings (i.e., expected change in parameter estimates of > 10; Brown & Moore, 2012; Muthén & Muthén, 2015). Specifically, allowing all three Tolerance items, all three Absorption items, two of the Appraisal items (items 7 and 10), and item 13 of Regulation to cross-load onto another domain-specific factor would substantially improve model fit. Modification indices also


18

suggested that model fit would benefit from allowing three items (2, 7, and 15) to cross-load on three factors. These results suggest that DTS items may reflect unidimensional rather that domain-specific content. Structural Regression Model. Because three of the four DTS domain-specific factors (Tolerance, Absorption, and Appraisal; see Table 3) exhibited poor performance within the bifactor model (i.e., non-significant residual variance and/or a preponderance of items that exhibit content redundancy with the general factor), only the Regulation factor and general factor were included as predictors in the structural regression.2 The structural regression model provided adequate fit to the data, with all fit indices approximating specified guidelines: χ2(585) = 1981.16, p < .001; RMSEA = .05 (90% CIs = .051 – .056); CFI = .91; TLI = .90; SRMR = .05. The general factor significantly predicted all outcome variables at p < .001, with standardized coefficient estimates in the expected direction: depression, β = -0.46; anxiety, β = -0.38; stress, β = -0.41. After accounting for the general factor, the Regulation factor also significantly predicted all outcome variables at p < .001; however, all standardized coefficient estimates were in a theoretically inconsistent direction: depression, β = 0.76; anxiety, β = 0.83; stress, β = 0.84.3 In order to test for possible net suppression effects, the structural regression was modified, such that depression, anxiety, and stress were regressed onto only the Regulation domain-specific factor (i.e., the general factor was modeled but did not serve as a predictor of these outcomes). This second structural regression yielded a pattern of results in opposition to the first. Significant negative associations between the Regulation factor and criterion variables (depression, β = -0.86; anxiety, β = -0.91; stress, β = -0.94, ps < .001) suggest that the unexpected positive relationships between the Regulation domain-specific factor and these clinical outcomes, when controlling for the general factor, most likely represent a suppression


19

effect. Further, the high standardized beta weights between this domain-specific factor and criterion variables suggests construct overlap.4, 5 Measurement Invariance. The final DTS bifactor model (the general factor and the Regulation domain-specific factor) was retained for an analysis of measurement invariance. Fit indices of the configural, metric, and scalar invariance models are displayed in Table 2. The bifactor model exhibited adequate fit in both males and females when modeled separately and simultaneously (i.e., evidence of configural invariance). While there was a change in AICc greater than 10, no significant decrement in model fit was observed between the configural and metric models as evidenced by Δχ2 (Δχ2[16] = 14.56), overlapping RMSEA 90% CIs, and ∆CFI of .002. χ2 differences between the metric and scalar models (Δχ2[13] = 41.76) and the scalar and configural models (Δχ2[29] = 58.70) were significant at p < .001. However, RMSEA CIs across all invariance models overlapped, and ∆CFI did not exceed Cheung and Rensvold’s (2002) guideline. Thus, overlapping RMSEA CIs and ∆CFI < .01 were interpreted as evidence of scalar invariance between males and females. In addition, latent mean scores of the DTS general factor and Regulation domain-specific factor were compared between sexes. Significant differences were not observed when comparing the latent general factor (difference = 0.11, p = .14) and latent Regulation factor means (difference = 0.06, p = .17) between males and females. Discussion The primary aim of this investigation was to address the paucity of empirical evidence in support of the multidimensional structure of the DTS. Consistent with standard methodology for comparing model fit (Brown, 2015; Kline, 2016), we compared the fit of a bifactor model of the DTS to a correlated four-factor model, a one-factor model, and a hierarchical model, and thereby moved beyond the measurement models examined in previous studies. The adequacy of the


20

hierarchical model proposed in the original publication of the DTS (Simons & Gaher, 2005) was replicated in this study; however, the hierarchical model and correlated four-factor model did not differ in their fit to the data. The bifactor model of the DTS, in which all indicators loaded onto a general factor in addition to their respective orthogonal domain-specific factors, evidenced better fit relative to all other tested models. In evaluating the bifactor model further, item loadings onto the domain-specific factors were substantially attenuated, and residual variance of two of the four domain-specific factors (i.e., Tolerance and Absorption) was rendered non-significant after accounting for the general factor. Moreover, the majority of DTS items demonstrated content redundancy at the item level. The general factor accounted for most of the variance in DTS scores, and no domain-specific factor met specified benchmarks of adequate construct replicability or factor determinacy. Succinctly, results provide support for a strong general factor within the DTS but do not support the use of purported domain-specific factors. This investigation also sought to test the incremental utility of the DTS domain-specific factors in predicting clinical constructs after accounting for the general factor. To that end, a structural regression model, in which depression, anxiety, and stress served as outcome variables, was performed. The Regulation factor was the only domain-specific factor whose residual variance remained significant in the bifactor model and whose items could be regarded as providing enough unique variance beyond that of the general factor to be considered a separate domain. Thus, only the Regulation factor and the general factor served as predictors in the structural regression. The general factor significantly predicted depression, anxiety, and stress, indicating that emotional distress intolerance shares a significant relationship with these clinical outcomes.


21

After accounting for the general factor, the Regulation factor of the DTS also significantly predicted depression, anxiety, and stress, but in a theoretically inconsistent direction. These unexpected results necessitated an examination of net suppression. While domain-specific factors cannot exhibit suppression effects on other domain-specific factors due to their orthogonal specification in the bifactor model, content redundancy may still emerge from the simultaneous loadings of all items on both the general factor and respective domain-specific factors. In other words, the Regulation factor may still have limited practical value on its own. As reviewed by Paulhus, Robins, Trzesniewski, and Tracy (2004), suppression and/or redundancy effects can occur in regression models when multiple predictor variables with strong correlations are entered simultaneously (for an example of the effect of suppression in a bifactor framework, see Patrick, Hicks, Nichol, & Krueger, 2007). All latent factors of the DTS have evidenced strong, positive correlations in past studies (e.g., Ameral et al., 2014; Leyro et al., 2011), and standardized correlations between the domain-specific factors in the current study were also quite high, falling within a range of .72 to .94. A second structural regression in which variance from the DTS general factor was not controlled for yielded a pattern of results in opposition to the first regression, providing strong evidence for net suppression. Further, the magnitude of the associations between the Regulation domain-specific factor and our outcome variables suggests content redundancy. That is, the Regulation factor likely has little to no incremental utility beyond the general factor in predicting depression, anxiety, or stress, because it may in fact be redundant with measures of these clinically relevant outcomes. Results such as these are not new to researchers of individual differences (see Wolgast, 2014). This study provided the first investigation of measurement invariance in the DTS across males and females. Past research has suggested that females may demonstrate lower scores on


22

this measure (i.e., poorer tolerance of emotional distress) than males (Cougle et al., 2013; Dennhardt & Murphy, 2011). To our knowledge, however, no other investigation has sought to explore whether this difference reflects a true deficit in emotional distress tolerance in females or measure-specific variance of the DTS items across sexes. Our results provide evidence that the DTS exhibits adequate measurement invariance across males and females and supports the use of identical scoring and modeling of the DTS in both sexes. Contrary to past findings, no significant differences were found in the latent means of the DTS’s general factor or Regulation domainspecific factor between sexes. Results of the current investigation support a unidimensional conceptualization of the DTS, contraindicate independent use of the DTS domain-specific factors, and provide evidence of measurement invariance across males and females. However, it should be noted that fit of the unidimensional model was relatively poor, suggesting a potential need for improvements to this commonly used measure of emotional distress intolerance. For instance, given that modification indices suggested multiple cross loadings, and that two of the four domain-specific factors exhibited non-significant residual variance after modeling the general factor (i.e., redundant item content), future investigations might prioritize refining the DTS by removing truly redundant and underperforming items. Refinement of the DTS in this way would likely result in a shorter, more parsimonious measure with greater ease of interpretability and potentially better unidimensional model fit. Results should be viewed in light of certain limitations. First, although research supports the use of MTurk for data collection (Chandler & Shapiro, 2016), and a substantial amount of literature exists to aid researchers in quality-control procedures and other issues related to online data collection (see Paolacci et al., 2010, for a discussion), MTurk samples should not be


23

considered representative of the general population. For example, evidence suggests that MTurk samples tend to be more highly educated and younger than the general population (Paolacci & Chandler, 2014). As such, replication of study results in both general population and clinical samples, with more individuals scoring at the high end of the continuum of anxiety and depression, will help to ensure that results generalize. Second, the current sample, though large, comprised a female majority and was relatively homogeneous in terms of race and ethnicity. The acceptability of online sampling notwithstanding, future research with the DTS would do well to examine the psychometric performance of this measure and to cross-validate its structure in samples with greater diversity. Third, the current investigation was cross-sectional in design. Future research would benefit from longitudinal data and experimental paradigms that allow researchers to better characterize the relationship between emotional distress intolerance and relevant clinical constructs. For example, research designs that recruit participants prior to developmentally sensitive periods (e.g., late adolescence, early 20s) and follow such participants across time would be better suited to examine the prospective relationship between emotional distress intolerance and the psychogenesis of various forms of psychopathology. Further, in line with a diathesis-stress model, longitudinal designs across multiple time points could help to expound on past findings related to emotional distress intolerance and the occurrence of stressful life events (e.g., traumatic exposure and the development of PTSD). Lastly, an additional limitation of the current study pertains to the growing body of research on distress tolerance as a multifaceted, transdiagnostic individual difference factor in the etiology and maintenance of psychopathology. As reviewed by Zvolensky, Vujanovic, Bernstein, and Leyro (2010), distress tolerance is believed to comprise five facets of experience that elicit psychological distress: uncertainty, ambiguity, frustration, physical discomfort, and


24

emotional distress (for empirical support for this conceptualization, see Bardeen et al., 2013b). The DTS was developed to reflect only emotional distress intolerance, and thus, represents only a part of the expansive work on the broader construct of distress tolerance. The purpose of this study was to evaluate the psychometric performance of the DTS within different models and across sexes. In doing so, we have contributed to this expansive literature base, but must acknowledge the narrowness of our focus relative to the breadth of the distress tolerance literature.


25 References

Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary Aeviews: Computational Statistics, 2(4), 433-459. Akaiki, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723. Ameral, V., Palm Reed, K. M., Cameron, A., & Armstrong, J. L. (2014). What are measures of distress tolerance really capturing? A mixed methods analysis. Psychology of Consciousness: Theory, Research, and Practice, 1, 357-369. Anestis, M. D., Bagge, C. L., Tull, M. T., & Joiner, T. E. (2011). Clarifying the role of emotion dysregulation in the interpersonal-psychological theory of suicidal behavior in an undergraduate sample. Journal of Psychiatric Research, 45, 603-611. Anestis, M. D., Lavender, J. M., Marshall-Berenz, E. C., Gratz, K. L., Tull, M. T., & Joiner, T. E. (2012). Evaluating distress tolerance measures: Interrelations and associations with impulsive behaviors. Cognitive Therapy and Research, 36, 593-602. Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychological Assessment, 10, 176-181. Bardeen, J. R., & Fergus, T. A. (2016). Emotional distress intolerance, experiential avoidance, and anxiety sensitivity: The buffering effect of attentional control on associations with posttraumatic stress symptoms. Journal of Psychopathology and Behavioral Assessment, 38, 320-329.


26

Bardeen, J. R., Fergus, T. A., Hannan, S. M., & Orcutt, H. K. (2016). Addressing psychometric limitations of the Difficulties in Emotion Regulation Scale through item modification. Journal of Personality Assessment, 98, 298–309. Bardeen, J. R. Fergus, T. A., & Orcutt, H. K. (2013a). Experiential avoidance as a moderator of the relationship between anxiety sensitivity and perceived stress. Behavior Therapy 44, 459-469. Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2013b). Testing a hierarchical model of distress tolerance. Journal of Psychopathology and Behavioral Assessment, 35, 495-505. Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2014). The moderating role of experiential avoidance in the prospective relationship between anxiety sensitivity and anxiety. Cognitive Therapy and Research, 38, 465-471. Bardeen, J. R., Kumpula, M. J., & Orcutt, H. K. (2013). Emotion regulation difficulties as a prospective predictor of posttraumatic stress symptoms following a mass shooting. Journal of Anxiety Disorders, 27, 188-196. Bardeen, J. R., & Michel, J. S. (2017). The buffering effect of religiosity on the relationship between intolerance of uncertainty and depressive symptoms. Psychology of Religion and Spirituality, 9, 90-95. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813. Bernstein, A., Marshall, E. C., & Zvolensky, M. J. (2011). Multi-method evaluation of distress tolerance measures and construct(s): Concurrent relations to mood and anxiety psychopathology and quality of life. Journal of Experimental Psychopathology, 2(3), 386-399.


27

Bernstein, A., Vujanovic, A. A., Leyro, T., & Zvolensky, M. J. (2011). Distress tolerance: Research synthesis and future directions. In M. J. Zvolensky, A. Bernstein, & A. Vujanovic (Eds.), Distress tolerance: Theory, research, and clinical applications (pp. 263-280). New York, NY: Guilford Press. Bonifay, W., Lane, S. P., & Reise, S. P. (2017). Three concerns with applying a bifactor model as a structure of psychopathology. Clinical Psychological Science, 5, 184-186. Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York, NY: Guilford Press. Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011). AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. Behavioral Ecology and Sociobiology, 65, 23-35. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5. Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53-81. Cheung, C. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. Corstorphine, E., Mountford, V., Tomlinson, S., Waller, G., & Meyer, C. (2007). Distress tolerance in the eating disorders. Eating Behaviors, 8, 91-97. Cougle, J. R., Bernstein, A., Zvolensky, M. J., Vujanovic, A. A., & Macatee, R. J. (2013). Validation of self-report measures of emotional and physical distress tolerance. Journal of Psychopathology and Behavioral Assessment, 35, 76-84.


28

Dennhardt, A. A., & Murphy, J. G. (2011). Associations between depression, distress tolerance, delay discounting, and alcohol-related problems in European American and African American college students. Psychology of Addictive Behaviors, 25, 595-604. Dueber, D. M. (2016, December). Bifactor Indices Calculator: A Microsoft Excel-based tool to calculate various indices relevant to bifactor CFA models. Available at http://sites.education.uky.edu/apslab/resources. Ebesutani, C., McLeish, A. C., Luberto, C. M., Young, J., & Maack, D. J. (2014). A bifactor model of anxiety sensitivity: Analysis of the Anxiety Sensitivity Index-3. Journal of Psychopathology and Behavioral Assessment, 36, 452-464. Ebesutani, C., Smith, A., Bernstein, A., Chorpita, B. F., Higa-McMillan, C., & Nakamura, B. (2011). A bifactor model of negative affectivity: Fear and distress components among younger and older youth. Psychological Assessment, 23, 679-691. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum. Graham, J. W., & Schafer, J. L. (1999). On the performance of multiple imputation for multivariate data with small sample sizes. In R. Hoyle (Ed.), Statistical strategies for small sample research (pp. 1-29). London, England: Sage Publications. Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. Du Toit, & D. Sorbom (Eds.), Structural Equation Modeling: Present and Future (pp. 195-216). Lincolnwood, IL: Scientific Software International. Henry, J. D., & Crawford, J. R. (2005). The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 44, 227–239.


29

Hsu, S. H., Collins, S. E., & Marlatt, G. A. (2013). Examining psychometric properties of distress tolerance and its moderation of mindfulness-based relapse prevention effects on alcohol and other drug use outcomes. Addictive Behaviors, 38, 1852-1858. Hu., L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55. Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). Thousand Oaks, CA: Sage Publications. Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). New York, NY: The Guilford Press. Leyro, T., Bernstein, A., Vujanovic, A. A., McLeish, A., & Zvolensky, M. J. (2011). Distress Tolerance Scale: A confirmatory factor analysis among daily cigarette smokers. Journal of Psychopathology and Behavioral Assessment, 33, 47-57. Leyro, T. M., Zvolensky, M. J., & Bernstein, A. (2010). Distress tolerance and psychopathological symptoms and disorders: A review of the empirical literature among adults. Psychological Bulletin, 136, 576-600. Linehan, M. (1993). Cognitive-behavioral treatment of borderline personality disorder. New York, NY: The Guilford Press. Lovibond, S. H., & Lovibond, P. F. (1995). Manuals for the Depression Anxiety Stress Scales (2nd ed.). Sydney, Australia: Psychology Foundation. McHugh, R. K., Kertz, S. J., Weiss, R. B., Baskin-Sommers, A. R., Hearon, B. A., & Björgvinsson, T. (2014). Changes in distress intolerance and treatment outcome in a partial hospital setting. Behavior Therapy, 45, 232-240.


30

McHugh, R. K., & Otto, M. W. (2012). Refining the measurement of distress tolerance. Behavior Therapy, 43, 641-651. Muthén, B., Kaplan, D., & Hollis, M. (1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52, 431-462. Muthén, L. K., & Muthén, B. O. (2015). MPlus (Version 7.4) [Computer software]. Los Angeles, CA: Muthén & Muthén. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419. Patrick, C. J., Hicks, B. M., Nichol, P. E., & Krueger, R. F. (2007). A bifactor approach to modeling the structure of the Psychopathy Checklist-Revised. Journal of Personality Disorders, 21, 118-141. Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004). Two replicable suppressor situations in personality research. Multivariate Behavioral Research, 3, 303328. Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696.


31

Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129-140. Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92, 544-559. Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137-150. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In C. C. Clogg, & A. von Eye (Eds.), Latent variables analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA: Sage Publications. Shorey, R. C., Strauss, C., Elmquist, J., Anderson, S., Cornelius, T. L., & Stuart, G. L. (2017). Distress tolerance and intimate partner violence among men in substance use treatment. Journal of Family Violence, 32, 317-324. Simons, J. & Gaher, R. (2005). The Distress Tolerance Scale: Development and validation of a self-report measure. Motivation and Emotion, 29, 83-102. Stucky, B. D., & Edelen, M. O. (2015). Using hierarchical IRT to create unidimensional measures from multidimensional data. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183-206). New York, NY: Routledge. Stucky, B. D., Thissen, D., & Edelen, M. O. (2013). Using logistic approximations of marginal trace lines to develop short assessments. Applied Psychological Measurement, 37, 41-57.


32

Sugiura, N. (1978). Further analysts of the data by akaike's information criterion and the finite corrections: Further analysts of the data by akaike's. Communications in Statistics-Theory and Methods, 7, 13-26. Suhr, D. D. (2005). Principal component analysis vs. exploratory factor analysis. SUGI 30 Proceedings. Retrieved from https://pdfs.semanticscholar.org/ 936e/ 76d8f2203cb534a3952d3ea7032fc57f8ad0.pdf Timpano, K. R., Buckner, J. D., Richey, J. A., Murphy, D. L., & Schmidt, N. B. (2009). Exploration of anxiety sensitivity and distress tolerance as vulnerability factors for hoarding behaviors. Depression and Anxiety, 26, 343-353. Veilleux, J. C., Pollert, G. A., Zielinski, M. J., Shaver, J. A., & Hill, M. A. (in press). Behavioral assessment of the negative emotion aspect of distress tolerance: Tolerance to emotional images. Assessment. Vujanovic, A. A., Bernstein, A., & Litz, B. (2011). Traumatic stress. In M. Zvolensky, A. Bernstein, & A. Vujanovic (Eds.). Distress tolerance: Theory, research, and clinical applications. New York, NY: The Guilford Press. Vujanovic, A. A., Bonn-Miller, M. O., Potter, C. M., Marshall, E. C., & Zvolensky, M. J. (2011). An evaluation of the relation between distress tolerance and posttraumatic stress within a trauma-exposed sample. Journal of Psychopathology and Behavioral Assessment, 33, 129-135. Wang, M., & Russell, S. S. (2005). Measurement equivalence of the Job Descriptive Index across Chinese and American workers: Results from confirmatory factor analysis and item response theory. Educational and Psychological Measurement, 65, 709-732.


33

Wolgast, M. (2014). What does the Acceptance and Action Questionnaire (AAQ) really measure? Behavior Therapy, 45, 831-839. Zvolensky, M. J., Bernstein, A., & Vujanovic, A. A. (Eds.). (2011). Distress tolerance: Theory, research, and clinical applications. New York, NY: The Guilford Press. Zvolensky, M. J., & Otto, M. W. (2010). Affective intolerance, sensitivity, and processing: Advances in clinical science introduction [Special series]. Behavior Therapy, 38, 228233. Zvolensky, M. J., Vujanovic, A. A., Bernstein, A., & Leyro, T. (2010). Distress tolerance: Theory, measurement, and relations to psychopathology. Current Directions in Psychological Science, 19, 406-410.


34 Footnotes

1

A measure by the same name (Corstorphine, Mountford, Tomlinson, Waller, & Meyer,

2007) has been developed as another measure of distress tolerance. However, this measure was developed and validated in a sample of treatment-seeking females with disordered eating, and there are notable conceptual differences between the purported constructs of this measure and the DTS of interest in the present study (e.g., an emphasis on emotion regulation strategies rather than tolerance of emotional distress per se). Therefore, Corstorphine et al.’s measure and confirmatory factor analytic work were not considered in our literature review. 2

Confirmatory factor analysis was used to test the post-hoc hypothesis that a modified

bifactor model of the DTS (i.e., with the general factor and the Regulation domain-specific factor) would provide adequate fit to the data. Model fit for the modified bifactor model was as follows: χ2(87) = 556.97 (p < .001), RMSEA = .081 (90% CI = .075 - .087), CFI = .918, TLI = .901, SRMR = .042. Significant Δχ2[12] of 140.13 (p < .001) suggested that the modified bifactor model provided significantly worse fit to the data than the full bifactor model. However, RMSEA 90% CIs between the two models were overlapping. 3

Readers are directed to supplemental Figure 1 for a depiction of the structural regression

model and parameter estimates. 4

Depression, anxiety, and stress shared positive correlations that were large in magnitude,

rdep-anx = .81, rdep-stress = .83, ranx-stress = .86, ps < .001. Thus, consistent with previous research (Bardeen, Kumpula, & Orcutt, 2013), the DASS-21 was also modeled as a general distress construct in a second structural regression model. Specifically, general distress was modeled as a second-order factor; correlations between lower-order factors (i.e., anxiety, depression, stress) were omitted and direct pathways were modeled from the second-order factor (i.e., general


35

distress) onto each lower-order factor. The DTS general factor and domain-specific factors, from the bifactor model, were simultaneously regressed onto the general distress latent factor, as measured via the DASS-21. Consistent with the first structural regression model, only the Regulation factor and general factor were included as predictors in the model. The results of this regression were as follows. Model fit was adequate, χ2(586) = 1734.62 (p < .001), RMSEA = .049 (90% CI = .046-.051), CFI = .926, TLI = .921, SRMR = .052. The general factor of the DTS significantly predicted general distress in a theoretically consistent direction, β = 0.45, p < .001, while the Regulation domain-specific factor significantly predicted general distress in a theoretically inconsistent direction, β = -0.13, p < .01. 5

The full item-level correlation matrix is presented in supplemental Table 1.


36

Table 1 Review of Psychometric Literature Simons & Gaher, 2005

Leyro et al., 2011

Hsu et al., 2013

Sample size

Sample 1: 642 students

173 adult cigarette

168 adults seeking

and

(70% female);

smokers from

treatment for substance

composition

Sample 2: 823 students

community (45%

use (36% female)

(67% female)

female)

EFA and CFA

CFA

PCA

1 - 15

1 - 13, 15

1 - 5, 7 - 15

(item 14 omitted)

(item 6 omitted)

1 general factor

Statistical approach

Items used

Structure

1 general factor, 4

1 general factor, 4

derived

specific factors

specific factors

(Tolerance,

(Tolerance, Absorption,

Absorption, Appraisal,

Appraisal, and

and Regulation)

Regulation)

Areas of

Mean differences in

Omission of item 14;

Omission of item 6;

concern

general distress

standardized residuals

generalizability of PCA

tolerance between

≥ 1.96; potential item-

results with past CFA

males and females

cross loadings; strong

work

inter-factor correlations (rs > .80) Note. EFA – exploratory factor analysis; CFA – confirmatory factor analysis; PCA – principle component analysis.


37

Table 2 Model Comparisons Scaling

RMSEA [90%

Factor

CI]

432.12* (84)

1.308

35424.30

824.78* (90)

Hierarchical

34910.83

Bifactor

34841.69

AICc

χ2 (df)

34895.60

One-Factor

Model Correlated FourFactor

Configural invariance Metric invariance Scalar invariance ConfiguralMetric Δ ConfiguralScalar Δ Metric-Scalar Δ

34874.90

34847.24

34857.43

CFI

TLI

SRMR

.071 [.064-.078]

.939

.924

.039

1.343

.099 [.093-.106]

.871

.850

.054

445.61* (86)

1.313

.071 [.065-.078]

.937

.923

.041

390.70* (75)

1.256

.071 [.064-.078]

.945

.923

.036

1.304

.083 [.076-.090]

0.915

.898

.045

1.267

.080 [.074-.087]

0.913

.904

.049

1.249

.080 [.074-.086]

0.908

.908

.052

663.56* (174) 692.92* (190) 735.82* (203)

27.66

14.56 (16)

.003

.002

-.006

-.004

17.47

58.70* (13)

.003

.007

-.010

-.007

-10.19

41.76* (29)

.000

.005

-.004

-.003

Note. AICc = sample-size corrected Akaike information criterion; * indicates p < .001; df = degrees of freedom; RMSEA = root mean square error of approximation; CI = confidence interval; CFI = comparative fit index; TLI = Tucker-Lewis fit index; SRMR = standardized root mean square.


38

Table 3 Bifactor Evaluation Indices and Standardized Factor Loadings General Factor

Tolerance

Absorption

Appraisal

Regulation

ω/ωS

.95

.83

.88

.87

.84

ωH/ ωHS

.91

.11

.07

.10

.35

ECV

.80

.16

.12

.16

.42

H

.95

.27

.25

.42

.55

FD

.96

.67

.69

.75

.84

Residual variance

.89

.37^

.44^

.06

.26

Item Number

I-ECV

Factor Loadings

1

.71

.72

.46

3

.90

.80

.27

5

.97

.66

.13

2

.70

.75

.49^

4

1.00

.83

.06^

15

.97

.81

.13^

6†

1.00

.42

.02^

7

.93

.66

.19

9

.97

.76

.13^

10

.97

.82

.14^

11

.53

.63

.59

12

.83

.73

.34

8

.72

.65

.40

13

.50

.64

.64

14

.54

.53

.49

Note. ω = omega; ωS = omega subscale; ωH = omega hierarchical; ωHS = omega hierarchical subscale; ECV = explained common variance; H = Hancock and Mueller (2001) replicability indicator; FD = factor determinacy; all factor loadings and residual variances significant at p < .05, except for those identified by ^; ^ indicates p > .05; † indicates reverse-scored item.

Table S1 Full Inter-Item Correlation Matrix 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 Mean SD n

DASS1 DASS2 DASS3 DASS4 DASS5 DASS6 DASS7 DASS8 DASS9 DASS10 DASS11 DASS12 DASS13 DASS14 DASS15 DASS16 DASS17 DASS18 DASS19 DASS20 DASS21 DTS1 DTS2 DTS3 DTS4 DTS5 DTS6 DTS7 DTS8 DTS9 DTS10 DTS11 DTS12 DTS13 DTS14 DTS15 SEX

1 1.00 0.42 0.54 0.43 0.47 0.50 0.40 0.54 0.44 0.45 0.54 0.65 0.52 0.52 0.52 0.51 0.47 0.54 0.38 0.47 0.45 0.13 0.17 0.19 0.27 0.09 0.23 0.13 0.10 0.23 0.23 0.23 0.25 0.15 0.06 0.24 0.00 1.04 0.99 820

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

1.00 0.47 0.45 0.36 0.35 0.37 0.38 0.37 0.36 0.38 0.36 0.38 0.36 0.40 0.39 0.39 0.35 0.42 0.39 0.38 0.07 0.09 0.13 0.17 0.10 0.18 0.11 0.07 0.16 0.12 0.17 0.17 0.08 0.03 0.16 -0.04 0.92 1.04 818

1.00 0.53 0.56 0.54 0.43 0.50 0.52 0.70 0.58 0.57 0.70 0.53 0.57 0.71 0.67 0.55 0.45 0.56 0.68 0.23 0.20 0.28 0.33 0.13 0.32 0.22 0.15 0.30 0.30 0.26 0.31 0.21 0.10 0.31 -0.07 0.74 0.94 821

1.00 0.43 0.43 0.48 0.48 0.47 0.39 0.42 0.40 0.44 0.40 0.51 0.46 0.41 0.37 0.57 0.56 0.41 0.12 0.13 0.17 0.22 0.06 0.21 0.07 0.05 0.13 0.16 0.15 0.22 0.11 0.04 0.19 -0.06 0.50 0.84 821

1.00 0.49 0.32 0.46 0.49 0.57 0.55 0.53 0.61 0.41 0.43 0.62 0.53 0.49 0.37 0.46 0.52 0.18 0.24 0.23 0.27 0.11 0.21 0.18 0.11 0.27 0.27 0.24 0.26 0.15 0.10 0.29 0.03 1.15 1.03 821

1.00 0.43 0.51 0.55 0.48 0.62 0.55 0.53 0.48 0.52 0.51 0.50 0.60 0.40 0.52 0.51 0.19 0.20 0.25 0.32 0.09 0.31 0.15 0.11 0.27 0.31 0.23 0.27 0.16 0.08 0.28 0.02 0.92 0.98 821

1.00 0.54 0.51 0.39 0.42 0.41 0.38 0.39 0.48 0.39 0.40 0.41 0.47 0.54 0.45 0.08 0.07 0.15 0.18 0.03 0.17 0.10 0.08 0.10 0.16 0.18 0.24 0.11 0.00 0.15 -0.08 0.41 0.76 819

1.00 0.59 0.47 0.58 0.61 0.48 0.50 0.60 0.50 0.48 0.52 0.44 0.56 0.47 0.19 0.18 0.22 0.27 0.11 0.21 0.20 0.13 0.23 0.23 0.25 0.33 0.20 0.10 0.28 -0.05 0.76 0.94 818

1.00 0.51 0.55 0.52 0.49 0.43 0.62 0.51 0.56 0.48 0.43 0.60 0.53 0.22 0.19 0.29 0.33 0.12 0.30 0.20 0.17 0.29 0.31 0.32 0.37 0.22 0.13 0.34 -0.04 0.68 0.95 821

1.00 0.58 0.53 0.71 0.46 0.52 0.74 0.73 0.52 0.39 0.51 0.74 0.23 0.24 0.29 0.33 0.17 0.29 0.19 0.17 0.31 0.33 0.29 0.33 0.20 0.11 0.31 -0.01 0.86 1.05 821

1.00 0.73 0.62 0.57 0.57 0.61 0.55 0.66 0.41 0.50 0.53 0.23 0.23 0.29 0.31 0.15 0.25 0.17 0.14 0.28 0.33 0.29 0.32 0.18 0.12 0.32 -0.04 1.08 0.98 821

1.00 0.61 0.55 0.60 0.62 0.53 0.62 0.40 0.50 0.51 0.21 0.25 0.29 0.30 0.14 0.24 0.17 0.12 0.26 0.31 0.27 0.32 0.17 0.11 0.31 -0.01 1.14 1.04 820

1.00 0.50 0.55 0.72 0.70 0.59 0.40 0.50 0.68 0.25 0.25 0.29 0.38 0.18 0.31 0.20 0.17 0.34 0.34 0.30 0.34 0.21 0.12 0.34 0.04 1.01 1.03 819

1.00 0.53 0.53 0.48 0.55 0.38 0.45 0.44 0.16 0.14 0.19 0.25 0.06 0.22 0.14 0.10 0.18 0.20 0.20 0.25 0.15 0.09 0.22 -0.03 0.80 0.94 819

1.00 0.62 0.58 0.56 0.50 0.63 0.53 0.19 0.18 0.27 0.33 0.13 0.28 0.17 0.14 0.25 0.27 0.28 0.36 0.22 0.10 0.30 -0.01 0.59 0.90 821

1.00 0.70 0.58 0.43 0.55 0.68 0.21 0.21 0.29 0.33 0.14 0.29 0.20 0.15 0.31 0.30 0.29 0.33 0.23 0.14 0.34 0.00 0.82 1.02 820

1.00 0.57 0.40 0.57 0.78 0.25 0.24 0.30 0.37 0.18 0.30 0.23 0.16 0.32 0.32 0.35 0.34 0.20 0.10 0.32 -0.03 0.77 1.05 822

1.00 0.38 0.49 0.53 0.17 0.17 0.23 0.26 0.09 0.20 0.15 0.09 0.23 0.26 0.23 0.24 0.12 0.06 0.24 0.02 0.94 0.97 820

1.00 0.54 0.41 0.09 0.10 0.13 0.17 0.04 0.21 0.05 0.04 0.14 0.13 0.14 0.20 0.10 0.00 0.17 -0.07 0.71 0.93 820

20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 1 DASS1 2 DASS2 3 DASS3 4 DASS4 5 DASS5 6 DASS6 7 DASS7 8 DASS8 9 DASS9 10 DASS10 11 DASS11 12 DASS12 13 DASS13 14 DASS14 15 DASS15 16 DASS16 17 DASS17 18 DASS18 19 DASS19 20 DASS20 1.00 21 DASS21 0.60 1.00 22 DTS1 0.19 0.25 1.00 23 DTS2 0.14 0.25 0.63 1.00 24 DTS3 0.25 0.30 0.70 0.68 1.00 25 DTS4 0.30 0.36 0.58 0.65 0.70 1.00 26 DTS5 0.07 0.15 0.54 0.54 0.56 0.58 1.00 27 DTS6 0.24 0.26 0.29 0.28 0.31 0.39 0.18 1.00 28 DTS7 0.19 0.22 0.52 0.43 0.55 0.53 0.47 0.27 1.00 29 DTS8 0.10 0.20 0.51 0.53 0.50 0.46 0.53 0.19 0.56 1.00 30 DTS9 0.24 0.34 0.51 0.55 0.60 0.63 0.45 0.39 0.55 0.47 1.00 31 DTS10 0.23 0.34 0.56 0.57 0.63 0.70 0.51 0.39 0.56 0.55 0.68 1.00 32 DTS11 0.27 0.30 0.46 0.43 0.52 0.52 0.37 0.27 0.54 0.40 0.55 0.59 1.00 33 DTS12 0.33 0.34 0.55 0.49 0.58 0.62 0.45 0.30 0.53 0.45 0.58 0.67 0.66 1.00 34 DTS13 0.19 0.23 0.48 0.43 0.49 0.47 0.48 0.23 0.49 0.67 0.47 0.53 0.41 0.53 1.00 35 DTS14 0.06 0.11 0.42 0.39 0.38 0.38 0.44 0.13 0.40 0.54 0.36 0.46 0.35 0.40 0.65 1.00 36 DTS15 0.26 0.31 0.58 0.67 0.61 0.69 0.52 0.34 0.50 0.51 0.64 0.68 0.55 0.64 0.55 0.50 1.00 37 SEX -0.11 -0.06 0.03 0.08 0.03 0.03 0.06 -0.03 -0.07 0.06 0.04 0.07 0.03 0.01 0.06 0.11 0.10 Mean 0.55 0.66 2.82 2.87 2.59 2.48 2.87 2.58 2.58 2.85 2.69 2.56 2.51 2.33 2.83 3.00 2.81 SD 0.88 1.01 1.31 1.36 1.32 1.42 1.36 1.27 1.28 1.27 1.37 1.37 1.40 1.35 1.29 1.28 1.37 n 822 819 824 824 825 822 823 823 826 822 825 825 822 822 824 823 823 Note: DASS – Depression Anxiety Stress Scales; DTS – Distress Tolerance Scale; SD – standard deviation; n – number of cases with available data per variable

37

1.00 1.61 0.49 821

Figure 1. Structural regression with the DTS modified bifactor model. Note: dts# = DTS items; gendt = general factor of the DTS bifactor model; reg = Regulation domain-specific factor of the DTS bifactor model; dep = Depression subscale of the DASS-21; anx = Anxiety subscale of the DASS-21; str = Stress subscale of the DASS-21; dass# = DASS-21 items; all parameter estimates fully standardized (i.e., STDXY standardization); all estimates significant at p < .01.