3 Factor Structure and Incremental Validity of the

5 downloads 0 Views 479KB Size Report
originally hypothesized a four-factor solution in which Impulse and Goals were subsumed by ..... The Depression Anxiety Stress Scales-21 item version (DASS-21). ..... Bardeen, J. R., Fergus, T. A., Hannan, S. M., & Orcutt, H. K. (2016).
Running head: DIFFICULTIES IN EMOTION REGUALTION SCALE

1

Tables: 3

Factor Structure and Incremental Validity of the Original and Modified Versions of the Difficulties in Emotion Regulation Scale Natasha Benfer1, Joseph R. Bardeen1, Thomas A. Fergus2, & Travis A. Rogers1 1

Department of Psychology, Auburn University, United States

2

Department of Psychology and Neuroscience, Baylor University, United States

NOTICE: this is the author’s version of a work that was accepted for publication in Journal of Personality Assessment. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Journal of Personality Assessment. Author Note Correspondence concerning this article should be addressed to Joseph R. Bardeen, Department of Psychology, Auburn University, Auburn, AL 36832. Voice: 334-844-6647; Email: [email protected].

DIFFICULTIES IN EMOTION REGULATION SCALE

2

Abstract The Difficulties in Emotion Regulation Scale (DERS; Gratz & Roemer, 2004) is a self-report measure that assesses six facets of emotion dysregulation. A modified version of the DERS (MDERS) was developed to address psychometric limitations of the original measure (Bardeen et al., 2016). Although the factor structure of the M-DERS (i.e., two models: correlated trait and second-order models) has been supported via confirmatory factor analysis (CFA), the tenability of a bifactor model of the M-DERS has yet to be examined. Preliminary research suggests that a bifactor model of the DERS is tenable. In the present study (Ns of 993 and 578), results from a series of CFAs indicated adequate fit of the M-DERS and poor fit of the original DERS across several tested models (e.g., correlated trait, second-order, bifactor). Although a considerable amount of variance was accounted for by the general factor, statistical indices from the bifactor model supported a multidimensional conceptualization of the M-DERS. The Nonacceptance and Goals subscales evidenced incremental utility, after accounting for the general factor, in predicting general distress (Nonacceptance only) and intolerance of uncertainty. Implications for future use of the DERS and M-DERS are discussed. Keywords: Difficulties in Emotion Regulation Scale (DERS); emotion regulation; bifactor; confirmatory factor analysis (CFA); general distress.

DIFFICULTIES IN EMOTION REGULATION SCALE

3

The Factor Structure and Incremental Validity of the Original and Modified Versions of the Difficulties in Emotion Regulation Scale As research continues to emphasize the importance of transdiagnostic risk factors for psychological disorders, interest in the construct of emotion regulation continues to increase (see Aldao, 2013, and Gross, 2015 for reviews). Emotion regulation, broadly defined, is the ability to monitor and alter emotional states in pursuit of goal-directed behavior (Gratz & Roemer, 2004). Deficits in emotion regulation have been linked to both internalizing and externalizing symptom presentations, such as depression, substance use, and anxiety (see Aldao, Nolen-Hoeksema, & Schweizer, 2010, for a review). Emotion regulation deficits have been shown to prospectively predict various forms of psychopathology (Bardeen, Kumpula, & Orcutt, 2013; Kim & Cicchetti, 2010; Tull, Bardeen, DiLillo, Messman-Moore, & Gratz, 2015). As such, treatments have been developed in which deficits in emotion regulation are targeted in the service of reducing psychological distress (e.g., Emotion Regulation Therapy [ERT]; Mennin & Fresco, 2013). To improve our understanding of psychopathology marked by emotion regulation deficits and to evaluate treatment outcomes, there is a critical need for psychometrically sound measures of emotion regulation difficulties. The Difficulties in Emotion Regulation Scale (DERS) is one of the most widely used measures of emotion regulation, with over 2,600 citations attributed to the initial validation study (Gratz & Roemer, 2004). Gratz and Roemer (2004) found that a six-factor correlated trait model (i.e., six related, but distinct, domains of difficulties in emotion regulation) ultimately best operationalized the original 36-item DERS. The factors included Nonacceptance, Goals, Impulse, Strategies, Awareness, and Clarity. It is important to note that Gratz and Roemer (2004) originally hypothesized a four-factor solution in which Impulse and Goals were subsumed by

DIFFICULTIES IN EMOTION REGULATION SCALE

4

one factor and Awareness and Clarity were subsumed by another factor. Contrary to that expectation, an exploratory factor analysis (EFA) supported either a five- or six-factor solution, and the six-factor solution was retained for ease of interpretation (Gratz & Roemer, 2004). A number of studies examining the structure of the DERS have been conducted since that time, with mixed support for the six-factor correlated trait model (see Bardeen, Fergus, & Orcutt, 2012, for a review). Across published studies, the Awareness factor does not consistently relate to (a) the other factors of the DERS and (b) criterion variables as would be expected (e.g., Giromini, Velotti, de Campora, Bonalume, & Zavattini, 2012). Specifically, the Awareness factor consistently exhibits low and/or nonsignificant correlations with the other DERS factors and fails to relate to clinically relevant criterion variables in a theoretically expected manner (Fowler et al., 2014; Lee, Witte, Bardeen, Davis & Weathers, 2016; Perez, Venta, Garnaat, & Sharp, 2012). This pattern of findings may be the result of a method effect, as the Awareness factor is the only factor of the DERS for which all items are reverse-coded. No other factor of the DERS consists of an item pool with more than two reverse-coded items. To address this concern, Bardeen, Fergus, Hannan, and Orcutt (2016) created a modified version of the DERS (M-DERS) in which all of the reverse-coded items of the DERS were reworded in a straightforward manner and then subjected to EFA. Consistent with Gratz and Roemer’s (2004) original hypothesis, items from Awareness and Clarity loaded on the same factor in the EFA, which Bardeen et al. (2016) named Identification. Additionally, Bardeen et al. (2016) removed one item of the DERS that cross-loaded onto two factors, two items that had poor factor loadings, and reduced the Identification subscale by removing the four lowestloading items, thus resulting in a 29-item measure. Results from confirmatory factor analysis (CFA) supported a five-factor correlated trait model of the M-DERS, as well as a second-order

DIFFICULTIES IN EMOTION REGULATION SCALE

5

model in which each factor loaded onto a higher-order emotion dysregulation factor. To date, no other psychometric evaluations of the M-DERS have been published. DERS items are routinely summed to create a total score and subscale scores (e.g., Fox, Axelrod, Paliwal, Sleeper, & Sinha, 2007; Salters-Pedneault, Roemer, Tull, Rucker, & Mennin, 2006). For example, the DERS total score has been used to predict a variety of psychological phenomena, including non-suicidal self-injury (NSSI; McKenzie & Gross, 2014), binge eating (Whiteside et al., 2007), and post-traumatic stress (Tull, Barrett, McMillan, & Roemer, 2007). Use of the total score assumes that the lower-order factors represent the same overarching emotion dysregulation construct. Second-order models (i.e., those in which direct pathways are modeled from an overarching general construct [higher-order factor] onto uncorrelated domainspecific constructs [lower-order factors]) can be used to examine this assumption by assessing the degree to which covariation among factors is captured by a general factor (Brown, 2015; Clark & Watson, 1995). The hierarchical structure of the DERS has been examined in a number of studies, and although some results from CFA have indicated adequate fit (Bardeen et al., 2012), the majority of published evidence suggests poor fit of this model to the data (e.g., Fowler et al., 2014; Lee et al., 2016). In contrast, preliminary evidence supports a second-order model of the M-DERS (Bardeen et al., 2016). Importantly, a second-order model cannot test the assumption that items of the specific factors provide unique information beyond the higher-order construct. Testing this assumption is necessary, because, as noted, DERS items are routinely summed to create subscales that have shown differential relations with constructs of interest (e.g., substance use: Fox et al., 2007; worry: Salters-Pedneault et al., 2006). In contrast to a second-order model, a bifactor modeling approach examines the degree to which items of a measure are representative of a general factor and more specific domain factors.

DIFFICULTIES IN EMOTION REGULATION SCALE

6

As such, bifactor modeling can be used to determine whether a total score is sufficient to operationalize the construct assessed by a given measure (Reise, 2012). Similar to a second-order model, direct effects from domain specific factors to indicators are modeled with a bifactor approach. However, unlike a second-order model, direct effects from the general factor to each indicator are also modeled; domain-specific factors are not nested. Thus, a bifactor model isolates the unique contributions of both domain-specific factors and a general factor, and therefore reveals the degree to which each domain-specific factor is redundant with the general factor. Evidence that a domain-specific factor is redundant with the general factor suggests that the corresponding subscale does not contribute unique information. If redundancy is observed across all domain-specific factors, the measure may be best represented as a unidimensional construct and points to use of a total score. In the only bifactor analysis of the original DERS, Osborne, Michonski, Sayrs, Welch, and Anderson (2017) removed direct paths from the Awareness items of the DERS to the general factor, but allowed the Awareness factor to remain in the model and correlate with the Clarity subscale alone. While the overall model demonstrated adequate fit, the findings from this examination are limited in their utility, as the current conceptualization of the DERS and associated clinical application (i.e., use of a total score which includes Awareness items) were not directly tested. That is, the value of this finding, with the Awareness items remaining in the model, but separate from the general factor, does not clarify our conceptual understanding of the nature of emotion dysregulation, as assessed by the DERS, nor does it provide a clear direction for pragmatic use of the measure. Given the widespread use of the DERS total score, it is essential when using a bifactor approach to model the items of the measure in a manner consistent with the original factor structure (Gratz & Roemer, 2004) and current clinical use.

DIFFICULTIES IN EMOTION REGULATION SCALE

7

That means allowing all of the items of the DERS to load onto the general factor and their respective domain-specific factors in a bifactor model. Additionally, to date, the M-DERS has yet to be examined via a bifactor modeling approach. Existing evidence suggests the M-DERS may have superior psychometric properties compared to the DERS (Bardeen et al., 2016), and therefore, a more rigorous evaluation of its proposed multidimensionality, via bifactor analysis, is needed. In light of recent criticisms of the factor structure of the DERS, and limited psychometric evidence in support of the M-DERS, the current study sought to examine model fit of these measures via a series of CFAs (i.e., one-factor, correlated trait, second-order, bifactor). Based on the research described above, second-order and bifactor models of the original DERS were not expected to provide adequate fit to the data. In contrast, and consistent with preliminary research (Bardeen et al., 2016), as well as theory suggesting that the emotion dysregulation construct is multidimensional (Gratz & Roemer, 2004), we expected that the bifactor model of the M-DERS would provide significantly better fit to the data than competing models. Some have suggested that the utility of comparing bifactor models to alternative models is limited because bifactor models often provide better fit to the data due to their flexibility (Bonifay, Lane, & Reise, 2017; Reise, Kim, Mansolf, & Widaman, 2016). As such, we also examined a number of additional statistical indices to evaluate the proposed bifactor model (Rodriguez et al., 2016). An additional benefit of bifactor modeling is that the relation between domain-specific factors and criterion variables can be examined while holding the general factor constant (Brown, 2015). This approach might provide evidence of the incremental utility of domainspecific factors, beyond the general factor, in predicting psychological constructs theoretically relevant to emotion dysregulation. More specifically, through the use of bifactor modeling and

DIFFICULTIES IN EMOTION REGULATION SCALE

8

structural regression, we can determine whether the domain-specific factors of the M-DERS account for unique variance in relevant criterion variables after accounting for the M-DERS general factor. Measures of general distress and intolerance of uncertainty were selected as criterion variables. General distress was selected because it is a clinically relevant construct. Intolerance of uncertainty, described as “an individual’s dispositional incapacity to endure the aversive response triggered by the perceived absence of salient, key, or sufficient information, and sustained by the associated perception of uncertainty” (Carleton, 2016, p. 31), is conceptualized as a facet of distress intolerance (Zvolensky, Vujanovic, Bernstein, & Leyro, 2010), and thus, should demonstrate meaningful but non-redundant associations with emotion dysregulation. Methods Participants and Procedure Sample 1. As part of a larger study (masked for review), general population adults from the United States (N = 993), recruited via Amazon Mechanical Turk (MTurk), completed the original DERS. MTurk’s online crowd sourcing platform allows individuals to participate in research in exchange for compensation. In exchange for completion of a battery of self-report questionnaires, which took approximately 30 minutes to complete, participants were paid $0.50, an amount similar to that paid to MTurk workers for completing questionnaire batteries of similar length (Buhrmester, Kwang, & Gosling, 2011). To minimize the effect of random responding or inattentiveness, three catch questions (e.g., “Select ‘Much’ if you are paying attention right now”) were imbedded in the online survey (Oppenheimer, Meyvis, & Davidenko, 2009; Paolacci, Chandler, & Ipeirotis, 2010). To ensure the quality of data in the present study, only participants who accurately responded to two out of three catch questions were included in

DIFFICULTIES IN EMOTION REGULATION SCALE

9

the final sample (for precedence, see Bardeen et al., 2016; Bardeen, Fergus, & Orcutt, 2013a, 2013b; Bardeen & Michel, 2017). Forty-eight participants did not respond correctly to any of the catch questions, 101 responded correctly to one question, 128 respond correctly to two questions, and 716 responded correctly to all three questions. Additionally, 19 participants who did not complete any of the items of the DERS were excluded from this study, resulting in a final sample of 825 participants. Concerning race, 81.1% of the sample self-identified as White, 6.9% as Black, 6.4% as Asian, 0.1% as American Indian or Alaska Native, 2.9% as “other”, while 1.6% preferred not to respond. Additionally, 6.8% of the sample identified as Hispanic. The majority of the sample was female (60.7%). The average age of the final sample was 34 years (SD = 12.5, range = 18 - 74). Sample 2. As part of a larger study (masked for review), general population adults from the United States (N = 578 MTurk participants) completed the M-DERS. The same quality control procedure used in Sample 1 was used for Sample 2 (i.e., accurate responding on at least two of three catch questions). Two participants did not respond correctly to any of the catch questions, 11 responded correctly to one question, 55 respond correctly to two questions, and 511 responded correctly to all three questions. The final sample consisted of 564 participants. Participants completed this battery of questionnaires in approximately 35 minutes and were paid $1.50 as compensation. Concerning race, 82.3% of the sample self-identified as White, 8.7% as Black, 5.9% as Asian, 0.4% as American Indian or Alaska Native, and 2.8% as “other”. Additionally, 6.8% of the sample identified as Hispanic. The majority of the sample was female (59.4%), and the average age was 36 years (SD = 11.3, range = 18 - 65). Measures

DIFFICULTIES IN EMOTION REGULATION SCALE

10

Difficulties in Emotion Regulation Scale (DERS). The DERS (Gratz & Roemer, 2004) contains 36 items used to measure six facets of emotion regulation (i.e., Nonacceptance, Goals, Impulse, Strategies, Awareness, Clarity). Consistent with standard scoring procedures, scores on the eleven reverse-coded items of the DERS were recoded in a straightforward manner to be consistent with the remaining twenty-five items before analysis. Items are rated on a 5-point scale based on how often participants believe each item pertains to them (1 = almost never to 5 = almost always). Higher scores indicate greater difficulty regulating emotions. Internal consistency in the current study (sample 1 only) was α = .96. Subscale αs were as follows: Nonacceptance = .93, Goals = .89, Impulse = .76, Strategies = .80, Awareness = .80, Clarity = .82. The mean total score was 82.6 (SD = 27.2). Modified Difficulties in Emotion Regulation Scale (M-DERS). The M-DERS (Bardeen et al., 2016) contains 29 items worded in a straightforward manner. These items are used to measure five facets of emotion regulation (i.e., Nonacceptance [“When I’m upset, I feel guilty for feeling that way”], Goals [“When I’m upset, I have difficulty getting work done”], Impulse [“When I’m upset, I feel like I can’t remain in control of my behaviors”], Strategies [“When I’m upset, I believe that I’ll end up feeling very depressed”], Identification [“When I’m upset, I don’t pay attention to how I feel”]). The item rating scale of the M-DERS is consistent with the DERS. Internal consistency in the current study (sample 2 only) was α = .97. Subscale αs were as follows: Nonacceptance: .94; Goals: .96; Impulse: .93; Strategies: .95; Identification: .91. The mean total score was 58.4 (SD = 24.7). The Depression Anxiety Stress Scales-21 item version (DASS-21). The DASS-21 (Lovibond & Lovibond, 1995) is a 21-item measure with scales assessing depression, anxiety, and stress symptoms in the past week. Each subscale contains seven items rated on a 4-point

DIFFICULTIES IN EMOTION REGULATION SCALE

11

scale (0 = did not apply to me at all to 3 = applied to me very much, or most of the time). Higher scores indicate higher symptom levels. The DASS-21 subscales have demonstrated strong convergent validity (Antony, Bieling, Cox, Enss, & Swinson, 1998), discriminant validity (Henry & Crawford, 2005), and internal consistency (Antony et al., 1998; Osman et al., 2012). Because the DASS-21 subscales are highly correlated, some have suggested that the subscales should be considered lower-order domains of a broader general psychological distress construct (Clark & Watson, 1991; Moras, di Nardo, & Barlow, 1992). As such, and consistent with previous research (e.g., Bardeen, Kumpula, & Orcutt, 2013; Bradbury et al., 2008; O’Brien et al., 2016; Osman et al., 2011), the DASS-21 subscales were used as indicators of a general distress construct in the present study. Internal consistency for the DASS-21 (sample 2 only) was as follows: total score: α = .96; Depression: α = .93; Anxiety: α = .86; Stress: α = .90. Intolerance of Uncertainty Scale—Short Form (IUS-12). The IUS-12 (Carleton, Norton, & Asmundson, 2007) is a 12-item measure that assesses maladaptive cognitive and emotional responses to uncertainty. Items are rated on a 5-point scale (1 = not at all characteristic of me to 5 = entirely characteristic of me). Higher scores indicate greater intolerance of uncertainty. The IUS-12 has demonstrated convergent validity with measures related to anxiety and worry (Carleton et al., 2007) and good to excellent internal consistency (αs ranging .83-.93; Hale et al., 2016). A recent psychometric investigation suggests that the IUS-12 consists of a strong general intolerance of uncertainty factor and that previously identified subscales should not be used because they exhibit poor reliability and account for a relatively small proportions of variance (Hale et al., 2016; Shihata, McEvoy, & Mullan, in press). As such, the IUS was modeled as a single latent construct in subsequent analyses. Internal consistency of the total score in the current study (sample 2 only) was α = .93.

DIFFICULTIES IN EMOTION REGULATION SCALE

12

Data Analytic Strategy Confirmatory Factor Analysis. We examined the correlated trait and one-factor models for both the DERS and M-DERS. The correlated trait model of the DERS (six correlated domain-specific factors) consists of six items with loadings on Nonacceptance, five items with loadings on Goals, six items with loadings on Impulse, eight items with loadings on Strategies, six items with loadings on Awareness, and five items with loadings on Clarity. For the M-DERS, the correlated trait model (five correlated domain-specific factors) consists of six items with loadings on Nonacceptance, five items with loadings on Goals, six items with loadings on Impulse, six items with loadings on Strategies, and six items with loadings on Identification. For both correlated trait models, no secondary loadings were modeled and all domain-specific factors were allowed to correlate. For both one-factor models, all items (i.e., DERS = 36 items, MDERS = 29 items) loaded onto a general factor. Correlations between items were not allowed. For both second-order models (DERS and M-DERS), correlations among the factors in the correlated trait models were removed and direct pathways were added from a second-order factor to lower-order factors. Finally, bifactor models were examined for both the DERS and M-DERS, in which all items were simultaneously loaded onto a general factor and their respective domainspecific factors. In the bifactor models, all factor covariances were fixed to zero (Brown, 2015). Structural Regression Models. Two structural regression models were used to examine whether the domain-specific factors of the M-DERS were associated with criterion variables (i.e., intolerance of uncertainty, general distress), after accounting for the general M-DERS factor. In the first structural regression model, the M-DERS general factor and domain-specific factors, from the bifactor model, were simultaneously regressed onto the general distress latent factor, as measured via the DASS-21. Consistent with previous research (e.g., Bardeen et al.,

DIFFICULTIES IN EMOTION REGULATION SCALE

13

2013), general distress was modeled as a second-order factor; correlations between lower-order factors (i.e., anxiety, depression, stress) were omitted and direct pathways were modeled from the second-order factor (i.e., general distress) onto each lower-order factor. In the second structural regression model, the M-DERS general factor and domain-specific factors were simultaneously regressed onto the intolerance of uncertainty factor. Consistent with factor analytic evidence in support of a strong general factor for the IUS-12 (e.g., Hale et al., 2016), intolerance of uncertainty was modeled as a single latent construct; all of the IUS-12 items loaded on a single latent factor. For each of the structural regression models, regression coefficients from the M-DERS domain-specific factors to the outcome variable (i.e., general distress or intolerance of uncertainty) provided an indication of the unique association between the domain-specific factor and the outcome, after accounting for the M-DERS general factor. Model Estimation and Evaluation. Approximately 1% of data were missing. Missing data patterns were analyzed using Little's (1988) missing completely at random (MCAR) test, which produced a nonsignificant normed chi-square value (χ2 [1294] = 1181.70, p = .99). Thus, concern regarding the implications of results due to missing data was minimal. All models were examined using MPlus (version 7.4; Muthén & Muthén, 2015). Robust maximum likelihood (MLR) estimation was used to test all models because it is well-suited for use when (a) sample sizes are not excessively large, (b) the number of response options exceeds four, and (c) some data are missing at random (Bentler & Chou, 1987; Brown, 2015; Kline, 2016; Flora & Curran, 2004; Rhemtulla, Brosseau-Laird, & Savalei, 2012). With MLR, full information maximum likelihood estimation is used to calculate parameter estimates from all available data. This approach provides less biased parameter estimates than listwise or pairwise deletion (Dong & Peng, 2013). All reported chi-square values represent the Satorra-Bentler scaled chi-square

DIFFICULTIES IN EMOTION REGULATION SCALE

14

(Satorra & Bentler, 1994). Goodness of fit was evaluated using commonly used fit statistics and criterion benchmarks outlined by Hu and Bentler (1999). Benchmarks included comparative fit index (CFI) and Tucker-Lewis index (TLI) values close to or greater than .95, root mean square error of approximation (RMSEA) values close to or less than .06, and standardized root mean residual (SRMR) values close to or less than .08. Scaled chi-square difference testing was conducted to compare nested models (using the method outlined by Satorra & Bentler, 2001). Significant differences between two comparable models indicates a significant worsening in model fit in the more restricted model. However, chisquare difference testing is influenced by sample size, with trivial differences being identified as significant in large samples. Therefore, alternative tests for comparing models were used (Brown, 2015; Kline, 2016). Specifically, we examined non-overlapping RMSEA 90% confidence intervals (CIs) and changes in CFI (ΔCFI). ΔCFI values less than either .01 (Cheung & Rensvold, 2000) or .002 (Meade, Johnson, & Braddy, 2008) have been used to determine significant differences in model fit. The bifactor model was further evaluated using fit indices put forth by Rodriguez, Reise, and Haviland (2016). Calculation of these indices were performed using the bifactor indices calculator provided by Dueber (2016). Specifically, we examined: Omega (ω), Omega H (ω H ), Omega HS (ω HS ), explained common variance (ECV), item-level ECV (I-ECV), and percentage of uncontaminated values (PUC). Omega (ω) is a model-based reliability estimate for latent variable modeling, and thus can be interpreted similarly to alpha coefficients. Omega H (ω H ) reflects the proportion of variance in M-DERS scores attributable to a single general factor, whereas Omega HS (ω HS ) reflects the proportion of variance in scores attributable to each domain-specific factor after removing the variance due to the general factor. ECV is the ratio of

DIFFICULTIES IN EMOTION REGULATION SCALE

15

variance explained by the general factor to variance explained by both the general and domainspecific factors. Thus, ECV values serve as a better index of unidimensionality than ω H . I-ECV indicates the amount of common variance for each item that is attributable to the general factor (Stucky, Thissen, & Edelen, 2013). ECV is interpreted in combination with PUC; higher PUC indicates higher parameter bias, and alters the interpretation of ECV (Bonifay, Reise, Scheines, & Meijer, 2015). Finally, we examined average relative parameter bias (ARPB), which serves as an indicator of the bias across parameters if items are forced into a unidimensional structure. Results Confirmatory Factor Analysis (DERS). As can be seen in Table 1, all models (i.e., onefactor, correlated trait, second-order, and bifactor) of the original DERS exhibited poor fit. With the exception of the SRMR for the correlated trait and bifactor models, none of the fit statistics met the pre-specified guidelines. Although fit was poor for the original DERS models, latent correlations among domain-specific factors were all significant in the correlated trait model (see Table 2). However, in line with previous research (Bardeen et al., 2016; Tull, Gratz, Latzman, Kimbrel, & Lejuez, 2010), the Awareness factor exhibited small to medium correlations with four of the five domain-specific factors (rs: .19 - .23, ps < .001), with the exception of the Clarity factor (r = .47, p < .001). Additionally, while five of the six domain-specific factors in the second-order model exhibited large loadings on the higher-order factor (range: .74 - .98, ps .05).3 Discussion

DIFFICULTIES IN EMOTION REGULATION SCALE

19

In the present study, we examined the factor structure of the DERS and M-DERS in two separate, large samples of community adults. Although Osborne et al. (2017) examined an alternate bifactor model of the DERS, this was the first study to examine a bifactor model of the DERS in its entirety (i.e., allowing all of the items of the measure to load onto the general factor and respective domain-specific factors). Additionally, the present study provided the first known test of a bifactor model of the M-DERS. Bifactor modeling is important for determining whether the degree of multidimensionality in a given measure is sufficient to support using subscales. For the DERS, none of the examined models provided adequate fit to the data, thus adding to mounting concerns regarding the factor structure of this measure (Bardeen et al., 2012; Lee et al., 2016). In contrast, the correlated trait, second-order, and bifactor models of the M-DERS all demonstrated adequate fit to the data. The superior model fit of the M-DERS, compared to the DERS, is consistent with Gratz and Roemer’s (2004) original conceptualization of emotion regulation in which awareness and clarity of emotions was encapsulated by a single factor (i.e., M-DERS Identification). Importantly, statistical indices from the bifactor model suggest that there are non-negligible amounts of multidimensionality that can be attributed to the domainspecific factors of the M-DERS. The general emotion dysregulation factor accounted for the majority of variance in MDERS scores. However, our results suggest that the multidimensionality of the M-DERS is substantial enough to warrant the use of the subscales, with the exception of M-DERS Strategies. For four of the five subscales, not including Strategies, at least 20% of reliable variance was independent of the general emotion dysregulation factor. Furthermore, average I-ECV values across these four subscales support a multidimensional model with a strong general factor. The majority of variance from items comprising Identification was attributed to the Identification

DIFFICULTIES IN EMOTION REGULATION SCALE

20

factor, rather than the general factor, and a considerable proportion of variance from Impulse, Nonacceptance, and Awareness was attributed to each domain-specific factor, over the general factor (range: 26 - 38%). In contrast, just 9% of reliable variance from the Strategies subscale appeared distinct from the general factor, and variance of the items of the Strategies subscale suggest that these items reflect content that is redundant with general factor rather than being domain-specific. This finding is consistent with previous research examining the hierarchical model of the DERS, which suggested the possibility that the Strategies factor was redundant with the general factor (Bardeen et al., 2016; Osborne et al., 2017). Specifically, the Strategies factor has exhibited loadings onto the general emotion dysregulation factor that were high enough to suggest redundancy (loadings of .96 and .94, respectively). Osborne et al. (2017) found similarly weak loadings of items from the Strategies domain on the domain-specific factor. The Strategies subscale was originally derived from items from the Generalized Expectancy for Negative Mood Regulation scale (NMR; Catanzaro & Mearns, 1990), a measure that is often used to assess general emotion dysregulation (Gratz & Roemer, 2004), but more accurately reflects an individual’s perceived ability to regulate emotions (e.g., “When I’m upset, I believe that I’ll end up feeling very depressed”). The other domains consist of content that is important, but perhaps more peripheral and narrower in scope than content that is at the core of the emotion dysregulation construct, as assessed by the DERS. This content-based explanation may aid in better understanding present and past results related to this specific domain. Also of note, some of the statistical indices used to assess the bifactor structure suggest the possibility that the Impulse factor may not provide enough unique information beyond that of the general factor to be considered its own separate domain. For example, the Impulse factor was the only domain-specific factor to exhibit nonsignificant residual variance, which may suggest a

DIFFICULTIES IN EMOTION REGULATION SCALE

21

weak factor with variance largely explained by the general factor. The Impulse factor was not removed from the structural regression model because, unlike the Strategies factor, the follow-up statistical indices used to determine factor redundancy suggest that the Impulse factor might be reasonably independent from the general factor (Rodriguez et al., 2016). Despite concerns regarding the redundancy of the Strategies factor, and to a lesser degree the Impulse factor, evaluation of the bifactor model suggests that the M-DERS consists of three to four domainspecific factors that are meaningfully independent of the general factor. As such, use of multidimensional latent variable model specifications that account for the general and domainspecific factors (e.g., bifactor models) should be considered when examining structural models that include the M-DERS. Two separate structural regression models further clarified the incremental utility of MDERS domain-specific factors. Results from the first structural regression model suggest that the Nonacceptance domain-specific factor from the M-DERS demonstrates incremental utility in predicting general distress after accounting for the general emotion dysregulation factor. Acceptance of emotions is a core component of several models of psychological health and flexibility (e.g., Acceptance and Commitment Therapy [ACT]; Hayes, Luoma, Bond, Masuda, & Lillis, 2006). This may help explain why M-DERS Nonacceptance (i.e., nonacceptance of emotional responses when experiencing negative emotions) emerged as uniquely related to general distress, beyond general emotion dysregulation. Results from the second structural regression model indicate that the that both the Nonacceptance and Goals domain-specific factors account for unique variance in predicting intolerance of uncertainty after accounting for the M-DERS general factor. Both intolerance of uncertainty and nonacceptance of emotions reflect a rigidity toward staying in contact with

DIFFICULTIES IN EMOTION REGULATION SCALE

22

negative internal states. In support of this idea, the IUS has demonstrated significant associations with measures of experiential avoidance (e.g., Lee, Orsillo, Roemer, & Allen, 2010). The Goals domain-specific factor is proposed to tap into difficulty engaging in goal-directed behavior when distressed (Bardeen et al., 2016) and the IUS-12 contains items that assess behavioral restraint in the face of uncertainty. Thus, it is theoretically consistent that the Goals domain-specific factor would share an association with the IUS-12, even after accounting for the general emotion dysregulation factor. Based on these results, some might suggest discontinuing the practice of using other M-DERS subscales. However, this conclusion is premature, as only two criterion variables were investigated in the present study. More research is needed to investigate whether other domain-specific factors demonstrate incremental utility in predicting other clinical outcomes that have been associated with the DERS or M-DERS (e.g., substance use, posttraumatic stress symptoms), as well as investigating associations with theoretically related constructs. Existing research has demonstrated that DERS subscales do have predictive utility. For example, Racine and Wildes (2013) found that the Impulse subscale, in comparison to the five other subscales, provided incremental utility in predicting recurrent objective binge eating and purging in an inpatient population. Additionally, Tull and Roemer (2007) found that individuals with a history of panic, versus those without, had significantly higher scores on the Clarity and Nonacceptance subscales. Therefore, before a firm conclusion can be reached regarding the incremental utility of the M-DERS subscales, future research should focus on replicating of our pattern of results using a wider range of criterion variables that are particularly relevant to emotion dysregulation (e.g., borderline personality disorder symptoms, disordered eating behaviors, substance use).

DIFFICULTIES IN EMOTION REGULATION SCALE

23

Study limitations must be acknowledged. The use of two large community samples in the present study ensured that our analytic approach was significantly powered to detect the hypothesized effects, and research supports the use of MTurk for the collection of high quality data (Chandler & Shapiro, 2016). However, MTurk samples may not be representative of the general population, with a tendency toward being younger and more highly educated than the general population (Paolacci & Chandler, 2014). Additionally, the samples used in the present study consisted largely of white females. The generalizability of study findings would be increased through replication in general population samples. Additionally, the factor structure of the M-DERS should be examined in clinical populations with heightened levels of emotion dysregulation (e.g., borderline personality disorder) and more variability in endorsement of mood and anxiety disorder symptoms. Limitations notwithstanding, the present results provide support for a bifactor model of the M-DERS, and importantly, provided no support for any tested model for the original DERS. The multidimensionality of the M-DERS appears substantive enough that the continued use of M-DERS subscales is recommended. Our results suggest that the Strategies subscale, however, should not be interpreted independent of the total score. This recommendation is consistent with that of Osborne et al. (2017). We found evidence of incremental utility of the Nonacceptance and Goals M-DERS subscales in predicting general distress (Nonacceptance only) and intolerance of uncertainty (Nonacceptance and Goals). It is recommended that future research examine the incremental utility of M-DERS subscales in predicting other relevant clinical constructs. Taken together, study findings support continued use of the M-DERS as (a) it has superior model fit in comparison to the DERS across several structural models, (b) it is more parsimonious, (c) the

DIFFICULTIES IN EMOTION REGULATION SCALE wording is straightforward, and (d) our findings suggest non-negligible amounts of multidimensionality.

24

DIFFICULTIES IN EMOTION REGULATION SCALE

25

References Aldao, A. (2013). The future of emotion regulation research: Capturing context. Perspectives on Psychological Science, 8, 155-172. Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010). Emotion-regulation strategies across psychopathology: A meta-analytic review. Clinical Psychology Review, 30, 217-237. Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychological Assessment, 10, 176-181. Bardeen, J. R., Fergus, T. A., Hannan, S. M., & Orcutt, H. K. (2016). Addressing psychometric limitations of the Difficulties in Emotion Regulation Scale through item modification. Journal of Personality Assessment, 98, 298-309. Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2012). An examination of the latent structure of the Difficulties in Emotion Regulation Scale. Journal of Psychopathology and Behavioral Assessment, 34, 382-392. Bardeen, J. R. Fergus, T. A., & Orcutt, H. K. (2013a). Experiential avoidance as a moderator of the relationship between anxiety sensitivity and perceived stress. Behavior Therapy 44, 459-469. Bardeen, J. R., Fergus, T. A., & Orcutt, H. K. (2013b). Testing a hierarchical model of distress tolerance. Journal of Psychopathology and Behavioral Assessment, 35, 495-505. Bardeen, J. R., Kumpula, M. J., & Orcutt, H. K. (2013). Emotion regulation difficulties as a prospective predictor of posttraumatic stress symptoms following a mass shooting. Journal of Anxiety Disorders, 27, 188-196.

DIFFICULTIES IN EMOTION REGULATION SCALE

26

Bardeen, J. R., & Michel, J. S. (2017). The buffering effect of religiosity on the relationship between intolerance of uncertainty and depressive symptoms. Psychology of Religion and Spirituality, 9, 90-95. Bentler, P. M., & Chou, C. P. (1987). Practical issues in structural modeling. Sociological Methods & Research, 16, 78-117. Bonifay, W., Lane, S. P., & Reise, S. P. (2017). Three concerns with applying a bifactor model as a structure of psychopathology. Clinical Psychological Science, 5, 184-186. Bonifay, W. E., Reise, S. P., Scheines, R., & Meijer, R. R. (2015). When are multidimensional data unidimensional enough for structural equation modeling? An evaluation of the DETECT multidimensionality index. Structural Equation Modeling: A Multidisciplinary Journal, 22, 504-516. Brown, T.A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York, NY: The Guilford Press. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science, 6, 3-5. Bradbury, C. L., Christensen, B. K., Lau, M. A., Ruttan, L. A., Arundine, A. L., & Green, R. E. (2008). The efficacy of cognitive behavior therapy in the treatment of emotional distress after acquired brain injury. Archives of Physical Medicine and Rehabilitation, 89, S61S68. Carleton, R. N. (2016). Into the unknown: A review and synthesis of contemporary models involving uncertainty. Journal of Anxiety Disorders, 39, 30-43.

DIFFICULTIES IN EMOTION REGULATION SCALE

27

Carleton, R. N., Norton, P. J., & Asmundson, G. J. G. (2007). Fearing the unknown: A short version of the Intolerance of Uncertainty Scale. Journal of Anxiety Disorders, 21, 105117. Catanzaro, S. J., & Mearns, J. (1990). Measuring generalized expectancies for negative mood regulation: Initial scale development and implications. Journal of Personality Assessment, 54, 546-563. Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53-81. Chen, F. F., West, S. G., & Sousa, K. H. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189-225. Cheung, G. W., & Rensvold, R. B. (2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. Journal of Cross-Cultural Psychology, 31, 187-212. Clark, L. A., & Watson, D. (1991). Tripartite model of anxiety and depression: Psychometric evidence and taxonomic implications. Journal of Abnormal Psychology, 100, 316. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309-319. Dong, Y., & Peng, C. Y. J. (2013). Principled missing data methods for researchers. SpringerPlus, 2, 1-17. Dueber, D. M. (2016, December). Bifactor Indices Calculator: A Microsoft Excel-based tool to calculate various indices relevant to bifactor CFA models. Available at http://sites.education.uky.edu/apslab/resources.

DIFFICULTIES IN EMOTION REGULATION SCALE

28

Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491. Fowler, J. C., Charak, R., Elhai, J. D., Allen, J. G., Frueh, B. C., & Oldham, J. M. (2014). Construct validity and factor structure of the Difficulties in Emotion Regulation Scale among adults with severe mental illness. Journal of Psychiatric Research, 58, 175-180. Fox, H. C., Axelrod, S. R., Paliwal, P., Sleeper, J., & Sinha, R. (2007). Difficulties in emotion regulation and impulse control during cocaine abstinence. Drug and Alcohol Dependence, 89, 298-301. Rhemtulla, D. B., Brosseau-Laird, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. Giromini, L., Velotti, P., de Campora, G., Bonalume, L., & Zavattini, G. C. (2012). Cultural adaptation of the Difficulties in Emotion Regulation Scale: Reliability and validity of an Italian version. Journal of Clinical Psychology, 68, 989-1007. Gratz, K. L., & Roemer, L. (2004). Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the Difficulties in Emotion Regulation Scale. Journal of Psychopathology and Behavioral Assessment, 26, 41-54. Gross, J. J. (2015). Emotion regulation: Current status and future prospects. Psychological Inquiry, 26, 1-26.

DIFFICULTIES IN EMOTION REGULATION SCALE

29

Hale, W. Richmond, M., Bennett, J., Berzins, T., Fields, A., Weber, D., Beck, M. & Osman, A. (2016). Resolving uncertainty about the Intolerance of Uncertainty Scale-12: Application of modern psychometric strategies. Journal of Personality Assessment, 98, 200-208. Hayes, S. C., Luoma, J. B., Bond, F. W., Masuda, A., & Lillis, J. (2006). Acceptance and commitment therapy: Model, processes and outcomes. Behaviour Research and Therapy, 44, 1-25. Henry, J. D., & Crawford, J. R. (2005). The short‐form version of the Depression Anxiety Stress Scales (DASS‐21): Construct validity and normative data in a large non‐clinical sample. British Journal of Clinical Psychology, 44, 227-239. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55. Kim, J., & Cicchetti, D. (2010). Longitudinal pathways linking child maltreatment, emotion regulation, peer relations, and psychopathology. Journal of Child Psychology and Psychiatry, 51, 706-716. Kline, R.B. (2016). Principles and practice of structural equation modeling (4th ed.). New York, NY: the Guildford Press. Little, R. J. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198-1202. Lee, J. K., Orsillo, S. M., Roemer, L., & Allen, L. B. (2010). Distress and avoidance in generalized anxiety disorder: Exploring the relationships with intolerance of uncertainty and worry. Cognitive Behaviour Therapy, 39, 126-136.

DIFFICULTIES IN EMOTION REGULATION SCALE

30

Lee, D. J., Witte, T. K., Bardeen, J. R., Davis, M. T., & Weathers, F. W. (2016). A factor analytic evaluation of the Difficulties in Emotion Regulation Scale. Journal of Clinical Psychology, 72, 933-946. Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33, 335-343. McKenzie, K. C., & Gross, J. J. (2014). Nonsuicidal self-injury: An emotion regulation perspective. Psychopathology, 47, 207-219. Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568-592. Meyer, J. F., & Brown, T. A. (2013). Psychometric evaluation of the Thought-Action Fusion Scale in a large clinical sample. Assessment, 20, 764-775. Mennin, D. S., & Fresco, D. M. (2013). Emotion Regulation Therapy. In J.J. Gross (Ed.), Handbook of emotion regulation (2nd ed.). New York, NY: Guilford Press. Moras, K., Di Nardo, P. A., & Barlow, D. H. (1992). Distinguishing anxiety and depression: Reexamination of the reconstructed Hamilton scales. Psychological Assessment, 4, 224227. Muthén, L. K., & Muthén, B. O. (2015). Mplus (Version 7.4). Los Angeles, CA. O'Brien, K. S., Latner, J. D., Puhl, R. M., Vartanian, L. R., Giles, C., Griva, K., & Carter, A. (2016). The relationship between weight stigma and eating behavior is explained by weight bias internalization and psychological distress. Appetite, 102, 70-76. O’Connor, B. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, &

DIFFICULTIES IN EMOTION REGULATION SCALE

31

Computers, 32, 396-402. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867-872. Osborne, T. L., Michonski, J., Sayrs, J., Welch, S. S., & Anderson, L. K. (2017). Factor structure of the Difficulties in Emotion Regulation Scale (DERS) in adult outpatients receiving dialectical behavior therapy (DBT). Journal of Psychopathology and Behavioral Assessment, 39, 355-371. Osman, A., Freedenthal, S., Gutierrez, P. M., Wong, J. L., Emmerich, A., & Lozano, G. (2011). The Anxiety Depression Distress Inventory‐27 (ADDI‐27): a short version of the Mood and Anxiety Symptom Questionnaire‐90. Journal of Clinical Psychology, 67, 591-608. Osman, A., Wong, J. L., Bagge, C. L., Freedenthal, S., Gutierrez, P. M., & Lozano, G. (2012). The Depression Anxiety Stress Scales-21 (DASS‐21): Further examination of dimensions, scale reliability, and correlates. Journal of Clinical Psychology, 68, 13221338. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Current Directions in Psychological Science, 23, 184-188. Perez, J., Venta, A., Garnaat, S., & Sharp, C. (2012). The Difficulties in Emotion Regulation Scale: Factor structure and association with nonsuicidal self-injury in adolescent inpatients. Journal of Psychopathology and Behavioral Assessment, 34, 393-404.

DIFFICULTIES IN EMOTION REGULATION SCALE

32

Racine, S. E., & Wildes, J. E. (2013). Emotion dysregulation and symptoms of anorexia nervosa: The unique roles of lack of emotional awareness and impulse control difficulties when upset. The International Journal of Eating Disorders, 46, 713-720. Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. Reise, S. P., Kim, D. S., Mansolf, M., & Widaman, K. F. (2016). Is the bifactor model a better model or is it just better at modeling implausible responses? Application of iteratively reweighted least squares to the Rosenberg Self-Esteem Scale. Multivariate Behavioral Research, 51, 818-838. Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73, 5-26. Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Applying bifactor statistical indices in the evaluation of psychological measures, Journal of Personality Assessment, 98, 223-237. Salters-Pedneault, K., Roemer, L., Tull, M. T., Rucker, L., & Mennin, D. S. (2006). Evidence of broad deficits in emotion regulation associated with chronic worry and generalized anxiety disorder. Cognitive Therapy and Research, 30, 469-480. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In Clogg, C.C. & von Eye, A. (Eds.), Latent variables analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA: Sage Publications, Inc. Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66, 507-514.

DIFFICULTIES IN EMOTION REGULATION SCALE

33

Shihata, S. S., McEvoy, P. M., & Mullan, B. A. (in press). A bifactor model of intolerance of uncertainty in undergraduate and clinical samples: Do we need to reconsider the twofactor model? Psychological Assessment. Stucky, B. D., & Edelen, M. O. (2015). Using hierarchical IRT to create unidimensional measures from multidimensional data. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183-206). New York, NY: Routledge. Stucky, B. D., Thissen, D., & Edelen, M. O. (2013). Using logistic approximations of marginal trace lines to develop short assessments. Applied Psychological Measurement, 37, 41-57. Tull, M. T., Bardeen, J. R., DiLillo, D., Messman-Moore, T., & Gratz, K. L. (2015). A prospective investigation of emotion dysregulation as a moderator of the relation between posttraumatic stress symptoms and substance use severity. Journal of Anxiety Disorders, 29, 52-60. Tull, M. T., Barrett, H. M., McMillan, E. S., & Roemer, L. (2007). A preliminary investigation of the relationship between emotion regulation difficulties and posttraumatic stress symptoms. Behavior Therapy, 38, 303-313. Tull, M. T., Gratz, K. L., Latzman, R. D., Kimbrel, N. A., & Lejuez, C. W. (2010). Reinforcement sensitivity theory and emotion regulation difficulties: A multimodal investigation. Personality and Individual Differences, 49, 989-994. Tull, M. T., & Roemer, L. (2007). Emotion regulation difficulties associated with the experience of uncued panic attacks: Evidence of experiential avoidance, emotional nonacceptance, and decreased emotional clarity. Behavior Therapy, 38, 378-391.

DIFFICULTIES IN EMOTION REGULATION SCALE Whiteside, U., Chen, E., Neighbors, C., Hunter, D., Lo, T., & Larimer, M. (2007). Difficulties regulating emotions: Do binge eaters have fewer strategies to modulate and tolerate negative affect?. Eating behaviors, 8, 162-169. Zvolensky, M. J., Vujanovic, A. A., Bernstein, A., & Leyro, T. (2010). Distress tolerance: Theory, measurement, and relations to psychopathology. Current Directions in Psychological Science, 19(6), 406-410.

34

DIFFICULTIES IN EMOTION REGULATION SCALE

35

Footnotes 1

Goodness-of-fit Statistics, using adjusted weighted least squares estimation rather than

robust maximum likelihood estimation to test all models, are presented in supplemental Table 1. 2

Post-hoc EFA’s were conducted, per Reviewer request, on the DERS (N = 825) and M-

DERS (N = 564) using parallel analysis with principal axis factoring (O’Connor, 2000). An uninterpretable nine-factor solution was identified via parallel analysis for the DERS. The first to second eigenvalue ratio was 14.89 to 3.71. Consistent with previous research (Bardeen et al., 2016), a five-factor solution was identified for the M-DERS. The first to second eigenvalue ratio was 15.63 to 2.60. Factor one accounted for 53.95% of variance, while 19.66% of variance was attributed to the remaining four factors. These results provide additional support for the assertion that the M-DERS is a multidimensional measure with a strong general factor. Structural regression models and parameter estimates are depicted in supplemental

3

Figure’s 1 and 2. The standardized factor loadings for the manifest indicators in both of these structural regressions is presented in supplemental Table’s 2 and 3.

DIFFICULTIES IN EMOTION REGULATION SCALE

36

Table 1 Goodness-of-fit Statistics for Tested Models χ2 Model

(Scaling correction

RMSEA

CFI

TLI

SRMR

df

RMSEA

594

.111

.108

.113

.64

.62

.11

579

.071

.069

.074

.86

.84

.08

588

.072

.070

.075

.85

.84

.09

558

.070

.068

.073

.86

.85

.08

377

.120

.117

.124

.67

.65

.10

367

.048

.044

.053

.95

.94

.06

372

.051

.047

.055

.94

.94

.07

348

.041

.036

.046

.97

.96

.05

1144

.040

.038

.043

.94

.94

.05

751

.047

.044

.050

.94

.93

.06

LL

UL

factor) DERS One-factor

6600.49 (1.32)

Correlated six-factor

2997.72 (1.29)

Second-order

3107.77 (1.30)

Bifactor

2837.78 (1.27)

M-DERS One-factor

3464.39 (1.68)

Correlated five-factor

851.54 (1.61)

Second-order

915.36 (1.61)

Bifactor

679.45 (1.57)

DASS structural regression

2179.81 (1.43)

IUS structural regression

1670.13 (1.38)

Note. DERS N = 825; M-DERS N = 564. χ2 = Satorra-Bentler scaled chi-square; df = degrees of freedom; RMSEA = root mean square error of approximation; LL = lower limit 90% confidence interval; UL = upper limit 90% confidence interval; CFI = comparative fit index; TLI = Tucker–Lewis fit index; SRMR = standardized root mean square residual. All models computed using maximum likelihood robust estimators (MRL). χ2 values were significant at p < .001.

DIFFICULTIES IN EMOTION REGULATION SCALE

37

Table 2 Correlations between Domain-specific Factors DERS

1

2

3

4

5

1. Awareness

--

2. Clarity

.47

--

3. Impulse

.19

.69

--

4. Goals

.13

.53

.73

--

5. Nonacceptance

.23

.66

.70

.62

--

6. Strategies

.19

.71

.87

.81

.80

6

--

M-DERS

1

2

3

4

1. Identification

--

2. Impulse

.67

--

3. Goals

.42

.69

--

4. Nonacceptance

.64

.67

.61

--

5. Strategies

.60

.80

.80

.74

Note. DERS N = 825; M-DERS N = 564. All correlations significant at p < .001.

5

--

DIFFICULTIES IN EMOTION REGULATION SCALE

38

Table 3 Bifactor Evaluation Indices and Standardized Factor Loadings from Bifactor Model

ω/ωs ω H/ ω HS ECV Residual variance

General Factor

Identification

Impulse

Goals

Nonacceptance

Strategies

.98 .90 .68 .15

.92 .56 .60 .40

.94 .22 .26 .01^

.96 .30 .32 .50

.94 .35 .38 .30

.95 .09 .10 .21

Item # 1 3 4 5 6 7 2 11 15 20 23 27 10 14 16 22 28 8 9 17 19 21 25 12 13 18 24 26 29

Factor Loadings .38 .60 .65 .46 .53 .37 .78 .70 .77 .72 .76 .71 .70 .75 .74 .76 .79 .64 .61 .64 .73 .71 .69 .80 .81 .81 .86 .80 .85

.62 .58 .52 .67 .69 .67 .09^ .56 .45 .31 .47 .58 .58 .51 .54 .53 .37 .48 .61 .64 .30 .54 .52 .40 .35 .27^ .24^ .15^ .14^

Note. ω = omega; ω S = omega subscale; ω H = omega hierarchical; ω HS = omega hierarchical subscale; ECV = explained common variance. DERS N = 825; M-DERS N = 564. All factor loadings and residual variances, except for those identified by ^, were significant at p < .05. ^ = p > .05.

DIFFICULTIES IN EMOTION REGULATION (supplement) Supplemental Table 1 Goodness-of-fit Statistics when adjusted weighted least squares estimation was used to test all models RMSEA

χ

df

RMSEA

One-factor

13184.58

594

Correlated six-factor

5913.20

Second-order Bifactor

Model

2

CFI

TLI

LL

UL

.164

.162

.167

.83

.82

579

.108

.106

.111

.93

.92

7354.11

588

.121

.119

.124

.91

.90

7729.18

558

.128

.126

.131

.90

.89

One-factor

5995.00

377

.163

.159

.166

.89

.88

Correlated five-factor

1585.46

367

.077

.073

.081

.98

.97

Second-order

1898.55

372

.085

.082

.089

.97

.97

Bifactor

1371.53

348

.072

.068

.076

.98

.98

DASS structural regression

2532.13

1144

.046

.044

.049

.98

.98

IUS structural regression

2599.58

751

.066

.063

.069

.97

.96

DERS

M-DERS

Note: DERS N = 825; M-DERS N = 564. χ2 = chi-square; df = degrees of freedom; RMSEA = root mean square error of approximation; LL = lower limit 90% confidence interval; UL = upper limit 90% confidence interval; CFI = comparative fit index; TLI = Tucker–Lewis fit index. χ2 values were significant at p < .001.

DIFFICULTIES IN EMOTION REGULATION (supplement)

Supplemental Table 2 Standardized Factor Loadings for DASS-21 Structural Regression Model MM-DERS Ident Impulse Goals Nonaccept DASS- Depression Anxiety Stress DERS General 21 Item # Factor Item # 1 .36 .62 3 .81 3 .58 .60 5 .75 4 .62 .54 10 .89 5 .43 .69 13 .85 6 .52 .70 16 .86 7 .35 .66 17 .84 2 .76 .15 21 .83 11 .68 .58 2 .53 15 .74 .49 4 .75 20 .70 .36 7 .68 23 .74 .50 9 .77 27 .69 .60 15 .85 10 .69 .60 19 .73 14 .74 .52 20 .85 16 .73 .56 1 .72 22 .73 .56 6 .76 28 .77 .40 8 .77 8 .62 .51 11 .81 9 .60 .62 12 .81 17 .63 .65 14 .68 19 .73 .32 18 .75 21 .69 .57 25 .66 .55 12 .86 13 .87 18 .85 24 .89 26 .81 29 .87 Note: Corresponding structural regression model presented in Supplemental Figure 1; all parameter estimates fully standardized (i.e., STDXY standardization); all item loadings are significant at p < .001; DASS-21 = Depression Anxiety Stress Scales-21 item version; M-DERS = Modified Difficulties in Emotion Regulation Scale; Ident = Identification; Nonaccept = Nonacceptance.

DIFFICULTIES IN EMOTION REGULATION (supplement)

Supplemental Table 3 Standardized Factor Loadings for IUS-12 Structural Regression Model MM-DERS Ident Impulse Goals Nonaccept IUS-12 IUS DERS General Item # General Item # Factor Factor 1 .36 .62 1 .75 3 .58 .60 2 .59 4 .62 .54 3 .78 5 .44 .69 4 .55 6 .52 .70 5 .71 7 .35 .66 6 .76 2 .76 .15 7 .83 11 .69 .58 8 .63 15 .74 .49 9 .78 20 .70 .36 10 .77 23 .74 .50 11 .57 27 .69 .60 12 .78 10 .69 .60 14 .74 .52 16 .73 .56 22 .73 .56 28 .78 .40 8 .62 .50 9 .60 .62 17 .63 .65 19 .73 .32 21 .68 .57 25 .66 .55 12 .86 13 .87 18 .85 24 .89 26 .81 29 .87 Note: Corresponding structural regression model presented in Supplemental Figure 2; all parameter estimates fully standardized (i.e., STDXY standardization); all item loadings are significant at p < .001; IUS-12 = Intolerance of Uncertainty Scale—Short Form; M-DERS = Modified Difficulties in Emotion Regulation Scale; Ident = Identification; Nonaccept = Nonacceptance.

DIFFICULTIES IN EMOTION REGULATION (supplement)

Supplemental Figure 1. Structural regression model with the bifactor model of the Modified Difficulties in Emotion Regulation Scale regressed onto general distress. Note. all parameter estimates fully standardized (i.e., STDXY standardization). ***p < .001.

DIFFICULTIES IN EMOTION REGULATION (supplement)

Supplemental Figure 2. Structural regression model with the bifactor model of the Modified Difficulties in Emotion Regulation Scale regressed onto intolerance of uncertainty. Note. all parameter estimates fully standardized (i.e., STDXY standardization). *p < .05. ***p < .001.