Predicting Match Outcomes: Science, Practice, and ... - PsycNET

Training and Education in Professional Psychology 2014, Vol. 8, No. 1, 68 – 82

© 2014 American Psychological Association 1931-3918/14/$12.00 DOI: 10.1037/tep0000030

Predicting Match Outcomes: Science, Practice, and Personality Jennifer L. Callahan and Lindsey R. Hogan

Elizabeth A. Klonoff

University of North Texas

San Diego State University and University of California, San Diego

Frank L. Collins, Jr.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

University of North Texas Internship applicants (N ⫽ 601) recruited from Council of University Directors of Clinical Psychology member PhD programs completed surveys before and after Match Day about their academic achievements and clinical training, personality, and match outcome characteristics. Results revealed strong evidence that the single best predictor of matching among these students is the number of interview offers attained. A low number of attained interviews (6 or fewer) forecasts increased likelihood of going unmatched. Entering the match a second or more time is also associated with not matching. Additional analyses indicated intervention and assessment hours significantly interact to impact number of interview invitations, suggesting that monitoring the accrual of proportionate clinical hours (rather than focusing simply on attaining more client contact hours) is important throughout preinternship training in this sample. Although associated with only a small amount of variance, facets of personality do appear to be significantly associated with the number interview offers obtained and internship match outcomes. Finally, a significant interaction between science, as indicated by research achievements, and practice, as indicated by proportionate clinical hours, was also observed. This represents the first empirical demonstration of the Boulder model’s philosophical premise regarding the training of clinical psychologists. Specific mentoring suggestions are offered, as well as recommendations for future directions in training, policy, and research. Keywords: internship, APPIC, clinical training, match imbalance, Boulder model

The internship represents a capstone training experience, providing an integration of the knowledge acquired during one’s graduate career (Collins, Callahan, & Klonoff, 2007; Lamb, Baker, Jennings, & Yarvis, 1982). Bridging the gap between graduate training and entrance into the professional world, internship is also a requirement of all American Psychological Association (APA) accredited programs (Commission on Accreditation, 2009; Guidelines and Principles, Doctoral Programs, Domain A.4). For the last two decades, however, our field has been increasingly faced with the problem of an internship shortage (Baker, McCutcheon, & Keilin, 2007; Keilin, Baker, McCutcheon, & Peranson, 2007; Keilin, Thorn, Rodolfa, Constantine, & Kaslow, 2000; Oehlert & Lopez, 1998; Thorn & Dixon, 1999). Conceptualized by many as a supply-and-demand problem (e.g., Baker, McCutcheon, & Keilin, 2007; Keilin, Baker, McCutcheon, & Peranson, 2007; Rodolfa, Bell, Bieschke, Davis, & Peterson, 2007), the number of applicants exceeds the number of available internship positions. With the percentage of students unmatched each year falling within the range of about 21 to 25% since 2007 (Association of Psychology Postdoctoral and Internship Centers [APPIC], 2013), the imbalance does not appear to be self-correcting. There have been a number of suggested interventions to address the imbalance, stemming from many perspectives, including seeking federal support to fund more internship sites, requiring training programs to disclose match rate outcomes, creation of various work forces, and finding ways to grow existing internship sites and create new ones, to name a few (e.g., Baker, McCutcheon, &

JENNIFER L. CALLAHAN earned her PhD in Clinical Psychology from the University of Wisconsin-Milwaukee, completed her internship and postdoctoral training at Yale University, and holds board certification in Clinical Psychology. She is currently Associate Professor and Director of Clinical Training for the Clinical Psychology program at the University of North Texas, where she directs the Evidence-based Training and Competencies Research Lab. LINDSEY R. HOGAN is a doctoral candidate in Clinical Psychology at the University of North Texas. She has broad interests in training and supervision, psychotherapy process and outcome, and evidence-based practice. ELIZABETH A. KLONOFF is Professor of Psychology at San Diego State University (SDSU) and Professor of Psychiatry at University of California San Diego, where she serves as the SDSU Co-Director of the Joint Doctoral Program in Clinical Psychology. In addition to her work in education and training, her research focuses on health, cancer disparities, and minors’ access to tobacco. FRANK L. COLLINS, JR. earned his PhD in Clinical Psychology from Auburn University. Before his death, Frank was Professor and Director of Clinical Training for the Clinical Health Psychology and Behavioral Medicine program at the University of North Texas. Frank contributed actively to the national training and education scene for nearly 20 years receiving numerous honors for his contributions. CORRESPONDENCE CONCERNING THIS ARTICLE should be addressed to Jennifer L. Callahan, University of North Texas, 1155 Union Circle #311280, Denton, TX 76205. E-mail: [email protected] 68


INTERNSHIP MATCH OUTCOMES

Keilin, 2007; Collins et al., 2007; Hatcher, 2011; Madson, Hasan, Williams-Nickelson, Kettman, & Sands Van Sickle, 2007; Rodolfa et al., 2007). There have also been calls for a shift at the philosophical or conceptual level, when considering a solution to the imbalance problem. Perhaps most notably, Hatcher (2011), in a special section of Training and Education in Professional Psychology (Volume 5, Issue 3), conceptualized the internship supply as a “common-pool resource,” or CPR, drawing from sharedresource management literature and asserting the need for governance to manage the scarce resource of internship sites. His pivotal remarks drew attention to the multifaceted, multiorganizational nature of the imbalance problem. Others responded by echoing the need for a more collaborative approach (e.g., DeMers, 2011; McCutcheon, 2011), and consideration of not only the imbalance itself but also the implications for the quality of our profession as a whole. For instance, though some propose that relaxing accreditation and licensing standards might quickly increase the available supply of internships, others point out the potential quality-control concerns across our field associated with lowering training standards. Such a concern seems to have some face validity. In economics, when demand exceeds supply, prices tend to rise. Correspondingly, when demand for internship positions exceeds the available supply, then the owners of the supply (internship sites) may raise the cost associated with securing an internship. In short, despite applicants having been vetted by their programs as ready for internship, internship sites may functionally raise the requirements of what is needed to match because they have the ability to be highly selective. However, this does not necessarily assure that the internship sites will obtain higher quality applicants—variables upon which sites are making selections might not be validly tied to applicants’ competencies. Notably, there are no standardized measures of clinical or professional competency contained within the application for internship. Another implication of this increased selectivity is that the methods used to select applicants are not transparent processes for applicants and do not facilitate a collaborative process among the full scope of stakeholders. What is clear is that as applicants are faced with the predicament of the ongoing imbalance, competition among applicants vying for internship positions is intense. In light of the significant imbalance, an immediate research focus of benefit to existing students is to elucidate what, if anything, they can do personally to increase their chances of successfully securing an internship at one of the sites to which they have chosen to apply (Callahan, Collins, & Klonoff, 2010; Ginkel, Davis, & Michael, 2010).

What Makes a Good “Match” in the Match? The assumption is often made by both graduate students and faculty that students who acquire a competitive number of clinical hours, publications, integrated reports, and so forth during their graduate training will be more likely to successfully match (Bieschke, Bell, Davis III, Hatcher, Peterson, & Rodolfa, 2011). Applicants often expend great amounts of energy working within this strategic, achievement-oriented approach. These efforts are not entirely in vain; it is true that type of training and other training characteristics appear to have some influence on match rates. For instance, students trained within a scientist-practitioner model

69

demonstrate significantly higher match rates (Callahan et al., 2010; Neimeyer, Rice, & Keilin, 2007). In fact, some have concluded, as a result of these findings, that a free-market approach to the imbalance problem is best, in which less accomplished applicants will not match and therefore not enter the profession. However, emphasizing academic achievements that are thought to be latent indicators of salient clinical and professional competencies in addressing the match is problematic. The science pertaining to the measurement of competencies is nascent. There are no published or known existing studies examining whether theoretically derived competencies map onto the achievement-centered academic variables captured on the APPIC Application for Psychology Internships (AAPI). We do not yet have enough information about the presence or strength of association among clinical or professional competencies and AAPI variables. One step toward progressing this much-needed area of inquiry is to determine which variables from the existing AAPI are associated with matching. This approach may facilitate developing preliminary corresponding benchmarks within training programs to assist students in their development of needed competencies prior to applying for internship (Collins et al., 2007). Callahan et al. (2010) provided a first effort at exploring applicant and program characteristics associated with matching. By looking at predictors of matching, trainee preferences regarding “fit” are inherently, albeit imprecisely, considered. Similarly, internship sites’ evaluations of fit with specific applicants are also incorporated. Although rudimentary, it provided a basis for beginning to empirically explore the issue of fit. Findings from that study revealed that the number of interview offers an applicant garnered was the only variable significantly associated with match outcome (matched/not matched), after controlling for the number of applications submitted. A hierarchical regression analysis demonstrated that the most variance in match outcome was associated with offers to interview (20.5%), followed by educational variables (such as clinical hours and completion of the dissertation proposal; 4.4%), variables under less control of the student (such as citizenship and cohort size; 1.9%), and number of applications submitted, which was associated with only 0.4% of the variance in match outcome. Given the importance of attaining interviews, further examination of correlates to interview success is merited. Within the broad employment literature, research indicates that applicant personality, as reflected by Big Five personality traits, is important in both obtaining interviews and in interview performance. Applicants who were more extraverted and conscientious were invited for more follow up interviews (Caldwell & Burger, 1998). Furthermore, with respect to job performance, conscientiousness and emotional stability (low neuroticism) have been found to be positively associated traits (Barrick & Mount, 1991; Salgado, 1997). Personality may impact the job application process more distally as well. Caldwell and Burger (1998) found that job applicants who were more extroverted, conscientious, and open to experience tended to use more social methods of preparing for the interview, such as talking with others who may be familiar with the potential employer, whereas applicants with higher conscientiousness were more likely to prepare themselves through familiarizing themselves with the employer via reading materials about the company (e.g., brochures). To our knowledge, no research to date has empirically explored the role of personality in internship match outcomes. However, our conversations with fac-

CALLAHAN, HOGAN, KLONOFF, AND COLLINS

70

ulty, directors of clinical training, internship training directors, and applicants all frequently mention applicant personality as potentially salient to interview performance.


Applicant Personality and Fit Conceptually, there is some indication that the concept of “fit” might reflect personal characteristics of applicants, including personality, which may provide a foundation for developing competencies (Collins et al., 2007). It is generally acknowledged that graduate programs in psychology value applicants with certain personality characteristics, including emotional maturity and integrity (Appleby, Keenan, & Mauer, 1999), as well as other common selection criteria such as research experience (Landrum, Jeglum, & Cashin, 1994; Norcross, Kohout, & Wicherski, 2005). Investigations of inclusion and exclusion criteria utilized by internship site directors have similarly found fairly consistent results, with educational and program factors such as APA accreditation, broad clinical experience, letters of recommendation, and lack of uncompleted academic requirements carrying a significant amount of weight behind fit (Rodolfa et al., 1999). Importantly, a more recent investigation by Ginkel et al. (2010) reexamined selection criteria used by internship training directors and found that “personality characteristics of the applicant” (p. 216) were statistically significantly more important as an inclusion criterion than previously reported by (Rodolfa et al., 1999). Ginkel and colleagues ranked applicant personality as sixth among nearly 40 selection criteria (exceeding the ranking of graduate education, number of supervised practicum hours, APA status of doctoral program, publications, or letters of recommendation, to name just a few). In considering the role of personality characteristics, it is notable that both Rodolfa’s team and Ginkel et al. found the internship interview to be a significant exclusion criterion in the selection of interns, with the interview ranked as much more important in 2010 than in 1999. Ginkel and colleagues also found that applicant personality, applicant demeanor, and personal reactions to the applicant were within the top 10 exclusion criteria, and suggested “personality characteristics might be the deciding factor that separates [applicants] from their peers” (p. 217). However, no research has been conducted to determine which, if any, specific personality traits may be associated with securing an internship in the APPIC match.

Hypotheses We conceptualized our hypotheses as aligning with the three major benchmarks for applicants in the internship process: applying, interviewing, and matching. With respect to applying to internship sites, on the base of the broader employment literature (Caldwell & Burger, 1998), we expected that personality variables would be associated with the number of applications submitted by trainees seeking internship. We also expected to find further support for the small body of empirical findings linking applicant characteristics to the application process. In particular, on the basis of the Callahan et al. (2010) study findings, we expected the following variables to be significantly associated with the number of applications submitted by trainees seeking internship: marital status, geographic restriction, responsibility for dependents, citizenship status, and degree plan (PhD vs. PsyD).

Relative to attaining offers to interview for internship, we again drew from the broader employment literature (e.g., Caldwell & Burger, 1998), as well as internship director survey findings regarding the importance of personality (Ginkel et al., 2010), and hypothesized that low neuroticism and high conscientiousness would be associated with the number of interview offers attained. Given that our recruitment plan was centered on Council of University Directors of Clinical Psychology (CUDCP) program applicants from predominantly scientist-practitioner (aka, the Boulder model; Belar & Perry, 1992; Raimy, 1950) programs, we also made the following theoretically derived investigations of the data: (a) in light of a position statement by CUDCP (2011), articulating an expectation that client contact hours interact meaningfully with supervision hours in fostering readiness for internship, we explored whether an empirically demonstrated interaction between intervention and assessment hours with supervision hours would be associated with the number of offers to interview attained; and (b) we hypothesized that an interaction between science and practice variables would be evident (publications, as indicative of science; clinical hours, as indicative of practice) and associated with readying trainees for the profession (with number of offers to interview as a proxy indicator of readiness to enter the field). Regarding match outcomes (match/no match), in light of employment literature (Caldwell & Burger, 1998) and internship director survey findings regarding the importance of personality (Ginkel et al., 2010), we hypothesized that personality variables of neuroticism, extraversion, and conscientiousness would be associated match outcome. Given the findings reported by Callahan et al. (2010), we also expected that the number of interview offers attained by applicants would forecast match outcome (match vs. no match) and be associated with the most variance in match outcomes. Finally, we sought to deepen our understanding of the link between offers to interview and eventual match outcome by (a) exploring within this sample the probability of matching as a function of number of interviews attained, and (b) identifying a reasonable critical value for number of attained interview offers that distinguishes matched from unmatched applicants.

Method Participants Recruitment for this study targeted students from member programs of the CUDCP, which primarily utilize a scientistpractitioner approach to training. We wanted to be able to directly compare the current findings with those reported by Callahan et al. (2010), who also recruited CUDCP students. It was also hoped that students from these programs might be more likely to reflect the characteristics associated with securing an internship; the previous study by Callahan et al. (2010) found that these students matched at a collectively higher rate in 2009 than the overall sample in the APPIC survey. For the current study, applicants to the 2010 and 2011 internship match were recruited for participation. In 2010, 236 applicants completed a prematch survey. At the postmatch follow-up, 84.7% of these students participated. A total of 365 students participated in a prematch survey in the 2011 sample, with 70.4% participating in the Phase I Match follow-up survey, and 53.1% participating in the Phase II Match follow-up survey.


71

Table 1 Applicant Demographics in 2009, 2010, and 2011 Among Applicants for Internship 2009


Variable Age Gender Female Male Race/Ethnicity White (non-Hispanic) Asian/Pacific Islander African American/Black Hispanic/Latino Biracial/Multiracial Other Citizenship U.S. Canada Other Sexual orientation Heterosexual Lesbian Bisexual Gay male Other Disability status Not disabled Identified disability Disability disclosed or otherwise evident in application Marital status Married/Partnered Not married/Partnered Responsible for dependents No Yes Self-reported youth economic standing Lower class Lower-middle class Middle class Upper-middle class Upper class Education-related debt

n

%

2010

M

SD

n

29.64

4.82

233

%

2011

M

SD

n

29.00

3.70

363

%

313 68

81.9 17.8

185 49

79.1 20.9

280 82

77.3 22.7

301 30 13 14 22 3

78.4 7.8 3.4 3.6 5.7 .8

194 15 8 8 3 6

82.9 6.4 3.4 3.4 1.3 2.6

285 20 17 18 14 6

78.7 5.5 4.7 5.0 3.9 1.7

354 10 18

92.7 2.6 4.7

211 12 10

90.6 5.2 4.3

340 13 8

94.2 3.6 2.2

354 5 10 9 3

92.9 1.3 2.6 2.4 .8

217 8 6 1 1

92.7 3.4 2.6 .9 .4

333 7 10 11 2

91.7 1.9 2.8 3.0 .6

372 10

97.4 2.6

227 6

97.4 2.6

346 16

95.6 4.4

4

33.3a

4

80a

3

50.9 49.1

129 104

55.4 44.6

187 176

51.5 48.5

328 54

85.9 14.1

84.6 15.4

303 59

83.7 16.3

18 56 135 109 15 329

5.4 16.7 40.3 32.5 4.5

198 36 199 10 21 78 82 8 197

5.0 10.6 39.2 41.2 4.0

10 46 86 99 8 238

4.0 18.5 34.5 39.8 3.2

$4,885

$41,423

$43,517

SD

29.16

3.78

$36,403

$39,363

23.1a

195 188

$47,104

M

Note. Due to occasional missing data, the total number of participants represented is not constant across variables within a given sample. Data pertaining to 2009 applicants is drawn from the Callahan et al. (2010) study. Although these variables were included in Callahan et al.’s analyses, they rarely reported basic frequency or descriptive statistics in that publication. To facilitate comparisons across samples, data regarding their study as presented in this table was obtained via personal communication with Callahan and represents the prematch survey of 2009 applicants. Thus, statistics reported here do not always exactly reproduce the limited descriptive information provided in the earlier article, which drew only from applicants who also completed a postmatch survey. a Indicates the percentage among those reporting disability.

Applicants in this sample closely resemble the 2009 sample described by Callahan et al. (2010; see Table 1 for a breakdown by match year). In our total sample of 601 applicants, the majority of applicants are female (78%), U.S. citizens (92.8%), White (nonHispanic ⫽ 80.4%), and heterosexual (92.1%), with an average age of 29.10 years (SD ⫽ 3.75). Applicants also reported information about their amount of education-related debt (M ⫽ $39,559.27, SD ⫽ $43,291.74) and their socioeconomic status (SES) growing up. The largest proportion of students in our sample categorized their familial SES as upper-middle class (40.4%), followed by middle class (36.6%), lower middle class (15%), lower class (4.5%), and upper class (3.6%). Table 2 displays educational and program characteristics of applicants by match year, and shows the majority of applicants

came from clinical psychology PhD programs (90.4%). The match rate for our sample was 88.6%, with applicants submitting an average of 15.44 applications (SD ⫽ 4.01). Of students who matched, the largest proportion (43.0%) did so at their top-ranked internship site (see Table 3 for descriptive statistics by match year). All participants in this study were treated with adherence to the Ethical Principles of Psychologists and Code of Conduct (APA, 2010), following institutional review board approval.

Procedures To accomplish recruitment of participants in this study, directors of clinical training at CUDCP member programs were contacted


72

Table 2 Training and Program Characteristics Among 2009, 2010, and 2011 Applicants for Internship 2009


Variable Program type Clinical Combined Counseling School Degree sought PhD PsyD Accredited program Yes No Training model Scientist-practitioner Scholar-practitioner Clinical scientist Local clinical scientist Other Previous training No prior graduate coursework Graduate coursework in psychology, without earned degree Graduate coursework, not in psychology, without earned degree Master’s degree in psychology Master’s degree in related field Master’s degree in unrelated field Doctoral Training Size of entering doctoral cohort Number of students in doctoral program applying for internship Years enrolled in doctoral program Total intervention and assessment hours Total supervision hours Number of integrated reports Completed comprehensive exam prior to applying for internship Dissemination Activities Number of total publications Number of peer-reviewed publications, specifically (includes in press) Number of presentations Graduate record examination scores Verbal Quantitative Dissertation Status Proposed prior to applying for internship Defended prior to match Practicum setting experience Child guidance Community mental health center Department clinic Forensic Medical clinic/hospital Inpatient psychiatric hospital Outpatient hospital University counseling center School Veterans Affairs Medical Center Other

n

%

371 12 0 0

M

2010 SD

n

%

96.9 3.1 0.0 0.0

231 3 0 0

346 37

88.0 9.4

205 29

378 4

99.0 1.0

294 31 50 3 5

76.8 8.1 13.1 .8 1.3

177 24 27 4 1

267

69.7

16 7 69 11 13

M

2011 SD

n

%

98.7 1.3 0.0 0.0

347 10 5 1

95.6 2.8 1.4 .3

87.6 12.4

334 25

92.3 6.8

71 100 0 0.0

352 9

97.5 2.5

76.0 10.3 11.6 1.7 .4

287 24 51 0 0

79.5 6.6 14.1 0.0 0.0

173

73.9

265

74.9

4.2

12

5.1

14

4.0

1.8 18.0 2.9 3.4

2 35 3 9

.9 15.0 1.3 3.8

0 63 1 11

0.0 17.8 .3 3.1

382

9.31

4.66

233

9.47

5.42

382 383 362 359 358

8.47 4.99 997.63 472.96 35.66

5.12 1.02 480.85 226.24 114.37

233 234 224 221 222

8.82 4.94 915.41 450.80 40.17

5.67 .95 398.97 199.35 68.90

359

98.9

225

99.6

372 365 330 329

M

SD

362

9.13

6.05

362 357 323 321 325

7.65 5.03 969.26 453.13 37.50

4.78 1.02 419.45 213.78 54.98

98.5

364

3.52

3.63

226

4.62

5.02

339

335

4.69

5.12

344 364

2.22 8.87

2.42 7.33

216 225

2.88 10.36

3.32 8.08

326 339

322 335

3.16 11.57

4.26 7.89

383 383

435.74 488.56

251.58 285.67

169 171

624.32 680.70

73.86 70.43

254 255

249 250

629.60 686.96

80.22 74.07

306 15

84.1 4.1

207 19

91.2 8.4

300 17

89.8 5.1

21 146 306 73 198 120 103 128 106 72 140

5.5 38.0 79.7 19.0 51.6 31.3 26.8 33.3 27.6 18.8 36.5

15 85 185 47 131 65 57 72 50 56 63

6.4 36.0 78.4 19.9 55.5 27.5 24.2 30.5 21.2 23.7 26.7

20 141 255 65 186 125 96 109 86 62 120

5.5 38.6 69.9 17.8 51.1 34.2 26.3 29.9 23.6 17.0 32.9

Note. Due to occasional missing data, the total number of participants represented is not constant across variables within a given sample. Data pertaining to 2009 applicants is drawn from the Callahan et al. (2010) study. Although these variables were included in Callahan et al.’s analyses, they rarely reported basic frequency or descriptive statistics in that publication. To facilitate comparisons across samples, data regarding their study as presented in this table was obtained via personal communication with Callahan, and represents the prematch survey of 2009 applicants. Thus, statistics reported here do not always exactly reproduce the limited descriptive information provided in the earlier article, which drew only from applicants who also completed a postmatch survey.


73

Table 3 Match Outcome Characteristics Among 2009, 2010, and 2011 Applicants for Internship


2009 Variable

n

Number of applications submitted Number of interview offers Number of submitted rankings Entered match as a couple Applied with geographical restriction Number of years applied First time Second time Third or more time Match status Matched in February (aka Phase I) Not matched in February (aka Phase I) Withdrew, prior to Phase I Match Secured internship via clearinghouse, Phase II Match, or other means Matched at first-ranked internship Matched at second-ranked internship Matched at third-ranked internship Matched at fourth-ranked or lower internship Matched at APA- or CPA-accredited internship Matched at APPIC member internship site Location of internship U.S. Canada Position Type Full-time internship Paid internship Internship setting Armed Forces medical center Child/Adolescent psychiatry Community mental health center Consortium Medical school Prison/correctional center Private general hospital Private outpatient clinic Private psychiatric hospital Psychology department School district Public hospital (state/county/other) University counseling center Veterans Affairs medical center Other

383 383 382a 12 147

%

2010 M

SD

n

14.61 7.84 9.09

4.13 3.47 5.55

3.1 38.5

235 234 230 2 80

356 26 1

93.0 6.8 0.3

290 49 0

%

2011 M

SD

n

%

14.66 8.01 9.09

4.10 3.40 4.27

0.9 34.2

363 363 359 6 124

1.7 34.2

215 21 0

91.1 8.9 0.0

334 26 1

92.5 7.2 0.3

85.5 14.5 0.0

175 25 0

87.5 12.5 0.0

229 24 3

89.5 9.4 1.2

12 119 64 32 67

24.5b 42.2 22.7 11.3 23.8

10 78 34 28 33

40.0b 45.1 19.7 16.2 19.1

10 88 54 25 46

41.7b 41.3 25.4 11.7 21.5

149 100

98.7 66.2

173 137

93.5 74.1

221 150

96.5 65.5

143 7

95.3 4.7

178 7

96.2 3.8

217 8

96.4 3.6

121 120

80.1 79.5

155 155

83.8 83.8

179 178

78.2 77.7

0 50 14 24 48 7 7 13 10 7 5 17 12 47 6

0.0 33.1 9.3 15.9 31.8 4.6 4.6 8.6 6.6 4.6 3.3 11.3 7.9 31.1 4.0

3 39 20 21 64 3 21 9 11 10 5 30 19 61 12

1.6 21.1 10.8 11.4 34.6 1.6 11.4 4.9 5.9 5.4 2.7 16.2 10.3 33.0 6.5

1 71 22 28 69 9 9 18 14 7 9 25 20 78 11

0.4 31.0 9.6 12.2 30.1 3.9 3.9 7.9 6.1 3.1 3.9 10.9 8.7 34.1 4.8

M

SD

15.95 7.94 9.43

4.00 3.51 4.58

Note. Due to occasional missing data, the total number of participants represented is not constant across variables within a given sample. Data pertaining to 2009 applicants is drawn from the Callahan et al. (2010) study. Although these variables were included in Callahan et al.’s analyses, they rarely reported basic frequency or descriptive statistics in that publication. To facilitate comparisons across samples, data regarding their study as presented in this table was obtained via personal communication with Callahan, and represents the prematch survey of 2009 applicants. Thus, statistics reported here do not always exactly reproduce the limited descriptive information provided in the earlier article, which drew only from applicants who also completed a postmatch survey. APA ⫽ American Psychological Association; CPA ⫽ Canadian Psychological Association; APPIC ⫽ Association of Psychology Postdoctoral and Internship Centers. a One outlier was removed prior to computing this statistic. b The number of unmatched applicants after February’s match day was used as the denominator to compute this percentage.

and asked to solicit all students in their program who were participating in the match. To make sure that we gathered the most up-to-date information possible prior to match day (e.g., number of offers to interview), we targeted our recruitment efforts to immediately follow the deadline for submission of rank order lists to the match service. Because it was anticipated that the outcome of match day might influence student responses on the survey, it was

required that surveys be submitted prior to match day. At the conclusion of the prematch survey, applicants were directed to a new page (not linked to their previously submitted data) to provide follow-up contact information. This information was then used to directly contact applicants immediately after match day to request continued participation in a brief postmatch survey. To maintain confidentiality, applicants were asked to use an impersonal iden-


74

tifier to anonymously link their pre- and postmatch data. Due to space constraints, it is not practical to include complete surveys in this medium; however, all surveys are freely available upon request from the corresponding author.

0.77 (Agreeableness); alpha coefficients for the facet scores ranged from 0.58 to 0.91, with a mean coefficient of 0.78.

Results


Standardized Measures The International Personality Item Pool (IPIP) is a multiscale measure of normal personality that consists of 300 self-report items, answered using a 5-point scale. The version of the IPIP used here was designed to reliably measure the domains and facets of personality captured by Costa and McCrae’s (1992) NEO Personality Inventory, Revised (NEO-PI-R). The IPIP has five domain scales, each representing a Big Five personality trait, with each domain measuring several facets of the specified trait (i.e., Neuroticism [anxiety, hostility, depression, self-consciousness, impulsivity, and vulnerability], Extraversion [warmth, gregariousness, assertiveness, activity, excitement-seeking, and positive emotions], Openness [fantasy, aesthetics, feelings, actions, ideas, and values], Agreeableness [trust, modesty, compliance, altruism, straightforwardness, and tender mindedness], and Conscientiousness [competence, self-discipline, achievement-striving, dutifulness, order, and deliberation]). Within the current sample, alphas associated with Big Five trait scores were 0.84 (Neuroticism), 0.78 (Extraversion), 0.64 (Openness), 0.79 (Conscientiousness), and

Applicant Characteristics Associated With Applying for Internship Table 4 provides a summary of the replications of Callahan et al. (2010) analyses pertaining to applicant characteristics associated with the number of applications submitted for internship. With respect to personality, no personality trait measures (Neuroticism, Extraversion, Openness, Agreeableness, Conscientiousness) were significantly associated with the number of applications submitted. In the interest of obtaining as much information as possible about applicant characteristics that may be associated with the internship process, we chose to then examine personality at the facet level. A few facet measures of personality were significantly correlated with the number of applications submitted. Specifically, facets measuring Cooperation (Agreeableness trait; r ⫽ .11, p ⫽ .02), Sympathy (Agreeableness trait; r ⫽ .11, p ⫽ .02), and Orderliness (Conscientiousness trait; r ⫽ ⫺.12, p ⫽ .01) were positively correlated with the number of applications submitted by trainees.

Table 4 Summary of Replication Analyses Pertaining to Number of Applications Submitted for Internship Variable Gender Female Male Marital status Married/partnered Not married/partnered Responsibility for dependents Yes No Geographic restriction Restricted Not restricted Citizenship status U.S. Citizen International student Disability status Disabled Not disabled Degree plan PhD PsyD Ethnicity African American/Black American Indian/Alaska Native Asian/Pacific Islander Hispanic/Latino White, non-Hispanic Biracial/Multiracial Other

Effect sizea

Finding replicates 2009 study

⫺0.35, 1.24

.09

Yes

⬍.001

0.53, 1.84

.29

Yes

2.76 (592)

.01

0.36, 2.16

.23

Yes

14.57, 4.69 15.09, 3.67

3.81 (592)

⬍.001

0.65, 2.02

.31

Yes

15.52, 3.98 14.77, 5.28

1.17 (589)

.24

⫺0.51, 2.02

.10

No

15.73, 5.28 15.44, 4.04

⫺0.32 (591)

.75

⫺2.04, 1.46

.03

Yes

15.07, 3.88 19.42, 4.15

⫺7.33 (61.32)b

15.21, 2.99 14.50, 0.71 16.62, 0.59 14.65, 3.51 15.32, 4.20 16.59, 2.58 17.17, 4.99

1.33 (6, 586)

M, SD

t or F (df)

p

15.35, 4.21 15.79, 3.65

1.10 (591)

.27

14.89, 4.37 16.07, 3.66

3.55 (592)

14.38, 4.65 15.64, 3.95

95% CI (LL, UL)

⬍.001

⫺5.54, ⫺3.16

1.87

Yes

.24

13.94, 16.47 8.15, 20.85 15.42, 17.81 13.24, 16.07 14.94, 15.69 15.26, 17.91 14.00, 20.34

.01

Yes

Note. df ⫽ degrees of freedom; CI ⫽ confidence interval; LL ⫽ lower limit; UL ⫽ upper limit. Cohen’s d for t test; eta squared for analysis of variance. b Represents the degrees of freedom associated with this analysis when equal variances are not assumed.

a


75


Applicant Characteristics Associated With Obtaining Interview Offers Bivariate correlations were conducted in order to determine which, if any, applicant characteristics were correlated with the number of offers to interview. The number of applications submitted was significantly correlated with the number of interview offers attained (r ⫽ .40, p ⬍.001). Because the findings reported in Table 4 indicated some characteristic differences in regard to the number of applications submitted by applicants, subsequent correlations controlled for the number of applications submitted. Only the following variables demonstrated significant correlations with the number of interviews offered to applicants. The total number of intervention and assessment hours (r ⫽ .23, p ⬍.001), total number of publications (r ⫽ .18, p ⬍.001), number of supervision hours (r ⫽ .16, p ⬍.001), number of peer-reviewed publications (including those that are in press; r ⫽ .14, p ⫽ .001), and number of presentations (r ⫽ .11, p ⫽ .02) on the applicant’s curriculum vitae at the time of application were all significantly positively correlated with the number of interview offers attained. With respect to personality measures, only the facet of Trust (Agreeableness trait; r ⫽ .09, p ⫽ .045) demonstrated statistical significance. Most of the associated variables reflected aspects of the scientist-practitioner training model; as such, these variables would, in theory, be expected to interact with one another meaningfully. We therefore empirically examined whether theoretically derived interactions among these variables were predictive of the number of offers to interview. After standardizing our variables, two interaction terms were created and tested. The first term reflected the interaction of intervention/assessment hours and supervision hours. A hierarchic linear regression was conducted with the standardized score for total number of intervention and assessment hours (b ⫽ 0.89, SE ⫽ 0.16) and standardized score for total number of supervision hours (b ⫽ 0.36, SE ⫽ 0.15) entered in the first block, F(2, 533) ⫽ 19.79, p ⬍.001, and associated with 6.9% of the variance in number of offers to interview for internship. The second block contained the interaction term (b ⫽ ⫺0.48, SE ⫽ 0.14) and significantly increased the predictability of number of offers to interview by associating with a small amount of additional variance (1.9%), ⌬F (1, 532) ⫽ 11.25, p ⫽ .001. Figure 1 provides an illustration of this interaction. In light of the finding that intervention and assessment hours significantly interacted with supervision hours, we created a proportion to represent the ratio of intervention and assessment hours to supervision hours (M ⫽ 2.41, SD ⫽ 1.53). After standardizing this proportion, we created another interaction term. This term represented the interaction of the standardized clinical-hours proportion and the standardized total number of publications. This interaction sought to look at the interaction between science training, as indicated by research achievements (total number of publications), and practice training, as indicated by proportionate clinical hours, in predicting the number of offers to interview. A hierarchic linear regression was conducted with the standardized score for proportionate clinical hours (b ⫽ 0.20, SE ⫽ 0.18) and standardized score for total number of publications (b ⫽ 0.19, SE ⫽ 0.15) entered in the first block, which was nonsignificant. The second block contained the interaction term (b ⫽ ⫺0.18, SE ⫽

Figure 1. Moderation of the effect of intervention and assessment hours on number of interview offers by number of supervision hours.

0.09) and was associated with a small, but statistically significant, increase in the predictability of number of offers to interview (1%), ⌬F (1, 530) ⫽ 3.77, p ⫽ .05. Figure 2 provides an illustration of this interaction.

Applicant Characteristics Associated With Matching As hypothesized, while controlling for number of applications submitted, the number of obtained interview offers yielded a significant correlation with successfully matching (rpb ⫽ .31, p ⬍.001), replicating earlier reports (Callahan et al., 2010). Taking a slightly different approach and correlating the proportion of interview offers to applications submitted with match status produced essentially the same result (rpb ⫽ .32, p ⬍.001). As with offers to interview, the following variables were significantly positively associated with matching after controlling for number of applications submitted: number of presentations (rpb ⫽ .14, p ⫽ .01), number of publications (rpb ⫽ .13, p ⫽ .01), number of publications that are peer reviewed (rpb ⫽ .10, p ⫽ .03), and number of supervision hours (rpb ⫽ .10, p ⫽ .04). Unlike with offers to interview, the total number of intervention and assessment hours was not significantly correlated with matching. Applicants who were entering the match a second time (or more) demonstrated a negative correlation with successfully matching (rpb ⫽ ⫺0.12, p ⫽ .01). The only other applicant characteristics found to be significantly correlated with match outcome (matched/not matched), still controlling for number of applications submitted, pertained to applicant personality. Although no trait level measures of personality were significantly correlated with successfully matching, several personality facet scales yielded small, though statistically significant, correlations. Measures of personality facets reflecting Artistic Interests (Openness trait; rpb ⫽ .11, p ⫽ 02), Self-Efficacy (Conscientious trait; rpb ⫽ .11, p ⫽ .02), and Friendliness (Extraversion trait; rpb ⫽ .10, p ⫽ .04) were all positively associated with matching. The facet measuring Assertiveness (Extraversion trait;


76


Figure 2. Interaction of proportionate clinical hours and number of publications on number of interview offers.

rpb ⫽ .09, p ⫽ .05) was on the cusp of statistical significance with a small effect.

Forecasting Match Outcomes First, a replication of the theoretically derived (see Collins et al., 2007) regression approach previously reported in the literature (Callahan et al., 2010) was accomplished. Hierarchical logistic regression was performed to examine the amount of variance in match outcomes that is associated with the number of offers to interview. Results (see Table 5) closely mirror the 2010 report in the literature. Second, another hierarchical logistical regression (see Table 6) was conducted to see if the regression approach could be improved in light of the univariate findings in this study with a larger sample size. In this analysis, the first block again entered control variables and included the following: marital status, responsibility for dependents, geographic restriction, and type of degree sought. Each of these variables had already demonstrated a significant univariate correlation with number of applications submitted. Collectively, they were associated with 0.05% of the variance in match outcome. The second block included personality facet scores that demonstrated univariate significance: Trust, Cooperation, Sympathy, Orderliness, Artistic Interests, Self-Efficacy, and Friendliness. The variance in match associated with these variables collectively was 7.35%. The third block included training variables within applicants’ control that had demonstrated significant univariate correlations. Entered into this block was the proportion of intervention and assessment hours to supervision hours, total number of publications,1 and number of presentations. Together these variables were associated with an additional 5.6% of the variance in match outcomes. The fourth block contained the number of applications submitted, and was not associated with any additional explained variance. The fifth block was associated with the most variance, 15.8%, and reflected the number of interview offers attained. As an approximation to ordinal least squares R2, the

Nagelkerke’s R2 strength of this association was associated with 28.8% of the overall variance in match outcome. Third, given the importance of interview offers, we calculated the probability of matching as a function of number of interviews in this sample. As shown in Figure 3, the probability of matching mirrors the actual match rate in this sample (88.6%) with six attained interview offers. Similarly, in applying Jacobson and Truax’s (1991) formula for calculating a cut score, we found that 6.92 interview invitations was the separation point demarking matched from unmatched applicants in this sample, with a reasonable balance between sensitivity (0.72) and specificity (0.71). For greater utility at the individual-student level, likelihood ratios were also computed. The positive likelihood ratio was found to be 2.45, and the negative likelihood ratio was 0.40. Stated another way, the likelihood that a student will match is nearly 2.5 times greater for those attaining seven or more interviews than those students who are offered fewer interviews. As shown in the histogram within Figure 3, most applicants in this sample attained slightly more offers to interview (M ⫽ 7.97, SD ⫽ 3.46). The probability of matching reached 99% (95% CI [93.3%, 99.8%]) when 12 offers to interview were secured.

Discussion The match rate of applicants in this sample (88.6%) was slightly higher than the match rate reported in the study by Callahan et al. (2010; 85.2%), but is generally consistent with previously reported match rates of CUDCP programs (Collins & Callahan, 2006). In our sample, 95.2% of applicants who matched did so with an APA or Canadian Psychological Association accredited internship site. Furthermore, about two thirds of applicants in our sample matched at their first- or second-ranked internship site. The strongest predictor of match success was the proportion of interview offers attained to applications submitted. Consistent with conceptual and empirical support in the broad literature regarding the association of personality to interview success, facets of applicant personality were correlated with the variables of interest (applying, interviewing, and matching) and were associated with significant variance in match outcomes among applicants in this study.

Comparison With APPIC Reports In comparison with results from APPIC’s 2010 (APPIC, 2010) and 2011 (APPIC, 2011) surveys of match participants, applicants in this study appear largely similar to the APPIC samples. However, some differences are notable, which likely reflect the broader range of applicants within the APPIC sample, relative to our sample of CUDCP students. The match rate in our sample of CUDCP trainees (88.6%) was higher than in the APPIC samples (80%), as was the number of interviews offered (approximately eight for both years), despite the number of applications submitted approximating those in the APPIC reports. As expected, compared with APPIC samples, many more applicants in our sample hailed from PhD programs. Relatedly, students in the APPIC samples 1 Although the number of peer-reviewed publications was also significantly correlated with obtaining offers to interview, this variable was omitted from the regression because it could create multicollinearity with the entered variable on total number of publications.


77

Table 5 Hierarchical Logistic Regression Analysis Predicting Match Outcome (No Match/Match) Step

␤

Wald ␹2

Odds ratio

95% CI (LL, UL)

0.60 ⫺18.12 0.03

0.46 0.00 0.53

1.83 0.00 1.03

0.32, 10.50 0.00, 0.00 0.95, 1.12

Variable

1

0.03 Citizenship (reference: U.S. citizen) Type of Degree (reference: PhD) Size of applicant’s class


2

0.09 Status of dissertation proposal (reference: completed prior to applying) Number of peer-reviewed publications Number of intervention/assessment hours Number of supervision hours Number of integrated reports

⫺0.41 0.06 0.00 0.00 0.00

2.91 0.87 0.55 0.70 0.07

0.67 1.06 1.00 1.00 1.00

0.42, 1.06 0.94, 1.20 0.10, 1.00 0.10, 1.00 0.10, 1.00

Number of applications

⫺0.08

2.92

0.92

0.83, 1.01

0.39

28.98

1.48

1.28, 1.70

3

0.09

4

0.25 Number of interview offers

Note.

Nagelkerke R2

CI ⫽ confidence interval; LL ⫽ lower limit; UL ⫽ upper limit.

reported larger doctoral cohorts than the current samples. Similar to previous findings (Callahan et al., 2010), the median number of clinical hours in the APPIC samples was lower than our sample. Number of assessment and intervention hours, as well as supervision hours, were also lower in the APPIC samples. With respect to research benchmarks, students in our sample published more articles in refereed journals, and a greater percentage proposed their dissertations prior to submitting applications.

Summary and Contextualization of Findings Although Callahan and colleagues (2010) focused on exploration of a large number of applicant and program characteristics,

raising the risk of alpha inflation and spurious findings, the findings here are largely consistent with their report. Although much variance in match outcome remains unexplained, the number of interview offers was the strongest predictor of matching and was associated with the most variance in match outcome (15.8%). Analyses also demonstrated that a number of other training variables, including presentations, total number of publications, number of publications that are peer reviewed, and the total number of supervision hours, were associated with matching. The addition of supervision hours, and corresponding omission of intervention and assessment hours, to the significant correlates of matching is particularly notable and merits following over time. However, unlike Callahan et al., neither completing the dissertation proposal

Table 6 Hierarchical Logistic Regression Analysis Predicting Match Outcome (No Match/Match) ␤

Wald ␹2

Odds ratio

95% CI (LL, UL)

0.18 0.00 ⫺0.24 ⫺18.17

0.19 0.00 0.31 0.04

1.20 1.00 0.79 0.00

0.53, 2.73 0.34, 2.99 0.34, 1.82 0.00, 0.00

Trust Cooperation Sympathy Orderliness Artistic interests Self-efficacy Friendliness

⫺0.09 0.02 ⫺0.00 0.02 0.08 0.06 0.05

3.84 0.22 0.00 0.43 3.19 0.97 1.51

0.91 1.02 0.10 1.02 1.08 1.06 1.05

0.94, 1.12 0.91, 1.10 0.97, 1.01 0.99, 1.18 0.95, 1.19 0.97, 1.13 0.72, 1.27

Interview/assessment hours: supervision hoursb Total number of publications Number of presentations

⫺0.05 0.05 0.04

0.10 0.86 1.85

0.95 1.05 1.04

0.94, 1.18 0.98, 1.10 0.81, 1.01

Number of applications submitted

⫺0.10

3.19

0.91

0.81, 1.01

0.39

25.48

1.48

1.27, 1.72

Step

Variable

Nagelkerke R2a

1

0.01 Marital status (reference: married/partnered) Responsibility for dependents (reference: no) Geographic restriction (reference: no) Type of degree (reference: PhD)

2

3

4

0.07

0.13

0.13

5

0.29 Number of interview offers

Note. CI ⫽ confidence interval; LL ⫽ lower limit; UL ⫽ upper limit. Rounded to two decimal places, which results in slight departures from the more precise numbers presented in the narrative. calculated by dividing the number of intervention and assessment hours by the number of supervision hours for each individual.

a

b

This proportion was



78

Figure 3. Straight, dotted line demarks the match rate within this sample. Remaining lines plot the probability of matching as a function of number of interviews in this sample (dashed lines indicate 95% confidence interval). Histogram displays the frequency at which the number of offers to interview was attained in this sample.

nor the total number of intervention and assessment hours was associated with matching. As previously noted, the current sample was comprised entirely of students from programs that are members of CUDCP, and this commonality among applicants may be salient to contextualizing the current findings. In light of the imbalance problem, CUDCP has engaged in extensive discussions over the past few years that may have influenced the mentoring provided to students in their doctoral programs, how their applications were evaluated by internship sites, or both (see CUDCP, 2011, for a summary of the organization’s expectancies regarding the dissertation proposal and the relative importance of intervention and assessment hours to supervision hours). Alternatively, the lack of support for the importance of the dissertation proposal may simply reflect limited variance, as the preponderance of applicants in this study had successfully proposed their dissertation prior to applying for internship. It is unclear whether the reduced variance over time is due to shifts in mentoring, pressure applicants are self-imposing as they try to enhance their competitiveness, or some other as yet unidentified variable(s).

Personality and the Match As discussed, applicant personality was explored for possible relationships to match outcome and obtaining interview invitations, with the suggestion that the idea of “fit” may include applicant personality traits. None of the Big Five personality traits demonstrated significant utility in this study. Because we sought to maximize the amount of information we were able to provide about applicant characteristics salient to internship, we chose to then examine personality at the facet scale level. A few measures of personality facets did appear to be weakly associated with the process of applying for internship among CUDCP students (Cooperation, Sympathy, Orderliness), obtaining offers to interview (Trust, Orderliness), or matching to a site (Friendliness, Assertiveness, Self-Efficacy). After considering the other variables in predicting match outcome, these personality characteristics were associated with a small amount of

variance in match outcome (7.35%). This finding, though very small, supports existing indications that internship sites consider personality features among their inclusion and exclusion criteria (Collins et al., 2007; Ginkel et al., 2010; Rodolfa et al., 1999). Future investigation regarding personality and the internship match process should consider the possibility that it may be difficult to detect significant variability among the personalities of applicants, without ruling out the possibility that personality plays a at least a small role in the internship match process. It also may be important to study personality in a larger sample; more variability may be found outside of a CUDCP sample. Although we acknowledge that not all graduate students have the same personality traits, we also suspect there may be certain traits commonly found among students who enter our field, and that these traits may be different than personalities of other groups of people. Furthermore, it should be considered that the decision of applicants to participate in the survey may be correlated with certain personality traits, further increasing any homogeneity within the sample. A recent study (Rieck & Callahan, 2013) reported, for instance, that graduate students enrolled in clinical practicum at a departmental training clinic had higher neuroticism, openness, extraversion, and agreeableness than normative samples developed during standardization of the NEO Five-Factor Inventory. The graduate students were also found to have lower conscientiousness scores than found in the normative sample. Thus, moving forward in this area of research may require consideration of the differences in personality (if any) between graduate students and other groups of individuals. We used the IPIP scales for their public domain convenience and with the hope that psychology students would be less likely to be familiar at the item level of the measure (which may have been true with other more commonly taught personality measures such as the NEO-PI-R). However, it remains possible that students, like other participants, may have been inclined to try and manage their impression, particularly in terms of personality. The use of additional measures of personality may be warranted in future research.



Additionally, there are likely other factors associated with successful interviewing that may direct future research regarding applicant characteristics associated with matching. For instance, the employment literature discusses the impact of personal appearance and communication skills (Bretz, Rynes, & Gerhart, 1993), nonverbal cues (Gifford, Ng, & Wilkinson, 1985), and emotional intelligence (Fox & Spector, 2000), to name a few. Furthermore, the impact of personality on match status is in part dependent upon an interviewer’s perception of certain personality traits in the applicant, perhaps during an interview. Self-perception skills and adaptive impression management may play a considerable role in an interview situation.

Limitations The generalizability of these findings to non-CUDCP samples is unknown. We strongly encourage the National Council of Schools and Programs of Professional Psychology and the Council of Counseling Psychology Training Programs to conduct similar studies within their organizations, as these each represent other large suppliers of applicants to the internship match. Additionally, because the present study examined a group of students collectively more likely to match than other students, a more representative sample of internship applicants may yield other important findings that distinguish match outcomes. As with previous research (Callahan et al., 2010), this study sought to maximize the amount of information provided about applicant variables salient to internship. Thus, no alpha corrections were made, which may have increased the possibility of Type I errors. We chose to use this more liberal approach to the data, though, to avoid overlooking possibly meaningful findings (Type II errors) informing the critical issue of internship match. Despite this approach, few significant findings were observed. In light of the small effect sizes, it is likely a more conservative approach to analyses would have generated even fewer significant findings. An absence of significant association between prominent AAPI variables and match outcomes would be highly concerning, though it might prompt fruitful discussion on whether outcomes more closely reflect a lottery system than a match system. In light of the potential alpha inflation herein, we encourage researchers who wish to replicate our findings to focus on effect sizes when comparing findings. Despite the large number of variables examined here, there remain some potentially important factors that were excluded. For instance, in order to protect the privacy of the students participating in our study, we did not collect data about specific programs and were therefore unable to examine the impact of a specific program’s reputation on training directors’ decisions. However, although we are aware from discussion with colleagues that reputational strength may influence some evaluators as they review internship applications, recent evidence is to the contrary. More specifically, among clinical psychology doctoral programs, training outcomes, including match rates, are not significantly correlated with reputational strength (Callahan, Ruggero, & Parent, in press). Additionally, structural equation modeling (SEM) is likely a more optimal approach to future analysis of internship match data. Number of interview offers appears to be the strongest predictor of match outcome, and it is likely that there are a multitude of variables, or latent constructs, that predict whether an applicant will obtain more offers to interview. SEM lends itself well to

79

elucidating the complex potential relationships among some of these variables and receiving interview offers. However, given the large number of applicant and program variables used in our analyses, it was determined that a much larger sample size would be necessary in order to successfully use SEM. As previously noted (Callahan et al., 2010), APPIC’s data would provide a significantly larger sample size in which SEM would be beneficial. All internship applicants are included in APPIC’s data set, and this would eliminate the need for any use of sample-dependent methods for increasing sample size, such as bootstrapping. Furthermore, the APPIC data includes variables not acquired from our CUDCP samples that may help further clarify the construct of “fit” between applicants and internship sites. Specifically, the APPIC data provides information regarding types of sites to which students applied, as well as how internship sites ranked students with certain characteristics. This information could then be analyzed to determine if there are characteristic differences among students faring better (or worse) at some sites than others. Although this study provides advancement in our empirical study of the match process, it is not enough to simply consider what factors contribute to a successful match overall. Instead, information is also needed about what types of applicants and internship sites are a best match for one another. With respect to training characteristics, a competent, well-qualified applicant with a training focus in one area may not be preferred by an internship placement with a focus in a different area— degree of fit between an applicant’s training and type or setting of the internship has been suggested as one of the most important factors considered by internship sites (Ginkel et al., 2010; Neimeyer et al., 2007; Rodolfa et al., 1999). This study was not able to fully address this concept. Although many applicant characteristics were examined, examination of internship sites’ perspectives was not possible. As previously discussed, it may be that an applicant’s educational achievements qualify him or her for a certain type of internship site, but not for others. Important data for future analyses would include the sites at which an applicant did not match, as well as sites at which the applicant was ranked. If an applicant does not apply to sites for which she or he is a good fit, then matching may become less likely. More research focused toward investigating what qualities are desired in applicants is needed, as well as differences in applicants who match and do not match with various types of internship sites. Approaching the idea of “fit” in this manner may yield additional important findings.

Suggestions for Mentoring Ultimately, we hope that this work will be of some assistance to applicants and their mentors as they prepare for the internship match experience. With few significant associations across multiple samples relative to a large number of applicant characteristics, one can feel frustrated with the potential implication that applicant preparation, competencies, and achievements, as reflected by the quantifiable AAPI variables, might not be strongly associated with match outcomes. Rather than the match process involving excessive randomness or error variance, it remains our hope that perhaps more research will further demystify match outcome phenomena. Nevertheless, there do appear to be some points of clarity within the data that may inform the mentoring of trainees coming from scientist-practitioner training model programs.


80


Below we offer some suggestions for advising students as they prepare to apply for internship. However, given that our sample consisted only of students in CUDCP programs, we advise caution in applying the following suggestions to internship applicants more generally. More hours is not necessarily better. Trainees with comparatively few intervention and assessment hours typically still attain more than sufficient offers to interview when their hours are closely supervised. Throughout training, trainees and mentors should monitor the accrual of hours, with the goal of attaining a proportionate blend of intervention and assessment hours with supervision hours (see Figure 1). The short story here is that more is not better; better is better. Publications matter. Although the vast majority (⬎90%) of applicants to internship have yet to complete their dissertation project, the typical applicant already has several publications in press or in print (M ⫽ 3.52, SD ⫽ 3.63). The number of attained interviews may function as a canary in the coal mine. Directors of clinical training may identify students’ high risk for not matching by inquiring how many offers to interview they have attained. Specifically, six or fewer attained offers to interview may signal the trainee is at heightened risk to not match (see Figure 3). Not matching in an earlier cycle heightens risk of not matching again. Unfortunately, the reasons for this are unclear. The time between not matching (late March for Phase II) and beginning to search for sites in the next cycle (mid-June is when site information is updated annually) is actually only a matter of weeks apart. This leaves very little time for applicants to strengthen their portfolio. One way of lengthening the time applicants have to improve their portfolio is to review, with applicants identified after uniform notification day in mid-December as high risk for going unmatched (based on insufficient offers to interview), ways that they may strengthen their portfolio over the next 6 to 9 months. This allows trainees to take thoughtful action during the spring semester with respect to improving important variables (i.e., proportionate clinical hours, number of publications) over which they have control. Submitting more applications is a poor strategy for the match. This finding has been reported by APPIC for several years and was also reported by Callahan and colleagues (2010). Findings here again underscore the point. If a trainee is identified as high risk to not match, they should not be encouraged to simply submit more applications to late application deadline sites. Such a strategy may result in the trainee incurring additional costs (depending upon how many they have already submitted, APPIC may charge an additional fee for these applications) and is not likely to improve their likelihood of matching. Instead, we suggest that these trainees first consider, preemptively, whether they intend to participate in the Phase II Match process in the event of not matching in Phase I. In deciding to participate in Phase II, we strongly encourage applicants to carefully consider what improvements are possible in the short term, from mid-December to mid-March, in order to increase the chances of matching in Phase II. Considerations may include improving interview skills, reevaluating one’s fit with internship sites, addressing weak letters of reference, reformatting the CV, composing more tailored cover letters regarding the specific fit with a site, making progress on research milestones, or improving practicum experiences, to name just a few possibilities. In light of how little we know about Phase II Match, empirical research that may inform how mentors may

best assist applicants entering the Phase II Match would be extremely valuable. After applications are in, variables related to personality may be among the salient and potentially modifiable factors remaining. In particular, applicants who present as friendly, assertive, and having self-efficacy fare better in match outcomes. Working with high-risk-to-not-match applicants on their interpersonal presentation with respect to these variables prior to going on interviews is not likely to be harmful and may be helpful. Although those with artistic interests also appear to benefit in the match, we do not recommend applicants suddenly commit themselves to art classes, music lessons, or dance troupes. Rather, we encourage applicants to consider that while they are being interviewed, one consideration potentially being made is whether the applicant seems like they would be interesting and enjoyable to work with throughout the coming year. Artistic interests represent only one pathway of that determination. We suggest that applicants be encouraged to “be themselves” and let people get to know them a bit during interviews rather than simply present as a walking reflection of the facts found on their vitae or within the AAPI. A large number of attained interviews does improve likelihood of matching. Having seven or more interviews portends that the applicant is 2.5 times more likely to match than not match. The probability of matching reaches 99% (95% CI [93.3%, 99.8%]) when 12 offers to interview are secured. This does not mean that the applicant must attend all interviews offered and incur the associated costs (both in financial resources and time consumed). Applicants with a very high volume of interviews may benefit from mentoring that helps them sort out which interview offers to accept. At the same time, it should be noted that, anecdotally, we know that many training directors have mentored highly competent, accomplished students who obtained many interviews but were not matched. More research is needed to help elucidate whether personality, interviewing skills, or both (or some other factors) are playing a role in these situations.

Future Directions In light of findings evidencing little information about what contributes to a successful match, we hope that continued research can offer students entering the match more empirical direction about how best to secure an internship. Specifically, future studies that inform whether the variables associated with matching vary systematically among the different types of internship settings are needed. Additionally, research examining the relationship between applicant’s clinical competencies, as captured on a valid and reliable measure, and AAPI variables is strongly needed. APPIC should consider developing a new AAPI that focuses on competencies and draws from test construction approaches to measure development (the existing approach is largely unchanged since inception). It appears that the lines of research and discussion focusing on match imbalance and applicant and program characteristics have been treated as somewhat separate areas of investigation. However, we argue that findings about applicant characteristics aimed at helping students successfully match can also inform policy decisions and discussion among multiple governing bodies attempting to remedy the internship supply and demand imbalance. In recognizing that both the imbalance problem itself and the factors that contribute to a successful match are much more com-



plex than they appear at first glance, we begin to reframe the discussions about the match imbalance in a way that addresses multiple perspectives and adequately considers all components of a complex issue. Finally, this study, though it examined a limited group of programs, provides an important precedent for studies and/or commentaries regarding the Boulder model of training. In searching the existing literature, we found only 80 peer reviewed journal entries with “Boulder model” indexed in PsycINFO, despite the model dating back more than a half century (Raimy, 1950). In restricting our search to only empirical studies, we found a mere 18 publications, and none of them specifically tested whether an interaction effect was observable between science and practice on any outcome. Rather, these studies generally took the approach of documenting interests (Parker & Detterman, 1988; Zachar & Leong, 2000), training preferences (Norcross, Gallagher, & Prochaska, 1989; Tyler & Clark, 1987), time spent in activities (Himelein & Putnam, 2001), relative emphases (Merlo, Collins, & Bernstein, 2008), differences between Boulder and non-Boulder graduates and trainees (Barrom, Shadish, & Montgomery, 1988; Rodriguez-Menendez, 2000; Templer, Stroup, Mancuso, & Tangen, 2008) or enthusiasm for the model (Norcross, Karpiak, & Santoro, 2005; O’Sullivan & Quevillon, 1992). This study provides an example for future research of how a training philosophy, in this case the Boulder model, can be subjected to empirical examination. Based on the theoretical Boulder model, our expectation was that science training would interact with practice training in readying trainees for the profession. To operationalize this into a testable hypothesis, we selected publications as our proxy measure of science achievements, proportionate clinical hours as our indicator of practice achievements, and number of offers to interview as a proxy indicator of readiness to enter the field. The results supported our hypothesis, with no indication that the model was statistically misspecified. But whether our operationalization of the Boulder model is valid is perhaps still a matter of opinion. Our proxy measures seem to have some face validity: sound training in science is likely beneficial to authoring publications; practicum hours are likely helpful in acquiring clinical competencies. The finding that they interact significantly in a predictive model for an emerging professional benchmark appears supportive of the Boulder model philosophy. However, future researchers might consider different proxy measures and test the Boulder model further. Based on our findings, though, the significant interaction here between science and practice variables on perceived readiness for internship, which is a clinical year, suggests that the rationale for requiring simultaneous training on conducting research with becoming a clinician is not spurious (see Frank, 1984, for an excellent review of pivotal criticisms of the Boulder model to this effect). The findings here, as displayed in Figures 1 and 2, suggest that research training is not coming at the expense of clinical training. Rather, the findings suggest that training in research is a key element that interacts significantly with the accumulation of clinical hours under close supervision to adequately preparing one to emerge from the doctoral program into the internship year. Although the findings appear supportive of the Boulder model, we do not imply that the findings indicate anything negative about any other training model. There may well be multiple viable training paths. Additional movement toward empirical investigation of training philosophies is a strongly needed direction for the future.

81 References

American Psychological Association. (2010). Ethical principles of psychologists and code of conduct, including 2010 amendments. Retrieved from http://www.apa.org/ethics/code/index.aspx Appleby, D., Keenan, J., & Mauer, B. (1999). Applicant characteristics valued by graduate programs in psychology. Eye on Psi Chi, 3, 39. Association of Psychology Postdoctoral and Internship Centers. (2010, August). 2010 APPIC Match: Survey of internship applicants: Summary of survey results. Retrieved from http://www.appic.org/Match/ MatchStatistics/ApplicantSurvey2010.aspx Association of Psychology Postdoctoral and Internship Centers. (2011, May). 2011 APPIC Match: Survey of internship applicants: Summary of survey results. Retrieved from http://www.appic.org/Match/ MatchStatistics/ApplicantSurvey2011Part1.aspx Association of Psychology Postdoctoral and Internship Centers. (2013, February). 2013 APPIC Match statistics – Phase I: Match report from the APPIC Board of Directors. Retrieved from http://www.appic.org/ Match/MatchStatistics/MatchStatistics2013PhaseI.aspx Baker, J., McCutcheon, S., & Keilin, W. G. (2007). The internship supplydemand imbalance: The APPIC perspective. Training and Education in Professional Psychology, 1, 287–293. doi:10.1037/1931-3918.1.4.287 Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26. doi:10.1111/j.1744-6570.1991.tb0068.x Barrom, C. P., Shadish, W. R., & Montgomery, L. M. (1988). PhDs, PsyDs, and real-world constraints on scholarly activity: Another look at the Boulder model. Professional Psychology: Research and Practice, 19, 93–101. doi:10.1037/0735-7028.19.1.93 Belar, C. D., & Perry, N. W. (1992). National conference on scientistpractitioner education and training for the professional practice of psychology. American Psychologist, 47, 71–75. doi:10.1037/0003-066X.47.1.71 Bieschke, K. J., Bell, D., Davis, C., III, Hatcher, R., Peterson, R., & Rodolfa, E. R. (2011). Forests, grazing areas, water supplies, and the internship imbalance problem: Redefining the paradigm to implement effective change. Training and Education in Professional Psychology, 5, 123–125. doi:10.1037/a0024902 Bretz, R. D., Rynes, S. L., & Gerhart, B. (1993). Recruiter perceptions of applicant fit: Implications for individual career preparation and job search behavior. Journal of Vocational Behavior, 43, 310 –327. doi: 10.1006/jvbe.1993.1050 Caldwell, D. F., & Burger, J. M. (1998). Personality characteristics of job applicants and success in screening interviews. Personnel Psychology, 51, 119 –136. doi:10.1111/j.1744-6570.1998.tb00718.x Callahan, J. L., Collins, F. L., Jr., & Klonoff, E. A. (2010). An examination of applicant characteristics of successfully matched interns: Is the glass half full or half empty and leaking miserably? Journal of Clinical Psychology, 66, 1–16. doi:10.1002/jclp.20664 Callahan, J. L., Ruggero, C. J., & Parent, M. C. (in press). Hidden gems among clinical psychology training programs. Training and Education in Professional Psychology. Collins, F. L., Jr., & Callahan, J. L. (2006). CUDCP mini-survey: Internship match outcomes. Unpublished report, Council of University Directors of Clinical Psychology (Available from Jennifer Callahan, UNT Department of Psychology, 1155 Union Circle #311280, Denton TX 76203–5017). Collins, F. L., Jr., Callahan, J. L., & Klonoff, E. A. (2007). A scientistpractitioner perspective of the internship match imbalance: The stairway to competence. Training and Education in Professional Psychology, 1, 267–275. doi:10.1037/1931-3918.1.4.267 Commission on Accreditation. (2009). Guidelines and Principles for Accreditation of Programs in Professional Psychology. Washington, D.C.: American Psychological Association.


82


Costa, P. T., Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO–PI–R) and NEO Five-Factor Inventory (NEO–FFI) professional manual. Odessa, FL: Psychological Assessment Resources. Council of University Directors of Clinical Psychology. (2011). Council of University Directors of Clinical Psychology expectations for internship eligibility. Retrieved from http://cudcp.us/files/Reports/ CUDCP%20Internship%20Expectations%20(2011).pdf DeMers, S. T. (2011). An ASPPB perspective on Hatcher’s “The internship supply as a common-pool resource: A pathway to managing the imbalance problem”. Training and Education in Professional Psychology, 5, 141–143. doi:10.1037/a0024901 Fox, S., & Spector, P. E. (2000). Relations of emotional intelligence, practical intelligence, general intelligence and trait affectivity with interview outcomes: It’s not all just ’G’. Journal of Organizational Behavior, 21, 203–220. doi:10.1002/(SICI)1099-1379(200003)21:2⬍203:: AID-JOB38⬎3.0.CO;2-Z Frank, G. (1984). The Boulder model: History, rationale, and critique. Professional Psychology: Research and Practice, 15, 417– 435. doi: 10.1037/0735-7028.15.3.417 Gifford, R., Ng, C. F., & Wilkinson, M. (1985). Nonverbal cues in the employment interview: Links between applicant qualities and interviewer judgments. Journal of Applied Psychology, 70, 729 –736. doi: 10.1037/0021-9010.70.4.729 Ginkel, R. W., Davis, S. E., & Michael, P. G. (2010). An examination of inclusion and exclusion criteria in the predoctoral internship selection process. Training and Education in Professional Psychology, 4, 213– 218. doi:10.1037/a0019453 Hatcher, R. L. (2011). The internship supply as a common-pool resource: A pathway to managing the imbalance problem. Training and Education in Professional Psychology, 5, 126 –140. doi:10.1037/a0024658 Himelein, M. J., & Putnam, E. A. (2001). Work activities of academic clinical psychologists: Do they practice what they teach? Professional Psychology: Research and Practice, 32, 537–542. doi:10.1037/0735-7028.32.5.537 Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19. doi:10.1037/ 0022-006X.59.1.12 Keilin, W. G., Thorn, B. E., Rodolfa, E. R., Constantine, M. G., & Kaslow, N. J. (2000). Examining the balance of internship supply and demand: 1999 association of psychology postdoctoral and internship centers’ match implications. Professional Psychology: Research and Practice, 31, 288 –294. doi:10.1037/0735-7028.31.3.288 Keilin, W. G., Baker, J., McCutcheon, S., & Peranson, E. (2007). A growing bottleneck: The internship supply-demand imbalance in 2007 and its impact on psychology training. Training and Education in Professional Psychology, 1, 229 –237. doi:10.1037/1931-3918.1.4.229 Lamb, D. H., Baker, J. M., Jennings, M. L., & Yarvis, E. (1982). Passages of an internship in professional psychology. Professional Psychology, 13, 661– 669. doi:10.1037/0735-7028.13.5.661 Landrum, R. E., Jeglum, E. B., & Cashin, J. R. (1994). The decisionmaking processes of graduate admissions committees in psychology. Journal of Social Behavior & Personality, 9, 239 –248. Madson, M. B., Hasan, N. T., Williams-Nickelson, C. W., Kettman, J. J., & Sands Van Sickle, K. (2007). The internship supply and demand issue: Graduate student’s perspective. Training and Education in Professional Psychology, 1, 249 –257. doi:10.1037/1931-3918.1.4.249 McCutcheon, S. R. (2011). The internship crisis: An uncommon urgency to build a common solution. Training and Education in Professional Psychology, 5, 144 –148. doi:10.1037/a0024896 Merlo, L. J., Collins, A., & Bernstein, J. (2008). CUDCP-affiliated clinical psychology student views of their science training. Training and Education in Professional Psychology, 2, 58 – 65. doi:10.1037/1931-3918.2.1.58

Neimeyer, G. J., Rice, K. G., & Keilin, W. G. (2007). Does the model matter? The relationship between science-practice emphasis in clinical psychology programs and the internship match. Training and Education in Professional Psychology, 1, 153–162. doi:10.1037/1931-3918.1.3.153 Norcross, J. C., Gallagher, K. M., & Prochaska, J. O. (1989). The Boulder and/or the Vail model: Training preferences of clinical psychologists. Journal of Clinical Psychology, 45, 822– 828. doi:10.1002/10974679(198909)45:5⬍822::AID-JCLP2270450521⬎3.0.CO;2-E Norcross, J. C., Karpiak, C. P., & Santoro, S. O. (2005). Clinical psychologists across the years: The division of clinical psychology from 1960 to 2003. Journal of Clinical Psychology, 61, 1467–1483. doi:10.1002/jclp.20135 Norcross, J. C., Kohout, J. L., & Wicherski, M. (2005). Graduate study in psychology: 1971–2004. American Psychologist, 60, 959 –975. doi: 10.1037/0003-066X.60.9.959 Oehlert, M. E., & Lopez, S. J. (1998). APA-accredited internships: An examination of the supply and demand issue. Professional Psychology: Research and Practice, 29, 189 –194. doi:10.1037/0735-7028.29.2.189 O’Sullivan, J. J., & Quevillon, R. P. (1992). 40 years later: Is the Boulder model still alive? American Psychologist, 47, 67–70. doi:10.1037/0003066X.47.1.67 Parker, L. E., & Detterman, D. K. (1988). The balance between clinical and research interests among Boulder model graduate students. Professional Psychology: Research and Practice, 19, 342–344. doi:10.1037/07357028.19.3.342 Raimy, V. C. (1950). Training in clinical psychology. New York, NY: Prentice Hall. Rieck, T., & Callahan, J. L. (2013). Emotional intelligence and psychotherapy outcomes in the training clinic. Training and Education in Professional Psychology, 7, 42–52. doi:10.1037/a0031659 Rodolfa, E. R., Bell, D. J., Bieschke, K. J., Davis, C., III, & Peterson, R. L. (2007). The internship match: Understanding the problem-seeking solutions. Training and Education in Professional Psychology, 1, 225–228. doi:10.1037/1931-3918.1.4.225 Rodolfa, E. R., Vielle, R., Russell, P., Nijjer, S., Nguyen, D. Q., Mendoza, M., & Perrin, L. (1999). Internship selection: Inclusion and exclusion criteria. Professional Psychology: Research and Practice, 30, 415– 419. doi:10.1037/0735-7028.30.4.415 Rodriguez-Menendez, G. (2000). In response to the 1998 APA Division 40 presidential address: So where’s the beef? The Clinical Neuropsychologist, 14, 202–211. doi:10.1076/1385-4046(200005)14:2;1-Z;FT202 Salgado, J. F. (1997). The five-factor model of personality and job performance in the European Community. Journal of Applied Psychology, 82, 30 – 43. doi:10.1037/0021-9010.82.1.30 Templer, D. I., Stroup, K., Mancuso, L. J., & Tangen, K. (2008). Comparative decline of professional school graduates’ performances on the examination for professional practice in psychology. Psychological Reports, 102, 551–560. doi:10.2466/pr0.102.2.551-560 Thorn, B. E., & Dixon, K. E. (1999). Issues of internship supply and demand: A survey of academic counseling and clinical programs. Professional Psychology: Research and Practice, 30, 198 –202. doi: 10.1037/0735-7028.30.2.198 Tyler, J. D., & Clark, J. A. (1987). Clinical psychologists reflect on the usefulness of various components of graduate training. Professional Psychology: Research and Practice, 18, 381–384. doi:10.1037/0735-7028.18.4.381 Zachar, P., & Leong, F. T. L. (2000). A 10-year longitudinal study of scientists and practitioner interests in psychology: Assessing the Boulder model. Professional Psychology: Research and Practice, 31, 575–580. doi:10.1037/0735-7028.31.5.575

Received May 29, 2013 Revision received September 9, 2013 Accepted October 16, 2013 䡲