Topics in Training

e60(1) C OPYRIGHT Ó 2018

BY

T HE J OURNAL

OF

B ONE

AND J OINT

S URGERY, I NCORPORATED

Topics in Training Evaluation of the Content and Accessibility of Web Sites for Accredited Orthopaedic Trauma Surgery Fellowships M. Kareem Shaath, MD, Michael G. Yeranosian, MD, Joseph A. Ippolito, MD, Mark R. Adams, MD, Michael S. Sirkin, MD, and Mark C. Reilly, MD Investigation performed at the Rutgers New Jersey Medical School, Newark, New Jersey

Background: Orthopaedic trauma fellowship applicants use online-based resources when researching information on potential U.S. fellowship programs. The 2 primary sources for identifying programs are the Orthopaedic Trauma Association (OTA) database and the San Francisco Match (SF Match) database. Previous studies in other orthopaedic subspecialty areas have demonstrated considerable discrepancies among fellowship programs. The purpose of this study was to analyze content and availability of information on orthopaedic trauma surgery fellowship web sites. Methods: The online databases of the OTA and SF Match were reviewed to determine the availability of embedded program links or external links for the included programs. Thereafter, a Google search was performed for each program individually by typing the program’s name, followed by the term “orthopaedic trauma fellowship.” All identified fellowship web sites were analyzed for accessibility and content. Web sites were evaluated for comprehensiveness in mentioning key components of the orthopaedic trauma surgery curriculum. By consensus, we refined the final list of variables utilizing the methodology of previous studies on the topic. Results: We identified 54 OTA-accredited fellowship programs, offering 87 positions. The majority (94%) of programs had web sites accessible through a Google search. Of the 51 web sites found, all (100%) described their program. Most commonly, hospital affiliation (88%), operative experiences (76%), and rotation overview (65%) were listed, and, least commonly, interview dates (6%), selection criteria (16%), on-call requirements (20%), and fellow evaluation criteria (20%) were listed. Programs with ‡2 fellows provided more information with regard to education content (p = 0.0001) and recruitment content (p = 0.013). Programs with Accreditation Council for Graduate Medical Education (ACGME) accreditation status also provided greater information with regard to education content (odds ratio, 4.0; p = 0.0001). Otherwise, no differences were seen by region, residency affiliation, medical school affiliation, or hospital affiliation. Conclusions: The SF Match and OTA databases provide few direct links to fellowship web sites. Individual program web sites do not effectively and completely convey information about the programs. The Internet is an underused resource for fellow recruitment. The lack of information on these sites allows for future opportunity to optimize this resource.

Disclosure: There was no source of external funding for this study. On the Disclosure of Potential Conflicts of Interest forms, which are provided with the online version of the article, one or more of the authors checked “yes” to indicate that the author had a relevant financial relationship in the biomedical arena outside the submitted work (http://links.lww.com/JBJS/E720).

J Bone Joint Surg Am. 2018;100:e60(1-5)

d

http://dx.doi.org/10.2106/JBJS.17.01112

e60(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 100 -A N U M B E R 9 M AY 2, 2 018 d

d

d

The Orthopaedic Trauma fellowship match was established in 2008, when the Orthopaedic Trauma Association (OTA) partnered with the San Francisco Match (SF Match) to coordinate fellowship appointments. In addition to overseeing the match, the OTA maintains a program list and provides information to prospective fellow applicants. Along with the OTA, the SF Match contains a database of participating programs. These 2 databases are often the first sources used. One of the purposes of this study was to assess the accessibility and comprehensiveness of information from these resources. Previous evaluations of online content in other subspecialties of orthopaedic surgery have shown underutilization of these resources for education and recruitment1-3. It is unknown how orthopaedic trauma fellowships use the Internet as a tool to disseminate information and recruit potential fellowship applicants. The purpose of this study was to determine the availability of information on orthopaedic trauma surgery fellowships obtainable through the OTA and SF Match databases. We also analyzed and determined the accessibility of program information from national databases and evaluated the education and recruitment content of orthopaedic trauma fellowship web sites. We hypothesized that orthopaedic trauma fellowship web sites would contain a dearth of information for prospective applicants. Materials and Methods Database Listings The OTA and SF Match maintain a complete list of current accredited orthopaedic trauma fellowships at http://www.ota. org and http://www.sfmatch.org. The lists of orthopaedic trauma surgery programs on the web sites of the OTA and SF Match were accessed on January 24, 2017. The number of individual offered programs and the numbers of fellowship positions were extracted. Web Site Accessibility All web sites were assessed independently. The criteria for web site content were determined by 2 of the authors (a current orthopaedic trauma fellowship applicant and an orthopaedic trauma fellowship director) and were based on previous study in the field1-3. A Google search was conducted by querying the program name (as a phrase) plus “Orthopaedic Trauma Fellowship.” This was done to determine the accessibility of program web sites from outside the OTA and SF Match databases. The Google searches were performed on January 25, 2017. The accessible program web sites were analyzed for content with regard to fellow education and recruitment (Table I). Fellow Education and Recruitment As previously described3, each web site was assessed for the presence of components of fellowship education, including rotation overview, didactic instructions, journal club, academic conferences, research requirements, research interests, meetings attended, operative experiences, office time, on-call requirements, and evaluation criteria. Also, web sites were evaluated for the presence of information with regard to recruitment, including hospital affiliation, links to the OTA,

C O N T E N T A N D AC C E S S I B I L I T Y O F W E B S I T E S F O R AC C R E D I T E D O RT H O PA E D I C T R AU M A S U R G E RY F E L LOW S H I P S

TABLE I Information Provided by Orthopaedic Trauma Fellowship Web Sites Information

No. of Programs*

Fellowship education Operative experience

39 (76%)

Rotation overview

33 (65%)

Didactic instruction Research requirement

27 (53%) 25 (49%)

Journal club

24 (47%)

Research interests

23 (45%)

Meetings attended

20 (39%)

Office time

17 (33%)

Academic conferences

15 (29%)

On-call requirements

10 (20%)

Fellow evaluation criteria

10 (20%)

Fellowship recruitment Program description

51 (100%)

Affiliated hospitals

45 (88%)

Association links

30 (59%)

Eligibility criteria

29 (57%)

Application links

21 (41%)

Fellow listing

18 (35%)

Graduate information

16 (31%)

Faculty listing Salary

15 (29%) 13 (25%)

Selection criteria

8 (16%)

Interview dates

3 (6%)

*The values are given as the number of programs, with the percentage in parentheses.

links to the SF Match, faculty listing, current fellow listing, graduated fellow listing, fellow eligibility criteria, salary and benefits, application links, interview dates, and fellow selection criteria. Statistical Analysis Fellowship programs were stratified by geographic region, number of fellows per year, Accreditation Council for Graduate Medical Education (ACGME) accreditation status, affiliation with a residency program, and top-20 status among affiliated orthopaedic hospitals4 and medical schools5 as per U.S. News & World Report. Within each stratification group, education and recruitment characteristics were analyzed independently, utilizing the chi-square test. For stratification groups, including the number of fellows (1 compared with ‡2), accreditation status (ACGME-accredited or not), residency affiliation, top-20 medical school status, and top-20 orthopaedic hospital status, odds ratios (ORs) were generated with computation of confidence intervals (CIs) utilizing the Baptista-Pike method performed using GraphPad Prism version 7.00 for Mac OS X (GraphPad Software, www.graphpad.com). In all tests, significance was set at p < 0.05.

e60(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 100 -A N U M B E R 9 M AY 2, 2 018 d

d

d

C O N T E N T A N D AC C E S S I B I L I T Y O F W E B S I T E S F O R AC C R E D I T E D O RT H O PA E D I C T R AU M A S U R G E RY F E L LOW S H I P S

TABLE II Comparison of Web Site Content by Program Characteristics*

Program Characteristics

No. of Programs

Education Content

10

39%

Region Northeast

OR

P Value

NA

0.015

Recruitment Content

OR

P Value

NA

0.171

46%

South

17

36%

34%

Midwest

11

50%

40%

West

13

50%

34%

1

31

35%

‡2

20

56%

2.4

0.0001

41%

1.5

0.013

8 43

47% 18%

4.0

0.0001

46% 37%

NA

0.090

NA

0.367

NA

0.937

Yes

45

43%

37%

No

6

49%

38% NA

0.969

Yes

10

45%

No

41

43%

NA

0.605

No. of fellows 31%

ACGME-accredited Yes No Affiliated residency

Top-20 medical school

NA

Top-20 orthopaedic hospital

0.614 37% 37%

NA

0.661

Yes

14

45%

36%

No

37

43%

38%

*NA = not applicable.

Results We found 54 OTA-accredited fellowship programs, offering 87 positions. Twenty programs offered >1 position, totaling 53 positions. All programs listed in the OTA database were found on the SF Match database. The OTA database did not provide working links to individual programs. The SF Match database had 15 links (28%) that linked directly to information about the fellowship. In the Google search, we found 51 programs (94%) that had web sites dedicated to their orthopaedic trauma fellowship and the remaining 3 programs (6%) did not. We found 48 programs (89%) that are affiliated with a residency program and 8 programs that are ACGME-accredited. Ten programs (19%) are affiliated with top-20 medical schools, 8 (15%) are affiliated with a top-20 orthopaedic hospital, and 14 (26%) are affiliated with a top-20 orthopaedic department. Fellow Education Analyzed web sites contained a variable amount of content pertaining to fellow education (Table I). Programs that are ACGME-accredited were more likely to provide information with regard to education content (47% compared with 18%; OR, 4.0 [95% CI, 2.4 to 6.4]; p = 0.0001). Also, programs with ‡2 fellows were more likely to provide information on their web site with regard to education content (56% compared with 35%; OR, 2.4 [95% CI, 1.7 to 3.4]; p = 0.0001). When each

variable was analyzed independently within the number-offellows subgroups, fellowships with ‡2 fellows were more likely to provide information including rotation overview (85% compared with 52%; OR, 5.3 [95% CI, 1.2 to 19.2]; p = 0.015), didactics (75% compared with 35%; OR, 7.3 [95% CI, 1.9 to 23.0]; p = 0.002), academic conferences (65% compared with 35%; OR, 3.4 [95% CI, 1.0 to 11.2]; p = 0.039), and research requirements (70% compared with 35%; OR, 4.2 [95% CI, 1.3 to 13.0]; p = 0.016). No differences in education content were seen by geographic region, residency affiliation, top-20 medical school affiliation, or top-20 orthopaedic hospital affiliation (Table II). Fellow Recruitment Analyzed web sites contained a variable amount of content pertaining to fellow recruitment (Table I). Programs with ‡2 fellows provided more overall recruitment content (41% compared with 31%; OR, 1.5 [95% CI, 1.1 to 2.2]; p = 0.013). When each variable was analyzed independently within the number-of-fellows subgroups, fellowships with ‡2 fellows were more likely to provide information including links to the SF Match and OTA (55% compared with 27%; OR, 3.2 [95% CI, 1.3 to 7.7]; p = 0.005) and eligibility criteria (80% compared with 42%; OR, 5.5 [95% CI, 1.5 to 17.5]; p = 0.007). No differences in recruitment content were seen by geographic region, ACGME accreditation status, residency affiliation, top-20 medical school affiliation, or top20 orthopaedic hospital affiliation (Table II).

e60(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 100 -A N U M B E R 9 M AY 2, 2 018 d

d

d

Discussion The Internet is usually the first place an applicant consults for finding information concerning prospective fellowship opportunities1-3,6,7. Several studies have emphasized the ability of residency program web sites to serve as recruitment tools for prospective applicants8-13. The results of this study indicate that orthopaedic trauma fellowship programs are not optimally exploiting this resource. This is not a finding unique to orthopaedic subspecialty fellowship web sites7-11,14,15. The OTA provides a document for prospective fellows entitled “Tips for Applicants-Choosing a Program” (http://ota. org/media/380528/Fellowship-Applicant-Tips.pdf). This document contains a list of factors that a prospective fellow may want to consider prior to applying to a fellowship. There are a number of questions listed that correspond with data from our study, and web sites’ utility in answering these questions was variable. One of the questions listed was: Do you want to be in a fellowship with other trauma fellows? In our study, we found that less than half (35%) of web sites would provide an applicant the answer to this question. Another question was: Do you want to apply to a large program with multiple faculty members? We found that 29% of the web sites contained the answer to this question. A large portion of this document is dedicated to research. However, we found that less than half of the programs noted a research requirement (49%) or described their research interests (45%). Another large portion of this document was dedicated to call. Call is an integral part of every trauma fellowship, yet only 20% of programs mention their call requirement on their respective web sites. These are details that any prospective applicant would want to know prior to applying to a fellowship. Similar studies have been conducted assessing residency web sites. In their survey of emergency medicine applicants, Gaeta et al. found that nearly half (41%) of applicants did not apply to programs because of the poor quality of their web sites and more than three-fourths (78%) of applicants decided to apply to a program solely because of the information on its web site10. Mahler et al. found that emergency medicine applicants valued program web sites and their mentors’ advice equally when deciding where to apply11. In a survey of internal medicine applicants, most of them used program web sites when deciding where to interview and to ultimately rank programs16. These studies have shown that the absence, inaccessibility, or poor quality of program web sites may negatively impact applicant recruitment, but an informative web site may offer a competitive advantage in recruiting potential applicants. One motivation of this study was to assess differences in the online content of various orthopaedic trauma fellowship programs. We found that larger programs with multiple fellowship positions provided more information on education and more on recruitment content. This was also found in orthopaedic spine fellowships3. We found no differences in content when analyzing web site content based on program location or the affiliation with a residency, top medical school, or top orthopaedic hospital. This indicates that the Internet is underutilized as a recruitment source among all orthopaedic trauma fellowships.

C O N T E N T A N D AC C E S S I B I L I T Y O F W E B S I T E S F O R AC C R E D I T E D O RT H O PA E D I C T R AU M A S U R G E RY F E L LOW S H I P S

A possible explanation for the lack of informative content could be the fact that orthopaedic traumatology is a subdiscipline of an already specialized surgical field. Given the relatively small community of orthopaedic traumatologists, intangible factors such as fellowship reputation or mentors’ advice may be more influential than web site content. Despite this, the availability of information, including a description of the program, list of current and previous fellows, recent publications and current research projects, links to society web sites, an outline of the conference and rotation schedules, a description of the cases completed by fellows, the office or clinic requirement, and the call responsibility, would still be valuable for prospective fellowship applicants. There were several limitations to the present study. The actual utilization of web sites by fellowship applicants remains unknown. The criteria for web site content were determined by consensus of 2 of the authors (a current orthopaedic trauma fellowship applicant and an orthopaedic trauma fellowship director) and were based on previous studies1-3. However, we still believe that these criteria are universally important to all orthopaedic fellowships. It is possible that certain criteria may have varying importance among fellowship applicants. These data were self-reported and did not reflect the quality of information. Another limitation was that SF Match information is only available to prospective applicants who are actively involved in the fellowship application process. Finally, our analysis represented a snapshot. It is possible that programs have updated their content since our query. Our results demonstrate an underutilization of the Internet by orthopaedic trauma fellowship programs for fellow recruitment. Orthopaedic trauma fellowships should strive to improve the accessibility and quantity of their content. This may be achieved by standardizing the information presented on fellowship web sites, with an emphasis on providing details on the educational and recruitment components analyzed in this and previous studies; the OTA may play a role in this regard. Improvements in the availability of information on fellowship web sites may assist applicants interested in applying to orthopaedic trauma fellowships. Although the findings of this study are limited to orthopaedic trauma fellowship web sites, identified areas for improvement are likely applicable across all orthopaedic subspecialties. n

1

M. Kareem Shaath, MD Michael G. Yeranosian, MD1 1 Joseph A. Ippolito, MD 1 Mark R. Adams, MD 1 Michael S. Sirkin, MD 1 Mark C. Reilly, MD 1

Rutgers New Jersey Medical School, Newark, New Jersey

E-mail address for M.K. Shaath: [email protected] ORCID iD for M.K. Shaath: 0000-0002-4171-0434

e60(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O LU M E 100 -A N U M B E R 9 M AY 2, 2 018 d

d

d

C O N T E N T A N D AC C E S S I B I L I T Y O F W E B S I T E S F O R AC C R E D I T E D O RT H O PA E D I C T R AU M A S U R G E RY F E L LOW S H I P S

References 1. Davidson AR, Murphy RF, Spence DD, Kelly DM, Warner WC Jr, Sawyer JR. Accessibility and quality of online information for pediatric orthopaedic surgery fellowships. J Pediatr Orthop. 2014 Dec;34(8):831-4. 2. Mulcahey MK, Gosselin MM, Fadale PD. Evaluation of the content and accessibility of web sites for accredited orthopaedic sports medicine fellowships. J Bone Joint Surg Am. 2013 Jun 19;95(12):e85. 3. Silvestre J, Guzman JZ, Skovrlj B, Overley SC, Cho SK, Qureshi SA, Hecht AC. The Internet as a communication tool for orthopedic spine fellowships in the United States. Spine J. 2015 Apr 1;15(4):655-61. Epub 2014 Nov 29. 4. U.S. News and World Report. Best hospitals for orthopedics. 2017. http:// health.usnews.com/best-hospitals/rankings/orthopedics. Accessed 2018 Feb 5. 5. U.S. News and World Report Education. 2018 Best medical schools. 2018. http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/topmedical-schools. Accessed 2018 Feb 6. 6. Lagwinski N, Hunt JL. Fellowship trends of pathology residents. Arch Pathol Lab Med. 2009 Sep;133(9):1431-6. 7. Perron AD, Brady WJ. Sources of information on emergency medicine residency programs. Acad Emerg Med. 2002 Dec;9(12):1462-3. 8. Chu LF, Young CA, Zamora AK, Lowe D, Hoang DB, Pearl RG, Macario A. Selfreported information needs of anesthesia residency applicants and analysis of applicant-related web sites resources at 131 United States training programs. Anesth Analg. 2011 Feb;112(2):430-9. Epub 2010 Nov 16.

9. Reilly EF, Leibrandt TJ, Zonno AJ, Simpson MC, Morris JB. General surgery residency program websites: usefulness and usability for resident applicants. Curr Surg. 2004 Mar-Apr;61(2):236-40. 10. Gaeta TJ, Birkhahn RH, Lamont D, Banga N, Bove JJ. Aspects of residency programs’ web sites important to student applicants. Acad Emerg Med. 2005 Jan;12(1):89-92. 11. Mahler SA, Wagner MJ, Church A, Sokolosky M, Cline DM. Importance of residency program web sites to emergency medicine applicants. J Emerg Med. 2009 Jan;36(1):83-8. Epub 2008 Apr 24. 12. Rozental TD, Lonner JH, Parekh SG. The Internet as a communication tool for academic orthopaedic surgery departments in the United States. J Bone Joint Surg Am. 2001 Jul;83(7):987-91. 13. Kumar A, Sigal Y, Wilson E. Web sites and pediatric residency training programs in the United States. Clin Pediatr (Phila). 2008 Jan;47(1):21-4. Epub 2007 Aug 10. 14. Hashmi A, Policherla R, Campbell H, Khan FA, Schumaier A, Al-Mufarrej F. How informative are the plastic surgery residency websites to prospective applicants? J Surg Educ. 2017 Jan-Feb;74(1):74-8. Epub 2016 Oct 4. 15. Silvestre J, Tomlinson-Hansen S, Fosnot J, Taylor JA. Plastic surgery residency websites: a critical analysis of accessibility and content. Ann Plast Surg. 2014 Mar;72(3):265-9. 16. Embi PJ, Desai S, Cooney TG. Use and utility of Web-based residency program information: a survey of residency applicants. J Med Internet Res. 2003 Jul-Sep;5(3): e22. Epub 2003 Sep 25.