Accepted Manuscript Evaluating the performance of the Minute Feedback System: A web-based feedback tool for medical students Patrick E. Georgoff, Gabrielle Shaughness, Lisa Leininger, Vahagn C. Nikolian, Gurjit Sandhu, Rishindra Reddy, David T. Hughes PII:
S0002-9610(17)30615-3
DOI:
10.1016/j.amjsurg.2017.08.047
Reference:
AJS 12599
To appear in:
The American Journal of Surgery
Received Date: 4 April 2017 Revised Date:
11 May 2017
Accepted Date: 22 August 2017
Please cite this article as: Georgoff PE, Shaughness G, Leininger L, Nikolian VC, Sandhu G, Reddy R, Hughes DT, Evaluating the performance of the Minute Feedback System: A web-based feedback tool for medical students, The American Journal of Surgery (2017), doi: 10.1016/j.amjsurg.2017.08.047. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
ACCEPTED MANUSCRIPT
ABSTRACT
Background: Medical students often report dissatisfaction with the feedback they
RI PT
receive on their clerkships. This study evaluates the performance of the Minute Feedback System (MFS), a web-based tool designed to facilitate medical student acquisition of same day written feedback from surgery residents and faculty.
SC
Methods: System-generated data, targeted surveys, and end of clerkship questionnaires were used to evaluate MFS performance over a one-year period.
M AN U
Results: 170 students made 3,190 feedback requests and received 1,978 responses (62% response rate). Students felt the system was easy to use (90%), provided useful feedback (74%), and allowed them to obtain more feedback than they would have in its absence (81%). Concerns were raised regarding the quality of electronic feedback and whether the data generated would be used for summative assessment.
TE D
Conclusions: The MFS encourages same-day assessment and increases documented medical student feedback. Further development is required to improve feedback quality,
AC C
EP
response rates, and feedback application.
ACCEPTED MANUSCRIPT TITLE PAGE Evaluating the performance of the Minute Feedback System: a web-based feedback tool for medical students
1
University of Michigan, Department of Surgery, Ann Arbor, MI
Corresponding author:
AC C
EP
TE D
M AN U
SC
Patrick Georgoff, MD
[email protected] 269-598-3203 2185 Churchill Dr. Ann Arbor, MI 48103
RI PT
Patrick E Georgoff, MD1; Gabrielle Shaughness, BA1; Lisa Leininger, MPH1; Vahagn C Nikolian, MD1; Gurjit Sandhu, PhD1; Rishindra Reddy, MD1; David T. Hughes MD1
ACCEPTED MANUSCRIPT ABSTRACT Background: Medical students often report dissatisfaction with the feedback they receive on their clerkships. This study evaluates the performance of the Minute Feedback System (MFS), a
RI PT
web-based tool designed to facilitate medical student acquisition of same day written feedback from surgery residents and faculty.
Methods: System-generated data, targeted surveys, and end of clerkship questionnaires were used to evaluate MFS performance over a one-year period.
SC
Results: 170 students made 3,190 feedback requests and received 1,978 responses (62%
M AN U
response rate). Students felt the system was easy to use (90%), provided useful feedback (74%), and allowed them to obtain more feedback than they would have in its absence (81%). Concerns were raised regarding the quality of electronic feedback and whether the data generated would be used for summative assessment.
Conclusions: The MFS encourages same-day assessment and increases documented medical
and feedback application.
TE D
student feedback. Further development is required to improve feedback quality, response rates,
EP
SUMMARY: The Minute Feedback System is a web-based tool designed to facilitate medical
AC C
student acquisition of written feedback from surgery residents and faculty. It is easy to use, encourages same-day assessment, and increases documented medical student feedback. Further development is required to improve feedback quality, response rates, and data application.
KEYWORD: Education, medical student, feedback, evaluation.
FUNDING SOURCES: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. 2
ACCEPTED MANUSCRIPT INTRODUCTION Feedback is defined as specific information that describes a learner’s performance in a given activity that is shared with the intent of improving on that performance(1). Feedback is
RI PT
critical to medical student learning and professional development. This is especially true in the clinical care environment. Despite its importance, feedback during medical student rotations has been shown to be limited in quantity and quality(2-7). While feedback is a relatively simple concept the delivery of frequent high quality feedback is actually quite challenging. Without an
SC
explicit process for doing so and in the absence of any formal education many clinical educators
M AN U
fall short of feedback goals, often giving feedback that is neither specific nor actionable(8, 9). Additionally, learners may not actively seek out feedback, or may purposefully avoid it if they feel it might negatively impact their final evaluations. Even when feedback is given learners may not recognize it, which leads to frustration for teacher and learner alike(10). This could be a result of learners’ limited ability for self-assessment or due to the content of delivered feedback.
TE D
In an effort to address the challenges of providing frequent, timely, and meaningful feedback to medical students during their surgery clerkship we created the Minute Feedback System (MFS), a web-based feedback tool. This system allows medical students to request and
EP
receive same day written feedback about their performance in the clinical care setting. A pilot
AC C
study showed the MFS to be an easy to use tool that enhances medical student education(11). By documenting feedback, clerkship directors are able to utilize specific comments for mid-rotation feedback sessions with students. Data collected with the MFS also allows for the analysis of resident and faculty response rates as well as the quality of feedback provided to medical students. The purpose of this study is to evaluate the performance of the MFS one year after its implementation, including its utilization, usefulness, and overall impact on the surgery clerkship.
3
ACCEPTED MANUSCRIPT METHODS The MFS was created by the authors using the commercially available Qualtrics© survey software platform. The feedback system was designed to allow medical students to initiate a
RI PT
request for feedback about their performance from surgery residents and faculty throughout their third year surgery clerkship. The system requires the student generate a feedback request via a unique survey link, which is sent to them via email at the beginning of the rotation. The student selects a resident or faculty, identifies how long they worked together (at least half a day), and
SC
requests feedback on a specific clinical skill (Figure 1). Students were expected to make
M AN U
requests and residents and faculty were expected to respond on the same day the interaction took place. Feedback categories include direct observation (physical exam or patient history) and performance (oral presentations, technical skills, or general performance). Each category includes a text field in which the student can ask a specific question. Upon completion of the feedback request the selected resident or faculty receives an email link corresponding to a unique
TE D
feedback response form that identifies the student, the length of interaction, and the feedback request (Figure 2). The resident or faculty then enters feedback into a single free text field by typing or dictating if using a mobile phone. The time required to complete the feedback survey
EP
is intended to be less than one minute. Upon completion, feedback is immediately emailed to the
AC C
requesting student. All data generated using the MFS was stored in a secure database accessible by the clerkship directors and coordinators. Before utilizing the system, all students, residents, and faculty underwent a brief tutorial on its use. This tutorial did not include specific training elements designed to improve the quality of feedback requests or responses. The MFS demonstrated construct validity when utilized during third year medical student surgery rotations in the 2015 academic year(11). Here, we study the system’s performance over the course of one academic year (May 2015 – April 2016). Three data sources were used to evaluate the system’s functionality and identify areas for improvement: 4
ACCEPTED MANUSCRIPT (1) System generated data: This includes the number of requests made, type of requests, response rate, and feedback given. The time between feedback request and response was not available.
RI PT
(2) Surveys: Surveys for students, residents, and faculty were created using the Qualtrics© survey software platform. These surveys were designed to assess the system’s
functionality, advantages, and shortcomings. Surveys were sent via email to students at the completion of their clerkship and to residents and faculty in January 2016 and May
SC
2016.
M AN U
(3) End of clerkship student questionnaire: This standardized questionnaire is administered to students following each of their core clerkships, including surgery. It provides general information about the student’s experience and allows for comparison between rotations. Questions specific to feedback are also posed, including one in which students are asked to rate the “quality of feedback on your performance in this clerkship.” Data was
comparison.
TE D
collected from questionnaires completed the year before the MFS was introduced for
At the end of the academic year data was collected and analyzed. PG and GS performed
EP
manual qualitative thematic analysis of system-generated data and surveys. Feedback was
AC C
iteratively read and coded. Codes were reviewed and discussed with VN and DH to ensure consensus. All data was de-identified and confidential. Approval was obtained from the University of Michigan Institutional Review Board. Unpaired t-tests were used to calculate statistical significance.
RESULTS During the 2015-2016 academic year a total of 173 third year medical students completed their surgery clerkship. Of these, 170 used the MFS at least once. There were 3,190 feedback 5
ACCEPTED MANUSCRIPT requests submitted by students and 1,978 responses from residents and faculty (overall response rate 62%). The average number of requests made per student over the 2-month clerkship was 19 (SD = 5.6). Junior residents (post-graduate year (PGY) 1-2) had a mean 60% response rate,
RI PT
senior residents (PGY3-7) 59%, and faculty 66%. Requests for feedback regarding “general performance” was most frequent (63%), followed by “technical skills” (10%), “physical exam” (7%), “patient history” (6%), and “oral presentation” (5%). Overall, 40% of students used free text boxes to make specific feedback requests. However, only 17% of specific feedback requests
SC
were answered with specific feedback. Table 1 demonstrates representative high and low quality
M AN U
interactions using the MFS. System-generated data was used to provide feedback to medical students during mid-rotation reviews, assess resident and faculty participation, and to meet documentation requirements set forth by the Liaison Committee on Medical Education (LCME). Surveys were sent to all 173 students who were provided access to the MFS. Of these, 113 (65%) completed the entire survey. The majority of students thought the MFS is easy to use
TE D
(90%), provides useful feedback (74%), and allows them to obtain more feedback than they would have in its absence (81%). Overall, 55% of students rated the system favorably (4 or 5 on a 5-point Likert scale). However, only 36% of students had a good understanding of how data
EP
generated from the system would be used. The majority of students cited confusion as to
AC C
whether the MFS was being used as a formative assessment tool, summative assessment tool, or both. When asked how they would like to see this data used, 52% favored its use as a formative and summative assessment tool while 48% favored its use as a formative assessment tool only. No students felt that the system should be used solely for summative evaluation. Table 2 shows representative free text survey results from students. Multiple themes were identified, including the timeliness, utility, and quality of electronic feedback, subpar response rates, and confusion regarding the use of feedback for formative assessment, summative assessment, or both.
6
ACCEPTED MANUSCRIPT Thirty residents and 35 faculty were surveyed midway through the academic year and at its completion. The response rate was 87% for residents and 69% for faculty. The majority of residents and faculty had used the MFS at least once (89%) and thought it was easy to use (97%).
RI PT
Overall, 75% of residents and faculty rated the system favorably (4 or 5 on a 5-point Likert scale). There were a number of reasons residents and faculty did not complete feedback
requests, including “inadequate exposure to the student” (86% of resident respondents who
reported not completing feedback, 48% of faculty respondents who reported not completing
SC
feedback), “feedback was or will be given in person” (44%, 30%), “lack of time” (28%, 13%),
M AN U
“concerns regarding the use of data for grading” (19%, 9%), and “other” (8%, 22%). When asked how they would like to see this data used, 66% of residents and 43% of faculty favored its use as a formative assessment tool only, 2% and 11% as a summative assessment tool only, and 32% and 45% as a formative and summative assessment tool. Table 2 shows representative free text survey results from residents and faculty. Multiple themes were identified, including the
TE D
systems ease of use and timeliness, the quality of electronic feedback, clerkship-wide evaluation and grading deficiencies, and confusion regarding the use of feedback for formative assessment, summative assessment, or both.
EP
End of clerkship student questionnaires were utilized to compare students’ perceptions of
AC C
feedback before and after the implementation of the MFS. Data was collected from the 20142015 academic year (pre-MFS, n=115 complete surveys) and the 2015-2016 academic year (post-MFS, n=192). Feedback-related questions were identified and pre- and post-MFS results were compared. Following implementation of the MFS “quality of feedback” on the surgery rotation rose from 3.39 to 3.63 on a 5-point Likert scale (p = 0.06), the percentage of students stating they received any feedback from a faculty member increased from 90 to 93% (p = 0.6), and the overall quality of the clerkship improved from 3.75 to 4.01 (p = 0.03).
7
ACCEPTED MANUSCRIPT DISCUSSION The Minute Feedback System is easy to use, encourages same-day assessment, and increases the quantity of documented medical student feedback. The majority of students,
RI PT
residents, and faculty rated the system favorably and thought it was easy to use. Documentation from the MFS was also used to fulfill LCME requirements regarding student feedback and
observation. While successful, the MFS requires further development to improve feedback
appropriate use as a formative assessment tool only.
SC
quality and response rates. Supplementary education is also needed to ensure the system’s
M AN U
The MFS played a prominent role in the clinical education of third year medical students on their surgery rotation. On average, each student utilized the MFS 19 times over the course of his or her two-month surgery experience. When surveyed, the vast majority of users felt the system was quick and simple to use. Over 80% of students thought the system allowed them to obtain more feedback than they would have in its absence. Furthermore, over 70% of students
TE D
felt this feedback was useful. By increasing the quantity of feedback during clinical rotations the MFS helps address an important deficiency in clinical education. It also establishes a predictable process for doing so. Data generated by the system was used to guide mandatory mid-rotation
EP
feedback sessions between students and clerkship directors. An informal survey of clerkship
AC C
directors found that having access to this data resulted in more robust and detailed discussion. From an administrative standpoint the MFS is simple to operate and maintain, and its use has helped ease administrative burdens that existed previously because it replaced printed direct observation cards used to fulfill Liaison Committee on Medical Education documentation requirements. While the MFS has demonstrated its utility there are legitimate concerns about the value of electronic feedback and its role in clinical education. Providing quality feedback is not an innate skill that all physicians possess. To be effective, feedback must be specific, actionable, 8
ACCEPTED MANUSCRIPT and tailored to the individual trainee. Feedback is also part of a complicated social interaction that is influenced by institutional culture, individual values, and student expectations(9). Ideally, feedback should be delivered in person shortly after a learner’s performance is observed: this is
RI PT
the gold standard. However, it comes as no surprise that the gold standard is rarely being met(4). In a companion paper, we found that the quality of feedback being provided via the MFS is also low. Overall, ineffective feedback was given three times more often than effective feedback; with most clinicians provide vague or non-specific feedback(12). So where does electronic
SC
feedback fit in? The MFS is designed to supplement timely in-person feedback, not to replace it.
M AN U
Students should be encouraged to actively seek out in-person feedback and integrate it into their clinical practice. In settings where this is challenging or insufficient the MFS can offer another modality to increase the amount of feedback students receive. Furthermore, the MFS is an excellent source of feedback data that be used for educational purposes to improve the quality of feedback being provided by faculty and residents.
TE D
One particularly important survey finding relates to concerns about the use of systemgenerated data for grading. This is interesting because the MFS was designed solely for use as a formative assessment tool. This was explained to all students, residents, and faculty before being
EP
granted access to the system. Despite this, many users expressed concern that the MFS would be
AC C
used for summative evaluation. One student wrote that he was “skeptical that this is not being used for grading” while another stated that “I wouldn’t use the system on days that I did not perform as well as I could because I didn’t want that to be reflected in my grade.” Residents and faculty were also suspicious. One noted that it is “hard to separate from grading, especially when there is so little data available at the end of a rotation.” This last response highlights a major deficiency in all clinical rotations: grading is imprecise and highly variable(13). A severe shortage of summative data, perhaps even more so than formative data, can make it challenging to assign a student’s clerkship grade. It is reasonable, then, that any system that documents a 9
ACCEPTED MANUSCRIPT student’s performance might be considered by users to be a source for summative data. Since these survey results have become available, extra efforts have been made to ensure all MFS users understand the system’s dedicated role as a tool for formative feedback only.
RI PT
There are a number of limitations to this study. First, the use of the MFS was limited to a single institution and a single clinical rotation. Second, the response rate to feedback requests was modest (62%) and varied widely (0-100%). Therefore, active users of the system are overrepresented in the data. Third, data collected via electronic surveys are limited by response rate
SC
and question quality. Fourth, we have limited data regarding the acquisition of in-person
M AN U
feedback by medical students during their surgery rotation. As such, it is unclear how in-person feedback complements electronic feedback obtained via the MFS. Fifth, while the MFS was designed to improve the timeliness of feedback, we have no data that shows the time between feedback request and response. Finally, an in depth evaluation of the quality of feedback
manuscript.
TE D
produced by the MFS is beyond the scope of this paper and will be covered in a separate
The Minute Feedback System is an electronic feedback tool with wide applicability. It is easy to use, encourages timely feedback, and increases the quantity of documented medical
AC C
education.
EP
student feedback, which can be used by students and clerkship directors to enhance clinical
10
ACCEPTED MANUSCRIPT REFERENCES
AC C
EP
TE D
M AN U
SC
RI PT
1. Ende J. Feedback in clinical medical education. Jama. 1983 Aug 12;250(6):777-81. PubMed PMID: 6876333. Epub 1983/08/12. eng. 2. Al-Mously N, Nabil NM, Al-Babtain SA, Fouad Abbas MA. Undergraduate medical students' perceptions on the quality of feedback received during clinical rotations. Medical teacher. 2014 Apr;36 Suppl 1:S17-23. PubMed PMID: 24617779. Epub 2014/03/13. eng. 3. Bing-You RG, Trowbridge RL. Why medical educators may be failing at feedback. Jama. 2009 Sep 23;302(12):1330-1. PubMed PMID: 19773569. Epub 2009/09/24. eng. 4. De SK, Henke PK, Ailawadi G, et al. Attending, house officer, and medical student perceptions about teaching in the third-year medical school general surgery clerkship. Journal of the American College of Surgeons. 2004 Dec;199(6):932-42. PubMed PMID: 15555978. Epub 2004/11/24. eng. 5. Hattie J TH. The power of feedback. Rev Educ Res. 2007;77:81-112. 6. Milan FB, Dyche L, Fletcher J. "How am I doing?" Teaching medical students to elicit feedback during their clerkships. Medical teacher. 2011;33(11):904-10. PubMed PMID: 22022901. Epub 2011/10/26. eng. 7. Moss HA, Derman PB, Clement RC. Medical student perspective: working toward specific and actionable clinical clerkship feedback. Medical teacher. 2012;34(8):665-7. PubMed PMID: 22830325. Epub 2012/07/27. eng. 8. Anderson PA. Giving feedback on clinical skills: are we starving our young? Journal of graduate medical education. 2012 Jun;4(2):154-8. PubMed PMID: 23730434. Pubmed Central PMCID: PMC3399605. Epub 2013/06/05. eng. 9. Lefroy J, Watling C, Teunissen PW, Brand P. Guidelines: the do's, don'ts and don't knows of feedback for clinical education. Perspectives on medical education. 2015 Dec;4(6):284-99. PubMed PMID: 26621488. Pubmed Central PMCID: PMC4673072. Epub 2015/12/02. eng. 10. Kogan JR, Bellini LM, Shea JA. Have you had your feedback today? Academic medicine : journal of the Association of American Medical Colleges. 2000 Oct;75(10):1041. PubMed PMID: 11031157. Epub 2000/10/14. eng. 11. Hughes DT, Leininger L, Reddy RM, et al. A novel Minute Feedback System for medical students. American journal of surgery. 2016 Dec 02. PubMed PMID: 27939007. Epub 2016/12/13. eng. 12. Shaughness G.; Georgoff P.E.; Sandhu G.; Leininger L.; Nikolian VR, R.; Hughes, D.T. Assessment of clinical feedback given to medical students via an electronic feedback system. Manuscript submitted for publication. 2017. 13. Alexander EK, Osman NY, Walling JL, Mitchell VG. Variation and imprecision of clerkship grading in U.S. medical schools. Academic medicine : journal of the Association of American Medical Colleges. 2012 Aug;87(8):1070-6. PubMed PMID: 22722356. Epub 2012/06/23. eng.
11
ACCEPTED MANUSCRIPT
Minute Feedback System Representative high and low quality interactions High quality
RI PT
Request: What are some technical areas that I could improve upon in the OR? Response: Work on faster handling of the needle driver so that in the OR you can really focus on handling live tissue (which is hard to practice). Need to identify the dermal-epidermal junction and practice making your suture tails come out on the same side of the loop. Good job though, keep working on it. Low quality
SC
Request: Any feedback or things to focus on?
M AN U
Response: Very thoughtful and careful. Enjoyed her willingness to help.
AC C
EP
TE D
Table 1: Representative high and low quality interactions collected from the Minute Feedback System. In this setting, a high quality interaction was defined by a request that was clear and specific and a response that reflected on observed behaviors, used nonjudgmental language, and connected feedback to specific, correctable behaviors. Low quality interactions did not meet these criteria.
12
ACCEPTED MANUSCRIPT
Minute Feedback System Representative survey results Timely and useful “More feedback than I would have probably gotten without it.” – student “This system is far superior to other clerkships, where we got little or no feedback.”– student “Attendings and residents actually remember how you did that day.” – students
Ease of use
“Quick and painless.” - resident “Easy to use. Keeps it simple.” – faculty
“Most of the feedback was generic and therefore not very useful.” – student “I feel like this is stuff we should be having as face-to-face conversations daily.” – resident “There are downsides to giving identifiable negative feedback in writing.” – faculty
Formative vs. summative assessment tool
“Great feedback tool but hard to separate from grading, especially when there is so little data available at the end of a rotation.” – faculty “Important to distinguish whether this is aimed at feedback or grading because it is unclear.” – resident “I am skeptical that it is not being used for grading.” – student “Because its use in grading was unclear, I wouldn’t use the system on days that I did not perform as well as I could because I didn’t want that to be reflected in my grade.” – student
TE D
M AN U
SC
Quality of electronic feedback
RI PT
AC C
EP
Table 2: Representative free text from students, resident, and faculty surveys, organized by theme.
13
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
Figure 1. Minute Feedback System student feedback request form.
14
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
Figure 2. Minute Feedback System resident and faculty response form.
15
ACCEPTED MANUSCRIPT
TITLE PAGE Evaluating the performance of the Minute Feedback System: a web-based feedback tool for medical students
1
RI PT
Patrick E Georgoff, MD1; Gabrielle Shaughness, BA1; Lisa Leininger, MPH1; Vahagn C Nikolian, MD1; Gurjit Sandhu, PhD1; Rishindra Reddy, MD1; David T. Hughes MD1 University of Michigan, Department of Surgery, Ann Arbor, MI
AC C
EP
M AN U
TE D
Patrick Georgoff, MD
[email protected] 269-598-3203 2185 Churchill Dr. Ann Arbor, MI 48103
SC
Corresponding author:
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
