Outcomes Mapping: A Method for Dental ... - Journal of Dental Education

Outcomes Mapping: A Method for Dental Schools to Coordinate Learning and Assessment Based on Desired Characteristics of a Graduate Galen B. Schneider, D.D.S., Ph.D., FACP; Marsha A. Cunningham-Ford, B.S., M.S., R.D.H.; David C. Johnsen, D.D.S., M.S.; Mary Lynn Eckert, B.A.; Michael Mulder, B.S. Abstract: This project, utilizing a seldom-used approach to dental education, was designed to define the desired characteristics of a graduating dental student; convert those characteristics to educational outcomes; and use those outcomes to map a dental school’s learning and assessment programs, based on outcomes rather than courses and disciplines. A detailed rubric of the outcomes expected of a graduating dental student from this school was developed, building on Commission on Dental Accreditation (CODA) standards and the school’s competencies. The presence of each characteristic in the rubric was mapped within and across courses and disciplines. To assess implementation of the rubric, members of two faculty committees and all fourth-year students were asked to use it to rate 1) the importance of each characteristic, 2) the extent to which the school teaches and assesses each, and 3) the extent to which each counts toward overall assessment of competence. All thirty-three faculty members (100 percent) on the committees participated, as did forty-six of the fifty-five students (84 percent). The groups gave high scores to the importance of each characteristic, especially for knowledge and technical competence (then separate categories but merged in the final rubric) and for self-assessment, as well as the extent to which they are being taught and assessed. Respondents most commonly named critical thinking as the area that should be emphasized more. Mapping the curriculum and creating its related database allow the faculty and administration to more systematically coordinate learning and assessment than was possible with a course-based approach. Dr. Schneider is Executive Associate Dean, The University of Iowa College of Dentistry & Dental Clinics; Prof. CunninghamFord is Associate Professor, Department of Preventive and Community Dentistry, The University of Iowa College of Dentistry & Dental Clinics; Dr. Johnsen is Dean, The University of Iowa College of Dentistry & Dental Clinics; Ms. Eckert is Academic Clinical Program Management Specialist, The University of Iowa College of Dentistry & Dental Clinics; and Mr. Mulder is Senior Systems Administrator, The University of Iowa College of Dentistry & Dental Clinics. Direct correspondence and requests for reprints to Dr. Galen B. Schneider, The University of Iowa College of Dentistry & Dental Clinics, 100 Dental Science Building, Iowa City, IA 52242; 319-335-7146; [email protected]. Keywords: dental education, dental curriculum, curriculum reform, outcomes, competencies Submitted for publication 9/30/13; accepted 12/24/13

H

ealth professions curricula provide course information and are the building blocks for any program of study,1,2 while individual courses feature objectives, content, synopses, and logistics. Collectively, courses constitute the essential components of a curriculum, but they may be insufficient to shape the student’s path to achieving the school’s desired outcomes and thus be inadequate to serve as a guide for learning and performance assessment. It is difficult to identify the characteristics of a graduate competent to practice entry-level general dentistry from a list of courses, just as the presence of a culture of inquiry and respect essential for much of learning will not be apparent from a course list.3 A course-based curriculum also lacks a cohesive way to bring together and balance outcomes regarding knowledge, technical proficiency, critical thinking,

1268

ethics, social responsibility, etc.1,2,4 since training and assessment in those areas cross disciplines and academic years and can appear to be fragmented throughout the curriculum.5 Though progress has been made in expanding the practice of dentistry beyond rote knowledge and technical skill, cultural and logistical barriers between disciplines and curriculum years remain obstacles to a comprehensive view of a student’s development of the many abilities required to be a competent dental practitioner. Similarly, the mission of a predoctoral program to produce graduates with an entry-level competence in general practice may not be reflected in grading systems in schools that rank students by knowledge levels.5 Yet significant progress has been made toward ways to encourage a culture of rigor, inquiry, and respect, which helps

Journal of Dental Education ■ Volume 78, Number 9

faculty members work toward common goals and get beyond disciplinary silo thinking. Many dental schools are taking incremental actions to change the framework of teaching, learning, and assessment beyond simply “moving the boxcars.”1,2,5 We think that an ideal approach to teaching and learning in dental education would instead begin with student outcomes and use them as the predominant guide for the curriculum. A first step toward that end is to direct the discussion of student learning toward desirable outcomes. With this goal in mind, the objectives of our project at The University of Iowa College of Dentistry & Dental Clinics were as follows: 1) to develop a set of desired characteristics of our graduating dental students for use as educational outcomes; 2) to survey our faculty and students regarding current implementation of these characteristics in the curriculum; and 3) to develop a “thematic map” of existing learning programs within the college using these guiding characteristics rather than disciplinebased rubrics. The resulting thematic map is used to chart, retrieve, and review course listings, syllabi, application of the college’s competencies, compliance with Commission on Dental Accreditation (CODA) standards, characteristic outcomes, formative/summative assessments, and points of competence in any combination. This article describes the process of its development, results of the faculty and student surveys, and examples of its use.

Defining Desirable Characteristics of a Dental Graduate To shift the orientation to educational outcomes, one approach has been to articulate the desired characteristics of a dental graduate,4,6 so our initial efforts focused on defining those as outcomes. Faculty focus groups were asked to think freely in developing an agreed upon set of characteristics we want our graduating students to have upon completion of their predoctoral dental education. In addition, first-year dental students were asked to develop a set of characteristics with the intent of reviewing it through each year of dental school. Each faculty group, as well as individual members of those groups, developed a set of characteristics separately; then, the groups discussed similarities among these outcomes and developed a matrix based on educational outcomes versus

September 2014 ■ Journal of Dental Education

methodologies.4,6 The educational outcomes defined in our strategic plan are knowledge, technical skill, critical thinking, ethics, practice management, and social responsibility. The outcomes were plotted against the methodologies of definition, cultivation format, measures, institutional coordination, and competence determination. An especially striking aspect of the exercise was the profound difference in methodologies between knowledge and critical thinking. Also, the outcomes for definition, cultivation format, assessments, institutional coordination, and competence determination were fundamentally different. It was interesting to observe the range of times individual faculty members took to complete the task. The task of articulating student outcomes does not come naturally to all, but all group members did report that once they turned their attention to the task, it required only a couple of hours to complete. When the results were shared and collated, the overlap in ideas was striking. A composite graphic was compiled consisting of seven main characteristics and about 100 words.6 It is important to note that the characteristics were not prioritized, but were defined as equally important. A rubric of the characteristics expected of a graduating dental student was then developed (Figure 1), which was shared with the college’s Executive Committee (twenty individuals including department heads, associate deans, and faculty/staff leaders) and then with the entire faculty. It was remarkable that while the presentation/format of the characteristics evolved, the content changed little from that developed by the focus groups. The graphic “Characteristics of a College of Dentistry Graduate” is now shared with first-year students and the Curriculum Committee, Admissions Committee, and Academic and Professional Performance Committee. We also use this figure as a screensaver on our dental clinic computers.

Assessment of the Characteristics Rubric After developing the initial “Characteristics of a College of Dentistry Graduate,” we sought and received exemption from the university’s Institutional Review Board to assess the rubric. We surveyed groups of faculty members and the senior students anonymously, asking them to rate their perception of 1) the importance of each characteristic (Figure 2, panel A); 2) the extent to which each is taught in the curriculum (Figure 2, panel B); 3) the extent 1269

Figure 1. Desired characteristics of a graduate from The University of Iowa College of Dentistry & Dental Clinics Source: Image used with permission from MedEdPORTAL.

to which each is assessed (Figure 2, panel C); and 4) the extent to which each counts toward overall assessment of competence (Figure 2, panel D). We also asked the participants which characteristic they thought should receive more or less emphasis 1) in teaching (Figure 3, panel A), 2) in assessment (Figure 3, panel B), and 3) in weighting progress toward competence (Figure 3, panel C). The three groups surveyed were the Executive Committee; a committee of thirteen faculty clerkship directors; and senior students three months before graduation. All thirty-three members of the two faculty committees (100 percent) participated. A number of seniors were on extramural rotations at the time of the survey, so 46/55 (84 percent) of this group participated. Of the forty-six participating seniors, forty-five responded to every question. The results of these surveys revealed potentially important similarities among groups. All three groups gave high scores to the importance of each

1270

characteristic, showing that the characteristics judged to be important in the initial survey were all thought to be important a year later. The characteristics knowledge and technical skill (then two separate categories, later merged in the final rubric) received high scores for what is taught, what is assessed, and what counts (how it is weighted) (Figure 2, panels A-D). While all other characteristics received high scores for importance, all the scores dropped off in the respondents’ perceptions of what is being taught, assessed, and counted/weighted. The close proximity of ratings for knowledge and technical skill was not surprising since technical skill has also been called “procedural knowledge” and is referred to that way in the article by our focus group.4 In it, we showed how, in parallel with developing the rubric, focus groups developed a matrix showing the outcomes plotted against methodologies. The outcomes were reflected in the rubric. The methodologies were definition, cultivation format, measures (summative/


formative, objective/subjective), institutional coordination, and competence determination. Methodologies for knowledge and technical skill (procedural knowledge) were similar in key ways. One finding of particular interest was the profound difference between critical thinking and either knowledge or technical skill. Critical thinking involves articulation of what is intended for the student to do, while knowledge involves what it is for the student to know. For critical thinking, there are both objective and subjective assessment components, whereas for knowledge, objective tests can stand alone. Critical thinking takes place over time, while knowledge can be learned and tested in a more high stakes format. This analysis helps to pave the way for program development along the lines of outcomes as much as by disciplines. Other trends were also worth noting. After knowledge and technical skill, the next highest-rated categories for all groups for extent of teaching, assessment, and what counts/weighted were critical thinking and self-assessment (Figure 3, panels A-C). The case can be made that self-assessment is part of critical thinking. It is also important to note that the individual characteristics were perceived similarly for teaching level, assessment extent, and what counts/weighted. For example, practice management had similar ratings for extent taught, extent of assessment, and extent of it counting/weighted by all three groups surveyed. Critical thinking was easily the most common response for what should be emphasized more. While cause and effect are difficult to prove, our research in this area continues.

The Thematic Map The electronic framework for thematic mapping (Figure 4) allows each faculty member to enter data for his or her course. Information entered includes course number, standardized syllabus, relevant competencies of the college, how each competency is assessed and addressed, progress toward competence (formative), competence achieved (summative), reference to the rubric, and relevant CODA accreditation standards.7 The map allows review of each characteristic and outcome relative to or independent of the CODA standards and the college’s competencies.

Thematic Map Development The curriculum mapping project was initiated in spring 2012, when a committee was convened to September 2014 ■ Journal of Dental Education

outline the project. The objectives of the committee were to 1) determine the type of data to collect to map the curriculum, 2) determine what format that data should take, and 3) create an electronic instrument to capture that data. An action plan was initiated to determine how to achieve these objectives. The committee solicited input from the college’s Information Technologies unit. It was determined that a mapped curriculum would have the advantages of allowing us to move towards a more horizontally integrated curriculum; transition to a more competency-based curriculum; respond to the new CODA accreditation standards; identify redundancies across the four-year curriculum; create flow between disciplines and academic years; move towards an interdisciplinary education; integrate the characteristics of a graduate into the curriculum; and follow the progression and professional development of individual students across the four-year curriculum relative to the outcomes described. An additional part of the project was to link course syllabi to the rubric. The syllabus database links course objectives, characteristics of a dental graduate, CODA standards, and the college’s competencies. Issues such as social responsibility, ethics, professionalism, values, critical thinking, technical competence, knowledge, self-assessment, patient and practice management, and function in a professional environment are mapped within the module with and through the course syllabus. The map thus allows us to determine how the formative and summative assessment process in each course contributes to the overall desired outcomes. The goal is also to be able to keyword search the completed database of syllabi. The mapping module allows faculty members to sort and search the database by specific criteria as well.

Capabilities of Thematic Mapping The electronic thematic map gives us a simultaneous, multipurpose perspective on the curriculum and indications of progress relative to specific courses, CODA standards, or the college’s competencies (Figure 5, panel A), student characteristic (Figure 5, panel B), or syllabus individually or collectively within or across courses and within or between departments over the four years. The database allows retrieval of this data entered in any combination. Selection and entry of specific CODA standards or the college’s competencies relative to those standards are available (Figure 5, panel A). The college’s competencies and CODA standards are linked upon entry

1271

of data in the mapping module (Figure 5, panels A and B, red arrow and red underline). Once the data are entered, we can ask where the students are achieving competence (summative) or how are they are progressing toward competence (formative) (Figure 5, panel A, green square). The course director selects how the course is addressing this progress and in what format its faculty are assessing this progress (Figure 5, panel A, green arrows and green underlines). The same process is accomplished relative to our student characteristics rubric to select which characteristics are taught in a course (Figure 5, panel B, red circle and red arrow), allowing the course director to enter how that characteristic is addressed and assessed (green square and green arrows). The electronic map also identifies where a specific characteristic is being taught. For example, Figure 2 indicates that the survey respondents reported

1272

that knowledge and technical skill were adequately taught and assessed. However, in Figure 3, it is evident that critical thinking and judgment require more emphasis. Using the thematic map, we searched for where the characteristic of critical thinking and judgment is entered (Figure 5, panel C, red circle). We were able to determine where in our educational process and in what courses this characteristic of critical thinking was being taught (Figure 5, panel C, green square and green arrow). The mapping module also allows an overview of these concepts within a specific course (Figure 5, panel D, red circle and red arrows). This provides an overview of any one course and an analysis of how that course relates to the overall educational program. Analyses are conducted within a course, within a department’s course, within a specific year, across years, and even within or between departments. The process is updated yearly


and is dynamic in its evolution as an instrument to evaluate our outcomes and provide a framework to modify an evolving curriculum.

Utilizing the Thematic Map to Increase Curriculum Efficiency Another advantage of the thematic map is that it shows redundancies within a dental curriculum. While repetition is viewed as beneficial, redundancy could be culled, allowing for balance within a course, courses, and integrated or interdisciplinary modules. The thematic map permits faculty members to address time usage and decompression issues. Our faculty, for example, expressed a need to allot more time for scholarship, course enhancements, and faculty development. The mapping module identified redundant instruction that could

be reduced or removed to release time from the curriculum contact schedule. This process continues to help faculty members identify topics that are being repeated and thus allow for shared teaching and educational efforts. Shared teaching loads, for the addition of alternative topics, and more advanced presentation of a specific topic within similar courses are also possible outcomes of the mapping program. The tool helps faculty members manage their time more efficiently and collaborate to present similar material in different courses. These efforts help to decompress students’ course schedule, as well. An example of such an activity was conducted in 2012 by the course directors of the first-year histology and gross anatomy courses. By working together to map both courses, they identified and removed redundant instruction and course materials, which were replaced with different or more advanced

Figure 2. Survey respondents’ ranking of perceived importance of each characteristic (panel A), extent to which each is taught (panel B) and assessed (panel C), and weight in overall assessment of students’ progress to competence (panel D) Note: The three groups were the Executive Committee (twenty faculty members), thirteen faculty clerkship directors, and senior students three months before graduation. All thirtythree members of the two faculty groups (100 percent) responded. Forty-six of the fifty-five seniors (84 percent) responded. At the time of the survey, the rubric had eight characteristics since this was before Knowledge and Technical Skill were combined.


1273

educational projects. Based on this mapping analysis, the two courses were combined into one, without loss of content and with significant reduction in hours of student contact time. Preparation time for each hour of student contact was also reduced, thus decompressing time commitments for both course directors and students. The new dynamic, interactive course was well received by students. These improvements resulted from identifying redundancy, highlighting opportunities for shared effort, and discussing horizontal integration of course content among faculty members teaching very similar courses. No longer are the two courses vertically integrated, but rather horizontally integrated. 1274

Discussion The final rubric, “Characteristics of a Graduating Dental Student,” incorporates all the ideas and discussion points generated by the faculty and student surveys as well as an Educational Outcomes Focus Group. These points are collated into a graphic with the seven main characteristics: critical thinking and judgment; ethical and professional values; social responsibility; function in a professional environment; patient/practice management; self-assessment; and technical skills and knowledge. While the presentation of the characteristics evolved, the content


Figure 3. Survey respondents’ perceptions of which characteristics should be emphasized more in teaching (panel A), assessment (panel B), and weighting students’ progress to competence (panel C), by percentage of respondents in each group Note: The three groups were the Executive Committee (twenty faculty members), thirteen faculty clerkship directors, and senior students three months before graduation. All thirty-three members of the two faculty groups (100 percent) responded. Forty-six of the fifty-five seniors (84 percent) responded. The lower percentages in panel A do not reflect respondents’ attitude that critical thinking is not important, but rather that the college does not need to do more because it is doing enough already.

changed little from that developed by the focus groups or when shared with first-year dental students, the Curriculum Committee, the Admissions Committee, and the Academic Performance Committee. The survey results gave high scores to the importance of each characteristic. Knowledge and technical skill scored higher than the others for extent taught and assessed for all respondents. Self-assessment also received relatively high scores on importance. Critical thinking was the most common category needing more emphasis. We also realize that many of the characteristics we think important will be more difficult in which to guide learning and assess performance.


The survey also indicated substantial differences between what is being done in knowledge and technical skill and the other characteristics. The survey clearly showed areas where more progress can be made. We did not interpret these findings as what is already being done, but rather areas needing additional work. This information gives us a sense of where we are now and a direction in which to proceed. A focus on outcomes is a cultural shift as well as a logistical challenge. Reaffirmation of the characteristics a year later indicates a continued focus on educational outcomes. The survey also highlighted characteristics receiving more or less emphasis.

1275

Figure 4. Thematic mapping platform at The University of Iowa College of Dentistry and Dental Clinics Note: This framework calls for development of a module for each course to include CODA standards met, the college’s competencies (developed from CODA standards), progress toward competence (how assessed, how addressed), competence achieved, reference to characteristics rubric, course number, and standardized syllabus.

While a direct task sequence is difficult to establish, elevated awareness and support for additional work on characteristics such as critical thinking provide the impetus to initiate and support programs in these areas. Since the initial efforts of the Educational Outcomes Focus Group, several specific activities have been developed. While cause and effect are difficult to prove, this approach offers encouragement to individual faculty members to take the initiative and for administration to offer support. For example, to link outcomes with evolving assessments, students are presented with the characteristics graphic in year one and are then assessed on those same characteristics as outcomes in year four.6 The framework for thematic mapping enables development of a map of learning programs that reflect the rubric themes and outcomes associated with the frequency with which faculty members use these characteristics to guide learning and assess performance. We are not aware of attempts to map the locations of formal attention to specific characteristics in learning programs. Mapping these outcomes

1276

and programs allows systematic coordination of guided learning and assessment of performance. The thematic map of learning programs capable of crossreferencing outcomes, competencies, and individual courses provides the opportunity for us to identify ongoing activities and gaps for potential program development. Mapping the curriculum also enables us to maintain compliance with the new CODA accreditation standards and our new competencies for the college. We are able to identify course content that needs to be added to or removed from the curriculum to maintain standards. Mapping identifies the absence or presence of interdisciplinary themes as concepts that cross disciplines and curricular years across the four-year program. Examples of this application are found in the areas of ethics, evidenced-based dentistry, implants, technology, interdisciplinary topics, and basic science translational applications. The mapping module allows us to begin to integrate our student outcomes rubric by indicating where each outcome is being addressed and where educational


Figure 5. Sample mappings for competencies, student characteristics, and courses Note: The electronic thematic map provides a simultaneous multipurpose view of the curriculum and indications of progress relative to specific courses, CODA standards, or the college’s competencies (panel A), student characteristics (panels B and C), and how characteristics are addressed and assessed collectively within or across courses within or between departments over the four years (panel D).


1277

gaps exist. This map of learning programs can be “mined” because it reflects the rubric themes and outcomes associated with the frequency with which faculty members use these characteristics to guide learning and assess performance. The map can also aid in constructing learning programs by showing the frequency with which faculty members use characteristics to guide learning and assess performance. While thematic mapping still falls short of the ideal (which would be to start with educational outcomes and then plan the curricular experience derived directly from those outcomes), we submit that this thematic mapping effort is a significant step in that direction. Given the deep cultures emphasizing disciplines, knowledge/technical skills, and “making what we are doing better,” this effort reflects a cultural shift as well as a logistical one in our college.

Conclusion The process of developing and implementing the “Characteristics of a College of Dentistry Graduate” and its mapping database has had a profound effect on our culture of teaching and learning, far beyond the administrative and logistical changes involved. The map offers a first step to identify and systematically coordinate learning and assessment of student performance based upon the outcomes desired of a graduate. With the basic function of the map determined to be sound, the next steps are to improve ease of use and plan for more specific utilization relative to the college’s educational processes.

1278

Acknowledgments

We thank Rich Tack for his contribution and assistance with the clicker exercises, Pat Conrad and Kathryn Befeler for graphics support, and Dr. Deborah Dawson for data use verification.

REFERENCES

1. Harden RM. AMEE guide no. 21: curriculum mapping—a tool for transparent and authentic teaching and learning. Med Teach 2001;23(2):123-37. 2. Mazurat R, Schönwetter DJ. Electronic curriculum mapping: supporting competency-based dental education. J Can Dent Assoc 2008;74(10):886-9. 3. Johnsen DC. Student learning: improving practice. In: Critical thinking: focal point for a culture of inquiry. Hauppauge, NY: Nova Science, 2013. 4. Johnsen DC, Marshall TA, Finkelstein MW, et al. A model for overview of student learning: a matrix of educational outcomes versus methodologies. J Dent Educ 2011;75(2):160-8. 5. Johnsen DC, Lipp MJ, Finkelstein MW, CunninghamFord MA. Guiding dental student learning and assessing performance in critical thinking with analysis of emerging strategies. J Dent Educ 2012;76(12):1548-58. 6. Straub-Morarend C, Spector M, Johnsen D, Holmes D. Dental student evaluation in the predoctoral comprehensive care clinic: II. quarterly (formative/summative) and final (summative) evaluation. MedEdPORTAL, August 21, 2013. At: https://www.mededportal.org/publication/9495. Accessed: September 30, 2013. 7. Commision on Dental Accreditation. Accreditation standards for dental education programs. Chicago: American Dental Association, 2010.
