Early Intervention Surveillance Strategies (EISS)

Educational Methodologies

Early Intervention Surveillance Strategies (EISS) in Dental Student Clinical Performance: A Mathematical Approach Marc Tennant, B.D.Sc., Ph.D.; Estie Kruger, B.Ch.D., M.Ch.D. Abstract: Graduating dental practitioners requires the mastery of a number of skills and a significant body of basic information. Dental education is a complex combination of didactic and physical skill learning processes. It is necessary to develop appropriate tools to measure student clinical performance to allow the provision of interventional strategies at the right time targeted at the right individuals. In this study, an approach to early intervention surveillance strategies was developed that is cost-effective, transparent, and robust based on mathematical predictions of student clinical achievements. Using a cohort of students’ clinical activity profile, a polynomial pair was developed that represents the predictive function of low and high achieving students. This polynomial pair can then be applied to students to predict their final achievement based on their current status. The polynomial methodology is adaptable to local variation such as access to clinical facilities. The early intervention surveillance strategy developed in this study provides a simple, cost-effective, predictive risk assessment system that relies on data sets already collected in most dental schools and can be completed without the need for significant human intervention. The mathematical approach allows the focusing of educational support towards students that require the assistance, thus augmenting the better use of resources. Dr. Tennant is Associate Professor and Director, Centre for Rural and Remote Oral Health; and Dr. Kruger is Research Officer, Centre for Rural and Remote Oral Health—both at the University of Western Australia, Perth, Australia. Direct correspondence and requests for reprints to Dr. Estie Kruger, Centre for Rural and Remote Oral Health, University of Western Australia, 35 Stirling Highway, Crawley, Western Australia 6009; 61-8-9346-7248 phone; 61-8-9346-7237 fax; [email protected]. Key words: dental education, surveillance strategy, early intervention Submitted for publication 3/2/05; accepted 8/18/05

G

raduating competent practitioners requires the mastery of a number of skills and a significant body of basic information. Dental training is a complex combination of didactic and physical skill learning processes. Dental students require considerable supervision and academic intervention to ensure that appropriate skills are developing within the training time or course available. Dental education, like other clinically based higher educational programs, faces challenges related to the high cost associated with the provision of clinical teaching.1 Against the backdrop of diminishing resources, there is an increasing demand for competency on graduation. The development of appropriate tools to measure student clinical performance to allow the provision of timely target interventional strategies is an essential part of the dental education mandate. Historically, dental schools have based their student assessment and early intervention surveillance strategies (EISS) on educational practices in

December 2005

■

Journal of Dental Education

other disciplines. Examples include the use of learning diaries, logbooks, student self-assessment, structured clinical operative tests, personalized interviews, and supervisor feedback.2-9 However, these strategies are resource intensive, and in some instances (unless sustained resources are committed) they risk being unsuccessful.10 In addition, the high variation between individual assessors can often influence the efficacy of the results unless rigorous standardization of assessors is undertaken.11 All early intervention surveillance methods demand reliability, validity, and feasibility. In this study, an alternative approach to EISS was developed that is cost-effective, transparent, and robust.

Methods In Australia, clinical activity can be recorded using the Australian Dental Association (ADA) item numbers. This system is a nationally accepted codi-

1353

fication of treatment provided to patients and thus allows the detailed recording of clinical activity. The Australian Department of Veterans Affairs (DVA) publishes a fee schedule that uses the ADA item numbers as its codification system. This fee schedule is used widely as the basis of pricing dental care. In this study, the DVA schedule was not used as a monetary fee schedule but was applied as a measure of the complexity of different items of care. A complete dataset of clinical activity (ADA item number by month) was collected for a single cohort of students (n=35) as they moved through the last two years of their dental education (2002-03). These data were used to develop the mathematical EISS. At the end of the cohort’s clinical education, six senior experienced clinical academics were asked to stratify the cohort based on their knowledge and experience of the students’ performance. These independently collected stratifications were collated (by frequency analysis), and the stratification produced a list of the four highest and the four lowest achieving students. The six clinical academics were selected from the Board of Examiners. They represented a wide spread of disciplines, and all had more than six hours per week previous exposure to the students. There were no criteria guidance to rank the students, but all the raters were clinical supervisors with knowledge of the students’ work. To effectively complete dental training requires a student to complete a series of care plans on patients. All steps of the care plans are reviewed and agreed to by senior clinicians; this process also ensures the treatment is carried out in a clinically acceptable manner. Care that requires completion by someone other than the student is not recorded (in the data set) against the student, but against the clinician who completed the procedure. This methodology does not allow the measurement of the “quality” of the outcome, but it sets a basic level of competency (for the item to be recorded against the student). The EISS developed in this study is a surveillance tool that facilitates the early prediction of students at risk and requiring educational support. It is not designed to be an assessment tool. It is universally acknowledged that some dental procedures are more complex to undertake than others. More complex care (e.g., crown and bridge work) is not undertaken by students until there is mastery of simple procedures. However, students who advance rapidly in their skill base may achieve this earlier. To ensure that any predictor accounts for rapid mastery, each procedure needs to be weighted for relative dif-

1354

ficulty (i.e., undertaking a simple intraoral radiograph needs to be weighed differently as compared to a full gold crown). Although it might be tempting to develop a weighting system de-novo, it can be simply acknowledged that price (in a non-demand-driven student environment) reflects relative difficulty. As such, in this study the DVA schedule fee of each item was used as a weighting factor. However, this does not preclude others from developing separate weighting scales to match their unique environment.

Polynomial Development Using the dataset of activity against time, the cumulative DVA value of care for each student was calculated at each month of the two final years of education. In each month the low achieving subset and the high achieving subset were calculated separately. (The two subsets were obtained from the expert appraisal frequency analysis.) A polynomial curve of best fit was then developed for each cohort subset: high and low (Figure 1). These two polynomial functions predict, based on the month of experience (over the two years), the value of care at any given time. It should be noted that polynomial functions were chosen as they most closely fitted the distribution of the cumulative value of care. These functions (one for the low and one for the high achiever subsets) are five deep polynomial functions with an R2 approaching 0.99. Other functions (e.g., power and exponential) were tested but were found to be poor fits with R values significantly less than the polynomial functions. The functions for these two close fit curves are: y = 0.0006x 5 + 0.0044x 4 - 0.4928x 3 + 4.7555x2 + 30.099x - 41.411 (high) y = 0.0021x 5 - 0.0825x 4 + 0.9434x 3 1.3379x2 - 5.3018x + 59.323 (low) where x is the month and y is the predictive cumulative schedule value of care for that month. Obviously, as more data become available (as more cohorts move through the course), it is possible to refine the functions to be even closer predictors of performance.

Relative Risk Ratio The derived polynomial functions can then be applied to the value of care at a given time for any student to predict his or her future value of care. Most

Journal of Dental Education ■ Volume 69, Number 12

importantly, it allows a simple mathematical approach to predict the value of care a student will produce at the end of their two years of clinical training. This end-point value of care can then be used to determine a relative risk ratio (RRR) for a student. The RRR gives a risk estimate of the student’s predicted ability to meet the care outcomes that would be expected to provide adequate clinical experience. It is calculated using the difference between the test student’s value of care and the average between the high and low risk polynomials. The RRR eliminates dollars from the reported data, thus eliminating the “perceived risk” of students being driven by value of care.

Noting that the RRR is below zero, and although some improvement can be seen midway through fourth year, it is predicted that the RRR at the end of the fourth year would be significantly negative and highlights a need for intervention to prevent a future “risk position” with this student’s ability to achieve. A second example (using hypothetical data) is presented where the RRR becomes positive (Figure 5). This hypothetical situation predicts that a student is relatively safe in the predicted outcome.

Increasing the Robustness of EISS Clearly, the opportunity for clinical activity by students is determined by their access to clinical facilities. This access is dependent on a host of factors including the design of the curriculum. This means that each education facility would have to develop its own unique polynomial pair. It also means that changes in curriculum (and other factors) that result in a change in student access to facilities will influence the shape of the polynomial. Thus, an additional development on the polynomial calculation would be to calculate

Application of the Polynomials A randomly chosen student’s data for the third year is presented compared to the high and low polynomials (Figure 2). Applying the formulas to the test student’s data delivers three endpoints for the fourth year: a high and low prediction (Figure 3). The RRR for the test student is superimposed on Figure 4.

1000

UGC UGD

800

Poly. (UGC) Poly. (UGD)

600

400

Oct

Aug

Jun

Apr

Feb

Nov

Sep

Jul

May

0

Mar

DVA Dollars

200

Months Figure 1. The fitting of five deep polynomial predictive functions to the high (UGD) and low (UGC) groups

December 2005

■


1355

High

900

Low

800

Test Students (LP)

700

Test Students (HP)

600 500 400

DVA Dollars

300 200 100 0 -100

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Months (from start of 4th year) Figure 2. The depiction of data for a test student and its comparison with the low and high functions

High

900

Low

800

Test Students (LP)

700

Test Students (HP)

600 500 400

DVA Dollars

300 200 100 0 -100

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Months (from start of 4th year) Figure 3. Predicting the test student end of fourth-year position

1356


High Low Test Students (LP) Test Students (HP) Ratio

1000 800 600 400 200 0

DVA Dollars

-200

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

-400 -600 -800

Months (from start of 4th year) Figure 4. Overlay of the RRR with the predictive functions

High Low Test Students (LP) Test Students (HP) Ratio

1000 800 600 400 200

DVA Dollars

0 -200

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

-400 -600

Months (from start of 4th year) Figure 5. A hypothetical example of a test student where the RRR is greater than 0

December 2005

■


1357

the value of care relative to hours of access each week. This would be undertaken prior to curve fitting and would produce a polynomial independent of facility access, thus improving the robustness of the predictive nature in a time when rapid curriculum changes are taking place. However, for simplicity of presentation, this additional calculation has not been presented in this article.

Conclusion In the current climate of diminished resources for dental education and an ongoing desire by the profession to produce competent, effective practitioners, the EISS developed in this study provides a simple, cost-effective predictive risk assessment system. This system relies on datasets already collected in most dental schools and can be completed without the need for significant human intervention. The mathematical approach allows the focusing of educational support resources towards students that require more assistance, thus directing resources to where they are needed.

REFERENCES 1. Tennant M, McGeachie JK. Dental school structures in Australia: heading to the 21st century. Aust Dent J 1999;44:238-42.

1358

2. Dahllof G, Tsilingaridis G, Hindbeck H. A logbook for continuous self-assessment during 1 year in paediatric dentistry. Eur J Paediatr Dent 2004;5(3):163-9. 3. Wanigasooriya N. Student self-assessment of essential skills in dental surgery. Br Dent J 2004;Sep Suppl:11-4. 4. Mattheos N, Nattestad A, Christersson C, Jansson H, Attstrom R. The effects of an interactive software application on the self-assessment ability of dental students. Eur J Dent Educ 2004;8(3):97-104. 5. Ericson D, Christersson C, Manogue M, Rohlin M. Clinical guidelines and self-assessment in dental education. Eur J Dent Educ 1997;1(3):123-8. 6. Macluskey M, Hanson C, Kershaw A, Wight AJ, Ogden GR. Development of a structured clinical operative test (SCOT) in assessment of practical ability in the oral surgery undergraduate curriculum. Br Dent J 2004; 196(4):225-8. 7. Chadwick RG, Mason AG. Development, application and effectiveness of a novel logbook checklist assessment scheme in conservative dentistry. Eur J Dent Educ 1997;1(4):176-80. 8. Tennant M, Scriva J. Clinical assessment in dental education: a new method. Aust Dent J 2000;45(2):125-30. 9. Manogue M, Brown G, Foster H. Clinical assessment of dental students: values and practices of teachers in restorative dentistry. Med Educ 2001;35(4):364-70. 10. Lindemann RA, Jedrychowski J. Self-assessed clinical competence: a comparison between students in an advanced dental education elective and in the general clinic. Eur J Dent Educ 2002;6(1):16-21. 11. Scott BJ, Evans DJ, Drummond JR, Mossey PA, Stirrups DR. An investigation into the use of a structured clinical operative test for the assessment of a clinical skill. Eur J Dent Educ 2001;5(1):31-7.
