Predictive validity of the personal qualities ...

3 downloads 0 Views 232KB Size Report
Medical School, The Mackenzie Building, Kirsty Semple Way, Dundee, DD2 4BF, UK. Tel: 44 (0) ... University in Australia (for further information, see www.pqa.
2011; 33: e485–e488

WEB PAPER

Predictive validity of the personal qualities assessment for selection of medical students in Scotland JON DOWELL1, MARY ANN LUMSDEN2, DAVID POWIS3, DON MUNRO3, MILES BORE3, BOIKANYO MAKUBATE1 & BEN KUMWENDA1 1

University of Dundee, Scotland, 2University of Glasgow, Scotland, 3University of Newcastle, Australia

Abstract Background: The Personal Qualities Assessment (PQA) was developed to enhance medical student selection by measuring a range of non-cognitive attributes in the applicants to medical school. Applicants to the five Scottish medical schools were invited to pilot the test in 2001 and 2002. Aims: To evaluate the predictive validity of PQA for selecting medical students. Methods: A longitudinal cohort study was conducted in which PQA scores were compared with senior year medical school performance. Results: Consent to access performance markers was obtained from 626 students (61.6% of 1017 entrants in 2002–2003). Linkable Foundation Year (4th) rankings were available for 411 (66%) students and objective structured clinical examination (OSCE) rankings for 335 (54%) of those consenting. Both samples were representative of the original cohort. No significant correlations were detected between separate elements of the PQA assessment and student performance. However, using the algorithm advocated by Powis et al. those defined as ‘nonextreme’ (51.5 SD from the cohort mean scores; SD, standard deviation) character types on the involved-detached and on the libertarian–communitarian moral orientation scales were ranked higher in OSCEs (average of 7.5% or 25 out of 335, p ¼ 0.049). Conclusions: This study was limited by high attrition and basic outcome markers which are insensitive to relevant noncognitive characteristics. However, it is the largest currently available study of predictive validity for the PQA assessment. There was one finding of significance: that those students who were identified by PQA as ‘not extreme’ on the two personal characteristics scales performed better in an OSCE measure of professionalism. Futures studies are required since psychometric testing for both cognitive and non-cognitive attributes are increasingly used in admission process and these should include more and better measures of professionalism against which to correlate non-cognitive traits.

Background

Practice points

Medical schools are challenged with the task of selecting applicants who will not only excel academically but also possess personal qualities befitting a career in medicine. In ‘Tomorrow’s Doctors,’ the UK General Medical Council (2009) stipulates that doctors are required as professionals, to be competent, empathic and ethical practitioners. Accordingly, many medical schools are seeking new ways of taking into account a range of personal characteristics in addition to academic achievement and potential in their selection procedure. The UK Clinical Aptitude Test (UKCAT) is currently piloting a ‘non-cognitive’ section which seeks to measure a range of non-academic personal characteristics (including a modified version of the Personal Qualities Assessment, PQA). While applicants’ cognitive abilities can reliably be measured using academic grades at the point of application (in the UK context, both GCSE and A level results have established predictive validity), selection on the basis of

. Medical school admissions remain an imprecise process with considerable variation and limited evidence. . Non-cognitive measures are widely acknowledged to be important but complex to measure, evaluate and use. . The Personal Qualities Assessment (PQA) tests were developed specifically for health care profession and are now being piloted widely through the UKCAT assessment and used in some centres in Australia and several other countries. . It is important to assess the potential of such measures but this requires detailed longitudinal studies which are rarely funded. . This study reports equivocal findings regarding the PQA tests. ‘Predictive validity of the Personal Qualities Assessment for selection of medical students in Scotland’.

Correspondence: J. Dowell, Director of Undergraduate Community Medicine, Division of Clinical and population Sciences and Education, Dundee Medical School, The Mackenzie Building, Kirsty Semple Way, Dundee, DD2 4BF, UK. Tel: 44 (0) 1382 420022; fax: 44 (0) 1382 420010; email: [email protected] ISSN 0142–159X print/ISSN 1466–187X online/11/090485–4 ß 2011 Informa UK Ltd. DOI: 10.3109/0142159X.2011.599448

e485

J. Dowell et al.

personal characteristics remains unproven (McManus et al. 2005). It would be very helpful if it could be shown that non-cognitive attributes assessed through self-report questionnaires have predictive validity for clinical professions such as medicine. In the 2001/2002 and 2002/2003 admissions cycles, two cohorts of school applicants to the four Scottish Medical Schools were invited to sit the PQA tests (Munro et al. 2005). This is a battery of psychometric tests developed at Newcastle University in Australia (for further information, see www.pqa. net.au). This article reports on two instruments: one tailored to measure applicants’ moral orientation in terms of being Libertarian versus Communitarian (LibCom), and the other assessed the four traits Narcissism, Aloofness, Empathy and Confidence (in dealing with people), hence ‘NACE’. It also provides a combined score known as ‘ECAN’ in which scores for empathy and confidence are scored positively and offset by scores for narcissism and aloofness which are scored negatively (Lumsden et al. 2005). A higher LibCom score indicates that the respondent gives greater value to the needs, rights, laws and norms of society rather than the individual (Communitarian). In contrast, a low LibCom score indicates that the respondent gives greater value to the needs, rights and well-being of individuals (Libertarian; Bore et al. 2005a, b). A high ECAN score indicates a greater tendency to be involved with others (high Empathy and Confidence) and a low score greater detachment from others (high Narcissism and Aloofness; Munro et al. 2005). While some evidence exists for the validity of the PQA tests in relation to other personality markers and medical student behaviours (Bore et al. 2005a, b; Munro et al. 2005; Adam et al. submitted), this article seeks to establish evidence of their predictive validity in terms of medical school performance, including simulated clinical exams such as Objective Structured Clinical Examinations (OSCE; Table 1).

Methods We sought comparable performance data from the four Scottish clinical medical schools to match with PQA scores gathered prior to admission. Consent from students was sought both soon after entry to medical school and through an online survey conducted in final year prior to graduation in 2007 and 2008. Students providing consent, whose data could be matched with certainty (school, name and one other ID such as DoB), and who entered in 2002 or 2003 and graduated in

Table 1. Showing the Pearson correlation with LibCom, ECAN and foundation year and OSCE rankings.

LibCom Narcissism Aloofness Confidence Empathy ECAN

e486

N

No. of items

Mean

SD

Alpha

12,245 9301 9301 9301 9301 9301

49 24 24 24 24 96

116 53 46 70 74 285

15.4 8.6 8.0 8.6 7.0 22.5

0.90 0.84 0.86 0.85 0.82 0.92

2007 or 2008, formed the study cohort. After an analysis revealed significant differences in the assessment systems, each school was asked to provide a ‘rank-within-year’ for these graduates based upon written (knowledge-based) and OSCE ( performance-based) achievements in years 4 or 5, as appropriate for the school’s clinical assessment system. In addition, ranks were available for all students after 4th year as part of the standard UK allocation system for junior (Foundation) hospital posts, based upon each university’s weighted system that includes some recognition of work done in the earlier years; this provided a single comparable estimate of rank within year group. To cater for differing year group sizes within respective medical schools, an individual’s rank within his/her year group was converted into a percentile ranking so that direct comparisons were possible between those at different institutions. This approach effectively assumes that the cohorts at different schools were similar at entry and that students ranked top or bottom at each school were comparable. Unfortunately, because our follow-up was retrospective, we could not determine reasons for study drop-out, which could include both good reasons (e.g. intercalated degrees) as well as bad (academic failure or health/professional issues). Thus, we are unable to comment on progression rates. Comparisons were made between the NACE subscale scores, ECAN and LibCom measures and the overall Foundation and OSCE rankings (though data provided by some schools did not allow us to rank their students within each exam, numbers are reduced for the comparison based on OSCE performance). A further comparison was based upon whether students’ PQA scores put them into one of the ‘moral types’ or groups of outliers (more than 1.5 SDs from the cohort norm on either the Libertarian–Communitarian or the Involved-Detached dimension) proposed by Lumsden et al. in 2005 to see if these performed differently from those in the central or ‘normal’ ranges on both these variables (Lumsden et al. 2005; Powis et al. 2005).

Results Participants Of the original pool of 1017 (63% female) medical school entrants from whom PQA data were gathered in 2001 and 2002, we secured consent from 626 to seek their subsequent performance data. Data from one school were not released, meaning foundation year ranking data, ECAN and LibCom scores, were available on only 411 of these 626 students. Of these, 284 (69%) were female; 224 (54.5 %) attended Glasgow, 113 (27.5%) Dundee and 74 (18%) Aberdeen. OSCE performance data (that aims to test practical and professional skills rather than knowledge) were available for 335 of these students.

Analysis Data were analysed with SAS version 9.1. Kolmogorov– Smirnov tests were used to compare the two study samples

Predictive validity of the personal qualities assessment

Table 2. Descriptive statistics (norms) and reliability of LibCom and NACE and its subsets (unpublished data from PQA accumulated database). Foundation year rankings and PQA variables (n ¼ 411) Combined year 4 rankings Combined year 4 rankings 1.000 p-Value OSCE rankings and PQA variables (n ¼ 335) OSCE percentile OSCE percentile 1.000 p-Value

LibCom 0.009 0.851

LibCom 0.065 0.232

Narcissism 0.080 0.104

Narcissism 0.085 0.119

Aloofness 0.009 0.858

Aloofness 0.011 0.842

Confidence 0.043 0.381

Confidence 0.046 0.400

Empathy

ECAN

0.040 0.421

0.036 0.467

Empathy 0.042 0.444

ECAN 0.004 0.948

Figure 1. Scatter plot showing moral types (n ¼ 411). Axes show ‘Z’ scores: standardized mean ¼ 0, SD ¼ 1.

(411 and 335) with the original cohort of 1017. No significant differences were found in terms of gender, LibCom or ECAN scores; so it appeared that our sample was representative of the original study group. Pearson Coefficients were used to assess the correlation between performance within the student’s study group and LibCom or ECAN scores or any of the NACE subscale scores. No correlations were found (Table 2). Using the schema described above, we compared students by ‘moral type’. A total of 313 (76%) of the applicants fell within the central square, whose limits are set at 1.5 SD of the sample mean score standardized ‘Z’ scores; 26% were defined as ‘extreme’ (Figure 1). The Mann–Whitney test comparing Foundation Year ranking between extreme and non-extreme students demonstrated no difference ( p ¼ 0.829). However, observing the performance in the final OSCE Examination, it was found that non-extreme applicants performed slightly better, with an average rank 7.5% higher (or 25 out of 335, p ¼ 0.049).

Discussion This study represents the largest cohort and most prolonged follow-up available for the Personal Qualities Assessment to date. However, it must be recognized that it has a number of weaknesses. First, there was a disappointing follow-up rate due to a combination of factors. Consent to follow-up was not obtained at the time of sitting the assessments but had to be sought retrospectively after the Data Protection Act came into force. There were also identification discrepancies, students leaving their original year groups for a number of reasons (unknown) and the reluctance of one school to release performance data. In addition, it is not possible to explore reasons for attrition as these data could not be obtained retrospectively without consent. These issues are not unique but highlight the challenges of multicentre long-term follow-up studies without core funding. It was clear from the outset that the outcome measures available would probably be insensitive to the non-cognitive attributes which are the focus of the PQA. However, the

e487

J. Dowell et al.

assessments used were reliable, discriminated effectively between students and are known, in general terms, to measure qualities relevant to postgraduate performance. Although knowledge-based achievements might not be expected to correlate with non-cognitive tests, an association with OSCE performance would be of great interest. Despite these reservations, it is important to determine if performance in these broad areas is associated with global PQA scores or individual elements thereof. The positive relationship found in one area thus bears closer scrutiny because the UKCAT is now also testing a very similar set of scales on a far wider basis. It is difficult to know what weight can be put on the link between better OSCE performance and those classified as ‘non-extreme’ according to the combined LibCom and ECAN scales but this does relate to a novel ‘comprehensive approach’ proposed by Bore et al. (2009). Clearly, further work is required in this area with close attention to course attrition rates and reasons, and establishing summative measures of professionalism within medical school assessment procedures. In summary, this study aimed to establish the predictive validity of the PQA assessments using 5-year follow-up in Scottish medical schools and, while identifying one relationship with outcome, it has not otherwise found a clear link. However, there are a number of reasons why this might be the case and this study has provided some insight. It is important that further predictive validity studies, perhaps focused on the UKCAT test instruments, are performed before such tests become used for selection. This will require more sophisticated follow-up techniques and appropriate outcome markers where these are available (Cook et al. 2010).

Acknowledgements The authors are grateful to the Scottish Board for Academic Medicine for funding this study and the Scottish Medical Admissions Working Group for assisting with data collection. Declaration of interest: David Powis, Miles Bore and Don Munro have developed the instrument being evaluated.

Notes on contributors JON DOWELL, MD, is a Reader in General Practice and Admissions Convener at Dundee Medical School, Scotland. His main interest is the development of robust, fair and practical selection techniques for undergraduate medicine. He is currently the lead of the UKCAT Research Panel.

e488

BOIKANYO MAKUBATE, PhD, is a research assistant in the School of Medicine, Dentistry and Nursing. His main interests are Medical Statistics and the use of routine datasets to answer clinical questions. MILES BORE, PhD, is a Senior Lecture at the University of Newcastle Australia. He lectures and conducts research in the areas of personality, values, morality and psychometrics. Miles is a member of the Personal Qualities Assessment research team and has contributed to the literature on the selection of medical students. DON MUNRO, PhD, is a Conjoint (retired) Associate Professor in the School of Psychology, University of Newcastle, Australia, and a member of the Personal Qualities Assessment research team. His main interests are in personality, testing, psychometrics and job selection. MARY ANN LUMSDEN, MD, is a professor of Gynaecology and Medical Education at the University of Glasgow. She was the Associate dean for Admissions for 8 years during which she was involved in setting up the UKCAT admissions test as well as studies of psychometric testing in the admission process and approaches for widening participation. BEN KUMWENDA, MRes, is Research Administrator in Medical Education at Dundee University. He was responsible for the development of Scottish Admissions Follow-up Study database and subsequent preparation of the manuscript. DAVID POWIS, PhD, has been a university lecturer and researcher in physiology and medical education since 1972. At university of Newcastle, Australia, he developed a professional interest, and worked extensively, in the area of medical student selection with the aim of establishing fair principles and appropriate strategy.

References Bore MR, Munro D, Kerridge I, Powis D. 2005a. Not moral ‘reasoning’: A Libertarian-Communitarian dimension of moral orientation and Schwartz’s value types. Aust J Psychol 57:38–48. Bore M, Munro D, Kerridge I, Powis D. 2005b. Selection of medical students according to their moral orientation. J Med Educ 39:266–275. Bore MR, Munro D, Powis D. 2009. A comprehensive model for the selection of medical students. Med Teach 31(12):1066–1072. Cook DA, Andriole DA, Durning SJ, Roberts NK, Triola MM, Cook DA, Andriole DA, Durning SJ, Roberts NK, Triola MM. 2010. Longitudinal research databases in medical education: Facilitating the study of educational outcomes over time and across institutions. Acad Med 85(8):1340–1346. General Medical Council 2009. Tomorrow’s doctors: Outcomes and standards for undergraduate medical education. London: GMC. Lumsden MA, Bore M, Millar K, Jack R, Powis D. 2005. Assessment of personal qualities in relation to admission to medical school. Med Educ 39(3):258–265. McManus IC, Powis DA, Wakeford R, Ferguson E, James D, Richards P. 2005. Intellectual aptitude tests and A levels for selecting UK school leaver entrants for medical school. Br Med J 331(7516):555–559. Munro D, Bore MR, Powis D. 2005. Personality factors in professional ethical behaviour: Studies of empathy and narcissism. Aust J Psychol 57:49–60. Powis DA, Bore M, Munro D, Lumsden MA. 2005. Development of a psychometric test battery as a tool for selecting medical students. J Adult Cont Educ 11:3–14.