International Journal of Clinical and Biological Sciences

Manuscript Title

:

Authors

:

On the use of Multivariate Receiver Operating Characteristic curve analysis with a special reference to Diagnostic Medicine Sameera G and Vishnu Vardhan R

International Journal of Clinical and Biological Sciences This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Cite this article as: Sameera G and Vishnu Vardhan R (2016) On the use of Multivariate Receiver Operating Characteristic curve analysis with a special reference to Diagnostic Medicine, International Journal of Clinical and Biological Sciences, 1(1): 76-85 For more details www.ijcbs.com

about

the

journal

and

submission

guidelines

please

visit

International Journal of Clinical and Biological Sciences Volume 1, Issue 1, Jan-June 2016, pp 76-85

On the use of Multivariate Receiver Operating Characteristic curve analysis with a special reference to Diagnostic Medicine Sameera G and Vishnu Vardhan R* Department of Statistics, Pondicherry University, Puducherry – 605 014. Abstract In handling classification problems, there is a wide variety of statistical classification tools/ techniques in literature. In this paper, the presentation is focused on highlighting an advanced and sophisticated model branching to a multivariate extension of Receiver Operating Characteristic (ROC) curves. ROC curve analysis is one such tool which has a wide spread of applications in biological and clinical diagnosis. Recently, a team of researchers have worked on a classification problem which combines multiple markers to give out a best linear combination in order to classify the individuals into one of the two groups with greater accuracy. The practical use of MROC analysis is illustrated using a real dataset and the outcome of the study signifies the facts that the model is capable of classifying the individuals with 93.7% of accuracy. Keywords: Classification, Multiple markers, MROC curve, Optimal threshold, ROC curve. Received: October 31, 2015; Revised: December 22, 2015; Accepted: December 27, 2015

Abbreviations: SGPT: Serum glutamic pyruvic transaminase; SGOT: Serum glutamic oxaloacetic transaminase; mcv: Mean corpuscular volume; ALP: alkaline phosphatase

Introduction The term “Classification” indicates the method of allocating a group of objects or individuals to one of the predefined status of health (with condition/without condition or benign/malignant or healthy/diseased or alive/dead). In this scenario, the process of allocation purely depends on decision making tools available in statistical literature, namely Discriminant Analysis, Logistic Regression, Classification Trees etc. The hub of these tools is usually referred to as Statistical Decision Theory (SDT). The present scenario in diagnostic medicine is completely dependent on evidences which are clinical in nature. However, the decision making process will generate a platform to provide a validation for such clinical evidences using reliable and convincing statistical arguments. The interface between clinical and statistical evidences has lead to the path of Medical Decision Making. In the stream of Medical Decision Making, radiologists and practitioners of diagnostic medicine have extensively used a classification tool namely Receiver Operating Characteristic (ROC) Curve analysis which has practical relevance and logical sense in identifying the sensible information and assessing performance of the diagnostic test. Even though the tool had its origin *

Corresponding author: Dr Vishnu Vardhan R, Assistant Professor, Department of Statistics, Pondicherry University, Puducherry – 605 014, email: [email protected]

International Journal of Clinical and Biological Sciences Volume 1, Issue 1, Jan-June 2016, pp.76-85

during World War II for analyzing radar images, its applicability in the field of diagnostic medicine was first introduced by Lusted [1] in 1971 to identify the health status of individuals using radiographic images. Even though the tool was invented for classification purpose, it ventured into analyzing and assessing the performance of diagnostic test and also to compare them. For instance, it was used in Psychology to study the perceptual detection of stimuli [2]. ROC Curve has wide spread interdisciplinary applications such as human perception and decision making [3], industrial quality control [4], military monitoring [5] etc. Later, the applications of ROC Curve are seen in many other fields such as experimental psychology, engineering, machine learning, biosciences and many more [6]. Apart from applications of ROC curve, several researchers contributed towards its theoretical development by means of statistical properties. The seminal work on the construction of ROC curve was done empirically and in later years parametric approaches were used to obtain smooth curves. A few notable works of theoretical kind are by Ogilvie and Creelman [2], Dorfman and Alf [3], Egan [4], Bamber [5], Hanley and McNeil [6,7], Hanley [8], McClish [9], Campbell and Ratnaparkhi [10], Hanley [11], Metz et al. [12], Faraggi and Reiser [13], Zou, Gastwirth and McNeil [14], Hall et. al. [15], Lasko et al. [16], Huang and Pepe [17], Vishnu Vardhan and Sarma [18, 19], Vishnu Vardhan, Pundir and Sameera [20], Xu et al. [21], Balaswamy et al. [22, 23] etc. An ROC curve can be defined as the tradeoff between“1- specificity” and “sensitivity” Sensitivity is the probability that the test result is positive when the condition is present and Specificity is the probability that the test result is negative when the condition is absent. The area under the curve (AUC) is an important accuracy measure which emphasizes the ability of a diagnostic test in classifying the group of subjects into predefined populations using a gold standard. For instance, to identify the individuals who are suffering from tuberculosis (TB), there are two biomarkers available namely Adenosine De Aminase (ADA)and Interferon- γ (IFNγ) [24]. Of the two, ADA is user friendly and cost effective when compared to IFN-γ. Apart from this, the percentage of correct classification observed using ADA is comparatively better than IFN-γ. Here, the illustration about the working methodology of ROC curve is confined to ADA. The established gold standard of ADA is 36 IU/L [25]. If any individual‟s ADA value is greater than 36 IU/L, then that particular individual is classified into TB group, otherwise. The significance of the gold standard is that it provides 100% correct classification in turn reflecting the importance of ADA marker in determining the true status of an individual. However, every marker does not have an established gold standard. In such cases, ROC curve analysis has an important role in identifying the threshold value which leads to better classification and is termed as “gold standard or optimal threshold”.

Sameera and Vishnu Vardhan

77


Materials and methods In medical diagnosis, most often decision about the true status of an individual has to be taken by using multiple markers at hand. Such situations will create a platform for the development of robust mathematical structures to define a classifier rule for the allocation of individuals. For example, to identify the presence of liver disorder, a thorough investigation of biomarkers like SGPT, SGOT, mcv, ALP etc. is required. A practitioner depends on his/her intuition and experience to assess the values of these markers obtained through diagnostic tests to identify the true status. This sort of exercise does not provide a statistical evidence for the outcome and may sometimes lead to wrong conclusions thus creating a high risk to the individual‟s survivability. Hence, multivariate statistical tools like Logistic Regression and Discriminant Analysis are to be used to avoid such wrong conclusions. The present paper's theme brings out the practical importance and usefulness of a Multivariate ROC curve in the field of diagnostic medicine with its mathematical feasibility and an ease in providing a better understanding about the outcomes of a particular study. Further, the technique is capable of providing Sensitivity of a diagnostic test which is very important for a medical practitioner. So far in the literature there are few studies pertaining to handle the above circumstances. In this aspect, ROC analysis has made its landmark in handling multiple markers and further emphasizing the outcomes in a sophisticated manner when compared to other multivariate tools. It embeds regression framework to study the effects of characteristics of an individual like age, gender etc. which play a predominant role in predicting the true status. The following are some of the research papers which address this scenario, Tosteson and Begg [26], Thompson and Zucchini [27], Toledano and Gatsonis [28], Pepe [29-31], Alonzo and Pepe [32], Faraggi [33], Dodd and Pepe [34], Cai and Pepe [35], Zhang and Pepe [36]. Further, another stream of work was initiated by assigning weights to each marker. These weights help in linearly combining the markers to produce one single value which can be compared to the threshold to identify the status. Researchers like Su and Liu [37], Faraggi and Reiser [38], Pepe and Thompson [39], Liu et al. [40], Gao et al. [41], Liu et al. [42] etc. provided various ways of defining the weights necessary for linearly combining the markers. Recently, Sameera et al. [43] developed an advanced model namely Multivariate ROC (MROC) curve which linearly combines multiple markers and is further shown to have mathematical ease over Discriminant analysis and the model by Su and Liu [37]. The main aim of their work was to establish a linear combination which minimizes the percentage of misclassification. The added advantage is that their methodology provides an optimal threshold which can be used for identifying the status of an individual. The present paper focuses on highlighting the use and application of MROC curve in diagnostic medicine with the help of Statlog (heart) dataset taken from UCI repository [44]. Sameera and Vishnu Vardhan

78


Numerical Illustration Data Characteristics The Statlog (heart) dataset consists of 270 samples of which 120 (44.4%) are diagnosed with presence of heart disease and 150 (55.6%) with absence of heart disease. The parameters age, sex (Male: 183, 67.78%& Female: 87, 32.22%), chest pain type (4 nominal values), resting blood pressure, serum cholesterol, fasting blood sugar, resting electrocardiographic (ECG) results (0,1 & 2), maximum heart rate achieved, exercise induced angina, old peak, the slope of the peak exercise ST segment, number of major vessels (0-3) colored by fluoroscopy and thal (normal, fixed defect & reversible defect) are considered for diagnosis. Results and discussion The following table depicts descriptive summary for the quantitative parameters (resting blood pressure, serum cholesterol, Max heart rate, old peak) with respect to grouping factor i.e., presence or absence of heart disease. The mean age of the individuals who are suffering from heart disease is observed to be slightly greater than their counterparts. From the data environment, it is noticed that the characteristics resting BP, maximum heart rate and old peak have a significant mean difference between the groups at comparison. However, the mean serum cholesterol levels in both the groups do not differ significantly. Table 1: Descriptive Summary Std. t value (sig.) Deviation Absence 150 52.706 9.509 Age Presence 120 56.592 8.116 Absence 150 128.866 16.457 Resting BP 2.575 (0.011*) Presence 120 134.442 19.095 Absence 150 244.213 54.019 Serum cholesterol 1.946 (0.053NS) Presence 120 256.466 47.969 Absence 150 158.333 19.283 Max heart rate 7.394 (0.000*) Presence 120 138.858 23.131 Absence 150 0.623 0.801 Old peak 7.172 (0.000*) Presence 120 1.584 1.282 '*' represents significance and 'NS' represents Not Significant at 0.05 level Parameters

Heart disease

N

Mean

The table 1 contains the descriptive summary of the quantitative parameters of the study. It also contains comparison results between the two groups to identify the significant parameters in identifying the heart disease. MROC curve analysis has been performed (using an R code, a programming language which supports high end computations) to obtain a linear combination which further helps to know the status of a new individual basing on the „U‟ value. Along with Sameera and Vishnu Vardhan

79


this, related measures of MROC curve such as correct classification rate, threshold value, AUC, sensitivity and specificity are also computed. The MROC curve is depicted in figure 1 and linear combination obtained is 𝑈= −0.022 ∗ 𝐴𝑔𝑒 + 1.323 ∗ 𝑆𝑒𝑥 + 0.829 ∗ 𝐶𝑕𝑒𝑠𝑡 𝑝𝑎𝑖𝑛 𝑡𝑦𝑝𝑒 + 0.019 ∗ 𝑅𝑒𝑠𝑡𝑖𝑛𝑔 𝑏𝑙𝑜𝑜𝑑 𝑝𝑟𝑒𝑠𝑠𝑢𝑟𝑒 + 0.005 ∗ 𝑆𝑒𝑟𝑢𝑚 𝐶𝑕𝑜𝑙𝑒𝑠𝑡𝑒𝑟𝑜𝑙 − 0.724 ∗ 𝐹𝑎𝑠𝑡𝑖𝑛𝑔 𝑏𝑙𝑜𝑜𝑑 𝑠𝑢𝑔𝑎𝑟 + 0.358 ∗ 𝑅𝑒𝑠𝑡𝑖𝑛𝑔 𝐸𝐶𝐺 𝑟𝑒𝑠𝑢𝑙𝑡𝑠 − 0.025 ∗ 𝑀𝑎𝑥𝑖𝑚𝑢𝑚 𝑕𝑒𝑎𝑟𝑡 𝑟𝑎𝑡𝑒 + 1.091 ∗ 𝐸𝑥𝑒𝑟𝑐𝑖𝑠𝑒 𝑖𝑛𝑑𝑢𝑐𝑒𝑑 𝑎𝑛𝑔𝑖𝑛𝑎 + 0.424 ∗ 𝑂𝑙𝑑𝑝𝑒𝑎𝑘 + 0.398 ∗ 𝑆𝑙𝑜𝑝𝑒 𝑜𝑓 𝑡𝑕𝑒 𝑝𝑒𝑎𝑘 𝑒𝑥𝑒𝑟𝑐𝑖𝑠𝑒 𝑆𝑇 𝑠𝑒𝑔𝑚𝑒𝑛𝑡 + 1.269 ∗ 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑎𝑗𝑜𝑟 𝑣𝑒𝑠𝑠𝑒𝑙𝑠 + 0.534 ∗ 𝑡𝑕𝑎𝑙. For example, let the profile of an individual be Age = 70, Sex = 1, Chest pain type = 4, Resting blood pressure = 130, serum cholesterol = 322, fasting blood sugar = 0, resting ECG results = 2, maximum heart rate = 109, exercise induced angina = 0, old peak = 2.4, slope of peak exercise ST segment = 2, number of major vessels = 3 and thal = 3. On substituting an individual‟s marker values in the above linear combination, we arrive at a U score which has to be compared with the optimal threshold value. 𝑖. 𝑒. , 𝑈 = −0.022 ∗ 70 + 1.323 ∗ 1 + 0.829 ∗ 4 + 0.019 ∗ 130 + 0.005 ∗ 322 − 0.724 ∗ 0 + 0.358 ∗ 2 − 0.025 ∗ 109 + 1.091 ∗ 0 + 0.424 ∗ 2.4 + 0.398 ∗ 2 + 1.269 ∗ 3 + 0.534 ∗ 3 ⇒ 𝑈 = 12.3926 The optimal threshold value (obtained using Youden‟s Index, J = max (Sensitivity + Specificity – 1),[48]) for identifying heart disease in an individual when the above mentioned characteristics studied is 7.27 with accuracy (AUC) of 93.7%. Since U = 12.3926 > 7.27, the individual will be allocated to heart disease group. The obtained threshold is observed to have 86.2% of sensitivity and 13.8% of 1-specificity (false positive rate). This means that the threshold is able to identify the true status of individual in a sensible manner with 86.2% by allowing 13.8% of false positive cases i.e., healthy misdiagnosed as diseased. This features out that the performance of the threshold has to be improved in such a way that the percentage of false positive rate can be minimized. In classification analysis, the entire results pertaining to the status of individuals has to be presented in the form of 2 X 2 contingency table, also called confusion matrix. The counts and percentages of the confusion matrix will reveal the nature of the classifier, percentages of misclassification and correct classification. From the confusion matrix, the percentage of diseased individuals that are classified as healthy is 13.33% while the percentage of individuals without heart disease classified as suffering from disease is 15% with overall misclassification Sameera and Vishnu Vardhan

80


rate 14.07% and correct classification rate 85.93%.This states that around 86 out of 100 individuals are classified correctly as suffering from heart disease or not. Figure 1: MROC Curve for Heart data

Table 2: Confusion Matrix for Statlog (Heart) Data set Status

Observed

Predicted

Total

Presence

Absence

Presence

130 (86.67%)

20 (13.33%)

150

Absence

18 (15.00%)

102 (85.00%)

120

148

122

270

Total

The table 2 consists of the details regarding the number of correctly classified individuals and misclassified individuals in each group. Conclusion In medical diagnosis, classification of individuals is a pivotal system which requires statistical validation. The present paper highlighted the importance and use of a statistical decision making and classification tool, namely Receiver Operating Characteristic (ROC) curve. Further, the focus is emphasized on the multivariate extension of ROC (MROC) curve proposed by Sameera et al. (2015)[48]. A real dataset namely Statlog (Heart) data is used to demonstrate the practical ease of the MROC curve analysis. A linear combination is obtained with an accuracy of 93.7%, optimal cutoff 7.27, 86.2% of sensitivity and 13.8% of 1-specificity (false positive rate). If the computed U score is greater than the optimal threshold, then the individual is classified into the Sameera and Vishnu Vardhan

81


group with heart disease, otherwise. The linear combination so obtained provides a sophisticated platform to the practitioners to predict the true status of the individual. However, the application and use of MROC curve analysis is not limited to medical diagnosis. It can be branched to many fields of science and engineering where binary classification with multiple markers is to be handled. Acknowledgement The author, Sameera G, acknowledges DST for supporting her research under DST-INSPIRE fellowship programme (IF130958). Conflict of Interest We declare that we have no conflict of interest References 1. Lusted, L. B. (1971) Signal detectability and medical decision making. Science 171: 12171219. 2. Swets, J. A. (1996) Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers, Lawrence Erlbaum Associates, New Jersey. 3. Green, D. M., and Swets, J. A. (1966) Signal Detection theory and Psychophysics. New York, NY: Wiley. 4. Drury, C. G., and Fox, J. G. (1975) Human reliability in quality control. New York, NY: Halsted. 5. Swets, J. A. (1977) Egilance: Relationships among theory, physiological correlates and operational performance. New York, NY: Plenum. 6. Krzanowski, W. J., and Hand, D. J. (2009) ROC curves for continuous data, Monographs on Statistics and Applied Probability. New York, NY: CRC Press, Taylor and Francis Group. 7. John C. Ogilvie and C. Douglas Creelman (1968) Maximum – Likelihood Estimation of Receiver Operating Characteristic Curve Parameters. Journal of Mathematical Psychology 5: 377 – 391 8. Donald D. Dorfman and Edward Alf, Jr. (1969) Maximum-Likelihood Estimation of SignalDetection Theory and Determination of Confidence Intervals – Rating-Method Data. Journal of Mathematical Psychology 6: 487-496. 9. Egan (1975) Signal Detection Theory and ROC analysis. Newyork, Academic Press. 10. Bamber, D. (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology 12: 387-415. 11. James A Hanley, Barbara J Mc Neil (1982) A Meaning and Use of the area under a Receiver Operating Characteristics (ROC) Curves. Radiology 143: 29 – 36


82


12. James A Hanley, Barbara J Mc Neil, (1983) A method of Comparing the Areas Under Receiver Operating Characteristics Analysis derived from the same cases. Radiology 148: 839 - 843 13. James A Hanley (1988) The Robustness of the “Binormal” Assumptions Used in Fitting ROC curves. Medical Decision Making 8: 197-203. 14. McClish, D. K. (1989) Analyzing a portion of the ROC curve. Medical Decision Making 9:190-195. 15. Gregory Campbell and Makarand V. Ratnaparkhi (1993) An Application of Lomax Distributions in Receiver Operating Characteristic (ROC) Curve Analysis. Communications in Statistics – Theory and Methods 22(6): 1681-1687. 16. James A Hanley (1996) The use of the „binormal‟ model for parametric ROC analysis of quantitative diagnostic tests. Statistics in Medicine 15: 1575-1585. 17. Metz, Charles E., Benjamin A. Herman, and Cheryl A. Roe (1998) Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Medical Decision Making 18(1): 110-121. 18. Faraggi, David, and Benjamin Reiser (2002) Estimation of the area under the ROC curve. Statistics in Medicine 21: 3093-3106. 19. Kelly H Zou, Joseph L. Gastwirth and Barbara J McNeil (2003) A Goodness-of-Fit Test for a Receiver Operating Characteristic Curve from Continuous Test Data. Lecture NotesMonograph Series, Crossing Boundaries: Statistical Essays in Honor of Jack Hall, 43: 59-68. 20. Peter Hall, Rob J. Hyndman and Yanan Fan (2004) Nonparametric Confidence Intervals for Receiver Operating Characteristic Curves. Biometrika 91(3): 743-750. 21. Thomas A. Lasko, Jui G. Bhagwat, Kelly H. Zou and LucilaOhno-Machado (2005) The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics 38: 404-415. 22. Ying Huang and Margaret Pepe (2007) A Parametric ROC Model Based Approach for Evaluating the Predictiveness of Continuous Markers in Case-control Studies. UW Biostatistics Working Paper Series, Paper 318.http://www.bepress.com/uwbiostat /paper318. 23. Vishnu Vardhan R and Sarma KVS (2010) Estimation of the Area under ROC curve using Confidence Interval of Means. ANU Journal of Physical Sciences 2: 1: 29-39. 24. Vishnu Vardhan R and KVS Sarma (2012) Determining the Optimal Cut-point in an ROCCurve: A Spreadsheet Approach. International Journal of Statistics and Analysis 2(3): 219-225. 25. Vishnu Vardhan R, Sudesh Pundir and Sameera G (2012) Estimation of Area under the Roc Curve Using Exponential and Weibull Distributions. Bonfring International Journal of Data Mining 2(2): 52-56.


83


26. Weichao Xu, Jisheng Dai, Y.S. Hung, Qinruo Wang (2013) Estimating the area under a receiver operating characteristic (ROC) curve: Parametric and nonparametric ways. Signal Processing 93: 3111- 3123. 27. Balaswamy S, Vishnu Vardhan R, Rao M.B (2014) A Divergence Measure for STROC curve in Binary Classification. Journal of Advanced Computing 3(2): 68-81. 28. Balaswamy S, Vishnu Vardhan R and KVS Sarma (2015) The Hybrid ROC Curve and its Divergence Measures for Binary Classification. International Journal of Statistics inMedical Research 4: 94-102. 29. S. K. Sharma. et.al (2006) Diagnostic Accuracy of Ascitic Fluid IFN – γ and Adenosine Deaminase Assays in the Diagnosis of Tuberculosis Ascities. Journal of Interferon and Cytokine research 26: 484-488. 30. Vishnu Vardhan R, Sarma K.V.S, Alladi Mohan and Suchitra M.M (2013) Optimality Criteriafor Classification through ROC Curve Analysis in the Presence of Outliers. InternationalJournal of Advanced Computer and Mathematical Sciences 4(2): 136-142 31. Tosteson, A. A.N. and Begg, C. B. (1988) Ageneral regression methodology for ROC curve estimation. Medical Decision Making 8: 204-215. 32. Thompson M. L. and Zucchini W. (1989) On the Statistical Analysis of ROC curves. Statistics in Medicine 8: 1277-1290. 33. Alicia Y Toledano and Constantine Gatsonis (1996) Ordinal Regression Methodology for ROC curves derived from Correlated Data. Statistics in Medicine 15: 1807-1826. 34. Pepe MS (1997) A regression modeling framework for receiver operating characteristic curves in medical diagnostic testing. Biometrika 84(3): 595-608 35. Pepe MS (1998) Three approaches to regression analysis of receiver operating characteristic for continuous test results. Biomterics 54:124-135 36. Pepe MS (2000) An interpretation for ROC curve and inference using GLM procedures. Biometrics 56: 352-359 37. Todd A. Alonzo and Pepe M.S (2002) Distribution – Free ROC analysis using binary regression techniques. Biostatistics 3(3): 421 – 432 38. David Farragi (2003) Adjusting receiver operating characteristic curves and related indices for covariates. The Statistician 52: 179-192. 39. Lori E. Dodd and Margaret Sullivan Pepe (2003) Semiparametric Regression for the Area Under the Receiver Operating Characteristic Curve. Journal of the American Statistical Association 98(462): 409-417. 40. Tianxi Cai and Margaret Sullivan Pepe (2003) Semiparametric Receiver Operating Characteristic Analysis to Evaluate Biomarkers for Disease. UW Biostatistics Working Paper Series, Working Paper 185.


84


41. Zheng Zhang and M.S.Pepe (2005) A Linear Regression Framework for Receiver Operating Characteristic (ROC) Curve Analysis. UW Biostatistics Working Paper Series, Paper 253. http://bepress.com/uwbiostat/paper253 42. John Q. Su and Jun S. Liu (1993) Linear Combinations of Multiple Daignostic Markers. Journal of the American Statistical Association 88(424): 1350-1355. 43. Reiser B and Farragi D (1997) Confidence Intervals for the generalized ROC criterion. Biometrics 53: 644-652 44. Margaret Sullivan Pepe and Mary Lou Thompson (2000) Combining diagnostic test results to increase accuracy. Biostatistics 1(2): 123-140 45. Aiyi Liu, Enrique F. Schisterman and Yan Zhu (2005) On Linear Combinations of Biomarkers to Improve Diagnostic Accuracy. Statistics in Medicine 24: 37-47. 46. Feng Gao, ChengjieXiong, Yan Yan, Kai Yu and Zhengjun Zhang (2008) Estimating Optimum Linear Combination of Multiple Correlated Diagnostic Tests at a Fixed Specificity with Receiver Operating Characteristic Curves. Journal of Data Science 6:1-13. 47. Chunling Liu, Aiyi Liu and Susan Halabi (2011) A min-max combination of biomarkers to improve diagnostic accuracy. Statistics in Medicine 30: 2005-2014. 48. Sameera G, Vishnu Vardhan R and Sarma KVS (2015) Binary Classification using Multivariate Receiver Operating Characteristic curve for Continuous Data. Journal of Biopharmaceutical Statistics. DOI: 10.108010543406.2015.1052479 49. Michie D, D.J. Spiegelhalter D.J, and Taylor C.C (eds.) (1994). Machine Learning, Neural and Statistical Classification. Ellis Horwood Limited. UCI Machine Learning Repository [https://archive.ics.uci.edu/ml/datasets/Statlog+%28Heart%29].

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Cite this article as: Sameera G and Vishnu Vardhan R (2016) On the use of Multivariate Receiver Operating Characteristic curve analysis with a special reference to Diagnostic Medicine, International Journal of Clinical and Biological Sciences, 1(1): 76-85


85