Department of Pathology and Quality Control Division, ... of quality-control data for the clinical pathology ... ings can be used to delineate further an instrument.
Quality-control Statistical Interpretation by Microcomputer PHILIP R. FOULIS, M.D., ALAN M. NORBUT, M.D., FRANK J. LOSOS, III, MT(ASCP), AND G. FREDERICK KESSLER, JR., M.D.
Foulis, Philip R, Norbut, Alan iM., Losos, Frank J., Ill, and Kessler, G. Frederick: Quality-control statistical interpretation by microcomputer. Am J Clin Pathol 74: 636-644, 1980. A comprehensive summary report of quality-control data is of great value in monitoring the accuracy and precision of the clinical chemistry laboratory. This report allows a retrospective appraisal alerting laboratory personnel to possible test control material degradation or instrument malfunction. A microcomputer-based program package is described, designed to reduce the errors and lengthy preparation inherent in the manual generation of such a report. Quality-control summary data is automatically compared with a predefined set of statistical criteria, and any aberrant values are flagged, thus eliminating subjective and nonuniform data interpretation. Statistical comparisons include: number of control points, delta mean, delta standard deviation, standard deviation index, and F-ratio. The summary report is well accepted by the laboratory staff, and its incorporation into the decision-making process allows for an efficient, critical, and uniformly rigorous examination of analytic proficiency. (Key words: Quality control; Microcomputer; Statistics; Computer interpretation.)
QUALITY ASSURANCE in the clinical pathology laboratory has primarily been confined to the manual interpretation of statistical parameters.'•2-5-"'-"->:i Attempts to computerize these quality-control functions into a standard interpretative format have met with limited success."'"" The laboratory computer systems currently available provide only basic quality-control functions with analysis of individual control values, obtained from either assayed or unassayed material, 4 " 110 commonly graphically represented by the computer. The subsequent visual inspection and interpretation of this graphic display may prove inadequate in evaluating subtle control degradation or instrument shift.8-"-13 Better control of laboratory proficiency may be obtained through the reduction of the summary data to several statistical values, including the calculation of mean, standard deviation, and coefficient of variation. The laboratory staff is, unfortunately, limited to a subjective evaluation of analytic proficiency by the manual review of this data. This retrospective appraisal has also proven to be time-consuming, tedious, and error-prone. With the continued advancements
Department of Pathology and Quality Control Division, Clinical Laboratories, Montefiore Hospital, and Department of Pathology University of Pittsburgh Health Center, Pittsburgh, Pennsylvania
in computer technology, we have found that a relatively inexpensive microcomputer incorporating a set of programs can assist in the preparation, interpretation, and printing of a comprehensive quality-control summary report, and such a microcomputer is now routinely employed for this purpose in our clinical pathology laboratory. Materials and Methods The system hardware consists of a North Star Horizon-II microcomputer* containing a ZILOG Z80-AI as the CPU (central processing unit). The system has 48,000 bytes of RAM (random access memory). Two Shugartt SA400 minifloppy disk drives, each doubledensity, single-sided, are employed with 180,000 bytes of storage per disk drive using five-and-one-quarterinch hard-sectored diskettes. In addition, the computer has one parallel and two serial ports used for communication. Currently interfaced are a Soroc§ IQ-120 CRT (cathode ray tube) terminal for visual display and program development and an EXPANDOR.H (I23P) ten-character-per-second impact printer. The Monitor, Disc Operating System (Version 5.0), and North Star extended Disk BASIC were provided by the manufacturer. The quality-control program package, as well as the word processor, was developed in our department by two of us (P.R.F. and A.M.N.), without professional programmer assistance, and was written in North Star BASIC. The programs are therefore readily adaptable to other computer systems of similar hardware configuration employing BASIC as the programmable language. Montefiore Hospital is an adult acute-care 500-bed medical/surgical facility associated with the University of Pittsburgh Medical Center. The clinical laboratory
Received November 7, 1979; received revised manuscript and accepted for publication February 4, 1980. Supported by Public Health Service Biomedical Research Grant No. 5S07 RR05592-08. Address reprint requests to Dr. Kessler: The Montefiore Hospital, Fifth Avenue at Darragh. Pittsburgh, Pennsylvania 15213.
* North Star Computers Inc., Berkeley, California 94710. t ZILOG Inc., Cupertino, California 95014. t Shugart Associates, Sunnyvale, California 94086. § Soroc Technology Inc., Anaheim, California 92801. H Expandor Inc., Upper St. Clair, Pennsylvania 15241.
0002-9173/80/1100/0636 $00.95 © American Society of Clinical Pathologists
636
Vol. 74 • No. 5
637
QUALITY-CONTROL MICROCOMPUTER SYSTEM 1.
Create d i c t i o n a r y of terms
rPlace all English phrases (eg. instrument, control, shift) to be used in summary report into on-line computer file via word processor
FIG. 1. Flow chart of process employed to define the format of the summary report.
Define summary report format
4L
1) Select term from previously defined file 2) Assign hierarchy
(major
heading, subheading(s), control
Store target values
Entry of target values for each pre-defined control generated by summary report format
processes approximately 2.1 million specimens annually, constituting 5.5 million CAP (College of American Pathologists) workload units. There are 405 individual controls evaluated monthly. Thirteen months of quality-control data for the clinical pathology laboratory are stored on a single diskette. Results System
Design
The quality-control program package was designed without a specific time limitation and, as a result, can be employed to generate a summary report of data accumulated over any period. Possible constraints to the generation of this cumulative report include the size of the laboratory together with the ability to accumulate statistically sufficient data points, 2 as well as the time limitations placed on laboratory personnel. Taking these factors into consideration, it was determined to generate the summary report once a month in our laboratory. Consequently, in the results and discussion that follow, the quality-control summary report is referred to as "monthly." This artificial temporalization does not preclude the evaluation of quality-control data over another interval, but merely represents the most workable time frame for our laboratory. The quality-control statistical package employs a set of programs that define a statistical format for data handling, achieve automatic interpretation of calculated results with subsequent storage, and print a
monthly quality-control summary report (Fig. 1). Initially, the names of all instruments, divisions, controls, and other English phrases that will be used to define the format of the final report are randomly entered in the computer via a word processor. Corrections, additions, deletions, or substitutions may be made to the file at any time, allowing the monthly report to reflect current laboratory instrumentation and reference controls. The program to define the format of the summary report has four hierarchies of the headings. The major or initial heading may be used to define a division within the laboratory (chemistry), a section within a division (manual chemistry), or a major instrument (SMAC**, DuPont ACAtt). These major headings can be subsequently used to generate partial summary reports for each specific division of the laboratory. The second and third hierarchies of headings define a smaller instrument (spectrophotometer) or a specific assay (sodium, potassium, quinidine). These subheadings can be used to delineate further an instrument by work shift ( A . M . , P . M . , night) or assay ( P c o ^ P 0 2 ,pH). Individual controls constitute the final heading. An additional program searches for the appropriate control and stores the corresponding group mean and group standard deviation (SD), which are subsequently used as a baseline for comparison with the current month's summary data. In addition, the program stores the CAP standard deviation for a corresponding ** Technicon Instrument Corporation, Tarrytown, New York. tt DuPont Instruments, Wilmington, Delaware.
A.J.C.P. • November 1980
FOULlSCT/lt.
638 12
methodology and instrument. The calculated in-house standard deviation is also saved for each individual control. Individual lots of chemistry control sera are obtained in sufficient quantity to last 12 to 15 months, eliminating the need to update these values frequently. (See Glossary for a definition of terms.) The final program (Fig. 2) creates and prints the quality-control files in the form of a monthly summary report. The user initially identifies the month and then enters the number of individual control determinations, standard deviation, and mean interactivity through a CRT display (Fig. 3). Values obtained that are greater than three standard deviations are excluded from subsequent calculations.2 To assist in data entry, the location within the file and the name of the control are always listed. If the last month's control lot number is identical to that of the present month, a comparison of the two is automatically performed. The user may verify or may edit
any value that appears unsatisfactory. A final printout of the data with accompanying computer interpretation is then obtained for review and distribution throughout the laboratory. Appropriate comments or explanations may be appended to the final report via the word processor, with subsequent storage in a separate data file. Our hospital employs a DEC PDP 11/34H computer with LCI§§ software that is used extensively within the pathology laboratory. It performs routine administrative data handling and basic quality-control functions. Specifically, it is able to calculate the mean and standard deviation for each control and generate appropriate Levy-Jennings charts.8 For departments not utilizing a large laboratory based computer, an additional program on the microcomputer tt Digital Equipment Corporation, Maynard, Massachusetts. §§ Laboratory Consulting Inc., Madison, Wisconsin.
Input Individual Control 1) Number of Points 2) Standard Deviatior 3) Mean
Computer compilation and interpretation of parameters
Print Summary Report
Return to same control to accept new user input
Proceed to next control CRT Display of results \ with identification of statistical significance/
Store Results
Yes
No
FIG. 2. Flow chart of qualitycontrol system to enter and store results, with generation of summary report.
vol. 74 . No. 5
QUALITY-CONTROL MICROCOMPUTER SYSTEM
639
SMAC GLUCOSE CONTROL - Z
NUMBER OF POINTS? STANDARD DEVIATION? MEAN? FIG. 3. CRT display for entry of monthly data. The listing of all headings assists the user in locating the correct control. The user inputs all underlined numbers and letters. Once the values are accepted, the program stores these values and proceeds to the next control.
D0
114 .238 8.5
you WANT TO COMPARE
WITH LAST MONTH'S DATA (Y OR N) ?
Y
GROUP NUM
MEAN
S.D.
C.V.
MIN/MAX
S.D.I- F-Ratio
114
8.50
.238
2.80%
8.0/9.0
0.00
0.91
0.98
10?
8.45s
.261&
3.09%
7.9/9.0
-0.20
1.09
1.18
ARE THESE VALUES ACCEPTABLE FOR STORAGE (Y OR N)?
is available to store daily quality-control points and calculate the necessary parameters for each control: number of points, mean, and standard deviation. Operative Statistical
CAP
Parameters
Six parameters (Table 1 and Appendix) are then automatically calculated and interpreted by the summary programs, including: (1) number of control points; (2) significance of the change in mean from last month to present month (delta mean); (3) significance of the change in standard deviation from last month to present month (delta SD); (4) significance of present month's standard deviation index (SDI—accuracy index) 10 ; (5) significance of present month's group Fratio (precision index); (6) significance of present month's CAP F-ratio (precision index). Changes of statistical significance (P = 0.05)7 in any control parameter are identified on the summary form by one of three computer-generated characters. The absence of any modifying character signifies the acceptability of that parameter. (1) Number of Control Points. A question mark
F-RatiO
Y
(?) is placed next to the total number of points if less than 21 were tallied for the month. Since further interpretation is statistically unreliable, an ampersand (&) is placed beside the subsequently derived mean, standard deviation, SDI, and F-ratios. (2,3) Delta Mean and Delta SD. The delta mean and delta SD checks3-4-7 compare the current mean and standard deviation with last month's mean and last month's standard deviation (see Glossary). A significant delta mean check signifies the presence of substantial systematic error and always indicates the need for a thorough examination of the quality-control plots for the control(s) indicated. Typical systematic errors arise from poor calibration technics, inaccuracy of calibrating standard, reagent deterioration, or drift in instrument electronics. A significant delta SD check indicates substantial random error. These typically arise from pipetting errors, variations in mixing, or the effects of light, evaporation, temperature, or physical inconsistencies in the control sera. If either the delta mean or delta SD check is significant, an asterisk (*) appears next to the corresponding value for mean or standard deviation.
640
FOULIS
CONTROL-Y
A.J.C.P. • November 1980
ETAL.
FOR LDH 1 2 1 1 1
> 460 548 -
-
453
1
11
448
1
I
1 - 1
11
1
1
1 - 1 1
1 1
1
443
1
1
438
1 1
433 428 -
-
423
1
1
2
1 - - 1 - 1
11 11 1
418
1 1 2 1 1 1
1
413
1
2
1
408 1 403 1
1
1
1
1
FIG. 4. Both the delta mean and the Standard deviation index (SDI) demonstrate a loss of accuracy. The graphic display reveals an upward shift in the control values starting on the thirteenth of the month. Investigation into the problem showed a change of substrate for LDH starting on that day.
1
1 1
5 4
< 396 1
1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 DATE
1 2 4 5 6 7 8 9 0 1 2 3 4 6 7 9 0 1 2 3 4 5 6 7 8 9 0 GROUP NUM
MEAN
S.D.
C.V.
MIN/MAX
S.D.I.
F-Ratio
CAP F-Ratio
LDH Control-Y
69
428.2* 29.5
6.88%
369/487
+2.29*
2.97*
43
396.8
6.05%
349/445
+0.46
1.97
24.0
Criteria used to determine the statistical significance of these parameters are discussed in the Appendix. (4) Standard Deviation Index (SDI). In contrast to the delta mean and delta SD checks, which are com-
parisons employing last month's values, the SDI is calculated with values obtained from outside the laboratory. A group mean is subtracted from the current mean. This difference is then divided by a group
Table 1. Definition and Significance of Interpreted Parameters Definition
Statistically unreliable if less than 21 points
1. Number of control points 34
Significance
2. Delta mean check '
Current mean - last month's mean
Greater than or equal to one in-house standard deviation
3. Delta SD check"
Current SD - last month's SD
Greater than or equal to one half the in-house standard deviation7
4. Standard deviation index
Current mean - group mean Group SD
Greater than 1.5 or less than - 1.5
2
(Current SD) 5. Group F-ratio
(Group SD)2
If greater than 2.12
6. CAP* F-ratio
(Current SD)2 (CAP SD)2
If greater than 2.12
* College of American Pathologists.
Vol. 74 • No. 5
641
QUALITY-CONTROL MICROCOMPUTER SYSTEM CONTROL-Z FOR IRON (FE) > 191
1 2
1 1
191
1
189 186
2
1
183
2-
180 - - 177 FIG. 5. Both the delta SD and group F-ratio demonstrate a loss of precision. The graphic display reveals random scatter of values above and below the mean (180), with the majority of points within allowable limits (2 SD). The cause of imprecision was of indeterminate origin and was alleviated by routine preventive maintenance.
1
174 1 2 2
2
1
1
- 2 1 2 1 2 1 1 1
1
1 2
1
1
1 1 1
1 - 1
2 21
171 1 1
1
1
1 - 1 1 1
1
1
1
2 1 1 1 1 1 1
1
2
1 1
1
3
1
1
169 1 - -
