by David H. Gustafson, John J. Kestly - NCBI

5 downloads 10067 Views 1023KB Size Report
Computer-aided medical diagnosis using Bayes' theorem (a formally optimal ... versity of Wisconsin College of Engineering, 1513 University Ave., Madison, Wis. 53706. ... obtained from Williams and Fitzgerald at the University of Florida [7,8].
Initial Evaluation of a Subjective Bayesian Diagnostic System

by David H. Gustafson, John J. Kestly, John H. Greist, and Norman M. Jensen A computer-aided diagnostic system using subjectively estimated probabilities for symptom-disease relationships is described and applied to a sample of 200 cases divided among hypothyroid, euthyroid, and hyperthyroid diagnoses. The subjective system is evaluated by comparing it with one using actuarial probabilities developed in standard fashion and one using separate actuarial probabilities for suspected hypothyroid and suspected hyperthyroid categories. Analysis of the data indicates that the subjective model's developmental cost and time requirement is much lower, while it performs as well as either actuarial model.

Computer-aided medical diagnosis using Bayes' theorem (a formally optimal method of revising prior opinion in the light of new evidence) has been a promising area of research for some time but has had little real impact on the practice of medicine. Among the reasons for this, discussed by Woodbury [1], Gustafson et al. [2], and Bruce and Yarnall [3], may be mentioned insufficient data bases resulting from the inaccessibility and poor quality of medical records; incorrect aggregation of data resulting from conditional dependence of data; and an inability to incorporate new information into the diagnostic model because of the difficulty of oollecting sufficient data to develop new likelihood estimates. One solution to some of these problems would be to delay further research and implementation of computer-aided diagnostic systems until adequate data bases have been developed. Then conditional dependence could be identified and accounted for by existing statistical methodologies, although the addition of new symptoms would still not be feasible. Research being conducted in this area [4,5,6] is essentially focusing on the development of better medical information systems, including computerized interviews and record systems. While their potential and need are apparent, these systems have generally not been implemented outside their research-based environments; thus the data bases collected are quite small and often describe special populations. Another solution to the data-base problem is to obtain the likelihood estiSupported by Research Grant HS-00316 from the National Center for Health Services Research and Development. Address communications and requests for reprints to David H. Custafson, Ph.D., University of Wisconsin College of Engineering, 1513 University Ave., Madison, Wis. 53706.

204

Health Services Research

SUBJECTIVE BAYESIAN DIAGNOSIS

mates required for Bayes' theorem through sources other than medical records. The principal author suggested in an earlier paper that subjective estimates of likelihood ratios by medical experts would overcome several of the problems mentioned above [2]. This article reports on the development of a prototype subjective model for the diagnosis of thyroid disease. The performance and cost-effectiveness of this model are compared, across several dimensions, with two actuarial models, employing data from medical records of thyroid disorder obtained from Williams and Fitzgerald at the University of Florida [7,8]. The subjective model described here uses only the 26 signs and symptoms and results of four laboratory tests available from the Florida data. A larger subjective model is being developed that will include a total of 70 signs, symptoms, and labora-tory tests and will also take into account the conditional dependencies among symptoms. The techniques used in constructing the subjective diagnostic and evaluative models are substantiated by recent behavioral studies. Three methods for training physician estimators [9], four philosophies of estimator interaction [10], and five methods for obtaining likelihood estimates were studied and evaluated to ensure that the best available methodologies were used. The Subjective Bayesian Model

The study used the odds form of Bayes' theorem: Q=LQo

~~~~~~~(1)

= the prior odds; the perceived odds that disease A rather

where u° =

than disease B is the correct diagnosis before observing any diagnostic data L=

P(P(S1,..., Si,... , SMI Sm DB) DA) = the likelihood ratio; the odds that the diagnostic data (Sl,... , Sm) will occur given disease A rather than disease B

P=

(DA

Sl,

.

Sm) = the posterior odds; the modified odds that

disease A rather than disease B is the correct diagnosis once the diagnostic data (Si, . .. , Sm) have been observed A random sample of 200 (designated "test cases") was drawn from 1528 abstracts of thyroid patient records comprising 63 clinically confirmed hypothyroid, 1341 euthyroid, and 124 hyperthyroid patients. The 200 test cases were first diagnosed on the basis of 26 symptoms and physical signs (presence or absence) alone; then the results of four laboratory tests (at increased, normal, or decreased levels) were included, making a total of 38 symptoms and test results. To illustrate how the absence, as well as the

Fall 1971

205

Gustafson et al. presence, of

symptoms is used as diagnostic

information, let S7 represent the

absence of Symptom 7 (S7). Then

P(DA |SI,-..., 97, .. . , St) |SI, . , S7, * *., Sn )

P (DB

P(S1 DA), *

[1-P(S7 I DA)],

,P(Sn I DA) P(DA) (2) P(SlIDB),*..,[l-P(S71D,I)],...,P(SnIDB) P(DB) Input to the diagnostic program thus included two prior odds and 38 likelihood ratios. The priors were calculated actuarially on the basis of analysis of t-he 1328 abstracts not included in the sample; the likelihoods were estimated subjectively by physicians; and a computer program calculated the posterior probabilities. The subjective likelihood estimates were made as follows: Four third-year medical residents from the University of Wisconsin hospitals, all with at least one month of clinical experience in the endocrinology department, each considered ten 3 x 5 cards at a time, each card containing one piece of diagnostic information, and ordered them according to their diagnostic value. After discussing their rankings in order to resolve differences in understanding of terms and correct errors in logic leading to differences in perceived diagnostic value, they reranked the cards. Next they estimated likelihood ratios with the aid of a logarithmically calibrated scale of odds, and inconsistencies between the ranking of the cards and the magnitude of the estimates were pointed out to them; they then reestimated the likelihood ratios. The final ratio used as input to the diagnostic program was the geometric mean of the four final estimates. (This function was used to aggregate estimates because it reduces the inflence of any one

,

deviant estimate.)

Evaluation The relative effectiveness of computer-aided diagnostic systems has in the past been evaluated largely by the percentage of errors in diagnosis. Evaluation of diagnostic systems, however, should reflect more than simple accuracy: Sterling, Nickson, and Pollack [11] have implied that a cost should be associated with each missed diagnosis, and Gorry and Barnett [12] believe that any measure of diagnostic performance should be based on both the cost of testing and the cost of misdiagnosis. In this study, therefore, not only the frequency of errors but also their relative seriousness was evaluated. Six errors were possible: a hypothyroid case could be incorrectly diagnosed as either euthyroid or hyperthyroid; a euthyroid case could be diagnosed as hypothyroid or hyperthyroid; and a hyperthyroid case could be diagnosed as euthyroid or hypothyroid. It is characteristic of such complex evaluation problems that several evaluation criteria are imp-ortant and that the relative importance of the criteria varies from one estimator to another; moreover, the extent to which the criteria are satisfied is not always directly measurable on an interval scale, and the criteria are sometimes interdependent. The criteria here (the 206

Health Health Services Services Research Research

SUBJECTIVE BAYESIAN DIAGNOSIS Table 1.

Relative Seriousness of the Six Possible Diagnostic Errors (Weights W1)

Actual patient state

Hypothyroid

0 Hypothyroid ....... 1127 Euthyroid ....... Hyperthyroid ...............................2241

Model diagnosis Euthyroid Hyperthyroid

.1375 0 .1438

.2205 .1614 0

relative importance of the six errors) were weighted by the geometric means of subjective rankings assigned independently by four physicians to the seriousness of each error [13]. The final ratios of relative seriousness are shown in Table 1. The evaluation (E) of a medical diagnostic system, since it is some aggregate of the evaluation of individual criteria, may be expressed as n

E=

i=1

wigp (Xi)

(3)

In this equation, wi represents the weighted seriousness of each error, and /3 ( xi) represents the preference function associated with a given level of satisfaction of criterion xi, which is directly related to the error rate. Since a high value of wi indicates a more serious error, a more effective model would yield a smaller value of E. This model has been successfully applied to other evaluation problems; examples include the measurement of severity of illness [13], determination of patient health status [14], prediction of job choices [15], and evaluation of R&D projects [16]. The advantages of having a single measure of effectiveness include ease of comparison, sensitivity to different types of errors, and elimination of more costly multivariate analysis. The diagnoses made by the "subjective" system were compared with diagnoses of the 200 test cases by a "standard actuarial" model and a "suspect actuarial" model. The "standard actuarial" model uses likelihood ratios calculated by analysis of the 1328 abstracts not included in the test cases. An auxiliary program tallied the incidence of each symptom, sign, or test result, then divided it S Dj) for each of the 38 by N, the total number of observations, to obtain P(Si pieces of diagnostic data and the three disease states. The P( Dj) prior probabilities were calculated by dividing the incidence of each state by N. The odds required for the diagnostic program were constructed from these probabilities. The "suspect actuarial" model was developed to test the hypothesis that diagnostic information is lost because of unclear population definition. In previous studies population definition has varied. Crooks et al. [17], for instance, used patients referred to their institute for radioiodine studies; Billewicz et al. [18] used only patients suspected of hypothyroidism; and Fitzgerald and Williams [8] used patients referred to their clinic with suspected thyroid disease. Likelihoods and prior probabilities for patients referred to the clinic with sus-

Fall 1971

207

Gustafson et al.

Table 2.

Likelihood Ratios Relating Diagnostic Data to Hypothyroidism and Euthyroidism

Symptom, sign,

or

lab test

(uIv) Stubjective

*T3 normal ............................ T3 decreased ......................... PBI decreased ........................ *PBI increased ......................... *I... (24 hr.) increased .................. Facial edema present ................... *PBI normal ........................... I"s' (24 hr.) decreased .................. *I"31 (24 hr.) normal .................... I"31 (6 hr.) decreased ................... Recent onset of lethargy ................ *Fine finger tremor present ............... Recent increase in cold sensitivity ........ *T3 increased .......................... *Warm moist skin ...................... *1"31 (6 hr.) normal ..................... *Recent increase in heat sensitivity ........ *I"3l (6 hr.) increased ................... Dry, coarse, yellow skin ................. *Recent weight loss ..................... Lethargic movements .................. Recent decrease in sweating ............. Thyroid gland tender .................. *Skin normal .......................... Recent weight gain .................... *Sensitivity to temperature unchanged ..... Hyperkinetic movements ................ Fine finger tremor absent ............... *Recent onset of nervousness ............. *Facial edema absent ................... *Weight unchanged ..................... Recent decrease in appetite .............. *Appetite unchanged .................... *Recent increase in sweating ............. *Recent increase in appetite .............. Eye signs present ...................... *Eye signs absent ...................... *Sweating unchanged ...................

58.80 46.50 25.90 16.90 15.60 13.37 12.00 11.82 10.00 9.68 7.54 7.46 6.75 6.41 6.25 5.88 5.26 5.10 5.00 4.55 3.95 3.60 3.50 3.33 2.55 2.22 1.92 1.56 1.54 1.43 1.37 1.32 1.30 1.00 1.00 1.00 1.00 1.00 I, I

Standard actuarial

2.70 5.87 13.90 111.10 142.80 7.24 5.88 6.86 6.25 12.10 4.25 3.12 5.10 21.30 125.00 20.00 200.00 200.00 4.68 5.00 7.55 5.39 1.98 2.85 3.15 1.85 3.33 1.14 2.08 1.92 1.27 1.46 1.49 2.00 1.78 1.52 1.05 1.04 = .783 r = .345

(III)

Suspect actuarial 2.44 7.27 15.10 111.10 71.40 4.43 10.00 5.52 10.30 11.97 1.64 1.39 2.42 7.69 15.20 15.90 30.30 125.00 2.03 4.76 2.71 2.11 3.56 2.78 1.79 2.27 7.69 1.02 1.32 2.08 1.16 1.09 1.04 1.25 1.03 2.29 1.05 1.14

* Ratios favor euthyroidism over hypothyr oidism.

pected hypothyroidism and found to be euthyroid could be significantly different from those for patients referred with suspected hyperthyroidism and found to be euthyroid. In order to test this hypothesis, two sets of likelih-oods and priors were developed, one for suspected hypothyroid patients and one for suspected hyperthyroid patients. Since reasons for referral were not available, four physicians subjectively classified each of the 1528 abstracts in "suspected hypo-

208

Health Services Research

SUBJECTiVE BAYESIAN DIAGNOSIS

Table 3.

Likelihood Ratios Relating Diagnostic Data to Hyperthyroidism and Euthyroidism

Symptom, signi, or lab test

(I)

Suubjective

T3 increased ......................... *T3 normal ........................... *I1" (24 hr.) decreased ................. *PBI normal .......................... *PBI decreased ........................ Ip1 (24 hr.) increased .................. *I3l (24 hr.) normal .................... *Recent weight gain ................... *T3 decreased ......................... Eye signs present ..................... PBI increased ........................ Recent weight loss .................... Warm moist skin ...................... I1S1(6 hr.) increased ................... Recent increase in heat sensitivity ........ *Dry, coarse, yellow skin ................ *13 (6 hr.) normal ..................... Fine finger tremor ..................... Hyperkinetic movements ............... *I1' (6 hr.) decreased .................. *Recent increase in cold sensitivity ........ Recent increase in appetite. *Recent decrease in appetite ............. *Facial edema present .................. *Fine finger tremor absent ............... Recent increase in sweating ............. *Sweating unchanged ................... *Sensitivity to temperature unchanged ..... *Appetite unchanged ................... *Weight unchanged .................... Recent onset of nervousness ............. *Recent onset of lethargy ................ *Skin normal .......................... *Lethargic movements .................. *Eye signs absent ...................... Facial edema absent ................... Recent decrease in sweating ............ Thyroid gland tender ..................

53.40 45.50 33.30 30.30 29.40 22.90 18.20 16.70 11.40 10.00 9.60 7.60 7.50 7.44 7.01 5.55 5.00 4.50 4.47 4.17 4.00 3.57 3.22 3.22 2.94 2.87 2.86 2.78 2.63 2.22 2.18 1.82 1.67 1.45 1.11 1.02 1.00 1.00

ri, II = .911 * Ratios favor euthyroidism over hyperthy roidism.

(III)

(II)

Standard actuarial

1.85 2.08 1.56 1.01 1.18 1.06 =

actuarial 16.10 2.63 6.25 62.50 3.45 5.27 13.10 2.63 4.55 2.92 7.06 1.57 2.20 4.46 1.91 1.79 10.00 2.36 3.68 5.00 4.55 3.17 2.08 1.45 2.78 1.53 1.43 1.89 1.75 1.85 1.63 1.32 1.61 2.94 1.54 1.02 1.44 1.35

31.20 3.00 4.55 35.70 2.63 6.62 15.60 2.78 2.43 4.79 8.39 2.45 2.20 5.07 3.38 1.47 10.00 4.06 6.47 3.23 3.77 4.73 1.92 1.04 2.86 2.52 1.61 2.50 1.08 2.50 2.34 1.69

rII

Suspect

.608

thyroid" or "suspected hyperthyroid" categories on the basis of history information only. The diagnostic program then referenced the appropriate set of ratios according to the suspected state of each test case. Tables 2 and 3 show the likelihood ratios used by each diagnostic model and the coefficients of correlation between the diagnostic values of the indicated ratio sets.

Fall 1971

209

Gustafson et al.

Table 4. Percent of Patients in Each Disease State as Diagnosed by Subjective and Actuarial Models Actual patient state

Model

Model diagnosis Hypothyroid Euthyroid Hyperthyroid WITHOUT LAB TESTS

'Subjective

Hypothyroid .... Standard actuarial Suspect actuarial Euthyroid

......

(Subjective Standard actuarial Suspect actuarial

(Subjective

Hyperthyroid .... Standard actuarial Suspect actuarial

64.3 57.1 28.5

28.6 35.7 71.4

7.1 7.1 0

4.8 2.6 0

87.7 91.2 97.8

7.5 6.1 2.2

0

10.0

90.0

0 0

5.0 45.0

95.0 55.0

WITH LAB TESTS

Subjective

71.4 71.4 50.0

21.4 14.3 42.9

7.1 14.3 7.1

'Subjective

1.8 1.8 0

92.5 93.8 97.4

5.7 4.4 2.6

Subjective

0 0 0

10.0 10.0 15.0

90.0 90.0 85.0

Hypothyroid .... Standard actuarial Suspect actuarial Euthyroid

......

Standard actuarial Suspect actuarial

Hyperthyroid .... Standard actuarial Suspect actuarial

Results

The condition with the largest posterior probability assigned by the diagnostic program was designated the model diagnosis. Diagnostic models were compared in three ways: the rate at which each condition was correctly diagnosed, the average probability assigned to the correct diagnosis, and the seriousness of the errors made. Table 4 compares the actual and the model diagnoses for the test cases both with and without the results of the four laboratory tests. The percentages of correct diagnoses are not appreciably different among the three systems, but the subjective model appears to diagnose hypothyroid patients correctly slightly more often than the actuarial models, whereas the actuarial models perform slightly better for euthyroidism. The reason for this is that the actuarial likelihood ratios for hypothyroidism were based on considerably fewer observations than those for euthyroidism and hence have lower diagnostic accuracy. Interestingly, the "suspected actuarial" model appears biased toward euthyroidism; it diagnoses most patients as euthyroid but does poorly on hypothyroid 210

Health Services Research

SUBJECTIvE BAYESIAN DIAGNOSIS

Table 5. Average Probability Assigned to Correct Diagnosis by Subjective and Actuarial Models Suspect Subjective Standard actuarial actuarial Diagnostic data History and physical exam infornation only .........

.88

.90

.90

History, physical exam, and lab data .............

.92

.93

.94

patients. It may be speculated that the ex post facto method used to classify the patients into suspected states was the cause of this bias. Further research with patients known to have been referred with suspected hypo- or hyperthyroidism is being conducted to determine whether a splitpopulation data base would lead to increased diagnostic accuracy. There is little difference in the average probabilities assigned to -the correct diagnosis by the three models, as shown in Table 5; but on the basis of the frequency and seriousness of the errors made by each system, incorporated in a summary statistic (computed from Eq. 3) in Table 6, the subjective and the standard actuarial models performed slightly better than the "suspect actuarial" model. Costs specific to the development of each model were also compared (with costs common to all models not considered). Total marginal cost to obtain the subjective estimates was $64 in reimbursement for two hours time for each of the four residents at $8 per hour. The marginal cost of the data base for the actuarial models must have been much higher: although actual figures for costs incurred at the source are no longer available, 879 of the medical records used in this study were abstracted retrospectively, and experience indicates that each abstract requires 20 to 30 minutes of time from a trained medical record librarian or a senior medical student, paid $5 per hour. The abstracting of these 879 records would thus require approximately 300 hours of abstracting time and would cost nearly $1500 plus keypunching costs. More important, the time required to develop the data base would be months, rather than the hours needed for the subjective model. (A five-year period was required for collection of the remaining 449 records prospectively.) Table 6. Summary Statistic Expressing Seriousness of Errors Made by Subjective and Actuarial Models Diagnostic data

Without lab tests ........... With lab tests ..............

Fall 1971

Subjective 8.7 7.1

Sacatuarad aSctuaect 8.5 7.5

16.7 10.1

211

Gustafson et al.

Discussion An analysis of individual mistakes made by the models shows that most errors involved patients who had recently taken drugs that confounded their lab-test results. The diagnostic program currently in use at the University of Florida uses heuristic methods to correct for such patients. This characteristic was not included in the study models, because the object was to compare the influence of estimation on Bayesian diagnosis. The more sophisticated subjective model being developed will handle drug and lab-test interactions by clustering them into conditionally independent complexes. There is wide potential for improving the diagnostic models. For one thing, the subjective estimates given by Wisconsin physicians may not be as effective for Florida patients' symptom-disease relationships as for those found in Wisconsin. This may have adversely biased the performance of the subjective model. Additionally, the subjective estimates were obtained from residents in medicine with limited practical experience in endocrinology. The hypothesis that diagnostic performance increases with the expertise of the estimators is currenitly being tested by obtaining estimates from endocrinologists and endocrinology fellows at the Mayo Clinic. Furthermore, in taking advantage of the aggregating power of Bayes' theorem, the conditional dependencies among symptoms should not be overlooked, and work is now going forward on the effects on model performance of clustering data into conditionally independent complexes. Finally, the study used a limited number of symptoms because of the limited data available for the Florida patients. It is believed that diagnostic accuracy would be increased by considering more patient information. For the actuarial models, however, this would require the costly and time-consuming reabstracting of the medical records. A word of caution is needed here. Research has shown that techniques used to obtain subjective probability estimates have a great effect on the quality of the estimates given. People tend to overestimate small likelihood ratios, for instance, and to underestimate large ones [19]. Moreover, there is a particular tendency toward this phenomenon if the estimators are asked to consider simultaneously the impact of multiple data [20]. The method of obtaining the estimates influences the magnitude of these errors [9,10]. Researchers interested in using subjective estimates should become familiar with the relevant behavioral sciences literature on decision theory. Acknowledgnents. The authors express their appreciation to Drs. F. Larson and E. Albright of the University of Wisconsin Medical School, who have advised on the development and implementation of nearly all components of this research study; Dr. C. Williams and Mr. L. T. Fitzgerald of the University of Florida Medical School, who have provided their data and insights into the diagnostic problem; and Mr. R. Ludke of the University of Wisconsin departmnent of industrial engineering, who has reviewed the article and suggested several improvements.

212

Health Services Research

SUBJECTIvE BAYESIAN DIAGNOSIS REFERENCES 1. Woodbury, M. A. Inapplicabilities of Bayes' theorem to diagnosis. Proceedings of the Fifth International Conference on Medical Electronics, Liege (Belgium), July 22-26, 1963. 2. Custafson, D. H., W. Edwards, L. D. Phillips, and W. V. Slack. Subjective probabilities in medical diagnosis. IEEE Trans. 10:61 September 1969. 3. Bruce, R. A. and S. R. Yamall. Computer-aided diagnosis of cardiovascular disorders. J. Chron. Dis. 19:473, 1966. 4. Slack, W. V., C. P. Hicks, C. E. Reed, L. J. Van Cura, and W. F. Carr. A computerbased medical history system. New Eng. J. Med. 274:194 Jan. 27, 1966. 5. Slack, W. V., B. M. Peckham, L. J. Van Cura, and W. F. Carr. A computer-based physical examination system. JAMA 200:224 Apr. 17, 1967. 6. Weed, L. L. Medical Records, Medical Education, and Patient Care. Cleveland: Case Western Reserve University Press, 1969. 7. Overall, J. E. and C. M. WVilliams. Conditional probability program for diagnosis of thyroid function. JAMA 183:307 Feb. 2, 1963. 8. Fitzgerald, L. T. and C. M. Williams. Mlodified program for computer diagnosis of thyroid disease. Radiology 82:237 Feb. 1964. 9. Shukla, R. K. The Effects of Training and Group Processes on Subjective Likelihood Ratio Estimation. Master's thesis, University of Wisconsin, 1970. 10. Custafson, D. H., R. K. Shukla, A. L. Delbecq, and C. W. Walster. A Comparative Study of Differences in Subjective Likelihood Estimates Made by Individuals, Interacting Groups, Delphi Groups, and Nominal Groups. Technical Report No. 6, Medical Decisionmaking Research Project, University of Wisconsin, 1971. 11. Sterling, T. D., J. Nickson, and S. V. Pollack. Is medical diagnosis a general computer problem? JAMA 198:281 Oct. 17, 1966. 12. Gorry, C. A. and C. 0. Barnett. Sequential diagnosis by computer. JAMA 205:849 Sept. 16, 1968. 13. Gustafson, D. H., I. Feller, K. Cranett, and D. C. Holloway. A Decision Theory Approach to Measuring Severity in lUness. Technical Report No. 9, Medical Decisionmaking Research Project, University of Wisconsin, 1971. 14. Holloway, D. C. The Development and Testing of a Model for Predicting Physicians' Evaluations of Health Status. Doctoral dissertation, University of Wisconsin, 1971. 15. Huber, C. P., R. Daneshgar, and D. Ford. An empirical comparison of five utility models for predicting job preferences. Organ. Behav. & Human Perform. 6:267 May 1971. 16. Custafson, D. H., C. K. Pai, and C. C. Kramer. A "weighted aggregate" approach to R&D project selection. AIIE Trans. 3:22 March 1971. 17. Crooks, J., I. P. C. Murray, and E. J. Wayne. Statistical methods applied to the clinical diagnosis of thyrotoxicosis. Quart. J. Med. 28:211 April 1959. 18. Billewicz, W. Z., R. S. Chapman, J. Crooks, M. E. Day, J. Cossage, E. Wayne, and J. A. Young. Statistical methods applied to the diagnosis of hypothyroidism. Quart. J. Med. 38:255 April 1969. 19. Peterson, C. R. and L. R. Beach. Man as an intuitive statistician. Psychol. Bull. 68:29, 1967. 20. Kleinmuntz, B. (ed.). Formal Representation of Human Judgment. New York: Wiley, 1968.

Fall 1971

213