Mental health needs assessment: Beware of false ... - Springer Link

Mental Health Needs Assessment: Beware of False Promises David Royse, Ph.D. Kenneth Drude, Ph.D.

ABSTRACT: Needs assessment is considered by experts to be an essential part of mental health planning. Unfortunately, almost anything can pass for a needs assessment. This article briefly examines the methodological limitations of the diverse and unstandardized approaches as well as the conceptual problems associated with needs assessment. The authors suggest that needs assessments could become much more useful if policy-setting and funding bodies would more explicitly state their expectations, if there were greater use of standardized instruments, and if efforts could be directed towards the cataloging of known needs assessment instruments. Further, the state of the art would be advanced if an expert committee could be formed to develop national standards for needs assessments.

Assessment of mental health needs in Western society is said to have begun with Richard Powell's analysis of the data contained in the Case Register started in 1775. In 1810 he reported to the Royal College of Physicians that the number of cases of insanity had increased in the years between 1775 and 1809 (Schwab, 1976). Since that early effort, there have been enormous changes in what is considered mental illness, its treatment, and resources available for application to the problem. With the allocation of greater resources have come increased demands for accountability. For instance, with the passage of Public Law 94-63, the Community Mental Health Center Amendments of 1975, federally funded community mental health centers were required to evaluate their programs in terms of meeting the needs of those in their catchment areas. Public Law 94-63 also mandated that federally funded centers develop an internal program evaluation capability to address such areas as availability, accessibility and acceptability of services, patterns of use, cost of operation, quality assurance, and the impact of services upon the mental health catchment residents. Warheit, Holzer and Robbins (1979) have attributed this growing demand for accountability to the planning and evaluation deficiencies of the "New Frontier" and "Great Society" programs of the Kennedy-Johnson administrations, to greater competition for diminished funds available for public programs, and to Dr. Royse is Director, Research and Evaluation, Greene-Clinton Community Mental Health Board, 168 E. Main Street, Xenia, Ohio 45385. Dr. Drude is Assistant Professor, School of Professional Psychology, Wright-State University, Dayton, Ohio. Community Mental Health Journal Vol. 18 (2) Summer 1982 0010-3853/82/1400-0097502.75 9 1982 Human Sciences Press

97

98

Community Mental Health Journal

strong inflationary pressures in the economy. Concern with needs assessment is one way in which this demand for accountability, especially in the field of mental health, was made manifest. With the repeal of both the Mental Health Systems Act of 1980 and the Community Mental Health Centers Act, the trend for greater accountability has been significantly modified. In order to receive federal funding, state authorities under the Omnibus Budget Reconciliation Act of 1981 only need to certify "(t)hat the State has identified those populations, areas, and localities in the State with a need for mental health, alcohol abuse and alcoholism, and drug abuse services" (P.L. 97-35). Clearly, states will have much greater latitude in complying with such needs assessment requirements than earlier more explicit requirements. Needs assessment is still considered an essential part of mental health planning despite the apparent reversal in federal requirements. Needs assessments are employed by those associated with the delivery of mental health services to identify target populations and to direct programming. The concept of needs assessment can be considered analogous to the first step in the scientific method--definition of the problem. Although needs assessment has been appropriately defined as an important informational input to the planning process (Siegel, Attkisson & Cohn, 1977), the term needs assessment all too frequently appears to mislead. In part, this is due to a wide range of meanings attributed to a term with numerous operational definitions. The term implies that, through a needs assessment process, all existing needs and the extent of these needs will be determined. However, seldom have needs assessments been observed to have been so comprehensive. It is the premise of this paper that needs assessments are important in the delivery of mental health services, but that the concept of needs assessment has unclear parameters and is in need of further conceptual refinement. The authors raise concern that mental health planners and administrators too often place blind trust in untested or limited methodologies from which it is expected that, much like a crystal ball, data having profound programmatic implications will be revealed. Since it is likely that needs assessments in some form will continue to be conducted, it is the intent of the authors to review some of the basic weaknesses of needs assessments and to offer recommendations for improving the state of the art.

DEFINITION OF NEED AND ASSESSMENT One reason why conducting and using needs assessments can be disappointing is that there is no clear agreement on the two components which comprise the term: (1) what constitutes a need; and (2) what constitutes an assessment. Kahn (1969) has observed: In effect"'needs"are socialdefinitions,representinga view of what an individualor group requires in order to play a role, meet a commitment, participate adequately in a social process, retain an adequate level of energyand productivity--at a givenmomentof history. (p. 63)

David Royse and Kenneth Drude

99

Bradshaw (1977) has gone several steps further and conceptualized four types of social need: (1) normative need--being that which an expert defines as need; (2) felt need--consisting of input from an actual population as to what they feel they need; (3) expressed need--a demand for service, e.g., waiting lists for services, and finally, (4) comparative need--an inferred measure of need determined by examining the characteristics of those receiving services and then locating those characteristics in the population. Need, then, is a term without conceptual boundaries and which requires an operational definition in each usage. Assessment refers to a range of different techniques by which conditions, needs, or resources may be identified. Operationally, these vary considerably and have been conceptually classified as collectively providing different information about needs to form a comprehensive needs assessment or as conceptually independent types of needs assessment that can be utilized independently of each other (Stewart, 1979).

DIVERSE METHODOLOGIES As indicated above, practically any data collection methodology with just about any sample size can be called a needs assessment. Although the term was just beginning to come into vogue, the widely referenced report of Booz, Allen (i973) observed that "the state of the art of needs assessment and resources assessment was not generally well developed," More recently, Kimmel (1977) has critically examined the area of needs assessment and made the following observations about the available methodologies: there is nothing common or unifying about the approaches; there are no systematic procedures for relating data from one approach to the next; none of the approaches has a well developed set of techniques for analysis, models or theoretical procedures associated with them beyond commonly used statistical procedures; there are no guides for data interpretation and analysis in the context of resource allocation or priority-setting. Kimmel (1977) concluded, "At best, the approaches lead to descriptive statements of needs but none of them provides a way to 'assess' them."

METHODOLOGICAL LIMITATIONS Because of inherent weaknesses in all needs assessment methods, experts have not advocated the use of a single approach, but recommend the use of multiple approaches or "convergent analysis" (Milord, 1976; Siegel et al., 1977; Warheit, Bell & Schwab, 1977). As recently as 1979 Warheit and colleagues noted: As community mental health centers and other human service agencies have attempted to respond to these mandates, they have been hindered by an absence of trained personnel and financial resources. Perhaps more importantly, they have not had methodologies whose validity and reliability have beenempirically demonstrated. No single method has emerged as the best one to assess needs. (p. 95)

100


Some of the inherent shortcomings of the major needs assessment approaches are reviewed below. These may not always be recognized by users of needs assessment data. Impressionistic or subjective approaches include community forums, public hearings, nominal groups, or key informants. This form of needs assessment invites individuals to present their opinions about various needs or problems. A major conceptual weakness is that the individuals selected may not be representative of the community or might not be familiar with community needs. Numerically small, but vocal, groups can dominate meetings and thus influence the findings. Finally, no statistical probability statement about needs can be garnered from this data. Even interviewing or surveying of various "knowledgeable" service providers or experts in the community suffers from a lack of scientific rigor, with the resultant data largely a function of those selected for the needs assessment. For instance, a service provider might be quite familiar with the needs of his or her clientele, but could be quite ignorant of the needs of that portion of the population which" does not seek services at his or her agency. Dentists, if interviewed, might be well aware of the dental needs of the community but unable to give more than a wild guess as to the number of persons in need of mental health services. Community surveys and epidemiological studies represent a second major needs assessment approach. Community or household surveys are popular because of the range of questions which can be asked and because interviews with 50 to 100 respondents can be obtained relatively easily. Unfortunately, many of these community surveys are hastily conceived and executed. When sampling procedures are not based on randomization, no statistical probability statement about needs in the community can be made. Epidemiological surveys are often recognized as preferable to other forms of needs assessment because much more confidence can be placed in the data, due to greater attention to sampling procedure and to validity and reliability concerns about the data obtained. Statistical probability statements can be made about the findings. It is not a technique, however, that is readily available to many agencies because of the resources it usually requires. Trained interviewers are needed as well as the skills of one knowledgeable of survey methods, sample design, questionnaire design, and data analysis. If a large survey is contemplated, computer processing almost becomes mandatory. Even if funding for a survey form of needs assessment is adequate, the significance of the products of these surveys may be somewhat perplexing. In a review of 44 major psychiatric epidemological studies, Dohrenwend and Dohrenwend (1969) found that the rates of mental disorder or psychiatric impairment reported in these studies ranged from less than 1% to over 60%. Epidemiologists have encountered difficulty in measuring the extent of actual mental disorder because of the lack of agreed upon operational definitions of mental illness. Other problems associated with survey approaches include: respondents being hesitant to acknowledge symptoms or problems which might indicate emotional disorder, problems associated with accurate recall of events


101

and memory span, and the problem of the honesty of the person providing the information. Glass (1979) has identified a number of problems related with the content of survey instruments ranging from the placement of questions, the type of information requested, to the wording of questions. Social indicator approaches use descriptive statistics or secondary data which are already in existence. These approaches can refer to a range from very crude techniques, such as ranking census tracts by a risk index, to highly sophisticated designs using canonical factor regression or multiple regression equations. Social indicator approaches are based on the assumption that mental health needs can be inferred from variables which have been found to be highly correlated with need for mental health services. Commonly the principal categories of such data are measures of social problems such as unemployment, poverty, crime rates, suicide and sociodemographic variables such as female heads of households. While census data are generally considered to be reliable data, one problem with using secondary data sources is that their reliability is not usually known. Another problem is that it is impossible to draw inferences at the individual level from data observed with census tract, enumeration district or county as the unit of study. A larger conceptual problem is that these indicators are only indirect measures of the need for mental health services. Formulas or need projections based on demographic data, from one population may not be easily generalized to other populations.

OTHER ISSUES: CONSTRUCT VALIDITY AND MICROLEVEL FOCUS As emphasized implicitly throughout this paper, the construct validity of the concept of needs assessment is a fundamental issue that refers not only to the validity of the concept itself, but also to the meaning of the results obtained from the techniques used. Given the variety of operational definitions used, it is understandable that each technique will produce somewhat different data with different implications. If multiple approaches are utilized, it is conceivable that investigators may find themselves with several sets of data that may not clearly translate into needed services. At times these data may even be in conflict. Although this may produce confusion, it is a predictable result of the broad boundaries that characterize the state of the art of needs assessment. Resolution of seemingly conflicting needs assessment findings places mental health planners into a political realm in which the realities of available funding sources and the biases of agency administrators and investigators become important factors. Determining the priorities of specific needs is a value-laden process that is not automatically evident from needs assessment data. Indeed, even the selection of a methodology has political implications. For instance, a community forum approach might be considered politically expedient because it suggests "going to the people" and the importance of a grassroots perspective versus expert opinion derived from service utilization or secondary data sources.

102


Another major weakness with the current use of needs assessments is that the vast majority of them are being conducted at a microlevel. This leads to multiple needs assessments within a given geographic area with potentially little ability to integrate or generalize findings from each. Without common agreement of what methods to use or standardized instruments and procedures, several agencies within a given area may produce only a disjointed picture of area or macrolevel needs. It is unfortunate that while federal health planning and accountability efforts have led to regional planning (i.e., health systems agencies) and regional service review (i.e., professional standards review organizations), there has not been a comparable development for mental health. Federal and state legislation for mental health should more explicitly promote comprehensive and integrative approaches to the identification of mental health service needs at the regional level or multiple mental health service areas. Although federal and state healthplanning agencies have technically been responsible for physical and mental health needs assessment, mental health has tended to get much less emphasis than physical health. At the time this article is written (Fall, 1981), the federal health planning agency structures appear headed for significantly reduced roles. Dismantling federally funded health systems agencies will result in greater dependency upon states to require and organize needs assessments in a coherent and integrated manner. At a time when resources for mental health services grow increasingly sparse, it is even more important that coordinated and integrated needs assessment efforts be encouraged.

RECOMMENDATIONS Since each needs assessment method has its own unique limitations (as well as advantages), investigators are cautioned to be aware of them before, during, and after their use. Needs assessments are not the panacea of planning nor can they "tell" mental health agencies what services or programs to provide. Typically needs assessments provide some information about what one or more groups of people feel to be the need for, or what can be inferred about their need for mental health services. The type of data generally provided from needs assessments does not allow administrators and directors of programs to decide whether to cut the resources of one program by 10 % and to increase the resources of another by 10 %. The state of the art is such that needs assessments provide very little practical information which can be used for administering ongoing programs. This situation is primarily due to conceptual problems with needs assessments which has led to the common acceptance, by those requiring them, of diverse and unstandardized approaches. Currently, one can interview a sample of people before they go into the supermarket, talk to a few knowledgeable service providers, randomly select persons from a phone directory to interview, look at service utilization statistics, or convene public forums; any of the various approaches can be used to justify


103

services or the allocation of resources. All methodologies appear equivalent in the eyes of those mandating needs assessments. How can needs assessments be made more meaningful? There are steps which can be taken immediately and there are measures which will require a longer period for implementation. First of all, it is time that all those currently conducting and funding needs assessments acknowledge that the term is fraught with ambiguity and that practically any data collection methodology of any sample size will usually suffice. This should prepare the way for the National Institute of Mental Health (NIMH), the Joint Commission on Hospital Accreditation (JCAH), state departments of mental health, and other policysetting bodies to define more explicitly their expectations in regard to needs assessments. It would not, for instance, appear unreasonable for these bodies to recommend one or two methodologies over the others, to provide guidelines for appropriate sample sizes, or to recommend standardized items or instruments which could become part of the needs assessment effort. As an interim step, those planning a needs assessment should not attempt to design new instruments, but should use standardized instruments already available. The use of standardized instruments facilitates the needs assessment process in that time is not required for instrument development or psychometric assessment. Further, standardized instruments allow results to be compared across studies or across subgroups (e.g. urban-rural comparisons). Not only would the data produced from the use of these instruments have more value for planners and administrators at the microlevel, but comparable data generated from other geographic areas would make an important contribution to researchers examining the nation's mental health at a macrolevel. There is no shortage of standardized instruments which could be employed as part of a mental health needs assessment (Chun, Cobb, & French, 1975, Comrey, Backer, & Glaser, 1973). However, some of the available instruments are much better than others. An instrument developed by the Center for the Study of Aging and Human Development at Duke University is one which seems to have exceptional promise because it is comprehensive, multidimensional, provides information about functional status, usage of 24 generic services, and is readily available. In addition, there is considerable evidence of its psychometric adequacy (Fillenbaum, 1978; Fillenbaum & Smyer, 1980). The Older American Resources and Services Questionnaire (OARS) provides information on the functional status of older adults in five domains: (1) social resources, (2) economic activities, (3) mental health, (4) physical health, and (5) activities of daily living. Concerned with the whole person rather than a single area of functioning, the instrument yields summary assessments of functional status for each of the five dimensions. Each summary score lies along a six-point continuum from "excellent functioning" to "totally impaired." While up to this point the OARS instrument appears to have been used almost entirely with older adult populations, there is no reason why this instrument could not be administered to other populations, either in its entirety or in part. Other needs assessment instruments, not designed for such a specific population

104


and with substantial data bases, are also available. The Florida Health and Family Life Instrument, designed to permit judgments about the health needs and kinds of services being utilized by communities, is such an example. This instrument, which has been used with well over 5,000 respondents, obtains information in three main areas: (1) social psychiatric impairment, (2) general health, and (3) types and amounts of health and welfare services being used by the population. Additionally, extensive demographic information is gathered on the respondent so that needs and service patterns can be analyzed by personal, familial and social characteristics (Warheit et al., 1977). A modified version of this instrument has been recently used with probability samples in over 20 counties in Ohio. Considerable time, talent and energy are spent each year by persons attempting to design needs asessments which often have many of the same features as found in these instruments. Usage of these or other standardized instruments would make for much more efficient use of scarse resources by eliminating duplication of effort, and quite possibly provide more usable and interpretable data. Utilization of better instruments would be encouraged if national professional organizations in the field (such as the American Psychological Association, the National Council of Community Mental Health Centers, NIMH, etc.) could provide the leadership to bring about or jointly sponsor a collection of needs assessment instruments. The cataloging of these instruments would be most useful for the mental health planner or practitioner charged with the responsibility for designing a needs assessment. The catalog could list all known applications of the instrument thus providing sources of data for comparison purposes, as well as information on its reliability and validity. A good example of an organization which has done this s0rt of thing is the National Institute of Drug Abuse with its 1976 publication of the Drug Abuse Instrument Handbook which summarizes 40 instruments used in drug research and epidemiological surveys. Without such a reference tool for needs assessment instruments, much duplication of effort, waste of public dollars and loss of comparability of data is produced. Regardless of the group which assumes the responsibility for cataloging needs assessment instruments, mental health planners, administrators and practitioners should still urge their professional associations and organizations such as JCAH and NIMH to establish an expert committee to prepare needs assessment standards. This committee could be similar to the one supported by the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education in publishing in 1966 Standards for Educational and Psychological Tests and Manuals. After reviewing the collected needs assessments, this expert committee would begin the task of preparing standards. It might, for instance, decide to scrap the broad term needs assessment, or at least to distinguish among needs assessments which have the potential to make statements about their reliability and validity, and those without such capability.


105

To assist them further in this difficult task of creating standards, this committee may wish to develop a "grand" needs assessment design. What is proposed is that systematic research be conducted to evaluate a number of approaches simultaneously in a given area. If funding could be secured and a suitable catchment located, it could then be empirically demonstrated which of the approaches lead the investigators to the same conclusions about the extent of needs in that catchment. It is conceivable that an expert committee analyzing the data from such research would be led to disallow certain current needs assessment approaches as valid methodologies. While there have been a few efforts which have analyzed data obtained from two different approaches (Bloom, 1975; Zautra & Simons, 1978; Warheit et al., 1979), there have been no reports of the extensive type of research as has been proposed. This kind of empirical test is crucially needed so that judgments about the utility and validity of the diverse needs assessment approaches can be made. This information in itself would not be sufficient for the development of needs assessment standards, but is necessary to advance the state of the art. Finally, until standards are developed, reports of needs assessments should clearly state their limitations and the tentative nature of their findings as well as such psychometric information as is available on the instrument. In this way legislators, public officials, citizens, administrators and others interested in reports of needs assessments will become aware of the caution required in interpreting needs assessments data. As the larger community becomes aware of the inadequacies of present day needs assessments, more effort will be directed towards their refinement and improvement. tt is reasonable to expect that a good bit of the funds allocated to the planning and conduct of needs assessments will be spent unwisely until the field of needs assessment becomes conceptually more sophisticated, and greater methodological rigor is required. These writers believe that the active pursuit of the recommendations presented will bring about the changes needed to deal with the methodological and conceptual problems with needs assessments described in this paper.

REFERENCES Bloom, B. Changing patterns of psychiatric care. New York: Behavioral Publications, 1975. Booz, Allen Public Administration, Inc. Assessing social service needs and resources: An implementation manual. Washington, D.C.: Social and Rehabilitation Service, 1973. Bradshaw, J. The concept of social need. In N. Gilbert & H. Specht, (Eds.), Planning for social welfare: Issues, models and tasks. Englewood Cliffs, New Jersey: Prentice Hall, 1977. Chun, T., Cobb, S., & French, J.R.P. Measures for psychological assessment: A guide to 3,000 original sources and their applications. Ann Arbor, Michigan: Survey Research Center, 1975. Comrey, A.L., Backer, T.E., & Glaser, E.M. A source book for mental health measures. Los Angeles: Human Interaction Research Institute, 1973. Dohrenwend, B.D., & Dohrenwend, B.P. (Eds.), Social status and psychological disorder: A causal inquiry. New York: John Wiley & Sons, 1969.

106


Fillenbaum, G.G. Validity and reliability of the multidimensional functional assessment questionnaire. In E. Pfeiffer (Ed.), Multidimensional functional assessment: The OARS methodology. Durham, North Carolina: Center for the Study of Aging and Human Development, 1978. Fillenbaum, G.G., & Smyer, M.A. The development, validity and reliability of the OARS multidimensional functional assessment questionnaire. A manuscript available from the Center for the Study of Aging and Human Development. Durham, North Carolina, 1980. Glass, J.J. Human need assessment surveys: Selected methodological problems, Journal of Health and Human Resources Administration, 1979, February, 346-363. Kahn, A.J, Theory and practice of social planning. New York: Russell Sage Foundation, 1969. Kimmel, W.A. Needs assessment: A critical perspective, Washington, D.C.: U.S. Government Printing Office, 1977. Milord, J.T. Human service needs assessment: Three nonepidemiological approaches. Psychologie Canadienne, 1976, 17(4), 260-269~ Omnibus Budget Reconciliation Act of 1981 (PL 97-35), Title IX, Subtitle A, Part B, Section 1915 (c) (10). Schwab, J. Identifying and assessing need: A synergism of social forces, In R.A. Bell, M. Sundel, J.R. Aponte & S.A. Murrell (Eds.), Proceedings of the Louisville national conference on needs assessment in health and human seroices. Louisville, Kentucky: University of Louisville, 1976. Siegel, L.M., Attkisson, C.C. & Cohn, A.H. Mental health needs assessment: Strategies and techniques. In W.A. Hargreaves, C.C. Attkisson, L.M. Siegel, M.H. McIntyre & J.E. Sorensen, (Eds.), Resource materials for community mental health program evaluation. Washington, D.C.: U.S. Government Printing Office, 1977. Stewart, R. The nature of needs assessment in community mental health. Community Mental Health Journal, 1979, 15 (4), 287-295. Warheit, G.J., Bell, R.A., & Schwab, J.J. Needs ass~: -~ approaches: Concepts and methods. WashLngton, D.C.: U.S. Government Prin~ing Office, • Warheit, G.J., Hoher, C.E., & Robbins, L. Social indicators and mental health planning: An empirical case study. Community Mental Health Journal, 1979, I5(2), 94-103. Zautra, A., & Simons, L.S. An assessment of a community's mental health needs, American Journal of Community Psychology, 1978, 6(4), 351-362.