EVALUABILITY ASSESSMENT A Catalyst for Program Change ... - STES

10.1177/0163278703252264 Evaluation & the Health Professions / June 2003 Thurston et al. / EVALUABILITY ASSESSMENT

ARTICLE Using a local cross-cultural health service program as a framework, the authors describe the process of an evaluability assessment (EA) and illustrate how it can be a catalyst for program change. An EA is a process that improves evaluation. The key product was a logic model, which traces the links between objectives, activities, and outcomes. Four key insights emerged. First, the distinction of who was included and excluded in the target population, originally ambiguous, was clearly defined. Second, through the development of the logic model, staff members were able to analyze their goals and assumptions and critically explore possible gaps between expected outcomes and activities. Third, the EA enabled reflection on and clarification of both process and outcome measures. Finally, global goals were pared down to better match the project capacity. Developing an evaluability assessment was a cost-effective way to collaborate with staff to develop a clearer, more evaluable project. Keywords: evaluability; mental health; cultural competence; immigrant

EVALUABILITY ASSESSMENT A Catalyst for Program Change and Improvement WILFREDA E. THURSTON JENNIFER GRAHAM University of Calgary

JENNIFER HATFIELD Hatfield Consulting Group, Calgary, Alberta

AUTHORS’ NOTE: We would like to thank the staff and clients of the Calgary CrossCultural Mental Health Consultation Project. Several colleagues provided feedback and asked helpful questions throughout the evaluability assessment. We thank Kathy Dirk for copyediting. Finally, we thank the anonymous reviewers and the editor for very helpful suggestions. Correspondence may be sent to Dr. Wilfreda E. Thurston, Associate Professor, Department of Community Health Sciences, Faculty of Medicine, University of Calgary, 3330 Hospital Dr. NW, Calgary, Alberta, Canada, T2N 4N1; e-mail: thurston@ ucalgary.ca.

EVALUATION & THE HEALTH PROFESSIONS, Vol. 26 No. 2, June 2003 206-221 DOI: 10.1177/0163278703252264 © 2003 Sage Publications

206

Thurston et al. / EVALUABILITY ASSESSMENT

207

valuability assessment (EA) is not new. Authors have described it as “a set of procedures for planning evaluations so that stakeholders’ interests are taken into account in order to maximize the utility of the evaluation” (Rossi & Freeman, 1993, p. 104; Wholey, 1977). Early writers on health program evaluation recommended “a rapid feedback evaluation” to determine the evaluability of a program (Horst, Nay, Scanlon, & Wholey, 1979). It has been argued that this body of work needs to be rejuvenated and that the evaluation process should start with the program planning process (Thurston & Potvin, 2002). The desired products of an EA are a thorough description of the program, the key questions to be addressed by the evaluation, an evaluation plan, and an agreement among the stakeholders on all of these (Rutman, 1977; Wholey, 1977). In the best of worlds, an EA would be conducted as a parallel process to program planning from the proposal writing stage to the end of the program. In this way, the two processes of EA and program planning would be linked and would inform each other (Thurston & Potvin, 2002). One of the aims of EA is to prevent “useless evaluation attempts” (Horst et al., 1979), but as this article will show, program improvement may be the most significant outcome.

E

BACKGROUND

The Calgary Cross-Cultural Mental Health Consultation Project (CMHP) was established to increase access to mental health services (particularly upstream, nonacute services) for immigrants and to increase the cultural competence of mental health practitioners in Calgary. The hospital that the CMHP originated from is located in a part of the city known to have a high proportion of residents who recently immigrated to Canada; hence, this hospital has sponsored a number of initiatives regarding ethnocultural diversity in that setting. The CMHP complements and builds upon an earlier pilot project, the Multicultural Awareness Program (MAP), initiated at the same hospital between 1999 and 2000. The MAP focused on improving access to and delivery of culturally competent care to immigrant populations across a wide range of health care services in the Northeast only, constituting a model for the CMHP. The CMHP focuses specifically on increasing awareness and beginning the process of providing

208

Evaluation & the Health Professions / June 2003

culturally competent mental health service across the region. Both projects were stimulated by a series of needs assessments conducted regionally that showed that immigrants tended to underutilize services, making use of acute care services in emergency departments rather than earlier preventative or community services (V. Wasil, personal communication, February 2002). In the MAP pilot project, a cultural awareness model was used as a point of entry. Planners believed that awareness activities would be nonthreatening and would build trust and buy-in so that future projects could address more critical thinking and skill development. More recently, the CMHP has integrated a more complex model of cultural competency into its project philosophy. Whereas formerly the focus was on increasing knowledge that was, in turn, believed to increase sensitivity (Chin, 1999; Hoang & Erickson, 1985; Welch, 1997), the focus has evolved to incorporate knowledge, attitudes, and skill development for culturally competent service delivery (Chin, 1999; Welch, 1997). Cultural awareness operates on the premise that understanding cultural norms of people from specific ethnocultural groups would enable providers to be more sensitive to clients from a particular background (Hoang & Erickson, 1985; Ogunranti, 1995). Sensitivity and awareness were based on the notion that “doctors who have alien [italics added] patients need to know about alien [italics added] cultures” (Ogunranti, 1995, p. 67). Several assumptions underlie the cultural awareness model, which can lead to an artificial sense of confidence and sensitivity but which may actually be counterproductive.

METHOD

To meet program expected outcomes efficiently, the CHMP commissioned an evaluation. The purpose of conducting the EA with the CHMP was to increase the utility of the evaluation. A framework was used for conducting the EA developed by Thurston (1991) and based on the work of Wholey (1977), Rutman (1977), and Rossi and Freeman (1989). The framework includes seven elements: (a) bounding


209

the program by identifying goals, objectives, and activities that make up the program; (b) reviewing documents; (c) modeling resource inputs, intended program activities, intended impacts, and assumed causal links; (d) scouting the program or getting a firsthand look at how it operates; (e) developing an evaluable program model; (f) identifying evaluation users and other key stakeholders; and (g) achieving agreement to proceed on an evaluation. Several methods were used to gather data. The written proposal for the CMHP as well as any reports from CMHP and the earlier MAP pilot were collected and reviewed. These documents provided a framework for subsequent interviews and guided the development of a first draft logic model. A logic model is a chart that traces the flow of reasoning linking goals to activities to outcomes (both expected and unexpected) in order to identify any discrepancies or gaps (Mohr, 1995; Rossi & Freeman, 1993; Unrau, 1993). By seeking multiple viewpoints, the authors attempted to determine both expected and unexpected outcomes from project activities and the priorities of different stakeholders. Semistructured interviews were conducted (and tape-recorded) with all project staff. These interviews enabled the authors to understand how the program implementers viewed and prioritized the activities, goals, and outcomes of their program. Small focus group sessions, lasting over an hour, were held with clients from the previous MAP pilot project to gain insight into program activities and outcomes, as understood by program clients. Focus groups with health practitioners were difficult to schedule due to the busy timetables of these individuals; therefore, group sizes were reduced to approximately three participants. Throughout the data collection, researchers were in frequent contact with CMHP staff to ask for clarification and elaboration. The authors met frequently to discuss findings and to decide on subsequent steps in gathering information from and/or reporting information to key stakeholders. In this way, the EA was iterative. This iterative process allowed project staff to verify the interpretations made, thus increasing validity of the data. Two of the authors worked part-time for approximately one and one-half months to collect all the data necessary for the EA.

210


RESULTS BOUNDING THE PROGRAM

The interviews, focus groups, and document review provided the evaluators with a solid description of what the goals and objectives and the planned activities of the project were. This enabled the evaluators to work with the staff to try to bound the project or to delimit the boundaries of the sphere of influence of the CMHP. Although program staff kept in mind the overall goal of creating an accessible, culturally competent mental health care system, close critical examination during the bounding of the program led to refocusing on a smaller scope for the project. Bounding the program required a balance of the tension between the capacity of the small program (small number of staff and budget) and the desire to achieve the overall goal. The CHMP has two distinct branches of activities, those targeting recent immigrants and those targeting mental health practitioners. The CMHP was also intended to serve as a means of linking the mental health sector and another local initiative, the Culturally Competent Professionals Network. IDENTIFYING KEY STAKEHOLDERS

One of the main elements of an EA is identifying the people who have a central role to play in the program and who may use a future evaluation in making decisions about the program. There were four categories of key stakeholders involved with the CMHP: the project staff, external support agencies, people in the mental health sector, and immigrant populations. One full-time and two part-time staff of the Calgary Health Region (CHR) were running the CMHP, and a research consultant was contracted to assist in the development of an evaluation and monitoring framework. The full-time staff member was responsible for planning of initiatives, facilitating workshops, and serving as a liaison between practitioners and immigrant service organizations. The part-time staff members provided administrative and program support. External support came from funding organizations, political leaders who supported the project through policy, other regional health authorities, and other mental health projects. The mental health sector,


211

for the purposes of this project, was comprised of mental health practitioners, volunteers, and interpreters within the Calgary Health Region. The immigrant population was represented by the Culturally Competent Professionals Network, a group of visible minority physicians dedicated to providing culturally competent care for the diverse populations of Calgary; community organizations with culturally competent strategies; and immigrant-serving agencies. DEVELOPING THE LOGIC MODEL

Various staff, because of their discipline and focus, identified components of the logic model that others may have missed or stated differently. As they were being identified, components of the model were linked together to follow the logic of the project developers. The logic model identified several areas in which there were possible gaps. These are identified in Table 1. The logic model was combined with a summary of the background of the CMHP, and this was presented to the project staff for their feedback. The staff and evaluators used the logic model as a tool for looking at the program with a critical lens. It allowed them to deconstruct program assumptions, recommend program changes, and revise problematic objectives. Uncovering and articulating assumptions helped the staff to explore the connections between planned activities and program objectives. For example, to achieve a culturally competent mental health system, CMHP staff identified the need to eliminate barriers to access. The program, however, had up to that point focused on increasing practitioner knowledge and skills (limited to skills in working with a translator), clearly not adequate in achieving systemwide cultural competence (Table 1). As a result of discussions, it was decided that workshops should at least include opportunities for more critical self-reflection by practitioners on their own attitudes. LESSONS LEARNED DURING THE EA

The EA acted as a catalyst for change for the CMHP. There were four key insights gained through doing the EA: (1) target clients of the program were not clearly defined; (2) the logic model contained gaps between planned activities and expected outcomes, and underlying assumptions required review and clear articulation; (3) a shift from a

212 TABLE 1

Analysis of the Causal Model Reflected in Proposed Activities Prior to the EA Objective/Activity

a

Purpose

b

Assumptions Made in Linking the Purpose to the Planned Activity

Establishing mental health service links

To increase awareness among practitioners of the existence of the CMHP

If practitioners know about the services of the CMHP they will make use of them. There exists a lack of dialogue and support for mental health workers (and providing that support will lead to better service).

Education and consultation with/for health providers

To assess current levels of awareness/ Through teaching the front line workers, the mental health system knowledge and develop a curriculum and will become more culturally competent. Multicultural education modules for training complements the client-centered approach, a policy adopted by the region.

Culturally competent mental health services

To develop a culturally competent mental health service through informal strategic interventions and teaching modules

Resistance to change by practitioners has not been planned for at this point. Change can be driven up from the front line providers. An initiative sanctioned by administration will effect change in the system (there is a role for administration in the sustainability of the initiative)

Establishing a community network with organizations providing services for immigrants

To establish contacts with groups and organizations working with immigrants in Calgary

Community organizations will be a valuable source of information about immigrant clients for mental health practitioners.

Education and consultation with communities

To educate different ethnocultural groups about the kinds of mental services in Calgary and how to access them and to provide a vehicle for feedback on the mental health system from ethnocultural groups.

If immigrants know about services and how to access them they will make more timely use of services (i.e., lack of knowledge is a key barrier to accessing services). By targeting four ethnocultural groups—South Asian, Chinese, Southeast Asian, and Newcomer— the majority of the immigrant population currently underutilizing the mental health system will be reached. Recent immigrants will be able to articulate their needs in a group setting in English.

Promotion of mental health services and prevention of mental health crises

To educate immigrant populations about prevention of mental health crises and to promote noncrisis services

The current service delivery mechanisms will be appropriate for immigrants (there is a recognition that services must adapt to better meet needs, and the project is seeking to identify these needs). Preventing/averting crises is a better use of resources and easier on individuals and families. Lack of knowledge is effecting utilization of services.

Improved confidence in mental health services

To increase the level of confidence of immigrant families in the competence and effectiveness of the mental health services available in Calgary

Increased knowledge and increasingly culturally competent mental health practitioners will lead to increased confidence in the services by immigrants. Increased confidence will lead to increased (and more timely) utilization.

NOTE: EA = evaluability assessment; CMHP = Calgary Cross-Cultural Mental Health Consultation Project. a. The objective or activity stated in the project documentation. Objectives were similar to activities; therefore, the first column contains only the objectives as originally stated. b. The purpose or expected outcome of the activity.

213

214


process orientation to an outcome orientation was needed; and (4) broad goals needed to be pared down to make them more congruent with available resources. 1. Defining the Target Clients

The project staff revealed that they were interested in increasing the level of awareness and skills of practitioners to enable them to better deal with diversity; however, the project goals and mission statement specified a focus on dealing with immigrants. Project staff eventually identified their target population as immigrants, and after further discussion, the target population was focused even more to recent immigrants, particularly from non-English speaking countries. 2. Gaps in the Logic Model

For the CMHP, evaluators and staff used the logic model to bring to light several gaps between planned activities and what the goals and objectives identified as expected outcomes. The project activities used a community development model to a) improve immigrant and refugee knowledge and utilization of mental health services and b) develop culturally competent mental health care providers. The planned activities, however, focused on the development of skills, knowledge, and attitudes surrounding a much broader population. Thus, although the target population specified was recent immigrants, the activities focused on visible minorities, particularly people from diverse linguistic, religious, and ethnocultural backgrounds. There was, therefore, a gap between the specified target group and the expected outcomes of planned activities. Other gaps in theory development were also identified. Table 1 illustrates some of the assumptions uncovered in the construction of the logic model. The first problematic assumption was that mental health practitioners who knew about the CMHP services would use them. This is problematic, as it may hold true only for practitioners who value cultural competence and are motivated to provide culturally competent care. Those practitioners, however, who have not


215

yet reached a level of awareness where they feel culturally competent care is necessary and who would therefore benefit greatly from the CMHP might not access the program. The EA provided a mechanism to investigate if the CMHP was making assumptions that had been found to be problematic in similar projects in other jurisdictions (e.g., a common assumption that education leads to behavior change). Behavior change is a complex process, of which knowledge is only a minor component (Crosby, DiClemente, Wingood, & Harrington, 2002; Duncan, Jones, & Moon, 1996). One common assumption made in projects with this emphasis is that if immigrants only knew about services, they would use them. Utilization of services will not increase, however, if they are not provided in a culturally competent way, transportation is impossible, or culturallybased stigma exists against using such services. Although project developers were aware of this problematic assumption, the only activities planned at this point were educational in nature. Another issue brought forward during the EA was that cultural awareness building activities, although potentially motivating and generating interest in the project, could have negative unintended consequences of stereotyping and generating negative cultural comparisons. The latter could actually act to compromise the cultural competence of individuals. Closely linked to the assumption that practitioners who know about CMHP services will use them was the final problematic assumption that needed to be critically analyzed by project staff. The assumption was that resistance to change by practitioners would be minimal and not act as a barrier to the systemic change hoped for in the project. Although this assumption would hold true for many mental health practitioners, not all practitioners will be open to cultural competency training or the idea of culturally competent care. The individuals who resist the notion of cultural competency are arguably the most important people for the project. There has been a recognition that cultural competence can be a threatening subject for some people, and cultural awareness models have been used in the pilot project as an entry point to try to increase enthusiasm and trust to lead to more analytical, skillfocused workshops.

216


3. Shifting From a Process to Outcome Orientation

A major development that came about through the EA was the realization that the project was focused on measuring processes rather than outcomes. Process-oriented indicators measure whether activities have taken place. Outcome-oriented indicators measure the effect of project activities on target groups. It is important to record processes for replicability of projects; however outcome-oriented indicators will provide information on the effectiveness of the interventions. Although the previous MAP pilot project had focused on processes (number of people trained, number of consultations, or whether activities had taken place), the staff foresaw the need for measuring outcomes to assist them in understanding the effects of their interventions. Knowledge of the outcomes of the project was sought to help in securing funding for the future. The staff also wanted information that would help them improve their project activities. During the EA, the project staff reworked their goals to focus more on expected outcomes rather than processes, for example, changes in the ways mental health practitioners use community service agencies as part of their overall treatment plan. The staff were able to clearly articulate the expected outcomes of the project activities, which led to outcome-focused objectives. 4. Realistic Goal Development

The logic model, in particular, was a useful tool for assisting project staff in paring down the global goals of the CMHP. Although the project aimed to create a culturally competent mental health system in the region, the logic model allowed the staff to see the discrepancy between planned activities, staff and other resources, and the overreaching ultimate goal of the project. With only three staff and 2 years of funding, the scope of the global goals was inappropriately large. A gap was also identified between the planned activities and the original goals that specified the expected outcomes. The project staff realized that either the activities or the goals had to change in order to bridge the gap. After the submission of a draft of the EA, the project staff began to examine the scope of the CMHP, and by the completion of the


217

EA, the goals had been pared down to better suit the resource capabilities of the project. The EA also enabled project staff to realize that whereas cultural competence training is used to develop skills and awareness for diverse working environments, the skills and attitudes necessary for working with recent immigrant clients are somewhat different. The educational component for the mental health practitioners, planned to start the following year, was based on broader models of cultural competence without identifying skills specific to practitioners working with recent immigrant clients. The project staff realized that, although improving practitioners’ skills to competently deal with immigrant clients was a starting point for increasing the cultural competence of the organization, the CMHP was not capable of dealing with such an enormous task with only one full-time program staff. At the end of the EA, project staff members were conducting a needs assessment with mental health practitioners to learn about their readiness for cultural competence training and were reviewing current literature to develop a more specific list of key skills and attitudes important in working with recent immigrants.

DISCUSSION

Mohr (1995) argued that a good analysis of the impact of a program is partially dependent on understanding the problem to be addressed, the ultimate outcomes sought, the outcomes of interest to the program, and the outcomes associated with each program activity and how these are related to the outcomes of interest. In fact, what Mohr and others (Chen, 1990; Macaskill et al., 2000; Milstein, Wetterhall, & CDC Evaluation Working Group, 2000; Rossi & Freeman, 1993; Unrau, 1993; Wimbush & Watson, 2000) required of evaluators is a good understanding of program theory. Mohr defined a program theory as that which “tells what is to be done in the program and why— what is to result from the program and how. It is, in short, a testable assertion that certain program activities and subobjectives will bring about specified results” (p. 18). Problems arise in program development and subsequently in evaluation when people fail to adequately describe the problem to be

218


addressed. This failure can lead to a number of missteps (Mohr, 1995): (a) becoming fixated on outcomes that are not related to the problem, (b) inability to see that one outcome may actually represent several problems that require alternative solutions or activities, (c) stating and therefore measuring outcomes that do not represent the real possibilities of the program, (d) not addressing the issue of how extensive a program must be to solve the stated program, and (e) losing sight altogether of the social good that was intended to result from the program. Clearly, the program theory articulated through a logic model is critical to developing the evaluation plan (Wimbush & Watson, 2000). It is also critical for utilization of evaluation findings by people outside of the program who need to know what worked, for whom, and where. The results of the evaluability assessment described here were similar to the results of a formative evaluation and hopefully will prevent the missteps described above. The EA itself and the iterative process of writing the EA acted as a catalyst for change within the CMHP. Through the building of a logic model, gaps between what the objectives would achieve and the overall goal of the project were exposed. The process of writing the EA facilitated a parallel process by project staff of tailoring objectives. The result was the ongoing examination of objectives and goals to pare them down to be realistic, achievable, and outcome based. Bounding the project, partly through identification of key stakeholders, allowed the staff of the CMHP to realize the project strengths and limitations. A snapshot of the planned activities, the way activities matched with objectives was crucial for critical analysis of the intended direction for the CMHP. Through the EA, project staff were able to more clearly articulate their target populations and expected outcomes. Measuring the effectiveness of a project is only possible when objectives are outcome based. The EA allowed CMHP project staff to better tailor their objectives to the expected outcomes. Work was ongoing at the end of this EA to develop indicators that would measure the achievement of project objectives and expected outcomes. For the CMHP, developing a logic model highlighted several important gaps between activities and anticipated outcomes, which needed to be addressed in order for the project to work toward achieving the goals stated in their project literature. By bringing to light the underlying assumptions of the CMHP, project staff were able to


219

examine the assumptions, challenge them, and make appropriate changes to the activities or goals. The link between activities and goals is bridged with assumptions that require critical analysis before program activities proceed, and the logic model was a tool that facilitated the analytical process. The need for a clearer understanding of the complex web of barriers that are restricting access to mental health services for recent immigrants was identified. There were some difficulties encountered during the EA. Communication was sometimes difficult because both program staff and evaluators had many responsibilities other than this evaluation. It may have been more effective to have regular meetings with program staff and all evaluators. Furthermore, as the EA progressed, the context changed as the health authority to which the hospital was responsible announced a new diversity initiative. This increased the desire on the part of CMHP champions that it be seen as a positive program. Among themselves, the evaluators discussed the need for sensitivity and to plan their communications with the CMHP so as to minimize the chances that staff commitment, skills, and values were being questioned. The CMHP, although based on an earlier pilot program, was under development and changing and adapting to the environment. The EA provided valuable insight, and staff implemented changes to resolve problems making written reports somewhat redundant. The process rather than the EA report was effective, causing some to question the need for and value of a report. The evaluators limited the time spent on writing an in-depth report of the EA but saw it as a historical document that might be important for future reference. Further, the evaluators had to clarify more than once the purpose of an EA. The discourse of best practices and evidence-based medicine has heightened awareness of the need for outcomes evaluation, further obscuring the need for the type of evaluation work described here. An EA cannot assess whether a program is meeting its goals but focuses more on whether the goals are appropriate. There was disagreement among program staff and evaluators as to whether some of the gaps in logic identified during the EA warranted attention. In a climate in which there is great ambiguity in terms relating to culture, race, and ethnicity (Krieger, 1992), it is easy for program planners to overlook important differences in terms and therefore subpopulations. The evaluators questioned whether a focus on

220


immigrants was appropriate and thought this could lead mental health practitioners to confuse the needs of immigrants with those of other populations of ethnic minorities. It was not possible for evaluators and program staff to agree on all issues, however, and the evaluators took the position that they could provide an evaluation framework that program staff could use to make strategic decisions but were not in a position to insist on a given approach. In the end, the EA led to a clear definition of target clients, something that is imperative for developing activities that will meet their needs and achieve the overall goals of the project.

CONCLUSION

EA is a cost-effective strategy for organizational change. It cannot ignore the political tensions that frequently surround health programs or program evaluation and requires the same attention to human relationships as other evaluation activities. Because of its iterative nature and focus on program development, it creates a relatively safe space for the critical examination of assumptions and can operate as a catalyst for change. Whether the changes are optimal and the program is successful remains to be assessed in other evaluation activities. Sometimes the evaluators and program planners will disagree. For the relatively small cost of two part-time staff over 2 months, the project discussed in this article was able to clearly identify client groups, refocus goals and objectives, and work toward developing indicators that would measure expected outcomes rather than simply processes, all of which would result in an improved project. EA is a cost-effective solution recommended by the authors for projects in formative stages looking for assistance in direction, evaluation, and setting up a monitoring system.

REFERENCES Chen, H. T. (1990). Theory-driven evaluation. Newbury Park, CA: Sage. Chin, J. L. (1999). Cultural competence and health care in Massachusetts: Where are we? Where should we be? (Issue Brief 5). Massachusetts: Massachusetts Health Policy Forum. Retrieved Feb. 12, 2002, from www.ihp.brandeis.edu/mhpf/pages/publications.htm


221

Crosby, R. A., DiClemente, R. J., Wingood, G. M., & Harrington, K. (2002). HIV/STD prevention benefits of living in supportive families: A prospective analysis of high risk AfricanAmerican female teens. American Journal of Health Promotion, 16, 142-145. Duncan, C., Jones, K., & Moon, G. (1996). Health-related behavior in context: A multilevel modeling approach. Social Science & Medicine, 37, 725-733. Hoang, G. N., & Erickson, R. V. (1985). Cultural barriers to effective medical care among Indochinese patients. Annual Review of Medicine, 36, 229-239. Horst, P., Nay, J. N., Scanlon, J. W., & Wholey, J. S. (1979). Program management and the federal evaluator. In H. C. Schulberg & F. Baker (Eds.), Program evaluation in the health fields (Vol. 2, pp. 73-109). New York: Human Sciences Press. Krieger, N. (1992, Winter). The making of public health data: Paradigms, politics, and policy. Journal of Public Health Policy, 412-427. Macaskill, L., Dwyer, J. J. M., Uetrecht, C., Dombrow, C., Crompton, R., Wilck, B., et al. (2000). An evaluability assessment to develop a restaurant health promotion program in Canada. Health Promotion International, 15, 57-69. Milstein, B., Wetterhall, S., & Centers for Disease Control Evaluation Working Group. (2000). A framework featuring steps and standards for program evaluation. Health Promotion Practice, 1, 221-228. Mohr, L. B. (1995). Impact analysis for program evaluation (2nd ed.). Thousand Oaks, CA: Sage. Ogunranti, J. O. (1995). Cultural and biological diversity in medical practice. World Health Forum, 16(1), 66-68. Rossi, P. H., & Freeman, H. E. (1989). Evaluation: A systematic approach. Newbury Park, CA: Sage. Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury Park, CA: Sage. Rutman, L. (1977). Formative research and program evaluability. In L.Rutman (Ed.), Evaluation research methods: A basic guide (pp. 59-71). Beverly Hills, CA: Sage. Thurston, W. E. (1991). Program planning and evaluation. Unpublished course documents, University of Calgary, Canada. Thurston, W. E., & Potvin, L. (2002). Evaluability assessment: A tool for incorporating evaluation in social change programs. Manuscript submitted for publication. Unrau, Y. (1993). A program logic model approach to conceptualizing social service programs. Canadian Journal of Program Evaluation, 8, 117-134. Welch, M. (1997). Culturally competent care for diverse populations. Retrieved February 12, 2002, from http://dgim.ucsf.edu/pods/html/divpeople.html Wholey, J. S. (1977). Evaluability assessment. In L. Rutman (Ed.), Evaluation research methods: A basic guide (pp. 41-56). Beverly Hills, CA: Sage. Wimbush, E., & Watson, J. (2000). An evaluation framework for health promotion: Theory, quality and effectiveness. Evaluation, 6, 301-321.