Notes On Methodological Issues Encountered During ...

48 downloads 0 Views 41KB Size Report
major provider of resources for the development of the intervention, the Legal Services. Corporation (LSC). During the past two decades the history of LSC ...
Notes On Methodological Issues Encountered During A Field Evaluation Of A Pro Per Intervention James W. Meeker, J.D., Ph.D. Richard Utman, J.D. Department of Criminology Law & Society School of Social Ecology University of California, Irvine paper prepared for the NLADA/AARP Evaluation Meeting March 11 & 12, 2002 Washington, DC

The authors were asked to design and implement an evaluation of an ongoing intervention to increase access to justice for pro per litigants by the Legal Aid Society of Orange County (LASOC). This intervention is based on emerging technology applying an interactive web/kiosk computer system to assist pro per litigants in preparing pleadings for a limited number of legal actions. In order to increase access to the community the project calls for these kiosks to be placed not only in LASOC but also the courthouse and the community. The intervention is called I-CAN! (Interactive Computer Assisted Network) We proposed to conduct a multi-methodological evaluation based on a process evaluation of I-CAN!’s development, field interviews of the developers of the intervention, surveys of court personnel and other relevant professionals in the field that interacted with pro per users of the legal system, field observations and survey of a sample of users of the intervention, and a questionnaire that was built into the I-CAN! system filled out by users in the process of completing their pleadings. Classic evaluation/research literature cites the need to have baseline data, clearly defined goals and outcome measures, the necessity of comparison groups, as well as the need to control for threats to external and internal validity of the evaluation. When actually conducting an evaluation of an ongoing project that is in the field and evolving during its implementation, the necessary elements cited in the literature often cannot be met. The following discussion outlines some of the problems we encountered during our evaluation and are issues that are likely to be encountered by other evaluators of interventions designed to increase access to the legal system for the poor.

1

Meeker & Utman I-CAN! Observations A. Problems Of Conflicting Goals When the need for clearly defined goals is discussed, it is often assumed that different players in the decision making process of the intervention share similar goals. In fact, this is not always the case. When you have a large intervention that involves multiple players such as a legal aid organization, the court, ancillary groups to the court, and different community organizations, you increase the likelihood of different perceptions of what the goal of the intervention should be. This can be further complicated when the development of the intervention brings together highly different professional groups with different expectations of the project.

1. Conflicts With Community Organizations A primary goal of LASOC with I-CAN! was the development of an intervention that could assist many pro per users of the court. This required decisions on where to place the kiosks to be of maximum benefit to the target community of users. Clearly this required placement of the kiosks outside of the offices of LASOC. An obvious location would be the courthouses that handled the pleadings for which the various modules of ICAN! were developed. However, this creates potential conflicts with the goals of a major provider of resources for the development of the intervention, the Legal Services Corporation (LSC). During the past two decades the history of LSC funding for field legal aid organizations has been characterized by increased restrictions on who are eligible clients for federally funded legal assistance. These restrictions cover income eligibility requirements, citizenship restrictions, and types of legal assistance. While types of legal assistance are not problematic with this particular type of intervention, the other two are. When placed in a public courthouse with open access it is difficult to control who will use the services. Indeed, this issue was further complicated with LASOC making modules accessible through the Internet. While it is possible to build in qualification questions in the modules to partially meet these restrictions there is no guarantee of the validity of those responses. A direct conflict occurred when LASOC approached the public library system for location of kiosks. As a public institution, this organization took the position that all services available on their premises should be open to the public regardless of income or citizenship status. This restriction eliminates the use of built in qualification questions and represents a direct conflict in goals between LSC and the public library system. 1 1

This is also true of abuse victims’ shelters, which want the Domestic Violence module and do not screen victims for income, and the District Attorney’s office. Four of the eight I-CAN! modules assist what are primarily child support obligors, without regard to their income, and indeed, possibly, because they have an income. 2

Meeker & Utman I-CAN! Observations While these organizations clearly have conflicting goals, over time an apparent stable compromise has evolved. LASOC has developed the I-CAN! modules to target their client population and provides no additional follow up services to anyone who does not qualify for their assistance. However, for the kiosks in public areas, no restrictions have been placed on who can use them.

2. Goal Conflict Among Developers Of An Intervention One on the hallmarks of I-CAN! is the introduction of a relatively new technology to the delivery of legal services to the poor. As a consequence, this calls for the joint effort of professionals that don’t have a long established tradition of working together. This creates the potential of different goals and expectations for the project that can have problematic effects for the intervention. The development of I-CAN! required a close working relationship between a legal aid organization, poverty attorneys and information technology professionals in developing the interactive modules designed to create court pleadings. There was a fairly long learning curve of exploring potential different artificial intelligence technologies and their suitability to providing the type of legal assistance contemplated. Part of this delay was the result of imperfect communication between attorneys and information technology professionals. The legal professionals involved in the project clearly had the goal of increasing access to the legal system for the poverty community. As such, their goal was the development of a system that would be freely available to other legal aid organizations. The original information technology professional in the project saw the venture as the development of a system that would generate profits for the developer. This difference in goals created problems in terms of intellectual property ownership, lawsuits and delays in the project.

3. Conflicting Goals Between Service Providers Malcom Feely has observed that criminal courts are fragmented formal organizations with multiple groups of professionals that have different and some times conflicting objectives 2. The same can be said for civil courts, especially in terms of competing goals. What judges may want out of a particular intervention is not necessarily the same as what attorneys, clerks, and other support personnel want. Of course all of these players may have different ideas of appropriate goals than the legal aid organization trying to implement the intervention. 2

Malcom Feely, Court Reform On Trial: Why Simple Solutions Fail, A Twentieth Century Fund Report (1983).

3

Meeker & Utman I-CAN! Observations One of I-CAN!’s most used modules is the domestic violence module. It is designed to help users complete the pleadings appropriate for a temporary restraining order (TRO). In the Orange County family law courthouse, there is a nonprofit organization that has a contract with the county to assist domestic violence victims in obtaining TROs. When LASOC first proposed to install a kiosk in the courthouse, this created a potential conflict between LASOC and this organization. Indeed, if I-CAN! is perceived as meeting the same need as this organization by the county supervisors it could potentially jeopardize their contract. In order to minimize potential conflicts between the organization and LASOC, a compromise solution was worked out. The kiosk was placed in the organization’s space in the courthouse, and users are referred to it after receiving an initial orientation from the organization’s counselors. While this potential turf conflict has at least been solved for the time being, it creates potential problems with evaluating the intervention. This situation makes it difficult to disentangle intervention effects from outside organizational influences. While both organizations have a similar goal to assist with TROs, they have dissimilar goals on how best to achieve this.

B. Data Problems In order for an evaluation to be effective it must be based on valid and reliable data. To adequately assess the impact of an intervention, good baseline data is needed. In addition, since legal interventions often involve multiple players in the process, for example legal assistance providers and the court, it important to have comparable data across different organizations. Finally, it is important that evaluators and intervention developers work closely together so that internally collected data has maximum utility for the evaluator.

1. Baseline Data Classic evaluation/research literature stresses the need for good baseline data to determine the impact of an intervention. Ideally, the baseline data is created during a needs assessment study that accurately measures and defines the problem to be addressed by the intervention. This can go a long way in solving problems on what outcome measures to use and how to define what is a successful intervention. Unfortunately, this does not always occur with legal interventions and did not occur with the I-CAN! project. Part of the problem is money. It is universally recognized among legal aid providers that the current level of resources available to meet the legal needs of the poor does not come close to meeting that need 3. Given the level of need, there is strong 3

In 1993 the Project Advisory Group (PAG) estimated it would take $3.6 billion to meet the legal needs for the poor (Project Advisory Group, Inc., Equal Justice for People in Poverty: The Long-Term Goal of

4

Meeker & Utman I-CAN! Observations pressure to employ what resources are available to the delivery of legal services. Indeed, it has only been fairly recent in the history of the delivery of legal services to the poor that the need for evaluation of different interventions and that funding an intervention should set aside at least ten percent of the budget for evaluation has been recognized. While it is difficult enough to set aside money for evaluation, it is even more problematic to obtain funding to gather baseline data in the first place. The I-CAN! project was developed in an environment that recognized a high level of unmet legal need coupled with the rise of pro per users of the court system and the consequences this was having for the courts and court users. It was seen as a possible solution to part of this problem. However there were no resources available to accurately define and measure the extent of the problem prior to the intervention. There was no needs assessment to determine the number of hand written and/or incomplete pleadings, or the number of pro per users of the courts or what their expectations or outcomes were. Indeed, it was taken as a given both by the courts, legal aid providers and funding agencies that this was a problem that needed an intervention. While there were resources available to evaluate I-CAN! these resources where not available prior to the development of I-CAN! to develop baseline data and define relevant outcome measures by which to assess the intervention. This competition for resources between defining a problem versus addressing the problem is not unique to the I-CAN! project. It is not uncommon to have a situation where there is a legal intervention implemented and the evaluation comes as an after thought 4. Even when the evaluation is funded as part of the intervention, as was the case with I-CAN!, the lack of good baseline data does create problems for accurately measuring the level of impact the intervention has made.

2. Data Compatibility Problems An evaluation of a legal intervention will often rely on data from multiple sources. Often this data is not originally gathered with evaluation or alternative analytical strategies in mind. Evaluating I-CAN! has encountered two types of data compatibility problems. The first data problem encountered has been limitations in dealing with court data. Court data management systems are not designed with evaluation of legal interventions in mind. Because they are part of a larger court management system there is often limited flexibility to incorporate additional data fields to track information necessary for evaluation. Even if key decision makers in the system want to cooperate, their data systems may have limited flexibility to accommodate data collection needs. If Legal Services 14 (1993)(paper presented to the Board of Directors of the Legal Services Corporation, on file with the Legal Services Corporation. 4 See James W. Meeker, John Dombrink, and Beth Quinn "Competitive Bidding and Legal Services for the Poor: An Analysis of Scientific Evidence," Western State University Law Review, Vol. 18:611(1992)

5

Meeker & Utman I-CAN! Observations the flexibility is possible, then the issue of cost can become a barrier. If the necessity of collecting court data is not taken into consideration early in the project, then the added resources necessary to cover the additional data collection costs may not have been adequately budgeted. A second data problem encountered involves the close collaboration that is necessary between the evaluator and the technological developers. Since I-CAN! involves an interaction between computer technology and a user, it has the capability to gather evaluative information during the interactive process. Indeed, I-CAN! has a series of satisfaction measures that are incorporated into the modules. However, as currently structured, they are totaled giving global measures on user responses for individual measures. The data were not collected in such a manner to create a data matrix of individual uses and their specific responses. This limits the ability to conduct multivariate analyses to check different response patterns by user characteristics (for example, differences between men and women, ethnic groups, etc). This stresses the importance of close collaboration between the developers of the intervention and the evaluators throughout the process.

C. Design Problems To effectively evaluate a legal intervention, the evaluation design needs to be as strong as possible. One way to control for many possible threats to validity of a study is to randomly assign individuals into treatment groups and then test differences between the groups. Another way to ensure that intervention effects are not contaminated by other changes in the environment is to ensure an unambiguous start to the intervention, so that clean pre post intervention data can be gathered. Both of these design issues have been problematic with I-CAN!.

1. Random Assignment Random assignment of subjects to treatment groups is a very strong methodological tool designed to handle a number of threats to validity5. However, random assignment is not always possible in the field, especially when dealing with legal interventions. This has been well documented in the criminal law area 6 and also occurs in the civil law poverty context 7. There are a number of reasons why random assignment can be difficult in the legal intervention context, such as lack of control of user 5

See Thomas D. Cook and Donald T. Campbell, Quasi-Experimentation: Design & Analysis Issues for Field Settings Boston: Houghton Mifflin Co. (1979) 6 See Laurence H. Ross and Murray Blumenthal, “Sanction for the Drinking Driver: An Experimental Study” Journal of Legal Studies Vol 3:53 (1974), Arnie Binder and James W. Meeker "Experiments as Reforms," Journal of Criminal Justice, Vol. 16: 347 (1988). 7 See Meeker, Dombrink, & Quinn cite supra note 4

6

Meeker & Utman I-CAN! Observations involvement in the intervention, ethical problems of providing a service to some users but not others, and self-selection issues with users. I-CAN! was not designed with random assignment of subjects in mind. Indeed, random assignment of users could run counter to the major goal of increasing access for users and trying to provide relief to the court system dealing with large numbers of pro per users. While not all pro per users of the court are using I-CAN! to develop relevant pleadings, the lack of random assignment makes comparisons between I-CAN! users and nonusers problematic.

2. Evolution Of Intervention Over Time Ideally, an intervention will have a clear and unambiguous start date so that the evaluator can link changes in outcome measures with the intervention. This can become problematic with an intervention that changes over time. I-CAN! has been changing over time. Not only have there been changes in project managers and information technology professionals involved, but the scope of the project both in terms of module design, modules available, numbers of kiosks, and location has expanded during the evaluation process. While the evaluation has tried to limit the impact of these changes by focusing on a subset of modules and locations, this evolution of the intervention can influence the findings in unknown ways. This can be particularly problematic when interviewing court personnel on their interactions with the system, because the evaluators cannot be sure that the responses are in terms of the narrower focus of the evaluation or in response to the broader elements of the I-CAN! project that has changed during the evaluation.

D. Summary Evaluation in theory can be very different from evaluation in practice. When dealing with legal interventions in the field, and with limited resources, certain limitations will be encountered and compromises made. This is not to suggest that evaluation is a fruitless endeavor. Quite the contrary, it is essential if we are to make progress in increasing access to justice for the many who need it. The intent of this discussion is to outline some of the limitations in the context of a specific evaluation so that others can avoid the problems that have been encountered, or if not avoid, them then recognize their possibility and be ready to confront them.

7