DR Program Evaluation

Presentation By Dispute Prevention and Resolution Services,
Department of Justice to the Philippine Delegation

September 22, 2003

Why Evaluate?

Evaluation is “an internal effort to define and improve operations over time while providing descriptive information to the field” (Janis Roehl)
Way to determine whether a DR program is meeting its goals and objectives
Allows program administrators to establish what works, what does not work, and to discontinue, modify or expand a DR program
Reveals the strengths and weaknesses of the DR program
Promotes consistent and proactive approach to continued DR program improvement
Identifies administrative needs (e.g. staffing requirements, bottlenecks in the process, etc.)

Important that evaluation planning begin BEFORE the DR program is implemented (e.g. Develop an evaluation framework)
Evaluation can be undertaken at different times during the life of a DR program
Factors to consider include:
- Whether the program has been in operation long enough to ensure that there are sufficient cases to examine
- Whether the program has gotten the early bugs out
- If pilot, whether the evaluation will be completed early enough to be a factor in the decision to continue/expand the program
- Whether there are other deadlines relating to future decision-making which will affect the usefulness of the evaluation results
Formative (pilot stage) and summative evaluations (when DR program at mature point)

2 main types of DR evaluations:

Program Effectiveness Evaluations (impact/outcome/summative)
- focus on whether DR program meeting its goals and/or having the desired impact
Program Design and Administration Evaluations (process/formative)
- focus on how a DR program can be improved
- Comprehensive evaluations should measure tangible and intangible benefits using both quantitative and qualitative data

DR Program Managers typically seek to evaluate some/all of the following measures of “success” *:

Cost savings to both the organization and the parties
Time savings to both the organization and the parties
Participation rates of the parties in the DR process
Participant satisfaction with the fairness of the DR process
Settlement rates
Quality of settlements (durability, creativity, improvements to ongoing relationships)

(*See: Performance Indicators for ADR Program Evaluation included in materials)

To evaluate the success of a given program, evaluators must structure their observations, measurements and reports to highlight a few characteristics/variables which are particularly significant
E.g. DR Fund sought to evaluate 4 results:
1. Reduction in costs and time spent in managing disputes
2. Increased party satisfaction with resolution outcomes
3. Funded organizations would foster further internal DR developments
4. Funded projects would serve as catalysts and/or models for other organizations

Basic Steps in the Evaluation Process:

1) Identify Participants (i.e. who uses the system – e.g. clients, lawyers, neutral service providers, internal staff, etc.)
2) Identify Program Goals: (e.g. reduce costs, reduce delay, maintain/improve disputant satisfaction, preserve the equity of outcomes, promote a less contentious environment, etc.) - must be absolutely clear about what program is trying to accomplish in implementing a particular DR process
3) Identify Performance Measures/Indicators Appropriate to Measure Desired Outcomes:
- e.g. if desired outcome is cost reduction -> whose costs? (agency’s/parties/both), what costs? (legal fees, administrative costs, etc.)
4) Collect the Right Type of Data For the Measures Identified:
- quantitative (file records, surveys, etc.) and/or qualitative data collection methods (interviews, focus groups, observations, participant surveys, etc.)
5) Choose an Appropriate Study Design:
most effective is the true control group study:
1. ensure that the population in the DR program is like that in the control group (or status quo)
2. hold all else constant
3. random assignment of cases to DR stream/traditional process
4. both processes operating contemporaneously
6) Collect and Analyze the Data
7) Discuss Findings (oral/written/both)
8) Make Necessary Changes to the DR Program

Key factors for effective evaluation:
- Objectivity (i.e. no stake in the outcome), experience in conducting program evaluations, sufficient technical expertise in terms of data collection and analysis
Internal v. external evaluators:
- External:
  - Pros – credibility, objectivity, impartiality, specialized evaluation skills
  - Con – expensive
- Internal:
  - Pros – specialized knowledge of organization, more cost effective
  - Con – potential perceptions of lack of impartiality

May wish to have “advisory committee” of key stakeholders to assist in evaluation design, implementation and reporting

(Source: Federal ADR Program Manager’s Resource Manual)

A Checklist for Evaluating Federal ADR Programs: Long Form
Performance Indicators for ADR Program Evaluation
Evaluating Agency Alternative Dispute Resolution Programs: A User’s Guide to Data Collection and Use
Federal ADR Program Manager’s Resource Manual, Chapter 8, Evaluating ADR Programs
Assessing Efficiency, Effectiveness and Quality: An Evaluation of the ADR Program of the Immigration Appeal Division of the Immigration and Refugee Board
An Evaluation of the Notice to Mediate Regulation under the Insurance (Motor Vehicle) Act
Evaluation of the Family Justice Registry (Rule 5) Pilot Project: Final Report
Evaluation of the Ontario Mandatory Mediation Program (Rule 24.1): Executive Summary and Recommendations
Mandatory Parenting after Separation Pilot: Final Evaluation Report