Evaluation of the Indigenous Courtwork Program
3. Methodology
The evaluation utilized multiple lines of evidence, including primary and secondary sources of information.
3.1. Document, Data and Literature Review
The evaluation involved a review of program documents such as previous program evaluations (2008 and 2012), the ICW Program Information Profile (2017), jurisdictional profiles, the ICW Strategic Plan (2016) and Implementation Plan (2017), Departmental Performance Reports and Reports on Plans and Priorities, the Minister’s Mandate Letter, meeting agendas, minutes and files from the TWG and the FPT Working Group, and the Terms of Reference for the Aboriginal Courtwork-Access to Justice Services Collaborative Working Group (2015).
The review of data included analysis of performance measurement data submitted by each participating jurisdiction, a review of program budgets and expenditures, and a review of national and regional projects funded under the ICW. The literature reviewed included research on key issues including over-representation in the criminal justice system, guilty pleas, mental health, addictions and Fetal Alcohol Spectrum Disorder (FASD), as well as collaboration among the family, child protection and criminal justice systems, evaluations of a Gladue Court and a First Nations Court, gender-based analyses, administration of justice offences, the Truth and Reconciliation Commission report, and outputs from funded projects and activities.
3.2. Surveys and Interviews
Surveys were conducted with:
- 124 judicial and court officials including 32 judges and 4 justices of the peace, 23 Crown counsels, 24 defence counsels, 11 probation officers, 16 court clerks, and 14 other court representatives who are familiar and have been involved in the ICW Program. The survey gathered information on the need for the program activities, its impact on clients and justice officials, partnerships that may have been developed, and key factors that influence effectiveness and opportunities for improvement. Those surveyed were drawn from contact lists of 220 potential respondents, providing a response rate of 56%.
Jurisdiction |
Judge/ Justice of the Peace |
Crown Counsel |
Defence Counsel |
Probation Officer |
Court Clerk |
Other |
Total |
|---|---|---|---|---|---|---|---|
Alberta |
5 |
3 |
4 |
0 |
0 |
5 |
17 |
British Columbia |
6 |
3 |
4 |
4 |
0 |
0 |
17 |
Manitoba |
6 |
2 |
2 |
0 |
0 |
0 |
10 |
Northwest Territories |
0 |
0 |
4 |
0 |
0 |
1 |
5 |
Nova Scotia |
1 |
0 |
1 |
0 |
0 |
0 |
2 |
Nunavut |
0 |
3 |
1 |
0 |
0 |
2 |
6 |
Ontario |
11 |
6 |
6 |
5 |
5 |
2 |
35 |
Quebec |
2 |
3 |
0 |
0 |
2 |
0 |
7 |
Saskatchewan |
5 |
2 |
2 |
2 |
9 |
1 |
21 |
Yukon |
0 |
1 |
0 |
0 |
0 |
3 |
4 |
Total |
36 |
23 |
24 |
11 |
16 |
14 |
124 |
Source: Survey of Judicial and Court Officials, 2017
- 114 Courtworkers to obtain their perceptions of the ICW Program, particularly with respect to client needs, the relevance of the services provided, Program impacts, the effectiveness of the Program design, and opportunities for improvement. Of the 185 Courtworkers who were targeted, 114 responded, providing a response rate of 62%.
- 35 key informants including six Department of Justice representatives, 16 provincial and territorial representatives, and 13 SDA representatives. Of these key informants surveyed, 33 participate in the TWG. The response rate for the key informant survey was 83% (42 people were approached to participate, of whom 35 completed). The purpose was to gather information on the Program need and responsiveness, alignment with government priorities, impacts of the Program, key factors that influence effectiveness, the TWG and opportunities for improvement.
- 823 ICW Program clients. Indigenous interviewers familiar with the ICW Program conducted a client survey consisting of close-ended questions. Some interviewers spoke the local language. The purpose was to collect information on client satisfaction with the information and referrals received, the decision-making process, understanding of the information received, whether additional help was needed, and perceptions of the justice system.
Of the clients who were surveyed, 116 consented to participate in a follow-up telephone interview and provided valid emails and telephone numbers. Of those, 62 (51%) were interviewed. The telephone interviews consisted primarily of open-ended questions, were interactive, and focused more directly on the importance and impact (particularly on decision making) of the information and support received through the ICW Program.
3.3. Case Studies
The case studies involved visits to four communities including two communities in British Columbia (Vancouver and Nanaimo), one in Alberta (Edmonton), and one in Ontario (Tyendinaga Mohawk Territory). Thirty-three individuals were interviewed during the case studies including eight Courtworkers, ten court and justice officials, nine stakeholders, four SDAs and three provincial government representatives. The case studies focused on issues related to service delivery, potential gaps in services available, areas where services need to be expanded, and best practices. As part of the case studies, a review of best practices, activities and approaches to service delivery and supports for Indigenous persons involved in the criminal justice system was conducted, including:
- Downtown Community Court (Vancouver, British Columbia);
- Bail Worker Program (Edmonton, Alberta);
- Kind Heart Services – providing culturally relevant and FASD informed mentoring services for Indigenous persons involved in justice system (Edmonton, Alberta);
- Restorative Justice and Changing Directions for Youth Program (Nanaimo, British Columbia);
- Indigenous Youth Engagement with Bench & Bar Pilot Projects (Ontario).
The findings of case study interviews and documents reviewed are integrated with other lines of evidence in this report. Please note that case studies did not assess the impact of these activities (some are pilot projects that have been recently implemented), but rather identified success factors and innovative approaches to service delivery.
3.4. Evaluation Limitations
The evaluation encountered several challenges and limitations. The timing of the evaluation did not allow for an assessment of the increase in ICW Program budget in 2016-17. The methodological challenges and limitations, as well as mitigation strategies, are described in the following table.
Evaluation Limitations and Challenges |
Mitigation Strategies |
|---|---|
Respondent bias. Most people surveyed were directly involved in the ICW Program (providing services or benefiting from services) which can result in positive response bias. |
Several measures were taken to reduce the effect of response bias and validate interview results including (i) the use of multiple lines of evidence, particularly validating findings through other primary and secondary research; (ii) the instruments clearly communicated the purpose of the evaluation, its design and methodology, and strict confidentiality of responses to participants; and (iii) the key informants and justice officials were asked to provide a rationale for their ratings. |
The performance measurement system has evolved in terms of its structure and was implemented over several years, which somewhat restricts the availability and comparability of program data over years. Data was not always reported consistently across jurisdictions. |
For some indicators, the evaluation relied most heavily on data in 2015-16, a year for which most regions reported data. The findings clearly identify the data used and potential limitations, or the changes made over time. |
The ICW Program is designed and delivered somewhat differently across jurisdictions, reflecting differences in scope, services, role of Courtworkers, and other available resources. Thus, some observations about the ICW Program may apply to some jurisdictions and not others. |
While the ICW Program has been designed to be delivered in a flexible way, methodologically this creates challenges for the evaluation by not measuring entities equally across jurisdictions. As a national evaluation, it has focused primarily on broad issues and trends rather than on jurisdictional differences. |
There were challenges calculating the response rate for the client survey due to technical difficulties with the survey software, which meant that not all clients who refused to respond were recorded in the system. |
The client survey was conducted in the ten jurisdictions where ICW services are provided. The results presented in the evaluation are based on completed surveys. |
Overall, the weaknesses were mitigated through the use of multiple lines of evidence to triangulate findings and increase data reliability, significant sample sizes incorporating the perspectives of all key stakeholder groups involved with the ICW Program, and the use of both quantitative and qualitative data.
- Date modified: