Appendix B: Evaluation methodology

• Previous Page
• Table of Contents

The following section describes the evaluation methodology employed.

1. Document, Data and Literature Review

A variety of documents, data, and literature were reviewed as part of the ICW Program evaluation including program data and documents such as the ICW Program’s Performance Information Profile, budget-related documents, performance measurement data, and annual reports; data from ICW gatherings such as survey results and evaluations; COVID-19 research data such as Indigenous Resilience and Adaptability: The National Indigenous Courtwork Program Response to COVID-19 (2021) prepared by the Native Courtworker and Counselling Association of British Columbia and Reciprocal Consulting; background documents and briefings; and research, statistics, and literature on the needs and context with respect to the ICW Program. Approximately 140 documents, data and literature were reviewed between February and August 2022.

2. Key Informant Interviews

Interviews were conducted with 33 key informants between August and October 2022 including 16 provincial and territorial representatives, 13 SDA representatives, and 4 Justice Canada departmental representatives. Interviews were conducted via videoconference in a semi-structured format that followed an interview guide. Key informants were provided with the interview guide in advance and given the option to participate in the interview in the official language of their choice. Table 4 presents the breakdown of key informants by jurisdiction and type.

All quotes presented in the report were reviewed and validated by the individuals who provided them as part of the evaluation.

3. Surveys of Judicial and Court Officials and Courtworkers

Surveys were conducted with 82 Courtworkers and 94 judicial and court officials between September and November 2022 (Table 6). Contacts for potential survey respondents were provided by provincial and territorial representatives. A separate survey questionnaire was developed for each respondent group and focused on the effectiveness and efficiency of the ICW Program. Respondents were contacted by email in both official languages to complete a voluntary online survey (20-30 minutes to complete). Some contacts were invited directly, while others were asked to forward an open survey link to other individuals involved in the ICW Program to invite them to complete the survey.

Table 7 provides a breakdown of survey respondents by jurisdiction and type. Surveys were conducted with individuals covering most provincial and territorial jurisdictions that had an ICW Program at the time of the evaluation.

* Contacts were not provided for Courtworkers in BC, and Judicial and Court Officials in Quebec and PEI at the time of the evaluation.

Prospective survey respondents were followed up by email and telephone to solicit participation and obtain referrals for other individuals who may be better suited to complete the survey. A total of 176 individuals completed the surveys among the 285 invited, resulting in a response rate of 62% (Table 8).

* Does not include the number of individuals who forwarded the survey.

4. Survey of ICW Program Clients

Surveys were conducted with 482 ICW clients between August and December 2022. The survey of ICW Program clients was administered by SDAs implementing the ICW Program across Canada. Justice Canada Evaluation Branch staff delivered training sessions to those administering the survey in August and September 2022. Each SDA was then provided with a web link to the survey as well as a Word document in case internet connectivity was an issue. The survey web link remained open from August and October 2022 in most jurisdictions. However, survey administration continued into November and December 2022 for some jurisdictions.

A total of 525 surveys were distributed and 482 surveys were completed, resulting in a completion rate of 92%. The survey was administered primarily by telephone (69%), in court (16%), in remand/jail (3%), and in other locations such as in person/in the community, in the Courtworker office, in a residential home, and in a Friendship Centre.

Among the 482 clients who provided information:

• 59% identified as a man, 41% identified as a woman, and 1% identified as another gender;
• 13% were 18-24 years old, 33% were 25-34 years old, 39% were 35-50 years old, and 16% were 51+ years old; and
• 86% received services in English, 15% received services in French, and 5% received services in another language (mostly Innu) (note that some clients received services in multiple languages).

In terms of their involvement in the justice system, 85% of clients were involved in Criminal Court, while 15% were involved in Family Court matters. The most frequent types of charges they faced included assault (28%), other Criminal Code offences (15%), provincial offences (12%), administration of justice (8%), weapons and explosives (7%), property damage/mischief (7%), and public order offences (7%). For a majority of clients, it was their first time receiving ICW services (61%), while 39% had received ICW services before.

5. Case Studies

Three (3) case studies were conducted, which addressed different themes: 1) economic value of Courtworkers; 2) client impact stories; 3) role of Courtworkers in family justice matters. Data collection for each case study included interviews, a review of survey and key informant interview data, and a review of program documents and files and publicly available information (e.g., websites). Case studies were summarized in individual case study reports. In cases where a case study included a specific quote, we validated the information and obtained consent from the individual who provided the information. Table 10 provides details about the approach employed for each case study.

• Previous Page
• Table of Contents