3. Evaluation methodology
The evaluation was guided by an evaluation matrix (evaluation questions, indicators, and data sources) which was developed through the evaluation scoping and planning phase. The evaluation included multiple lines of evidence, and employed the data collection methods described below. Annex A contains a list of the evaluation questions, and Annex B provides a more detailed description of each methodology.
3.1 Document and Data Review
Approximately 140 secondary data sources relevant to the ICW Program were reviewed. The types of documents reviewed included the following:
- Program documents related to departmental mandate and priorities, program foundational and implementation documents, literature related to the ICW Program;
- Performance Information - ICW Program performance measurement data and COVID-19 research data; and
- Financial budget information.
3.2 Key Informant Interviews
A total of 33 interviews were conducted with stakeholders representing the following groups:
- Provincial and territorial representatives (n=16)
- Justice Canada representatives (n=4)
- Service Delivery Agency representatives (n=13)
Interviews were conducted with a representative from each province and territory in which the ICW Program was operating at the time of the evaluation.
3.3 Courtworkers and Judicial/Court Officials Surveys
Surveys were conducted with 82 Courtworkers and 94 judicial and court officials between September and November 2022. Among the Courtworkers surveyed, almost all provided services in Criminal Court (96%) and a majority provided services in Family Court (67%). The Courtworkers surveyed were located in every provincial and territorial jurisdiction with an operating ICW Program at the time of the survey, except British Columbia and Manitoba. Among the judicial and court officials surveyed, respondents included Crown counsel (58%), judges (14%), probation officers (13%), defense counsel (10%), and court clerks (3%). The judicial and court officials surveyed were located in every provincial and territorial jurisdiction with an operating ICW Program at the time of the survey, except Quebec and Prince Edward Island.
3.4 Client Survey
Surveys were conducted with 482 ICW Program clients between August and December 2022. The client survey was administered with the assistance of SDAs implementing the ICW Program across Canada. Many SDAs had resource constraints, inhibiting their ability to collect ICW client contact data and administer the survey. As a result, some jurisdictions are not represented in the client survey data. Of the 525 client surveys distributed, 482 ICW clients completed the survey, resulting in a response rate of 92%.
The ICW clients surveyed were located throughout Canada: Quebec (32%), British Columbia (30%), and Alberta (26%), with some located in Ontario (7%), Nunavut (4%), and Northwest Territories (1%). A total of 67 communities were represented. In terms of their involvement in the justice system, 85% of clients were involved with Criminal Court, while 15% were involved with Family Court matters. For a majority of clients (61%), it was their first time receiving ICW services.
In reporting the findings from the client survey, Courtworkers and judicial/court officials surveys, the following scale was used: a few = <20% of respondents; some = 25-49% of respondents; half = 50% of respondents; majority = 51-70% of respondents; most = 71-90% of respondents; and almost all = >90% of respondents.
3.5 Case Studies
Three case studies were conducted to highlight and provide more in-depth information in three areas: 1) economic value of Courtworkers; 2) client impact stories; and 3) the role of Courtworkers in family justice matters. Data collection for each case study included survey and interview results, additional interviews specific to each case study and a review of relevant program documents, files and publicly available information (e.g., websites).
3.6 Evaluation Limitations
The evaluation encountered several methodological limitations or challenges that should be noted.
- Attributing the impact of federal funding on the ICW Program was challenging because the Program is cost-shared and delivered by provincial and territorial governments through various delivery structures. The role of Justice Canada with respect to the delivery of Courtwork services is limited since the delivery structures and agreements with SDAs delivering ICW services rest with provincial and territorial governments. This limitation was mitigated by focusing on the relevance, effectiveness, and efficiency of ICW Program activities that are the responsibility of Justice Canada.
- There exists potential for respondent bias in the qualitative evidence. The evaluation relies on qualitative methodologies such as key informant interviews, surveys and case studies. These methodologies have the potential for respondent bias. This limitation was mitigated by comparing responses by different respondent groups. In addition, respondents were asked to provide examples to better understand and contextualize responses.
- There exists a gap in performance data. Complete performance data for the ICW Program was not available as some jurisdictions did not provide performance data for FY 2018-19 and FY 2019-20, and performance information for FY 2021-22 was not yet available. The lack of performance data made it difficult to assess changes in Courtworkers’ workload. To mitigate this limitation, qualitative information was collected regarding changes in workload. In addition, some information outside the period of the evaluation (e.g., performance data for FY 2016-17) was included for comparative purposes and to provide context.
- There are limitations to the client survey in terms of its implementation and completeness. The surveys were administered by multiple different SDAs across Canada, and in many jurisdictions, organizational capacity constraints limited the number of clients that participated in the survey. This situation created the potential for inconsistencies in how the client survey was administered and/or how the survey questions were interpreted. To mitigate this, the evaluation exercised caution in analysing client survey data, and triangulated with other lines of evidence.
- There were few case study interviews conducted. Interviews with SDA representatives, Courtworkers, and clients was intended as a key data collection method for the case studies. Despite multiple attempts to schedule interviews with potential interviewees, a very limited number of individuals accepted to participate in an interview. For the case study regarding the role of Courtworkers in family justice matters, only two interviews were conducted, and only one interview was conducted for the client impact stories case study. As such, few client impact stories were collected, and minimal information regarding the history, context, and role of Courtworker services in Family Court was gathered. As a result, these case studies did not yield the expected information.
- There are gaps and limitations to the analysis of cost savings to the Justice system. The analysis of estimate cost savings to the justice system attributable to the ICW Program should be viewed with caution. The analysis relies on Courtworker recollections and opinions about how their interactions with clients contributed to a reduction in the number of client arrests and clients in custody. External research sources were used to infer the reduced costs to the justice system as a result of Courtworker actions and it was assumed that these costs were accurate and generalizable across Canada. The analysis also does not include a control group or counterfactual assessment (e.g., of costs that would have been incurred without Courtworker involvement).
- Date modified: