Evaluation of the Youth Justice Initiative

• Previous Page
• Table of Contents
• Next Page

Appendix B: Evaluation Methodology

The evaluation of the YJI relied on multiple lines of evidence that were triangulated and form the evidence base for the study conclusions and recommendations. The data collection occurred between April and September 2020. The lines of evidence included a review of literature, documents and data, a review of YJSFP and IRCS project files, key informant interviews and case studies. A more detailed description of various data sources is provided below.

Document, Literature and Data Review

Document review: A document review was conducted to provide background information for the YJI, including context, design and implementation, and was also used as evidence to address the evaluation issues and questions. The types of documents that were reviewed included:

• program foundational documents to understand the context and purpose of the program and its overall structure (e.g., legislation, policy objectives, committee terms of reference, FPT agreements);
• program implementation documents (e.g., overviews/summaries, definitions/fact sheets, Terms and Conditions, briefing notes and materials/meeting minutes, records of decision, workplans);
• performance information (e.g., previous evaluation and audit reports and associated action plans and follow-up, reporting templates, performance reports, Departmental Performance Reports/Results Reports);
• other special studies or reports; and
• financial information on planned and actual funding by fiscal year.

Literature review: A focused review of grey literature and academic journal articles published in the last several years was undertaken to provide broader contextual information on the youth justice landscape and observed trends. Two types of literature were reviewed: (i) peer-reviewed articles; and (ii) published and unpublished reports and other documents produced mainly by and for government, agencies and non-governmental organizations (grey literature). This literature was useful in understanding legal trends/policy gaps/emerging or persistent issues and needs related to youth justice, as well as exploring the effectiveness and best practices related to key directions under the YCJA such as diversionary tools and measures.

Data review: Youth criminal justice statistics from the Canadian Centre for Justice Statistics (CCJS)^{Footnote 1} were used to explore trends in youth crime and the youth justice system. Key sources included: 1) police-reported crime data collected through Statistics Canada’s incident-based Uniform Crime Reporting Survey; 2) the Integrated Criminal Court Survey which collects information on court cases, court decisions, use of remand and processing times; and 3) Youth Custody and Community Services Survey and Integrated Correctional Services Survey for sentencing and incarceration data. These data addressed evaluation questions related to relevance and performance.

Statistical materials were sourced from existing analyses undertaken by government departments and agencies (e.g., for example, CCJS Juristat publications) and existing custom tabulations from the CCJS requested by the Research and Statistics Division of Justice Canada and the Policy Implementation Directorate.

File Review

A review of the YJSFP and available IRCS files was conducted to enable an assessment of the basic characteristics of YJSFP projects and IRCS cases funded during the fiscal years covered by the evaluation. The file review component examined two types of files: annual reports for YJSFP; and IRCS case management files (Face Sheets).

YJSFP: Provinces and territories provide an annual report to Justice Canada, including descriptive information on the service delivery model (delivery of youth justice services and programs in the past year), as well as financial expenditure information. The review of YJSFP files captured a pre-defined set of variables from the service delivery model portion of the annual report, using the most recently submitted annual report. A quantitative analysis was performed for closed-ended questions and financial information, and a qualitative, thematic analysis performed for provincial/territorial narrative responses.

IRCS: With respect to IRCS, provinces and territories that receive IRCS funding use a Face Sheet to track/monitor their IRCS Parts B and C cases. The Face Sheet gathers information in the following categories:

• Part 1: Sentence Information / Diagnosis / Other Information:
• Part 2: Background Information / Chronology of Major Milestones - Custody
• Part 3: Transfers from Secure to Open Custody or Release from Custody to Community or Change of Status
• Part 4: Chronology of Major Milestones - Community
• Part 5: Completion of Community Portion or Change of Status
• Part 6: Consent / Funding Information
• Part 7: Completion / Termination of Sentence or Funding Eligibility

In total, 105 IRCS Face Sheets were reviewed of the 153 cases that were completed or terminated within the time period under study. Selected data from the IRCS files was captured using a programmed survey-style form and analyzed at an aggregate level across the files. Data included tombstone information on IRCS cases and treatment interventions, as well as results indicators such as clinical professionals’ assessment that the youth will be successful post sentence; assessment of whether the youth’s mental health improved or deteriorated; and whether IRCS Part B sentence/Part C case funding was believed to have been of benefit.

Key Informant Interviews

Key informant interviews were conducted to gather in-depth information from Justice Canada management and staff and stakeholders on the YJI, its implementation, performance and suggestions for improvement. In total, 35 interviews were conducted with 39 individuals. Interviews were conducted between June and July 2020. Justice Canada provided a list of potential interviewees, including internal and external key informants. The distribution of interviewees is presented below (Table 3).

Tailored guides were developed in both French and English for each of the respondent groups. Interviews were conducted with key informants via teleconference in the official language of choice and were approximately 45 to 75 minutes in duration. A copy of the key informant guide was provided in advance of the interview to allow respondents an opportunity to review/reflect on the questions beforehand.

Interview notes were taken electronically or by hand during the interview and then captured in a customized Excel template organized by interview question and cross referenced to the applicable evaluation indicator. The qualitative interview data were analyzed by respondent type, and main themes identified by indicator and evaluation question. Given the small numbers of respondents for each category, the responses were aggregated as follows:

• Internal Justice Canada (senior Justice Canada officials, YJI program managers and staff);
• Provincial/territorial representatives (YJI CCSO and Youth Justice Cost-Sharing and Programs WG provincial-territorial members); and
• Other stakeholders (external YJF project leads and other external stakeholders).

Because the number of responses within each of the above categories was small, a simplified three-point qualifier scheme was used to describe the level of agreement/frequency of responses. The qualifiers used to represent the findings are:

• A few/several – two to three respondents;
• Some – more than a few, but less than half of respondents;
• Many/most – more than half of the respondents.

Case Studies

Case studies, as an approach, combine various sources of information to provide illustrations and examples of selected activities or components of a program. For the evaluation of the YJI, the purpose of conducting case studies was to help illustrate various types of high-priority programs and services funded through the YJSFP.

Five provinces and territories representing various geographical regions (North, Western, Central and Atlantic) volunteered to participate in case studies. The studies were also selected to represent different areas of YJSFP priority programming (Table 4).

The approach for each case study was planned in consultation with the participating provinces and territories. While the methodology was somewhat customized for each case study, the methods generally included:

• document/data review – foundational documents describing the program or service, administrative data (e.g., program uptake, other statistics), evaluative/review materials;
• interviews – conducted with program managers/staff, service delivery partners, other stakeholders (e.g., academic partners, expertise);
• focus groups – conducted with program staff; and
• surveys – (Ontario only) of program staff responsible for delivering YJSFP-funded programs and services.

A template for the case study narratives was developed to ensure consistency in reporting across all the case studies. The narrative reports were used to inform evaluation questions related to performance.

• Previous Page
• Table of Contents
• Next Page

Date modified:

2021-08-13