Evaluation of the Justice Canada Federal Victims Strategy

3 Evaluation Methodology

An evaluation matrix was developed to guide the evaluation, which included relevant indicators and data collection methods for each of the evaluation questions. The evaluation matrix was reviewed by the Evaluation Working Group, along with associated data collection methods. The specific evaluation issues and questions are included in Appendix B.

3.1 Data Collection Methods

A combination of methods was used to collect data in support of the evaluation including:

  • Documents and Literature: Background documents and literature were reviewed, including foundational federal government documents (e.g., Throne Speeches, Ministerial Mandate Letters, etc.), program related documentation (e.g., Justice FVS Performance Information Profile, Victims Fund Terms and Conditions, previous Justice FVS evaluation reports, etc.), and literature on victim issues and the nature of victimization in Canada (e.g., Victims of Crime Research Digest, including an article on Victims and Survivor Participation in Indigenous Justice Program Restorative Justice Processes, General Social Survey, etc.). Additionally, given the 20-year evolution of the Justice FVS, the evaluation also included a review of historical documents (e.g., previous evaluation reports and other program documents) of the Strategy to provide a brief retrospective review of the changes to the Justice FVS response to victim needs since the Strategy began.
  • Administrative and Financial Data: Administrative data was reviewed for the fiscal period of 2015-16 to 2019-20 and included Victims Fund applications, approvals and amendments, funding streams and calls for proposals (CFPs), and Gs&Cs service standards. Financial data were also reviewed during the same period to examine Justice FVS allocations, expenditures, and lapses.
  • Victims Fund Files: Victims Fund files were reviewed for the fiscal period of 2015-16 to 2019-20 to examine funded activities of PTs and NGOs. Documents included funding agreements and associated project reports related to the PT five-year agreements, TAs in the context of implementation, FILUs, supports for families of MMIWG, as well as a sample of initiatives focused on providing supports for victims of sexual assault, particularly adult victims.
  • Victims Fund and Policy Initiatives Surveys: A review and analysis of data from pre-existing surveys was conducted for the fiscal period of 2015-16 to 2019-20. This included data from the Victims Fund financial assistance surveys (i.e., victims assessing the Parole Board and Victims Abroad funding streams) and policy initiative event surveys (e.g., PCVI led knowledge exchanges and webinars).
  • Case Studies: Findings from case studies of Justice FVS activities for the current reporting period were reviewed and incorporated. The case studies included Supports for Families of MMIWG (2019) and TAs (2019).
  • Key Informant Interviews: A total of 36 semi-structured individual and group interviews were conducted with 45 key informants over the telephone. Participants included departmental personnel, federal partners, PT stakeholders, and NGOs. The number of key informants from each group are provided in Appendix B. Representatives from the Evaluation Working Group helped identify key informants and reviewed draft interview guides.

The analytical approach to synthesizing and triangulating data collected from the above noted methods, as well as the scale used to report on the frequency of key informant responses, are provided in Appendix B.

3.2 Limitations

There were some methodological limitations and challenges encountered during the evaluation. Table 3 outlines the key limitations and the strategies employed to mitigate them.

Table 3: Limitations of the Methodology and Mitigation Strategies
Limitation Mitigation Strategy
Victims Perspective: Interviews were not conducted with victims and survivors of crime during the current evaluation
  • Included data from case studies containing interviews with front line staff (as a proxy) across Canada
  • Included data from previously conducted surveys of victims accessing direct financial support from the Victims Fund
Response Bias: Key informant interviews may introduce response bias due to their vested interest in the program
  • Broad range of stakeholders interviewed, across different jurisdictions and roles
  • Multiple lines of evidence used to verify findings against other sources and perspectives
Victims Fund Files: Lack of consistent reporting across PTs and CBOs/NGOs who received Gs&Cs (e.g., #/types of activities, # of participants, etc.)
  • Higher-level analysis of types of activities/examples of uptake conducted
  • Additional examples gathered through interviews
Financial Data: Variances in how budgeted and expended resources are captured at the departmental and programmatic levels
  • Focused on more detailed analysis of Gs&Cs budgeted and expended amounts
Administrative Data: Lack of access to easily operable databases related to applications, rejections, and amendments
  • Manually coded PDF files and conducted multiple rounds of coding for accuracy
  • Selected samples to reduce coding burden
Date modified: