Crimes Against Humanity and War Crimes Program Evaluation

Appendix B: Evaluation Matrix

Evaluation matrix
Issue/question Indicators Data sources and methods
Relevance - Issue 1: Continued need for the Program
1.1 Is there a continued need for the Program? 1.1.1 Evidence and perception as to whether the context or environment related to war crimes and crimes against humanity have changed (e.g., war crimes continue to be committed/international community still committed to prosecutions)
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies

Baseline information:

  • Annual Reports for the Program for 2003–04 to 2007–08
  • Relevant international reports and studies for 2003–04 to 2007–08
1.1.2 Evidence and perception as to whether national and/or international legislative or policy changes or new international accords have enhanced or reduced the need or requirement for the Program
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
1.1.3 Evidence and perception as to the extent that the Program as currently configured meets partner department and international partner needs for identification and follow-up regarding war crimes and crimes against humanity
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
1.1.4 Evidence and perception as to whether partner department and international partner needs and priorities are reflected in the current configuration of the Program
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
Evidence and perception of the continuing demand for the Program
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
1.2 To what extent are the objectives of the Program still relevant? 1.2.1 Level and perceived changes of international commitment to this issue and pressure on Canada to participate
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff

Baseline information:

  • Annual Reports for the Program for 2003–04 to 2007–08
  • Relevant international reports and studies for 2003–04 to 2007–08
1.2.2 International evidence of the impact of other similar initiatives
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Country Studies
Relevance - Issue 2: Alignment with government priorities
2.1 To what extent does the Program meet the policy priorities of the government with respect to war crimes and crimes against humanity? 2.1.1 Evidence and perception as to whether the objectives are consistent with the federal government’s policy priorities with respect to war crimes and crimes against humanity
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Speech from the Throne
2.2 To what extent does the Program align with the departmental strategic outcomes of each partner? 2.2.1 Evidence and perception as to whether the objectives are consistent with departmental strategic outcomes of each Program partner
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
Relevance - Issue 3: Alignment with federal roles and responsibilities
3.1 Is there still a role for the federal government to deliver the Program? 3.1.1 Extent to which the federal government and/or international entities believe the GoC should deliver the CAHWCP or aspects of it Structured Interviews with Key Stakeholders
3.1.2 Extent to which international obligations require the GoC to deliver the CAHWCP or aspects of it Structured Interviews with Key Stakeholders
Performance (effectiveness, efficiency, and economy) - Issue 4: Achievement of expected outcomes
4.1 To what extent has the Program contributed to an increase in knowledge and awareness of the Program among stakeholders? 4.1.1 Level of outreach activities (e.g., dissemination activities, tools, training, international support)
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
4.1.2 Extent to which training adequately prepares staff to exercise their responsibilities in relation to the Program
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
4.1.3 Level of delivery partners’ knowledge of other components of the Program
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
4.1.4 Perceived extent and adequacy of knowledge management
  • Review of Performance Information, Files, and Databases
  • Structured Interviews with Key Stakeholders
4.2 How well have allegations been managed under the CAHWCP with respect to determination? 4.2.1 Change from 2008-09/2011–12 to 2012–13/2015–16 in the number of decisions rendered by Federal Court where revocation was considered

Review of Performance Information, Files and Databases

Baseline information:

Total inventory of revocation cases for period 2008–09 to 2011–12

4.2.2 Perception among stakeholders of Canada’s ability to address allegations
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
4.2.3 Change from 2008–09/2011–12 to 2012–13/2015–16 in the number of removals, extraditions and successful defence by respondent/defendant

Review of Performance Information, Files and Databases

Baseline information:

Total inventory of removals, extraditions and successful defences for period 2008–09 to 2011–12

4.3 To what extent has the Program deterred and prevented persons believed to have committed or been complicit in CAHWC from coming to Canada? 4.3.1 Level and type of publicity surrounding cases Review of Performance Information, Files and Databases
4.3.2 Trend analysis of the allegation inventory and case outcomes from 2008-09/2011–12 to 2012–13/2015–16

Review of Performance Information, Files and Databases

Baseline information:

Total inventory of allegations and case outcomes for period 2008–09 to 2011–12

4.3.3 Perception among stakeholders that the Program has deterred persons believed to have committed or been complicit in CAHWC from coming to Canada
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
4.4 How and to what extent has Canada demonstrated leadership regarding CAHWC issues? 4.4.1 Perception among stakeholders that Canada is a leader in CAHWC issues
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Country Studies
4.4.2 Extent to which other countries continue to learn from and emulate the Canadian model
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Country Studies
4.4.3 Adequacy of Canada’s legislative framework to address CAHWC issues
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
4.5 To what extent has the Program assisted Canada to meet its international obligations? 4.5.1 Perception among domestic and international stakeholders that Canada has met its international obligations regarding CAHWC
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • UNHCR reports
  • Country Studies
4.5.2 Benefit of Canada meeting its international obligations as signatories to the Genocide Convention; the Geneva Convention concerning war crimes and Additional Protocols; the Convention Against Torture; and the Rome Statute creating the International Criminal Court
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
  • Country Studies
4.6 To what extent does Canada protect Canadians and successfully remove persons believed to have committed or been complicit in CAHWC through the Program? 4.6.1 Perception that the Program contributes to the protection of Canadian residents, particularly those formerly from regions where CAHWC have been committed, from the actions of persons believed to have committed or been complicit in CAHWC through the removal of such persons from Canada
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
4.6.2 The number of offenders prosecuted in Canada Review of Performance Information, Files and Databases
4.6.3 The number of inadmissible individuals removed from Canada Review of Performance Information, Files and Databases
4.6.4 The number of inadmissible individuals whose Canadian citizenship is revoked based on misrepresentation Review of Performance Information, Files and Databases
4.6.5 The number of inadmissible individuals refused visas Review of Performance Information, Files and Databases
4.6.6 Justice/RCMP Inventory of suspected war crimes, crimes against humanity, and genocide cases Review of Performance Information, Files and Databases
4.7 To what extent does Canada contribute to the domestic and international fight against impunity and is not a safe haven for persons believed to have committed or been complicit in war crimes, crimes against humanity or genocide? 4.7.1 Perception among stakeholders that Canada is not a safe haven
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
4.7.2 The number of inadmissible individuals refused visas Review of Performance Information, Files and Databases
4.8 Were there any unintended impacts? 4.8.1 Lessons learned from the delivery of the CAHWCP
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
4.8.2 Evidence the horizontal approach contributed/detracted from the achievement of outcomes
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
Performance (effectiveness, efficiency, and economy) - Issue 5: Demonstration of efficiency and economy
5.1 To what extent has the Program achieved its results to date efficiently? 5.1.1 Description of the resources (FTEs, operations and program funding) allocated to the Program each year from 2012–13 to 2015–16 plus additional departmental/partner resources contributed to achieve objectives
  • Review of Performance Information, Files and Databases
  • Structured Interviews with Key Stakeholders
  • Cost comparisons
5.1.2 Evidence that the resources were used for the purposes intended
  • Review of Performance Information, Files and Databases
  • Structured Interviews with Key Stakeholders
  • Cost Comparisons by Remedy
5.1.3 Evidence and perception of the extent to which each Program partner could have increased outputs with the same level of inputs, or whether the same level of outputs could have been achieved with a lower level of inputs
  • Review of Performance Information, Files and Databases
  • Cost Comparisons by Remedy
  • Structured Interviews with Key Stakeholders
5.1.4 Evidence the horizontal approach contributed to the efficiency and economy of the initiative
  • Review of Performance Information, Files and Databases
  • Cost Comparisons by Remedy
  • Structured Interviews with Key Stakeholders
  • Case Studies
5.2 To what extent has the Program achieved its results to date economically? 5.2.1 Evidence and perception as to whether there are alternative ways of achieving Program objectives that might be less costly than the current approach, and description of any alternative approaches
  • Cost Comparisons by Remedy
  • Structured Interviews with Key Stakeholders
  • International comparisons
  • Case Studies
Performance (effectiveness, efficiency, and economy) - Issue 6: Design and Delivery
6.1. Was the Program designed appropriately and did it operate in the manner intended? 6.1.1 Evidence and perception of the factors that influenced the horizontal collaboration
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
6.1.2 Evidence and perception that the governance is effective (evidence that it supports accountabilities, decision making, control and risk management)
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
6.1.3 Evidence and perception of the impact of different departmental cultures on the management of the Program
  • Structured Interviews with Key Stakeholders
  • Survey of Departmental Staff
  • Case Studies
Date modified: