Evaluation of the Youth Justice Initiative Funding Components

APPENDIX E:
Evaluation Framework

Evaluation Framework for the Youth Justice Initiative – Funding Components
Issues/questions Indicators Data sources
Relevance
1. Are the objectives of the funding components of the YJI consistent with the strategic objectives of the Department of Justice and federal priorities?
  • Comparison of objectives with departmental strategic objectives and federal priorities
  • Document review
2. Are the three funding components of the YJI still relevant/needed? What would be the impact on the youth justice system if they were not in place?
  • Identified need for programs
  • Impact of YJI funding components in youth justice system
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients): Case studies:
Survey:
3. Does the program area or activity continue to serve the public interest?
  • Assessment of contribution to public interest
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients):
4. Is there a legitimate and necessary role for government in each program area or activity? Is the current role of the federal government appropriate, or is the program a candidate for realignment with the provinces?
  • Assessment of appropriateness of government role and level of government involvement
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients):
Program Design and Implementation
General
5. To what extent have the YJI funding components been implemented as planned? What has changed and why? Were there any impacts attributable to those changes? Comparison of actual with intended design in terms of:
  • organizational structure
  • funding programs
  • stakeholder relationships
  • performance measurement strategy
Perception/evidence of impacts of changes		 Document review
Key informant interviews (Department of Justice personnel)
6. To what extent is the design of the YJI funding components appropriate to support the objectives, intended outcomes, identified needs, and emerging issues?
  • Assessment of appropriateness of design
  • Assessment of proposal development / approval processes and recommendations for changes (YJF)
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients): Case studies
Survey
7. To what extent have partnerships/relationships been working effectively to support the achievement of YJI funding objectives?
  • Assessment of effectiveness of project partnerships (YJF)
  • Assessment of effectiveness of federal-provincial/territorial working relationship (YJSFP and IRCS)
  • Number and nature of joint initiatives and working groups with federal and provincial/territorial partners (YJSFP and IRCS)
 Document and file review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients): Case studies
Survey
8. Have there been any changes in YJSFP service delivery over time? Were changes a result of federal funding? (YJSFP)
  • Changes in high priority services and programs
  • Assessment of reasons for changes in service delivery
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives
9. How is the basic capacity and case funding used by jurisdictions? (IRCS)
  • Description of use of IRCS funding, including incremental funding
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives
10. What assistance do Department of Justice staff provide to YJF funding applicants and funded projects? To what extent does this assistance help projects to move forward? (YJF)
  • Nature and extent of assistance provided/received
  • Assessment of assistance provided/received
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, funding recipients)
Case studies
Survey
Performance Measurement Strategy
11. To what extent has the Department of Justice implemented an effective performance measurement strategy to:
provide timely management/project management information
assist the Department of Justice in ongoing performance monitoring and program adjustment
capture adequate and useful information for assessing the impact of YJI funding?
  • Evidence that agreements/projects are being monitored
  • Capability of GCIMS to generate relevant reports (YJF)
  • Assessment of performance measurement strategy
  • Assessment of data to support evaluation
  • Assessment of reporting requirements for agreements/projects
 File review
GCIMS
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, funding recipients
Case studies
Survey
Success
12. In what manner and to what extent have the activities and outputs contributed toward achieving the objectives and intended outcomes of the YJI funding components? (to be addressed for each funding component separately) YJSFP
  • Number of agreements with provincial/territorial partners
  • Number of cases involving youth by jurisdiction
  • Youth incarceration rates by jurisdiction
  • Ability to maintain/continue existing and high priority programs and services after funding cap
  • Assessment of alignment of provincial/territorial youth justice programs with federal youth justice policy objectives
  • Availability and use of alternatives to court and incarceration
  • Assessment of use of court process and custody by seriousness of offence
  • Assessment of proportionality of accountability measures to the severity of the offence and degree of responsibility of the offender
  • Changes in integration and/or coordination in approach to youth justice issues
 Document and file review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives)
IRCS
  • Number of agreements with provincial/territorial partners
  • Number of IRCS cases
  • Availability of specialized services required for the administration of an IRCS sentence
  • Assessment of changes in provincial/territorial capacity to provide specialized services, including rehabilitation and reintegration services, for serious violent young offenders suffering from a mental illness or disorder
  • Assessment of impact of incremental funding
  • Assessment of proportionality of accountability measures to the severity of the offence and degree of responsibility of the offender
  • Changes in integration and/or coordination in approach to youth justice issues
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives)
YJF
  • Number of applications received
  • Number and types of projects funded
  • Number of projects in each funding component (provincial/territorial partnerships, PLEI, innovative programming, cities and community partnerships) and priority area (GGD, YJADS).
  • Number of projects with new/non-traditional partners
  • Nature of non-traditional partnerships
  • Number of projects involving provincial/territorial partners
  • Evidence that projects are reaching targeted groups through projects/number of youth reached
  • Assessment of changes in responsiveness to emerging youth justice issues and youth with unique needs including youth in care, youth with mental health issues, youth with FASD, Aboriginal youth, gang-involved youth and violent offenders, and drug-addicted youth
  • Level of community involvement in youth justice system, including collaboration between traditional and non-traditional youth justice partners
  • Evidence of knowledge development amongst youth justice system stakeholders, including sharing and application of best practices, skills development, and awareness of new or enhanced models
  • Development, availability, and use of enhanced rehabilitation and reintegration services, including community-based youth programs and initiatives
  • Changes in integration and/or coordination in approach to addressing emerging youth justice issues
 Document and file review:
GCIMS data
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients)
Case studies
Survey
13. Are there more effective ways of achieving the objectives of YJI funding?
  • Identified possible alternatives
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients):
Case studies
14. Are the YJI funding components supported by stakeholders?
  • Number and nature of shared initiatives with stakeholders
  • Level of stakeholder support for funding approaches
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives)
Case studies
Survey
15. What unintended or unanticipated impacts were generated by the YJI funding components? How have they impacted achievement of the intended outcomes?
  • Identified unintended impacts
Document and file review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients) Case studies
Survey
16. What factors are contributing to or constraining the achievement of the objectives and intended outcomes?
  • Identified challenges and best practices
  • Nature of factors contributing to or constraining success
 File review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, other federal representatives, funding recipients): Case studies
Survey
17. Is the Department of Justice facing any resource challenges in terms of available funding, organizational structure/staffing, or internal support that are affecting its ability to achieve its objectives?
  • Description of allocation and resource demands
 Key informant interviews (Department of Justice personnel)
Cost Effectiveness and Alternatives
18. Is the resultant package of programs and activities affordable? If not, what programs or activities would be abandoned?
  • Assessment of affordability and potential impact of cuts
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives)
19. If the program or activity continues, how could its efficiency be improved?
  • Documented evidence of inefficiencies
  • Identified possible efficiencies
 Document and file review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives, funding recipients
20. What activities or programs should or could be transferred in whole or in part to the private/voluntary sector?
  • Assessment of transferable activities or programs
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives, funding recipients)
21. Is federal government funding keeping pace with provincial/territorial expenditures? (YJSFP)
  • Level of federal, provincial, and territorial expenditures
  • Assessment of balance of federal/provincial/territorial contributions
 Document review
Key informant interviews (Department of Justice personnel, provincial/territorial representatives)
22. Are Canadians getting value for their tax dollars?
  • Assessment of program value for money to Canadian taxpayers
 Key informant interviews (Department of Justice personnel, provincial/territorial representatives, funding recipients)