Litigation Branch Evaluation Final Report

Appendix B: Evaluation Matrix

Evaluation Matrix — Evaluation of the Litigation Branch
Issues/questions Indicators Data sources Timing of data collection Responsibility for data collection
Relevance

1. Is there a continued need for Litigation Branch legal services?

TB core issue 1

  • Trends in demand for Litigation Branch services
  • Number of actively managed files by fiscal year and type of service (litigation, advisory, other)
  • Total hours of legal services by fiscal year and type of service (litigation, advisory, other)
  • Legal practice trends (type of legal issues by fiscal year)
  • Trends in file complexity and risk level by fiscal year and type of service (litigation, advisory, other)
  • Document review (e.g., iCase, environmental scan of practice trends, action plan reports)
Ongoing Litigation Branch
  • Legislative and policy requirements
  • Document review (e.g., legislation and policy documents)
Evaluation Evaluation Division

2. Does the delivery of legal services by the Litigation Branch continue to respond to federal government priorities?

TB core issue 2

  • Alignment of the Litigation Branch mandate and activities with federal priorities/policy commitments
  • Assessment of how the Litigation Branch contributes to the priorities and objectives of the federal government
  • Document review (documentation describing Litigation Branch mandate/ activities and relevant federal priorities)
  • Key informant interviews
Evaluation Evaluation Division

3. Is the provision of legal services by the Litigation Branch consistent with the Department of Justice Act and the departmental mandate?

TB core issues 2 and 3

  • Assessment of extent to which Litigation Branch legal services are consistent with Department of Justice Act and departmental mandate
  • Document review (legislation and policy documents)
  • Key informant interviews
Evaluation Evaluation Division
Performance of Litigation Branch(effectiveness): TB core issue 4
Immediate outcomes

4. Does the Litigation Branch provide timely, responsive, high-quality litigation services and legal advice?

  • Compliance with Service Standards
  • Assessments by the Litigation Branch and the client departments on litigation services and legal advice in terms of:
  • clarity
  • timeliness
  • meeting deadlines
  • understanding nature of problem
  • providing updates/progress reports
  • involving the client department in development of legal strategy and positions
  • developing legal strategies appropriate to client department policy and/or program objectives
  • effectively working with the client department to identify and manage legal risks
  • overall quality
  • Document review (Justice Canada Client Feedback Survey)
Ongoing Litigation Branch
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

5. To what extent does the Litigation Branch provide consistent advice and take consistent legal positions?

  • Existence of structures to ensure consistency (e.g., Litigation committees, practice groups, practice directives, knowledge management system, Justice National Security and Intelligence Committee)
  • Use of consultations for the purpose of ensuring consistent advice is given and consistent legal positions are taken across government
  • Assessment of the consistency in approach to legal issues
  • Document review
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

6. Are legal risks identified and assessed in a timely and consistent manner?

  • Use of processes, tools, and standards for assessing legal risk
  • Document review
  • Key informant interviews
  • File review
  • Case studies
Evaluation Evaluation Division
  • Number and percentage of files “unable to assess” and length of time before assessment entered
  • Average number of risk assessments/reassessments per file
  • Document review (iCase reports)
  • File review
  • Case studies
Evaluation Evaluation Division
  • Consistency of language used in assessing and communicating legal risk
  • Consistency of risk ratings
  • Timeliness of assessments/ reassessments of legal risk
  • Use of consultations to discuss and assess legal risks
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

7. Has the Litigation Branch contributed to the enhanced understanding of legal issues, their implications, and potential risks by decision-makers?

  • Memoranda of understanding outlining roles and responsibilities related to legal risk
  • Communication of legal risk management strategies and mitigation measures to client departments
  • Assessment of the Litigation Branch contribution to improved understanding of legal issues and legal risks by clients
  • Document review
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

8. Do counsel have the appropriate training, tools, structures, and resources to support the delivery of legal services?

  • Nature and type of tools and structures
  • Best practices
  • Practice directives
  • Practice groups
  • Committees
  • Use of tools and structures
  • Document review
  • Key informant interviews
  • File review
  • Case studies

Evaluation

Evaluation Division

  • Satisfaction with tools, structures, and resources
  • Nature and frequency of training
  • Alignment of training with individual learning plans
  • Number of counsel and support staff trained
  • Document review (reports on training activities)
Ongoing Litigation Branch
  • Satisfaction with training
  • Key informant interviews
Evaluation Evaluation Division
  • Level and sufficiency of resources (human, financial, information, training, technological) to support Litigation Branch work
  • Document review
  • Key informant interviews
Evaluation Evaluation Division

9. To what extent does the Litigation Branch contribute to informed ministerial decision making?

  • Provision of timely legal advice with options, implications, and recommendations
  • Development of operational frameworks and policies to guide decision making (e.g., Legal Services Contracting Framework)
  • Decision-making processes are followed and documented (e.g., for decisions on court interventions, appointing agents, and whether to surrender individuals to foreign countries)
  • Key informant interviews
  • File review
Evaluation Evaluation Division

10. To what extent are delegated ministerial functions effectively managed?

  • Decision-making and monitoring processes are followed and documented (e.g., for delegated responsibilities under the Extradition Act, the Mutual Legal Assistance in Criminal Matters Act, and section 38 of the CEA)
  • Appropriate information management practices are followed in offices with delegated functions to support accurate measurement of resource utilization
  • Key informant interviews
  • Case studies
  • File review
Evaluation Evaluation Division
Intermediate outcomes

11. To what extent does the Litigation Branch provide effective advocacy of a whole-of-government approach?

  • Assessment that an integrated, whole-of-government approach is used in provision of legal services, where appropriate
  • Counsel work collaboratively within Justice
  • Counsel consult with clients and other stakeholders to develop legal positions and strategies that consider the interests of government, where appropriate, as opposed to exclusively considering individual client interests
  • Use of peer review processes, such as Litigation committees
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

12. Is Litigation Branch legal advice considered by government decision-makers?

  • Consideration of Litigation Branch legal and legal policy advice by decision-makers in program and policy development (e.g., evidence of consideration of legal advice in client instructions)
  • Litigation Branch and client’s assessment of legal advice in terms of :
  • involving the client in development of legal strategy and positions
  • developing legal strategies appropriate to the client’s policy and/or program objectives
  • Document review (Justice Canada Client Feedback Survey)
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

13. Are legal risks managed and mitigated effectively?

  • Consideration of legal risk in the development and implementation of client policies, programs, and litigation strategies
  • Use of legal advice to prevent, mitigate, and/or manage legal risk
  • High-risk files are resourced and communicated appropriately (e.g., early warning reports, level of senior counsel involvement)
  • Contingency plans are used for high-risk files
  • Assessment of the effectiveness of management and mitigation strategies
  • Document review (Justice Canada Client Feedback Survey, iCase)
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

14. How does the Litigation Branch contribute to Justice Canada’s support of a bijural and bilingual legal system?

  • Consultations regarding bijural aspects of legal issues
  • Practice groups and other structures within the Litigation Branch that involve practitioners of both legal systems
  • Internal policies that reflect and support the bijural and bilingual legal system
  • Assessment of the Litigation Branch and litigation function as a centre of expertise for providing litigation services in a bijural and bilingual legal system
  • Document review
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

Ultimate outcome

15. To what extent does the Litigation Branch contribute to a federal government that is supported by effective and responsive legal services?

  • Contribution of Litigation Branch to provision of effective, responsive legal services
  • Document review
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

16. To what extent does the Litigation Branch contribute to a fair, relevant, and accessible justice system that reflects Canadian values?

  • Contribution of Litigation Branch to a fair, relevant, and accessible justice system that reflects Canadian values
  • Document review
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division

17. What factors are contributing to or constraining the achievement of expected outcomes?

  • Constraints on ability to achieve outcomes
  • Factors supporting achievement of outcomes
  • Document review
  • Key informant interviews
  • Case studies
  • Focus groups
Evaluation Evaluation Division
Performance of Litigation Branch(efficiency, economy): TB core issue 5

18. Is the Litigation Branch facing any challenges in terms of available funding, organizational structure/staffing, or internal support (training, technological, research)?

  • Forecasted resource needs compared to available resources
  • Identified resource challenges
  • Assessment of the effectiveness of Litigation Branch organizational structure
  • Document review
  • Key informant interviews
  • Case studies
  • Focus groups
Evaluation Evaluation Division

19. Could the work of the Litigation Branch be undertaken/conducted more efficiently and economically?

  • Mix of legal resources (FTEs/hours by group and level) by file complexity and level of risk
  • Resources are allocated in line with Business Plan and agreements with clients
  • Document review (including iCase data)
Ongoing Litigation Branch
  • Measures in place to manage efficiency and cost of legal services and their effectiveness (e.g., Law Practice Model; practice groups; use of standardized processes, practices, approach to assigning work and determining when to use agents, information management tools such as iCase)
  • Suggestions for improvements in efficiency
  • Document review
  • Key informant interviews
  • Case studies
  • Focus groups
Evaluation Evaluation Division
  • Alternative approaches to service delivery
  • Trends in law practice/service delivery models
  • Literature review
Evaluation Evaluation Division
  • Comparison of billing rates with private bar
  • Comparison of rates (iCase data with private bar rates from Canadian Lawyer survey and/or queries with large and mid-size law firms about rates for litigators based on years of experience)
   

20. What is the role of the client in the effectiveness, efficiency and economy of legal services?

  • Role and responsibility of the client in managing demand for legal services
  • Assessment of the client’s effectiveness in managing its demand for legal services
  • Use of early dispute resolution by the client to settle files without involving Litigation Branch services
  • Receptiveness to use of alternative dispute resolution practices
  • Timely involvement of Litigation Branch by the client
  • Document review
  • Key informant interviews
  • Case studies
  • Focus groups
Evaluation Evaluation Division

21. Is Litigation Branch able to achieve appropriate resolution of litigation cases in a timely, cost-effective manner?

  • Cost of legal services relative to file outcome (success rate), and impact
  • Hours spent per closed file by complexity and legal risk
  • Number and percentage of files settled/closed without going to court
  • Document review (iCase)
Ongoing Litigation Branch
  • Use and effectiveness of strategies to promote timely, cost-effective resolution
  • Early resolution strategies
  • Alternative dispute resolution practices, where appropriate
  • Assessment of whether litigation cases are resolved in timely, cost-effective manner
  • Key informant interviews
  • File review
  • Case studies
  • Focus groups
Evaluation Evaluation Division