Litigation Branch Evaluation Final Report
3. Methodology
The Litigation Branch evaluation made use of multiple lines of evidence in order to support robust findings. The methodology included five main lines of evidence: a document and data review, key informant interviews, a file review, case studies, and a survey of legal counsel.
The evaluation matrix (which links the evaluation questions, indicators and lines of evidence) and the data collection instruments were developed with the input of the Litigation Branch Evaluation Working Group. The evaluation matrix is included in Appendix B, and the data collection instruments are in Appendix C.
Each of the evaluation methods is described more fully below. This section also includes a brief analysis of methodological challenges.
3.1. Document and Data Review
The document and data review was conducted both to inform the development of data collection instruments and to address the majority of the evaluation questions.
Documents reviewed were obtained from internal, external and publically available sources. Departmental documents reviewed included: Departmental Performance Reports (DPRs), Reports on Plans and Priorities, the results from Public Service Employee Surveys (PSES), Footnote 7 and Client Feedback Survey results. Footnote 8 Internal Litigation Branch documents were also reviewed, as well as publically available information, such as budget speeches and Speeches from the Throne.
In addition to documents, the evaluation involved the review of iCase data from fiscal years 2008/09 to 2013/14. iCase is the Department's integrated case management, timekeeping and billing, document management, and reporting system. Given their small number of files, LPMC, National eDiscovery and Litigation Support Services and ADAG Litigation were not included in the iCase analysis.
3.2. Key Informant Interviews
The key informant interviews conducted for this evaluation addressed the majority of evaluation questions, and represented a key line of evidence in gathering information on the need for the Litigation Branch and the effectiveness of Branch activities. A list of potential key informants was prepared, and interview guides tailored to each key informant group were developed in consultation with the Evaluation Working Group. Interviews were conducted with 58 key informants representing the Litigation Branch (n=24); regional offices, including Regional Directors General, regional Litigation Committee chairs, and regional IAG delegates (n= 14); portfolio representatives and DLSUs (n=14); and client departments/agencies (n=6).
The following scale has been applied to report on interviews:
3.3. File Review
A review of a selection of closed files was conducted to allow for a more in-depth understanding of the life of a file in relation to the performance measures for the Litigation Branch. This method also allowed the evaluation to explore whether the information obtained from key informants on how the Litigation Branch conducted its work was supported by a review of selected case files.
The file review involved the examination of iCase data and, when available, hard copy files for 62 files: 33 CLS files, 32 of which were from six “suites” of related files at different levels of court; five NSG files; five IAG files; six related MCAMLU files; six LPMC files; and seven files (of which five were related) from regional offices that involved the Office of the ADAG Litigation and/or the NLC.
The sample of files was chosen with the input of the Evaluation Working Group and was considered to provide a good selection of the broad spectrum of the work of the Litigation Branch, but a particular focus was higher profile (risk and complexity) files. As files were not chosen randomly, and as the sample is not large (considering the thousands of files worked on by Litigation Branch counsel during the time period covered by the evaluation), the file review sample is not a strictly representative one. Rather, the file review was intended to be illustrative of the Litigation Branch’s approach to its work.
To protect confidential information, as well as solicitor-client privilege, staff from the Evaluation Division and, as needed, counsel from the Litigation Branch, conducted the file review. Footnote 9 To ensure that comparable information was collected from the files, counsel completed the standard file review template developed for the study. The template collected information to respond to the evaluation matrix and focused on factual information available in the files.
3.4. Case Studies
Three case studies of litigation files were conducted to allow for an exploration of best practices and lessons learned. For each case study, file review templates (completed as part of the file review conducted for the evaluation) were examined. In addition, telephone interviews were conducted to supplement documented information, to provide context for the work, and to allow for a more in-depth assessment of the way the file was handled and the effectiveness of the working relationship between the Litigation Branch, other areas of Justice (i.e., DLSUs, regional offices, specialized sections within headquarters), and the client representatives. Interviews involved Branch counsel, other counsel within Justice Canada, and client department/agency representatives who worked on the file that was the subject of the case study (including headquarters and regional representatives from each organization). Interviews were conducted with 11 stakeholders.
3.5. Survey of Legal Counsel and Paralegals
To gather the input of all Litigation Branch counsel and paralegals, the evaluation included an anonymous and confidential web-based survey that was delivered by the Department of Justice. The survey was online for a total of three weeks — from June 17 to July 9, 2014. Invitations were sent to all 138 counsel and paralegals of the Litigation Branch. A reminder was sent to participants in order to increase the response rate. Fourteen of these participants were not available during the time the survey was online. In total, 79 respondents completed the survey for a response rate of 64%. Footnote 10 Once the survey was finished, open-ended questions were coded and the survey data was analyzed using SPSS, a statistical software package.
Table 3 provides a profile of survey respondents and shows that, generally, respondents were representative of the population of Litigation Branch counsel and paralegals.
Table 3: Compçarison of Litigation Branch and Survey Respondent Profiles
| Characteristics | Litigation Branch (N=138) Footnote 11 | Survey Respondents (N=79) | ||
|---|---|---|---|---|
| Number | Percentage | Number | Percentage | |
| Paralegal (EC) | 46 | 33% | 26 | 33% |
| Counsel: LP-01 | 22 | 16% | 13 | 16% |
| Counsel: LP-02 | 37 | 27% | 18 | 23% |
| Counsel: LP-03 | 15 | 11% | 9 | 11% |
| Counsel: LP-04 | 9 | 7% | 6 | 8% |
| Counsel: LP-05 | 4 | 3% | 3 | 4% |
| Counsel: LC-01 to 04 | 5 | 4% | 4 | 5% |
| Characteristics | Litigation Branch (N=138) | Survey Respondents (N=79) | ||
|---|---|---|---|---|
| Number | Percentage | Number | Percentage | |
| >Less than 6 years ago | 33 | 24% | 17 | 22% |
| Between 6 and 10 years ago | 41 | 30% | 20 | 25% |
| More than 10 years ago | 64 | 46% | 42 | 53% |
| Characteristics | Litigation Branch (N=138) | Survey Respondents (N=79) | ||
|---|---|---|---|---|
| Number | Percentage | Number | Percentage | |
| CLS | 59 | 43% | 34 | 43% |
| MCAMLU | 10 | 7% | 4 | 5% |
| LPMC | 4 | 3% | 4 | 5% |
| NSG | 19 | 14% | 15 | 19% |
| IAG | 30 | 22% | 15 | 19% |
| National eDiscovery and Litigation Support Services | 10 | 7% | 4 | 5% |
| Office of ADAG | 6 | 4% | 3 | 4% |
| Characteristics | Litigation Branch (N=138) | Survey Respondents (N=79) | ||
|---|---|---|---|---|
| Number | Percentage | Number | Percentage | |
| Litigation | Not available | Not available | 54 | 68% |
| Advisory | 21 | 27% | ||
| Policy development | 2 | 2% | ||
| Other | 2 | 2% | ||
3.6. Limitations
The evaluation faced a few methodological limitations. These are listed below by line of evidence.
Document and data review: iCase data.
Overall, iCase was a useful source of information for the evaluation. There were, however, some changes in Justice policy that affected iCase data and, consequently, what data were included in the evaluation. Data on the number of files, number of hours, legal issues and outcomes were included for the full evaluation period (2008/09 to 2013/14). However, iCase data on legal risk and complexity were not included for the 2013/14 fiscal year for two reasons. First, the methodology for assessing legal risk has changed, so that the risk levels for that year are not comparable to earlier years’ assessments. Footnote 12 Second, when the new risk rating took effect for advisory and legislative files on September 30, 2013, all open files of those types had their risk rating set to “not yet evaluated” to increase the accuracy and reliability of the data going forward. However, based on earlier data, many Litigation Branch units were already assessing legal risk for their advisory and legislative files before September 2013. Consequently, when data were re-pulled to obtain 2013/14 data, those risk assessments were now gone. As a result, the decision was made to rely on data from 2008/09 through 2012/13 for legal risk and complexity.
Late in the evaluation, it was discovered that the iCase reports included time recorded by non-Litigation Branch staff to Litigation Branch files. In addition, the iCase reports did not include hours recorded by Branch staff to files held by other areas of Justice Canada. This situation does not, however, substantially change the results reported. In fact, for the four units of the Litigation Branch that are most engaged in legal advisory and litigation files, well over 90% of the time recorded to that unit’s files was recorded by Litigation Branch staff. In addition, almost all of the legal service hours for each unit are included in the iCase reports (see Table 4). Because the Office of the ADAG and the National eDiscovery and Litigation Support Services record substantial amounts of time to non-Litigation Branch files, they are not included in the iCase analysis of hours. The LPMC does not record time in iCase and is, therefore, also not included.
| Litigation Branch Unit | Percentage of total time recorded by staff from Unit | Percentage of legal services work of Unit accounted for in iCase data |
|---|---|---|
| CLS | 94.0% | 98.0% |
| MCAMLU | 95.0% | 96.0% |
| NSG | 99.6% | 99.7% |
| IAG | 97.3% | 99.9% |
Source: iCase data.
Interviews, case studies and the survey
The interviews with key informants and case study participants, as well as the survey of Litigation Branch staff, have the possibilities of self-reported response bias and strategic response bias. Self-reported response bias occurs when individuals are reporting on their own activities and so may want to portray themselves in the best light. Strategic response bias occurs when participants answer questions with the desire to affect outcomes.
File review
In any given year during the evaluation (2008/09 to 2013/14), the Litigation Branch actively managed in excess of 3,000 files. To obtain a representative sample was not feasible. Instead, the evaluation relied on the opinion of the Litigation Branch units to select files that they believed reasonably represented their work.
Mitigation strategy
The mitigation strategy for the above methodological limitations was to use multiple lines of evidence. The evaluation gathered information from the Litigation Branch and those using Branch services, from management and “front line” staff, and from a review of files and more comprehensive administrative data review (iCase). The mitigation strategy also included using both quantitative and qualitative data collection methods to answer evaluation questions. By using triangulation of findings from these different sources, the evaluation was able to strengthen its conclusions.
- Date modified: