National Anti-Drug Strategy Implementation Evaluation, Final Report

2. Methodology

This section of the report describes the methodology used to complete the Implementation Evaluation of the Strategy.

2.1. Evaluation framework

The approach taken for this evaluation was developed collaboratively by the partner departments and is included in the RMAF. In addition, it should be noted that although outcomes were to be included, the implementation evaluation was limited in its ability to report on outcomes, given the relative newness of the Strategy and the performance reporting capacity of the participating departments.

2.2. Data collection methods

The evaluation methodology consisted of document and file review, in-depth interviews (n=85), and three focus groups with departmental personnel (n=19). Triangulation was used to verify and validate the findings obtained through these methods and to arrive at the overall evaluation findings.

2.2.1. Document and file review

The document and file review included both Strategy-level files and departmental files. Strategy-level files include documents, briefing papers, and correspondence created by the Assistant Deputy Minister Steering Committee, working groups, and the DOJ Youth Justice, and Strategic Initiatives and Law Reform section. Departmental files include program-related documentation, research studies, and evaluations; project-related reports, evaluations, publications, and other products; program management information systems to manage cases, projects, and statistical and other administrative information; and departmental performance reports completed in response to Strategy requirements.

2.2.2. Interviews

A total of 85 individuals were interviewed, including:

Members of the Subcommittee on Evaluation and Reporting identified an initial list of potential interviewees. All individuals on the list received an initial form of communication, either from their department/agency (in the case of ADMs and departmental personnel) or from the Department of Justice (DOJ) Evaluation Division (in the case of external stakeholders and funding recipients). The initial communication explained the purpose of the evaluation and invited individuals to participate in an interview.

All of the individuals on the initial list were contacted for an interview. A small number declined to participate. In some cases, those who declined identified others within their organizations as more suitable candidates for an interview. The interviews were conducted by telephone in the preferred official language of interviewees.

The following scale has been used to analyze the key informant data for this report.

A few; Some; Around half; Most; All

2.2.3. Focus groups with departmental personnel


Finally, three focus groups were conducted with departmental personnel involved in Strategy implementation. The purpose of this line of evidence was to validate the evaluation findings, to explore possible conclusions, and, in light of the preliminary evaluation findings, to explore possible solutions for the final report.

The focus groups were conducted after submission of the preliminary results and presentation of the preliminary findings to the SER. By then, the salient evaluation issues had emerged. The SER identified three topics that it wanted to explore more fully in the final line of evidence. The moderator's guides were subsequently developed and were based on a preliminary analysis of the findings.

Each focus group was devoted to examining one of the following broad themes: 1) design and perceptions of the Strategy; 2) governance and horizontality; and 3) performance measurement and reporting. Participants were identified by members of the SER and included senior managers, such as Directors and Program Managers; although all partner departments and agencies were invited to identify participants, actual participation was optional. A total of 19 individuals participated in the focus groups; some individuals participated in more than one group. By department/agency, participation in the focus groups was as follows:

All of the focus groups were held on September 16, 2009 in Ottawa. The ideas and suggestions generated through the focus group discussions have been incorporated into the relevant sections of this report, and have helped to frame the overall conclusions of the evaluation.