Department of Justice Canada Client Feedback Survey

PDF Version

Annex A – Methodology

Introduction

With the expert guidance of the Statistical Consultation Group at Statistics Canada, the Department of Justice developed a standardized questionnaire and methodology for collecting client feedback on the degree to which the delivery of legal services is meeting the needs and expectations of clients. Statistics Canada played an important role by reviewing and challenging the proposed approach throughout the design and implementation stages, vetting the analyses of survey data and reviewing and commenting upon the presentation of findings contained in all reports.

The Department launched the second cycle of the Client Feedback Survey in November 2009. Potential respondents received invitations to complete the standardized questionnaire, which covers the legal advisory, litigation, legislative drafting, and regulatory drafting services provided by the Department of Justice Canada.

The first cycle of the Survey was launched in 2006, beginning with the Aboriginal Affairs Portfolio as the pilot project. Based on lessons learned during the first survey cycle, there were some changes made to the questionnaire and to the survey administration process. The questionnaire has been aligned to the Department’s Service Standards, offering an opportunity to obtain feedback on performance against the Standardized Legal Service Agreements. The survey administration process has also been improved, resulting in much higher response rates.

Potential Respondents and Census Approach

Invitations were sent to potential respondents at the EX minus one and equivalent levels through to Deputy Heads of client departments and agencies in the National Capital Region (NCR) and across the country. However, in an effort to maximise the response from clients outside the NCR and to give regional management a more accurate portrayal of client perceptions of legal services, invitations to complete the questionnaire were also sent to employees at EX minus two and equivalent levels who were located in the regions.

Justice adopted a census approachFootnote 12 to the Survey because the target population is of a manageable size and potential sources of error associated with sampling are avoided. The Survey was administered via a web-based questionnaire housed on a Department of Justice Canada server.

In total, 26,012 invitations to complete the questionnaire were successfully delivered via email to potential respondents across the country. Of these, 12,390 completed the questionnaire, resulting in an overall response rate of 48%. This is a significant improvement in the response rates achieved during Cycle I (33%) and boosts confidence in the precision of the Survey results.

Approximately 39% of respondents reported having used departmental legal services in the twelve months preceding the administration of the Survey. Unless otherwise noted, all reported results for Cycle II are based on the feedback from these 4,786 service users.

Interpreting Results

The Survey collected feedback from clients using a 10-point Likert scaleFootnote 13 with two anchors: not at all satisfied (1) and completely satisfied (10). Feedback was sought along three key dimensions of service quality – accessibility/responsiveness, usefulness, and timeliness – and collected through a number of individual elements of client satisfaction, many of which relate directly to the Department’s Service Standards for legal services. Further to this, service users were asked to rate their level of satisfaction with the overall quality of legal services.Footnote 14

It should also be noted that a weighting strategy adopted for the first Client Feedback Survey cycle has been discontinued at the recommendation of Statistics Canada. As a result, any references to the Cycle I and II survey scores, ratings, or results for elements of satisfaction now refer to comparable unweighted data. Due to this change in methodology, some results may differ from previously published values.

Margins of Error

In reviewing the results presented throughout this report, it is important to remember that survey results represent estimates of client population perceptions of service delivery. As such, there is an important caveat to bear in mind, namely the calculated margins of error. The magnitude of the margin of error is generally affected by the extent of variabilityFootnote 15 in respondent feedback and by the overall size of the respondent group.

There are two key elements to calculating the margins of error from survey findings. First, there is the confidence level which, in the most simplistic terms, refers to the extent to which it is believed the same results would be obtained if the Survey were administered repeatedly. For the purposes of the Justice Client Feedback Survey, a 95% confidence level was adopted for calculating results. Consequently, a statistically significant difference indicates that there is a less than 5% probability (p<0.05) that the result occurred by chance. Second, and more importantly, there is the confidence interval, which refers to the range in which the results will fall if the measurements are repeatedly taken.Footnote 16

The confidence intervals presented account for variability related to non-response. Had all service users responded to the Survey, there would be no variability, as all opinions would be accounted for. In the calculation of the confidence interval, it is assumed that non-response is independent of respondent characteristics but is affected by use of legal services (i.e. actual service users are more likely to answer the questionnaire). It is a reasonable assumption that a relatively large proportion of non-respondents are non-users. The Finite Population Correction Factor has been applied in the calculation of the margin of error in order to take the size of the total number of potential users into account; otherwise the margins of error would be overstated.

Date modified: