Aboriginal Justice Strategy Evaluation, Final Report

APPENDIX A: Evaluation of the Impact of the Aboriginal Justice Strategy on Rates of Re-offending

This appendix provides a condensed and non-technical summary of the study of the impact of the AJS on rates of re-offending.

Background

In 2000, an evaluation[28] of the Aboriginal Justice Strategy by the Department of Justice Canada included a preliminary statistical analysis of the impact of the Strategy on rates of re-offending (recidivism) for five community-basedprograms. This study was expanded in 2006[29], when the same statistical analysis was conducted on data from nine AJS-funded programs (four of which had also been included in the 2000 study). These studies were based on a comparison of recidivism rates for Aboriginal offenders who participated in one of the selected programs to a comparison group of offenders who were referred to a program but did not participate. While the results were generally positive in both studies, their results can only be interpreted with caution due to limited generalizability, a quasi-experimental approach, and a lack of true control group.

The current study builds on the two previous ones with the goals of including more programs, in a greater number of provinces/territories, and increasing the size of the participant group. The current study expanded the sample size, from 4,246 to 5,141; the size of the participant group, from 3,361 to 4,570; and the number of AJS-funded community programs represented in the analysis, from nine to 25.

Methodology

This study compared the likelihood of re-offending of individuals who participated in an AJS program from January 1, 1998 to December 31, 2007, with that of individuals who were referred to, but did not participate in, an AJS program during the same period of time. It was intended to provide insights into the impact of the AJS programs on clients' likelihood of re-offending over time.

The methodology developed for this study was based on the principle of replicating and extending the quasi-experimental approach taken for the two previous recidivism studies in 2000 and 2006. However, the number of programs participating in the study was greatly expanded: from nine in the 2006 study to 25. Seven of the nine programs who participated in the 2006 study consented to continued participation and provided updated information on individuals referred; this included four of the five programs who participated in the first study, conducted in 2000.

Survival analysis was the statistical approach used to model the likelihood of re-offending. This method is ideally suited to modeling the occurrence and timing of events, particularly where the event (re-offending) has not occurred to all individuals in the sample by the end of the observation period (known as “right-censored” data). Further, because there were a number of intervening variables to be controlled for, it was necessary to utilize a regression procedure that allowed isolation of the impact of participating in the program. The standard survival regression procedure used for these types of analyses is the Cox Proportional Hazards Model[30], which was also used in the 2006 and 2000 recidivism studies.

Criminal history information for adults and youth[31] was obtained from the Canadian Police Information Centre (CPIC) of the Royal Canadian Mounted Police.

Offenders who participated in an AJS program, whether or not they successfully completed the program, are referred to throughout this document as “program participants.” Offenders who were referred to an AJS program but did not participate are referred to as “comparison group members”. Criminal behaviour is defined in terms of criminal offences that result in convictions (or findings of guilt in the case of young offenders).[32]

Recidivism was defined, for the purpose of this study, as a criminal conviction following participation in the program. While all charge information is available from CPIC, restricting the definition of recidivism to convictions allows for a higher degree of certainty that an offence had been committed. Using the end date of the program as the reference point, convictions were identified as being either pre-program offences or post-program offences[33]. In keeping with past practice, administration of justice offences (such as violations of probation conditions, failure to appear, etc.) were excluded from the individuals' offence histories. It has been argued that these types of offences are not predicate offences and may not rise to the same level of seriousness as other offence types.

Consistent with the approach taken for the 2000 and 2006 studies, an offence that occurred while in the program was not considered to be a post-program conviction; typically, CPIC dates offences using the disposition date and in some cases there is uncertainty surrounding the timing of these offences. Coders manually verified whether offences near the time of participating in the program were to be considered as pre-program offences or post-program offences.

For the analysis, the time elapsed after completing the program until receiving a criminal conviction or, in the absence of a conviction, the time to the end of the observation period (December 31, 2010) was statistically modeled. This elapsed time was modeled as a function of age, sex, number of pre-program convictions (the intervening variables), and participation or not in an AJS program. The sample was analyzed as a whole for the purposes of this report, although we also conducted analyses at the provincial and program levels when the sample was sufficiently large.

Limitations to Methodology

Participation in this study was voluntary; therefore, the 25 programs willing to participate cannot be considered randomly selected as they agreed to participate with many others did not. Additionally, only programs from provinces and territories whose representatives had signed a formal agreement with the Department of Justice in support of the study were invited to participate.

The quasi-experimental design employed for this study did not permit the use of a randomized control group of individuals not participating in a program, nor did it permit for a randomization of individuals who participated in a program. Differences in the characteristics of the two groups are to be expected, reflecting the reasons for participating or not participating in the program. For example, exclusion from a program may have been the result of circumstances of the offence, previous criminal history or attitudinal issues. A simple comparison of the re-offending experience of program participants with non-participants could be misleading in such circumstances, as it is possible that the characteristics that draw offefnders to participate in AJS-funded community-based justice programs may also predispose them to lower recidivism rates than those offenders who were referred to, but did not participate in, those programs.

This quasi-experimental design leaves open the likelihood of selection bias in the results obtained. The fact that the comparison group consists of program referrals who did not ultimately participate in the program further increases the likelihood that selection bias is almost certainly present in the analysis. Examining the differences in characteristics between the two groups shows that there are some statistically significant differences, though not necessarily in the expected direction (for example, comparison group members tend to be slightly older but have relatively fewer previous convictions of some types of offences).

This problem was mitigated to an extent with the use of regression analysis which controls for these differences, but the ability to do so is restricted by limitations in the available information known about the individuals in the sample. We are largely limited to taking into account, and controlling for, the individuals' age, gender and previous criminal background. This allowed the analysis to control for underlying differences in offender characteristics which correlate with (but do not cause) criminal behaviour; in this way, recidivism rates among program participants and non-participants were compared in order to try to isolate any incremental impact attributable to the program itself.

Findings

The program participants and comparison group members tended to be similar in background characteristics but some key differences between the two groups were identified: comparison group members tended to have fewer prior convictions and to be slightly older.[34] These differences are presented in Table 8 on the following page.

Table 8: Characteristics of Offenders in the AJS Recidivism Study

Number of Prior Drug Convictions (%)
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
0 94.9 96.5 96.1
1-5 4.9 3.3 4.7
6 or more 0.2 0.2 0.2
Mean 0.09 0.06 0.09

Number of Prior Violent Convictions (%)
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
0 81.5 82.3 81.6
1-5 15.1 15.8 15.2
6 or more 3.4 1.9 3.2
Mean 0.65 0.45 0.63

Number of Prior Non-violent Convictions (%)
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
0 74.5 70.7 74.1
1-5 18.0 20.5 18.3
6 or more 7.5 8.8 7.7
Mean 1.40 1.53 1.41

Year of Program Completion (%)
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
1997-99 12.0 17.3 12.6
2000-03 33.7 39.2 34.3
2004-07 48.0 40.5 47.2
2008-10 6.3 3.0 6.0

Age of Program Completion (%)[35]
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
17 and under 12.6 0.2 11.2
18-24 42.2 51.8 43.2
25-34 22.0 25.2 22.3
35-44 14.1 14.2 14.1
45 and over 9.2 8.6 9.1
Mean 27 28 27

Gender of Program Completion (%)[35]
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
Male 58.0 56.2 57.8
Female 42.0 43.8 42.2

Province of Program Completion (%)[35]
  Program Participants
(n = 4,570)
Comparison Group
(n = 571)
Total Sample
(N = 5,141)
New Brunswick 3.4 0.0 3.0
Nova Scotia 1.0 0.0 0.9
Ontario 28,0 0.0 24.9
Saskatchewan 64.8 100.0 67.8
British Columbia 3.9 0.0 3.4

Four covariates were considered in the survival analysis: participation in an AJS-funded program, number of prior convictions, gender (man, woman), and age. The regression results for the national-level analysis are presented in Table 9. These results demonstrate that the overall model was statistically significant, suggesting that the combination of the covariates (i.e., AJS participation, age, sex, and number of prior convictions) is significantly associated with the probability of re-offence over time.

Table 9: Cox Regression of Recidivism – National Results (N = 5,141)
Variable Coefficient Estimate Standard Error Chi-Square P-Value Hazard
Ratio (eβ)
AJS Participation
(comparison: 0; program: 1)
- 0.553 0.068 66.18 < 0.0001 0.57
Gender
(women: 0; men: 1)
0.275 0.053 27.23 < 0.0001 1.32
Age
(years)
- 0.031 0.003 123.08 < 0.0001 0.97
Prior Convictions - Drugs
(number)
0.019 0.043 0.19 0.6634 1.02
Prior Convictions - Violence
(number)
0.088 0.009 85.86 < 0.0001 1.09
Prior Convictions – Non-violent
(number)
0.041 0.005 73.84 < 0.0001 1.04

These results indicate that the probability of re-offence over time was correlated with each of the covariates, after holding the other covariates constant. Most notably for the purposes of this study, participation in an AJS program was significantly correlated with a lower likelihood of recidivism[36].

In terms of the Cox regression model, the censoring variable was recidivism (no re-offence, re-offence) and the dependent variable was the time elapsed between the end of the participant's involvement in the program and either re-offence or the end of the data collection phase. The results presented in Graph 2 and Table 10 demonstrate the cumulative percentage of offenders who recidivated during the observation period, by program participation (the analysis that produced these results controlled for differences in age, sex, and number of prior convictions between the participant and comparison group):

Graph 2: AJS Average Recidivism Rate

img/graph2.jpg


Table 10: AJS Average Recidivism Rates
Time After Program Completion Cumulative Percent who have Re-offended
Participants Comparison Group
1 year 10.9 18.2
2 years 17.6 28.5
3 years 22.0 35.1
4 years 24.8 39.1
5 years 27.2 42.4
6 years 28.7 44.5
7 years 30.4 46.7
8 years 32.0 48.8

Note:  Recidivism rates are fitted from the proportional hazards model and are based on the average characteristics of the national sample: 

  • number of prior convictions – drug (mean = 0.09)
  • number of prior convictions – violence (mean=0.63)
  • number of prior convictions –non-violent (mean=1.41)
  • age (mean=27)
  • gender balance (0.58)

Rates of re-offending were found to be significantly lower among program participants at every point in time after completing the program:

  • At one year, 18.2% of comparison group members had been convicted of at least one other crime compared with 10.9% of AJS program participants.

  • At four years, 39.1% of comparison group members had re-offended compared with only 24.8% of AJS program participants.

  • At eight years, 48.8% of comparison group members had re-offended compared with 32.0% of AJS program participants.

Although these findings should be interpreted with caution, given the methodological limitations described in Section 3.5, they suggest that AJS-funded programs are associated with the intended long-term outcome of reducing crime. These findings are in line with results of recidivism studies conducted in 2000 and 2006. The differences between the likelihood of re-offending for AJS program participants and non-participants is particularly pronounced in the years immediately following the program, but the cumulative effects, even after 8 years, remain.

Conclusions

An intended outcome of the AJS is to contribute to decreasing rates of victimization, crime and incarceration among Aboriginal persons participating in AJS-funded programs. One indicator for measuring the effectiveness of the AJS in meeting this objective is through the rates of crime and recidivism among those offenders who participate in the community justice programs funded under the AJS. It is an aim of the AJS to reduce the rates of re-offending for offenders who have participated in AJS-funded community programs.

The analyses presented in this report were carried out in an effort to assess whether a measurable link could be established between offender participation in these community justice programs and their likelihood of re-offending. As a whole, findings derived from these analyses show strong support for the existence of such a link. Most notably, at the national, provincial and program levels, offenders who were referred to, but did not participate in, an AJS program were more likely to re-offend, compared with offenders who participated in the AJS program. Further, while it was not specifically measured as part of this study, a logical extension of this finding is that rates of incarceration may have been similarly reduced.


  • [28]  Justice Canada. Final Evaluation: Aboriginal Justice Strategy, October 2000.

  • [29]  Justice Canada. Evaluation of the Impact of the Aboriginal Justice Strategy on Rates of Re-offending, July 2006.

  • [30]  As a regression procedure, an advantage of the Cox approach is its ability to simultaneously control for intervening variables (“covariates”) and to provide a quantitative estimate of the impact of each covariate on the likelihood of recidivism. Standard OLS or logistic regression is not appropriate in cases of right-censored data. Another competing class of models includes Accelerated Failure Time (AFT) regression, but they are less flexible and often considered to be less robust than the Cox approach.

  • [31]  In accordance with the Youth Criminal Justice Act, judicial authorization was obtained prior to the release of information on young offenders (i.e., offenders under the age of 18).

  • [32]  Throughout the report, “convictions” refers to both convictions under the Criminal Code and also to “findings of guilt” under the Youth Criminal Justice Act.

  • [33]In cases where programs were only able to provide a start date, it was assumed that the program was completed after three months.

  • [34]  These group differences might be due to selection bias (i.e., the background characteristics of an offender may influence whether or not this individual is selected—or self-selects—to participate in an AJS program).

  • [35]For the comparison group, age at program completion is three months following referral to the program.

  • [36]  , the hazard ratio, is interpreted as the percent difference in the likelihood of re-offending per unit change in the covariate.

Date modified: