Evaluating Restorative Justice — According to Its Aims

The online conference held on 5 November 2021 ‘Measuring, researching, narrating: Discussing the (social) impact of restorative justice’ was incredibly inspiring to us, which is why the Editorial Committee chose to focus the next series of articles on these topics. We will cover the following four thematic areas with one or more articles:

• efficiency and evaluation of restorative justice programmes,
• from evaluation towards the social impact of restorative justice,
• the role of international and local organisations in fostering restorative justice with different approaches, and
• the various forms in which the impact of restorative justice can be narrated.

Hoping to have stimulated your curiosity, we have the pleasure of opening the series with an article by Joanna Shapland, Edward Bramley Professor of Criminal Justice at the University of Sheffield (UK), who will discuss why it is important to evaluate restorative justice and which aspects should be taken into account when doing so. (If you were not able to attend the conference, you can find more details and the recordings of the presentations given during the event by following the link above.)

(The editors of the series: Emily Molinari & Silvia Randazzo) 

Evaluating restorative justice or restorative practice sounds like a very technical thing, perhaps one to be left to researchers or specialists, and far removed from the day-to-day business of responding to referrals, telling potential participants about restorative justice and delivering a service to those who have been harmed, or have harmed. But, though there are some key principles to bear in mind, actually evaluation — and its precursor, monitoring — are key elements for all those delivering services. They let the scheme, and those funding it or thinking about participating, know whether what they are intending to deliver is actually being delivered. I first consider why it’s important to evaluate, and then how to do so.

There are some very practical reasons why restorative justice providers want to evaluate what their schemes are doing, and also some ethical reasons. The practical reasons include:

• Being able to report back to funders as to how their money has been used and to enable economic analyses, including cost benefit analysis;
• Being able to attract more funding;
• To build on training provision and facilitator confidence — essentially to build reflective practitioners;
• To document what the process is. The great unknown question about restorative justice and restorative practice is what kind of restorative justice is best for which individual cases. Unless it’s known what has been delivered, it will be impossible to tie up the effects with what was delivered.

The ethical reasons for evaluating include:

• To be accountable to participants.
• To be accountable more generally to those referring cases, to the general public and to any criminal justice process, including making sure that the process is helping and not causing harm.

Those participating in restorative justice or restorative practice are generally vulnerable people, whether through being victimised, or through earlier adverse experiences, so evaluation is actually part of safeguarding.

• To see whether what is being delivered is justice, is restorative and meets the other values of restorative justice.

Restorative justice has been said to be an ‘umbrella concept’ (Shapland et al., 2011). There’s no one accepted definition, with different national and international instruments having subtly different views. In addition, restorative justice embodies a considerable number of different processes, even when restricted to dealing with criminal offences. They include direct mediation, indirect or shuttle mediation, conferencing, circles, panels and, more broadly, forms of truth and reconciliation commissions. If we include restorative practice, the kinds of processes adopted can be much wider, including where those who have been harmed meet with those who have harmed others. For me, the essence of a process being restorative justice is where there is communication both ways between those who have been harmed (often termed ‘victims’ or ‘survivors’) and those who have harmed (‘offenders’) about an offence or offences which link them.

It’s not all vague though. Restorative justice and restorative practice have some core values, which run through all the different kinds of processes and which are key to the process being restorative. They are central in the international instruments, such as those from the Council of Europe and the European Union, and for good practice. The core values, in relation to restorative justice and restorative practice relating to crime, are:

• Participation by individuals must be voluntary (for those harmed and those who have harmed).
• Participation aims at being inclusive, with both the person harmed and the person who has harmed being actively involved, but often including as well those close to them (their supporters), and sometimes those from the community, as well as from criminal justice.
• Participation centres on facilitating communication between participants, with everyone having the opportunity to put their view and have a fair say, and no one being dominated.
• The process should be made as safe as possible, with any meetings being arranged in a safe space for all participants, and directed by a neutral, trained facilitator.

Making this process of communication inclusive may involve just a small number of participants for some offences — two or three, together with the facilitator (sometimes called the mediator). But it can involve large numbers of people, for example, in neighbourhood disputes, or with extended families. Moreover, there are always lots of stakeholders involved: those from the scheme, the lay participants, those referring, those funding the activity. All of this can make evaluation complex, because all these stakeholders’ views may need to be ascertained — and they do not always want the restorative justice/practice to feature exactly the same aims.

Restorative justice and restorative practice schemes have often been developed to follow particular theoretical views as to how restorative justice ‘works.’ These can emphasise shame and subsequent reintegration into the community (e.g. Braithwaite, 1989), or healing and restoration (e.g. Zehr, 1990), or procedural justice (e.g. Tyler, 1990). For each of these theoretical perspectives, the key aims of the scheme may differ.

The key principle of evaluation is that the scheme must be evaluated according to the aim(s) it espouses. It would be quite unfair to evaluate a scheme in relation to an outcome it was never seeking to attain.

The aims of the scheme may (or may not) include the following:

• Meeting the person harmed’s needs (as expressed by that person);
• Producing a procedurally just process;
• Providing financial compensation to the person/people harmed or other forms of material restoration;
• Reconciling the person harmed and the person who has caused the harm (particularly if they know each other);
• Preventing reoffending (as measured by reconviction or rearrest, depending on jurisdiction);
• Reducing the use of imprisonment or other more coercive criminal justice outcomes;
• Reducing the likelihood of reoccurrence of the same kind of offence or problem, or calls to police as a result of that kind of problem.

Hence the first step in evaluation is for the scheme itself and the evaluator to agree on what the key aims and desired outcomes of the scheme are. It is important to note that, as mentioned above, different stakeholders quite commonly have different outcomes in mind as to what constitutes ‘success.’ Government objectives, for example, often include reducing reoffending, but for the scheme and the participants, reconciling the participants or meeting the needs of the person harmed may be the most important. Ideally, the different stakeholders can be got to agree both on what the aims are, and what priority each should have. If not, those running the scheme and the facilitators will have a difficult time!

In terms of how to evaluate what a restorative justice scheme is doing, we need to distinguish between monitoring and evaluation.

Monitoring is what any scheme should be doing, itself, as routine. It means keeping proper records of cases, referrals, participants, contacts, what has happened at each contact, outcome agreements (where these occur), whether outcome agreements are fulfilled and brief feedback from participants. It’s essentially the routine documentation of cases and what happens to them. It means setting up good electronic databases so that any queries (from, for example, participants or referrers) can be answered easily and quickly. There is now some software that is specifically for restorative justice cases, rather than being criminal justice case management, but its development is still really in its infancy

As discussed above, monitoring is a basic duty of all schemes offering restorative justice and is part of being accountable to all those using the schemes. Because restorative justice sets itself up as a means of doing justice, and as being inclusive, that means that it needs to be accountable to those who agree to participate — so that they can, for example, find out (if they have forgotten) what the details of an outcome agreement were. In the same way as prosecutors and judges need to record the outcomes of criminal cases, so restorative justice providers need to record what happened in restorative justice.

One key element is also what exactly happened when the case is closed, as far as the scheme is concerned.

• Have participants been told the case is closed or are they still waiting to hear something?
• Was it promised that they would be told whether an outcome agreement’s tasks had been completed — and have they been told?
• Are they able to contact the scheme again if they need further support, or have they been given the contacts for other support providers?

Both in our original research in England and Wales in the early 2000s (Shapland et al., 2011), and in more recent research across Europe (Shapland et al., 2022), facilitators and schemes agreed that what was to happen after restorative justice was probably the weakest point of provision. Facilitators put very considerable effort into preparation and into any meeting or communication, but being clear about the end of the restorative justice process was vaguer. Yet that is also important. If, for example, those who have been harmed do not hear about whether outcome agreements have been completed, they may come to doubt the sincerity of the harmer and whether he or she really meant to promise to do what was agreed (Shapland et al., 2011)

Monitoring, then, is vital. Periodic evaluation is also important. Evaluation is whether the scheme or provider is achieving its aims. Doing evaluation often requires more research skills or experience than providers may possess in-house, so it is worth partnering with, or commissioning university researchers to help to do this. An independent evaluation also tends to carry more weight with funders and referrers than one done by the provider itself.

What should be looked at in the evaluation depends on what the scheme’s aims are. It might include looking carefully at whether the scheme is meeting the needs of those who have harmed — and that is likely to involve interviewing past users of the scheme. Many schemes ask participants to fill in a brief ‘satisfaction’ questionnaire and that may include some free space to comment on their experience. Those answers will provide some clues as to what it may be worth asking about more specifically. However, ‘satisfaction’ ratings by themselves usually do not give enough detail to see what is really appreciated, or where the scheme can do better (is it the process, the amount of information, the outcome, feedback opportunities?). Though participants’ responses often highly correlate between different aspects they have experienced, participants are able to rate and be more specific as to their most appreciated, or least appreciated, aspects (Shapland et al., 2011).

Other aims may include how those who have harmed feel they have been affected and whether there has been any change in their lives. That will mean locating and interviewing past users who have harmed — sometimes a lengthy process, because it means gaining access to those in custody. Or an aim may be to reduce reoffending. The difficulty here is that one is trying to measure differences in reoffending (normally through re-convictions) between those with similar criminal careers who have experienced restorative justice and those who have not. In other words, it is important to have a good control group and measure the re-convictions of both groups. The best way to do that is to set up a randomised controlled trial, because we know that those who agree to participate in restorative justice are different from those who refuse (so simply comparing the two does not give reliable answers). However, running a randomised controlled trial is a specialist activity requiring considerable research skills. Any of these types of evaluation are likely to be quite costly and to require funding for the evaluation itself.

The key elements of choosing the right research method for evaluation is that they need to relate to the key aims of the scheme. It is also important to be able to relate the findings re outcomes to the restorative justice process itself. So it is worth documenting clearly what the restorative justice process of the particular scheme is. And that will help with the monitoring as well. Overall, good implementation of restorative justice — keen facilitators, support from the top, enough referrals and funding — are more important than perfect evaluation research methods. But monitoring and evaluation are not optional.

Joanna Shapland is an Edward Bramley Professor of Criminal Justice at the University of Sheffield, in the United Kingdom.
Contact: J.M.Shapland@sheffield.ac.uk

Braithwaite, J. (1989). Crime, shame and reintegration. Cambridge: Cambridge University Press.

Shapland, J., Buchan, J., Kirkwood, S. and Zinsstag, E. (2022). Mitigation and risk in restorative justice: report to the Scottish Government. Restorative Justice Forum Scotland. (https://www.rjforum.scot/files/mitigation-risk-report-final-version-to-scottish-government.pdf)

Shapland, J., Robinson, G. and Sorsby, A. (2011). Restorative justice in practice: evaluating what works for victims and offenders. Abingdon: Routledge.

Tyler, T.R. (1990). Why people obey the law. London: Yale University Press.

Zehr, H. (1990). Changing lenses: a new focus for crime and justice. Scottdale PA: Herald Press.

Published on 29 November 2022.