How to Evaluate Social Impacts of Restorative Justice

The following reflection draws on the author’s experience in policy evaluation and in particular in the evaluation of restorative justice interventions at community level, for example, the ConTatto, the App@Con and the Un Futuro in Comune projects. The current reflection does not aim to deepen the evaluation design proposed for these evaluations, but rather to focus on lessons learnt from these experiences to provide learning on key aspects to be considered in assessing the social impacts of restorative justice.

Before getting into details on lessons learnt, it is useful to clarify the main concepts under analysis:

• social impact and
• social impact evaluation.

Various definitions of impact exist in the literature. The Organisation for Economic Cooperation and Development (2019, p. 10) defines impact as the:

"Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended". 

The European Commission speaks about outcomes and defines them as the:

"Specific dimension of well-being and progress for people that motivates policy action, i.e. what is intended to be changed, with the contribution of the interventions designed".

When it comes to social impacts, the International Association for Impact Assessment defines them as deployed or potential changes in one of the following areas:

• people’s way of life
• their culture
• their community
• their political systems
• their environment
• their health and wellbeing
• their personal and property rights
• their fears and aspirations (Vanclay et al., 2015, p. 2).

Many of these areas are particularly relevant for restorative justice interventions. A wide literature acknowledges the effects of restorative justice on the various social impact dimensions covered above, namely people’s lifestyles, people’s health and well-being, people’s fears and aspirations, people’s culture, community, people’s rights and the environment. Therefore, the social impacts of restorative justice interventions refer to the expected/unexpected, positive/negative impacts in one of the above-mentioned areas.

A heterogenous picture is also registered when it comes to the definition of impact evaluations, and in particular of social impact evaluations.

Gertler et al. (2016) define impact evaluation as assessing changes in the well-being of individuals, households, communities or firms that can be attributed to a particular project, programme or policy, whereas the European Commission as evaluations that answer the following questions:

• did the public intervention have an effect at all and if yes, how big – positive or negative – was this effect … ?
• why did an intervention produce intended (and unintended) effects?

These two definitions underline two different visions of impact evaluations that have been widely debated in the evaluation literature. The World Bank’s definition concentrates on measuring impacts, whilst attributing the difference in the obtained impacts to the input, that is, the policy/programme/project. The European Commission’s definition challenges this vision, acknowledging the complexity of relating social impacts to a sole input and moves beyond it, focusing on opening the black box of interventions, namely questioning why a specific impact has been obtained. The next section will further explore this debate.

When it comes specifically to the evaluation of social impacts, it is worth noting that this does not represent a unique technique as many versions co-exist depending the objective of the evaluation or on the unit of analysis considered (Melloni, 2017). In a prospective perspective, it aims to anticipate and mitigate the negative social impacts of a specific intervention and to strengthen the positive one, providing, thus, information for the selection of interventions. It has also been used in an ex-post facto perspective to verify the achievement of social impacts foreseen and to enhance the accountability of the interventions implemented. Furthermore, it has also been used for the continuous management of changes produced by the implemented interventions in order continuously to mitigate potential damage and strengthen potential benefits. As to the unit of analysis, social impact evaluations have been used to

• deliver complex interventions (policies, reforms, programmes),
• assess development cooperation projects and, more recently, social innovation interventions and social investments.

In all perspectives, it has been generally used prospectively, such as for instance in the case of the Social Impact Assessment implemented in the United States, the Impact Assessment implemented at the European Commission level or the Poverty Social Impact Assessment implemented by the World Bank (Melloni, 2017). However, some approaches have been used both in a prospective and ex-post facto perspective. This is, for instance, the case of the Social Return on Investment. Social Return on Investment, drawing on the return on investment and cost-benefit analysis, refers to the measurement and monetisation of the social and environmental value creation.

The social impact evaluation of restorative justice interventions refers, thus, to the ex-post facto assessment of social impacts achieved by the interventions under scrutiny irrespective of the approach adopted to do it, whether attributive, contributive or monetisation.

Evaluation has two main functions: on the one hand accountability towards the funder of an intervention, the stakeholders and beneficiaries of the intervention and on the other hand policy learning to inform the intervention (re)design and implementation and the wider policies related to it. While the accountability function is a relevant part of evaluation, it is the policy learning that is at the centre of the evaluative research. However, often these two functions are unbalanced on the ground, following funders’ focus especially on the control of what has been achieved with the investment realised, often combined with a limited evaluative culture and practice of the implementers. If in the past, this has been translated into a focus on the control of the achievement of outputs foreseen by an intervention, more recently there has been a shift from outputs to outcomes/impacts, especially in the context of an increase in the mainstream of the ‘pay by result’ logic into social interventions (as the restorative ones are). However, an unbalanced use of the social impact evaluation poses several risks.

• A first risk regards the limited capacity of the evaluation to influence the decision-making process and to unveil elements for improving not only the assessed intervention, but also the general policies in the fields touched upon an intervention, for example, criminal justice policy, education, housing, healthcare policies, etc. in the case of restorative justice interventions. A limited learning capacity also risks limiting the effective replicability of an intervention in other contexts.
• Another risk consists in the generation of a ‘crowding out’ effect, meaning that implementers may give priority to less vulnerable beneficiaries who guarantee a higher probability of achieving the expected social impacts (Melloni, 2017). This risk is particularly relevant in interventions that foresee a ‘pay by result’ approach.
• The third risk regards the hindering of the innovation and experimentation capacity of interventions designed, as the higher the innovation level the higher risk of failure. This may push some implementers to choose already tested interventions.

This unbalanced function of evaluation can be reduced through the creation of a culture of social impact evaluation as a learning activity aimed to continuous improvement of interventions at both funders’ and implementers’ levels.

While the creation of an evaluative culture can take a long time, it is the evaluator’s responsibility to clarify the purpose of the evaluation, taking into consideration the perspective of all relevant actors (funders, intervention managing organisations and partners, beneficiaries), even if divergent, and to stimulate the learning interest of the beneficiaries of the evaluation activity. When the social impact evaluation refers to a complex intervention, whether a reform, a policy or a programme, an initial phase of awareness raising and training on social impact evaluation and its benefits in terms of policy learning may prove useful to create a shared understanding of the evaluative logic and language among the implementers.

A wide range of approaches to the assessment of impacts of an intervention, including the social ones, are described in the evaluation literature (Stern et al., 2012), such as

• experimental (randomised control trials, quasi experiments, natural experiments),
• statistical (longitudinal studies, econometrics, statistical modelling),
• social value monetisation (social return on investment),
• theory-based (theory of change, process tracing, contribution analysis, realist evaluation, congruence analysis), case-based (naturalistic, grounded theory, ethnography, QCA, within-case analysis),
• participatory (empowerment evaluation, collaborative action research).

This section will focus on the discussion of three of these approaches that, to the knowledge of the author, are more often used to assess the social impacts of restorative justice interventions:

• attributive (in particular randomised control trials),
• theory-based and
• social value monetisation.

For a long time, social impact evaluation methods have been almost entirely focused on assessing if what is expected to happen has actually happened and on attributing that effect to the respective intervention (i.e. net impact). However, this approach has been challenged by several authors (Chen and Rossi, 1989; Weiss, 1995; Pawson and Tilley, 1997, etc.), as it does not open the black box of policies/programmes, namely what happens between the input and the outcome of a policy/programme/project (Stame, 2004). Promoters of theory-driven evaluation stress that without opening the black box evaluations, are ‘at best social accounting studies that enumerate clients, describe programmes and sometimes count outcomes’(Chen and Rossi, 1989, p. 299). Other authors have also challenged the actual possibility of distinguishing the net impact in a complex context.

It is precisely this influence of the context and of the actors on the outcomes of a programme that is emphasised in theory-based evaluations, such as the theory of change (Weiss, 1995), and in particular in realist evaluations. In fact, realist evaluation (see Pawson and Tilley, 1997) considers that it is not programmes that bring about changes in an undesired condition tackled by an intervention but rather people from a particular context who, when involved in a specific intervention, activate a mechanism, such as naming and shaming, self-efficacy or prestige, that generates change. Mechanisms explain what it is about a system that makes things happen. Thus, in the realist perspective, evaluation has to be based on a context-mechanism-outcome configuration. It is the evaluator’s duty to extrapolate mechanisms that generate causation in an intervention and to unveil for whom they generate change.

Monetisation of social value (SROI — Social Return on Investment) is an approach consolidated in several countries, for example, the US and UK, and more recently in others such as Italy. It consists in a synthetic measure of the monetary value of social investments. While its rapid communication to external stakeholders has increased interest in the social impact evaluation of social organisations, the evaluation literature points out several limitations. An initial and relevant limitation relates to the greater focus on the social accountability function of evaluation than on its policy learning role. Another limitation relates to the risk of focusing the implementers’ attention on social effects that are easily monetised and to exclude other impacts that might be more relevant, but also more difficult to monetise (Cordes, 2017), which in the end might create an imbalance in the offer of social programmes (Yates and Marra, 2017). The excessive reductionism of complex reality is another critique raised in the literature (Melloni, 2021). Furthermore, some authors question the utility and relevance of monetising intangible outcomes, such as, for instance, the sense of community (Gibbon and Dey, 2011). Other authors (King, 2014) stress that the selection of measurement criteria and especially of proxies for the calculation of the SROI is highly subjective and sometimes opaque.

The restorative justice field is not new to the debate on how to evaluate. While numerous evaluations, using attributive, such as randomised control trials, and monetisation (SROI) methods have been carried out, showing what works in restorative justice, far fewer evaluations have gone beyond the effects of restorative justice to understand why, how and for whom they work (Bolitho, 2017; O'Mahony and Doak, 2017; Saulnier and Sivasubramaniam, 2015). Suzuki and Yuan (2021) point out that this is hardly a new problem in restorative justice impact evaluation, being indicated more than a decade ago. Nevertheless, impact evaluation of restorative justice has continued to limit its focus to the ‘what works’ question, probably in the attempt to legitimate its validity. In this context, scholars draw attention on the need for evaluation, including also social impact evaluation, in this field to incorporate the ‘why, how and for whom’ questions to inform improvements in restorative interventions (Suzuki and Yuan, 2021; Walgrave, 2011; Bazemore and Green, 2007). In a context of embedding of restorative justice into wider policy frameworks, for example, restorative cities and education policies, a multi-sectoral, multidimensional and multi-stakeholder perspective on changes achieved and on how, why and for whom policy interventions work together to produce these changes is needed. Theory-based impact evaluations can best answer this need. However, this does not mean that theory-based evaluation has to operate in isolation from the attributive or monetisation evaluation approaches. Such approaches can be combined better to respond to stakeholders’ needs bearing in mind that, whatever the evaluation approach selected, it has to answer the intervention stakeholders’ learning needs in order to avoid turning the evaluation into a ‘tick the box’ activity.

Three main ingredients are essential for an effective evaluation of the social impacts of restorative interventions:

• Time. Restorative justice needs time to produce impacts. For instance, Blood and Thorsborne (2005) estimate that it takes 3–5 years for restorative justice interventions in schools to produce impacts on school values and management culture. Often the time needs of social impact evaluation barely coincide with the need for information from both implementers and funders. This is why impact evaluations should ideally be accompanied by ongoing evaluation during the intervention lifetime.
• Competences and independency. Evaluators of restorative justice interventions should possess both evaluation and restorative justice competences. Ideally a mixed group made of restorative practitioners and evaluators should be set up. This would also contribute to increasing the reputation of the evaluator and the legitimacy of the evaluation findings. Furthermore, the evaluator should be independent, that is, not involved in the delivery of the programme activity.
• Adequate financial resources. Often evaluation is the Cinderella of an intervention budget. However, the evaluation of complex interventions, such as restorative ones, takes time and requires adequate skills, which need to be acknowledged in terms of budget.
• Willingness to evaluate. This is particularly relevant for avoiding turning social impact evaluation into a ‘tick the box’ activity without any kind of policy learning and influence on the intervention design and delivery.
• Mainstreaming restorative values in the evaluation design. This implies in particular paying attention to
  • including and respecting all voices and perspectives on the intervention even when divergent,
  • voluntary participation in all evaluation activities,
  • being open and transparent,
  • fairness and
  • deciding together with stakeholders the rules of engagement, that is, the evaluation dimensions, questions and methods

Social impacts evaluation of restorative justice has an enormous potential for boosting both the legitimacy of restorative justice at society level, through unveiling its effects in all domains of social impacts considered in the literature, such as people’s lifestyles, fears and aspirations, culture, health and well-being and rights, community cohesion, democracy and participation and environment, and its effectiveness. However, this double potential depends on several conditions:

• interest in evaluation as a policy learning exercise and willingness to question the perceived effectiveness of the intervention adopting an evidence-based view to it;
• adopting an evaluation approach that does not limit itself to the ‘what works question’, but that analyses also why and how it works and for whom;
• ensuring adequate resources (time, knowledge, funds); and
• mainstreaming the restorative values and principles in the evaluation design and delivery.

Cristina Vasilescu is a policy analyst and an evaluator. 
Contact: cristinel.m.v@gmail.com

Aos, S., Miller, M. and Drake, E. (2006). Evidence-based public policy options to reduce future prison construction, criminal justice costs, and crime rates. Olympia: Washington State Institute for Public Policy.

Augustine, C.H., Engberg, J., Grimm, G.E., Lee, E., Wang, E.L., Christianson, K. and Joseph, A.A. (2018). Can restorative practices improve school climate and curb suspensions? An evaluation of the impact of restorative practices in a mid-sized urban school district. RAND Corporation. Https://doi.org/10.7249/RR2840.(https://www.rand.org/content/dam/rand/pubs/research_reports/RR2800/RR2840/RAND_RR2840.pdf)

Bazemore, G. and Green, D.L. (2007). “Yardsticks” for victim sensitive process: principle-based standards for gauging the integrity of restorative justice process. Victims & Offenders 2(3):289–301. Https://doi.org/10.1080/15564880701404031.

Blood, P. and Thorsborne, M. (2005). The challenge of culture change: embedding restorative practice in schools. Paper presented at the Sixth International Conference on Conferencing, Circles and other Restorative Practices: “Building a Global Alliance for Restorative Practices and Family Empowerment.” Sydney, Australia, March 3–5, 2005.

Bolitho, J. (2017). Inside the restorative justice black box: the role of memory reconsolidation in transforming the emotional impact of violent crime on victims. International Review of Victimology 23(3):233–255. Https://doi.org/10.1177/0269758017714549.

Bolivar, D. (2012). Victim-offender mediation and victim’s restoration: a victimological study in the context of restorative justice. Ph.D. thesis, KU Leuven, Leuven.

Brown, M.A. (2017). Being heard: how a listening culture supports the implementation of schoolwide restorative practices. Restorative Justice An International Journal 5(1):53–69. Https://doi.org/10.1080/20504721.2017.1294792.

Cellini, G. (2009). Alternative al carcere: la intensive probation e un esempio di trattamento intensivo efficace. Rassegna Italiana di Criminologia 3(1):37–55.

Chapman, T. and Campbell, H. (2016). Working across frontiers in Northern Ireland: The contribution of community-based restorative justice to security and justice in local communities. In: K. Clamp (ed.) Restorative justice in transitional settings, chap. 7. London: Routledge.

Chen, H.T. and Rossi, P.H. (1989). Issues in the theory-driven perspective. Evaluation and Program Planning 12(4):299–306. Https://doi.org/10.1016/0149-7189(89)90046-3.

Cordes, J.J. (2017). Using cost-benefit analysis and social return on investment to evaluate the impact of social enterprise: promises, implementation, and limitations. Evaluation and Program Planning 64:98–104. Doi:10.1016/j.evalprogplan.2016.11.008.

Davis, F.E. (2018). Whole school restorative justice as a racial justice and liberatory practice: Oakland’s journey. The International Journal of Restorative Justice 1(3):428–432. Http://dx.doi.org/10.5553/IJRJ/258908912018001003007.

European Commission: Directorate-General for Regional Policy (2015). Guidance document on monitoring and evaluation. Brussels: European Commission. (https://ec.europa.eu/regional_policy/sources/evaluation/2014/wd_2014_en.pdf)

Fronius, T., Darling-Hammond, S., Persson, H., Guckenburg, S., Hurley, N. and Petrosino, A (2019). Restorative justice in U.S. schools: an updated research review. San Francisco, CA: WestEd.

Gal, T. (2016). The conflict is ours’: community involvement in restorative justice. Contemporary Justice Review: Issues in Criminal, Social, and Restorative Justice 19(3):289–306. Https://doi.org/10.1080/10282580.2016.1185950.

Gertler, P.J., Martinez, S., Premand, P., Rawlings, L.B. and Vermeersch, C.M.J. (2016). Impact evaluation in practice. 2nd ed. Washington DC: InterAmerican Development Bank and World Bank. Doi:10.1596/978-1-4648-0779-4.

Gibbon, J. and Dey, C. (2011). Developments in social impact measurement in the Third Sector: scaling up or dumbing down? Social and Environmental Accountability Journal 31(1):63–72. Https://doi.org/10.1080/0969160X.2011.556399.

International Institute for Restorative Practice (2014). Evidence from schools implementing restorative practices: a report from the International Institute for Restorative Practice. IIRP. (http://www.fixschooldiscipline.org/wp-content/uploads/2020/09/3.Evidence_From_Schools_Implementing_Restorative_Practices.2014.pdf)

King, N. (2014). Making the case for sport and recreation services: the utility of Social Return on Investment (SROI) analysis. International Journal of Public Sector Management 27(2):152–164.

McIvor, G. (1991a). Community service work placements. Howard Journal of Crime and Justice 30(1):19–29. Https://doi.org/10.1111/j.1468-2311.1991.tb00693.x.

Melloni, E. (2017). La valutazione di impatto sociale: obiettivi, metodi ed evoluzioni. RIV

Rassegna Italiana di Valutazione 67(1):101–123. DOI 10.3280/RIV2017-067007.

Melloni, E. (2021). Evaluation in Italien. In: R. Stockmann and W. Meyer (eds.) Die Institutionalisierung der Evaluation in Europa, chap. 11, pp. 221–244. Münster: Waxmann Verlag.

O’Mahony, D. and Doak, J. (2017). Reimagining restorative justice. London: Hart Publishing.

Organisation for Economic Cooperation and Development (2019). Evaluation criteria: adapted definitions and principles for use. Development Cooperation Directorate: Development Assistance

Committee. (http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=DCD/DAC(2019)58/FINAL&docLanguage=En)

Pawson, R. and Tilley, N. (1997). Realistic evaluation. London: SAGE.

Petersilia, J. and Turner, S. (1993). Evaluating intensive supervision probation/parole: results of a nationwide experiment. Washington DC: U.S. Department of Justice, National Institute of Justice.

Rietman, M. (2017). Effectiveness of restorative justice practices: an overview of empirical research on restorative justice practices in Europe. Online. (https://www.euforumrj.org/sites/default/files/2019-11/a.2.7.-effectiveness-of-restorative-justice-practices-2017-efrj.pdf)

Rosenblatt, F.F. (2015). The role of community in restorative justice. Abingdon: Routledge. 978 1315 71762 3 (e-book).

Saulnier, A. and Sivasubramaniam, D. (2015). Restorative justice. New Criminal Law Review: An International and Interdisciplinary Journal 18(4):510–536.

Sherman, L.W. and Strang, H. (2007). Restorative justice: the evidence. London: Smith Institute. (http://www.iirp.edu/pdf/RJ_full_report.pdf)

Stame, N. (2004). Theory-based evaluation and types of complexity. Evaluation 10(1):58–76. Https://doi.org/10.1177/1356389004043135.

Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R. and Befani, B. (2012). Broadening the range of designs and methods for impact evaluations: report of a study commissioned by the Department for International Development. London: Department for International Development. (https://www.oecd.org/derec/50399683.pdf)

Suzuki, M. and Yuan, X. (2021). How does restorative justice work? A qualitative metasynthesis. Criminal Justice and Behavior 48(10):1347–1365. Https://doi.org/10.1177/0093854821994622.

Vanclay, F., Esteves, A.M., Aucamp, I. and Franks, D.M. (2015). Social Impact Assessment: guidance for managing and assessing the social impacts of projects. Fargo ND: International Association for Impact Assessment.

Wachtel, J. (2012). World’s first “Restorative City”: Hull, UK, improves outcomes of all interventions with young people, saves resources. IIRP. (https://www.iirp.edu/news/world-s-first-restorative-city-hull-uk-improves-outcomes-of-all-interventions-with-young-people-saves-resources)

Walgrave, L. (2011). Investigating the potentials of restorative justice practice. Washington University Journal of Law & Policy 36:91–139.

Weiss, C.H. (1995). Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families. In: J.P. Connell, A.C. Kubish, L.B. Schorr and C. Weiss (eds.) New approaches to evaluating community initiatives : concepts, methods, and contexts, chap. 3, p. 65ff. Washington, DC: Aspen Institute.

Yates, B.T. and Marra, M. (2017). Social Return On Investment (SROI): Problems, solutions . . . and is SROI a good investment? Evaluation and Program Planning 64:136–144. Doi: 10.1016/j.evalprogplan.2016.11.009.