Evidence base for perpetrator interventions

The Royal Commission highlighted the importance of investing in the still-emerging evidence base about perpetrators, their motivations and the types of interventions likely to be most effective in reducing their violence:

There are serious gaps in our knowledge about the characteristics of victims and perpetrators of family violence and about how the systems that respond to such violence are working. These gaps restrict the government’s ability to respond to family violence effectively and to plan for the future, and could well result in ineffective or wasteful expenditure on some responses and insufficient expenditure on others.¹

It also recommended introducing better evaluation practices and to continue to support research into family violence.

Overall, we find that progress has been made since the Royal Commission in evaluating new types of interventions for perpetrators and in conducting research to better understand their characteristics and motivations. Important efforts are also underway to establish a common evaluation framework to allow for better comparison of the effectiveness of different perpetrator interventions.

Perpetrator research

The Victorian Government has recently conducted two major reviews of perpetrator programs and available best practice literature. The first is a comprehensive desktop study examining the available evidence base on perpetrator interventions, which Family Safety Victoria commissioned the Department of Premier and Cabinet to develop in order to inform a theory of change on perpetrator accountability. The second is an overarching review conducted by the Department of Justice and Community Safety based on its internal evaluations of family violence interventions for justice clients. This review documents lessons learned and provides an opportunity for government to understand which aspects of family violence perpetrator
interventions work best for justice clients.

There is a need for ongoing research to better understand perpetrators, the drivers of family violence and how this intersects with special needs for diverse cohorts. The Victorian Government is continuing to invest in research to better understand perpetrators, both by funding state-based initiatives and by contributing $1.1 million in the 2022–23 State Budget to the Australian National Research Organisation for Women’s Safety (ANROWS).

Also, in August 2022, Victoria’s Minister for Prevention of Family Violence announced 13 successful projects funded under the Family Violence Research Program 2021–2024, of which three focus specifically on perpetrator responses. The topics to be covered include interventions for young perpetrators, early identification of LGBTIQ+ perpetrators, and participant engagement strategies to reduce attrition in perpetrator programs.²

Developing the Family Violence Outcomes Framework

Domain 3 of the Victorian Government’s Family Violence Outcomes Framework is ‘Perpetrators are held accountable, connected and take responsibility for stopping their violence’. It includes the six indicators listed in Figure 13.

Stakeholders consulted for this report had mixed views on the measurement of the indicator ‘Reduction in all family violence behaviours’ and how it can be used as a performance metric, given that figures showing a decrease in recidivism could also mean that breaches of FVIOs were not being reported and acted on.

Noting that Family Safety Victoria is currently engaged in a consultative process to determine the best way to measure the indicators in the Family Violence Outcomes Framework, we suggest that the indicators could also be further developed by making them more specific (for example, how is ‘perpetrators’ overall wellbeing’ defined?) and by including baselines, realistic targets and timeframes for achieving those targets. For example, what is the current average level of perpetrator retention in programs, and what would be an attainable target and within what timeframe? A government statistics expert highlighted that it is critical that services collect and enter indicator data (for example, number of perpetrators completing programs) into systems in a standardised way wherever possible to ensure progress in delivering services can be monitored and reported on.

Guidance could also be developed for the perpetrator service response sector to specify the data that is needed to measure the indicators, and the roles and responsibilities for data collection. For example, how is the ‘equity and safety of relationships’ defined, how is it measured and who is responsible for doing so? As we noted in our Reform Governance report:

There is no clear relationship between the major investments and their intended or projected bearing on high-level outcomes. While activities have been grouped under what they are expected to contribute to, there has been no articulation of the connection between the inputs (delivery of initiatives), the outputs (provision of services) and the results or outcomes these are having. It is not possible to determine the extent to which outputs are contributing to outcomes. Without this logic, governance mechanisms are limited in their ability to measure, monitor and report on progress. Oversight bodies should now be pivoting their efforts towards measuring short and medium-term outcomes – what is working and what is not – to determine progress towards achieving the vision.³

One area of progress is the development of a client-reported outcomes measurement approach for perpetrator interventions, which has been trialled in the Medium-term Perpetrator Accommodation Service (MPAS) since 2021. Two sets of questions have been developed as part of the MPAS evaluation - one for perpetrators and one for adult victim survivors – and these are to be asked at the point of program entry and program closure. These questions align with relevant outcome indicators in the Family Violence Outcomes Framework and will directly contribute to the evaluation of outcomes achieved by the MPAS. The approach is also expected to contribute to the broader work to measure and monitor outcomes across the family violence system.

Evaluation of perpetrator interventions

As outlined in section 2, since the Royal Commission both Family Safety Victoria and the Department of Justice and Community Safety have implemented several innovative pilot programs trialling approaches to individual case management and programs for perpetrators from diverse communities (refugee and migrants, LGBTIQ+ and so on) and those with complex needs (alcohol and other drug issues, mental health). These evaluations have given valuable insight into intended and unintended outcomes, contributed to identifying promising new interventions and indicated the potential benefits of further investment in these areas. They have also highlighted lessons learned to improve the implementation of future interventions. For example, the evaluation of the perpetrator accommodation program shows the importance of building awareness about new interventions among networks that can provide referrals and the importance of branding perpetrator interventions in a more positive way to encourage uptake.

Given the importance of evaluation activity in building the evidence base, we suggest the Victorian Government should continue to invest in quality evaluations, ensuring adequate funding is allocated at the program design stage and ensuring evaluation data collection starts early enough to capture key data, including baseline measures as appropriate.
We note the development of a common evaluation framework, and some common evaluation challenges, both of which are described below.

Developing a common evaluation framework

Since the EACPI delivered its final report in 2018, the Victorian Government has adopted the advisory committee’s advice on establishing a common evaluation framework for perpetrator interventions to ensure rigour and consistency in evaluations and better comparison of outcomes.⁴ Family Safety Victoria’s guidance document ‘Key Evaluation Questions to Examine the Implementation of Perpetrator Interventions’ provides solid and clear guidance regarding evaluation themes (see Table 2) and key questions, covering cost-benefit analysis, intended and unintended outcomes and lessons learned.

Table 2: Perpetrator Accountability: Key Evaluation Themes in the Common Evaluation Framework

Source: Based on Family Safety Victoria’s ‘Perpetrator Accountability – Key Evaluation Questions’

Evaluation experts highlighted that it was important to have this centralised guidance in place to avoid a ‘piecemeal’ approach with data not being comparable due to varying methodologies. Monash Gender and Family Violence Prevention Centre explained that striking a balance between standardisation and flexibility is needed for innovative projects:

To some extent we need to be comfortable with the messiness to this work when tailoring programming; we need more evidence and more consistency in evaluation practice, but at the same time data can’t necessarily be compared if different programs are doing different things with different people.

Key evaluation challenges and issues

As outlined by the EACPI, in general there are multiple inherent challenges associated with evaluating perpetrator interventions, whether in Victoria, the rest of Australia or internationally.⁵ While it is still critical to take advantage of opportunities to conduct evaluations, it is important to be aware of the following issues when considering evaluation design:

• Measuring impacts can be difficult when desired outcomes might only be achieved some time after a program has ended, and other factors may shape perpetrator behaviour.
• Many evaluations rely on a reduction in perpetrator reoffending (as measured by contact with the justice system) as the sole indicator of success, yet most family violence incidents go unreported.
• Perpetrator programs are not conducive to some impact evaluation methodologies. For example, randomised controlled trials usually require a control group to compare with the treatment group, which poses ethical problems in perpetrator program evaluations because it would mean refusing program access for otherwise eligible perpetrators. Other alternatives exist for conducting rigorous studies – for example, collecting pre and post measures from perpetrators, victim survivors and others in a position to observe perpetrator behaviour change.
• It can be challenging to recruit a significant sample if perpetrators and victim survivors are not motivated to participate in evaluations. In the absence of their active participation, comparing reoffending outcomes between program participants and a control group based on matching people on certain characteristics in system-level data is one alternative, although recidivism is not a complete measure of all family violence behaviour.

Stakeholders also raised several areas for improvement in the way perpetrator interventions are evaluated, including:

• The need for disaggregated data to shed light on the experiences of diverse cohorts: While perpetrator data on the Crime Statistics Agency’s Family Violence Data Portal is usually disaggregated by gender and Aboriginality, disaggregated data on perpetrators is much more limited for refugee and migrant communities, LGBTIQ+ people and disability status.
• The need to capture a wider range of outcomes: Stakeholders expressed interest in evaluations assessing the full range of outcomes from perpetrator interventions, including measures of recidivism and reported changes in victim survivor safety and perpetrator behaviour.
• The need to build (and fund) evaluation into program design across the board: Although independent evaluations have been funded for a number of innovative pilot projects, only a small number of evaluations have recently been carried out in Victoria for group men’s behaviour change programs, which continue to attract more than half of Family Safety Victoria’s perpetrator intervention funding. Stakeholders expressed that more evaluation funding was needed to improve program quality. It should be noted that while the approach of group men’s behaviour change programs themselves is not new, evaluation is still needed to assess the performance of service providers, value for money and the quality of the men’s behaviour change curriculum and facilitation. Especially in light of the revised Men’s Behaviour Change Minimum Standards, evaluations should also assess providers’ level of compliance with the need to refer perpetrators to other services, follow-up between sessions to maintain motivation and attendance, assess and manage risk and carry out their essential family safety contact function.
• The need for systemic collection of client feedback data: To design appropriate perpetrator interventions and assess their effectiveness, it is critical to allow for regular opportunities to receive feedback from victim survivors, perpetrators and frontline service workers. Data should be collected in a systematic way from the outset, with a clear data collection and analysis plan about which data will be collected, when/how it will be analysed and who is responsible. One evaluation conducted by Monash University for a Relationships Australia men’s behaviour change program in Victoria suggested that to overcome challenges in collecting evaluation data from participants, a ‘Research Champion’ should be embedded with the program coordination team to collect user experience data throughout the life of the program.⁶
• The need for evaluations of online interventions to identify best practices: Stakeholders highlighted that the COVID-19 response has driven a major shift in service delivery formats, and in the post-pandemic period it will still be valuable to assess how remote men’s behaviour change programs have benefited clients in rural areas. There is a need to develop a strong body of evidence to determine the most appropriate and effective use of technology with men’s behaviour change clients, including looking at issues such as the maximum group size and how to keep participants engaged in an online format

Footnotes

1. State of Victoria (2016): Royal Commission into Family Violence: Summary and Recommendations, p. 41.
2. Victoria Government (2022): Projects for Family Violence Research Grants Unveiled, Media Release, 9 August 2022.
3. Family Violence Reform Implementation Monitor (2022): Monitoring Victoria’s Family Violence Reforms, p. i.
4. Victorian Government (2018): Expert Advisory Committee on Perpetrator Interventions: Final Report, p. 102.
5. Ibid, pp. 101-102.
6. Monash University (2019): Evaluation of the Relationships Australia Victoria Men’s Behaviour Change Program and Men’s Case Management Pilot, p. 87.