Frameworks and evaluations

Frameworks guiding family violence prevention and early intervention effort in Aboriginal communities

Organisations we consulted spoke about using a variety of frameworks to guide their work. These included Our Watch’s Changing the Picture (see Figure 10), the Indigenous Family Violence Primary Prevention Framework (see Figure 11) and the Nargneit Birrang – Aboriginal Holistic Healing Framework for Family Violence. However, some stakeholders commented that these frameworks lack supporting resources and that organisations are left to work out how to apply them in practice. Certainly, Changing the Picture does not have the same practice tools and resources that accompany Change the Story.

Similarly, the Indigenous Family Violence Primary Prevention Framework, while providing a high-level overview of the key considerations and approaches to guide primary prevention effort, does not outline the comprehensive suite of actions needed to address family violence in Aboriginal communities. It also predates the development of Changing the Picture and would benefit from greater alignment with the national framework and subsequent Victorian mainstream prevention frameworks such as the Free From Violence strategy.

A refresh of the Indigenous Family Violence Primary Prevention Framework is a key commitment under the first action plan of the Dhelk Dja Agreement (see also Table 2 in section 5 of this report). We agree that enhancing the strategic approach to family violence prevention in Aboriginal communities would be supported through an updated Aboriginal primary prevention framework. This will ensure there is a shared understanding across government, Aboriginal-led organisations and communities about what primary prevention is in the Aboriginal context, where the priorities are, and how this work intersects with or sits alongside mainstream prevention effort.

While Dhelk Dja has a Monitoring, Evaluation and Accountability Plan that includes a high-level theory of change across the agreement’s five strategic priorities and associated actions, it does not provide a detailed articulation of the different types of prevention activity needed to achieve the strategy’s intended outcomes. Without outlining how outputs and activities feed into the desired outcomes, it is not user-friendly for small community organisations looking to define their program logic and how they fit into the state-level framework for preventing family violence.

Our companion report, Primary Prevention System Architecture, found the need for a clearly articulated theory of change and a strategic operating framework for the broad prevention system to give effect to the Victorian Government’s Free From Violence strategy. The planned update to the Indigenous Family Violence Primary Prevention Framework should include a similar approach that is conducted in tandem with the mainstream work. This would provide an opportunity to develop a much clearer and more detailed operational strategy to accompany the high-level Aboriginal-specific framework, which sets out essential actions, provides a theory of change and establishes roles and responsibilities between mainstream and Aboriginal-led organisations [relates to action 1]. The theory of change will also be important for informing a future investment strategy as discussed in the next chapter.

Stakeholders highlighted that the process of developing an Aboriginal prevention framework (and associated materials) needs to be led from within Aboriginal communities. We agree and note that a lead Aboriginal organisation (or group of organisations) will likely have to be funded to undertake this work on behalf of Dhelk Dja, in partnership with Respect Victoria and Family Safety Victoria. We also note that there is likely to be substantial value in engaging Our Watch in this process to utilise its expertise and ensure alignment with the national framework Changing the Picture.

Operationalising prevention and early intervention frameworks

In our consultations, stakeholders raised that individual Aboriginal organisations often don’t have the time, resourcing or specialist prevention expertise to operationalise the frameworks that exist. Prevention organisations such as Respect Victoria, Our Watch, Safe and Equal and government departments and agencies such as the Office for Prevention of Family Violence and Coordination and Family Safety Victoria (both within the Department of Families, Fairness and Housing) can contribute specialist prevention expertise and provide assistance. However, stakeholders have reinforced that for efforts to be sustainable, culturally relevant and reflective of local needs, the process has to be led from within Aboriginal communities. Key elements of operationalising frameworks include:

• designing initiatives that address the drivers identified in the framework and theory of change, and creating materials and resources to support implementation that are informed by evidence-based practice and tailored to local community needs
• training, support and guidance for staff delivering initiatives
• collecting and analysing data on initiatives being delivered, including undertaking program monitoring and evaluations to tailor delivery, support reporting back to funders, and contribute to the evidence base of effective approaches to inform future efforts.

The availability of data to inform efforts and evaluation capacity are two areas that were repeatedly identified during our consultations as needing further development.

Availability of data to guide efforts and support strategic investment

Access to data and data sovereignty (ownership of data and narratives around meaning) continue to be of significant concern to Aboriginal stakeholders. Inadequate data collection was identified as a key theme by the Royal Commission. The MARAM Framework also highlights the need to strengthen data on the prevalence and experience of family violence among Aboriginal communities. Seven years on from the Royal Commission, organisations consulted indicated that better data is still needed. As the Victorian Aboriginal Community Controlled Health Organisation explained:

… members have expressed that data relating to family violence in Aboriginal communities needs to be stronger, and better reflect the wide range of experiences of victims and people who perpetuate the violence – including the prevalence of family violence committed by non-Aboriginal men towards Aboriginal women. Family violence datasets need to be broadened to include causal data and factors leading up to family violence incidents such as alcohol and other drug use so that funding can be allocated to services that address the factors that lead to family violence.

ACCOs consulted for this report repeatedly raised that they do not have visibility of family violence figures or other relevant information for their community/region. They wished to know where the family violence ‘hotspots’ are to support additional intervention activities in those areas. Just as importantly, they wanted to see if family violence rates were lower in certain communities where prevention initiatives have taken place to understand what is working effectively. The Crime Statistics Agency released police family violence incident data for Aboriginal people for the first time in September 2020, with a range of data available by local government area and government regions. Some organisations have reported that the Crime Statistics Agency portal is difficult to navigate, and there is a need to bring together relevant data in an accessible way for use by Dhelk Dja members to support their strategic decisions about prevention and early intervention priorities.

Aside from the need for more in-depth data on the incidence and drivers of family violence in Aboriginal communities, the prevention mapping project highlighted that only 33 (13 per cent) of the 251 family violence prevention initiatives in Victorian Aboriginal communities collected data on participation or initiative reach. Challenges for smaller organisations in collecting and recording initiative data are further described in this and the following chapter.

Family Safety Victoria has funded a position to support Dhelk Dja members’ access to data, and the data working group of the Dhelk Dja Partnership Forum has been developing a suite of data aligned to the strategy that will be presented to the forum at an upcoming meeting. At the time of report consultations, the Family Safety Victoria funded role had not been filled, and while dedicated capacity within government to support Dhelk Dja is important, improving the provision of data across government (see also section 5 of this report) and supporting Aboriginal organisations to collect and make use of data is also critical [relates to action 5]. In progressing work under Dhelk Dja’s strategic priority 5 (Aboriginal-led and -informed innovation, data and research) consideration needs to be given to how each of these areas is delivered, including the funding requirements to develop internal data capacity within organisations.

Within Aboriginal communities, the Victorian Aboriginal Community Controlled Health Organisation, the peak body for Aboriginal health and wellbeing organisations, is planning to roll out a new platform to streamline and improve the quality of its members’ data:

The vision is to implement a Data Lake, through the Deadly Data™ portal that will bring together data from varying environments and will supply a source of truth for advocacy on behalf of Community. The data (aggregated and de-identified) will enable VACCHO and Members to tell stories with a community lens to support better health outcomes for the community and ensure data stays in Community and remains under community control.

This appears to be an enormously promising initiative, and consideration could be given to how family violence–related data can be incorporated to improve its availability for the community.

Evaluation of Aboriginal prevention and early intervention initiatives

Understanding what works, what doesn’t and why, and adapting programming accordingly, along with documenting and communicating achievements and lessons learned, is crucial for improving program quality and promoting better outcomes for communities.

The Royal Commission recommended that all family violence interventions in the Victorian Aboriginal community be evaluated in a culturally appropriate manner. Government’s approach to implementing this recommendation included:

• developing the Dhelk Dja Monitoring Evaluation and Accountability Plan, which establishes outcomes for data and research (see Figure 12)
• undertaking evaluation and capacity building with recipients of the Preventing the Cycle of Violence (PCV) Aboriginal Fund and the Community Initiatives Fund described below
• establishing a dedicated Aboriginal data position to support the Dhelk Dja Partnership Forum in accessing and interpreting family violence data.

This work builds on evaluation and capacity building for a range of Aboriginal family violence initiatives funded by the Department of Justice and Community Safety under the Koori Community Safety Grants Program in 2016.

In the most recent efforts, Family Safety Victoria engaged evaluation company Urbis Pty Ltd in partnership with Aboriginal consultants Karen Milward and Yatu Widders-Hunt to build the capacity of ACCOs implementing programs under the PCV and Community Initiatives Fund. This included 11 projects implemented under PCV and 22 under the Community Initiatives Fund, which ran for three years. The evaluation capacity building originally took the form of in-person workshops but shifted online during the COVID-19 pandemic. Under the project, the evaluation team supported ACCOs to design and deliver monitoring and evaluation activities such as designing a program logic, carrying out data collection and writing evaluation reports. An Aboriginal consultant involved in the project described how it had been successful in breaking through some of ACCOs’ initial apprehension about evaluation by demystifying technical jargon:

To get them to understand what a program logic is, we put a cultural lens on it rather than using terms such as ‘outputs’ and ‘outcomes’. The hardest thing is that no one explains what they are in laymen’s terms, so people were nervous about being involved in an evaluation project.

Other achievements included ‘building capacity on how to record data, not just record numbers but getting real data from participants and stakeholders’ and redesigning government evaluation templates to make them more accessible and meaningful to Aboriginal stakeholders. Participants received three rounds of feedback on their evaluation activities and reports.

While this project was valuable in building awareness of evaluation within the organisations involved, this and other past efforts have been time-limited and sporadic, and it is challenging for small community-based organisations to undertake quality monitoring and evaluation without ongoing support. Reflective of these challenges, of the 251 prevention initiatives included in the prevention mapping project, only 35 had been evaluated, which represents a missed opportunity to contribute to the evidence base of effective approaches. While some larger organisations such as Djirra, VACCA, and the Victorian Aboriginal Community Controlled Health Organisation have internal monitoring and evaluation teams and are better positioned to undertake this work, many smaller organisations do not. As one Aboriginal stakeholder explained,

‘Even though you can build capacity, people just want to deliver, not evaluate’.

This stakeholder suggested that it would be more strategic to fund Aboriginal organisations to commission and manage independent external evaluations, which would reduce the risk of distracting core staff from program implementation. However, other organisations want to build evaluation capability internally.

Program implementation staff do not necessarily need to know how to write an evaluation report; however, they do need to know how to manage and use evaluations to improve outcomes for their organisations and clients. Organisations also need to ‘own’ the evaluation process to ensure the methodology is culturally appropriate, and that the knowledge gathered will be useful for Aboriginal communities. At the program development stage, they also need to set the foundation for the evaluability of their project by establishing a clear program logic with outcome indicators, and by collecting consistent data over the life cycle of the project. This ensures there is a vision of what success looks like for the initiative, and how to measure it. The Dhelk Dja Monitoring Evaluation and Accountability Plan is clear that delivering the outcomes presented in Figure 12 (see above) is dependent on adequate funding being provided to build the capacity of Aboriginal organisations’ ‘data systems, infrastructure, capacity and governance’ and to ‘support achievement of an evidence base for programs’.

Supporting Aboriginal organisations to apply prevention and early intervention frameworks and evaluate the outcomes of initiatives

We find that there is a need for more sustained capacity building and support for ACCOs in applying prevention and early intervention frameworks, collecting data on and monitoring implementation of initiatives and strengthening evaluation of outcomes. This needs to be undertaken within ACCOs, supported by long-term funding and a mandate to promote quality implementation and workforce development across the family violence sector in Victorian Aboriginal communities.

One promising development arising from the Royal Commission into Victoria’s Mental Health System is the Balit Durn Durn Centre of Excellence for Aboriginal Social and Emotional Wellbeing. The centre is being established within the Victorian Aboriginal Community Controlled Health Organisation ‘to foster innovation and improvement in social and emotional wellbeing practice, policy and research’. This could be a useful model to better support family violence prevention and early intervention work in Victorian Aboriginal communities [relates to action 4]. Such an approach could concentrate Aboriginal-specific expertise on family violence prevention and early intervention within an ACCO to:

• bridge the gap between high-level frameworks and implementation on the ground through providing guidance and resources to translate and apply frameworks in developing initiatives
• provide a link with mainstream prevention agencies such as Respect Victoria, Our Watch and Safe and Equal
• provide guidance and support for other organisations in data collection, analysis, monitoring and evaluation
• house and disseminate resources, best practice guidance and learnings throughout the sector
• support workforce development (see also section 4) through an Aboriginal-led community of practice around prevention and early intervention work.

The Balit Durn Durn Centre uses the Aboriginal social and emotional wellbeing model that emphasises the importance of building strength, resilience and connectedness in Aboriginal people and communities (see Figure 13).

While ACCOs consulted for this report felt that it was positive that there is greater focus on prevention and early intervention since the Royal Commission, they noted the multiplicity of federal and state government strategies, frameworks, plans and agreements that apply across different government portfolio areas. For example, one organisation noted that every department has an Aboriginal strategy as well as subject-specific strategies and plans. Aligning their work with these multiple frameworks is challenging and resource-intensive, particularly for smaller organisations. In addition to considering how ACCOs are best supported to implement family violence prevention-specific plans and frameworks, there also appears to be an opportunity to consider consolidating and more clearly aligning governments’ Aboriginal strategies and plans under Victoria’s Closing the Gap outcomes and implementation approach.

Figure 13: The Aboriginal social and emotional wellbeing (SEWB) model identifies the key Aboriginal social and emotional wellbeing areas (inner circle) and the broader influences on that wellbeing (outer circle)

Source: Department of Health: Balit Murrup – Aboriginal Social and Emotional Wellbeing Framework 2017–2027, p. 24. Artist: Tristian Schultz, RelativeCreative. Reference: Gee, Dudgeon, Schultz, Hart & Kelly 2013 on behalf of the Australian Indigenous Psychologists Association

Note: The above wheel represents holistic healing and includes protective factors that support good mental health for Aboriginal Communities. This approach is particularly important for resilience and healing, including for men's groups.