Monitoring the impact of MARAM, and the effectiveness of various approaches to its implementation, is critical to inform future planning and effort.
Evaluation
It is positive that evaluations have been planned in some areas. A maternal and child health family violence evaluation will attempt to measure practice change as a result of MARAM and information sharing. The evaluation has been delayed due to significant demand and workforce pressures on the maternal and child health workforce due to the COVID-19 pandemic, but the report is expected to be available in late 2022. An evaluation of MARAM implementation in education settings has also been funded, but has not yet begun.
Hospitals have expressed frustration about evaluation not being built into the funding model for the SHRFV initiative. Representatives from the hospitals leading this initiative explained that there is a strong need for data and evaluation to help this work become business as usual by ensuring there is a documented, evidence-based case for the benefits of the program. For example, according to the Royal Women’s Hospital, there has not been a large increase in referrals to specialist family violence services due to family violence screening occurring in its antenatal service, and this is likely because the hospital is intervening earlier and working directly with some patients. However, due to a lack of evaluation resourcing, SHRFV has no capacity to collect corroborating evidence. Other hospitals, particularly those where social work is not consistently available, have indicated that they have seen an increase in referrals to specialist family violence services.
The SAFE audit conducted as a one-off project across multiple health services and hospitals with the support of philanthropic funding has provided a good snapshot of how well hospitals are set up to identify and respond to family violence. The SAFE Project report1 recommended:
- regular review of demand associated with family violence work
- regular auditing including SAFE Tool implementation.
However, doing this kind of auditing requires ongoing resourcing, as we were advised that the 50-file audit that is required takes approximately three weeks for one person. Some hospitals are initiating their own studies of the effectiveness of their approach to SHRFV implementation (for example, see Box 5), but these rely on clinician-reported knowledge and confidence.
Box 5: Evaluation activity at the Royal Melbourne Hospital
As part of its implementation of the SHRFV initiative, the Royal Melbourne Hospital implemented an evidence-based and in-depth training framework for its 6,000+ workforce, in conjunction with a clinical champions network (called the Family Safety Advocates Network). This was a network of staff who received more than nine hours of family violence–related training and consisted of health professionals throughout the hospital.
A study evaluated the clinical champions model in allied health staff. The study found that the model resulted in statistically significant and sustained improvements in levels of self-reported family violence knowledge, confidence and frequency of screening, when combined with a community of practice.
The hospital is also assessing the three-year impact of SHRFV through a 2020 evaluation (yet to be published). The evaluation found that, compared with the baseline knowledge of all staff, as assessed in 2017, there were statistically significant increases in knowledge, confidence levels and how often family violence screening was occurring. There were also self-reported improvements in staff knowledge about key indicators of violence and in their ability to ask about and respond to disclosures of family violence.
Source: Family Violence Reform Implementation Monitor, based on information provided by the Royal Melbourne Hospital.
System-level monitoring
Governance groups and bilateral meetings between Family Safety Victoria and departments supporting prescribed sectors provide an important communication line for raising implementation issues, and we suggest that such feedback be actively sought from frontline workforces [relates to action 12].
Data and evidence are also required. Family Safety Victoria worked with Cube Group in 2019 to develop a MARAM monitoring evaluation plan and framework. These highlighted existing mechanisms for data collection, such as those listed in Table 7, each with its own challenges and opportunities. The monitoring plan also proposed a series of measures that Family Safety Victoria hoped to capture via individual services, including those outlined below, but we are not aware of whether data against these measures could be obtained:
- number of referrals made to other framework organisations, by type of framework organisation
- number of referrals received by other framework organisations, by type of framework organisation
- number of requests for or provision of secondary consultations related to family violence
- number of referrals made and received for clients from Aboriginal communities, diverse communities and at-risk age cohorts
- number of victim survivors identified by universal or non-specialist services who are referred to appropriate services.
Table 7: Ongoing monitoring of MARAM implementation – challenges and opportunities associated with using existing data sources
| Existing data sources* | Challenges | Opportunities |
|---|---|---|
| Census of Workforces that Intersect with Family Violence |
According to reporting on the 2019 census, only 1% of the estimated 222,070 workers in broader workforces that intersect with family violence responded to the survey (which includes workforces in scope for this report as well as many others, including police, courts, etc.). | The 2020 process evaluation of the MARAM reforms suggested using future iterations of the census to determine the level of awareness, understanding and use of the MARAM Framework. |
| Post-training surveys | Preliminary findings from the evaluation of the Strengthening the Foundations Rolling Action Plan suggest that objective measures of learning are rarely used, undermining the veracity of some evaluations. | The limitations of these surveys need to be acknowledged and results coupled with more objective data on the impact on practice. |
| Framework organisations survey |
The survey has only been done once, in April/May 2021, and did not cover phase 2 organisations. Encouraging a strong response rate from universal services will In addition, the survey only captures the perspectives of the most senior staff within each organisation who are responsible for and involved in implementing MARAM in their organisation. |
Results for universal services, particularly broken down by workforce, will provide a good snapshot of MARAM understanding and implementation, indicating things like:
|
| Family violence– related data reported by agencies |
Much of the data reported by implementation agencies seems to focus on training numbers. Again, preliminary findings from the evaluation of the Strengthening the Foundations Rolling Action Plan noted that training numbers are not always connected to workforce planning, therefore the scale of achievement is unclear and the remaining need not quantified. |
We suggest further emphasis be placed on building the capacity of agencies to report on measures that can provide a greater sense of impact. Data on referrals to The Orange Door, including the source, is tracked, and it will be useful to also track incoming secondary consultations through the new data fields recently added to the client relationship management system. |
Source: Family Violence Reform Implementation Monitor, based on information provided by Family Safety Victoria.
* As identified in the MARAM monitoring and evaluation framework.
There are multiple other sources of data, such as surveys (for example, the Department of Education and Training’s principals survey, which includes a range of MARAM and information sharing–related questions) and evaluations. We suggest it is vital that monitoring and evaluation activity be built into ongoing practice wherever possible, and that this data be brought together to form a coherent and meaningful view of MARAM implementation across the system, with an emphasis on impact [relates to action 13].
Ongoing monitoring of MARAM across workforces could focus on answering questions such as:
- What is the family violence identification rate?
- How is practice changing for different workforces? How does it vary across locations? What training have staff done?
- What outcomes are being achieved for victim survivors?
- Where are clients experiencing family violence being referred? Are services meeting demand? Where are there blockages? What support is being provided by universal services in these situations?
- What is the percentage of people who can be appropriately supported without referring onto specialist family violence services?
- What is the frequency of, and time taken for, secondary consultations and information sharing? For example, the regulatory impact statement included a range of assumptions, including an estimated 30 minutes per MARAM response for phase 2 organisations. It will be important to determine whether these estimates are reasonable or require revisiting to inform future planning. It will also be useful to track which sectors are making these requests, what they are requesting and of whom [relates to action 5].
MARAM Maturity Model
A MARAM Maturity Model project is currently underway and will involve developing products such as a maturity matrix against which organisations can plot their progress, a range of resources to support alignment and an improvement cycle (as depicted in Figure 6). As discussed in our Reform Governance report, this work builds on the existing Organisational Embedding Guide, released in 2020. The improvement cycle is expected to operate at two levels: at the program area level, whereby sector maturity is assessed, improvement strategies put into place and then reviewed; and at the individual service level.
Once this model is in place, it has the potential to be a key source of information about the effectiveness of MARAM implementation. How departments will engage with the model, and how they will support prescribed organisations to engage with it, is yet to be determined.
The MARAM Framework is the foundation for a unique and ambitious whole-of-system approach to identifying and managing family violence risk. An ongoing emphasis on monitoring, evaluation and data is required to ensure progress can be accurately monitored, opportunities for improvement can be identified, and the framework can become embedded across the system.
Figure 6: Proposed improvement cycle for the MARAM Maturity Model
Source: Family Safety Victoria
Endnotes
- The Royal Women’s Hospital and the University of Melbourne (2021): The System Audit Family Violence Evaluation (SAFE) Project, Final Report, p. 48.
Updated