Making realist evaluations useful for government
What is it about an intervention that generates change? What conditions are required for it to be effective? How can we get more of the intended and less of the unintended outcomes?
If you have ever wanted to know the answer to these questions, you should be thinking about realist evaluation. Realism brings a sophisticated approach to understanding how programs ‘work’. Realists recognise it is how people interpret program activities rather than the activities themselves that generate change. That is, programs don’t emit a constant causal force – they work by providing resources or opportunities for people to reason, make decisions and behave differently (see image courtesy of Gill Westhorp). Crucially, different people will respond differently. Some will benefit; some will not; and some may be harmed.
Realist evaluation is about identifying the ‘mechanisms’ that make an intervention work and the ‘contexts’ in which these generate ‘outcomes’. For example, CCTV cameras might deter potential offenders or, conversely, attract them by signalling ‘rich pickings’. It might help catch offenders in the act, and lead to more successful prosecutions with video footage (which might reduce crime if it’s a small number of offenders, or have no impact if many others take their place). It might lead people to be more willing to park their car because they think it will be safer, or less willing because they think CCTV signals a risk, or that it invades their privacy. People may even become complacent and leave more valuables in their car, which could lead to increased crime through increased opportunity. These are all different ‘mechanisms’ by which CCTV might work, or fail to work, with different people in different circumstances. Additionally, in Sydney, people may just take the first space they can, regardless of CCTV presence, but in a country town, there may be more scope for decision making. The case of CCTV demonstrates that knowing what outcomes were achieved on average does not tell you much about what the best approach will be in any given car park.
So how can realist evaluation answer real world questions for policy makers?
- Work out who to target and how to modify an intervention to maximise overall outcomes: Realist evaluation can provide sophisticated advice about which people to target with an intervention, or how to modify the intervention to enhance future performance. You may need to think about limiting access to those that really benefit (and avoid people who may be harmed), or move from a one-size-fits all approach to designing a few options that work for a greater range of people.
- Identify what matters for future implementation: A realist analysis identifies the mechanisms that generate outcomes. This makes realist evaluation robust and means the findings from one evaluation can be transferred to another without blind adherence to program fidelity. For example, instead of finding that a pizza night leads to social bonding, it can identify that what you need is the ‘shared meal in an informal setting between mentors and mentees’ because it is that, and not the pizza, that leads to social bonding.
- Understand and benefit from failure: When a program appears to be failing, realist evaluation can help you understand why and help you find the ‘silver lining’ (i.e. the conditions under which certain components of the program do work). Realist evaluation is particularly useful for pinpointing why particular programs do not always work because of its attention to the contexts in which, and target groups for whom, the program activities actually fire mechanisms that generate change. In an age of innovation – of ‘fail fast and fail quickly’ – realist evaluation can provide a scientific approach to experimentation that is not simply testing good ideas, but ideas based on proper theories with a greater chance of success.
- Ensure a scientific approach to evidence-based policy: Science is in large part about developing and testing theories about how the world works. Scientific evaluation is about understanding the value of interventions into the world. Realists know a method does not make an approach scientific. Premature experimentation using Randomised Control Trials (RCTs) without a sufficient theory as to how and why a program should work is simply unscientific. It might tell you what happened, but it won’t tell you why, which parts were most and least valuable, and, crucially, what to do next in a future time or place to maximise the intended outcomes. Realists have a scientific approach – they identify middle range theories – not so specific as to be unique, but not so abstract as to be vague and unusable.
- Design new programs: Drawing on literature, experience and a well-defined problem realist evaluation can identify the mechanisms that might need to be fired for change. For example, when researching a program on nursing home visits, it can be hard to know how many visits are needed. Meta-analysis will usually be inconclusive or contradictory. But once the mechanism of ‘rapport’ is found to be the cause of change, then this outcome, rather than an output, can be the focus of planning and implementation.
You don’t need to produce a realist evaluation report that is full of all the realist jargon to make use of realist principles to better explain, replicate and maximise program outcomes. You just need to think more deeply about how change occurs in social settings.
