WHAT IS EVIDENCE AND WHAT IS IT TELLING US TO DO?
By Marion MacLeod, Policy Manager, Children in Scotland
Our public and voluntary sector services are currently facing some of the most severe challenges in a generation, both in terms of the issues with which they are grappling and the resources they have at their disposal to address them. Local authorities and health services are reducing or, in some cases, closing down services. As much third sector provision receives a significant proportion of its income though services being commissioned by public bodies, the consequential impact of reduction in public sector budgets on voluntary agencies and community groups is also profound and far-reaching.
In the last few years we have had two major enquiries aimed at improving the sustainability, responsiveness and effective impact of public services in Scotland. The Scottish Parliament Finance Committee held a wide-ranging enquiry into preventative spending, looking both at the types of services that prevent problems and the nature of evidence that demonstrates effectiveness. The Christie Commission on the Future Delivery of Public Services considered these issues alongside questions of governance and community involvement. Progress on implementing both has been limited, largely due to the reductions in public sector budgets described above. In services for communities and in adult health and social care, such developments as have been possible have focused largely on co-production of services within the community planning framework.
In relation to children’s services, a substantial and compelling body of evidence… underlines the importance of early life interventions and support for families.
These conclusions were not entirely without impact. Despite the restrictions on public sector funds (and the reduction in availability of our universal provision, such as health visiting and family centres) we have, in recent years, seen significant investment at national and local level in some elements of early childhood services. This investment has largely funded programmes aimed at improving aspects of parenting in children’s early life. The rationale for funding these programmes has been that they are strongly supported by evidence of effectiveness. Many of these were developed and evaluated in countries whose public services are structured in very different ways from Scotland’s. The evidence referred to is usually evaluation by randomised control trial [RCT], a method historically applied to drug testing where, out of a group of subjects, some are chosen at random to receive the drug with the others – the control group – receiving a placebo with no active ingredients or, indeed, nothing at all. Comparison of outcomes between the two groups demonstrates the efficacy or otherwise of the drug.
An example of such a programme is the ‘Family Nurse Partnership’, in which the Scottish Government invested £8.5m between 2010 and 2013. It was developed in the United States by Professor David Olds, somewhat ironically based on his observation of the effectiveness of universal health visiting in the UK. He developed and patented a home visiting programme, delivered by nurses to teenage mothers. The number and nature of contacts between nurses and mothers is highly specified and must be followed with absolute fidelity. He carried out many of the evaluations of the programme himself, comparing, over a twenty-five year period, outcomes for their children of mothers who had received the programme with those who had not.
The outcomes for the recipients of the programme were indeed better than those of the control group. We should, however, ask a lot more questions before we decide whether this, or any other expensive programme, constitutes best use of our increasingly limited public funds. Firstly, when a programme has been developed elsewhere, we should ascertain whether the public service context is similar enough for the programme to be readily transferable. The US has little in the way of free primary health care and community support services. In Scotland all families have the support of a health visitor (likely to be intensive support in the case of vulnerable young mothers), and family centres are available in many communities. It is therefore eminently possible that the differences in Scotland between recipients and non-recipients of the FNP will be much less significant than was found in the US.
We need, therefore, to be confident when investing in these programmes that we are doing the best we can to close the gap between richest and poorest year on year and on a sufficiently ambitious scale.
Thirdly, we should consider a much broader range of evidence than can be supplied by RCTs. It is not generally possible to evaluate the impact of universal services by RCT, yet the countries who achieve the best and most equitable outcomes have largely done so through investment in their universal services rather than through programmes. International comparisons of systems, structures and principles are highly informative in this respect.
Some basic information on the conduct of any evaluation is also important. Sustainability is critical – if we take, for example, smoking cessation interventions, fewer participants are likely to report that they remain smoke free six months after the intervention than six days after. The objectivity of the evaluators should also be considered. A quick search on the internet will show that many evaluations of programme based interventions have been carried out by those who developed the programme. Evidence should also be sought from less directly interested parties. Factors such as programme completion and dropout rates, whether they reached those most in need and rigorous comparison of their outcomes with those of alternative investment options should also be considered. It is also important to ‘deconstruct’ programmes and establish the components that generate effectiveness – whether it is the Family Nurse Partnership programme that is making the difference or whether home visiting, developing a relationship, frequency and regularity of contact and provision of good information are the things that matter.
A recent paper by Professor Pat Dolan of Galway University and colleagues both explores all these issues in greater detail and sets out an alternative, pluralistic approach to evaluation. The European Commission and other international bodies are giving serious consideration to how they view evidence, this paper having significant influence in this debate. We would do well in Scotland to reflect on these questions if we are to achieve good and equitable lives for our citizens and best value for our public pound.
is Policy Manager with Children in Scotland
By Marion MacLeod, Policy Manager, Children in Scotland
OTHER ARTICLES IN THIS ISSUE
- The changing nature of residential investment in Scotland
- The Benefits of Confidential Reporting
- Imagining a Fairer Fife
- Time to Refocus Regeneration Resources?
- What is evidence and what is it telling us to do?
- Analysing, advising, researching and arguing
- Digital exclusion - is the 3rd Sector missing the bus?
- Planning for a Fairer Scotland
- Stressed out? Let's do something about it.
- In working order? The state of Scotland's labour market
- Tartanising the Apprenticeship Levy
- Thinking Big on Affordable House Building - a common sense policy?
- Time for a Transient Visitor Levy?
- Community Finance can challenge money market failure
- Developing Carluke
SCOTLAND'S LOW CARBON OPPORTUNITY
Currently, Scotland is living a three planet lifestyle. This means if everyone in the world lived as we do, we would need three planets to survive. The world population is growing, resources are becoming scarcer and the effects of climate change are starting to be recognised. This is simply unsustainable.
- Carbon, energy and the environment: We need to move on from talking about climate change to acting on climate change.
- Finding balance in the low-carbon transition
- Greening the Centre of Scotland
- Meeting the cost of reducing carbon
- The Carbon Cycle?
- The role of nature-based solutions in combatting the climate crisis
Looking for a previous issue? Use the menu below to select an issue.
MOST READ ARTICLES
- Transport for Edinburgh - Integrated Transport for a Smart City
- Worth more than the First Minister? Senior Salaries in Scottish Quangos
- Social Business Can Transform Public Services
- Bringing alive the Digital Participation Charter for Scotland's citizens, communities and businesses
- Dundee: From Waterfront redevelopment to city economy regeneration
- A Planet of Smart Cities: Scotland's digital challenge
- Public Services Reform and Public Opinion
- Increasing digital participation levels in Scotland - what needs to happen next?
- The Evolving Public Sector Response to Budget Challenges
- Telehealth and Telecare for Older People