Issue 13

WHAT IS EVIDENCE AND WHAT IS IT TELLING US TO DO?

By Marion MacLeod, Policy Manager, Children in Scotland

Our public and voluntary sector services are currently facing some of the most severe challenges in a generation, both in terms of the issues with which they are grappling and the resources they have at their disposal to address them. Local authorities and health services are reducing or, in some cases, closing down services. As much third sector provision receives a significant proportion of its income though services being commissioned by public bodies, the consequential impact of reduction in public sector budgets on voluntary agencies and community groups is also profound and far-reaching.

In the last few years we have had two major enquiries aimed at improving the sustainability, responsiveness and effective impact of public services in Scotland. The Scottish Parliament Finance Committee held a wide-ranging enquiry into preventative spending, looking both at the types of services that prevent problems and the nature of evidence that demonstrates effectiveness. The Christie Commission on the Future Delivery of Public Services considered these issues alongside questions of governance and community involvement. Progress on implementing both has been limited, largely due to the reductions in public sector budgets described above. In services for communities and in adult health and social care, such developments as have been possible have focused largely on co-production of services within the community planning framework.

In relation to children’s services, a substantial and compelling body of evidence… underlines the importance of early life interventions and support for families.

In relation to children’s services, a substantial and compelling body of evidence was presented to both enquiries underlining the importance of early life interventions and support for families. Dr Suzanne Zeedyk, among others, gave oral evidence to the Finance Committee’s enquiry, where she outlined her research on brain development in infants. This research showed that early cognitive development was affected for good or ill by responsive nurturing and communication between babies and their caregivers and the consequent development of strong emotional attachment. The Finance Committee concluded that early life experiences were critically important in preventing and mitigating later difficulties.

These conclusions were not entirely without impact. Despite the restrictions on public sector funds (and the reduction in availability of our universal provision, such as health visiting and family centres) we have, in recent years, seen significant investment at national and local level in some elements of early childhood services. This investment has largely funded programmes aimed at improving aspects of parenting in children’s early life. The rationale for funding these programmes has been that they are strongly supported by evidence of effectiveness. Many of these were developed and evaluated in countries whose public services are structured in very different ways from Scotland’s. The evidence referred to is usually evaluation by randomised control trial [RCT], a method historically applied to drug testing where, out of a group of subjects, some are chosen at random to receive the drug with the others – the control group – receiving a placebo with no active ingredients or, indeed, nothing at all. Comparison of outcomes between the two groups demonstrates the efficacy or otherwise of the drug.

An example of such a programme is the ‘Family Nurse Partnership’, in which the Scottish Government invested £8.5m between 2010 and 2013. It was developed in the United States by Professor David Olds, somewhat ironically based on his observation of the effectiveness of universal health visiting in the UK. He developed and patented a home visiting programme, delivered by nurses to teenage mothers. The number and nature of contacts between nurses and mothers is highly specified and must be followed with absolute fidelity. He carried out many of the evaluations of the programme himself, comparing, over a twenty-five year period, outcomes for their children of mothers who had received the programme with those who had not.

The outcomes for the recipients of the programme were indeed better than those of the control group. We should, however, ask a lot more questions before we decide whether this, or any other expensive programme, constitutes best use of our increasingly limited public funds. Firstly, when a programme has been developed elsewhere, we should ascertain whether the public service context is similar enough for the programme to be readily transferable. The US has little in the way of free primary health care and community support services. In Scotland all families have the support of a health visitor (likely to be intensive support in the case of vulnerable young mothers), and family centres are available in many communities. It is therefore eminently possible that the differences in Scotland between recipients and non-recipients of the FNP will be much less significant than was found in the US.

We need, therefore, to be confident when investing in these programmes that we are doing the best we can to close the gap between richest and poorest year on year and on a sufficiently ambitious scale.

Secondly we should ask whether the outcomes are good enough. US evaluations of the FNP programme show that the children of the mothers who received the programme were, for example, rather less likely to drop out of high school and to be in prison as adults. We should first of all be sure that our universal services in Scotland have not achieved and cannot achieve a similar level of divergence. Scotland aspires to be ‘the best place in the world to grow up’. We need, therefore, to be confident when investing in these programmes that we are doing the best we can to close the gap between richest and poorest year on year and on a sufficiently ambitious scale.

Thirdly, we should consider a much broader range of evidence than can be supplied by RCTs. It is not generally possible to evaluate the impact of universal services by RCT, yet the countries who achieve the best and most equitable outcomes have largely done so through investment in their universal services rather than through programmes. International comparisons of systems, structures and principles are highly informative in this respect.

Some basic information on the conduct of any evaluation is also important. Sustainability is critical – if we take, for example, smoking cessation interventions, fewer participants are likely to report that they remain smoke free six months after the intervention than six days after. The objectivity of the evaluators should also be considered. A quick search on the internet will show that many evaluations of programme based interventions have been carried out by those who developed the programme. Evidence should also be sought from less directly interested parties. Factors such as programme completion and dropout rates, whether they reached those most in need and rigorous comparison of their outcomes with those of alternative investment options should also be considered. It is also important to ‘deconstruct’ programmes and establish the components that generate effectiveness – whether it is the Family Nurse Partnership programme that is making the difference or whether home visiting, developing a relationship, frequency and regularity of contact and provision of good information are the things that matter.

A recent paper by Professor Pat Dolan of Galway University and colleagues both explores all these issues in greater detail and sets out an alternative, pluralistic approach to evaluation. The European Commission and other international bodies are giving serious consideration to how they view evidence, this paper having significant influence in this debate. We would do well in Scotland to reflect on these questions if we are to achieve good and equitable lives for our citizens and best value for our public pound.

Marion MacLeod
is Policy Manager with Children in Scotland

By Marion MacLeod, Policy Manager, Children in Scotland

Issue 13

PREVIOUS ISSUES

Looking for a previous issue? Use the menu below to select an issue.