UK: rail safety research

[John Ashworth]

The impact of research – so hard to pin down

'Impact' is now a major component in research grant assessment, but how do you quantify it?

o Jonathan Wolff
o The Guardian, Tuesday 2 February 2010

I've been doing my best to ignore the "impact agenda", but I'm sorry to say that finally I feel the need to add my two euros' worth. Those following the debate will know that research grant applications now need to include an "impact statement", while the new research excellence framework assesses impact as a major component of the research quality of each academic department.

One working definition of impact is "making a demonstrable difference in a non-academic context". Sounds pretty laudable, assuming, of course, that the demonstrable difference is a positive one. But other than finding the topic a little wearing, I've kept quiet about impact partly because much of my own work is "policy engaged" and so I've smugly assumed that the agenda will work to my advantage. But now I'm having my doubts.

Consider, for example, some work I did a few years ago for the railway industry. In the aftermath of the accidents at Ladbroke Grove (1999) and Hatfield (2000), the government put a few million into research into railway safety. Some of this was used to think about ethical questions. And so, I like to boast, I was employed as a moral consultant to the railway industry.

The industry was under huge pressure, especially from the broadsheets, to spend what then seemed like a vast amount of money – several billions – on a new signalling system to stop trains running through red lights. If the system worked it could, on average, be expected to save perhaps a couple of lives a year. The industry executives, though, thought that this would be a very poor return for so much money, which could save very many more lives if spent, say, on the NHS. But they were afraid of the reaction if they made this argument in public. I was asked to provide an ethical commentary on the issues.

To cut a long, and, to me anyway, fascinating, story short, my advice to the railway industry was that if they wanted to capture public confidence, then they had better try to find out what, exactly, the public currently thought about the issues. We knew what the media said. Incessantly. And the unions. And the pressure groups. But do any of these groups speak for the public? We conducted a number of in-depth interviews to try to find out.

As it turned out, we found no evidence that the public had any interest in expensive, high-tech, flashy safety systems, especially if, whether by ticket price or by taxation, they would end up footing the bill. Rather, they were concerned that maintenance work was poorly supervised, and that subcontractors had financial incentives to cut corners. The root of the problem, thought the public, was bad management, not inadequate technology. Shortly after we submitted the report, a senior industry official called me to say that he had no idea that the British public were so sensible.

So what happened to our report? The commissioning group decided that it would be better not to publish it, but to treat it as an internal briefing document. A little later I was told that it had been a helpful input in refocusing the industry's communication and management strategy, so that instead of constantly pumping out facts and figures to convince the public of how safe it is to travel by train, they quietly attempted to rebuild their reputation for competence in managing safety.

And why didn't they publish the report? Because they didn't want media attention. They didn't want to give the papers an excuse for yet more stories about rail crashes. A couple of years later, they again refused, as by then the delay to publication would be likely to be treated as a "cover-up". So the report remains in the filing cabinet of a few people who have probably moved on to quite different jobs.

Did this work have impact? Quite possibly, although not on its own. In public policy, it is rare to have influence unless you are one of a crowd making the same point. Can I demonstrate impact? Of course not. Stupidly, I forgot to tape-record the conversations I had with industry executives, and nothing was ever put in writing. Influence in sensitive public-policy areas rarely leaves an audit trail. And if this is true of work directly commissioned by a public authority, what chance for work published in academic journals?

Jonathan Wolff is professor of philosophy at University College London. His column appears monthly