This week, in the course of writing an article for the Campaign Group Open Democracy, I’ve had the opportunity to speak to paediatric surgeons at Leeds General Infirmary (LGI), the object of much attention after its children’s cardiac surgery unit was closed late last week.
I’ve also looked into the facts behind the claim that the unit was closed because its mortality rate was ‘double the national average’ – or to be more precise, to look into them as far as is possible given that the claims were supposedly based on statistics that are neither published nor even finished.
The fact that the unit now looks set to be reopened doesn’t remove the concerns about the original decisions were reached. Nor does it undo the damage that has been done by the decision and what has been said and published around it. As the BBC News site observed yesterday, “if the unit does reopen, Leeds General Infirmary will have to reassure patients and families that safety of care is paramount.”
One of the reasons given by Monitor for its decision to put Mid Staffs into administration was that the Trust’s ‘brand’ had become ‘toxic’. Since a decision is still going to be made at some point about whether LGI’s unit is closed permanently as part of the rationalisation of children’s heart surgery into a smaller number of ‘centres of excellence’, ‘toxicity’ in the public perception will be extremely damaging to the chances that those who rely on LGI’s paediatric heart services will continue to have access to them.
So, it’s been a very interesting week. While I hadn’t looked closely at Leeds previously, as I had my head deep into the Mid Staffs statistics and the deceitful claims by the Tories about disability claims being abandoned because of their reforms, it was definitely next on my list.
Given what I’d learned from researching the Mid Staffs situation about the accuracy of mortality statistics and the way they’re handled by the media, the idea that a children’s heart unit could have double the national death rate clearly needed closer examination.
The parallels between the Leeds case and the recent ‘workfare’ events had also caught my attention. In Leeds, the supposed quality problems ‘came to light’ immediately after an inconvenient legal judgment to allow the original aim to go ahead. With the ‘Poundland’ workfare case, the government simply rewrote the rules to get around a similarly inconvenient judicial ruling – in the process breaking centuries of legal precedent that you can’t make a law and then make it apply to events in the past.
It’s safe to say that alarm bells were ringing.
Fighting dirty?
In the course of one of my interviews last week, a cardiac surgeon told me that one of the members of the committee which made the initial decision that the Leeds unit should close while Newcastle’s Freeman unit should remain open had been overheard telling a colleague,
On this one we’re going to fight dirty.
This is hearsay, so it can’t be taken to be fact – but it does make it very valid to look at the facts around the decision to see whether they suggest underhand dealings and hidden agendas. So let’s take that look.
The Conflict of Interest
The original decision to close LGI was criticised for the fact that it would have left only two, widely separated centres for children’s heart surgery in the eastern half of England – London and Newcastle. Seriously ill children would face repeated long journeys to distant centres for life-saving treatment, with all the attendant risks as well as considerable hardship for their families for years, perhaps even generations.
Surely for such a decision to be taken there must be over-riding reasons?
Perhaps so – but they’re well hidden. The original ‘scoring’ process used to reach the decision was extremely opaque and full of subjectivity, as the judge observed. Scoring included ‘multipliers’ that increased the score of some hospitals more than others, and which were based on such nebulous factors as ‘management accountability’. Crucial factors such as co-location of complete acute services, so that children could receive treatment for other problems without having to leave the unit, may or may not have been weighted more heavily than the vaguer factors. But because the weightings were kept secret, there is no way to know.
When LGI tried to obtain information on the judging criteria, this was withheld even in the face of a Freedom of Information request. When the information was eventually released there were several versions, with many discrepancies among them – and the crucial weightings were not included.
There is a potentially even more worrying aspect to the decision-making process. In reports on the original decision to select the more centrally-placed Leeds unit for closure, one important fact has generally been conspicuous by its absence – the presence on the decision-making committee of an individual with a personal stake in the outcome.
This committee was the JCPCT‘s “Safe and Sustainable” (SaS) steering committee, whose vice-chair is cardiac surgeon Leslie Hamilton. Mr Hamilton practises at Newcastle upon Tyne Hospitals NHS Foundation Trust – the same Trust whose children’s heart surgery unit faced closure if the LGI unit remained open.
As local MP Greg Mulholland told BBC Radio 4’s Today programme yesterday,
the only way that Newcastle stays open is if Leeds closes.
Mr Hamilton’s behaviour may well have been exemplary, but it doesn’t matter. There is still an inevitable conflict of interest when a senior member of the decision-making committee works for one of the units being considered for closure – and any decision between Leeds and Newcastle is inherently unsafe.
An article in this Thursday’s Guardian criticised a “turf war” among surgical units:
What is happening in Leeds is a fight over the bodies of small babies born with heart defects. It would have been good to think it could have been sorted out in quiet, compassionate and well-informed discussion exchanging evidence around a table.
Maybe. But when the decision to close your unit is being made by a group that includes someone whose own unit will close if yours doesn’t, I’m not going to blame you for fighting tooth and nail instead of meekly giving in to it.
Now let’s look at the statistics that bear on both the initial decision to close the LGI unit and the post-judicial decision to suspend its surgeries.
Flawed statistics
Surgeons have to be calm under pressure and extremely focused. However, when I talked to one of LGI’s children’s heart surgeons this week he was clearly incandescent about what is being said about the competence and outcomes of the Leeds unit – that there have been severe safety concerns, including allegations that its mortality rates were twice the national average. These allegations were swiftly challenged from various quarters. Problems with the data were pointed out, resulting in the referral of the statistics for more detailed analysis. However, yesterday’s Guardian carried a prominent article (the same one that criticised the ‘turf war’) stating that subsequent analysis of Leeds’ corrected statistics still “shows that its death rates are unacceptably high”.
This appears to be completely untrue. According to my consultant contact, as of Thursday morning when the article was published, the data was still with the Leeds team for its input.
For such a claim to appear in a prominent Guardian article suggests that the newspaper’s journalist was intensively ‘briefed’ by someone closely connected to the issue with a desire to influence public perception ahead of the release of more accurate analysis.
Professor Sir Brian Jarman, author of the ‘HSMR’ system of statistical analysis of clinical outcomes, has published statistics indicating that, far from being ‘double’ the national level, mortality rates for children’s heart surgery at Leeds are in fact below the national average. It’s not often that Professor Jarman and I have found each other on the same side of a mortality rate issue, but in this case we seem to be.
The SaS steering committee is apparently using a different data set (CCAD or NICOR) to calculate mortality rates. These statistics are neither published nor even finished, and certainly shouldn’t be used to make radical statements about mortality rates at a particular unit.
Whatever the differences between the statistical systems, though, the ‘raw’ numbers will be the same or very nearly so in both cases, as these simply reflect the numbers of children who died without adjusting for ‘case mix’ and other factors. A look at these figures will give a strong indication whether there is a real problem in Leeds.
The incandescent surgeon told me that a national measurement of children’s heart-surgery outcomes shows an average “mortality rate within 30 days of surgery” of around 2%. The graph below shows the mortality rates as a percentage of total surgeries for all of England’s children’s heart surgery centres, plus the rate for the country as a whole, for the period from 2009-2013:
Far from being double, the Leeds rate (in red) of 1.77% is slightly below the national average (in yellow) of 1.79%. So much for ‘double the national average’. Alder Hey and Birmingham have far higher mortality rates, yet will remain open under the JCPCT‘s plan.
Any adjustment to such small data sets that will take a hospital from being below average on raw numbers to being ‘double‘ the average in the ‘standardised’ results is inherently unsound – because any statistical adjustment or assessment based on such small numbers is always unsound.
In order to achieve any kind of reliable statistical measurement, a large enough number of ‘episodes’ has to be recorded. If this number is too small, chance variations have a disproportionate effect and cause false trends to appear. The numbers of cases per hospital – even over a 4-year period from 2009 to now – range from a low of 384 cases to a ‘high’ of 1,223. Even the uppermost figure is too small for reliable, meaningful statistics. You might argue for working from figures over a longer period to increase the sample size – but figures from the more distant past can hardly be said to be relevant to the present performance and future fate of a surgical unit.
To illustrate the problems of small samples being used for statistics, let’s take a look at one of Professor Jarman’s graphs on mortality rates in children’s heart surgery, to which I’ve added some annotation:
This graph shows the standardised mortality ratios (SMRs) for the various children’s cardiac units. ‘Standardised’ means that they’ve been adjusted to try to even out case mix factors and others that might affect the likelihood of deaths at separate units.
‘CI’ refers to the ‘confidence interval’ – basically the level of certainty with which you can identify the rates. The marks above and below each point on the graph reflect what you can say with 95% certainty. As a large centre for biomedical statistics points out, the “confidence interval is directly related to sample size” – in other words, if you have small numbers you can’t be very certain!
This is reflected in the graph. Look at the top and bottom marks for the Leeds SMR. These show that if you want to have 95% certainty of being right, you can only say that the Leeds SMR (with 100 being exactly average) was somewhere between about 50 and about 180 – according to Prof Jarman’s details, 46.9 and 180.3, to be exact.
If you wanted to say with 98% or 99% certainty (the standard used normally for hospital mortality statistics according to Prof Jarman’s testimony to the Francis inquiry), the distance between the upper and lower points would be far greater.
This brief look at the limitations of statistical analysis shows clearly that any adjusted statistics based on samples of only a few hundred cases can’t possibly tell us anything reliable about the performance of a unit that, as our incandescent surgeon told me, is ‘operating on patients varying from a 600g premature baby to a 16 year old’.
No matter what statistical system the SaS or NHS executives are considering for their decision, the leaked accusation that the Leeds unit has twice as many deaths as the national average is not merely nonsense, but culpable nonsense.
The Dutch surgeon
As with the furore around Mid Staffs, there are serious question marks over the media’s reporting of the issues. Not only has the publication of the ‘double the average’ accusation been reckless or based on ulterior motives, but even perfectly innocent occurrences are slanted to appear incriminating. Take this example, which is from the same Guardian article:
The Guardian has also learned that a Dutch surgeon was being paid by the Leeds Teaching Hospitals NHS Trust to fly in from her job in Aarhus, Denmark, to operate one weekend a month – a highly unusual move in this very specialised medical field.
Do you see the insinuation? “Flying a foreign surgeon in once a month? What’s going on? What was this surgeon having to put right and what deficiencies must there be in the LGI’s own team?”
The real situation is quite simple, and far from incriminating. The surgeon, Catharina van Doorn, used to work at the Leeds unit and, according to my consultant contact, is extremely capable and has certain rare, specialised skills. She is brought across to perform surgeries when more surgical capacity and those specialised skills are needed. Which is a perfectly sensible, responsible course of action.
That such an innocent and even positive fact is used to insinuate something sinister or furtive raises serious questions about the objectivity – or lack of it – of this article and raises worrying concerns about undue exertion of influence from quarters with a vested interest in the outcome of this matter. It also serves to highlight the much less subtle way in which much of the media has pitched the story.
Back to ‘toxicity’
Pro-LGI campaigners will be celebrating the announcement that their heart unit is going to reopen, and rightly so. However, as things stand at the moment this is merely a reprieve. Unless the NHS board and the Department of Health have a complete change of heart, either Leeds or Newcastle is almost certainly going to be closed at some point in the not-too-distant future.
The fact that – whether intentionally or negligently – the LGI ‘brand’ has been ‘tofixied’ by the questions raised in the public confidence doesn’t bode well for its prospects of surviving the ‘Safe and Sustainable Review’ that aims to reduce number of England’s paediatric heart centres from 10 to 7.
The way in which figures have been leaked, snap decisions have been made on the suspension of services based on inherently-unreliable data, and the media has been ‘briefed’ to portray the story in a negative and fear-inducing way should be of serious concern to anyone associated with or dependent on LGI’s children’s heart services.
It should also be of grave concern to anyone who cares about fairness, or about the way in which the NHS as a whole is being targeted and portrayed by the government and the national media.
Is someone ‘fighting dirty’? You’ve now hopefully got some facts that you won’t pick up in the mainstream coverage of the events and are in a better position to decide!
(If you’re also incandescent about the way they NHS is being treated and would like to contribute to the fight to defend it, please visit CCGWatch.org.uk and consider making a donation.)