This week, in the course of writing an article for the Campaign Group Open Democracy, I’ve had the opportunity to speak to paediatric surgeons at Leeds General Infirmary (LGI), the object of much attention after its children’s cardiac surgery unit was closed late last week.
I’ve also looked into the facts behind the claim that the unit was closed because its mortality rate was ‘double the national average’ – or to be more precise, to look into them as far as is possible given that the claims were supposedly based on statistics that are neither published nor even finished.
The fact that the unit now looks set to be reopened doesn’t remove the concerns about the original decisions were reached. Nor does it undo the damage that has been done by the decision and what has been said and published around it. As the BBC News site observed yesterday, “if the unit does reopen, Leeds General Infirmary will have to reassure patients and families that safety of care is paramount.”
One of the reasons given by Monitor for its decision to put Mid Staffs into administration was that the Trust’s ‘brand’ had become ‘toxic’. Since a decision is still going to be made at some point about whether LGI’s unit is closed permanently as part of the rationalisation of children’s heart surgery into a smaller number of ‘centres of excellence’, ‘toxicity’ in the public perception will be extremely damaging to the chances that those who rely on LGI’s paediatric heart services will continue to have access to them.
So, it’s been a very interesting week. While I hadn’t looked closely at Leeds previously, as I had my head deep into the Mid Staffs statistics and the deceitful claims by the Tories about disability claims being abandoned because of their reforms, it was definitely next on my list.
Given what I’d learned from researching the Mid Staffs situation about the accuracy of mortality statistics and the way they’re handled by the media, the idea that a children’s heart unit could have double the national death rate clearly needed closer examination.
The parallels between the Leeds case and the recent ‘workfare’ events had also caught my attention. In Leeds, the supposed quality problems ‘came to light’ immediately after an inconvenient legal judgment to allow the original aim to go ahead. With the ‘Poundland’ workfare case, the government simply rewrote the rules to get around a similarly inconvenient judicial ruling – in the process breaking centuries of legal precedent that you can’t make a law and then make it apply to events in the past.
It’s safe to say that alarm bells were ringing.
Fighting dirty?
In the course of one of my interviews last week, a cardiac surgeon told me that one of the members of the committee which made the initial decision that the Leeds unit should close while Newcastle’s Freeman unit should remain open had been overheard telling a colleague,
On this one we’re going to fight dirty.
This is hearsay, so it can’t be taken to be fact – but it does make it very valid to look at the facts around the decision to see whether they suggest underhand dealings and hidden agendas. So let’s take that look.
The Conflict of Interest
The original decision to close LGI was criticised for the fact that it would have left only two, widely separated centres for children’s heart surgery in the eastern half of England – London and Newcastle. Seriously ill children would face repeated long journeys to distant centres for life-saving treatment, with all the attendant risks as well as considerable hardship for their families for years, perhaps even generations.
Surely for such a decision to be taken there must be over-riding reasons?
Perhaps so – but they’re well hidden. The original ‘scoring’ process used to reach the decision was extremely opaque and full of subjectivity, as the judge observed. Scoring included ‘multipliers’ that increased the score of some hospitals more than others, and which were based on such nebulous factors as ‘management accountability’. Crucial factors such as co-location of complete acute services, so that children could receive treatment for other problems without having to leave the unit, may or may not have been weighted more heavily than the vaguer factors. But because the weightings were kept secret, there is no way to know.
When LGI tried to obtain information on the judging criteria, this was withheld even in the face of a Freedom of Information request. When the information was eventually released there were several versions, with many discrepancies among them – and the crucial weightings were not included.
There is a potentially even more worrying aspect to the decision-making process. In reports on the original decision to select the more centrally-placed Leeds unit for closure, one important fact has generally been conspicuous by its absence – the presence on the decision-making committee of an individual with a personal stake in the outcome.
This committee was the JCPCT‘s “Safe and Sustainable” (SaS) steering committee, whose vice-chair is cardiac surgeon Leslie Hamilton. Mr Hamilton practises at Newcastle upon Tyne Hospitals NHS Foundation Trust – the same Trust whose children’s heart surgery unit faced closure if the LGI unit remained open.
As local MP Greg Mulholland told BBC Radio 4’s Today programme yesterday,
the only way that Newcastle stays open is if Leeds closes.
Mr Hamilton’s behaviour may well have been exemplary, but it doesn’t matter. There is still an inevitable conflict of interest when a senior member of the decision-making committee works for one of the units being considered for closure – and any decision between Leeds and Newcastle is inherently unsafe.
An article in this Thursday’s Guardian criticised a “turf war” among surgical units:
What is happening in Leeds is a fight over the bodies of small babies born with heart defects. It would have been good to think it could have been sorted out in quiet, compassionate and well-informed discussion exchanging evidence around a table.
Maybe. But when the decision to close your unit is being made by a group that includes someone whose own unit will close if yours doesn’t, I’m not going to blame you for fighting tooth and nail instead of meekly giving in to it.
Now let’s look at the statistics that bear on both the initial decision to close the LGI unit and the post-judicial decision to suspend its surgeries.
Flawed statistics
Surgeons have to be calm under pressure and extremely focused. However, when I talked to one of LGI’s children’s heart surgeons this week he was clearly incandescent about what is being said about the competence and outcomes of the Leeds unit – that there have been severe safety concerns, including allegations that its mortality rates were twice the national average. These allegations were swiftly challenged from various quarters. Problems with the data were pointed out, resulting in the referral of the statistics for more detailed analysis. However, yesterday’s Guardian carried a prominent article (the same one that criticised the ‘turf war’) stating that subsequent analysis of Leeds’ corrected statistics still “shows that its death rates are unacceptably high”.
This appears to be completely untrue. According to my consultant contact, as of Thursday morning when the article was published, the data was still with the Leeds team for its input.
For such a claim to appear in a prominent Guardian article suggests that the newspaper’s journalist was intensively ‘briefed’ by someone closely connected to the issue with a desire to influence public perception ahead of the release of more accurate analysis.
Professor Sir Brian Jarman, author of the ‘HSMR’ system of statistical analysis of clinical outcomes, has published statistics indicating that, far from being ‘double’ the national level, mortality rates for children’s heart surgery at Leeds are in fact below the national average. It’s not often that Professor Jarman and I have found each other on the same side of a mortality rate issue, but in this case we seem to be.
The SaS steering committee is apparently using a different data set (CCAD or NICOR) to calculate mortality rates. These statistics are neither published nor even finished, and certainly shouldn’t be used to make radical statements about mortality rates at a particular unit.
Whatever the differences between the statistical systems, though, the ‘raw’ numbers will be the same or very nearly so in both cases, as these simply reflect the numbers of children who died without adjusting for ‘case mix’ and other factors. A look at these figures will give a strong indication whether there is a real problem in Leeds.
The incandescent surgeon told me that a national measurement of children’s heart-surgery outcomes shows an average “mortality rate within 30 days of surgery” of around 2%. The graph below shows the mortality rates as a percentage of total surgeries for all of England’s children’s heart surgery centres, plus the rate for the country as a whole, for the period from 2009-2013:
Far from being double, the Leeds rate (in red) of 1.77% is slightly below the national average (in yellow) of 1.79%. So much for ‘double the national average’. Alder Hey and Birmingham have far higher mortality rates, yet will remain open under the JCPCT‘s plan.
Any adjustment to such small data sets that will take a hospital from being below average on raw numbers to being ‘double‘ the average in the ‘standardised’ results is inherently unsound – because any statistical adjustment or assessment based on such small numbers is always unsound.
In order to achieve any kind of reliable statistical measurement, a large enough number of ‘episodes’ has to be recorded. If this number is too small, chance variations have a disproportionate effect and cause false trends to appear. The numbers of cases per hospital – even over a 4-year period from 2009 to now – range from a low of 384 cases to a ‘high’ of 1,223. Even the uppermost figure is too small for reliable, meaningful statistics. You might argue for working from figures over a longer period to increase the sample size – but figures from the more distant past can hardly be said to be relevant to the present performance and future fate of a surgical unit.
To illustrate the problems of small samples being used for statistics, let’s take a look at one of Professor Jarman’s graphs on mortality rates in children’s heart surgery, to which I’ve added some annotation:
This graph shows the standardised mortality ratios (SMRs) for the various children’s cardiac units. ‘Standardised’ means that they’ve been adjusted to try to even out case mix factors and others that might affect the likelihood of deaths at separate units.
‘CI’ refers to the ‘confidence interval’ – basically the level of certainty with which you can identify the rates. The marks above and below each point on the graph reflect what you can say with 95% certainty. As a large centre for biomedical statistics points out, the “confidence interval is directly related to sample size” – in other words, if you have small numbers you can’t be very certain!
This is reflected in the graph. Look at the top and bottom marks for the Leeds SMR. These show that if you want to have 95% certainty of being right, you can only say that the Leeds SMR (with 100 being exactly average) was somewhere between about 50 and about 180 – according to Prof Jarman’s details, 46.9 and 180.3, to be exact.
If you wanted to say with 98% or 99% certainty (the standard used normally for hospital mortality statistics according to Prof Jarman’s testimony to the Francis inquiry), the distance between the upper and lower points would be far greater.
This brief look at the limitations of statistical analysis shows clearly that any adjusted statistics based on samples of only a few hundred cases can’t possibly tell us anything reliable about the performance of a unit that, as our incandescent surgeon told me, is ‘operating on patients varying from a 600g premature baby to a 16 year old’.
No matter what statistical system the SaS or NHS executives are considering for their decision, the leaked accusation that the Leeds unit has twice as many deaths as the national average is not merely nonsense, but culpable nonsense.
The Dutch surgeon
As with the furore around Mid Staffs, there are serious question marks over the media’s reporting of the issues. Not only has the publication of the ‘double the average’ accusation been reckless or based on ulterior motives, but even perfectly innocent occurrences are slanted to appear incriminating. Take this example, which is from the same Guardian article:
The Guardian has also learned that a Dutch surgeon was being paid by the Leeds Teaching Hospitals NHS Trust to fly in from her job in Aarhus, Denmark, to operate one weekend a month – a highly unusual move in this very specialised medical field.
Do you see the insinuation? “Flying a foreign surgeon in once a month? What’s going on? What was this surgeon having to put right and what deficiencies must there be in the LGI’s own team?”
The real situation is quite simple, and far from incriminating. The surgeon, Catharina van Doorn, used to work at the Leeds unit and, according to my consultant contact, is extremely capable and has certain rare, specialised skills. She is brought across to perform surgeries when more surgical capacity and those specialised skills are needed. Which is a perfectly sensible, responsible course of action.
That such an innocent and even positive fact is used to insinuate something sinister or furtive raises serious questions about the objectivity – or lack of it – of this article and raises worrying concerns about undue exertion of influence from quarters with a vested interest in the outcome of this matter. It also serves to highlight the much less subtle way in which much of the media has pitched the story.
Back to ‘toxicity’
Pro-LGI campaigners will be celebrating the announcement that their heart unit is going to reopen, and rightly so. However, as things stand at the moment this is merely a reprieve. Unless the NHS board and the Department of Health have a complete change of heart, either Leeds or Newcastle is almost certainly going to be closed at some point in the not-too-distant future.
The fact that – whether intentionally or negligently – the LGI ‘brand’ has been ‘tofixied’ by the questions raised in the public confidence doesn’t bode well for its prospects of surviving the ‘Safe and Sustainable Review’ that aims to reduce number of England’s paediatric heart centres from 10 to 7.
The way in which figures have been leaked, snap decisions have been made on the suspension of services based on inherently-unreliable data, and the media has been ‘briefed’ to portray the story in a negative and fear-inducing way should be of serious concern to anyone associated with or dependent on LGI’s children’s heart services.
It should also be of grave concern to anyone who cares about fairness, or about the way in which the NHS as a whole is being targeted and portrayed by the government and the national media.
Is someone ‘fighting dirty’? You’ve now hopefully got some facts that you won’t pick up in the mainstream coverage of the events and are in a better position to decide!
(If you’re also incandescent about the way they NHS is being treated and would like to contribute to the fight to defend it, please visit CCGWatch.org.uk and consider making a donation.)
If the choice was between Leeds and Newcastle then based on the clinical results Newcastle rightly won. The legal challenge originally was about process. Now, it is a simple fact when it comes to rare and complex surgery that the more a surgeon performs it the better they get. I have a family, and to be honest I would be prepared to travel if it meant a longer distance if it meant a better chance of survival in cases such as this. One of the understandable reasons for the challenge is those families requiring ongoing support and treatment being faced with journey’s up to Newcastle. There doesn’t seem to be a way round this. Either both have to be kept, or if one is chosen it has to be Newcastle based on results between the two.
Having said all that, recent events really do smack of underhand tactics.
Reblogged this on patricktsudlow and commented:
Our lazy and incompetent media doing the work of a nasty and vicious government. Looking at the graphs, most of the hospitals are based in the South of the country. With the closure of Leeds or Newcastle, would leave the North of the country one unit for patients and parents to go to.
Graeme, you are right, to a point, about the more operations a surgeon does, the better he/she will be. What needs to be done, is for that point to be defined (based on evidence). That point has been defined by S and S as 400. THERE IS NO EVIDENCE FOR THAT. In fact the only evidence, from comparing the volume and outcomes of all the centres in the US, shows that anything above 250 and the mortality doesn’t change. If, as you suggest, the relationship is linear, we should only have 1 centre in the UK and everyone travels (after all we would all travel to the ends of the world for our children). So basically, with the recommendations as they stand, parents will have to travel, with no benefit in terms of outcome, after all, apart Oxford ALL the centres in England are doing more the 250 operations per year (and probably more than 300 by now). In addition the same review decided that the optimal number for our Scottish neighbours was 300 cases per year per centre. Roger Boyle’s famous quote about Sweden, and how well the centralisation has worked there, makes me laugh, as if the same standards were applied there, Sweden wouldn’t be able to have a children’s heart centre AT ALL! So perhaps what really needs to be done, is to look at who is pushing for the change (all those ex clinicians who are now really politicians) who witnessed the first Bristol scandal, determined a path at that point, and haven’t stopped to re examine where children’s heart surgery is now. Perhaps we should start listening to those surgeons who are actually working in the field NOW. How many of them can there be? 10 centres, 3 or 4 surgeons in each? Put them in a room together with a wood burner. We all wait outside, when the smoke changes colour, they can come out. Surely ownership of the changes by those actually running the service (that is true clinicians, not politicians dressed as clinicians) will lead to acceptance and driving of change, rather than constant accusations of NIMBYism.
Exactly!! In Germany 85% of their 4500 child heart ops are done in 26 – TWENTY SIX centres of which only 1 does more than 400.
But German mortality figures are as good as England, and so are Glasgow’s and Dublin’s- who also do <400.
S&S devised the 'standard' of 4 surgeons and 400/500 ops for other reasons!
Thank you – that’s coincidentally extremely timely info! Can you point me to any sources I can link to on those figures and outcomes?
https://twitter.com/GerardTubbSky/status/319804973472968704/photo/1
Here is a link with the figures by Welke, looking at over 55 000 ops. NO evidence that 400 is safer and will save more lives. This is a fallacy that has been generated by S and S and is now regularly reported (including incorrectly on radio 4 yesterday). Kennedy did not recommend reduction to 7 units, Monro did not recommend 400 cases per year (he recommended 300 per unit) and yet this myth presides that lives will be saved. Oxford was an outlier, it was the only unit doing small numbers. Now huge sums have been spent on a review, and it has to be justified. The total amount spent is still not being released under the FOI but we know that over £750 000 was spent on the initial PR prior to the start of the review! Now if nothing was seen to change, there will be a few people who would be looking very red in the face! The 400 cases per unit was not the only standard set, and apart 3 or 4 (including this one) the rest appear to be widely accepted by the clinical community. The introduction and strengthening of networks etc is invaluable and will improve the care available for families. However this remains a sticking point. Why we are now arguing over data, heaven only knows, as the review decided not to even consider it when determining which units to keep, which starts to raise even more questions. So we are back to this, a recommendation with no evidence to support it’s implementation and a lot of money spent on a review which now has to be justified. Hmmm.
There is nothing new here, except for idle gossip. I notice the other issues aren’t covered, like the surgeon who has been stopped twice for poor technique. I have no axe to grind as I work for the NHS in an area not affected by this issue. I am very disturbed by the actions of politicians in delaying the implementation of the SaS review which is vital if we are to make paediatric heart surgery safe. As a Labour Party member I’m also dismayed that Labour MPs are jumping on this bandwagon, though not to the extent that ConDem MPs are doing, I consider them to be totally irresponsible. I loathe what this government is doing to the NHS but this issue has NOTHING to do with politics.
While I understand the worries of some parents, and it’s by no means all, it should be the job of politicians to put safety first, not their number of prospective votes, and it really is time we put a stop to the empire building obsession of some NHS consultants.
Far too many people are involved in this that simply don’t understand the issues. Sadly there is no way of knowing yet how many lives are being lost or damaged by the delay in implementation.
You’re entitled to your opinion. But if a surgeon is getting stopped for poor technique, then the system is doing its job.
The idea that fewer, busier centres has become received wisdom, but it doesn’t seem to be borne out in practice – and if it isn’t, then the whole rationale for the SaS review is shot, because the ‘pain’ of the reduction in centres is guaranteed but the benefits look unlikely to materialise at all. But I share your loathing of what’s being done to the NHS by this government!
It’s not doing it’s job if a unit then has to employ locums or not provide 24/7 cover to make up for the deficit. I’m afraid the idea that clinical standards can be maintained in too many units is farcical. Same goes for transplant units. Maintaining clinical skills through training and practice is very, very important. Demanding that your local unit stays open is understandable, if misguided, but denying the need for rationalisation is ridiculous, and displays a much more dangerous agenda.
10 units is, what, 40 surgeons? Hardly an excessive national repository of skills in paediatric cardiac surgery. Any fewer would be vulnerable.
Leeds did almost 800 ops over 4 years. 200 a year is surely enough to maintain skills – and about double what Newcastle were doing. Since Newcastle supposed had better outcomes, that hardly supports the ‘more is needed’ hypothesis.
I’m not denying the need. I’m just recognising that it’s anything but proven – in fact, the evidence suggests the opposite so far, so the rationalisation supporters are the ones who need to prove their case.
If a “brand” is “toxic” and the reason for that is partly problems and mostly spin then if the problems are ironed out then why not “detoxify” by giving the public a clear and unequivocal message – well if you don’t, the intention is only too clear.
The recent happenings in South Wales vis-a-vis MMR vaccine show only too clearly that if the media are allowed to, or encouraged to “toxify” then unintended worse results may happen. For Stafford and Leeds in this context it’s all about access to services for the less privileged, for whom the NHS was set up in the first place.
Exactly!
Re the comment about locums – the impression given is that these were junior doctors, just out of med school. They were in fact experienced consultant surgeons. And I would imagine many hospital units would be using such staff over an bank holiday weekend. And I would like to know where the evidence has come from which says that Newcastle has better outcomes than Leeds? Sometime it feels like myths are being peddled as facts, including the regular one that lives are being lost through the delay in implementing the proposed changes. Even the chair of the S&S review has said regularly that all units are safe. Finally, please don’t have go at politicians over this. All our local MP’s of whatever persuasion are behind the campaign to keep Leeds Unit open and we need them to keep asking the questions. This is a political issue now.
The other point to make is that ALL centres try before they buy, ie consultants are generally given a locum appointment prior to being given a permanent contract. After all, most of us would agree that meeting someone once, and reading a CV doesn’t give you any real idea about how that person will fit into a team. Within cardiac surgery (probably more so than most specialities), having the support and trust of colleagues, where you are able to rub ideas off each other etc is essential and probably one of the keys to a successful unit. The irony of this review is that the surgeons from the ‘de designated’ units will be moved to the unit where their patients are going to go. SO Leeds surgeons to Newcastle! I’m not sure that I can envisage that ‘team’ working well together, especially in light of the whistleblower coming from Newcastle!! So just to clarify. The number of surgeons doesn’t change, just the location in which they operate will change (ie patients have to travel further to see the same surgeon) – it’s part of their NHS contract -special clause about if your unit closes. As I illustrated above, there is no evidence that units doing more than 400 will have any better outcomes. Basically more pain for the patient with no gain. Oh except it will be easier for surgeons to take annual leave, and they could even go away 2 at a time, rather than have to take it in turns!
Dear God, it just gets more and more insane!
Absolutely skwalker! Having watched developments closely since the first Bristol enquiry, there is far more to this than suggested in the media. Failure of the s and s team to disclose the cost, suggestion that they had an ‘open budget’ are all extremely concerning, particularly in light of lack of evidence, and failure to undertake a process which was fair and lawful (whether or not you want to argue it being a technicality). The DoH are now distancing themselves, and stating that they do not hold the information on expenditure (requested under foi act) The JCPCT (as was) refuses to comment, apart from the release of the £750 000 PR figure – published in HSJ. Huge sums are being spent and there doesn’t seem to be any accountability. The need to justify this expenditure is going to become more acute by the powers that be (whoever they are!!) Hence the drive to implement changes will be pushed and pushed, irrespective of the true best interests of our children.
In case you think this is as far as the insanity could go, the proposals for Mid Staffordshire (325 sq miles containing 300,000 people) are that ALL patients requiring acute care/maternity etc travel out of the area where, no doubt, they will meet staff who used to work in Mid Staffs and who are now travelling out of the area too!
Numbers of patients is often quoted as important – but there is VERY little evidence for this except in the most technically demanding specialities (and the numbers required are often not that high). The reason these things are trotted out is because they sort of sound right and are therefore hard to challenge especially in the sound-bite-o-sphere (have I just invented a word?).
Couldn’t agree more – big part of the reason why Mid Staffs and Leeds have my close attention!
Yes and, if bad data..
http://www.bbc.co.uk/news/health-22080650
Yup!
Reading many of the statements there seems to be some confusion over the terms data and analysis.
It is worth noting that the ‘bad data’ that was the trigger for closure is only a small part of the data picture.
Prof Jarman is quoted in the BBC article above, but as he pointed out on twitter (referring to his involvement in the initial Bristol scandal):
BrianJarman @Jarmann
31 Mar
@Seb4chuf @undunc At Bristol we had 4 databases, incl cardiac surgical register. We had to use HES cos clinical databases have missing data.
He has published the data and analysis using hospital episode statistics (HES) data – 100% complete and right up to date (although with it’s own limitations )
http://dl.dropbox.com/u/34930931/Paediatric%20cardiac%20surgery%2C%20April%202009%20to%20Feb%202013%2C%20under%205s%20%26%20under%2015s.doc …
This shows Leeds is not an outlier. However he did raise concerns that Liverpool might be:
BrianJarman @Jarmann
30 Mar
@Demotivatrix @GregMulholland1 Alder Hey has significantly higher SMR than the 3 lowest SMR PCS units. Would be good to check with CCAD data
One of the other databases that is used to collate data is PICANET – again Leeds was not an outlier. http://www.picanet.org.uk/
Remember that CCAD data is published a year in arrears and that the analysis performed, that led Leeds to be seen as an outlier, was using a new methodology which hadn’t been fully validated and (apparently) a number of systematic failures in the methodology had been identified since the ‘whistleblowing’ incidence. I am also a little in awe of Laura Donnelly’s article showing the graph depicting Leeds mortality alongside all the other units, particularly in light of the fact that only Great Ormond Street (where the new methodology and software is being developed in collaboration with CCAD) and one other unit has actually run their data through the new system. Where the other lines on the graph came from are a little mysterious!
So, apologies for the long winded rant, in conclusion, there are a number of methods which monitor mortality, and one new, not fully cooked method (but apparently with huge potential). The long standing methods have their limitations and caveats, but showed no issue. The new method, had not been fully adjusted – hence the anger from those designing it – and later to (apparently) demonstrated some issues, is the data and associated analysis that was used to close the unit. Also remember, from what I can understand, that CCAD data (the data used) runs a year in arrears. Does this add an additional slant to the use of data? Disclaimer. I do not live or work in Leeds. I have no connection with Leeds, however I do feel that only half the picture is out there!
http://nhscomplaint.wordpress.com/2013/03/30/why-the-ccad-data-was-likely-to-have-been-wrong-when-shutting-down-leeds-infirmary/
Latest from Prof Brian Jarman re why the CCAD was wrong