I wrote a week ago about the apparent ‘dirty fight‘ that is being conducted to achieve the closure of the children’s heart surgery unit at Leeds General Infirmary (LGI), highlighting some of the very suspicious aspects of the way surgery at the unit was suspended the day after pro-LGI campaigners won a key legal decision against the decision to close it.
A couple of days ago, I disclosed that the draft ‘Rapid Response Team’ report (which may or may not be released next week) looks set to call LGI’s unit ‘fully exonerated’ and ‘world class’ – in complete contrast to the leaked smears suggesting that its death rate was double the national average.
It appears that those who want to see LGI closed are determined to see through their plan and not to let the media battle get away from them in terms of how it sways public opinion. Today’s Guardian carried a prominent report with a headline that
Data shows that three hospitals – Leeds General Infirmary, Alder Hey and Evelina hospital were close to ‘alert’ levels
The headline presents a misleading picture, because the article goes on to quote the report as saying (emphasis mine)
the analysis does not show a significantly increased mortality rate
but the Guardian still states,
Leeds General Infirmary came “very close” to the “alert” threshold.
While the strict facts of the article are correct, the thrust of it is clearly intended to indicate that there are problems at Leeds, even when a section of the report says there are not.
However, the problems with the NICOR (National Institute for Cardiovascular Outcomes Research) report do not end with the way that it’s being covered by the media. The report itself, while trying to maintain a veneer of balance, contains elements that give serious cause for concern – not about the performance of LGI’s unit, but about the objectivity of the report.
Exceeding the brief
NICOR, and its report, are supposed to be addressing mortality statistics at heart rates across England’s 10 centres for paediatric heart surgery. However, it features a prominent criticism of the LGI that goes way beyond statistical analysis. In the context of deadlines and completeness of data submitted, NICOR states:
c) It was clear to the NICOR Steering Group that there were major deficiencies in the data submitted by Leeds (as exemplified by Table B). This has been addressed as actively as possible in the intervening period, as part of NICOR’s process to optimize data quality.
d) The new data included in the current analysis of 07/04/13 is the reason for the difference in the outcomes for individual Trusts from the preliminary data. The effectiveness of the data submission process could be considered as a measure of organizational culture and commitment to quality service delivery.
In other words, NICOR concludes that Leeds’ ‘failure’ to submit complete data in time is a sign of a lack of organisational competence and commitment to offering a good service.
A statistical data team can claim competence in the analysis of data. But it is way outside the brief and competence of such team to reach conclusions about the motivations for the dates and extent of a data submission. Moreover, the Leeds team has made it perfectly clear that it did not submit its data specifically because it was flawed – which in no way suggests any lack of commitment or quality.
For such a section even to be included in this report casts severe doubt on its objectivity and reliability – because it betrays a desire to cast a unit in a bad light that the data do not support.
The ‘wandering’ average
Even more worrying is something that casts doubt on the reliability of the data, the analytical process and the way in which it is portrayed. The report shows graphs of ‘standardised’ mortality plots for the 10 English children’s heart units for each of 3 years – 2009-10, 2010-11 and 2011-12 – and then one for the 3-year period as a whole.
But the graph for the total – which is the basis for the claim that LGI was ‘very close to the alert threshold‘ makes no sense.
To understand why it makes no sense, look at the graph below. For this graph I’ve superimposed the positions of LGI in each of the 3 years onto one graph, so you can see them together, and I’ve removed the plots for all the other heart units to remove any distraction. There appear to be only two plots for LGI, but that’s because the plots for two of the years fall in exactly the same position.
The green lines in the chart show the distance of each plot from the ‘alert’ zone – 2 years with a big distance, one year with a shorter distance.
Now look at NICOR’s graph that supposedly depicts LGI’s average position over the 3 years as a whole:
The green line for the distance to the alert zone is so small that it’s almost invisible.
If you take an average of 3 things, you add all 3 together and divide by 3. The average of two equally long lines and a shorter line will be a line that is shorter than the two long lines but longer than the shortest one – and nearer to the length of the two long lines, because there are two of them and only one shorter one. It cannot be shorter than all of them.
Or do it numerically. If two identical lines are 4cm long and a 3rd is 2cm long, their total length is 10cm. Divide by 3 and the average line length is 3.33333cm – nearer to 4cm than to 2cm.
NICOR would have us believe that the average of 3 years, in none of which LGI gets very close to the ‘alert’ line, is very close to the alert line.
It makes no sense and doesn’t stand up to scrutiny.
NICOR has released the graphs, but has not released the methodology for the ‘standardisation’ adjustments it makes to the ‘raw’ mortality data. But whatever those adjustments are, if they can take 3 years’ distances and average them to a distance smaller than any of them, there is something seriously amiss with their methodology – and with their results.
Edit: Professor David Spiegelhalter, the statistical advisor for the report, tells me via Twitter that the radically different position is a statistical effect because the greater number of results when the 3-year period is looked at as a whole means less uncertainty.
I have no reason to doubt Prof Spiegelhalter’s integrity about the numbers. However, as provider of ‘statistical advice and support’, he did not frame the way the information has been presented. As already noted, the report itself concedes that anything inside the funnel is not statistically significant. The report also admits,
The funnel shape arises because Units to the left of the plot are smaller and so we would expect more variability due to chance.
Here is the graph for the 3-year period showing the results for all the units:
You’ll notice that LGI’s plot is much further to the left than either of the other ‘close to alert’ units. So any result within the funnel is not statistically significant anyway – and Leeds’ result is far more uncertain than the two units it was grouped with in the report and the surrounding publicity.
Moreover, NICOR have not released the methodology of their calculations – the statistical weightings and adjustments applied to the raw data – and appear to be unwilling to do so. Since there is always an element of subjectivity and ‘(hopefully) educated guesswork’ in such statistical adjustments, the failure to offer these for public scrutiny also places a significant question-mark above the results.
Yet none of this critial nuance was included in the media reports, nor in the unprofessional statement to the media by Roger Boyle (who has advocated LGI’s closure and can hardly be considered an impartial witness).
You wouldn’t want to do that..
On the same day, Roger Boyle – who works for NICOR – issued a statement that appears extremely unprofessional from someone closely connected to a supposedly objective report, not to mention one completely disconnected from what the report actually says. As the Guardian reported his ‘bombshell’,
“We find they’re just on the edge of what we call an alert – in other words, showing that they were right on the edge of acceptability,” he told the BBC.
If his own children needed surgery, “I would go somewhere else”, he said. “I would go to Newcastle.”
A damning verdict. And yet the very same NICOR report that he refers to concedes that
Units inside the funnel have a relative risk that is not ‘significantly’ different from average
By definition, around half of all Units will have more deaths than ‘expected’. It is therefore inappropriate to label centres as ‘blameworthy’ for these deaths, as the analysis does not
show a significantly increased mortality rate.
Any plot within the ‘funnel’ does not have a statistically significant higher mortality rate, even if it is near the ‘alert’ line. But Leeds is nowhere near the alert line anyway – until it somehow miraculously cosies up to that line in that strange 3-year plot.
The NICOR report admits, as I have already shown with reference to the overall hospital mortality statistics that were supposed to show Mid Staffs to have high mortality, that there are serious drawbacks with any ‘average’ depiction of performance, because half of the units measured are always going to appear ‘worse’ than average.
So what’s really going on?
Skewed reporting, misleadingly-presented statistics, a report that goes way beyond its competence to make unfounded judgments about motives, and a public statement by someone who works for NICOR (and has consistently wanted Leeds closed) that doesn’t match NICOR’s own report on the same day.
All in complete contradiction of the conclusions of the ‘Rapid Response Team’ who actually inspected the unit and considered it ‘fully exonerated’ and ‘world class’ – a report whose findings were leaked to me by a source very close to events in Leeds, but which may not be officially released to influence public opinion.
Some people might consider that LGI is being set up to take a fall by vested interests who are determined to see it close no matter what. At the very least, there are serious concerns – and serious questions need to be asked about the provenance, methodology and motivations of an evidently-skewed set of data and conclusions.
What do you think?