As you’re likely to know if you’re visting this blog, I’ve written extensively on ‘Hospital Standardised Mortality Ratios‘, or HSMRs, and the many weaknesses and potential for inaccuracy of a system that was used by media, government and some campaign groups to bludgeon Stafford hospital to the verge of destruction and is now being used to target other hospitals for similar treatment.
This statistical system, which is so full of problems at the data-capture end as to be meaningless and whose use was riddled with potential for conflicts of interest because it might force ‘poorly-ranked’ hospitals into paying £35,000 a year to the company that publishes the statistics to remedy the ‘problems’, is currently being used by the government to target a further 14 NHS hospitals in England for investigation and potential break-up. That’s just a fraction under 10% of England’s 147 acute hospitals.
Well, I’ve come across a very interesting piece of information regarding the criteria used to target these hospitals – or actually a couple of pieces. Here’s a tweet from yesterday by Professor Sir Brian Jarman, the creator of the HSMR system:
What’s wrong with this picture and what does it tell us? Let’s take a look.
‘Twins’?
There are two main statistical competitors measuring mortality rates in hospitals. HSMRs, as we’ve already seen, are one. The other is ‘Summary Hospital-Level Mortality Indicators‘, or SHMIs. But in spite of the similar names, the two ‘twins’ are not very alike: because of key differences in how the statistics are calculated, SHMIs are the measure considered more reliable by most statisticians and clinicians.
What are these differences? Well, a very helpful webpage provided by the NHS tells us:
- SHMIs include all deaths, not just the 80% sample used by HSMRs. This means they are more likely to give an accurate reflection of the whole picture.
- The inclusion of deaths outside acute hospitals in the SHMI but not in the HSMR. This means hospitals are less likely to look better just because they have a big hospice nearby, for example – or because they discharge patients to die at home more often.
- The factors adjusted for vary between the two indicators – there are some differences in how death rates are ‘standardised’ to allow for variations in age of population etc, but the key difference is: SHMIs only count each death once.
Hang on. Let’s go over that last one again. HSMRs don’t only count a death once?! Yes, exactly. Here’s how the helpful NHS webpage puts it:
If you are treated in two hospitals in the run-up to your death – according to HSMRs, you died twice, once in each hospital.
This HSMR system is based on massively flawed data that even showed 17,000 men being given obstetrics (pregnancy) treatment last year and has a host of other potential problems. But that the system itself counts the same death more than once just about takes the biscuit.
To be (investigated), or not to be?
Professor Jarman’s tweet mentions, as if it was barely significant and just in passing, that the hospitals under investigation are
14 trusts high HSMRs over 2 yrs (5 chosen having high SHMI)
But let’s ‘unpack’ that statement. The government has chosen 14 hospitals for investigation and ‘special treatment’ based on them having high HSMRs. ‘After all, Stafford had high HSMRs and it was killing people, right?‘ Well, no – but for details on that, read the articles linked at the top of this post.
But whatever the rights and wrongs of the way Stafford has been portrayed (and they were almost all ‘wrongs’), 9 of the 14 hospitals showing high HSMRs do not have high SHMIs. That’s 9 of 14 – over 64%.
Imagine you’re the government. You have two sets of statistics to choose from: HSMRs – full of holes, built on very crumbly data foundations, not counting people sent home or to hospice to die, and counting some deaths twice. These stats let you target 14 hospitals.
Or SHMIs – stronger, counting all deaths (but only once!), even if they occur within 30 days of leaving hospital, and giving a much fuller, more accurate picture – but ‘only’ identifying 5 hospitals to target.
I don’t think there’s any question which a reasonable person would choose, do you?
Yet the government has chosen to include 9 hospitals with no mortality problem at all according to the better statistical system alongside the mere 5 that might have a problem according to both.
Since SHMIs are still dependent on the quality of coding that is very far from complete and accurage (remember those poor pregnant men), even the 5 hospitals might turn out not to really have a mortality problem at all.
Yet the government has knowingly chosen to prefer the shakier of the two systems to increase the number of hospitals it can target and then close or break up. If that makes you think that this situation has very little to do with death rates and improving patient safety, and everything to do with senior politicians who are glad of any opportunity to close down NHS hospitals, I think you’re absolutely correct.
And not only is the government targeting hospitals for closure that really don’t appear to have a problem and preferring a more-flawed system over a better one – but it’s also put the man who devised the shaky system on the panel that will recommend what to do with them.
Insult and injury
Professor Jarman admitted to the Francis inquiry that his statistics could not be used to calculate numbers of avoidable deaths – but that didn’t stop him appearing on BBC News, the BBC’s Newsnight programme and in the press claiming that his stats showed there had been 20,000 avoidable deaths because of NHS failings.
That statement is nonsense, because his system looks at the total number of deaths, calculates an average, and calls that ‘expected’. He’s then looking at deaths of the average and calling them ‘excess’ – but because it’s an average, that means that:
- For the 20,000 deaths above the average, there will be exactly the same number of deaths that didn’t happen in the below-average hospitals.
- The total number of deaths in the NHS is still the number that happened – and under his system is also the ‘expected’ number (‘expected’ is just based on that total number and not on any clinical estimation of what should happen.
- If you prevented the 20,000 deaths, you’d just have a new average – and you’d still have large numbers of deaths occuring in hospitals with above the average death rate. Because that’s how averages work.
The flaws are obvious – and when you realise the flaws, you realise how false the ‘NHS killed 20,000 people’ headlines have been (but if you want more details, read those linked posts at the top of this article).
The man who has fuelled them is now on the panel judging hospitals that have fallen foul of his system rather than being found to have clinical failings by clinical experts doing proper clinical assessments.
The usual suspects?
The 9 hospitals with good SHMIs that are being unfairly targeted for their HSMRs are:
BUCKINGHAMSHIRE HEALTHCARE NHS TRUST
UNITED LINCOLNSHIRE HOSPITALS NHS TRUST
MEDWAY NHS FOUNDATION TRUST
THE DUDLEY GROUP NHS FOUNDATION TRUST
GEORGE ELIOT HOSPITAL NHS TRUST
SHERWOOD FOREST HOSPITALS NHS FOUNDATION TRUST
NORTHERN LINCOLNSHIRE AND GOOLE HOSPITALS NHS FOUNDATION TRUST
BURTON HOSPITALS NHS FOUNDATION TRUST
NORTH CUMBRIA UNIVERSITY HOSPITALS NHS TRUST
If I were you and lived near any of these hospitals, I’d be out on the streets waving placards and demanding that the government keep its greed-motivated hands off my local hospital and stop claiming without justification that they’re killing patients.
For that matter, if I lived near any of the other 5, I’d be doing the same – since both SHMIs and HSMRs are based on data that is full of errors and omissions.
And then I’d be demanding that that any judgments on hospitals are based on objective, dispassionate, clinical assessment and that the government learn the real lesson of Stafford hospital and its HSMRs and stop underfunding and short-staffing hospitals so they have a chance to do the job we love them for.