9/14 hospitals investigated for high deaths DON’T have high mortality?

As you’re likely to know if you’re visting this blog, I’ve written extensively on ‘Hospital Standardised Mortality Ratios‘, or HSMRs, and the many weaknesses and potential for inaccuracy of a system that was used by media, government and some campaign groups to bludgeon Stafford hospital to the verge of destruction and is now being used to target other hospitals for similar treatment.

This statistical system, which is so full of problems at the data-capture end as to be meaningless and whose use was riddled with potential for conflicts of interest because it might force ‘poorly-ranked’ hospitals into paying £35,000 a year to the company that publishes the statistics to remedy the ‘problems’, is currently being used by the government to target a further 14 NHS hospitals in England for investigation and potential break-up. That’s just a fraction under 10% of England’s 147 acute hospitals.

Well, I’ve come across a very interesting piece of information regarding the criteria used to target these hospitals – or actually a couple of pieces. Here’s a tweet from yesterday by Professor Sir Brian Jarman, the creator of the HSMR system:


What’s wrong with this picture and what does it tell us? Let’s take a look.



There are two main statistical competitors measuring mortality rates in hospitals. HSMRs, as we’ve already seen, are one. The other is ‘Summary Hospital-Level Mortality Indicators‘, or SHMIs. But in spite of the similar names, the two ‘twins’ are not very alike: because of key differences in how the statistics are calculated, SHMIs are the measure considered more reliable by most statisticians and clinicians.

What are these differences? Well, a very helpful webpage provided by the NHS tells us:

  • SHMIs include all deaths, not just the 80% sample used by HSMRs. This means they are more likely to give an accurate reflection of the whole picture.
  • The inclusion of deaths outside acute hospitals in the SHMI but not in the HSMR. This means hospitals are less likely to look better just because they have a big hospice nearby, for example – or because they discharge patients to die at home more often.
  • The factors adjusted for vary between the two indicators – there are some differences in how death rates are ‘standardised’ to allow for variations in age of population etc, but the key difference is: SHMIs only count each death once.

Hang on. Let’s go over that last one again. HSMRs don’t only count a death once?! Yes, exactly. Here’s how the helpful NHS webpage puts it:


If you are treated in two hospitals in the run-up to your death – according to HSMRs, you died twice, once in each hospital.

This HSMR system is based on massively flawed data that even showed 17,000 men being given obstetrics (pregnancy) treatment last year and has a host of other potential problems. But that the system itself counts the same death more than once just about takes the biscuit.

To be (investigated), or not to be?

Professor Jarman’s tweet mentions, as if it was barely significant and just in passing, that the hospitals under investigation are

14 trusts high HSMRs over 2 yrs (5 chosen having high SHMI)

But let’s ‘unpack’ that statement. The government has chosen 14 hospitals for investigation and ‘special treatment’ based on them having high HSMRs. ‘After all, Stafford had high HSMRs and it was killing people, right?‘ Well, no – but for details on that, read the articles linked at the top of this post.

But whatever the rights and wrongs of the way Stafford has been portrayed (and they were almost all ‘wrongs’), 9 of the 14 hospitals showing high HSMRs do not have high SHMIs. That’s 9 of 14 – over 64%.

Imagine you’re the government. You have two sets of statistics to choose from: HSMRs – full of holes, built on very crumbly data foundations, not counting people sent home or to hospice to die, and counting some deaths twice. These stats let you target 14 hospitals.

Or SHMIs – stronger, counting all deaths (but only once!), even if they occur within 30 days of leaving hospital, and giving a much fuller, more accurate picture – but ‘only’ identifying 5 hospitals to target.

I don’t think there’s any question which a reasonable person would choose, do you?

Yet the government has chosen to include 9 hospitals with no mortality problem at all according to the better statistical system alongside the mere 5 that might have a problem according to both.

Since SHMIs are still dependent on the quality of coding that is very far from complete and accurage (remember those poor pregnant men), even the 5 hospitals might turn out not to really have a mortality problem at all.

Yet the government has knowingly chosen to prefer the shakier of the two systems to increase the number of hospitals it can target and then close or break up. If that makes you think that this situation has very little to do with death rates and improving patient safety, and everything to do with senior politicians who are glad of any opportunity to close down NHS hospitals, I think you’re absolutely correct.

And not only is the government targeting hospitals for closure that really don’t appear to have a problem and preferring a more-flawed system over a better one – but it’s also put the man who devised the shaky system on the panel that will recommend what to do with them.

Insult and injury

Professor Jarman admitted to the Francis inquiry that his statistics could not be used to calculate numbers of avoidable deaths – but that didn’t stop him appearing on BBC News, the BBC’s Newsnight programme and in the press claiming that his stats showed there had been 20,000 avoidable deaths because of NHS failings.

That statement is nonsense, because his system looks at the total number of deaths, calculates an average, and calls that ‘expected’. He’s then looking at deaths of the average and calling them ‘excess’ – but because it’s an average, that means that:

  1. For the 20,000 deaths above the average, there will be exactly the same number of deaths that didn’t happen in the below-average hospitals.
  2. The total number of deaths in the NHS is still the number that happened – and under his system is also the ‘expected’ number (‘expected’ is just based on that total number and not on any clinical estimation of what should happen.
  3. If you prevented the 20,000 deaths, you’d just have a new average – and you’d still have large numbers of deaths occuring in hospitals with above the average death rate. Because that’s how averages work.

The flaws are obvious – and when you realise the flaws, you realise how false the ‘NHS killed 20,000 people’ headlines have been (but if you want more details, read those linked posts at the top of this article).

The man who has fuelled them is now on the panel judging hospitals that have fallen foul of his system rather than being found to have clinical failings by clinical experts doing proper clinical assessments.

The usual suspects?


The 9 hospitals with good SHMIs that are being unfairly targeted for their HSMRs are:


If I were you and lived near any of these hospitals, I’d be out on the streets waving placards and demanding that the government keep its greed-motivated hands off my local hospital and stop claiming without justification that they’re killing patients.

For that matter, if I lived near any of the other 5, I’d be doing the same – since both SHMIs and HSMRs are based on data that is full of errors and omissions.

And then I’d be demanding that that any judgments on hospitals are based on objective, dispassionate, clinical assessment and that the government learn the real lesson of Stafford hospital and its HSMRs and stop underfunding and short-staffing hospitals so they have a chance to do the job we love them for.


  1. I agree with you on this.

    On the list – I have worked in a few. Let me give you my views.

    UNITED LINCOLNSHIRE HOSPITALS NHS TRUST – Not as bad as implied by the masses
    MEDWAY NHS FOUNDATION TRUST – does have lot of complaints etc.
    GEORGE ELIOT HOSPITAL NHS TRUST – Excellent trust. I have worked there.

    I do agree with you in that my first impressions of the defamed hospitals are that

    1. They are unable to defend themselves against bad publicity

    2. They should not be vilified in the media like this without tangible data substantiating the allegation. There isn’t any.

    3. It is a risk to public safety to defame hospitals falsely as it will have the negative effect of dissuading them to attend. This affects patients confidence and faith in the staff.

    I don’t think the media or Jarman know what they are doing by parading hospitals in the media like this.

    Anyway, well done. Another superb piece.

    Dr Rita Pal

  2. Very well put again. Once you grasp the concepts here, the nonsense is all too apparent.

    As I’ve said before, in all likelihood, if mortality stats were done perfectly, pretty much all hospitals would oscillate around 95-105. Reason – the medical staff are pretty well trained (and those over 50 phenomenally experienced) and advances are disseminated quickly – so with universal coverage (hooray!) the public get about as good a result as possible (yes there will be occasional errors and disasters).

    The main hinderance to all this is ACCESS. You all know about postcode lotteries for new treatments. Well imagine how much this would be magnified if your local facility is downgraded or shut and becomes further and further away. Then imagine you are poor, or less educated/well informed or disabled or …. you get the idea. Since none of this makes any clinical sense it MUST be a political act. And just in case you are wondering, it’s unlikely to save money.
    Firstly Trust mergers haven’t improved quality, access or finances (Bristol Group Study) so the next mad scheme can’t necessarily be trusted. Secondly, the downsides of hospital closures are never properly accounted for because the savings accrue in one budget but the downsides in another. For example you close Hospital X. Patients travel to Hospitals Y and Z. The Health economy saves 10m (it probably doesn’t, disappointingly, but stay with me). The Ambulance service spends another 3m whizzing people around the area; patients and relatives spend another 3m on travel, work lost etc (not available to spend in the local economy), other parts of the system (eg social services) spend 4m extra on account of having to deal with people they’ve never heard of from miles away (cock-ups, extra time spent – everyone knows this because we all deal with occasional out of area patients and they always take a huge effort and time but as there are only a few, it can be done) ….. I’m sure I’ve only scratched the surface. Remember the system is marketised/fractionated rather than collaborative and integrated so lots of these perverse incentives exist. If you drive the system primarily by finances (rather than quality and safety) then the concern for your own budget will inevitably rise and your concern for anyone else’s is bound to fall.

    This fight is Sooooo worth fighting!

  3. Oh and with a projected DoH UNDERspend against budget of £2bn, who is trying to pretend that struggling hospitals are unviable rather than simply in need of some assistance (financial or for that matter non-financial). I hope Mr Burnham is ready to ask that one when the year-end figure is announced because the NHS won’t be getting the £2bn back!

    1. Exactly – couldn’t agree more! I covered the ‘clawbacks’, surpluses and backdoor fund-removals in some detail a few weeks ago.

  4. Steve – is it “corruption” of figures also that Keogh is pandering to reference Leeds Heart story today? I wonder how long it will take them to shut up the MPs who are actually now saying Leeds is under attack and wrongly. Keep up the good work. Giselle.

    1. Unfortunately I don’t have data to make a judgment on Leeds – but the timing is extremely suspicious! This government has ‘form’ for moving the goalposts after a judicial defeat 🙁

  5. In a previous blog comment I mentioned the radio 4 programme “More or Less” (statistics/maths). It turns out they have looked at HSMRs and Stafford – in 2010 (“listen again” 3 minutes in, until 14 minutes at:


    though this only got halfway to the reality, as you can imagine).

    A number of listeners have suggested they have another look at it!

  6. There is an excellent article by Paul Taylor in current issue (11 April) of the London Review of Books. This gives a very good picture for the non-specialist of the history, problems and issues of hospital mortality data.
    The LRB has a fairly gentlemanly paywall which usually allows non-subscribers to view the odd article.


Leave a Reply

%d bloggers like this: