In the most recent issue of Private Eye magaine, the Medicine Balls columnist shows a graph of figures concerning the use of the Z51.5 palliative care code for deaths at a number of Midlands hospitals. He or she claims of this graph:
Here is a pretty picture. It shows how Trusts can disguise high death rates by recoding those who die as palliative care (code Z51.5). They are then “expected to die” and disappear from the published death rates.
Here is the graph the Eye showed:
This chart, which was taken from written evidence submitted by Prof Brian Jarman of Doctor Foster, shows 3 West Midlands Trusts, plus Medway Trust in Kent, and an average for all the other acute Trusts in England minus the 4 aforementioned Trusts.
Looks damning, doesn’t it? But does it show what we’re meant to think it shows?
Specialist subject: the bleedin’ obvious
Professor Jarman – and the Eye article – assert that the graph demonstrates how Trusts can ‘disguise high death rates’ by using the palliative care code. Prof Jarman told the inquiry that he believed the Trusts were ‘gaming’ his system to make themselves look better.
But look at the green line in the graph, the one which represents Walsall Trust.
Walsall’s line jumps from almost zero up to the end of 2006 to around 9% at the beginning of 2008 – to almost 80% by the 4th quarter of 2008. Can anyone seriously believe that Walsall Trust was trying to fool people that 4 out of every 5 people who died at its hospital were being treated for terminal illnesses?
Of course not. So that’s the ‘deliberate gaming’ theory shot down in flames just from a first glance.
TLC
What was happening at Walsall during the period where the line shoots up sharply was a misunderstanding among its coders about what constitutes palliative care.
Palliative care constitutes a defined set of treatments that are given in a defined set of circumstances, aimed at making someone comfortable during the very final stages of his or her life. The Liverpool Care Pathway, about which I’ve written previously, is one example. Administering drugs to sedate someone who is panicking because they can’t get their breath at the end of their life is palliative care – but administering similar treatment to someone who is not imminently dying is not.
The confusion at Walsall was between palliative care and what they were calling ‘TLC’ – similar care given to someone who was not imminently dying, but neither were they going to get better. Making someone comfortable, easing breathing and so on – with a view to sending them home (or to a care home etc) once they were settled, is ‘TLC’.
But Walsall’s coders were coding it as palliative care – hence the huge spike in the green line, which then drops sharply once the error was identified and corrected.
Hocus, Focus..
Now let’s strip out some of the extraneous information. We know Walsall’s line was very wrong, and why, so let’s remove that from the graph. Since Walsall’s figures must have a drastic effect on the average line for the 3 West Mids Trusts, we’ll remove that too. I’ve edited those out as cleanly as I can with my limited graphics skills, and here’s the resulting graph:
That’s clearer, isn’t it? Without so many lines you can see more clearly. As I wrote in my first post on Mid Staffs’ HSMR mortality rates, when the palliative care code was first introduced to make the coding across the whole country reflect more accurately what was actually happening, the Trust’s coding department did not make the switch to using it.
This is borne out by the graph, which shows Stafford’s line at or very near zero until the end of 2007, while the line representing national use of the code is climbing steadily from 2004. The graph also shows that the Medway Trust in Kent failed to realise the change at first, but it caught on earlier, and its line rises from the beginning of 2007 rather than the end.
So far so good. But the lines for Stafford (and George Eliot and Medway) still look significantly higher than the national average. So surely that’s a problem, isn’t it?
Well, no – but let’s see why.
A jury of its peers?
Prof Jarman’s Doctor Foster’s, along with the Healthcare Commission (HCC) had grouped Stafford into a ‘peer group’ of other Trusts. When asked by Jonathan Pugh, Information Manager at Mid Staffs, what the basis for this grouping was, the HCC didn’t know and Doctor Foster’s didn’t say. But as Mr Pugh told the Francis inquiry in his written statement:
Mid Staffs always argued that this group was not representative of the Trust and we put forward our own peer group.
So, Mid Staffs felt that the peer group was chosen to make them look bad, but even so, as Mr Pugh testified to the inquiry:
There is always someone at the top and a graduated reduction as you go down the graph. Without Medway and Walsall you could have easily included the next 8 in the graph.
In other words, Mid Staffs was put into a group for the purposes of the graph that reflected badly on it – but even in that group, if the others in the group had been included on the graph, we would have seen a spread of lines across the graph.
Seeing an even spread of lines would show that there was considerable variation among Trusts in their use of the Z51.5 palliative code – just as there was considerable variation in the handling of all data relevant to HSMRs. So much variation, in fact, that it made the HSMR results meaningless.
It almost looks as if the candidates chosen for the graph were deliberately selected to make Mid Staffs and other West Midlands Trusts look bad – look like they were faking the figures. Could this be true?
Back to Mr Pugh. In his statement to the inquiry, he points out that
Airedale NHS Foundation Trust, in quarter three of 2006, had a peak of 3.5 times the national average for coding of deaths with palliative care.
Wow. Mid Staffs’ palliative coding – at its worst – was about 3.8 times the average at the time (and less than twice the average in 2010). Surely Airedale was identified as a ‘gaming’ Trust and targeted for auditing and possible sanctions?
No. As Mr Pugh points out, Airedale had been
‘Small Trust of the Year’ in Dr Foster’s annual hospital guide on numerous occasions.
So, Mid Staffs is accused of fiddling the system while Airedale, with very similar use of the Z51.5 code, wins awards. Draw your own conclusions.
The big picture
We’ve already noted that Mid Staffs was allocated a ‘peer group’ by Doctor Foster’s – a group which the Trust disagreed with vehemently on the grounds that it made the Trust look bad. But even so, Mr Pugh’s evidence to the Francis inquiry included a table that showed a comparison of palliative care episodes for each of the peer-group Trusts, including Stafford.
That table bears close examination. Here it is:
This table tells a very clear story – but because it’s a lot of figures, it’s quite easy to miss it. So, at great expense (well, an hour’s worth of me typing the figures into a spreadsheet and creating a chart), here’s how those figures look when put into a graph:
What this chart shows is Mid Staffs (the red line) starting at almost zero in 07/08 and then climbing sharply as use of Z51.5 kicks in. But it is then rapidly overtaken by most of its peers and finishes 3rd-lowest out of the 7 peer Trusts (although there’s hardly anything separating most of them. So Stafford’s use of the palliative code is entirely in line with other hospitals considered its peers.
And remember, Stafford’s codes were audited by the Audit Commission – and found to be 97-98% correct, an unprecedented score. So there’s no chance it’s ‘gamed’.
This peer group was chosen by Dr Foster’s – yet the only one they chose to show on the chart presented to the Stafford inquiry was Mid Staffs, making Mid Staffs and its neighbouring Trusts look like ‘outliers‘ – like the odd ones out, the ones that ‘must’ be up to no good, because they’re so far above the national line.
And don’t forget, the chart you’re seeing above still gives a false impression of a lot of coding – until you notice that the top of the ‘Y’ axis showing the percentages denotes only 1.2% of all the codes entered by those hospitals in those years.
If I make the top of the ‘Y’ axis 100%, here’s what it looks like:
All the lines are so low – fractions of a percent – that you can’t even separate them.
So much for Mid Staffs ‘gaming’ the system.
The even bigger picture
You might now be thinking, ‘That’s all well and good – but Mid Staffs Z51.5 codes are still quite a bit higher than the national average. That’s still suspicious.’
I understand why you might think that – but you’d be wrong. What you can’t see from any of the tables or charts presented is what’s going on outside the figures that still affects them – and there are a factors that easily account for the difference.
What you don’t know might hurt you..
Prof Jarman presented his graph contrasting the West Midlands Trusts (plus Medway) against a line for the national average. We’ve already seen that a number of other Trusts – the ‘peer group’ that was selected, but then ignored, by Dr Foster’s – had very similar patterns of coding for palliative care. If those had been shown on the graph, Mid Staffs and its neighbouring Trusts wouldn’t have looked anything like so unusual.
But they weren’t. And if they weren’t, then it has to be asked: how many other similar Trusts were omitted which distorted the picture even further?
And at the bottom end of the range of figures lie a similarly-unknown number of Trusts with very low use of Z51.5 – and they must exist to bring down the ‘national average’ line low.
Perhaps a lot of Trusts are still not using the code as much as they should, just like Stafford and others did at the outset but later corrected. Any Trusts not using the code much – rightly or wrongly – will drag down the average and make other Trusts look worse on a graph like that presented by Prof Jarman.
And a lot will be using the code a lot less, for perfectly legitimate reasons:
The hospice factor
As I showed in my original HSMR article, HSMRs only measure people who die in hospital. Fair enough, you might think – but there is one big factor that can affect the HSMR scores among hospitals in a completely unseen way: local hospice care provision.
Some hospitals are situated in areas with a number of local hospices. My own area is one such, with a number of hospices situated within the catchment area of my local hospital. This means that people in the final stages of their life and who need palliative care are far less likely to receive that care in hospital – and if they don’t get it in hospital, it never shows up in HSMRs.
By contrast, some Trusts are in areas with little or no hospice provision. Their terminally ill patients are going to die either at home or, far more usually, in the hospital – and if the coding department is doing its job properly, the Z51.5 code will be used a lot.
This means we should expect a huge variation between hospitals with great hospice availability and those with poor or even none.
And Stafford? As one local resident put it, Stafford has
a piddling little hospice.
So, with very modest hospice provision you’d expect Stafford to be high on the chart for use of Z51.5 – but because some hospitals will have even less access to hospice care, not right at the very top.
Which, strangely enough, is exactly what we see in the charts – both mine and Prof Jarman’s.
Whether deliberately or not, the chart showed by Professor Jarman and his Dr Foster’s group present an extremely misleading picture – one that happens to make their HSMR system look useful while making Mid Staffs look like it must be ‘gaming’.
Unfortunately, Private Eye appears to have bought into the myth in the most emphatically wholesale fashion – and its article therefore merely adds to distortion and misunderstanding that is being foisted on the British public, to the detriment of not just Stafford, but of the NHS as a whole.
It’s taken me a few hours to look more closely at the facts and see that the tale presented by the Eye isn’t worth the paper it’s printed on. What a pity they didn’t do the same before they printed it, and avoid stoking the fires of suspicion an anti-NHS sentiment that the Tory-led government are intent on spreading for its own purposes.
Still, I had to pay £1.50 to buy the magazine and get a clear scan of its chart – so maybe for them swallowing a line is good business.
Private Eye’s MD is Dr Phil Hammond (phil@drphilhammond.com) if you want to get in touch.
I agree that this week’s graph doesn’t feel quite right – I definitely got the feeling that it needed more context, so thanks for digging it out. I don’t think that Private Eye are ‘swallowing a line’ out of any kind of cynical business decision though, rather that they have a shotgun-type approach to scandal combined with a tendency to assume the worst!
The ‘good business’ line was just a throw-away as a handy ending, really. But they definitely seem to have swallowed hook, line and sinker – as the article on Mid Staffs and David Nicholson showed, with its emphatic repetition of the ‘100s of deaths’ myth.
Thanks for the info, though!
Did the Levison report cover sloppy journalism? It certainly isn’t a neutral act, given the effect that misleading stories can have on real people and populations.
Data analysis is always disputed – as it was in Bristol, where they claimed their death rates were high because they operated on sicker kids. When experts investigated, they found this not to be the case. They were just very slow surgeons who lacked the expertise to do highly complex child heart surgery.
The point is that all HSMRs do is highlight the need for an independent investigation that pulls the notes, observes the practice and ascertains the causes. That’s the only way to find out if the care is substandard or the patients are just sicker. And there are some very good examples (starting with Bristol) with managers and clinical staff using HSMRs to uncover poor care, fixing it and then finding the HSMRs improved.
Your point that the average should never change is crap. Continuous quality improvement should see the curve shift to the right, so the average should improve year on year, and each year half of all hospitals will be below average, some by chance alone, some because of their sicker populations and some because of shit care. The only way to find out which is to go and look. But the bottom line is that data has to be accurate, and acted on appropriately to protect patients from avoidable harm.
We have enough statisticians in the NHS. What we need now is effective inspectors and regulators to investigate outlying data, user complaints and staff concerns. At Mid Staffs and Bristol, you had all 3, which is highly suspicious. Your points about Private Eye are so wide of the mark they discredit the legitimacy of your otherwise very thought-provoking analysis. Statistics only take us so far – care has to be obseerved close up.
Phil
I agree that HSMRs are – at best – a nudge to ‘go and look’. So why are you – or ‘MD’ if that’s not you – still repeating the ‘hundreds of deaths’ myth in Private Eye if my comments about the magazine’s position are so wide of the mark?
The case-note review of the deaths at Stafford found ‘perhaps one’ questionable death among the 50 personally reviewed by the head of the independent review. Since there were some 200 cases whose families asked for a review, that means perhaps 4 such deaths.
Avoidable deaths happen at every hospital in the world, of course. It’s an inevitable consequence of large-scale healthcare. But the question at Stafford was, ‘Is the hospital killing more people than other hospitals?’ And the answer, according to a reasoned analysis of the statistics, is emphatically ‘No’.
Yet the Eye as well as all the mainstream media are propagating a damaging myth. The article in the previous Eye had none of the responsible caveats about ‘hints to go and check’ and so on, and called for David Nicholson’s resignation based on the 100s of needless deaths that had supposedly happened on his watch. Were there needless deaths? Certainly – all over the NHS and every other healthcare system in the world. Would the same have been true under a different CEO? Absolutely.
Professor Jarman – on whose analysis you/MD have based the articles – conceded to Francis that HSMRs cannot be used to calculate avoidable deaths, and Francis’ report said the same. But Prof Jarman has, nonetheless, appeared on BBC News claiming 20k avoidable deaths on exactly the same basis that 400-1200 were claimed at Stafford, even though he agreed his system couldn’t be used to draw those conclusions.
Something stinks like shit, certainly. But it’s not just the poor care that happened in some – some – parts of Stafford, which appears not to have increased the mortality rate anyway, once the proper depth of coding was applied.
As for rebasing – it would be fine, as long as it wasn’t used for ‘needless death!!!’ headlines and the stats were used with the proper caution. But it has been, and they were not used with caution.
I agree we want to monitor continued improvement – but that would be far better achieved by calling ONE year ‘100’ and then measuring success/failure in reducing to 98, 95 and so on, year on year.
Changing ‘100’ every year to whatever the average was the previous year just makes hospitals look bad when they might not be. Improvement isn’t without variation to a line, and the different case mixes etc at hospitals might make different rates of improvement unavoidable, no matter how hard the logarithms try to allow for them. And that’s without the problems with consistency of input data and practice.
That said, whichever way you use it is probably ok – as long as you don’t publish misleading ‘league tables’ that seem designed to push hospitals to pay for monitoring systems to try to move up the league.
It’s an EXTREMELY complex, nuanced situation – and none of that nuance is reflected in how the media headlines and articles have been written. Including, sadly, those in the Eye.
Thanks. I don’t think high HSMRs are a nudge to go and look, I think teir an obligation too. Very retrospective case analysis, such as at a PI, is very hard to do accurately. Far better to have real-time monitoring and investigate at the time. Notes are often so poor and incomplete, with bits going missing, it simply isn’t possible to estimate deaths accurately so long after the event. After Bristol, I argued for the equivalent of a ‘crash investigation team’ to go in quickly to areas where serious concenrs are highlighted either by data, user or staff concerns. The report and recommendations would then be put into the public domain in a timely way. Very retrospective public inquiries are no subsitute. And, as I say, there is good evidence thattrusts that have improved safety and quality in response to high HSMRs (rather than just recoding), have seen the HSMRs fall. So I think they can be far more useful than you’re giving them credit for. And it provides a basis for estimating lives saved if all hospitals with high HSMRS acted similarly on them.
You’re right – excess deaths are estimates, and we will never know the precise number so long after. There is good evidence from many countries that healthcare harms 1 in 10, and the older and sicker you are, the more you are at risk from avoidable harm. In 2001, the Bristol Inquiry concluded as many as 25,000 people may die each year from avoidable harm in the NHS , Hunt says 30,000 excess chronic disease deaths occur in the UK compared to benchmarked European countries, the NHS Atlas of Variation finds shockingly wide variations in quality of care and outcome for common disease such as diabetes and kidney failure. All estimates, all very complex. To get closer to the truth, you have to have an ingrained safety culture in the NHS, to measure harm on every ward in real time and go and look quickly when you discover it. Labour wasted a huge oppurtunity to do this with the extra NHS fnding after Bristol, and until we do start measuring harm properly, we won’t know the true figures. I err on the upper estimate of harm to try to get politicians an the NHS to take it seriously. If they do and the true figure turns out to be much lower, I will say so.
HSMRs aren’t ‘real time’, though – even if they were accurate. Hospitals submit them and then have up to 3 months to tweak them (I’m working from memory, so it might be 2, but still a good while) before they’re finalised.
Stafford responded on both fronts – it instituted a ‘peer review’ of all cases on any ‘red bell’ warning from DFI’s RTM system AND got its coding independently audited (and the auditors found the HSMR scores were 25-30% wrong).
Stafford’s standardised and crude mortality rates were falling by the time the public furore started (and HSMRs fell further once the coding was corrected) – but you’d never guess that from the headlines or even from the information put out in the Francis report.
I’ve written in one of my earlier articles that HSMRs CAN be useful – but only if the input is standardised by thorough training and strict audit. But there’s no sign of that happening – all that is happening is that HSMRs are being exploited by the government as a club to beat down other hospitals with.
As for the 25k – the BMJ’s thorough study recently brought that estimate down a lot, to around 11k/year or lower. I wouldn’t trust anything Hunt says further than I could throw him – and I’d enjoy throwing him!
The problem with erring on the upper side isn’t politicians taking it seriously – it’s the current lot in government using it to attack public affection for the NHS so they can dismantle it even further. Under a less anti-NHS government (whatever their public statements say), it would be less critical.
You’re right. Politicians may well use HSMRs for their own agenda, merely to close hospitals. And the retrospective lifting of gagging orders could present Hunt with a handy shit storm of bad NHS stories
that happened under labour. However, the real story about Staffs was that very good and truly appalling care coincided at the same time in the same hospital. And the hospital simply didn’t sort out the truly appalling care. So whether it fulfilled its statistical requirements is neither here nor there. And the fact that it took a screen snapshot of the one occasion its HSMR came down to near 100 to support its foundation trust application suggests its motives weren’t entirely pure.
So I guess we come back to statistics or concerns pointing the way for people to visit to the ward to observe the standards of care. This is far more important than retrospective argument about statistics or case note analysis. When the health care commission finally did this at staffs, they found some devastatingly bad care. I don’t think that can be denied, even if the absolute number of avoidable deaths will always be disputed.
“The government” (and it’s always the government, whoever you vote for) could have brought in an air transport style open incident and near-miss reporting system years ago. Why didn’t they?
The above correspondence is all very interesting but it does not, fundamentally, answer the question as to why the press are sloppily recycling old news about Stafford Hospital (whether you buy into the whole thing or a proportion is immaterial) rather than informing the public that by all objective criteria AND on the ground (more inspections than anyone else or just ask the patients and staff like the CQC did) there has been a huge improvement. So much so that other Trusts are learning from that. The story ought to be – why is the government hell bent on removing all acute services from the area instead of taking this heaven sent opportunity to learn about quality improvement? Private Eye should investigate! Dr Hammond are you still out there? Dr Hammond ….
I await his reply with interest!
You highlight precisely my concern with public inquiries. This one happened at least three years too late, and itts findings should have informed the government’s health reforms, so the NHS could be rebuilt
on quality and safety rather than marketisation. Because we have precious little robust comparative data for different clinical specialties,
commissioning is largely still done blind. And is a current financial
climate, it’s likely that hospital closures and mergers will be driven by money rather than quality and safety. Add in the fact that the vast majority of NHS staff are disinterested and disengaged with the NHS
reforms, and trust neither the leadership of our politicians or the NHS
commissioning board, and I just don’t see where the cherished clinical leadership is going to come from. To save the NHS, wise and
compassionate clinical leaders need to step up and fight its corner.
I’ve tried to put safety at the centre of the NHS since I broke the story of the Bristol heart scandal over 20 years ago, and we still haven’t safely reorganised child heart surgery. So while I’d like to be optimistic
for the future of the NHS, I just can’t be at the moment. But please
keep me in touch with how the NHS copes in Staffordshire and surrounding areas.
Will do. But doesn’t the fact that staff are disengaged with reforms say all that needs to be said about those reforms?
A good deal of agreement seems to have broken out here.
As for Mid Staffs, instead of the local population benefitting from a very improved situation (the least they deserve surely) they are now busily writing to the press and organising marches so that there are any acute services at all.
There are two important things the NHS brought to the ordinary people of the UK. 1) Services free at the point of use paid for out of progressive taxation. 2) Services that could be accessed by those ordinary people. It is proposed that the people of Mid Staffs (and especially the poor and elderly) lose No 2 (and, therefore, to an extent, No 1).
Amen to that. It’s why the irresponsible (and usually malicious) headlines have driven me up the wall..
The number of community hospitals in the DGH catchment area also need to be factored in to make a peer group of DGHs anywhere near comparable.