NICOR: KNOWN data flaws and a ‘turf war’?

NICOR (National Institute for Cardiological Outcomes Research) is the organisation whose statistics were used to justify the suspension of services at Leeds General Infirmary (LGI). Leaked comments about this data led to headlines claiming that LGI had mortality rates for its children’s cardiac surgery around double the national average, even though its basic death rates were exactly on the national average.

Even after it was pointed out that the data were incomplete and the statistics were not ready for release, and a ‘corrective’ exercise was carried out, the media were still using the resulting figures in NICOR‘s report to claim that LGI’s mortality statistics were ‘very close to alert levels’.

NICOR‘s report stated (although this was largely ignored by the media) that any position within the ‘funnel’ of the statistical graph (as LGI’s was) did not mean that there was a mortality problem:

Units inside the funnel have a relative risk that is not ‘significantly’ different from average

However, there is an even more significant problem with the way in which LGI’s mortality statistics have been portrayed, and conclusions about them have been drawn.

I’ve come into possession of the minutes from the ‘Congenital CCAD stakeholder’s meeting’ in January this year – CCAD stands for the Central Cardiac Audit Database and is the data set on which NICOR bases its statistics. NICOR was well represented at the meeting, with 3 people participating. The minutes of this meeting are revealing – and worrying.

The data

Most crucially, the minutes reveal that NICOR knows the data on which it has based its damaging statistical conclusions are flawed – and continue to be flawed even after the supposed ‘corrections’. There is no single statement to this effect in the minutes, but the cumulative effect of a number of statements in the ‘NICOR update’ around data quality is inescapable:

Little has changed in the last year, with some centres carrying out congenital procedures (mostly catheter interventions rather than surgery) without sending us any data. It appears that BCIS still regard PFO and ASD closure as their domain, and are even considering setting up their own database for these procedures.

This statement concerns data for adult heart surgery, but if data can simply not be submitted for adult procedures, it raises serious questions about whether the same is happening for paediatric surgeries.

this year four new procedures will be added to the specific procedure outcomes shown on the public portal – ICD implants, transcatheter pulmonbary valve implants, right ventricular outflow stenting and ductal stenting.

So, at least 4 cardiac procedures that happen often enough to generate significant data are not included in the 2009-2012 statistics – these four are to be added from this year, but how many others are missing? If statistics do not cover all procedures, they cannot be assumed to provide a comprehensive and reliable picture of safety at any unit.

We need more volunteers for data validation visits and are pleased to involve trainees. DQIs for endocarditis data are still below par and it often proves difficult to validate data

Surgical centres are visited to validate their data – but the visits rely on volunteers to complete them. NICOR is behind on these visits to such an extent that it calls for more and is prepared even to involve ‘trainees’, who are by definition going to be inexperienced and inexpert. Unvalidated data is therefore making it through to the statistics – and even some of the data that is validated will have been checked by people who lack the expertise to do it.

And now the clincher, which is encapsulated in three short passages:

There were two false positive red liners in the initial analysis of the provisional data – both turned out to be errors, one due to erroneous submission of post infarct VSD closure and one due to a combination of inadequate coding and a glitch in our procedure allocation algorithm..

The importance of accurate recording of comorbidities for risk adjustment was emphasised.

The complex partial risk adjustment model (PRAiS) has been validated and, taking into account diagnoses as well as procedures, comorbidities, other risk factors and real mortality data from CCAD appears to be more robust than any previous model. Whilst it was emphasised that there can never be “perfect” risk adjustment in such a complex field, there was unanimous support for this work and for implementing it in our data analyses. It is clear that if Nicor do not publish such data it will be done far less well by others.

From these passages, we can parse the following clear conclusions:

1. There are known issues with data quality, including ‘erroneous submissions’ and ‘inadequate coding of comorbidities’.
2. The algorithms used in the standardisation have had at least one ‘glitch’, but that’s just what NICOR have identified – nobody knows what other glitches might remain that have not yet been spotted.
3. Even in ideal circumstances, NICOR knows that ‘there can never be “perfect” risk adjustment’ – and things are far from ideal.

As I’ve already shown, and is clear from testimony to the Francis inquiry about the Mid Staffs ‘HSMR’ mortality statistics, data errors and ‘lack of coding depth’ can lead to huge variations in the apparent death rates in ‘standardised’ statistics. These variations can only be amplified by the fact that the cardiac mortality data set covers a tiny number of cases relative to the overall NHS statistics for all condition.

As NICOR pointed out in its latest report on mortality relating to children’s heart surgery, “it is inappropriate to make comparisons based on crude mortality rates” – so statistical adjustments are made to try to even out the effect of factors such as the complexity of the operations carried out, other conditions suffered by the patient etc.

But NICOR knows there are problems in the quality of its data, as its minutes make clear – if it has to ’emphasise the importance’ of recording comorbidities properly, then clearly they are not always being recorded, and the 3 years worth of data on which its claims about Leeds are based are flawed – very probably fatally so.

Since the long-term future of LGI’s children’s heart unit is still under threat and significant damage has been done to public perception of its services, the fact that extreme conclusions and drastic actions against the unit have based on statistics known to be drawn from flawed data and on algorithms known to be imperfect at best is an extremely serious matter.

None of these nuances and doubts have been communicated in the headlines about the unit. Nor have they been reflected in statements such as the one made by Sir Roger Boyle that he would not send his children to the unit – for which he has since resigned. But why would a statistical institute release conclusions and a report based on data it knows to be faulty?

The why and the wherefore

A clue to this may lie in a ‘reading between the lines’ of other statements by NICOR in its section of the minutes:

The parents’ support groups as well as the other stakeholders present were all very much opposed to outcome data being released by different sources and strongly support the concept of a single trusted source (Nicor) endorsed by the professional societies, the DH and other stakeholders.

In all, three requests for data have been received from Dr Foster, all of which were rejected for a variety of reasons including concerns about their proposed methodology.

The original funding for PRAis (from NIHR) has been used, so we are left with the problem of funding the software to be provided to each centre.

They (the NHS specialist commissioning group) plan a publicly accessible source of outcome data for paediatric cardiac procedures but strangely had not contacted Nicor! Happily they will continue to liaise with us and, we hope, will choose Nicor as the platform for this data

It appears that BCIS still regard PFO and ASD closure as their domain, and are even considering setting up their own database for these procedures. Nicor’s meeting with commissioners suggested that commissioners wish to have a single source of audit data.

Taken together, these statements appear to show an organisation which is:

• short of cash
• aware that the commissioners of specialist services want to have a single source for outcome statistics
• engaged in a programme of establishing itself as that ‘single source’
• denying access to CCAD data to other organisations to protect its position

It really isn’t NICOR‘s job to judge Dr Foster’s ‘methodology’ etc – rather, this suggests a desire to prevent any alternative portrayals/conclusions from being available to compete with NICOR‘s ‘sole provider’ ambitions. Similarly, the indignant statement about BCIS regarding any data as ‘their own’ or ‘even’ considering setting up their own database suggests considerable fear of competition.

Could NICOR be releasing statistics and reports on data prematurely, as well as over-claiming for the reliability of the conclusions because of funding pressures and to further its ambitions to become the sole provider statistics in its field? If so, then the surgical ‘turf war’ criticised by the Guardian is not the only territorial crossfire that LGI finds itself caught in.

If this surmise is correct it adds yet another disturbing conflict of interest to the already-revealed issue with the presence of a cardiac surgeon from Newcastle on the ‘Safe and Sustainable‘ steering committee deciding whether LGI or Newcastle will close – in no less a position than that of vice-chair of the committee.

I’m preparing an article on the whole issue of whether there are really grounds for closing any of the existing children’s heart surgery units, as the evidence does not appear to support the claims from some quarters that fewer, bigger centres performing more operations are needed for best outcomes.

But if any centre is to close, then the decision as to which should be made with absolute impartiality and transparency, and without any hidden agendas and conflicts of interest. The judge who ruled the original closure decision illegal clearly recognised that there were serious problems with it in this regard – but now, as NHS England seeks to appeal against that ruling, it seems even more patently clear that this cannot be claimed of the process surrounding the decisions and events concerning Leeds so far.