Reporting Outcomes: On Whose Terms?
November 18, 2019
Google Reviews
November 18, 2019
FEATURE
Publishing clinical outcomes is a hugely complicated issue, but some doctors say the age of quality transparency is coming – whether professionals like it or not. The risk then becomes – if doctors aren’t leading this movement, who will?
BEFORE INSTANT REPLAY, 3D rendering, and hot spot technology, sports officials were Gods.
Their decisions were final. Coaches, players, and fans could shake their fists and throw their hats to the floor in disgust, but at the end of the day they had to suck it up. A call – even a bad call – was ineradicable.
But as technology evolved, multiple camera angles combined with frame-by-frame reviews gave sports fans a vantage point that was sometimes superior to that of the umpire or referee.
Almost overnight, the mist on the mountaintops where sport officials were revered completely dissipated, and transparency became the new buzzword.
A good case study on transparency is the US National Basketball Association. NBA referees are arguably one of the most analysed professionals in the world.
In 2015, the NBA started publishing the ‘Last Two Minutes’ – officiating reports that contain reviews of every call (or non-call) made by referees in the final two minutes of close games. The reports include the call type, committing and disadvantaged players, comments, the review decision, and a link to the video of the play. For the first time in print, referees’ mistakes were laid bare for all to see.
While referees may have been nervous about it initially, it has allowed them to dissect their performances and, ultimately, improve.
Data not only reveals mistakes; it can reveal bias. Even prior to publishing the Two Minute Reports, the NBA has been using data to reveal whether referees’ calls were biased on certain factors. Did they favour the losing team? Were they more likely to benefit the home team – thus the long-held theory of ‘home court advantage’? Did they have an implicit or unconscious racial bias? A study published in the Quarterly Journal of Economics in 2010 found they did. Specifically, their research found white referees tended to call more fouls on black players, while black referees tended to call more fouls on white players. However, after being made aware of their implicit bias, referees changed their behaviour. Later studies revealed a significant reduction in this type of implicit bias.
In 2017, the NBA expanded its focus on data-driven game review with the intention of creating objective referee measurement standards and track the progress of call accuracy and errors per game over multiple seasons.
Has data made NBA refs perfect? No. But they now have about a 95% accuracy rate on all calls and non-calls, which they’ve been able to maintain for the last few years.
Moving beyond basketball – AMA (NSW) President, Dr Kean-Seng Lim argues we should be applying the same principles of data-driven performance to medicine.
“If we are looking at a world where there is increasing transparency and accountability, then we as individuals and we, as an organisation, probably need to be thinking about that,” he says.
*If you’re really keen to hear more about NBA officials, instant replay and call accuracy, listen to Michael Lewis’ podcast Against the Rules – a fascinating seven-part series that unpicks Americans’ relationship to authority and rules.
AMA Position Statements: Indicators and outcomes
The health system has been looking at clinical and performance indicators since the 1980s. In the last four decades, clinical indicators have been developed for specific diseases and services, as well as patient safety and clinical governance areas.
Quality improvement is supported through several mechanisms, including the Australian Commission on Safety and Quality in Health Care (ACSQHC), Federal Government funded practice incentives, audit and peer review processes, accreditation, and continuing professional development.
The AMA’s Position Statement on Clinical Indicators, developed in 2012 and revised in 2016, states, “Monitoring health care quality is impossible without the use of clinical indicators. They create the basis for quality improvement and prioritisation, and when assessed over time, provide a method of assessing the quality and safety of care at a system level.”
In 2016, the AMA also released its Position Statement on Measuring Clinical Outcomes in General Practice.
In this statement, the AMA echoed the ability for clinical outcomes to be used to improve quality in healthcare.
“Medical practitioners use outcome measures intrinsically to assess when treatment should be changed, referred to another, or discontinued. While outcomes may not be the only measure of quality, they by and large provide the ultimate validation of the effectiveness and quality of medical care.”
Measuring outcomes involves using clinical indicators across several areas. This includes patient reported outcome measures (PROMS) and patient reported experience measures (PREMS). AMA’s position statement on clinical outcomes notes, “Medical practitioners have generally been more comfortable with the use of clinical indicators which measure practice processes and clinical activities, as these are components of care that can be reliably measured and that clinicians have more direct control over.”
Transparency in hospitals
While patient groups are increasingly demanding greater insight into safety and quality measures in hospitals, the drive for transparency at the individual clinician-level has raised concerns within the profession. What does the push to report surgical outcomes mean for proceduralists?
When media reported on obstetrician and gynaecologist Professor Stephen Robson’s decision to publish surgical outcomes data, the reaction from the medical community was mixed.
Australia’s Chief Medical Officer, Professor Brendan Murphy, applauded the move toward transparency.
‘‘Any doctor claiming superiority has an obligation to provide the data to back that up,’’ Professor Murphy told the SMH. “The overall standard of surgeons in Australia is very good, but there are some surgeons at the top of their specialties that are probably worth paying more for, and they are certainly not necessarily the people currently charging very high fees.’’
Professor Robson said his decision to publish the data was not made lightly. After a period of deliberation, he published a five-year audit of his surgical practice on his website, detailing the number of surgeries he had performed and benchmarking his complication rates against clinical indicators published by the Australian Council on Health Standards.
He told the SMH, ‘‘I just thought ‘oh, bugger it’. It’s time for me to help patients make an informed decision about the quality of care they get and maybe I can inspire my colleagues to do the same,’’ he said.
‘‘Surgery is like sex: everyone talks up their performance,’’ Professor Robson added. ‘‘But getting the objective data to verify their claims is much harder to come by. When something is veiled in secrecy it can feed mistrust and anxiety.’’
However, the Royal Australasian College of Surgeons (RACS) did not support the move. The College warned that the data could be misinterpreted by patients and there was a risk that this type of reporting could lead to comparisons between doctors that would unfairly rank those who took on more complicated patients, or doctors at the beginning of their careers would be unfairly judged. It also warned there could be a temptation to “cherry pick” less complicated cases.
Publishing Outcomes
Dr Fred Betros strongly supports collection of clinical data on patient outcomes, but suggests his concern (and that of others) is, how would it be used?
“The collection of data on patient outcomes is nothing new – it is what we are doing with it that is the main issue here,” he says.
“There is a real fear within our profession, that this type of reporting can be seen as having a ‘witch-hunt’ mentality. I believe that identifying a problem on its own is worthless. If you look for problems actively, you also need to be part of the solution and have a means to help rectify the situation.”
What are the Benefits?
Public health experts and health consumer groups argue there are a number of benefits to publishing clinical outcomes, including increased transparency for consumers which heightens a patient’s ability to make an informed decision about their healthcare, as well as opportunities for clinical improvement.
The Australian Commission on Safety and Quality in Health Care published a study “Public
reporting of safety and quality in public and private hospitals,” in March 2019 which examined findings from expert groups, as well as consumer and clinician focus groups to inform options for national reporting standards of safety and quality in health care across public and private hospitals in Australia.
The report’s findings speak to a consumer appetite for greater transparency around safety and quality information.
In looking at reasons why safety and quality information should be reported publicly, consumer focus groups believed it would allow patients to make more informed choices.
At the moment, there is little objective and transparent information available. In the absence of such information, many rely on information given by the healthcare system and healthcare providers, as well as the opinions of friends and family who have had similar experiences.
Consumers also indicated the reporting of safety and quality information would compel hospitals and providers to improve coordination of care, as well as processes and communication.
In addition, the consumer groups felt public reporting would arm patients with information about potential or actual areas of safety concerns within hospitals.
Lastly, consumer groups in the report suggested hospital-level information was a good first step, with a view toward clinician-level reporting.
Clinician focus groups, however, did not agree with clinician-level reporting, with one interviewee noting this “elevates the importance of one person above the clinical team”.
Clinician focus groups in the report did stress that public reporting of safety and quality information at the hospital-level would provide several benefits, including driving improvements in clinical quality and promoting transparency which would support patients in decision-making.
Clinicians also noted that in the absence of good public reporting, too often the gap is filled by overwhelmingly negative media stories. Public reporting was also cited as a means of enabling GPs to provide an objective assessment of performance between hospital providers and refer their patients accordingly.
They found data collection and reporting arms patients with information to make better decisions about their own healthcare. This includes the role GPs have in providing advice to consumers.
According to Dr Betros, “First and foremost, ensuring best outcomes for our patients should always be the leading benefit of this type of data collection / reporting. A patient should be reassured that when they see a practitioner, that doctor can benchmark themselves against their peers in an objective and data driven way. Additionally, in the long run this also protects the practitioners. It indicates to the purchaser of that care (whether it be the patient, Medicare, private insurers or the public hospital system), that there is real value in the service provided.”
What are the Challenges?
Publishing clinical outcomes is not without its challenges.
While the AMA supports mechanisms for quality improvement, it cautions against the misuse of clinical indicators in its AMA’s Position Statement.
“There is an inherent danger that a focus on achieving clinical indicators and performance against them diverts attention from patient care. This danger is particularly present if outcomes measured against clinical indicators are used to; dictate or impose levels of safety or quality or for pay for performance purposes.”
Other dangers, as highlighted by the RACS, are that some practitioners could be unfairly judged for taking on more complicated patients, or doctors at the beginning of their careers might be deemed less competent. There are also concerns some doctors might avoid high-risk cases. In addition, there could also be a tendency to over-treat patients and ignore whether that’s in the patient’s best interest.
Dr Betros stresses that if public reporting of outcomes was to become more commonplace, we would need to find a way to use this data to improve the care provided by practitioners who sit outside of the data norms.
One other potential challenge, which was cited by several clinicians in the ACSQHC’s “Public reporting of safety and quality in public and private hospital report” was that decision-making processes could be hindered by fears of negative consequences.
The report noted, “Several clinicians across all clinician focus group forums commented that a significant potential barrier to public reporting was the lack of political will to publish information. The potential for negative consequences from poor performance was viewed as impacting the decision-making process and hindering more transparent sharing of information.”
It is also noted by clinicians, that in team-based care scenarios, it is difficult to separate performance of one clinician. If there is a bad outcome, who’s fault is it?
One potential alternative would be to report outcomes at a hospital ward level, or another option would be to report outcomes by grouping clinicians by speciality within a hospital, but there is no consensus within the medical profession to wholly support those options.
AMA’s Statement on Clinical Indicators is clear that performance data should not be used for promotion or disparagement.
It also states that the selection of clinical indicators to be used for quality improvement “should be entirely voluntary” and driven by the local level as this improves the scope “for ‘buy-in’ from the practice workforce.” It also stresses that there should be no cost to use the clinical indicators and so practices’ administrative burden isn’t increased as a result.
Who should be responsible?
The question remains, if greater transparency and publication of clinical outcomes were to become commonplace, who should be leading this process?
The AMA’s Position Statement on Clinical Indicators, states, “For high standards of quality and safety to be assured it is essential that there is a strong clinical involvement in and ownership of the development of clinical indicators and the processes set up to measure and to assure safety and quality. Clinical indicators should therefore, independently of government, be developed and ratified by the relevant medical specialty.”
The focus groups – both consumers and clinicians – in the ACSQHC’s “Public reporting of safety and quality in public and private hospital report” did not have a consensus on where responsibility should lie for public reporting. However, there was a common view that ideally, a single, national body should be empowered to release information.
What should be reported?
Clinicians in the ACSQHC’s “Public reporting of safety and quality in public and private hospital report” took a general view that no information should be off limits, but there was concern that too much information without context would not be helpful.
Consumers identified indicators such as infection-rates, surgical mishaps, post-intervention and procedure outcomes, medication errors, readmission rates, length of stay and waiting times, as well as reporting outside of the hospital environment, as being most relevant.
Clinician groups in the report expressed a desire for increased transparency around relative risks and benefits of procedures such as caesarean sections.
How should it be reported?
There is a clear consensus that any data that is reported needs to be done in a way which is meaningful to both patients and doctors, that both helps in decision-making (in a patient’s point of view) and improvement (in a clinician’s point of view).
Case Studies
Hand Hygiene
In terms of using data to drive change, Australia’s handwashing campaign has been cited as a major success story. Since it was implemented in 2009, the National Australian Hand Hygiene Initiative (NHHI) has driven significant improvements in handwashing compliance and reduced the transfer of potentially harmful bacteria.
“One of the most transformational things we’ve done in healthcare, the thing that has really made a difference in people’s lives, is simply focusing on better hand hygiene,” says Dr Jaspreet Saini, general practitioner. “That was really a big ‘A Ha!’ moment for me.”
Research presented at the 2019 European Congress of Clinical Microbiology & Infectious Diseases found that for every 10% increase in hand hygiene compliance, there was a corresponding 15% decline in the incidence of Staphylococcus aureus infection in 132 public hospitals in 2016-17.
In the study, researchers analysed outcomes of the NHHI from Jan 2009 to June 2017. Hand hygiene compliance auditing is done three times a year by trained auditors who collect data using direct observation of their colleagues when in clinical encounters. In NSW, more than 220 facilities regularly submit data as part of the NHHI. The clinical impact of the scheme was analysed by linking data on hospital-level incidence of Staphylococcus aureus infection with hospital-level hand washing compliance.
In NSW, facilities use data to compile reports that can identify areas of high compliance or low compliance with hand hygiene. These areas include departments, professional groups, as well as compliance with specific ‘moments’.
Interestingly, a study released in 2018 by the University of NSW found handwashing compliance among doctors and nurses dropped from 94% to 30% when human auditors stopped monitoring their behaviour and researchers used automated surveillance.
This suggests data collection and the reporting of data collection has an influence on behaviour.
Surgical Variance Reports
Medibank teamed up with the Royal Australasian College of Surgeons (RACS) to create a series of Surgical Variance Reports reports which analyse clinical and other indicators for common procedures within surgical specialties, including general surgery, urology, ear, nose and throat surgery, vascular surgery and orthopaedic surgery.
The first of the reports were published in 2016, based on de-identified Medibank claims data from 2014. In 2017, a second series of reports were created on surgical variance within surgical specialties, as well as a review of same-day surgery for hernia repair.
Additional reports published in 2018 and 2019 respectively look at rehabilitation pathways following hip and knee arthroplasty and volume-outcome relationships in pancreatic-oduodenectomy.
“The Medibank Private initiative with the Royal Australasian College of Surgeons has some merits, but I am cautious with respect to some areas of their reporting,” says Dr Betros.
In Dr Betros’ area of practice, Medibank offered individual surgeons the ability to look at the data regarding that individual’s personal performance against their peers who perform similar services.
“I participated in this exercise as I was certainly interested to see how I performed personally, and I wanted to see the type of data Medibank was using in its reporting.”
The report analysed KPIs such as length of stay, unplanned readmission, unplanned ICU admissions, which Dr Betros says are “good markers of quality”. However, it also looked at measurements such as the average cost of an episode of care and the average cost of the prosthesis used.
“These are obviously important considerations for us as clinicians and whilst this may prompt some of our colleagues to look at their practice for areas where responsible, improved financial efficiency can occur, I am concerned it could have other unwanted effects.”
Dr Betros argues that to solely look at the cost of an admission, without ensuring quality outcomes are maintained is a concern.
“If we are to report on data that is driven by cost savings, there also needs to be a mechanism that ensures all drivers of change keep the outcome of the patient treatment as the highest priority,” he says.
“At the moment, I see no safeguards in place to protect our patients and our clinicians from having treatment options potentially influenced by some of these financial drivers if an organisation chooses to do so – especially when the organisation is the payer of these services.”
Whilst he acknowledges financial viability is important to organisations such as Medibank and they should have some input into how their funds can be spent in the most efficient way, we need to carefully consider how much input they have on treatment options.
“If we are going to report on outcomes and performance, it has to be in a way that ensures we provide the best possible care for our patients, by clinicians who can demonstrate an appropriate level of performance – standardised and accepted by their peers, using our limited resources in a clinically responsible way.”
Colonoscopy Certification and Recertification Program
An example of how service provision can be standardised, is to look at the current nationwide Colonoscopy Certification and Recertification Program. The Australian Commission on Safety and Quality in Health Care has provided for the Gastroenterological Society of Australia (GESA) to oversee a triennial program aimed at ensuring colonoscopic services provided in Australia meet a minimum standard set. It was initially a voluntary program, but now participation is mandatory, with clinicians being required to provide de-identified outcomes for three blocks of 50 colonoscopies performed by that clinician over a three-year period.
According to Dr Betros, “There are very clear standards that are required to meet ongoing certification, such as the rate of reaching the caecum / terminal ileum plus adenoma and sessile serrated adenoma detection rates. These standards have been proven to represent good quality colonoscopy and it is reasonable to expect practitioners providing this service to meet those standards.”