Patients should not use The Leapfrog Group's Hospital Safety Score, which gives hospitals an A to F letter grade based on their likelihood of causing harm, because it's inaccurate, the American Hospital Association said this week.
The scoring applies a 26-measure questionnaire that is biased, uses unreliable measures, applies variable weights to the same measures for different groups of hospitals, and contains significant errors, AHA President and CEO Rich Umbdenstock said in a five-page letter to Leah Binder, Leapfrog President and CEO.
Further, Umbdenstock said, "we are hearing from enough of our members about significant data issues that we are concerned about the manipulation of the data."
On Wednesday, Binder responded, addressing each of the AHA's three points. She said Umbdenstock is incorrect or misinformed in many of his claims. "With regard to the idea that Leapfrog deliberately manipulated data: This is a very serious charge for you to make without offering a single example to support it. We will launch a full investigation of any such example should you find one," she wrote.
The Leapfrog Group, a non-profit organization launched by employers who wanted to improve hospital quality of care, released safety scores on June 5 for 2,651 hospitals based on the occurrence of preventable events as falls, pressure ulcers, or bloodstream infections, and whether they have safety systems in place to prevent errors and complications. Hospitals retaliated in anger, saying they were caught off guard, and heatedly criticized the survey results.
Days later, some in the hospital industry went further, saying they suspected Leapfrog was trying to drum up more support for its annual hospital survey, alleging that if a hospital participated in Leapfrog's data collection, they got a better safety grade.
The AHA's three main criticisms and Binder's responses are as follows:
1. Bias toward hospitals participating in The Leapfrog Group's survey
"Chief among our concerns is that the methodology The Leapfrog Group uses appears to favor its own survey over other similarly reliable sources of information," Umbdenstock said.
Of all the hospitals in the U.S., only about 1,000 participate in the Leapfrog Group's voluntary reporting program, which collects information on whether the hospital uses a computerized provider order entry system (CPOE) and whether it has dedicated intensivists in the intensive care unit.
For hospitals that don't voluntarily participate in the Leapfrog survey, a secondary source was used, such as the American Hospital Association's own Health Information Technology annual survey.
Leapfrog participant hospitals can earn 100 points toward a high letter grade in each of those categories, but non-participating hospitals judged on the AHA's HIT survey, can get only 15 points.
"Therefore it is unclear why The Leapfrog Group would give so little weight to an answer to a similar question in the AHA survey," he wrote.
"By assigning vastly different point scores to similar information derived from reliable secondary sources, we are concerned that the scorecard can lead patients to inappropriate conclusions," he wrote.
Yale-New Haven Hospital is a good example. While it has a fully-functional electronic health record and 137 full-time equivalent intensivists on staff, it received only 30 points instead of 200 because it doesn't participate in the Leapfrog survey. That meant Yale-New Haven got a C "when we believe it rightfully should have received an 'A.' " Umbdenstock wrote.
Binder replied that with respect to the AHA's survey regarding the CPOE and intensivist measures, "is far less information than what is required of hospitals that report to the Leapfrog Hospital Survey." In order to earn full-credit, "hospitals not only have to demonstrate a high level of adoption, but also take a six-hour simulation test to prove their system works safely."
She said Umbdenstock's criticism "is a misreading" of the AHA survey's scoring algorithm.
With respect to a hospital getting a C that should have gotten an A, Binder replied that hospital's score "was hurt by its lower than average adherence to surgical care guidelines, rates of hospital-acquired conditions, and lower than average performance on other measures included in the Hospital Survey Score."
2. Use of unreliable measures
The AHA says that eight of the 26 measures used in the Leapfrog score have not been endorsed by the National Quality Forum. "To the best of our knowledge, it (Leapfrog's survey) has not been assessed for its reliability and validity by any independent organization."
Of the scoring methodology's 11 outcome measures, "nine are proposed for retirement from the Hospital Compare website in 2014 because studies have shown them to be unreliable." Regarding those that are hospital-acquired conditions (HACs), the Measure Applications Partnership or MAP has recommended they be removed because the lack of risk adjustment makes them inappropriate to compare hospitals.
"Despite this, The Leapfrog Group's methodology gives these measures higher weights than the demonstrably more reliable measure assessing prevention of surgical complications," Umbdenstock wrote.
The AHA contends that if some hospitals didn't supply data or answer all the questions, weights were shifted to the remaining questions unequally. "We believe this means hospitals’ grades are being calculated based on disparate weighting systems, and the results are not comparable," Umbdenstock wrote.
Binder replied that nine unbiased experts in hospital safety included measures based on "strength of evidence, opportunity for improvement, and impact to the patient," adding that the Leapfrog Survey which collects data "is as reliable and documented a data collection and public reporting tool as any health data collection effort in the United States, short of a full-scale accreditation process.
As for the criticism about the inclusion of HAC measures, Binder said that if the measures are retired, "they will be removed," but for now the safety score panelists believe they are useful.
With respect to unequal weight shifting from questions hospitals didn't answer, Binder replied, "Your analysis is incorrect," and impossible. She asked Umbdenstock to look again at the scoring methodology.
3. Significant errors in the Leapfrog's data
The AHA says in its letter that Leapfrog has made significant mistakes that have resulted in "misleading information being publicly displayed." For example, at least one specialty hospital was given a score when the survey's stated methodology excluded specialty hospitals. Furthermore, the AHA says it found several hospitals that should have qualified for a score but weren't graded.
Binder replied that all 2,600+ hospitals that were scored received a copy of the source data used in calculating their scores as well as the methodology five weeks before the score became public. "We have corrected the handful of errors reported to us, but the vast majority of what we have reported has been substantiated and stands as originally reported."
For hospitals that weren't scored, and which Umbdenstock maintains should have been, "this is typically because they did not provide enough publicly available data to meet our criteria," Binder explained.
Of Umbdenstock's charge that Leapfrog manipulated the data, Binder said it "suggests you are expressing something beyond sincere concerns about the methodology. So to be clear on the issue of 'manipulation of data': disappointing though many of these grades may be to many of your member hospitals, Leapfrog...did not make these scores up. The Hospital Safety Score grades reflect real problems threatening the lives of people who depend on America's hospitals. We urge you to address those problems quickly."