Many healthcare organizations fired back with umbrage Wednesday at the Leapfrog Group's courageous hospital safety report cards, charging that assigning each hospital a "Safety Score" letter grade, much like county inspectors score restaurants, is an unfair way to gauge whether any hospital is more likely to harm patients than another.
And they said they were shocked that so many of the 2,652 hospitals—including winners of prestigious awards who make national "best hospital" lists would flunk, (132) or barely get a passing grade (1,111).
On Leapfrog's website, each facility now appears next to a big bold capital A, B, or C in a square box, or "Score Pending," which means a D or an F.
I heard an earful of complaints.
Leah Binder, Leapfrog's CEO, countered every point. My take is that while this process may be off to a bumpy start, it will definitely stimulate the conversation about the absolute necessity to improve patient safety in our nation's hospitals.
But first I'll let the hospitals have their say about what's wrong with Leapfrog's effort.
Hospital officials blasted the measures Leapfrog picked, the weights they were assigned, and the methods the non-profit group used to announce the scoring system to the public. Several I spoke with discredited Leapfrog's entire operation, saying many hospitals no longer participate in its voluntary data collection because of concerns that the measures Leapfrog collects are prone to the reporting hospital's interpretation.
Asked if the Leapfrog Group methodology is fair, Martin Ciccocioppo, vice president of the Hospital and Healthsystem Association of Pennsylvania, answers bluntly, "No. This is a way of looking at a plethora of data and trying to dumb it down to a few data points. I don't want to seem pejorative, but this is not the way to drive a decision at which facility one should seek care."
"They haven't done very damn well at all with it, to be honest," says David Perrott, MD, senior vice president and chief medical officer for the California Hospital Association.
Not only is the scorecard flawed in major ways, he says, "They promised they were going to contact CEOS and the CEOs weren't contacted. They promised the hospitals were going to be able to reply, but the hospitals weren't given the information. So let me tell you, we didn't receive a lot of this information until the last 24 hours or so...That's the frustration we're having."
California has 260 hospitals eligible for inclusion in the Leapfrog Group report card, more than any other state, and saw 10% receive a grade of D or F, also more hospitals than in any other state.
Perrott, a surgeon from Salinas, explains that among the 26 safety scores in the algorithm are at least six that the federal government plans to eliminate from Medicare's Hospital Compare database by FY 2015, so hospitals will no longer be reporting them starting in October, 2014. I was unable to confirm that Wednesday.
Also, Leapfrog is unfairly including measures such as accidental puncture or laceration during surgery, he says. That's an invalid measure because how hospitals report it depends on the phrases surgeons use when they dictate it into the medical record. If the surgeon mentions the laceration occurred in a patient who already had a lot of scarring from prior surgeries, that would not count. But not all surgeons do it that way, he says.
The safety score also has embarrassed some of the leading healthcare institutions in the country. For example, Henry Ford Hospital in Detroit and Cleveland Clinic will have to explain their "C" grades.
Henry Ford's communications office sent this response:
“Although we no longer participate in Leapfrog, we understand that the criteria evaluated differs from one agency to another. As a result, hospital systems often rank very differently in Leapfrog, HealthGrades, US News, Baldrige, and the like.
"As a health system, we look at each of our hospital's scores, and try to learn from the best practices in order to provide a consistently high quality experience with every episode of care. In the Leapfrog data, Henry Ford Macomb Hospital received an A, so we will work to learn and improve throughout our health system.”
One major concern from hospitals is how Leapfrog weighted the measures in the algorithm. The scorecard uses safety information from two sources. One is the Medicare database, which is collected from all hospitals except those in Maryland.
But the rest comes from data that about 1,000 hospitals that participate in Leapfrog's program voluntarily submit, such as the extent to hospital's doctors use computerized physician order entry systems or employ a full time intensivist in the intensive care unit.
Hospitals that don't volunteer that information to Leapfrog had their scores based solely on Medicare data, and some saw that as pivotal.
Shannon Phillips, MD, patient safety officer for the Cleveland Clinic, says the scoring system "feels uncomfortable" because her healthcare system dropped out of Leapfrog some years ago.
"We found that because it's a self-assessment, and you take that seriously, you go out pulling cases and doing a lot of work, which is [a] tremendous amount. We found it wasn't fundamentally adjusting the priorities we set for ourselves. We decided that the resources spent to do a good job accurately completing that survey were better served providing quality care."
Jim Lott, vice president of the Hospital Association of Southern California, which includes 180 acute care facilities in six counties, said the Leapfrog scorecard is also flawed because it "grades hospitals on a curve, and that's just wrong. It should be that the hospital either meets the standard or it doesn't.
"Imagine what it would be like if you graded restaurants on a curve? No you don't." If they don't meet the standards, they are shut down.
In an e-mail yesterday, Binder countered each point. Among her responses were these:
First, she says, "We made many attempts to contact hospitals well in advance as a courtesy and to request their feedback" and sent a letter to each CEO and the AHA on May 4 for help reaching each hospital. "Almost 1,000 hospitals contacted us in May in response to our letter."
Second, Binder underscores that the measurement strategy was determined by nine of the nation's top safety experts, including Lucian Leape, MD, of Harvard; John Birkmeyer, MD, of the University of Michigan; Peter Pronovost, MD, of Johns Hopkins University; Robert Wachter, MD, of the University of California San Francisco and Ashish Jha, MD, of Harvard.
"In the long run, it is likely we will set fixed safety standards and grade hospitals against those standards, but first we need better information about what standards are reasonable and feasible. We can only know that through comparative analysis right now," she says.
The fact of the matter, she says, is that "there are significant differences among hospitals in their level of safety, and that is what is reflected in the score."
Binder says that if and when Medicare stops reporting on a measure, or the expert panel concludes it's no longer eligible for the Hospital Safety Score, "we will remove it."
As she told me several weeks ago when she described the purpose of this new scorecard: "People deserve to know that some hospitals are safer than others."