Skip to main content

Hospital Groups Strike Back at Hospital Rating Systems

 |  By cclark@healthleadersmedia.com  
   March 10, 2014

The Association of American Medical Colleges has issued a list of 25 guidelines to help hospital officials evaluate rating systems that rank hospitals. Hospitals are urged to consider the validity of measures used, and transparency about methodology.

As hospitals grow increasingly frustrated with contradictory commercial rating systems that grade, score, and rank their quality of care, the Association of American Medical Colleges and other hospital groups are striking back.

They've issued a list of 25 Guiding Principles to help hospital officials judge the scorecards themselves, so they can explain results to their patients, payers, philanthropists, and board members, and if there are fees associated, so they can determine whether a survey's marketing value is worth the asking price.

When hospitals are rated, "[they] want to know, 'How do we talk with our boards, why we should put a lot of emphasis on this rating system or why we shouldn't, and where is it important?'" explains Joanne Conroy, MD, chief healthcare officer for the AAMC, which spearheaded the project last year. "They need a clear set of principles that's not their opinion to justify why they should emphasize one [rating] system and not another."

In fact, none the rating systems published to date, including those from The Joint Commission, the Leapfrog Group, Consumer Reports, Hospital Compare, the Commonwealth Fund, Truven, Healthgrades, and U.S. News and World Report, meet all of the AAMC's guiding principles. "Nobody meets them all," Conroy says.

"Some of the [scorecards] are really great at identifying their purpose, and some are really transparent about their metrics. And others are totally not," she says.

"Some of these rating organizations use proxies, because that's all they have. They don't measure what they purport to measure."

"Differences in the measures, data sources, and scoring methodologies produce contradictory results that lead to confusion for the public, providers, and governing boards, and impair the public's ability to make well-informed choices about healthcare providers," the AAMC document says.

"For data to be understood and for results to be comparable, publicly reported data should adhere to a set of guiding principles."

The guidelines suggest that hospitals look at each scorecard's merits on the validity of measures used, transparency about methodology, and purpose. For example does the scorecard specify its intended audience? Are the measures used endorsed by the National Quality Forum? Are the outcome results risk-adjusted?

'A Lot of Commercial Interest'
Does the hospital have a business relationship with the ranking company?

"Quite a few of these are what I call commercial scorecards that offer services to help improve you, and to me that's a conflict," says Michael Henderson, MD, chief quality officer for the Cleveland Clinic and a member of the committee that compiled Guiding Principles. "Because guess what? Those who take their services go up to the top of their rankings pretty quickly."

It's confusing for everybody, Henderson says, because it seems that all hospitals are number one on someone's "best" or "top" list.

Henderson says that hospitals just have to get used to the noise. "I think it's going to be exceedingly difficult to get a single agreed-upon way of doing this, because there's a lot of commercial interest in these scorecards out there, and to break that will be very hard to do."

Many of the ranking systems require hospitals to pay to advertise an award, or to participate in a reporting program, which Conroy says has made hospitals feel "like they're being 'held up' a little bit, because a lot of them charge you to use their ranking publicly. And CEOs have to make hard decisions about that because they have limited resources, which should be directed toward patient care."

A Different Version for Hospitals
The three-page AAMC document is backed by America's Essential Hospitals, the American Hospital Association, the Catholic Health Association of the United States and the Federation of American Hospitals.  But the publicly distributed version stops short of actually saying which rating systems comply with which of the 25 aspects a good rating system should have.

Hospitals have access to a longer version of the document, which evaluates two Leapfrog scoring systems, Hospital Compare, Healthgrades, and U.S. News and World Report. But that version is not available to the public, Henderson says.

Why not publicly distribute who meets the new criteria?

"We didn't want to be an evaluator," Conroy says, "because there are other organizations that do that," although she says the AAMC group may do that in the future.

She pointed to the Healthcare Association of New York State's "Hanys' Report Card on Report Cards," issued last October, and the Informed Patient Institute, each of which evaluates these scorecards.

HANYS gave The Joint Commission and the Centers for Medicare & Medicaid Services' Hospital Compare three stars for meeting most of HANYS criteria.

But Leapfrog's Hospital Safety Score, Truven Health Analytics 100 Top Hospitals, Healthgrades, Consumer Reports Hospital Safety Rankings and U.S. News and World Report, got only one star.

How Should Hospitals Respond?
HANYS spokeswoman Melissa Mansfield explained in an e-mail:

"Some of the report card publishers do have an agenda, as they profit from the licensing fees, magazine subscriptions, or consulting services through for-profit corporations. Others use their report card results to negotiate contracts with health care providers. Individual hospitals must make their own decisions about whether and how to respond to report card organizations."

U.S. News & World Report editor Ben Harder, also responding by email, said:

"One easily overlooked source of differences among different public reporting services is that the services differ in their objectives—that is, each organization seeks to measure a different dimension of healthcare quality. U.S. News aims to evaluate hospitals on whether they excel in treating the most medically challenging patients, the kind for whom hospital choice holds the highest possible stakes. The best hospitals on that dimension may or may not stand out from the pack on other dimensions of quality, such as value-based purchasing, adherence to safe practices, or routine care."

Harder added,

"If the new principles were followed to their letter, public reporting services would tend to identify only extreme positive or extreme negative outliers, masking clinically important variability among the vast majority of hospitals."

'It's a Black Box'
Leah Binder, President and CEO of the Leapfrog Group, says that while the AAMC's principles "adhere to Leapfrog's standards for our own public reporting," geared for both patients and providers, presented with 100% transparency including the data and methodology, and use validated measures, the effort seems to deny free speech.

"We wish AAMC would focus their enormous expertise and leadership on expanding transparency, instead of this quixotic attempt to curtail it. In America, not even an organization with the impressive credentials of AAMC gets to decide which speech is acceptable."

Conroy says the hospital groups may try to go further in future documents, and publicly name organizations that have met most or all of the principles. But this time around, "we didn't want people to think that we were creating principles that were inherently biased."

She gave as one example, Healthgrades, which she says "has a proprietary algorithm, and that means there's no way for me to understand why I didn't score well, or did score well. It's a black box. They don't specify their data sources."

Hospitals, as well as the physicians who refer patients to them, should be prepared for more rankings going forward, especially with public quality measures becoming increasingly numerous, specific and used to structure payment, Conroy says. "There are at least 14 out there now; this is like a new enterprise, and they are certainly going to proliferate."

Conroy says that as hospitals digest these Guiding Principles, there will "be some pushback on these rating systems to improve their processes and be more transparent. And I think we'll all get better."

Tagged Under:


Get the latest on healthcare leadership in your inbox.