HANYS Grades the Hospital Graders

The Healthcare Association of New York State studies 10 of the most well-known report cards that attempt to quantify hospital performance—including those from the Leapfrog Group, Healthgrades, and U.S. News and World Report—and ranks them.

Healthcare likes data, so it's a safe bet it's going to love data about data. Stats lovers take notice: We now have another hospital report card. But instead of scoring hospital quality, this one scores the hospital report cards themselves.

For marketing departments that rely on those lists to state their case to the public, the results may be surprising.

The Healthcare Association of New York State (HANYS) studied 10 of the most well-known report cards that attempt to quantify hospital performance, including the Leapfrog Group's hospital report cards based on its hospital survey, America's Best Hospitals issued by Healthgrades, and U.S. News and World Report's Best Hospital Rankings.

The Report on Report Cards was released to the public on Monday; but HANYS membership got a peek at the nine-page document last week. "We hear from our members that all these report cards come out with the varied scores and methodology, and it's confusing," says HANYS spokeswoman Melissa Mansfield.

The annual report cards are meant, in part, to give consumers helpful information for making healthcare decisions. But these lists and grades are also an important component of hospital marketing strategy. For example, when Children's Hospital Los Angeles rebranded itself in 2011, the advertising campaign centered on the message that it was the best children's hospital in L.A.

DeaAnn Marshall, chief marketing and development office for CHLA, says she felt comfortable using that as a premise because of the grades the hospital received from various report cards.

"What we use in our marketing materials and our advertising is our U.S. News & World Report Honor Roll status," says Marshall. "We are [also] ranked as the fifth best children's hospital in the country by U.S. News and World Report. We are also a Magnet hospital, which is the highest designation for high-quality nursing care; we are also a Leapfrog hospital."

But, HANYS' report card awarded both Leapfrog's Hospital Safety Score, and U.S. News and World Report's Best Hospitals List only one star out of three. Only four report card-issuing organizations received HANYS' highest ranking of three stars:

• The Joint Commission's Quality Check
• DOH Hospital-Acquired Infection Report
• Center for Medicare & Medicaid Services' Hospital Compare
• DOH Hospital Profile Quality Selection

Mansfield says the report is meant to be a tool for hospital leaders to make sense out of the various report cards. It is not meant as endorsement of any one. From letters grades to check marks, the variation among the reports can be confounding, so HANYS developed nine criteria for grading the report cards:

• Transparent methodology
• Evidence-based measures
• Measure alignment
• Appropriate data source
• Current data
• Risk-adjusted data
• Data quality
• Consistent data
• Hospital preview

Leapfrog and U.S. News and World Report received only one star from HANYS because both organizations fully or partially met only a few of the criteria. The transparency of each organization's methodology was included as a criterion (one that Leapfrog passed, but U.S. News and World Report failed) because HANYS believes the results should be able to be replicated.

Most of the report cards were credited with using evidence-based measures to develop their individual rankings. HANYS's report, however, criticized the America's Best Hospitals Report, issued by Healthgrades, because it doesn't "provide scientific evidence" that shows the relationship between its 28 measures of complication and mortality and a hospital's quality. U.S. News and World Report's Best Hospitals List also received criticism for relying on "subjective perception".

The report cards or rankings that HANYS had the most faith in were from governmental or accrediting organizations. Both Hospital Compare, run by CMS, and the Joint Commission's Quality Check, were praised for meeting all or most of the nine measures HANYS used.

"The transparency and evidence-based measures were the strongest parts," says Mansfield, noting the report shows the organizations that ranked low on the used questionable data whether it was from administrative claims or data that wasn't validated.

Now, are these hospital lists meant to be compared side-by-side? Not really. But, the consumer doesn't know that. And if hospital rankings are being used in your organization as a selling point, you have to put yourself in the role of a patient or caregiver. When you do, it becomes clear that these lists are confusing.

HANYS graded 10 report cards, and marketers know there are many more local lists that doctors and hospitals also pay attention to. In Dallas, for example, D Magazine issues its own "Best Doctors" list, along with the best burgers, best cocktail, and so on. The marketing director of a specialty hospital in Big D has told me that it is imperative they make D Magazine's list every year because it's important to their patient clientele. The key phrase of that sentence is the latter part, "important to their patient."

Instead of looking at lists, it's time to start listening to patients.