While the federal government's star ratings may be a reliable measure of patient experience, that's about all they can be relied upon to provide, says a provider of healthcare quality and outcomes data.
Ever since CMS debuted its hospital quality star rating system at the end of July , studies highlighting the system's shortcomings have proliferated.
Criticism from healthcare executives, who along with members of Congress called for delay of the release, has continued.
The clamor is well-founded. Hospitals contend that the ratings unfairly disparage some hospitals by boiling down multifaceted, complex organizations to a single grade.
CMS Releases 5-Star Hospital Quality Rankings
While the star ratings may be a reliable measure of patient experience, that's about all they can be relied upon to provide, says Frank Mazza, MD, chief medical officer of Quantros, a provider of quality and outcomes data to employers choosing healthcare options for their employees.
When the CMS ratings came out, Mazza, an intensive care specialist by training, noticed the lack of five-star rated organizations that offer complex services such as intensive care.
"We, like everyone else, thought this didn't really sound right because we had a lot of skepticism that [HCAHPS] surveys, [which] basically measured whether docs were nice to you, could somehow translate to better clinical outcomes."
Fewer CMS Stars for Hospitals in 'Stressed' Cities
Howard Kern, the president and CEO of Norfolk, Virginia-based Sentara Healthcare, which is not a Quantros customer, largely agrees.
Though he doesn't believe many patients will base their choice of where to have an important surgery or other treatment on the star ratings, he says the system's shortcomings muddy the waters for patients who want to choose among their healthcare options based on data and analysis.
Following, according to Kern and others, are three reasons the system needs big improvements.
1. Ratings are Oversimplified
The biggest challenge to the data is that CMS is trying to roll up a diversity of data metrics and reference points into a single rating, says Kern. "That's extraordinarily hard to do."
Even Consumer Reports, the evidence-based testing and rating organization, has a better system, he says.
CR rates everything from washing machines to new cars by compiling a weighted overall rating that also shows critical subcategory ratings and allows the consumer to scan overall quality. This allows consumers to pinpoint the items that are important to them.
"A mother looking for a place to have her baby is looking for different things than someone who's having a hip replacement," Kern says.
"For most community hospitals and big academics, breaking out data from the overall score by service line or by category of clinical quality, safety, and customer experience, gives them a look on that level so they can weigh for themselves what's important to them."
Another factor: the data CMS relies upon for the ratings comes only from Medicare patients.
"There are not too many mothers having babies through Medicare," Kern quips.
"CMS may not have the ability to drag data from other payers—they may need legislative action for that—but other states have done it, most notably in Pennsylvania with heart surgeries."
2. Size Shouldn't Matter
Some 87.5% of hospitals that earned five stars under CMS's ratings scheme have fewer than 100 beds, says Quantros' Mazza. Some are critical access hospitals, which by definition have less than 25 beds, and 68% of five-star hospitals don't have an ICU, he says.
Rural Hospital Tracks Shift to Value in CMS Star Ratings
"Anyone who has clinical knowledge knows that when people are very sick with pneumonia, for example, they wind up in the ICU with special medicines," Mazza says.
Heart attack patients will be in the ICU with lines and drips, while congestive heart failure patients may be on artificial hearts.
"These are not things that are offered in very small critical access hospitals—they're offered in big medical centers with sophisticated capabilities," he says.
"We have a lot of critical access hospitals here in Texas. Many are wonderful places with wonderful people who do the best they can, but it is hard to believe they offer the highest quality of care with these three conditions."
For his part, Kern says the methodology will have to be greatly improved for the CMS system to be considered a viable measure of quality and outcomes, upon which patients and doctors should make treatment choices.
"I don't pretend to be an expert on the methodology, but the truth is that it's an issue," he says.
Under the current CMS methodology, rural hospitals, and for that matter, specialty hospitals, can benefit in that they don't have the volume to generate the results that would drive them statistically off the five-star list.
They also benefit by being able to be highly selective of the patients they do treat, he says.
"They can be pretty selective and for high-risk patients, they refer elsewhere," Kern says. "Larger academic medical centers and general community hospitals take all comers and manage as best they can."
3. Patients' Perceptions Aren't a Proxy for Quality
In the Quantros study, CMS one-star hospitals made up a large percentage of the top 10%.
More telling, Mazza says, the Quantros study found no correlation at all between CMS star ratings and clinical outcomes, even when it broke down the data to look separately at mortality and complication rates.
And there was no correlation with an organization's CMS star rating except in patient safety.
"CMS is essentially saying you can use patient experience scores as a proxy for outcomes and clinical quality, but you really can't," says Mazza.
"If you were to use the star rating system as a proxy for this, not only might you not wind up at a high-quality institution, you might end up at substandard institution. Experience is in some respects a measure of quality but not a measure of health outcomes."
Ratings System Should be Refined
Kern is hopeful that CMS will, over time, broaden the data that generate the star ratings and improve upon it. Even in its current format, he says, Sentara pays attention to it and is working toward improving its hospitals that scored relatively poorly on patient experience.
"We give pats on the back to high performers and try to improve the ones that could do better. At the same time, it's not just about getting a score."
Philip Betbeze is the senior leadership editor at HealthLeaders.