Surgeon Scorecard Needs Improvement

ProPublica's rating system for surgeons is under fire for using data from Medicare claims rather than quality-based, clinical outcomes. But despite those flaws, it's a step forward on the march to physician transparency.

I recently needed minor surgery on my arm, and I wanted a surgeon with some OR street cred. So I asked an MRI technician at the hospital where I would be having the surgery who she would recommend.

"I'm not supposed to tell you," she said. But she scribbled a name on a scrap of paper and handed it to me, as though it were a code to a nuclear detonator.

There's got to be a better way to vet surgeons. The not-for-profit public interest group ProPublica thinks it has one.

It's the Surgeon Scorecard, a searchable database of surgeons rated by mortality and complication rates. It's a tool for helping consumers select surgeons based on Medicare claims data associated with eight common elective surgeries.

But since its release last week, the scorecard has been kicking up a nuclear dust storm in the hallowed halls of many surgical departments and online.

Jeffrey B. Matthews, MD, professor of surgery and chair, department of surgery, at the University of Chicago, believes the scorecard's information is meaningless for both consumers and surgeons.

"It is a big stretch if you are using that scorecard's data to choose a surgeon," he says. A look at the measure of Chicago's surgeons who perform gallbladder surgery is extremely inaccurate based on the scorecard, he emphasized. "The confidence intervals are so wide that an apparently medium-risk surgeon might in fact be high-risk or low-risk, and then you are trying to compare that to, say an apparently low-risk surgeon with just-as-wide confidence levels."

Ashish Jha, MD, professor of International Health, director of the Harvard Global Health Institute in the Department of Health Policy and Management, advised ProPublica as it researched, collected, and analyzed the data. Other healthcare experts also guided them along the way.

He acknowledges that the scorecard has flaws, but says there is nothing better available and it's long past time for the healthcare system itself to develop a better tool for consumers.

Methodology
ProPublica used Medicare billing records on death and complication rates of surgeons who did knee replacements, hip replacements, three types of spinal fusions, gall bladder removals, prostate removals, and prostate resections.

The most often cited criticisms of the scorecard from surgeons I talked with is that Medicare data is based on administrative needs and delivered under the fee-for-service model, rather than on quality-based, clinical outcomes. The number of procedures per surgeon is also too small, they told me.

The data doesn't convey a good indication of a surgeon's ability, says John Birkmeyer, MD, executive vice president, Integrated Delivery System and chief academic officer, Dartmouth-Hitchcock Medical Center. "But I also believe that almost any data is better than no data at all."

"There are probably two or three major challenges with the scorecard's data," he says. "It is based on fee-for-service claims and on incredibly small numbers. Also, the types of procedures included account for only a minority of procedures a surgeon does."

Surgery is complicated and has many variables that determine the outcome, Matthews says. "It's very hard to boil surgeon quality down to a single number. It is especially difficult to do this from billing records."

"ProPublica tried to deal with risk adjustment for sicker patients; they should be given credit for that," Jha says. "They used the best methodology available. Is it perfect? No."

The rub is that the clinical data that's needed to achieve higher accuracy is only available from hospitals and healthcare systems. Despite calls for transparency, nobody in the healthcare system has shared it publicly in a useful way to consumers, says Jha.

Although he empathizes with surgeons' arguments against the scorecard, Jha says his response to them is, "'What are you doing about it? Do you have a better idea now?' Until I get a clear explanation, I'll defend the scorecard."

Toward Better Data Accuracy
The American College of Surgeons is collecting data and working toward more clinically relevant outcomes for analysis," says David Hoyt, MD, FACS, executive director of the American College of Surgeons. "But it takes time and effort to extract the data that is most useful."

"I personally think the professional societies have an obligation to generate standards of practice for their specialties and measurements for the standards, and then make some or all of the information publicly available, says Guy Clifton, MD, a former neurosurgeon and now CEO of Rockford, IL-based Actin Care Groups, which coordinates care for employees of self-insured employers.

Most surgeons I interviewed told me that despite its flaws, they believe the ProPublica scorecard is a good place to start when talking about the transparency of physician quality. "But it should be only one piece of information used when making the decision, says Charles Mick, MD, an orthopedic surgeon and former president of the North American Spine Society.