Peer Review of Surgeons' Skills Carries 'Threatening Undertones'
HLM: I want to come back to the idea that in the view of these participating surgeons, this was their best work they submitted, right?
JB: The advantage of having surgeons select videos of their best work makes this even more reproducible. If you didn't control for case difficulty, you'd introduce bias and measurement noise.
HLM: Were you surprised that fellowship completion or teaching hospital practice weren't linked to better outcomes?
JB: No. Keep in mind that we were focusing on surgeons who had been practicing for a mean of a decade. They had done thousands of bariatric operations.
HLM: How can you protect against bias among the peer reviewers? Perhaps some might favor a technique or sequence similar to their own.
JB. We found very little evidence of that. I worried we'd have hard graders and easy graders that would contaminate findings. We re-rated all these videos with a second expert panel, with no attachment to the Michigan collaborative, and they were slightly harsher. And all the best surgeons in the original rating were still the best, and the worst surgeons still the worst.
And we had the best and worst surgeons submit a second video, just to make sure there wasn't something weird about what they submitted on the first pass. And again, when we re-rated those, the best and worst got put in exactly the same place.
And when I showed these videos to non-surgeons, including lay people, with no labels or prompts, every single lay person said, "Oh my God, I definitely would want the first guy," and they were exactly right, of course. Everything wasn't quite as good.
Cheryl Clark is senior quality editor and California correspondent for HealthLeaders Media. She is a member of the Association of Health Care Journalists.
- Interventional Radiology No Longer a Sub-Specialty
- NFP Hospitals' Revenue Growth at 'All-Time Low'
- Acute Kidney Injury Gets New Focus
- Transforming Cancer Care
- Half of All Primary Care, Internal Medicine Jobs Unfilled in 2013
- Evidence-Based Practice and Nursing Research: Avoiding Confusion
- mHealth Tackles Readmissions
- CNO Leads $1M Charge for New Scrubs, Uniforms
- Sharp HealthCare Leaves Pioneer ACO Program
- Proton Beam Therapy Poised for Growth in US