A serious problem of 'surveillance bias' threatens the fairness of quality incentive payments and therefore, the integrity of public reporting about harm, say infection checklist pioneer Peter Pronovost, MD, and his Johns Hopkins research colleague Elliott R. Haut, MD.
In a commentary published Tuesday in the Journal of the American Medical Association, the researchers say that some hospitals may appear better or worse than their competitors because of faulty data that appears when hospitals look harder, and therefore find more evidence of complications in their patients.
"Performance measurement is essential for improving quality and reducing costs of medical care," they wrote. "However, most outcome measures in use do not sufficiently standardize surveillance for events and those at risk for events, likely introducing substantial measurement error."
They give the example of screening for deep vein thrombosis, a condition that is labeled as hospital-acquired when it occurs in orthopedic or other procedures. In those cases, Medicare won't pay hospitals for additional care that those avoidable outcomes require.
"Everyone wants to know, 'What is the best hospital?' "Where should I have my surgery?' Haut said in a news release. "People want to compare hospitals, but if the science can't keep up, maybe we're doing more harm than good when we report certain kinds of data. It raises a different question: Are the numbers being reported meaningful?"
Without standardizing the way hospitals look for these complications, data that points to better or worse quality care "may be worthless," he said.
DVT is a common, life-threatening complication that can occur with greater frequency in trauma patients, and some clinicians use duplex ultrasound to screen trauma patients even if they are asymptomatic.
"Other clinicians argue this approach is neither clinically necessary nor cost-effective and therefore do not routinely screen for DVT in trauma patients," they wrote.
"This clinical uncertainty leads to variability in the use of screening duplex ultrasound, creating variability in rates of DVT identified and reported – a typical example of surveillance bias."
That's "a clear incentive to avoid diagnosing this event," they wrote. This surveillance bias "may become extensive, leading to erroneous, undeserved payments for these biased measures of higher or lower quality."
Incentives to improve outcomes "may encourage clinicians to avoid appropriate diagnostic testing to minimize reported complications. Because performance measures do not specify surveillance, outcomes that are not sought ordinarily will not be detected."
The answer, the researchers suggest, is that when measures for reporting are set forth, they must be done based on principles from clinical research. "To do otherwise would be reckless and unjust."
And since surveillance is expensive, "key decisions involve who should pay these expenses and whether certain measurements are worth the financial investments."
They identified three steps to improve measurement.
1. Guidelines should specify which patients and events are at risk, the sensitivity and specificity of the screening tests, and the net risks and benefits associated with false positive results.
2. Costs and benefits of proposed measures should include rational prioritization of which measures to mandate. "Policymakers could require a formal post hoc review evaluating the risks, benefits and harms of the outcome measures after implementation."
3. Performance measures could link a process of care with adverse outcomes when defining incidence of preventable harm. "When standardized surveillance is too costly or risky, processes of care among those sustaining the outcome could be examined. For example, what percentage of patients who develop a DVT (outcome) received appropriate risk assessment and prophylaxis (process)?"
Pronovost this month was named director of the newly established Armstrong Institute for Patient Safety and Quality at Johns Hopkins. He was simultaneously named Johns Hopkins Medicine’s senior vice president for patient safety and quality.