Trial to Test Risk-Adjusting for Social Factors Called Inconclusive

The National Quality Forum sends risk-adjustment model designers back to the drawing board after their largely unsuccessful attempts to link social risk factors such as ethnicity with medical-treatment outcomes.

A two-year trial designed to examine the impact of risk-adjusting medical-treatment performance measures for social risk factors has raised far more questions than it has answered.

On July 18, the National Quality Forum (NQF) released a report on the trial, which began in April 2015. The trial is most noteworthy for what it did not prove and more research is necessary, the report says.

"One of the most striking findings of the trial was that measures with a conceptual basis for adjustment generally did not demonstrate an empirical relationship of the social risk factors to the outcome measured," authors of the report wrote.

The impact of social risk factors such as income and education level on medical-treatment outcomes at hospitals and other care settings has been debated hotly for several years.

Supporters of risk-adjusting medical-service payments for social factors contend that patients with social risks are more challenging to treat, placing an unrecognized financial burden on safety-net hospitals and other caregivers in disadvantaged communities

Opponents of risk-adjustment for social factors argue that setting the bar low for medical services in disadvantaged communities could institutionalize healthcare disparities across the country.

"The frequent use of NQF-endorsed measures for payment purposes underscores the importance of ensuring accurate comparisons of providers so that rewards or penalties are fairly assessed and based on true differences in performance," Shantanu Agrawal, MD, president and CEO of NQF, says in a prepared statement accompanying the release of last week's report, "Evaluation of the NQF Trial Period for Risk Adjustment for Social Risk Factors."

On Monday, the NQF Board of Directors released a statement announcing a new three-year study that will be designed to draw deeper conclusions about risk-adjusting performance models for social risk factors.

"We support the launch of a new, three-year initiative to build on NQF's leadership in this area and the significant knowledge gained through its recently completed trial. NQF will work with its stakeholders to secure the necessary financial support for this critical initiative," wrote Board Chairman Bruce Siegel, MD, MPH, and Board Vice Chairman Jim Chase, MHA.

"The new initiative will focus on unanswered questions about social risk adjustment of measures, including determining what data are needed and how to access them. NQF will continue to consider adjustment for social risk factors as part of its measure endorsement criteria. This vital work reflects NQF's commitment to effectively deploying measurement to decrease health disparities."

During the two-year trial, 303 performance measures were submitted to NQF for review, with about one-third of the measures categorized as "outcome or intermediate outcome" metrics. Out of those outcome measures, 93 utilized risk adjustment in some manner.

Risk-adjustment models during the trial featured more than a dozen social risk factors, including race, distance from clinics, marital status, language, and country of origin. "Race, ethnicity, and payer (including Medicaid status) were the most commonly examined variables," the report's authors wrote.

In addition to not finding expected correlations between risk-adjusting for social factors and healthcare-provider performance, last week's report says the two-year trial encountered several roadblocks:

• Paucity of data: Researchers encountered challenges finding comparable data sets of social risk factors at the individual level, and they struggled to include community-based data into risk-adjustment models. "NQF recognizes the current limitations of data availability but encourages developers to continue efforts to explore alternative data sources," the report's authors wrote.
• Conceptual basis of research: The approach to identifying potentially impactful social risk factors during the trial lacked consistency, with 65 risk-adjustment models picking social factors for inclusion based on literature reviews that were largely speculative on the impact of social risk factors, and 19 models selecting social risk factors for inclusion based on data from prior research. "The conceptual model has been identified as a potential area for greater specificity," they wrote.
• Risk-model remodeling: Accounting for social risk in medical-treatment performance measures remains an imprecise science and risk-adjustment for social factors must be optimized before conclusions are drawn on their impact.