Measuring Value in Clinical AI Means Going Beyond the Bottom Line

Jon Handler and Roopa Foulger of OSF HealthCare, participants in the HealthLeaders Mastermind program on Ai in clinical care, say the healthcare industry still has a lot to learn about ROI.

The trick to embracing Ai for clinical care is managing expectations. That includes understanding what ROI really means with this technology.

“The ROI piece is always interesting,” says Jon Handler, Senior Fellow for Innovation with OSF HealthCare. “There’s this concept of hard costs and hard ROI and soft ROI. … At the end of the day, the real-world impacts on the bottom line are the same regardless of how hard or easy it is to measure it.”

Jon Handler, Senior Fellow for Innovation, OSF HealthCare. Photo courtesy OSF HealthCare.

Handler and Roopa Foulger, Vice President of Digital and Innovation Development for OSF HealthCare, are taking part in the HealthLeaders Mastermind program on the use of AI in clinical operations. They say the Illinois-based health system is looking to be reasonable in finding the value of new tools and programs, with an eye not only on the bottom line but also long-term clinical value.

“How do we measure it?” Handler asks. “How do we assess it? How do we validate it? And I think that gets harder, not easier, with some of the new large language models and the generative AI that’s out there. Because now, instead of algorithms built on a use case by use case basis, you’ve got this general purpose model – how do you evaluate all the things that it can do?”

“There are so many other ways to measure the value that is created,” he concludes. “Determining the right things to measure, which may not always be the easiest things to measure, is critical.”

Foulger says the health system has been using AI in several areas, including some clinical programs around mortality and risk prediction and imaging reviews. Through OSF Innovation, they’re looking at small startups with unique ideas, in addition to implementing AI tools provided by their EHR vendor.

“We’re encouraging what might be different that we should keep an eye on,” she says. “At the same time we’re asking, ‘Why try to build something already available?’”

Both Handler and Foulger say they’ve been surprised at how fast AI has worked its way into healthcare, even as the industry has been using automation and predictive algorithms for more than a decade. But while they’re seeing adoption in several departments and showing success in improving efficiency and reducing administrative stress, they’re also seeing a lot of strong use cases fail to make an impact.

Roopa Foulger, Vice President of Digital and Innovation Development, OSF HealthCare. Photo courtesy OSF HealthCare.

“I’m surprised at what is working and what is not working,” says Foulger, who notes that AI tools have shown value in revenue cycle and finance by handling complex processes that take a lot of time and effort. She wonders if healthcare organizations are embracing new ideas too quickly, and not giving these tools time to prove their efficacy.

Handler says he’s surprised that some promising projects, like using AI to transcribe the doctor-patient encounter or generate draft replies to inbox messages, have seen mixed results in published literature. There may be a disconnect between the outcomes some expect from these new tools and the benefits they might more consistently provide, like reduced stress and burnout.

It may also be, he says, a good indication that healthcare still has a lot to learn about AI.

“It’s hard to know when you’re dealing with something that’s overhyped or not,” Handler says, noting the internet was once a shiny new tool that received mixed predictions of its impact before becoming universal. “So any prediction about the future [of AI] … is treading on dangerous territory because people who make predictions are very often wrong.”

“In addition to unexpected upsides, there may also be downsides that that we haven't anticipated or been able to manage because of the speed with which these things are happening,” he adds. “These are really, really important questions to wrestle with.”

Foulger sees a future where AI is part of smarter healthcare ecosystem, giving patients and providers instant access to decision support, best practices and health and wellness tips. She notes that the industry has access to vast amounts of data, but until now it hasn’t had the tools to make use of that information.

The key, Handler adds, is to find the right way to use those tools.

“My biggest hope is that we capitalize on it as effectively as possible to help improve the service we can provide to our fellow human beings.”

The HealthLeaders Mastermind program is an exclusive series of calls and events with healthcare executives. This Mastermind series features ideas, solutions, and insights on excelling in your AI programs.

To inquire about participating in an upcoming Mastermind series or attending a HealthLeaders Exchange event, email us at exchange@healthleadersmedia.com.