"It's a tough question," Stewart concedes. "I think about ROI in two ways: qualitative and quantitative. Most people want to focus on quantitative ROI."
As noted earlier, Mercy uses machine learning algorithms as part of a systemwide initiative to identify the total value of standardizing the care process, but AI was only one component of a large project with many moving parts. "We set a goal for three years to save $50 million. Fiscal '18 will be the third year of the project, and we are on track to hit that goal," Stewart says. "Now, out of that $50 million in savings, what proportion of that comes from our machine learning work versus what comes from standardizing the processes and operational things? That is the extreme difficulty in knowing how much the machine learning contributed to that overall savings. That's where it gets almost impossible to really know. Honestly, we just don't really attempt to do that."
Stewart says Mercy does not look at machine learning in isolation. "It's a tool. It is qualitative. You look at the process, you talk to the people who are using it, it definitely has a value," he says. "We aren't going to try to go down to the penny or dollar for what proportion of that was from machine learning. When they reduced the level of ‘not seens' in the ED by X percent, how many dollars did that translate to relative to the spend on the machine learning side? I don't know.
"If you add up all the spend on the machine learning side for that one use case it may be a negative in terms of the return," he says. "But knowing that we can apply that same protocol going forward to many other use cases that are highly beneficial, it's more of qualitative. We know there is value there, and this is stuff we have to commit to organizationally if we're going to have that ability."
As for metrics, Stewart says he believes that the overall success of initiatives in which AI played a role—such as standardizing care regimens systemwide—can provide a good sense that the new technology is on the right track.
"It's a lot like the concept of what is the ROI for an EHR. It's difficult to measure. You can measure dollars spent, but it is much harder to quantitate dollars on the back side, where it puts you in a competitive position; we have to have that EHR data," he says. "More or less we are viewing machine learning as similar."
John Commins is a content specialist and online news editor for HealthLeaders, a Simplify Compliance brand.