When it comes to revenue cycle management, AI must undergo more "evaluation and refinement" before it can be used for coding.
Artificial intelligence has become a popular solution in revenue cycle operations, but some tasks are best completed with a human touch.
A study from the Icahn School of Medicine at Mount Sinai has found that large language models, state of the art artificial intelligence systems, have limited accuracy when it comes to medical coding.
“Previous studies indicate that newer large language models struggle with numerical tasks,” Eyal Klang, MD, director of the D3M’s Generative AI Research Program, said in a statement. “However, the extent of their accuracy in assigning medical codes from clinical text had not been thoroughly investigated across different models.”
Researchers used over 27,000 unique diagnosis and procedure codes, excluding identifiable patient information, and asked LLMs from OpenAI, Google, and Meta to produce the most accurate medical codes. All three models showed limited accuracy in reproducing the initial medical codes.
“Our findings underscore the critical need for rigorous evaluation and refinement before deploying AI technologies in sensitive operational areas like medical coding,” Ali Soroush, MD, MS, assistant professor of data-driven and digital medicine at Icahn Mount Sinai, said in a statement.
The study’s findings will come as a disappointment to health systems struggling to hire medical coders and considering digital expansion to assist them.
After integrating AI into its bedside procedures in 2023 to assist with medical coding, Henry Ford Health was able to utilize staff in other areas that needed them.
“Regarding the big picture on the people side of Henry Ford Health, it reduces the daily workloads on physicians, medical coders, and billing administrators,” Joann Ferguson, vice president of revenue cycle at Henry Ford Health, previously told HealthLeaders. “Driving better financial and operational performance while improving our coders’ job satisfaction.”
Authors of the Icahn Mount Sinai study maintain that AI has potential but warn that there must be continuous development in order for it to be a reliable and efficient solution for the healthcare sector.
Jasmyne Ray is the revenue cycle editor at HealthLeaders.
KEY TAKEAWAYS
Testing three LLMs, researchers found that all three showed limited accuracy in reproducing the initial medical codes.
While they noted the potential of AI, researchers believe that it needs to be evaluated and refined further.