Skip to main content

UPMC Helps Fund AI-Powered Patient Tool to Record, Transcribe Medical Visits and Drive Patient Engagement

Analysis  |  By Mandy Roth  
   July 14, 2020

Technology that was inspired by patients' need to recall information exchanged during clinical visits is now being used during telehealth consults to enhance the patient experience and drive future clinician efficiencies.

When patients leave an encounter with a physician, can they recall details of the discussion? One health system is ensuring that they do with a tool that offers a recording and transcript of key segments of the visit.

UPMC has taken AI-enabled technology embedded in an app from Abridge one step further and, with patients' consent, some physicians at the Pittsburgh-based health system are using the same technology during phone consults to deliver a verbal and written record of the medical portions of the interaction.

As healthcare delivery transitions toward more virtual exchanges, this technology can help improve recall, communication, and the patient experience, says Shiv Rao, MD, a UPMC cardiologist. Rao serves as the CEO and is one of the co-founders of Abridge, in which UPMC Enterprises is a minority investor.  

The app was launched last year and is free for patients to download. Its original intent was for use by consumers during face-to-face clinical encounters. The pandemic created an opportunity to use the technology in a new way that bypasses the app; it can now be activated when a physician calls a patient through a toll-free number. In addition to offering this service—presently at no charge—to physicians, the company is interested in exploring partnerships with health systems beyond UPMC.

Currently, the technology works only with telephone calls. While there are plans to expand its capabilities to encompass video visits, patients who have downloaded the app on a tablet or smartphone can record the audio component of the conversation.

Developed as a Tool to Empower Patients

"Abridge was founded, first and foremost, to empower patients to stay on top of their health, but also to keep them, their families, and their clinicians all on the same page," says Rao. "It's really about improving the doctor-patient communication by recording those conversations and then creating an interactive transcript that highlights the key medical points."

Aiding patient's recollection can help improve healthcare, says Rao. "There's research … that suggests that sometimes people only remember 10% to 15% of what was said in the encounter, especially if they have a chronic disease."

Unlike traditional transcription, he says, the technology is trained to focus only on the medical portion of the encounter, and the transcript highlights key words. Rao says that the transcript functions like a "bookmark," and when the user touches the screen, the associated audio recording plays. "All those moments are pulled out like bookmarks for the audio so that you can experience it again and relive those moments that you might've forgotten."

The tool also defines key medical terms and medications mentioned during the visit, and provides links to discount coupons for some prescriptions. "The transcript provides the opportunity, Rao says, "to layer on health literacy, advocacy, and connection—things that can potentially, over time, improve the experience for everyone involved."

The system employs machine learning. To build its capabilities, the developers started with recordings of about 76,000 fully consented conversations between physicians and patients, Rao explains. Experts then transcribed the dialogue to correctly capture medical jargon, and organized that information into specific topics. "These were takeaways or next steps related … to the patient's history of present illness," Rao says. "All of those annotations then are fed into a machine learning system that helps us train models to automate it."

The Pandemic Led to Expansion of Capabilities

When COVID-19 forced clinicians and patients to interact in new ways, the technology evolved. "In this new paradigm, it's so important to be relevant in the remote space," says Rao. "We quickly transitioned to building capabilities for phone visits, and we've built a proof of concept for video visits. The ambition is to connect the dots across all these types of encounters to deliver the same kind of value."

The pandemic also created a need for the system to learn new terminology. "We quickly circled with experts at the CDC and put together a list of symptoms, diagnostics, and therapeutics related to coronavirus." The team tried to anticipate and capture any topic related to COVID-19 that clinicians and patients might discuss. "In a matter of hours, we trained our system to recognize treatments like Remdesivir in a very robust way so we can really go deep and build a system that goes above and beyond what a lot of the off-the-shelf speech transcription models can do."

UPMC began using the system in late March, initially with cardiologists, then expanded to other specialists and primary care physicians, according to UPMC's Chief Medical Information Officer Robert Bart, MD. As of late May, there were up to a hundred calls daily initiated by physicians. No app is necessary when the system is used this way. The physician calls the number, adds the patient to the call, ensures the patient consents, and records the conversation. After the call, the physician and patient both receive links via a text message to the visit summary.

An Innovation Inspired by Patients

Rao was inspired to create the original app based on his own family's experience with a rare disease, as well as his professional experiences. Before developing the tool with his partners from Carnegie Mellon University, he recalls consulting with a patient who was about to start a chemotherapy treatment that might affect her heart muscle. The medical visit was the first one in a 10-year history of breast cancer that her husband had not been able to attend with her. She was "clearly uncomfortable," Rao remembers, so he asked her what her husband did during these encounters. 

"She told me, 'When you're really sick, you remember very little of what's said,' " Rao recounts. Her husband would sit in the corner, quietly taking notes. "That meant she could feel liberated to be more present, focus on what the clinician was saying, and potentially build a better relationship [with the doctor]. They could go home and unpack all the notes that he took and potentially layer some of their own research and education on top of those notes. Then she could take a summary of those notes to the next doctor and retell it, feeling like the main character, as opposed to someone looking in from the outside." Abridge replicates that process, Rao says, "so the patient is getting some version of what her husband did in the corner of the room."

What's Next: Expanding Value for Clinicians

Giving patients a more complete understanding of their health "empowers them to take a more active role in their care," according to a UPMC media release about the technology. Bart says it also provides benefits to the health system and providers, particularly as the technology further evolves.

"Health systems are always looking for tools that make a patient stickier to their health system," says Bart, "and [UPMC] is always looking for tools that improve the engagement between our [clinicians and their patients]. Abridge is actually one of the few technology solutions that exists that has opportunity to provide value for both the patient-consumer and for the clinician delivering care."

While the transcript can be attached to the electronic medical record (EMR) as an after-visit summary, UPMC is working with the company to explore ways to integrate the content directly into the clinical notes.

The parsed conversation could become a template for the clinical note, Bart says. If that occurs, "now you've improved the clinician's efficiency in documenting the care they delivered. You've also probably improved the accuracy of the documentation of the care. As we all know, our recall is not as good at five in the afternoon as it is for the patient we see at 8:00 a.m. On top of that, we also know that many of our clinicians don't do their notes temporally as they see the patient. They might jot down a few notes to themselves. If they've seen 20 patients in the course of the day, once all the patients go home, that's when they actually sit down and type out or dictate out those 20 notes."

Being able to bypass that process will be a "huge win for clinicians," Bart says.

"Ultimately, that's also a huge win for patients because they now have this parsed conversation as a permanent reference of the interaction, but also what's documented of the interaction," Bart says. "That person's medical record is much more reflective of the care that was delivered and the decisions that were made together about the care in that visit. That's the piece that we're actively working through and exploring right now."

“Health systems are always looking for tools that make a patient stickier to their health system.”

Mandy Roth is the innovations editor at HealthLeaders.


The patient app can be downloaded for free; physicians also can bypass the app and initiate usage through a toll-free number.

The pandemic spawned further development of the technology, which can now be used for some telehealth consults.

Get the latest on healthcare leadership in your inbox.