How Multimodal LLM Assistants are Set to Revolutionize Patient Care: Insights from NY Hospital Exec
Multimodal large language models (LLMs) have the potential to transform patient care by creating virtual doctor’s assistants that can proactively track and diagnose medical conditions. This groundbreaking technology was discussed by Dr. Ashley Beecy, the medical director of AI operations at NewYork-Presbyterian (NYP), one of New York’s leading hospital systems, during VentureBeat’s AI Impact Tour event in New York.
Dr. Beecy explained that NYP is already experimenting with generative AI in various areas that provide value with minimal risk, such as summarizing conversations from patient visits. However, she envisions a future where hospitals can utilize this technology to build powerful and all-encompassing assistants that will revolutionize the way doctors practice medicine.
The use of multimodal LLM technology would allow doctors to gain insights from various sources, including imaging, echocardiograms, and electrocardiograms, which may reveal indicators of potential health issues before they occur. Dr. Beecy expressed her desire to have access to this information so she can act preemptively and provide better care for her patients.
While the technical capabilities for this technology are largely in place, there are challenges in adjusting internal workflows and processes to make it a reality. Dr. Beecy emphasized the need for change management and collaboration with national health organizations to bring about larger structural changes in healthcare systems. She outlined a step-by-step approach, starting with low-risk administrative tasks and gradually moving towards clinical diagnostics using generative AI.
Despite the risks associated with generative AI, such as potential mistakes and overreliance on technology, Dr. Beecy remains optimistic about its potential benefits. She believes that as long as doctors review the summaries and diagnoses generated by AI, the risk can be minimized. NYP is taking a measured approach to implementing this technology, ensuring that it aligns with the needs of both those who want to roll out generative AI tools and those who will use them.
Dr. Beecy also highlighted the democratizing nature of this technology, as it allows doctors and providers at NYP to have direct access to AI tools like ChatGPT. This empowerment of end-users has led to the identification of valuable use cases and increased engagement with the technology. However, questions of transparency and patient consent still need to be addressed, which may require input from sociologists and bioethicists.
Sarah Bird, global lead for responsible AI engineering for Microsoft, also discussed the potential for an all-encompassing doctor’s assistant. While Microsoft’s AI technology can provide the building blocks for such an assistant, there are challenges in ensuring accurate and reliable information in generative AI summaries. Microsoft is actively working on techniques to enhance the model’s understanding of medical information to improve the effectiveness of these summaries.
In conclusion, multimodal LLM assistants have the potential to revolutionize patient care by providing proactive and comprehensive support to doctors. While there are challenges and risks associated with implementing generative AI, the benefits and excitement around this technology are undeniable. With careful consideration and collaboration, hospitals like NYP can harness the power of AI to improve healthcare outcomes for patients.