Exécution de notre outil de synthèse sur un nouvel article
Vérifiez l’article ci-dessous, puis lancez sa synthèse.
Advancing conversational diagnostic AI with multimodal reasoning
Khaled Kamal Saab (Google DeepMind (United Kingdom)), Chunjong Park (Google DeepMind (United Kingdom), Google (United Kingdom)), Tim Strother (Google DeepMind (United Kingdom), Google (United Kingdom)), Jan Freyberg (Google DeepMind (United Kingdom), Google (United Kingdom)), David G. T. Barrett (Google DeepMind (United Kingdom)), Yong Cheng (Google DeepMind (United Kingdom)), Wei‐Hung Weng (Google DeepMind (United Kingdom)), David Stutz (Google DeepMind (United Kingdom)), Nenad Tomašev (Google DeepMind (United Kingdom)), Anil K. Palepu (Google (United States)), Valentin Liévin (Google DeepMind (United Kingdom)), Yash Paul Sharma (Google (United States)), Roma Ruparel (Google (United States)), Abdullah Ahmed Ali Ahmed (Google (United States)), Elahe Vedadi (Google DeepMind (United Kingdom)), Kimberly Kanada (Google (United States)), Cían Hughes (Google (United States)), Yun Liu (Google (United States)), Geoff Brown (Google DeepMind (United Kingdom)), Yang Gao (Google DeepMind (United Kingdom)), Sean Li (Google DeepMind (United Kingdom)), S. Sara Mahdavi (Google DeepMind (United Kingdom)), James Manyika (Google (United States)), Katherine Chou (Google (United States)), Yossi Matias (Google (United States)), Avinatan Hassidim (Google DeepMind (United Kingdom)), Dale R. Webster (Google (United States)), Joelle K. Barral (Google DeepMind (United Kingdom)), S. M. Ali Eslami (Google DeepMind (United Kingdom)), Pushmeet Kohli (Google DeepMind (United Kingdom)), Adam M. Rodman (Google (United States)), Vivek Natarajan (Google DeepMind (United Kingdom)), Mike Schaekermann (Google (United States)), Tao Tu (Google DeepMind (United Kingdom)), Alan Karthikesalingam (Google DeepMind (United Kingdom), Google (United Kingdom)), Ryutaro Tanno (Google DeepMind (United Kingdom), Google (United Kingdom))
Résumé : Abstract AI systems based on Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms can permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in conversation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal medical data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process which emulates that of experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, double-blind study of chat based consultations with 25 patient actors in the style of an Objective Structured Clinical Examination (OSCE). We constructed 105 evaluation scenarios, using common artifacts such as smartphone photos of skin conditions, ECG tracings, and PDFs of clinical documents across a diversity of conditions and demographics. We designed a rubric to assess multiple aspects of the multimodal capability of AMIE and PCPs, in addition to other clinically meaningful axes such as history-taking, diagnostic accuracy, management reasoning, communication skills, and empathy. Evaluation by 18 specialists revealed superior performance of AMIE relative to PCPs in handling and reasoning about multimodal data on 7 out of 9 axes while displaying similar superior performance in the non-multimodal metrics, including diagnostic accuracy, on 29 out of 32 axes. These results demonstrate clear progress in multimodal conversational diagnostic AI, although real-world translation necessitates further research.