AI chatbots can diagnose medical cases as accurately as doctors.
GPT-4 scored strongly against physicians on written diagnostic cases, and Google's AMIE matched or exceeded clinicians in simulated consultations. A pooled analysis of 83 studies found no significant difference from non-specialist physicians, though AI performed below experts. The evidence comes from test cases and simulated patients, not routine care of real patients.
Current
Cases on this claim
Supporting
1
Disputing
0
1 case from 1 contributor. These numbers count cases people have written. They are not votes, and Koinovera does not decide whether the claim is true.
Original case
The case IKECHUKWU put this claim forward with on 6 Oct 2026. It is the publisher's case, open to the same scrutiny as every other.
+Supports
The study suggests that an LLM can outperform physicians in certain diagnostic reasoning tasks when used independently, but giving the LLM to physicians did not significantly improve their diagnostic performance. This is because the LLM alone scored 16 percentage points higher than the conventional-resources physician group, whereas physicians using the LLM scored only 2 percentage points higher than those using conventional resources, a difference that was not statistically significant.[1]
Passage from the sourceAbstract
Importance
Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves physician diagnostic reasoning.
Objective
To assess the effect of an LLM on physicians’ diagnostic reasoning compared with conventional resources.
Design, Setting, and Participants
A single-blind randomized clinical trial was conducted from November 29 to December 29, 2023. Using remote video conferencing and in-person participation across multiple academic medical institutions, physicians with training in family medicine, internal medicine, or emergency medicine were recruited.
Intervention
Participants were randomized to either access the LLM in addition to conventional diagnostic resources or conventional resources only, stratified by career stage. Participants were allocated 60 minutes to review up to 6 clinical vignettes.
Main Outcomes and Measures
The primary outcome was performance on a standardized rubric of diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps, validated and graded via blinded expert consensus. Secondary outcomes included time spent per case (in seconds) and final diagnosis accuracy. All analyses followed the intention-to-treat principle. A secondary exploratory analysis evaluated the standalone performance of the LLM by comparing the primary outcomes between the LLM alone group and the conventional resource group.
Results
Fifty physicians (26 attendings, 24 residents; median years in practice, 3 [IQR, 2-8]) participated virtually as well as at 1 in-person site. The median diagnostic reasoning score per case was 76% (IQR, 66%-87%) for the LLM group and 74% (IQR, 63%-84%) for the conventional resources-only group, with an adjusted difference of 2 percentage points (95% CI, −4 to 8 percentage points; P = .60). The median time spent per case for the LLM group was 519 (IQR, 371-668) seconds, compared with 565 (IQR, 456-788) seconds for the conventional resources group, with a time difference of −82 (95% CI, −195 to 31; P = .20) seconds. The LLM alone scored 16 percentage points (95% CI, 2-30 percentage points; P = .03) higher than the conventional resources group.
Conclusions and Relevance
In this trial, the availability of an LLM to physicians as a diagnostic aid did not significantly improve clinical reasoning compared with conventional resources. The LLM alone demonstrated higher performance than both physician groups, indicating the need for technology and workforce development to realize the potential of physician-artificial intelligence collaboration in clinical practice.
Written cases; small trial; model-alone success is not evidence of safe clinical deployment.
The study claims that AMIE, an AI system designed for diagnostic dialogue, demonstrated greater diagnostic accuracy and overall performance than primary care physicians in simulated text-based consultations. This is because specialist physicians rated AMIE higher on 30 of 32 performance measures, while patient-actors rated it higher on 25 of 26 measures, including diagnostic accuracy.[2]
Passage from the sourceAbstract
At the heart of medicine lies physician–patient dialogue, where skillful history-taking enables effective diagnosis, management and enduring trust1,2. Artificial intelligence (AI) systems capable of diagnostic dialogue could increase accessibility and quality of care. However, approximating clinicians’ expertise is an outstanding challenge. Here we introduce AMIE (Articulate Medical Intelligence Explorer), a large language model (LLM)-based AI system optimized for diagnostic dialogue. AMIE uses a self-play-based3 simulated environment with automated feedback for scaling learning across disease conditions, specialties and contexts. We designed a framework for evaluating clinically meaningful axes of performance, including history-taking, diagnostic accuracy, management, communication skills and empathy. We compared AMIE’s performance to that of primary care physicians in a randomized, double-blind crossover study of text-based consultations with validated patient-actors similar to objective structured clinical examination4,5. The study included 159 case scenarios from providers in Canada, the United Kingdom and India, 20 primary care physicians compared to AMIE, and evaluations by specialist physicians and patient-actors. AMIE demonstrated greater diagnostic accuracy and superior performance on 30 out of 32 axes according to the specialist physicians and 25 out of 26 axes according to the patient-actors. Our research has several limitations and should be interpreted with caution. Clinicians used synchronous text chat, which permits large-scale LLM–patient interactions, but this is unfamiliar in clinical practice. While further research is required before AMIE could be translated to real-world settings, the results represent a milestone towards conversational diagnostic AI.
Summary of the cases and their sourcesAI analysisNot generated yet
AI summary
Written by AI from the published cases and the sources they cite. It describes what each side argues and what the readable sources show, and is not a ruling from Koinovera. The generation date does not guarantee that the claim wording or sources are unchanged. It only changes when someone asks for an update.
No generated summary yet. Koinovera can summarise the usable evidence attached to this claim and cite the passages used.
Every argument made about this claim, each with its own reasoning and sources.
Strongest orders cases by how readers have rated their reasoning, weighted by the standing of those readers and lowered when a case rests on poorly rated sources. Cases nobody has rated sit in the middle, newest first. It never counts how many cases or sources there are.