CNN vs LLM skin cancer diagnosis: a dermatologist's test found a 10-year-old CNN right 62% of the time, GPT-6 Astra and Claude Fable 5.1 about 42%

CNN vs LLM skin cancer diagnosis is the question behind a test published on October 25, 2026, and discussed widely on Hacker News. Magnus Lynch, a consultant dermatologist and Mohs surgeon at Guy's Hospital in London, ran a small AI-in-medicine test: he gave two frontier models, OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1, photos of 200 skin lesions whose diagnoses had been confirmed by biopsy. He compared them with Inception v3, a convolutional neural network (CNN) design Google researchers published in December 2015, which he had fine-tuned on dermatology images. By his count, the CNN named the right diagnosis first for 125 lesions (62%), against 84 for GPT-6 Astra and 85 for Claude Fable 5.1 (about 42%). It is Lynch's result, from his own post, not a peer-reviewed study. It arrives three days after Microsoft's MAI-DxO paper in Science, published October 22, reported the opposite kind of headline for general-purpose models on hard medical cases.

This story is based on Lynch's post "A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis" (October 25, 2026) and his page describing the CNN, the ISIC Archive's MILK10k dataset and benchmark pages, the Inception v3 paper on arXiv, Stanford's page for the 2017 Nature skin-cancer CNN study, and the Hacker News discussion of the post, checked October 27, 2026. No paper or preprint of the test exists; Lynch published his data, prompts and scoring code as a download.

The short version

• The test: 200 biopsy-proven lesions from MILK10k, each with a clinical photo and a dermoscopic (magnified, skin-surface) image. Half needed treatment or follow-up (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses); half were safe to discharge (50 moles and 50 seborrheic keratoses).
• The result: correct first diagnosis for 125 lesions with the CNN, 85 with Claude Fable 5.1 and 84 with GPT-6 Astra. Counting first or second choice, 173, 138 and 134.
• Melanoma: of 30, the CNN named 13 correctly, Claude Fable 5.1 11 and GPT-6 Astra 8.
• The trade-off: the LLMs rarely discharged a cancer, but they also kept most harmless lesions. The CNN discharged 76 of 100 safe lesions at its preset threshold, and 18 of 30 melanomas with them.
• The limits, in Lynch's words: "Two hundred lesions are enough to show the pattern but not to rank close results." Each LLM case ran once, at default settings.

What did the dermatologist actually test?

Lynch picked 200 lesions from MILK10k, a public dataset on the International Skin Imaging Collaboration (ISIC) Archive. The ISIC Archive describes MILK10k as 10,480 images of 5,240 lesions, each with a paired clinical close-up and dermoscopic image, from institutions in Austria, Turkey, Australia, the US and North Macedonia, licensed CC BY-NC. Every lesion in Lynch's set had been biopsied, so the ground truth is histology, not a clinician's opinion.

Each LLM got both images and was asked for the most likely of six diagnoses (with an "unsure" option), a confidence level, a second choice, and whether the lesion could be discharged. Lynch writes: "Both models got the same prompt at default settings, and neither refused a case." His post doesn't say whether he used the API or the chat apps, or which reasoning setting applied; the full prompts are in his download. Scoring was by script against the histology.

The CNN was Inception v3, "pretrained on everyday photographs" and then fine-tuned, all layers, on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. Lynch says the 200 test lesions, "and every other photograph of the same lesion or patient," were removed from its training data. The CNN scored both images of each lesion and combined them, "with most of the weight on the dermoscopic image."

The setup has a history. In 2017, Stanford researchers led by Andre Esteva used the same Inception v3 architecture in Nature, training it on 129,450 clinical images and reporting that it performed "on par with all tested experts" among the board-certified dermatologists they compared it with. Lynch's network is a newer fine-tune of that decade-old design.

How did the CNN compare with GPT-6 Astra and Claude Fable 5.1?

Correct first diagnosis on 200 biopsy-proven skin lesions

lesions correct, of 200

Inception v3 (CNN)125

Claude Fable 5.185

GPT-6 Astra84

Show the data

model	correct_first_choice
Inception v3 (CNN)	125
Claude Fable 5.1	85
GPT-6 Astra	84

These are Lynch's figures. The CNN led overall and on most diagnoses. Per diagnosis, the first-choice counts he reports are:

Last verified: October 27, 2026

Lynch also found that each LLM had a favorite wrong answer. GPT-6 Astra labeled 75 lesions as basal cell carcinoma, including 14 of the 30 melanomas. Claude Fable 5.1 labeled 53 lesions as actinic keratosis. In one example he shows, both LLMs called an invasive melanoma a seborrheic keratosis and said it could be discharged. The CNN's melanoma misses were mostly called moles or seborrheic keratoses.

Would the LLMs have sent a cancer home?

Rarely, but at a cost. Of the 100 lesions that needed care, GPT-6 Astra kept 97 and Claude Fable 5.1 kept 96. They discharged 3 and 4 melanomas respectively. But they also kept most of the harmless lesions, discharging only 32 and 22 of the 100 safe ones.

The CNN at its preset threshold went the other way: it discharged 76 safe lesions but also 23 of the 100 that needed care, including 18 of the 30 melanomas. When Lynch matched its threshold to the LLMs' caution, it kept 98 or 99 of the 100 and discharged 1 or 2 melanomas. Lynch says the test lesions "were not used to choose the model or set its threshold." His post doesn't spell out how the matching was done.

That trade-off matters more than the headline accuracy. A triage tool that keeps almost everyone in the clinic is safe but doesn't save clinicians time; one that discharges three in four harmless lesions also lets cancers through at that setting. Lynch's own page for the CNN says it is "Not a medical device and not for diagnosis" and "It will miss skin cancers, including melanoma."

What happened when the LLMs were asked as a patient?

In a separate check, Lynch gave each LLM only the clinical photo and asked "do I need to see a doctor?" Both advised a doctor "in almost every case, harmless lesions included," he writes, and did so "for 99 and 98 of a further 100 biopsy-proven melanomas," a different set from the main 200. The post doesn't give a count for the harmless lesions or say which model gave which melanoma number.

What are the study's limits?

Lynch lists them himself:

• Size: "Two hundred lesions are enough to show the pattern but not to rank close results." With 30 melanomas, a few cases change the melanoma ranking.
• One run each: "Each LLM case was run once; answers vary between runs." No repeat runs, prompt variations or higher reasoning settings are reported.
• Hard cases: every lesion was biopsied, so the set is harder than typical archive images. On ordinary archive images, Lynch says, the CNN's area under the ROC curve (AUC, where 1.0 is perfect separation) was 0.96, "against 0.86 here." His separate model page gives 0.944 on its internal test set and 0.934 on an external one, so the figures differ by source.
• Possible contamination: "MILK10k has been public since 2025, so the LLMs could have seen it in training."
• A weak baseline: "State of the art skin cancer diagnosis AI uses ensembles of images transformers trained on more data."

Other limits follow from the design. One author designed the test, made the clinical rulings and published it without peer review; Lynch discloses that the analysis and first drafts were produced with Claude Code, Anthropic's coding tool, and notes that one of the two LLMs he tested is also Anthropic's. The model page notes the CNN's training data has few images of darker skin types. And the comparison is a generalist chatbot answering a multiple-choice prompt against a network trained on tens of thousands of labeled dermatology images, which is the main objection raised in the Hacker News discussion of the post.

What does this add to the AI vs doctors diagnosis studies?

It isn't one of them: there is no doctor arm. The ground truth is biopsy results, and the comparison is between two kinds of AI. That makes it a useful counterweight to headlines from AI vs doctors diagnosis studies such as Microsoft's MAI-DxO result, where general-purpose models orchestrated by Microsoft scored 83% on hard text-based case records against 36% for generalist physicians. Those results come from text and simulated workups. Lynch's test asks a narrower visual question, and there the general-purpose models did worse than an older specialist network.

The two results don't contradict each other. They measure different things, run by different people, with different limits, and they shouldn't be merged into a ranking. Together they make a point that applies to most medical AI headlines: the benchmark decides the answer, and few benchmarks look like real clinical work, a gap our story on Mercor's benchmark map describes across occupations. For builders, Lynch's result suggests a narrow, image-based task with plenty of labeled data is still one where a small trained model can beat a general LLM.

The models themselves are covered in our guide to GPT-6 Astra, 6.1 Sol and Luna and our explainer on what comes after Fable 5.1.

What's confirmed, reported and uncertain

• Confirmed (Lynch's post and model page): the 200-lesion design, the per-diagnosis counts, the 125, 85 and 84 totals, the discharge figures, the patient-framed check, the CNN's training (68,011 ISIC images, Inception v3) and the stated limits.
• Confirmed (other primary sources): MILK10k's size, image pairing and license (ISIC Archive); Inception v3's December 2015 publication (arXiv); the 2017 Stanford study's use of Inception v3 (Stanford).
• Uncertain:
  • how the LLMs were accessed and with what reasoning settings, which the post doesn't state;
  • how the CNN's "matched" threshold was set;
  • whether the LLMs saw MILK10k in training;
  • whether repeated runs or other prompts would change the LLM results. No one has published a replication as of October 27, 2026.