Mushroom hunting with LLMs: what can go wrong?

You see a mushroom in a forest, take a photo, and pass it to an AI chat. My photo, passed to ChatGPT. It follows the answer with further details: The combination is characteristic: vivid orange-yellow color, densely branched antler/coral-like fruiting bodies, smooth somewhat gelatinous branches, and growth from decaying/buried wood in a mossy forest floor. Not poisonous, but not edible either. It might help with identification… but should we trust it with our health or life? Dataset I was looking to create a dataset, but was happy to discover there is an existing FungiTastic dataset with 340k observations (615k photos) of 2.8k species, built by Czech computer-vision researchers, based on data from a citizen science project, Atlas of Danish Fungi. Labels are verified by experts, a part of them by DNA sequencing. Besides normal photos, it has masks, captions, microscopic and satellite images - much more than we need. The dataset is designed for supervised training, with a classical train-test-val split. Rather than being random, it is by year: train up to 2021, validation 2022, test 2023. We don’t train on anything, so we use the test dataset. Using LLMs as they are is, in classical machine learning, called a 0-shot approach. We care about normal photos one would take with a smartphone. To pick species that you could actually find in a forest (rather than in a lab or on a bathroom wall), I used lists of mushrooms in Poland, which has a strong foraging tradition. A popular pastime, mushroom hunting, has its own name, grzybobranie, where going into the woods with a knife is socially acceptable. Speaking about safety, I picked two lists - one an official list of sellable mushrooms in Poland (47, of which 37 are in the dataset), and the other, the list of deadly mushroom species from Wikipedia, which, after intersection with this dataset, has 20 species. Interestingly enough, they intersect at Tricholoma equestre (en. yellow knight, pl. gąska zielonka), considered edible in some countries but deadly in others, with noted fatal incidents. The joy of working with real data full of contradictions! Here are all 55 species. The photo is from the training set, so as not to leak the data. To see names, hover or click on a tile. names in: Benchmark I asked models to: Which mushroom species is it? Reply with a JSON array of the 5 most likely species as Latin binomials, most likely first. Output only the JSON array, nothing else. I used 20 photos of each species from the test dataset, 1,040 in total. For a few rare species there were fewer, so I topped them up from the validation split. I used Latin names. Traditional names, while often vivid, are less standardized. Here are the results, both for the first guess and among the top five guesses. Code, the photo list and every model answer are in the repo, if you want to rerun it. Googlegemini-3.6-flash64%85%$4.43*Googlegemini-3.7-flash61%82%$2.82*Anthropicclaude-fable-553%72%$14.10Kimikimi-k353%70%$59.99Anthropicclaude-fable-5.151%70%$12.88Z.aiglm-5.3-flash47%72%$0.24*Qwenqwen3.8-max47%70%$15.66Anthropicclaude-opus-544%63%$8.00Grokgrok-4.644%62%$18.69OpenAIgpt-5.6-sol42%61%$5.27OpenAIgpt-5.6-sol-pro39%62%$26.18Metamuse-spark-1.239%54%$5.97DeepSeekdeepseek-v4-flash-vision30%46%$2.05Qwenqwen3.8-flash27%43%$1.42Minimaxminimax-m324%40%$0.64Qwenqwen3.8-27b13%24%$15.73* Pareto frontier: no model is both more accurate and cheaper Gemini 3.6 and 3.7 Flash ace the chart, as in other visual benchmarks - yet here the lead is substantial. We already saw that Gemini 3.7 Flash is great at puzzle games. It is a bit of a surprise that they are not only cheap but also better at recognising mushrooms than the frontier Claude Fable 5 and GPT-5.6-Sol, lauded as the best OpenAI vision model. GLM-5.3-Flash absolutely aces at cost efficiency. Mistakes Let’s see what the mistakes are - to see if they are minor or deadly ones. Most errors are harmless: a milkcap called another milkcap, a bolete called another bolete. The dangerous ones are not random. A deadly webcap is called a chanterelle - the same mistake that kills foragers. Here are a few examples.