Lilith.
⌕
Editorial illustration: MedChemBench gives GPT-6.2 Astra 84.5%, while compound selection remains weak
Lilith illustration · editorial remix

OpenAI has released MedChemBench, a multimodal benchmark built from 99 problems across 117 papers and patents. A model must read molecular structures, connect them to experimental results and decide which compounds merit further testing. GPT-6.2 Astra earned 84.5% of available credit at the xhigh setting.

Ninety-nine cases test choices among competing properties

The dataset contains 761 distinct molecular structures. Seven task areas span integrated lead selection, structure-activity forecasting, ADME, safety and retrosynthesis. A model receives only the information available at a chosen point in a historical programme, while later results inform the reference answer.

Scoring separates structure reconstruction, scientific rationale, ranking and candidate selection, and numerical property prediction. GPT-6 Astra earned 95.4% of structure credit at xhigh, yet only about half the available credit for compound selection and numerical prediction. That split says more than the aggregate score.

The benchmark separates reading chemistry from research judgement

Medicinal chemistry is governed by trade-offs. Greater potency can worsen solubility, membrane passage or metabolic stability. MedChemBench therefore tests more than vocabulary: the model must choose a next step under several constraints at once.

The error breakdown is the practical value for research teams. A model that reconstructs an 18-membered ring correctly may still recommend the wrong compound. It can accelerate preparation and comparison, but experimental evidence and a chemist's judgement still own the final priority.

Historical cases leave room for the model to have seen the answers

The authors acknowledge that source material may have appeared in training data. Adding a DOI or patent identifier changed mean scores by only minus 0.23 to plus 0.80 percentage points, with intervals including zero, but that does not exclude contamination. A single historical outcome may also penalise plausible alternatives.

Each response was produced in one turn without tools, feedback or new experiments. The benchmark therefore measures reasoning over a prepared evidence package, not the ability to run an iterative research programme.

Prospective blind tests will reveal laboratory value

The decisive follow-up is a set of new cases whose outcomes are unknown when prompts are created. It also needs independent medicinal chemists to assess alternative designs and an agent track with retrieval, computation and revision.

MedChemBench is a useful diagnostic for the interface between a molecular drawing and a decision. Practical deployment begins when a proposal reaches a wet lab and is measured before the answer appears in a paper.

Lilith's verdict

The model can now assemble an 18-membered ring correctly on paper. A drug programme is decided when a chemist tests the chosen compound and the instrument returns a different number.