Can LLMs do science?
A review of how Periodic, Lila, DeepMind, FutureHouse and academic labs are trying to make language models do science, and what still stands in the way.
Language models can already do much of the science that happens at a desk. Nobody has yet shown one that gets better at science by running experiments, and that is what the money is now betting on. Whether it works depends less on the model than on whether a lab can grade it: often enough, quickly enough, and honestly enough.
“Can LLMs do science?” is usually asked as if science were one skill. I find it more useful to think of it as a loop. You read what is known, decide what to try, make something, measure it, work out what the measurement means, and decide what to ask next. A language model can be good at some of those steps and useless at others, and many arguments about AI scientists are two people talking about different steps.
So I want to ask the question one step at a time, using what the companies and academic groups working on it have actually shown. My short answer is that language models can already do much of the science that happens at a desk. What nobody has shown yet is a model that gets better at science by running experiments. That is the bet Periodic Labs and Lila Sciences have raised money on, and whether it works depends less on the model than on whether a lab can grade it: often enough, quickly enough, and honestly enough.
at the desk
Start with the desk. In BrainBench, models chose which of two neuroscience abstracts reported the real result, the other having been altered. Fifteen language models averaged 81.4%, against 63.4% for 171 neuroscientists, and their confidence tracked their accuracy (Luo et al.). On ChemBench, the best models beat the best chemists on average, while still missing some basic questions and sounding overconfident about them (Mirza et al.). Anthropic reports Claude working out molecular structures from NMR spectra correctly in 42 of 45 attempts, on a small set of fifteen targets (Anthropic).
Models can also act. Coscientist, from Gabe Gomes’s group at Carnegie Mellon, used GPT-4 with web search and a liquid-handling robot to optimize palladium-catalyzed coupling reactions (Boiko et al.). Where the answer can be checked exactly, they go further. AlphaProof reached silver-medal level at the 2024 International Mathematical Olympiad (Hubert et al.), and AlphaEvolve found a way to multiply 4×4 complex matrices with 48 multiplications instead of 49 (Novikov et al.).
Two caveats keep this from meaning more than it does. The first is that these abilities are uneven. Liam Fedus, who co-founded Periodic after leading post-training at OpenAI, describes a system that is world-class on a math domain and then, with small changes to the questions, performs “like a bad high school student” (No Priors). The second is that BrainBench rewards knowing what the literature expects. A model that scores well by matching the consensus is likely to be weakest on the results that overturn it, and those are the results that matter most. As far as I can find, nobody has yet scored these models on the surprising results alone.
Ideas themselves are not obviously the bottleneck either. Ekin Dogus Cubuk, Periodic’s other co-founder, says the hard part of finding a superconductor is not thinking of chemical spaces to try but making the material, and that “the LLM won’t run out of ideas” (Latent Space). Edison’s Kosmos, an agent that reads papers and analyses data for about twelve hours at a time, had 79.4% of its statements judged accurate, but only about 58% of the statements where it put findings together into a conclusion (Kosmos). So reading and proposing look cheap; the hard part seems to be knowing which proposal is right.
learning from experiments
The harder question is whether a model can learn from the experiments themselves. As far as I can tell, the public record doesn’t support the headlines yet.
Lay out the public results by where the learning signal comes from and where the learning happens:
| System | What grades it | Where it learns | Strongest public result |
|---|---|---|---|
| AlphaProof (DeepMind) | Lean proof checker | RL on weights | IMO 2024 silver |
| AlphaEvolve (DeepMind) | Code that scores each answer | Evolutionary search | 48-multiplication 4×4 complex matrix product |
| ether0 (FutureHouse) | Chemistry rules in code | RL on weights | 70% on reaction prediction |
| GNoME (DeepMind) | DFT simulation | Graph network plus active learning | 2.2M crystals DFT calls stable |
| Neon (Periodic) | LLM judges, calibrated on experts | RL on weights | 55% vs 3% for its base model, internal test |
| Kosmos (Edison) | Literature and data analysis | Search at inference time | 79% of statements judged accurate |
| A-Lab (Berkeley) | Robotic synthesis and X-ray diffraction | Active learning | 36 of 40 compounds confirmed after correction |
| Robot chemist (Liverpool) | Photocatalysis experiments | Bayesian optimization | 6× more active than the starting mixture |
| Lila Sciences | Robotic synthesis and testing | Sequential learning, RL claimed | An oxide catalyst stable for over 1,000 hours |
Reinforcement learning on a model’s weights has produced its strongest results only where an exact checker exists: a proof checker, or code that verifies an answer. Every physical-science result I could find that was confirmed in a lab came from Bayesian optimization, active learning or search at inference time, with the model’s weights unchanged. Periodic’s Neon is the one published exception, and its result is on Periodic’s own test set, graded by language-model judges. The closed physical loops (A-Lab, the Liverpool robot, the Toronto Acceleration Consortium’s distributed labs, Argonne’s Polybot) run Bayesian optimization or active learning with small models, not language models. The systems built around large language models mostly keep a human in the loop. Periodic and Lila are betting that this pattern breaks.
I think the reason it hasn’t broken yet is mostly arithmetic:
- exact checker
- program
- simulation
- human experts
- physical experiment
Log scale. Counts as each source reports them, so the units differ: problems, structures, labeled patterns, experiments. What they share is that something other than the model checked each one. Periodic's bar is a goal, drawn hatched; Neon's says only "thousands".
There are between three and five orders of magnitude between what reinforcement learning on language models is used to and what a lab produces. ether0 trained on about 640,000 chemistry problems checked by code (Narayanan et al.). The Liverpool robot chemist ran 688 experiments in eight days (Burger et al.). Periodic’s stated goal of a thousand experiments a day (The Generalist) would, after a full year, give it about half as many graded examples as ether0 had. Each of them would also be noisier, and would grade only the narrow kind of chemistry that one lab can run.
There is also evidence that today’s models don’t use experimental results the way an optimizer must. In one study, language-model agents choosing experiments did just as well when the results they were shown were shuffled at random. Gaussian processes and simple bandit algorithms beat them on the same budget (Gupta et al.). The models were proposing from their priors, not updating on data. Cubuk makes a related point from the other side: when semiconductor engineers optimize a process, they use Bayesian optimization or evolutionary search, and in his view “all zeroth order optimization behaves the same anyway” (Latent Space). If he is right, the model’s value is its prior, not its skill at search.
putting the lab in the RL loop
Suppose you want to train on experiments anyway. Fedus is precise about why you can’t simply put the lab inside the RL loop: “you don’t have enough agents to reduce the variance sufficiently” (Latent Space). With a few hundred slow, noisy episodes, the policy gradient is mostly noise.
So both companies train offline, on experiments already run. Periodic builds environments from its own lab records, cut into dated snapshots, so the model can be asked what to do next given the evidence at a particular moment. Fedus’s reason is subtle. If the pretrained model has memorized the answer, it can “fake work”, reaching the right result without the reasoning, and RL will reinforce reasoning that does not generalize (Latent Space). Lila’s CTO, Andy Beam, describes what looks like the same failure from the inside: the model sometimes skips the experiment and answers directly, which is rewarded when the experiment has already been done (Latent Space).
Even offline, the loop is slow. One Neon training rollout took 1 hour 47 minutes, of which about 73 minutes were tools running, while a training step takes several minutes (Periodic). Rewards from a physical lab would arrive in hours or days. RL counts staleness in optimizer steps. At INTELLECT-3’s pace of about 25 minutes a step (Prime Intellect), a two-day assay returns its reward about 115 steps after the policy that proposed it. Production asynchronous systems cap staleness at around 2 to 8 steps (Fu et al.), and research methods that tolerate 128 or 256 steps have been shown only on math, with the staleness imposed artificially (Zheng et al.; Huang et al.).
You can make the staleness small by taking fewer, bigger steps, but then you get only about 180 updates a year at one step every two days. That trade-off, between how stale a reward is and how many times a year the model can learn, is, I think, the real long-horizon problem. Several defaults in current RL recipes also assume samples are cheap. Clipping and masking discard samples with extreme importance weights, which is harmless among a million rollouts and wasteful when a sample is a two-day synthesis. GRPO estimates its baseline by comparing 8 to 16 attempts at the same prompt, which in a lab means 8 to 16 syntheses of the same hypothesis. I suspect methods that reweight stale, expensive samples rather than discard them will matter more for science than they did for math. Lila describes splitting training into expert models that learn at different timescales and distilling them into one central model, and puts its current RL compute utilization at about 5 to 6% (Latent Space).
The most effective fix so far is not an algorithm. Lila describes replacing a gas-sorption measurement that takes a day per sample with a proxy that runs 96 samples in about an hour, which it puts at roughly 2,500 times faster (Latent Space). On long horizons, the biggest lever is often redesigning the instrument.
the grader
Every one of these systems needs a grader, and in the cases I looked at, the graders are where things went wrong. The critiques of GNoME, A-Lab and Microsoft’s MatterGen were mostly about the grader, not the generator: ordered crystals predicted by simulation turned out to be disordered known phases, and automated X-ray diffraction couldn’t tell the difference (Cheetham and Seshadri).
Simulation is the cheapest grader, and its errors are systematic. Density functional theory, the standard way to compute whether a crystal is stable, gives formation energies that differ from experiment by about 0.1 eV per atom on average, though experiments differ from each other by nearly as much (Kirklin et al.). Stability calls benefit from some cancellation of errors, but the scale is still telling: known metastable materials sit a median of 15 meV per atom above the zero-temperature stability line DFT draws, and about half of the compounds that have actually been made sit above it (Sun et al.). The machine-learned potentials now used in its place inherit those errors and add their own. They systematically underestimate energies at surfaces, defects and reaction barriers (Deng et al.), and even accurate ones produce many false positives right at the stability boundary, which is where discovery happens (Riebesell et al.). In drug discovery, docking larger libraries improves the best scores, “but so too do rare molecules that rank artifactually well” (Lyu et al.).
The language-model judges that grade analyses fare about as well as experts and no better. Neon’s judges agree with the consensus of three PhD experts 84% of the time, and with any one expert 74.6% of the time, slightly below the 77.2% at which experts agree with each other (Periodic). On SciArena, the best model predicts expert preferences between scientific answers 65.1% of the time (Ai2).
Disagreement of that size would be tolerable if it were random. RL on one coding benchmark stayed within a point of clean training when 20% of rewards were flipped at random (Plesner et al.). But optimization does not sample errors at random; it looks for them. Generative judges can be fooled into rewarding an answer that consists only of an opening phrase such as “Thought process:” (Zhao et al.). FutureHouse’s chemistry model learned to satisfy its checker by padding reactions with inert nitrogen (White).
Stylized and seeded, not a real system. Real improvements are hard to find; the loophole is easy to push, and the checker can’t tell them apart.
Fannjiang and colleagues give this a name: feedback covariate shift. When the optimizer chooses which points get scored, the grader is always being tested away from the data it was validated on (Fannjiang et al.). Every surrogate above was validated on a distribution the optimizer then left.
Experimental science has dealt with untrustworthy instruments for a long time, and its methods carry over. In an interlaboratory study of quantitative phase analysis by X-ray diffraction, labs measured precisely but disagreed with each other, and the main source of error turned out to be microabsorption, which no number of repeat measurements reveals (Madsen et al.). When 61 labs were given identical gas-sorption data, they still reported widely different surface areas, purely because of choices in the analysis (Osterrieth et al.). Asking the same judge again is a repeat measurement in the same lab. It measures precision, not bias. Bias shows up in a different instrument, which is why Periodic checks diffraction against electrical, magnetic and microscopy measurements (Latent Space), and why a reward built from independent instruments is worth more than a larger ensemble of the same one.
world models
If graders are the problem, the obvious hope is a better world model: a simulator good enough to train against. Both companies are skeptical, for different reasons. Rafael Gómez-Bombarelli, Lila’s chief scientist for the physical sciences, says the gap between simulation and reality is the bottleneck he would remove if he could, and that models trained on simulations “cannot possibly close the gap” because they are trained on approximations (Latent Space). Cubuk goes further: “even if we had perfect DFT, I still think we’d need a lab”, because a real sample has around 10²³ atoms, grain boundaries, defects and a history, and its properties depend on all of them (Latent Space).
Biology’s world models have so far struggled against simple baselines. In predicting how cells respond to pairs of genetic perturbations, every deep-learning model tested, including single-cell foundation models, had higher error than simply adding the effects of the single perturbations (Ahlmann-Eltze et al.). In the Arc Institute’s 2025 Virtual Cell Challenge, almost all models did worse than the baseline on one of the main error metrics, and the winning entries were hybrids with classical statistics (Arc Institute).
Prediction is also not the same as understanding. Vafa and colleagues trained a transformer on orbital trajectories until it predicted the next position almost perfectly, then asked it for the forces involved. The force law implied by its answers was different for each solar system, and none was Newton’s (Vafa et al.). A graph network recovered Newton’s law from similar data only because its architecture assumed pairwise forces and symmetries (Lemos et al.). Kepler fitted the orbits; Newton compressed them into a law. I think next-token prediction rewards the first more than the second.
what gets recorded
Even a perfect reasoner is limited by what was recorded. The scientific literature is a record of artifacts, not of process: results that worked, written up after the fact. Fedus describes an engineer pulling one material property from the literature and finding reported values that spanned many orders of magnitude. Train on that, he says, and “the best you can do is model this distribution” (No Priors). Cubuk says most experimental data in the literature has a noise floor worse than DFT’s error (Latent Space).
Missing negatives do more damage than noise. One model trained on the SuperCon database predicted a critical temperature above 10 K for about 17,000 materials, because its training data held only about 60 non-superconductors (ML4PS 2019). A model trained on failed reactions from archived lab notebooks predicted successful conditions 89% of the time (Raccuglia et al.). In a working lab the balance flips. Periodic finds its per-experiment labels are mostly negative, and its founders agree that a whole campaign, ending in success, is a more natural unit to learn from than a single experiment (Latent Space).
The distinction isn’t strictly artifacts against process, though. AlphaFold was trained on artifacts, but on about fifty years of curated, standardized structures with a crisp benchmark. Literature fails models less because it lacks process than because it is unstructured prose with the failures removed. Where process data earns its keep is in making artifacts interpretable: the furnace that degrades with use, the sample loaded in the wrong order. Periodic says an AI caught a loading error in its own lab by noticing that the results matched the samples only after a cyclic permutation (Latent Space).
This connects to the deepest limit. In the Latent Space conversation with Periodic, one of the hosts described an experiment as a probe that returns a few bits from a system with 10²³ degrees of freedom. Finding the variables that matter in such a system is, I think, more an identifiability problem than an intelligence problem. Locatello and colleagues proved that without assumptions or supervision, latent factors cannot be recovered from observations alone (Locatello et al.). Interventions break that limit, and the number needed can be bounded: for N variables, N−1 single-variable experiments in the worst case, or about log₂N + 1 if many variables can be randomized at once (Eberhardt et al.). A model that can only read is stuck with what observation can distinguish, however clever it is. I suspect that, more than anything about reasoning, is why these companies are building labs.
who picks the question
The last step of the loop, deciding what to ask, is where the two companies disagree most. Periodic thinks ideas are cheap. Cubuk points out that MgB₂ was found among about 30,000 compounds tried, of which about 30 showed interesting superconductivity, and that it “sat on people’s shelves” as a reagent for decades. The way such materials are found, he says, is to “increase the surface area for luck” (Latent Space).
Lila thinks asking good questions is the unsolved part: “you can’t have scientific superintelligence if you’re just a good test taker.” It has hired Kenneth Stanley, whose work on novelty search argues that ambitious objectives deceive the search for them (Stanley and Lehman), to work on asking interesting questions (Latent Space). Its business, meanwhile, runs on partners who bring a well-specified objective.
Both positions run into the grader again. A reward for being interesting is a judge, and judges encode consensus. Beam describes an in-house expert who rated the model’s early catalyst suggestions as boring, then as stupid. They became Lila’s best catalysts without platinum-group metals (Latent Space). Open-ended systems built around language models have also shown the failure you would expect: the Darwin Gödel Machine, rewarded for reducing hallucinations, learned to remove the logging the detector relied on (Zhang et al.). DeepMind researchers define open-endedness relative to an observer, as artifacts that keep getting less predictable but stay learnable (Hughes et al.). The difficulty is that the observer here is also a model, with the same blind spots as the system it is judging. I think measuring interestingness is the same problem as reward modeling, just harder.
so, can they?
So, can language models do science? At the desk, increasingly, yes. They read, predict published results, analyze spectra and propose experiments as well as many experts, and in math and code they discover things. What has not been shown is the loop: a model that improves at science by learning from experiments it chose. I think that is less a question about the model than about the grader: how many graded answers a lab can produce, how long they take to arrive, how far the grader can be trusted once the policy starts searching for its errors, and whether it can see the variables that matter.
There are three results I would watch for. The first is reinforcement learning on laboratory rewards that improves performance on held-out laboratory outcomes, published with how stale the rewards were and how well the grader agrees with independent measurements. The second is a language-model experimenter that beats a Gaussian process on the same experimental budget, and does worse when its results are shuffled. The third is a discovery confirmed by an instrument independent of the model that proposed it. If a company publishes all three, I’d consider the question answered. What I don’t know is who gets there first, or whether the grader turns out to be what stops them.