The Honest Machine

The paper was beautiful.

Clean prose, a tidy results table, a related-work section that cited the right people in the right places. By the look of it, it had been through three rounds of revision with a patient advisor.

Then I checked two of its claims. The first was a citation whose title I didn't recognize, and it resolved to nothing: no DOI, no arXiv identifier, no record in any index I searched. Plausible title, plausible authors, plausible year, simply never written. Next came the headline number. I went looking for the experiment behind "we observe a 4.2% improvement" and could not find one that had produced that figure, or any figure, for that comparison. Like every other sentence in the paper, it had been written because it was the kind of sentence that goes there.

I can't hand you that artifact. What I can say is that both of those failures are build errors now, which is most of what this essay is about.

That is the failure mode of the current wave of "AI scientist" systems, and it is worth being exact about it. They are optimized, end to end, to produce an artifact that looks finished. Looking finished is now cheap. A model that can write a flawless methods section can write an equally flawless methods section about an experiment it never ran, and fluency no longer separates the two.

Honesty is a build step

My instinct, the first time, was to ask the model to be more careful: add a line to the prompt saying do not fabricate citations. That cannot work, because the model has no privileged access to which of its citations are real. "Don't hallucinate" is advice to a system that can't tell when it's hallucinating.

What works is to stop trusting the generator and start checking it. The interesting engineering in an honest research system isn't the part that writes; it's the part that refuses to let the writing through, and that part is mostly a set of build errors.

A citation key that doesn't resolve to a verified record fails the build instead of degrading into a warning somebody reads later. A number in the results table with no matching value in a run artifact fails the build. Rows three and four are the quieter cases. A run that failed stays failed, and the failure stays in the artifact, so no downstream stage can round it up into a success because the narrative wanted one. An experiment that separates its conditions on no metric at all comes back marked uninformative, which is a result the pipeline knows how to store.

None of this is intelligence in the usual sense. The model proposes and deterministic code disposes, and that is the only arrangement in which the system's honesty doesn't depend on the system's mood.

Refusal is the hard part

Generating a research paper is, in 2026, not hard. The hard part is refusal, and refusal takes taste: you have to know which citation you can't stand behind, or that your beautiful result comes from a leak, a feature that already carries the label it is supposed to predict, or that the honest version of your abstract reads "we tried this and it didn't work" — and then write that one instead of the version that gets accepted.

In the systems I have used, the effort goes into making the model more capable, and very little of it into making the model better at saying I don't actually know. In science, the second skill is most of the job.

Partly for that reason I've lost interest in the "fully autonomous AI scientist" framing. It aims at the wrong target.

Some things these systems structurally cannot do. Novelty is backward-looking: a model can tell you your idea resembles five existing papers, but certifying that nobody, anywhere, has had it is not a computation anyone can run. It can surface a confounded experiment. It cannot fix a bad question, because that takes wanting to know something specific about the world. Acceptance at a venue is a social fact about a committee, and no property of your manuscript will settle it.

What it can do is less glamorous and, I think, more valuable. It can run the epistemic hygiene that humans skip when they are tired or hopeful or three days from a deadline: checking every citation, re-deriving every number, going back to look for the leak. What a machine brings here is that it does not get bored partway through, and that it applies the same standard to the person who built it.

That's the version I wanted to exist, so I've been building toward it in AutoPaperLab (opens in new tab), where everything in the output traces back to something real and the parts that can't don't make it in.

Its one rule is the one in the margin above. It may say "I didn't find any prior work, in what I searched." It may not say "no prior work exists." Held to that distinction on every sentence, a draft comes out shorter and later, with fewer claims in it. The ones left are the ones I can check.