Every request an inference engine takes is, from its point of view, the first. The prompt arrives whole. The engine tokenizes it, schedules it into a batch, reuses whatever prefix it has cached, streams tokens back, and drops the whole thing. Nothing about the request survives, and that forgetting is why the engine is fast: batching, prefix caching and scheduling all get simpler when no request carries obligations from the one before it.
An agent wants the opposite. A coding session is one conversation spread across dozens of model calls, with tool calls in the middle and their results fed back in, and by the third turn most of what the model needs to know is something it said itself earlier. Some of that was said to a tool and never to the user. Some of it was reasoning the client was never shown. Somebody has to hold all of it, and until recently that somebody was the client, resending the whole transcript on every turn.
vLLM's Agentic API (opens in new tab) is a Rust gateway that takes the job over. Reading its design records, I kept finding the same decision underneath each of them: the engine owns the tokens, the gateway owns the history, and much of the codebase is what that seam forces.
Where the seam goes
Figure 01 · the seam
The engine owns the tokens. The gateway owns the history.
three turns,
one store
One half of the rule is obvious and the other is not. Tokenization and chat templates stay in the engine, obviously. Every model family has its own template quirks and input sanitization, and the project declined to reimplement any of it; a neighbouring project that tried is converging back on reusing vLLM's. So the gateway calls vLLM's own stateless Responses endpoint, sends the full history every time, and lets the engine do what it already does.
Less obviously, the history belongs to the gateway and to nobody else. A turn arrives with a previous_response_id and only the new input. From that id the gateway loads a checkpoint, appends the input, and hands the engine a complete prompt. Neither side has to remember anything: the engine never learns that there was a previous turn, and the client never has to prove it remembers one. State that used to be duplicated in every client now lives once, next to the engine, in a database.
How it lives there is the first consequence. Each item is stored once, whether a user message, a model message, a tool call, a tool result or a reasoning item, and every response keeps an ordered list of item ids that serves as its continuation checkpoint. Embedding the whole history in every response row would have made storage grow with the square of the conversation length, since each turn repeats every earlier one and reasoning and tool payloads run far larger than the final text. Walking a chain of responses at request time would store less and read slower. The middle option reads one checkpoint, fetches its items in bulk, and restores their order in memory.
Clients that want none of this still get none of it. A request with no history, no storage and no gateway-owned tools goes through a byte-transparent proxy path to the engine, untouched. Statelessness stays available; it is no longer compulsory.
One turn, several rounds
Figure 02 · one turn
The client asks once. The engine may be asked several times.
up to ten rounds
behind one response
Because the gateway holds the history, the unit the client sees and the unit the engine sees can be different sizes. One request from the client asks for one response. Producing it can take several inference rounds, because the model may call a tool, and the tool's result has to go back into the next round before the answer can be finished. So the gateway runs that loop, at most ten rounds per turn, and projects the whole of it as one response and one continuous event stream, persisted once at the end.
Which tool runs where is decided by ownership, declared per tool before inference. Web search and MCP servers are gateway-owned: the gateway executes them mid-turn with bounded fan-out and feeds the results back. Plain functions, custom tools and the namespaced tool groups that coding harnesses declare are client-owned: their calls are returned unresolved, and the turn ends until the client answers. None of this distinction reaches the engine. Every tool is normalized to a plain function declaration before the prompt is built, and every call comes back as a function call that the gateway routes by looking its name up in a registry built at the start of the request. Mixed rounds are allowed, too. Whatever the gateway owns runs, whatever the client owns comes back, and the response says which is which.
Routing by registry is a small choice and a load-bearing one. Adding a new kind of tool means adding a handler on the gateway side; the engine keeps a single, simple notion of a tool, and the gateway keeps the rich one.
What remembering costs
Any checkpoint that grows forever will eventually exceed the context window, so the gateway compacts. It asks the model to summarize the history, stores the summary as a new item, and issues a new checkpoint that references the summary plus whatever recent items it kept. Old rows are left alone, and every earlier response id still works as a starting point. If the summary turns out to be poor, or a client wants the uncompacted branch, the earlier checkpoint is still there to continue from.
Deciding when to compact runs into the seam directly. Not owning tokenization means the gateway cannot count tokens, so it estimates instead: bytes of text divided by four, a fixed allowance for each item's framing, a flat allowance per image regardless of the image. Its documentation tells you to leave headroom because the estimate is not the tokenizer. Owning the history means owning its size, and not owning the tokens means measuring that size with a ruler that is approximate on purpose.
Reasoning items are the part I keep returning to. With current models, part of what the next turn depends on is content the client was never shown. It exists only in the gateway's store, replayed into each prompt, and the conversation, in the sense that matters to the model, is no longer something the client possesses. That is a strange place for a transcript to end up, and it is exactly where a design that lets the engine forget has to put it.