Making Room for the Voice

On a phone call, a second of silence after you finish speaking is just a pause. By ten seconds you're wondering whether the line has dropped.

Coding agents and voice agents looked to me like good tenants for one GPU, so I ran a voice agent taking two calls beside 24 replayed Claude Code sessions, all on a single consumer card running vLLM. On its own the voice agent answered within 1.8 seconds in nineteen turns out of twenty, and I set my target at that plus a quarter of a second. With the agents beside it, the same measure came to 10.5 seconds, and one reply in fourteen came after more than ten seconds of silence.

Figure 01 · the demo

The same line waits 9.6 s on tuned vLLM and 1.9 s with make-⁠room.

two recorded calls
turn 1, then ten

Recorded calls · every wait in real time

PlaySound

The clip plays turns from the recorded calls. In the upper lane, vLLM is tuned the way an operator would tune it, with the voice agent at the highest priority. The lower lane runs the same vLLM with make-room, the scheduler change I proposed to vLLM, and the rest of this post is how I got there.

Why I expected sharing to work

The whole plan rested on the agents and the caller wanting the GPU at different moments. A coding agent works on tool time. Each call resends a long context, often more than ten thousand tokens that are mostly the previous call's plus a little more, and then the agent goes quiet for seconds or minutes while a tool runs or its user reads the result. The 24 sessions here are replays of public Claude Code traces, pauses included, with each request shrunk to fit the card. Agents like these will take every cycle you give them for hours, and with nobody waiting on the line, an extra second on one of their calls goes unnoticed.

On a call, the voice agent mostly listens, and while it listens the GPU has almost nothing to do for it. Its requests are small. From the moment the caller stops talking until the first word of the reply, though, there's a person on the line waiting in silence, and they notice every moment of it.

So I expected the agents to soak up the idle time and each voice turn to borrow the GPU for the fraction of a second it needed. What I hadn't thought through was who holds the memory. A running agent call holds its whole context on the card as KV cache, and vLLM keeps it cached between calls, because recomputing a 16,000-token context takes more than three seconds and finding it still cached takes a tenth of one. Two or three running calls fill the cache, so when the caller finishes a sentence there may be nowhere to put the turn.

A gateway in front of vLLM

My first design left vLLM's code alone. I put a gateway in front of vLLM that watched for moments like a caller starting to speak or an agent's tool call coming back, and used them to keep room in reserve for the next voice turn. It deferred new agent requests, throttled agent prefill while a turn was being answered, and cut a voice reply short when room ran out. I called it Leaseholder, after the leases it would hold on each session's memory.

When I finally ran it alone in front of stock vLLM, without make-room, it didn't help. The slowest turn in twenty still took about eleven seconds, a little worse than vLLM without it, and the gateway's own counters showed why. Its headroom held back dozens of new agent requests in every run and freed nothing, because holding back new requests frees nothing that running ones already hold, and here the running ones held everything. Its throttle, which I'd been tuning, never deferred a single request, and the one part that acted on every turn was the cap that cut each voice reply short.

The deeper problem was where the gateway sat. A gateway sees requests, but the KV blocks those requests hold and the steps the engine runs are out of its view, so all it can do is hold back work that hasn't started, or cancel work that has and throw away what was computed. The voice turn needed memory that running requests already held, taken without wiping out their progress, and it needed the steps that decode its reply kept clear of agent prefill. In vLLM, those decisions belong to the scheduler.

Nothing else I tried without changing vLLM's code did much better. Other vLLM settings but one, caps on how many agent requests could run at once, SGLang's priority preemption, an emulation of llm-d's flow control and another router all left the slowest turn in twenty between 8.6 and 12.1 seconds. Reserving memory for voice ahead of time halved the tail and still missed the target by more than three seconds. The one policy that came close aborted agent requests that hadn't produced anything yet and resubmitted them once the voice turn had its first token. It got the tail to about 2.5 seconds by sending those requests back to wait again, and the agents' slowest requests went from waiting 106 seconds for their first token to waiting 193.

Where the seconds went

I should have timed a turn stage by stage, beside agents that fill the cache, before building anything.

Most of the wait for the first token is the turn waiting for memory. When the agents' running requests fill the KV cache, a new voice turn needs blocks that nobody will give up. vLLM's scheduler preempts running requests when another running request needs room to grow. A request that is still waiting can't make it do that, so the turn sits in the queue until an agent finishes, or until vLLM preempts an agent request to let another grow. In a probe that sent a short voice turn after two long agent requests had filled the cache, the turn waited five seconds for its first token.

My first version of that probe found no wait at all. Its voice turn got a first token right away, because the cache was still half empty when the turn arrived, so the probe had no way to fail. The wait showed up only once I sampled the engine's memory every half second and sent the voice turn after the cache had really filled.

Once the turn had its memory, the reply itself was slow. Every step that decoded it also carried chunks of the agents' prefill, the expensive first pass over a new long context. vLLM caps how many tokens go into a step, but the cap doesn't stop a step with prefill in it from taking a long time. Beside three long agent prefills, a voice reply's tokens came 60 to 66 milliseconds apart, and a first sentence runs 15 of them at the median and as many as 39.

Changing the scheduler

So the change had to go into vLLM's scheduler. I called it make-room.

With make-room, when the highest-priority waiting request can't get KV, the scheduler preempts lower-priority running requests to free the blocks it needs. It goes through vLLM's own preemption path and uses vLLM's own order for picking victims, and it takes at most two in a step, or none if two wouldn't be enough. Preemption costs an agent less than it sounds. vLLM's CPU tier already holds a copy of its KV, so when the agent resumes, it reloads whatever the tier still has and recomputes only what's missing.

The scheduler also holds back lower-priority prefill while a voice turn prefills and produces the first T tokens of its reply, a span the pull request calls the window. In the pull request, decoding carries on for everyone, so the agents' replies keep streaming through the hold and only their prefill waits.

By default the pull requests leave the change off. Among requests of the same priority the scheduler behaves exactly as vLLM's does, and the change works only from counts of blocks, tokens and steps, with no timers and no model of the hardware. The preemption and the hold are open as pull requests to vLLM, #65282 (opens in new tab) and #65283 (opens in new tab).

With only the preemption, make-room helped less than I expected. The wait for the first token fell from 2.81 seconds to 0.78, but the slowest turn in twenty still took 5.1 seconds, because the reply now decoded beside agent prefill. Adding the hold with its default window of 16 tokens brought the tail to 3.1 seconds.

Sixteen tokens is too short for a voice agent, which waits for a complete first sentence before handing anything to the speech synthesizer. The caller hears nothing until that sentence is ready. Replies here ran 23 tokens at the median, so a 16-token window ended before 79% of them did, often in the middle of the first sentence, and the rest of the reply decoded beside agent prefill again.

The window has to cover the first sentence. At 48 tokens it covers every first sentence I measured and all but one of the 5,571 replies in the evaluation, which is also where the sweep above stops improving. The longer hold cost the agents about 3% of their throughput, and a still longer one would cost nothing more, because the hold ends when the reply does.

With the preemption and a 48-token window, the slowest turn in twenty took 2.24 seconds, down from 10.48 and still 0.2 over my target, and the median reply came after 1.96 seconds, down from 4.61. The coding agents kept 97% of their throughput, and their own slowest requests waited about as long as they did without the change.

Since then

When I first wrote this at the end of October, the slowest turn in twenty was still 0.2 seconds over my target at the settings I'd evaluated, though cutting vLLM's cap on the tokens in a step from 512 to 256 had just halved the replies over it, and with four or six callers at once the tail missed it in every round. That gap has since closed. The changes that closed it sit outside make-room's pull requests, in how Flatshare serves the model. The scheduler's own traces showed where much of what remained came from, such as a call reloading its own context from the CPU, or waiting behind memory the agents had reserved for their loads. Fixing those, and finally holding the agents' decoding out of a reply's first 48 tokens along with their prefill, brought the replies over the target to 16 of 720 in two rounds on November 2, beside 24 agent sessions with two, four and six callers. The agents paid for each of these. The smaller steps cost them 7% of their token rate, the fixes another 2 to 5%, and the hold on their decoding 3.8% more, past the 3% I'd allowed it in advance. I kept it anyway for the voice tail.

The agents had a problem of their own, which showed up when I asked how many sessions one card could hold beside the calls. Each turn of an agent's session sends the whole conversation so far, and what the engine has already computed waits for it in the CPU tier, which drops whatever was used longest ago. With 24 sessions served in arrival order, each request waited longer than the tier kept its session, so it came back to find its own history gone and computed all of it again, which made the next request wait longer still. Nine tenths of what the agents sent was being computed again. Serving first the request with the least left to compute, counting what the GPU and the CPU tier still hold, broke the cycle. The share they computed fell to a fifth, and the agents' throughput more than doubled.

That fix made the calls slower. The queue now loaded far more from the CPU tier, and on this card's PCIe link each load ran on the GPU's copy engine, which kept the next step's small input copy waiting until the whole load had finished, about a fifth of a second for a large one. vLLM already ships a copy kernel that runs on the GPU's own cores, which it uses only for small pages, and sending the loads through it kept the steps moving. With the new order and the copy kernel, one card now holds 24 agent sessions beside the two calls, where it held four. In the three rounds that decided it, 3 of 177 replies came in over my target, and the agents' token rate was 2.2 times what it was at four. The price falls on the agents' slowest requests, which can now wait more than two minutes to start. I've proposed them to vLLM as well: #66558 (opens in new tab) lets vLLM's cache-aware admission, which already reorders waiting requests by what the GPU holds, count what the CPU tier holds too, and #66527 (opens in new tab) sends these loads through that kernel. They stay drafts until they have run on a GPU, and make-room's pull requests are still waiting for a maintainer's review.

Looking back

The change matters most when the agents fill the GPU's memory, and much less when there's room to spare. At 256 tokens a step the cache kept some room in the rounds that tested it: vLLM counted a handful of preemptions a run where it had counted dozens at 512, and the hold and the smaller steps did the work. All of this ran on one small model.

Most of what went wrong along the way came from the order I did things in. I designed a gateway before I had timed a turn beside agents that fill the cache, and when I finally did, the timing put the lost seconds inside the engine, where nothing in front of it could reach. The probe and the throttle were smaller versions of the same mistake. I believed the probe before making sure it could fail, and I counted the throttle as part of the design before reading the counters that showed it had never fired.

Percentiles can fool you in a similar way. One of vLLM's settings seemed to protect the voice tail until I noticed it was refusing 48 of 50 turns outright, and a refused turn never shows up as a slow one. Since then I count every failed turn as thirty seconds.

The window taught me the most. A default of sixteen tokens made sense from the engine's side, where everything is counted in tokens. I only saw the problem once I measured how long the agent's replies ran.

Beside 24 agent sessions, the slowest reply in twenty now comes 1.87 seconds after the caller stops talking, where on tuned vLLM it came after ten.

Links

• Code, data and figures: github.com/StevenWang-CY/flatshare (opens in new tab)
• The technical report, with the runs behind every number: docs/REPORT.md (opens in new tab)
• The pull requests: vllm-project/vllm#65282 (opens in new tab) and #65283 (opens in new tab)
• The proposals for the CPU tier: vllm-project/vllm#66558 (opens in new tab), the queue's order, and #66527 (opens in new tab), the copy path
• The full three-minute recording, ten turns with sound: release v1.27 (opens in new tab)

Flatshare is an independent project. It is not affiliated with or endorsed by the vLLM project or any other project named here.