On a phone call, a second of silence after you finish speaking is just a pause. By ten seconds you're wondering whether the line has dropped.
Coding agents and voice agents looked to me like good tenants for one GPU, so I ran a voice agent taking two calls beside 24 replayed Claude Code sessions, all on a single consumer card running vLLM. On its own the voice agent answered within 1.8 seconds in nineteen turns out of twenty, and I set my target at that plus a quarter of a second. With the agents beside it, the same measure came to 10.5 seconds, and one reply in fourteen came after more than ten seconds of silence.
Figure 01 · the demo
The same line waits 9.6 s on tuned vLLM and 1.9 s with make-room.
two recorded calls
turn 1, then ten
Recorded calls · every wait in real time
The clip is one turn from each of two recorded calls. In the upper lane, vLLM is tuned the way an operator would tune it, with the voice agent at the highest priority. The lower lane runs the same vLLM with make-room, the change to its scheduler that I ended up with, and the rest of this post is how I got there.
Why I expected sharing to work
The whole plan rested on the agents and the caller wanting the GPU at different moments. A coding agent works on tool time. Each call resends a long context, often more than ten thousand tokens that are mostly the previous call's plus a little more, and then the agent goes quiet for seconds or minutes while a tool runs or its user reads the result. The 24 sessions here are replays of public Claude Code traces, pauses included. Agents like these will take every cycle you give them for hours, and with nobody waiting on the line, an extra second on one of their calls goes unnoticed.
On a call, the voice agent mostly listens, and while it listens the GPU has almost nothing to do for it. Its requests are small. From the moment the caller stops talking until the first word of the reply, though, there's a person on the line waiting in silence, and they notice every moment of it.
So, without giving memory much thought, I expected the agents to soak up the idle time and each voice turn to borrow the GPU for the fraction of a second it needed. But an agent's context stays on the card as KV cache after its call ends, because recomputing a 16,000-token context takes more than three seconds and finding it still cached takes a tenth of one. Two or three of those contexts fill the cache, so when the caller finishes a sentence there may be nowhere to put the turn.
A gateway in front of vLLM
My first design didn't touch the engine at all. I put a gateway in front of vLLM that watched for moments like a caller starting to speak or an agent's tool call coming back, and used them to keep room in reserve for the next voice turn. It deferred new agent requests, throttled agent prefill while a turn was being answered, and cut a voice reply short when room ran out. I called it Leaseholder, after the leases it would hold on each session's memory.
It didn't help. The slowest turns still took about eleven seconds, a little worse than vLLM without it, and the gateway's own counters showed why. Beside these agents its throttle never deferred a single request, and all it really did was shorten every voice reply. Deferring wouldn't have helped much anyway, since holding back new requests frees nothing that running ones already hold, and here the running ones held everything. Most of what I'd been tuning had never fired.
The deeper problem was where the gateway sat. A gateway sees requests, but the KV blocks those requests hold and the steps the engine runs are out of its view, so all it can do is hold back work that hasn't started, or cancel work that has and throw away what was computed. The voice turn needed memory that running requests already held, taken without wiping out their progress, and it needed the steps that decode its reply kept clear of agent prefill. In vLLM, those decisions belong to the scheduler.
Nothing else I tried without changing vLLM's code did much better. Other vLLM settings, caps on how many agent requests could run at once, SGLang's priority preemption, llm-d's flow control and another router all left the slowest turns between 8.6 and 12.1 seconds. Reserving memory for voice ahead of time halved the tail and still missed the target by more than three seconds. The one policy that came close aborted agent requests that hadn't produced anything yet and resubmitted them after the voice turn began to speak. It got the tail to about 2.5 seconds by throwing that work away, and the agents' slowest requests went from waiting 106 seconds for their first token to waiting 193.
Where the seconds went
I should have timed a turn stage by stage before building anything.
Figure 02 · the voice turn
Beside the agents, almost all of the turn's extra time is spent inside the engine.
mean of every reply
seconds
With make-room the engine's part of the turn drops to 0.29 s for the first token and 0.26 s for the first sentence
On average a turn waited 2.81 seconds for the model's first token, against 0.09 alone, and then took 1.07 seconds to decode a first sentence it could speak, against 0.22.
Most of the wait for the first token is the turn waiting for memory. When the agents' running requests fill the KV cache, a new voice turn needs blocks that nobody will give up. vLLM's scheduler preempts running requests when another running request needs room to grow. A request that is still waiting can't make it do that, so the turn sits in the queue until an agent finishes, or until one agent grows into another and vLLM evicts one of them. In a probe that sent a short voice turn after two long agent requests had filled the cache, the turn waited five seconds for its first token.
My first version of that probe found no wait at all. Its voice turn got a first token right away, because the cache was still half empty when the turn arrived, so the probe had no way to fail. The wait showed up only once I sampled the engine's memory every half second and sent the voice turn after the cache had really filled.
Once the turn had its memory, the reply itself was slow. Every step that decoded it also carried chunks of the agents' prefill, the expensive first pass over a new long context. vLLM caps how many tokens go into a step, but the cap doesn't stop a step with prefill in it from taking a long time. Beside three long agent prefills, a voice reply's tokens came 60 to 66 milliseconds apart, and a first sentence runs 15 of them at the median and as many as 39.
Changing the scheduler
So the change had to go into vLLM's scheduler. I called it make-room.
Figure 03 · the scheduler change
The scheduler frees memory for the waiting turn and keeps agent prefill out of its first tokens.
illustrative
blocks and steps
In the probes the first token's wait fell from 5 s to 0.22–0.29 s · and reply tokens from 60–66 ms apart to 16–20 ms
With make-room, when the highest-priority waiting request can't get KV, the scheduler preempts lower-priority running requests to free the blocks it needs. It goes through vLLM's own preemption path and uses vLLM's own order for picking victims, and it takes at most two in a step, or none if two wouldn't be enough. Preemption costs an agent less than it sounds. vLLM's CPU tier already holds a copy of its KV, so when the agent resumes, it reloads whatever the tier still has and recomputes only what's missing.
The scheduler also holds back lower-priority prefill while a voice turn prefills and produces the first T tokens of its reply, a span the pull request calls the window. Decoding carries on for everyone, so the agents' replies keep streaming through the hold and only their prefill waits.
The change is off by default. Among requests of the same priority the scheduler behaves exactly as vLLM's does, and the change works only from counts of blocks, tokens and steps, with no timers and no model of the hardware. The preemption and the hold are open as pull requests to vLLM, #65282 (opens in new tab) and #65283 (opens in new tab).
With only the preemption, make-room helped less than I expected. The wait for the first token fell from 2.81 seconds to 0.78, but the slowest turns still took 5.1 seconds, because the reply now decoded beside agent prefill. Adding the hold with its default window of 16 tokens brought the tail to 3.1 seconds.
Sixteen tokens is too short for a voice agent, which waits for a complete first sentence before handing anything to the speech synthesizer. The caller hears nothing until that sentence is ready. Replies here ran 23 tokens at the median, so a 16-token window ended before 79% of them did, often in the middle of the first sentence, and the rest of the reply decoded beside agent prefill again.
Figure 04 · the window
The reply stays fast while the window covers it.
the window T
in tokens
The 40th token of a 40-token reply: 3.76 s at T = 16 · 1.04 s at T = 48, the window deployed
The window has to cover the first sentence. At 48 tokens it covers every first sentence I measured and all but one of the 5,571 replies in the evaluation, which is also where the sweep above stops improving. The longer hold costs the agents little, because it ends when the reply does.
With the preemption and a 48-token window, the slowest turn in twenty took 2.24 seconds, down from 10.48, and the median reply came after 1.96 seconds, down from 4.61. The coding agents kept 97% of their throughput, and their own slowest requests waited about as long as they did without the change.
Figure 05 · results
Each step of the change took seconds off the slowest turns.
the slowest turn
in twenty, p95
Replies slower than the target: 90% on tuned vLLM · 32% with make-room at T = 48
Looking back
When I first wrote this at the end of October, the voice agent's slowest turns were still 0.2 seconds over my target, and with four or six callers at once the tail missed it in every round. Most of that gap has closed since. The changes that closed it sit outside the two pull requests, in how Flatshare serves the model. Cutting vLLM's cap on the tokens in a step from 512 to 256 halved the replies over my target. The scheduler's own traces then showed where much of the rest came from, such as a call reloading its own context from the CPU, or waiting behind memory the agents had reserved for their loads. Fixing those, and finally holding the agents' decoding out of a reply's first 48 tokens along with their prefill, brought the replies over the target to 16 of 720 in the latest rounds, beside 24 agent sessions with two, four and six callers.
The agents paid for each of these. The smaller steps cost them 7% of their token rate, the fixes another 2 to 5%, and the hold on their decoding 3.8% more, past the 3% I'd allowed it in advance. I kept it anyway for the voice tail. The change matters most when the agents fill the GPU's memory, and much less when there's room to spare. All of this ran on one small model, and both pull requests are still waiting for review.
Most of what went wrong along the way came from the order I did things in. I designed a gateway before I had timed a single turn, and when I finally did, the timing put the lost seconds inside the engine, where nothing in front of it could reach. The probe and the throttle were smaller versions of the same mistake. I believed the probe before making sure it could fail, and I counted the throttle as part of the design before reading the counters that showed it had never fired.
Percentiles can fool you in a similar way. One of vLLM's settings seemed to protect the voice tail until I noticed it was refusing 48 of 50 turns outright, and a refused turn never shows up as a slow one. Since then I count every failed turn as thirty seconds.
The window taught me the most. A default of sixteen tokens made sense from the engine's side, where everything is counted in tokens. I only saw the problem once I measured how long the agent's first sentences ran.
If I were putting a real-time workload beside agents like these again, I'd go straight to the scheduler, let the waiting turn take memory from the agents, and keep their prefill out of the steps that produce its first sentence.
Links
- Code, data and figures: github.com/StevenWang-CY/flatshare (opens in new tab)
- The technical report, with the runs behind every number: docs/REPORT.md (opens in new tab)
- The pull requests: vllm-project/vllm#65282 (opens in new tab) and #65283 (opens in new tab)
- The full three-minute recording, ten turns with sound: release v1.2 (opens in new tab)
Flatshare is an independent project. It is not affiliated with or endorsed by the vLLM project or any other project named here.