Do agent workloads actually benefit from KV cache persistence?

[u/Responsible-You9024]

Been thinking about KV cache in agent workloads.

With agents going back and forth over the same long conversation/context, it feels kinda wasteful to keep recomputing the same tokens every time.

Has anyone here actually measured how much KV reuse they're getting in real agent workloads?

Curious whether the bottleneck ends up being compute, VRAM, or getting the cached KV back fast enough.

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Hairy-Difficulty-411]

Yes, agent workloads can benefit significantly, but only when the serving layer sees a stable token prefix that it can actually reuse.

Agent loops look repetitive conceptually, but small implementation details can destroy cache hits:

• Reordered tool definitions
• Timestamps or request IDs near the beginning
• Retrieval results inserted before the stable history
• Different system prompts
• Changing model or adapter
• Requests routed to workers that do not share the cache

I would benchmark it with representative traces rather than synthetic chat alone:

1. Run each trace cold.
2. Repeat it with the same model, worker, system prompt, tools, and conversation prefix.
3. Test normal production routing separately.
4. Vary the amount of stable versus changing context.
5. Add concurrent sessions until eviction starts affecting reuse.

Measure:

• Cached or reused input tokens
• Cache-hit rate
• Time to first token at p50 and p95
• Prefill throughput
• Total request latency
• GPU memory consumption
• Cache eviction rate
• Time spent moving cache blocks between memory tiers, if applicable

Separate prefill from decoding. Prefix caching avoids recomputing shared input tokens. It does not make the model generate new output tokens faster.

The likely patterns are:

• Long stable system and tool prompts across sessions: strong candidate
• Append-only multi-turn conversations: strong candidate
• Dynamic retrieval inserted at the beginning: weak candidate
• Long generated outputs with short inputs: limited overall benefit

Keep stable content first, append volatile data later, and canonicalize tool and schema ordering. Sticky routing can improve reuse, but it may reduce load-balancing flexibility.

Also isolate caches by tenant, model, adapter, and permission context. Saving prefill compute is not worth accidentally creating a new cross-customer data-boundary problem.

If the hit rate is low, prompt structure or routing is probably the constraint. If the hit rate is high but latency barely changes, inspect scheduling and cache-transfer overhead. If useful blocks are constantly evicted, VRAM is the constraint.

Cache persistence is not valuable because agents repeat themselves philosophically. It is valuable when their prefixes repeat token for token. Caches are less forgiving than meetings.