AGENTIC TRAFFIC NOW MAKES UP MORE THAN 70% OF ALL INFERENCE TRAFFIC 🚀

Agentic workloads are characterized by four elements:

🟠 Multi-turn: a session includes tens or hundreds of turns, leading to high potential KV-cache reuse.

🟠 Long context: system prompts, tool definitions, and the large number of turns make context accumulate quickly.

🟠 High prefix reuse: since the conversation progresses linearly, where output from turn n-1 is concatenated to turn n (typically), most context can be served from KV cache rather than recomputed (this depends on the amount of storage available to store KV tensors). As n grows, the ratio of cached input relative to uncached input typically tends towards 1.

🟠 Sub-agent bursts: a session launches multiple short-lived sub-agents with fresh context, which create bursty KV-cache patterns.