Nice paper to improve inference efficiency.
It's been a while we haven't seen good work on efficiency.
Here is why it matters:
A long-running agent's workspace outgrows its context window long before the task finishes.
The first approach commonly used, compaction, loses the fine-grained execution evidence. And text retrieval re-prefills content the model already processed.
KVMem keeps the overflow as paged KV state instead, spread across GPU memory, host memory and NVMe.
Lightweight attention-space indexes, native to the model, pick the relevant historical blocks and materialize a query-dependent view that fits inside the native context window.
On the DeepSWE long-context test with Qwen3.8-27B, task success goes from 43.8% under compaction to 48.4%.
The local deployment result stands out. It runs Qwen3.6/3.8-27B NVFP4 with MTP on a laptop with a 24GB RTX 5090, virtualizing an agent workspace up to 1M tokens, four times the model's native 256K window, at around 50 tokens per second.