Repeated prefill is one of the quietest wastes in LLM serving.
LMCache tackles the problem by saving and getting back KV cache.

• 10K+ Github stars
• Benchmark shows up to to a 10.7x speedup
• And vLLM plus LMCache delivers 3-10x improvements on AMD MI300X.

💾 LMCache is a KV cache management layer for LLM inference.
LMCache allows the serving stack to reuse the heavy attention state from the first read of a long prompt, so the GPU doesn’t have to do that work twice.

That attention state is called the KV cache, where KV means key-value tensors from the model’s attention layers.

Normally, this cache lives like short-term memory inside the serving engine, so it can vanish when the engine restarts, fill up GPU memory, or stay stuck to 1 machine.
LMCache turns it into a managed layer that can sit across GPU high-bandwidth memory, CPU RAM, local storage, and remote storage.

That gives you 3 useful advantage: lower time-to-first-token, higher throughput, and cheaper long-context serving.

My favorite part is that LMCache does more than basic prefix caching, which means that the text that needs to be cached has to appear at the beginning of the prompt.

It can reuse repeated KV blocks from repeated or overlapping text. This is the same pattern you see in coding agents, retrieval augmented generation, long document QA, and multi-turn assistants.

And it is not locked to NVIDIA GPUs either.

vLLM with LMCache runs on AMD MI300X through ROCm, AMD’s GPU software stack. Also, there are separate non-CUDA paths for work that only needs to run on CPU or other accelerators.

🧵 1.