Long‑context breakthrough paper from AMD.

Proves that long-context ability can be added after pretraining in a cheaper way, with far less memory use and only a small drop in normal short-context quality.

HyLo hybrid LLM architecture extends usable context up to 2M tokens with over 90% KV‑cache reduction and gains on long‑context benchmarks like RULER

The problem is that normal Transformer LLMs get costly and memory hungry on long inputs, while most newer hybrid models still need full pretraining from scratch.

HyLo’s answer is structural triage.

Keeps some attention layers for quality, replaces many others with cheaper layers that track long sequences with fixed memory, and starts from an already trained model.

They then train the converted model in 2 stages, first to stay close to the original model and then to work at 8K and 64K context with help from a larger teacher model.

Across Llama and Qwen models around 1B to 3B size, HyLo kept short-range results close to standard baselines and beat earlier upcycled hybrids on long-context tests.

That matters because it gives model builders a cheaper way to get much longer context and much lower memory use without throwing away a strong existing checkpoint.

Hybrid scaling may be less about inventing a brand new architecture, and more about deciding which parts of attention are truly worth keeping.

----

Paper Link – arxiv. org/abs/2604.24715v1

Paper Title: "Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling"