Zero Train–Inference Mismatch — now for linear attention, and under async RL 🎯 We got bitwise-exact trainer/generator parity for Gated DeltaNet (Qwen3.5-9B / 35B-A3B) on TorchTitan RL + vLLM, then asked the question nobody had actually tested in open source: does it help async RL? Two firsts (as far as we know): 1️⃣ First open-source stack to hit train/inference logprob diff = 0 on a linear-attention model 2️⃣ First to measure what zero mismatch is worth under async off-policy RL How — smaller than it sounds: • trainer and generator share one model definition (TorchTitan unified model). No more aligning trainer & generator op by op • recurrent kernel everywhere on the forward (prefill + decode + trainer), chunked only for backward • split-K off, batch-invariant GEMM, + a batch-invariant bmm for the MoE router • we never touch FLA kernel internals — the recurrent kernel is intrinsically batch-invariant: one sequence's own state, fixed order, no cross-sequence reductions • vLLM's linear-cache management + prefix caching keep working as-is 📊 Qwen3.5-9B + DAPO-Math, async at offpolicy = 4 / 12 / 32: • logprob_diff = exactly 0 at step 0 ✅ every nonzero after that is staleness, not precision • at offpolicy=32 the stock stack runs away past 0.065; ours holds ~0.035 • the unified model is free — the entire bill is the kernels • also verified on MoE 35B-A3B and a 64-turn / 64K-context terminal agent 🤔 Now the honest part: it depends on the config. On math at offpolicy=12, BI leads on train reward and peaks highest on held-out AIME. On the terminal agent it's slightly ahead. On search, and on other math configs, the curves are indistinguishable. What is consistent is the numerics — BI keeps the gap from exploding as you widen the window. Cost: 2–3× trainer throughput (≈5× on the agent). Our take: bitwise parity is a debugging tool, not a production default. Flip BI on for 20 on-policy steps — if the gap is 0 and your run is still broken, the infra is exonerated and the bug is in your data or your algorithm. That's worth 20 steps, not a whole run. Next: • make BI cheap (kill the 2–3×) • more workloads, more seeds • stay tuned for more complex agentic training recipes/tests 👀 Full writeup 🔗 https://yichuan-w.github.io/blog/GDN-train-inference-mismatch-asyncRL/ We collaborated closely with the TorchTitan team, and we sincerely encourage everyone to check out TorchTitan RL. With its clean, efficient unified model architecture and native async RL design, we believe it is a strong foundation for your next RL engine. #LLM #RL #agentic #pytorch #torchtitan