Abstract

The scaling laws hold that a language model grows more capable with more parameters and more training data, and Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous stored parameter bank for each token. That success is built on static pretraining data. A deployed model faces a different world, where much of the data that would make it more useful is not in its training set but in the live interaction it is currently handling, such as the facts a user supplies or the corrections they give. A conventional model cannot learn from this data, because its weights are frozen after training. Instead, the knowledge and behaviour supplied at run time are placed in the prompt, by retrieval or instruction, and re-read on every request only to be discarded once the request ends. We ask how an architecture could learn from live interaction by writing it into its weights. Taking inspiration from MoE, we propose the Infinite-Parameter LLM. A compact hypernetwork turns the data given at run time into a low-rank modulation of a shared base network, so the feed-forward weights are generated from live data rather than stored in a fixed bank. Where prior weight generators read the context once and freeze, we carry a Bayesian belief over the generator’s latent code and update it online, so the effective weight is re-derived from that evolving belief as the session proceeds rather than fixed after one read. The stored footprint stays fixed, yet the weights the model can compile are effectively infinite. For the knowledge and behaviour supplied at run time, carrying them in the weights rather than the prompt is amortized in compute, frees the context window, persists across turns, and can generalise better than in-context use. We specify an evaluation protocol that tests exactly this against in-context learning and retrieval.

1 Introduction

For half a decade, the reliable way to a more capable language model has been to train a bigger one on more data, and the scaling laws make the dependence on data precise. Capability rises predictably with the amount of training data, alongside parameters and compute (Kaplan et al., 2020; Hoffmann et al., 2022); data is a first-class input to capability. Two facts about where the data comes from now shape the problem. The first is that the supply of static pretraining text is finite. Current models are on track to exhaust the stock of public human-generated text between roughly 2026 and 2032 (Villalobos et al., 2024), so the easy gains from simply pretraining on more of it are running down. The second, and the one we build on, is that data has not stopped growing so much as changed form. Deployed models, and increasingly the agents built on them, generate an enormous and fast-growing stream of interaction data, from the questions users ask and the documents they bring to the corrections they give and the outcomes an agent observes. This data is produced at inference, from real use, and it is exactly the data a model most needs to become useful to this user on this task. If data is what buys capability, this is where the next of it will come from.

The trouble is that today’s models cannot learn from this data in the loop. A deployed model is frozen, and the interaction that just happened changes nothing about its weights. The workarounds all keep the data outside the weights. One family puts it in the prompt, where retrieval, long context, few-shot examples, and system prompts carry the knowledge a model draws on and the behaviour it should follow in the context window, re-read token by token on every request and discarded when the request ends. The other family builds around the frozen model with agent harnesses, tool orchestration, and external memories, engineering scaffolding that manages data without ever changing the network. Both avoid the harder question. If the valuable new data is generated by use, the model should be able to learn from it, which means its weights must change.

This is the question we pursue, and it concerns the architecture and its weight-update rule, not the harness around it. To learn from live interaction, a model needs weights that can take on new knowledge and behaviour cheaply, at inference, and hold onto what matters. We take our inspiration for such an architecture from the Mixture-of-Experts models already at the frontier.

A Mixture-of-Experts model stores a large bank of expert sub-networks and routes each token through only a few of them; DeepSeek-V3 holds 671B parameters yet activates 37B per token (DeepSeek-AI, 2024), and models such as Mixtral (Jiang and others, 2024), Qwen3 (Qwen Team, 2025), Kimi K2, and Llama-4 span a similar range. Seen through the right lens, such a model is less a collection of separate experts than a single network whose weights vary with the input. For a token $x$, an MoE layer applies the effective weight $W_{\text{eff}}\left(x\right)={\sum }_i{g}_i\left(x\right)W_i$, a combination of stored experts selected by an input-dependent gate, which is precisely the conditional-computation construction of Bengio et al. (2013) and Shazeer et al. (2017). A MoE is, in this sense, a dense feed-forward network endowed with dynamic, per-token weights, and it is this property, not its parameter count, that we take as our starting point. The weights of a model can be made a function of the input rather than fixed constants. Two things about the MoE realisation of that idea limit it for our purpose, however. Its dynamic weights are bought in memory, because although only a few experts are activated for any token the router may select any of them, so all must remain resident. And the bank it selects from is fixed once training ends, identical for every user and every moment of use, so a model serving a doctor and a novelist, at 9am and at midnight, reaches into the same unchanging palette. An MoE varies its weight with the token, but it can no more learn from the interaction in front of it than a dense model can. We keep the dynamic-weight idea and drop both limitations, generating the weights from a compact network instead of storing a bank, and letting them keep changing after the token that produced them.

Generation is what lets the weight carry what the prompt normally carries. It does not expand what a small model can store, since the information a network holds is bounded by its parameters, near two bits per parameter for MoE and dense models alike (Allen-Zhu and Li, 2024); a compact generator obeys the same bound as any other network its size. But storage is not the point. The comparison generation sets up is not against a bigger model, it is against the prompt. Today the knowledge and behaviour a model needs at run time are supplied in the context as retrieved facts, a task instruction, or a few demonstrations. A generator can instead compile that same data into the weights, so a user’s facts become a weight that answers without the facts in the prompt. This is the “hypernetwork as encoder” that a growing line of work has shown to work (Charakorn et al., 2025) and to scale, with injected knowledge improving as a power law in the generator’s size and generalising better out of distribution than the same knowledge left in context (Dhankhar et al., 2026). Behaviour compiles the same way, since an instruction or a set of demonstrations is data the generator can read into the adapter rather than into the prompt. And because task adaptation occupies subspaces of strikingly low intrinsic dimension (Aghajanyan et al., 2021), a compact generator and a low-dimensional code are the right-sized tools for this, not undersized ones. Carrying the data in the weights, rather than re-reading it from the context on every token, is amortized in compute, frees the context window, and persists past the turn. None of this is available to a prompt, which is re-read whole on every request and forgotten at the end of it.

There is, however, a gap between “generate from data” and “dynamic per token” that the existing generative work leaves open, and closing it is our contribution. The weight generators that turn a context into an adapter do so once. They read the whole context in a single pass, emit one adapter, and then hold it fixed while the model answers (Charakorn et al., 2025). That is turn-level, and it is memoryless, since the adapter does not evolve as more tokens arrive and nothing is carried from one turn to the next. To match the view we started from, in which an MoE varies its weight every token, and to let the model keep adapting within a session rather than resetting each turn, the latent code cannot be read once and frozen. It must be inferred online. We therefore treat the code as a latent variable with its own prior and carry a belief over it, updated as evidence arrives, coarsely once per turn or finely every token, by an amortized recursive Bayesian filter. The turn-level generator supplies the belief with a strong measurement from the live facts, and the online filter keeps the belief moving between measurements. Together they give a weight that is both generated from data and evolving in time.

Carried to its conclusion, this produces a model whose stored footprint is fixed but whose effective weights are not, since it keeps compiling fresh ones from whatever data the run brings. We call it an infinite-parameter LLM, and intend the term precisely and narrowly. It names the unbounded set of effective weights and behaviours the model can realise, a fresh expert per token, produced from a continuous code and moved by an evolving belief. The contrast is with an ordinary model, whose weights are fixed after training and whose only channel for anything new is the prompt. There the reachable behaviours are whatever the fixed weights plus a bounded context allow, whereas here the weights themselves are recompiled from live data at every step. Concretely, we replace the stored expert bank with a compact generator that synthesises each token’s expert on demand as a low-rank modulation of a shared base network, drive that generator from the run-time data, and carry a Bayesian belief over the latent code that the generator reads, updated online over the course of a session. Figure 1 places this regime in the landscape of ways a model turns data into capability; Figure 2 (§ 2.7) then locates our architecture against the specific prior designs it draws on.

媒体内容 · 前往原文查看

This closes the loop we opened with. The data that is still growing is generated by use, and a model whose weights are compiled from that data can turn it into capability at inference, in the loop, rather than only at the next pretraining run. As an interaction accumulates, more of it is written into the weights, and the model becomes more useful on the task at hand. We are careful about the scope of this claim. Session adaptation is bounded, low-dimensional, and reversible (§ 3.3), and it does not repeal the capacity law or substitute for pretraining. The loop we enable is that live data can enter the weights and be used, closing a path that a frozen model, prompted or scaffolded, leaves open, not that a model grows without limit from its own exhaust.

The design that follows is built around this weights-versus-prompt comparison. Knowledge and behaviour enter from the run-time data the generator reads, and what we generate and adapt is the low-dimensional part, namely which weight best fits the current context and how it should drift as the session goes on. Whether carrying data in the weights actually beats carrying it in the prompt, at matched budget, is what our evaluation measures (§ 4).

We situate the proposal within a natural progression along two axes at once, where the weight comes from, and whether it can change after training. Standard MoE and the bank-free variants ($\mu$MoE, $\infty$-MoE) select from a set that is fixed at deployment, and whether or not that set is unbounded, it is frozen. The recent weight generators generate the weight from context, but read the context once and then freeze the adapter for the turn. We take the last step, an expert space that is both generated from live data and adapted online, its weight moving with the data and with time, and make the following contributions.

1. A generative expert architecture and its design space (§§ 3.2 and 3.3). We make a shared base FFN’s weights dynamic through a generated low-rank delta driven by a latent code, with no stored expert bank, and set out a design space that positions $\infty$-MoE, $\mu$MoE, DFC, HyperMoE, and MoEGen by the axis on which each departs from this structure. The concrete choices that instantiate it — the base, and the form of the generator — are made in § 3.5.

2. The infinite-parameter view (§ 3.4). A precise statement of the sense in which the expert space is unbounded, a continuous generated family, one expert per token, extended over time by adaptation, distinguished from unbounded knowledge, with a guiding analogy to Bayesian-nonparametric mixtures of experts.

3. Online adaptation as a belief over the latent code (§ 3.3), the element that distinguishes us from one-shot weight generators. Rather than reading the context once and fixing the adapter, we carry a belief over the latent code and update it as the interaction proceeds, at three cadences (contextual, per-turn, per-token) under one probabilistic formulation, with uncertainty-gated retention, locating in-context learning, one-shot hypernetworks, continual-learning posteriors, and fast weights as points within it.

4. An evaluation protocol (§ 4) aimed at the comparison the design actually faces, carrying knowledge and behaviour in the weights versus carrying them in the prompt. The headline baselines are the prompt family, in-context learning and retrieval, with one-shot weight generators and point-estimate test-time training as adaptation baselines and stored-bank MoE as a reference point.

The ingredients each have precedent. Our contribution is their coupling, a shared-base low-rank generator driven from live data by a continuous latent code, made dynamic in time by recursive Bayesian inference over that code. Concurrent efforts that independently articulate the generate-instead-of-store thesis are discussed, and our differences delimited, in § 2.7.

2 Related Work

Our proposal touches several mature literatures; we organise them below and state our position against the closest work in § 2.7. We claim none of the individual ingredients in isolation.

2.1 Mixture-of-Experts: from stored banks to bank-free selection

The mixture-of-experts idea originates with adaptive mixtures of local experts and their hierarchical, EM-trained form (Jacobs et al., 1991; Jordan and Jacobs, 1994), and the underlying principle of conditional computation, activating input-dependent parts of a network for capacity without proportional cost (Bengio et al., 2013; Bengio et al., 2015). Under the scaling-law paradigm, where capacity reliably buys capability (Kaplan et al., 2020; Hoffmann et al., 2022), this made sparsity the default route to cheap capacity: an MoE layer applies a per-token effective weight $W_{\text{eff}}\left(x\right)={\sum }_i{g}_i\left(x\right)W_i$, a dense FFN whose weights are chosen conditionally on the input. Sparsely-gated MoE realised this at scale (Shazeer et al., 2017; Fedus et al., 2022), with subsequent work pursuing finer-grained experts and an always-on shared expert (DeepSeekMoE; Dai et al., 2024), a design that directly parallels our always-applied shared base FFN, and very large expert counts via retrieval (PEER; He, 2024) built on product-key memory (Lample et al., 2019) and related memory layers (Berges et al., 2024), with the returns to sparsity themselves the subject of MoE scaling laws (Clark and others, 2022; Krajewski et al., 2024; Abnar and others, 2025). All of these store their experts. Dense-to-MoE “upcycling” makes this explicit, replicating a dense FFN into a stored bank (Komatsuzaki et al., 2023), the replicate-and-store move we invert. Softer relaxations reduce discreteness but not storage: soft merging of stored experts (SMEAR; Muqeeth et al., 2024), scaled to autoregressive pre-training (Lory; Zhong et al., 2024). A separate line reaches an unbounded but frozen expert set without a stored bank: $\mu$MoE (Oldfield et al., 2024, § 2.2) factorises a fixed weight tensor, and $\infty$-MoE (Takashiro et al., 2026) draws a per-token continuous latent code from a Gaussian router and uses it as a top-$N\%$ activation mask over one shared FFN. $\infty$-MoE shares with us a shared base steered by a low-dimensional per-token latent code, but the resemblance is superficial. Its operator is a multiplicative mask that only reweights the existing neurons of a vanilla (non-gated) FFN, essentially giving an old-style FFN a GLU-like gate, so on a modern SwiGLU base, which already gates multiplicatively, the mechanism is largely redundant with the architecture. We instead generate an additive low-rank delta that steers neurons along new pre-activation directions, drive it from live data rather than a fixed router, and, the difference with no analogue in a frozen router, adapt the latent code online. We treat $\infty$-MoE and $\mu$MoE as the frozen-selection contrast, not as the precedent our method extends.

2.2 Compressing and factorising experts

A large literature makes experts cheaper. Low-rank or vector experts over a shared base recover most of full-expert quality at a fraction of the parameters (MoV/MoLORA, Zadouri et al., 2023; MixLoRA, Li et al., 2024; MoLE, Wu et al., 2024; X-LoRA, Buehler and Buehler, 2024; LoRAMoE, Dou et al., 2024), but retain a stored bank. Multilinear MoE ($\mu$MoE; Oldfield et al., 2024) is the closest “do not store experts” precedent: it represents the whole bank as a single CP- or Tensor-Ring-factorised weight tensor that is never materialised, routed by a differentiable entmax gate, so, like us, it stores no individual experts. The difference is that $\mu$MoE factorises a fixed tensor and routes linearly over it, confining each token’s effective weight to the convex hull of a fixed atom set, whereas we generate the factors from a latent code that is itself produced from live data (§ 3.2); it is also not adaptable. Because $\mu$MoE already achieves an un-materialised bank, we do not rest our contribution on the absence of storage but on generating the code from run-time data and adapting it online. Orthogonally, resident memory is reduced by quantising and decoding experts on the fly (QMoE; Frantar and Alistarh, 2023), pruning or skipping experts (Lu et al., 2024), merging them (HC-SMoE, Chen et al., 2025; MEO, He et al., 2023), offloading (Eliseev and Mazur, 2023), or distilling an MoE into a dense model (Xue et al., 2022), as surveyed by Liu et al. (2024). These compress a stored bank; we remove the bank and generate experts instead.

2.3 Hypernetworks and generated experts

Hypernetworks generate a target network’s weights (Ha et al., 2017); more generally, the dynamic-weight-tensor view treats any layer whose weights are an input-dependent function, made tractable by CP factorisation of the generated tensor (DFC; Babiloni et al., 2023), the general form our § 3.2 specialises to the FFN. Lightweight conditioning primitives such as FiLM (Perez et al., 2018) and (IA)³ (Liu et al., 2022) modulate a shared computation from an input-dependent signal, the same family as an additive or multiplicative weight modulation from a code. The most direct precedents generate a PEFT module for a frozen LLM in a single forward pass over a context or task description (HyperTuning, Phang et al., 2023; Text-to-LoRA, Charakorn et al., 2025; Doc-to-LoRA, Charakorn et al., 2026; SHINE, Liu et al., 2026; Drag-and-Drop LLMs, Liang et al., 2025; Zhyper, Abdalla et al., 2025), and the concurrent injection-scaling work (Dhankhar et al., 2026) shows this route scales: knowledge injected into a generated adapter improves as a power law in the hypernetwork’s size and generalises out of distribution better than a stored LoRA or full fine-tuning. This line is the closest to ours and the one we build on: it establishes that a hypernetwork acting as an encoder of live data injects knowledge that the model then uses without the data in context, the “knowledge from data, not from a bigger bank” leg of our design. These generators differ sharply in how they read the data, and at what cost. At one end, Text-to-LoRA reads only a short task description into a single embedding and generates the adapter from a small MLP, adding well under a percent of the base’s parameters (Charakorn et al., 2025). At the other, SHINE reads the full context by reusing the frozen backbone itself as the encoder, appending learnable memory tokens processed under an auxiliary “Meta LoRA” and mapping their all-layer hidden states to the adapter through a dedicated memory-to-parameter transformer; this reads context far more richly but adds on the order of a sixth of the base’s parameters (roughly $\left(L^{\prime }/L+2r/H\right)P$, about $17\%$ for their Qwen3-8B setting; Liu et al., 2026). This span, from a compact description-encoder to a backbone-reusing context-encoder, sets the sizing question our own encoder faces (§ 3.3). The gap we close is orthogonal to it: all of these read the context once and then freeze the adapter, so the generated weight is turn-level and memoryless, unchanged as the model reads on and reset from one turn to the next (SHINE’s recurrent variant chunks a long context but still produces a fixed adapter, not an evolving one). We keep the encoder-of-data generator and add what it lacks, a belief over the latent code that keeps moving as the interaction proceeds (§ 3.3). Within MoE, HMoE (Qu et al., 2022) and HyperMoE (Zhao et al., 2024) generate expert modulations from a low-dimensional latent code but retain the stored bank; Zhao et al. (2024) report that conditioning the generator directly on the token can underperform a standard MoE, the optimisation difficulty our compact latent bottleneck (§ 3.3) targets. The effort closest to our generator (see § 2.7) is MoEGen (Zeng et al., 2026), which generates instance-specific LoRA updates from a shared hypernetwork, though on the attention projections and from a per-prompt, top-$k$ discrete latent code, without online adaptation. A documented failure mode across weight generators is memorisation rather than generalisation (Zeng et al., 2025), which we treat as a first-class evaluation concern (§ 4).

2.4 Inference-time adaptation and fast weights

Adapting weights at inference descends from fast-weight programmers (Schmidhuber, 1992; Ba et al., 2016; Schlag et al., 2021). Test-time training updates weights by self-supervised gradient steps, as a sequence primitive (Sun et al., 2024), a long-term memory (Behrouz et al., 2025), reinforcement-learned self-edits (Zweiger et al., 2025), or per-task adapters that surpass in-context learning on novel structure (Akyürek et al., 2024); all produce point estimates. Methodologically, these approaches embed an updatable state within the sequence-mixing layer and update it by a hand-designed gradient or “surprise” rule; we instead leave attention unchanged, adapt only the generated FFN experts, and replace the hand-designed update with an amortized approximation to an explicit Bayesian filter (§ 3.3), yielding calibrated retention rather than a point estimate. Any per-token belief we carry is a low-dimensional FFN-side filter, adding no recurrent state to the attention/sequence-mixing path. Test-time compute can instead be spent on search or sampling against a verifier (Snell et al., 2024), a matched-budget baseline for us. Closest in spirit are online MoE adaptations: continuous rerouting via gradient-updated router-logit deltas (Rewiring Experts; Su et al., 2025) and gradient-free, uncertainty-guided Bayesian adaptation of expert confidence in medical vision-language models (MoBE; Imam et al., 2026). Both adapt the usage of a fixed expert bank, not the latent code of a generated manifold.

2.5 Continual, online, and Bayesian foundations for adaptation

Continual and online learning study exactly the problem of updating a model over time without erasing what it knows, the stability–plasticity trade and its failure mode, catastrophic forgetting (McCloskey and Cohen, 1989; Kirkpatrick et al., 2017). Its three families, regularisation (EWC; online EWC in Progress & Compress, Schwarz and others, 2018), replay (GEM, Lopez-Paz and Ranzato, 2017), and architecture growth (Progressive Networks, Rusu et al., 2016), together with distillation-based variants (Learning without Forgetting, Li and Hoiem, 2017) all target durable adaptation; van de Ven and Tolias (2019) taxonomise the settings, and recent work carries the problem to LLMs (Wu and others, 2024; O-LoRA, Wang et al., 2023). A complementary line shows that fixed-capacity networks progressively lose plasticity under continual updates (Dohare and others, 2024). We take two things from this literature. The framing: our uncertainty-gating is a stability–plasticity controller that spends plasticity where the posterior is uncertain and protects it where confident, so live adaptation increases the diversity of weight configurations realised over a session rather than the stored parameter count. The machinery: the recursive posterior-as-prior update (below). We differ by relocating this from full-weight, offline, task-sequential training to a low-dimensional, generated latent code updated online at inference, forward-only and anchored to base, so adaptation is bounded and reversible rather than a permanent consolidation.

Probabilistic treatments of MoE run from the original mixtures (Jacobs et al., 1991; Jordan and Jacobs, 1994) through Bayesian hierarchical mixtures of experts (Waterhouse et al., 1996), nonparametric infinite MoE via a Dirichlet-process gate (Rasmussen and Ghahramani, 2002), feature-allocation priors with unboundedly many latent features finitely active (Griffiths and Ghahramani, 2011), and modern identifiability/convergence theory for softmax gating (Nguyen et al., 2023). For LLM-scale adaptation, Bayesian posteriors over low-rank adapters are tractable (Laplace-LoRA, Yang et al., 2024; BLoB, Wang et al., 2024), and post-hoc structured Laplace has been applied to MoE expert layers (Bayesian-MoE; Dialameh et al., 2025). Our online update is recursive Bayesian filtering (variational continual learning, Nguyen et al., 2018; online Laplace, Ritter et al., 2018, building on Kirkpatrick et al., 2017; low-rank extended Kalman filtering, Chang et al., 2023), but applied to the generator’s low-dimensional per-layer latent code rather than to full weights or expert selection. Amortizing such a filter, training a recognition network to emit the state update in a forward pass, places us in the deep state-space / amortized-filtering lineage (deep Kalman filters, Krishnan et al., 2015; structured inference networks, Krishnan et al., 2017; deep variational Bayes filters, Karl et al., 2017; Kalman VAEs, Fraccaro et al., 2017), and we distinguish it from Kalman methods used as training-time optimizers over weights, whose observation is the loss rather than a predictive likelihood over a latent code (KOALA++; Xia et al., 2025).

2.6 Conditioning on run-time data through the prompt

The incumbent way to make a deployed model use run-time data is to place that data in the context. In-context learning conditions a frozen model on instructions or a few demonstrations supplied at inference (Brown et al., 2020), and can be read as implicit Bayesian inference over a latent concept the context selects (Xie et al., 2022); retrieval-augmented generation fetches relevant text into the context so the model can draw on knowledge it does not store (Lewis et al., 2020), with nearest-neighbour language models a non-parametric variant that interpolates an external datastore at the output (Khandelwal et al., 2020). Long-context modelling and soft prompt- or prefix-tuning (Lester et al., 2021) are further points on the same axis, enlarging or learning the conditioning signal while the model’s own weights stay fixed. All of these carry the run-time knowledge and behaviour in the context, where it is re-read on every request, competes for a bounded context window, and is discarded when the request ends; agent harnesses and external memories likewise manage this data around a frozen model rather than writing it into one. Our design targets the same goal by the opposite route, compiling that data into the weights, and § 4 makes in-context learning and retrieval the primary baselines against which the weight-carried alternative is measured.

2.7 Positioning: how this work differs

No confirmed prior work combines the full stack we describe, so we position against it on the two axes that survive the reframe: where the weight comes from, selected from a stored bank versus generated from data, and, once generation is granted, whether the weight keeps moving after it is produced, frozen for the turn versus carried as an online-updated belief. Underneath both sits the paradigm contrast that motivates the work, whether run-time knowledge and behaviour are carried in the weights or in the prompt; the whole generate-and-adapt family lives on the weights side of that line, and in-context learning and retrieval on the prompt side (we treat these as the primary evaluation baselines in § 4, not as architectural precedents). Figure 2 lays out the two architecture axes and the single cell each prior method occupies; Table 1 places the closest lineage, the context-driven weight generators, against the axes in detail; and Table 2 does the same on the adaptation axis specifically. We give stored-bank MoE only the two-axis summary and not a row-by-row scorecard: as § 1 argued, a stored MoE is the inspiration our design departs from and a reference point, not a method we compete with benchmark-for-benchmark, so the detailed comparisons below are with the generator and test-time-adaptation lines that are genuinely close to us.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Work	What drives the generated weight	$\Delta$ over shared base FFN	Weight after it is produced	Post- erior
HyperTuning (Phang et al., 2023)	task / context description	✓	frozen for the turn	$\times$
Text-to-LoRA (Charakorn et al., 2025)	task description	✓	frozen for the turn	$\times$
Doc-to-LoRA (Charakorn et al., 2026)	a document	✓	frozen for the turn	$\times$
SHINE (Liu et al., 2026)	in-context prompt	✓	frozen for the turn	$\times$
Zhyper (Abdalla et al., 2025)	conditioning / task	✓	frozen	$\times$
Injection scaling (Dhankhar et al., 2026)	fact corpus (train-time)	✓	frozen once baked	$\times$
MoEGen (Zeng et al., 2026)	per-prompt discrete code	$\times$	frozen for the prompt	$\times$
HyperMoE / HMoE (Zhao et al., 2024; Qu et al., 2022)	latent code	$\times$	frozen	$\times$
DFC (Babiloni et al., 2023)	raw input	$\times$	frozen per input	$\times$
$\mu$MoE / $\infty$-MoE (frozen sel.)	router over fixed atoms	$\sim$	frozen	$\times$
Rewiring / MoBE (fixed-bank)	— (adapts usage)	$\times$	evolves (bank usage)	$\sim$
Inf-params LLMs (This work)	live data + running evidence	✓	evolves online (belief over $z$)	✓

Read across these axes, the generate-instead-of-store thesis is by now partly anticipated. MoEGen frames the shift from expert selection to expert-conditioned generation, DFC generates weights from the input in general, the Text-to-LoRA / SHINE line generates adapters from context and shows the route scales, and $\infty$-MoE and $\mu$MoE both reach an un-materialised expert set, so we claim neither that thesis nor the absence of a stored bank as new. The two genuinely unclaimed elements are (i) the mechanism as a specific point in the design space of § 3.2, a generated low-rank additive delta over a single shared base FFN, driven by a latent code produced from data; and (ii) the coupling, in which the generator is driven from live data and the latent code it reads is not fixed but carried as a belief updated online by recursive Bayesian inference. The sharpest single distinction is against the generator line closest to us (Text-to-LoRA, SHINE): those read the context once and freeze the adapter, turn-level and memoryless, whereas we carry an evolving belief, so the weight keeps moving within a session. Distillation (Hinton et al., 2015), where we use it, is an enabling training choice and not a contribution; the reframed design does not rest on compressing a teacher bank. Each rival misses at least one axis: $\infty$-MoE masks rather than generates and is frozen; $\mu$MoE factorises a fixed tensor with linear routing and is frozen; DFC generates a factor but over the input directly, with no shared base or online adaptation; Text-to-LoRA/SHINE generate from context but freeze the adapter; MoEGen generates over attention with a per-prompt top-$k$ code and no online adaptation; Rewiring and MoBE adapt a fixed bank’s usage rather than a generated latent code.

媒体内容 · 前往原文查看

Work	What’s adapted	Where it lives	Update rule	Pt./ dist.	Granularity	Unc.	Forg.
Text-to-LoRA / SHINE	generated LoRA	FFN/attn adapter	read context once	pt.	per-turn (one-shot)	—	reset each turn
TTT (Sun et al., 2024)	inner-model weights	in sequence layer	gradient	pt.	per-token	—	implicit
Titans (Behrouz et al., 2025)	memory MLP	branch beside attn.	gradient + momentum	pt.	per-token	—	gate ${\alpha }_t$
SEAL (Zweiger et al., 2025)	full weights	whole model	RL $\to$ SFT	pt.	per-task	—	—
Rewiring (Su et al., 2025)	router logits	MoE router	gradient	pt.	per-segment	entropy (heur.)	reset
MoBE (Imam et al., 2026)	label statistics	frozen experts	gradient-free EMA	post. (labels)	per-sample	✓	—
This work (A–C)	generated latent code $z$	FFN-side, attn. frozen	amortized Bayes filter	dist. over $z$	in-ctx / turn / token	✓(prec.)	✓($Q$)

The adaptation axis tells the complementary story. The closest relatives on the “generate the weight” axis, Text-to-LoRA and SHINE, produce the adapter from context but then freeze it for the turn and reset each turn, so the weight does not evolve as the interaction proceeds. The test-time weight-adaptation methods do evolve the weight, but every LLM-side one updates a point estimate by gradient descent, inside the sequence-mixing layer (TTT; Titans), over the whole model by reinforcement (SEAL), or over a fixed bank’s router logits (Rewiring), while the sole Bayesian one keeps a posterior over labels, not parameters, by gradient-free moment-matching (MoBE). None both generates the weight from live data and carries a calibrated posterior over the generating latent code, updated by a distilled recursive filter with uncertainty-gating and principled forgetting, on the FFN side with attention untouched.

Concurrent work.

MoEGen (Zeng et al., 2026) appeared essentially concurrently and independently articulates part of the generate-instead-of-store thesis; we cite it as concurrent, delimit our differences above, and do not claim priority over the shared framing.

3 Method: Generating and Adapting FFN Experts

3.1 Overview

We build on a standard decoder-only transformer and leave attention untouched; only the feed-forward (FFN) sub-layer is changed, and only in a chosen subset of layers. At each such generative layer, three components replace the usual FFN (Figure 3): a shared base FFN, always applied; a compact generator $G_{\varphi }$ that maps a low-dimensional latent code to a structured low-rank modulation of that base; and a belief over the latent code, from which the code driving the generator is read and which is updated from live data and running evidence (§§ 3.2 and 3.3). The token’s effective expert is the base FFN plus the generated modulation. Crucially, no expert bank is stored: each token’s expert is generated from its latent code and discarded, so the resident parameters are the base, the generator, and the small inference map, all of fixed size, while the set of experts the model can produce is unbounded (§ 3.4).

This one mechanism carries all three of our claims. Infinite parameters: a fresh expert is generated from a latent code drawn from a continuous, data-materialised space, so the model deploys an unbounded family of effective weights rather than reusing a finite stored bank (§ 3.4). Knowledge and behaviour in the weights: because the code is produced from the data supplied at run time, the effective weights come to carry what a prompt would otherwise carry, such as facts, an instruction, or a few demonstrations, entering through the weights rather than being re-read from the context on every token (§ 3.2). Adaptation: because the expert comes from a latent code, and because we carry a belief over that code rather than reading it once, the model keeps specialising as a session proceeds, with the transformer left unchanged (§ 3.3). We first fix notation and set out the belief-over-code framework (§ 3.2); describe how live data writes the belief and how per-token inference moves it (§§ 3.2 and 3.3); make the infinite-parameter claim precise (§ 3.4); and only then commit to the concrete architectural choices that instantiate the framework (§ 3.5).

媒体内容 · 前往原文查看

3.2 The belief-over-code framework

Notation and setup.

We write $d$ for the model width, $h$ for the FFN hidden width, $d_z$ for the latent dimension, and $r$ for the rank of a generated modulation, with $r\ll d$. Layers are indexed by $\ell$ and tokens within a sequence by $t$; the input to a generative layer is the post-attention hidden state $x=h_{\ell ,t}\in {ℝ}^d$, which already integrates context through the layer’s attention and the residual stream. The frozen, shared base FFN has weights collectively denoted $W_0$ and is initialised from a strong dense model (§ 4.1). The three moving parts are a Data-to-Weight encoder $E_{\varphi }$ that reads run-time data into a latent code, a generator $G_{\varphi }$ that maps a code $z$ to a low-rank modulation $\Delta W\left(z\right)$ of the base, and a belief over the code that is updated online. The code budget $(r,d_z)$, the base architecture, and the concrete values these symbols take are choices we fix in § 3.5.

The primitive: a belief over the latent code.

The object at the centre of the design is not a weight and not a code but a belief over the code — a distribution $P\left(z\right)$ that the model carries and updates as it works (we reserve $q$ for the amortized approximation to it, § 3.3). Everything else is downstream of it: the code that drives the generator is a summary of the belief (its mean, or its most probable atom), the weight delta is a function of that code, and adaptation is inference on the belief. Fixing the belief as the primitive, rather than the weight or a point code, is what lets one mechanism serve the two channels of § 1: a measurement channel, by which run-time data writes the belief (this is where knowledge and behaviour enter the weights, later in this section), and an inference channel, by which the belief moves between measurements as evidence accumulates (§ 3.3). The form of the belief — a Gaussian over a continuous code, or a categorical distribution over a pool of materialised codes — is an architectural choice we defer to § 3.5; the framework, and the two channels, are the same either way.

The generated weight.

Given a code $z$, the effective weight of any modulated base projection $W_0$ is the base plus a generated low-rank additive delta,

$W\left(z\right)=W_0+\Delta W\left(z\right),\Delta W\left(z\right)=B\left(z\right)A{\left(z\right)}^{\top },A\left(z\right)\in {ℝ}^{d\times r},B\left(z\right)\in {ℝ}^{h\times r},$

(1)

with the factors $A\left(z\right),B\left(z\right)$ produced from the code. The delta is never materialised: we compute $y_{\text{delta}}=B\left(z\right)\left(A{\left(z\right)}^{\top }x\right)$, so the per-token application costs $𝒪\left(r\left(d+h\right)\right)$ per layer, negligible relative to the base FFN’s $𝒪\left(hd\right)$. This is the framework; the specific base (SwiGLU), which projections carry a delta, and how $A,B$ are produced are concrete choices made in § 3.5. What matters for the framework is only that the weight is a function of a code, and the code is drawn from a belief.

The measurement channel: writing the belief from live data.

The belief is written from the data supplied for a turn — the facts, instruction, or examples — by the Data-to-Weight encoder $E_{\varphi }$ (the encoder-hypernetwork of § 2.3). This is where the knowledge and behaviour that would otherwise sit in the prompt enters the weights: $E_{\varphi }$ turns supplied data into a code (or, categorically, into a new materialised code added to the pool, § 3.5), and it is a strong, content-rich measurement rather than a cheap per-token guess. A router over the running hidden state cannot, by itself, inject a fact the base was never given; only the measurement channel can, which is why the encoder and the per-token inference of § 3.3 are distinct modules.

The encoder is where the design’s cost concentrates, and the framework spans a spectrum of realisations trading footprint against reading fidelity; we set out the axis rather than fix a point on it. At the light end, the encoder reads the supplied data with the base model’s own forward pass (which must process those tokens regardless) and taps a small readout head on the resulting hidden states, so the head and generator are the only added parameters, at the scale of a description-conditioned hypernetwork (Text-to-LoRA’s smallest variant adds under a percent of the base; Charakorn et al., 2025). In the middle, the backbone is reused as a dedicated context-encoder with auxiliary read-time adapters and a memory-to-parameter network, as in SHINE (Liu et al., 2026), which reads long context more faithfully at the cost of on the order of a sixth of the base’s parameters. At the heavy end, the encoder is a completely separate hypernetwork, not tied to the base’s weights at all, as in the knowledge-injection hypernetworks of Dhankhar et al. (2026), whose evidence is that injection fidelity scales with this hypernetwork’s capacity. These are points on one axis — how much dedicated machinery reads the data into the code — and which is warranted is an empirical, footprint-versus-quality question (§ 4.3) rather than settled by fiat; a relevant consideration along the way is that the online belief (§ 3.3) can correct an imperfect one-shot read that a frozen generator cannot, which can relieve a lighter encoder of carrying the whole burden in a single pass. Attention and the base weights are frozen throughout.

3.3 Moving the belief: online inference over the code

The element that separates this design from the one-shot weight generators of § 2.3 is that the code is not read once and fixed; we carry the belief and update it as the interaction proceeds. Adaptation, in every variant, is therefore inference over the latent code, with the generator and base frozen. The transformer’s attention / sequence-mixing path is left unchanged throughout: no variant inserts a recurrent state into the sequence layer, in contrast to test-time-training methods that adapt the sequence path itself (Sun et al., 2024; Behrouz et al., 2025). The only state carried across steps is the low-dimensional belief, and it lives entirely on the FFN side.

Why a belief and not a point.

Existing test-time adaptation carries a point estimate of the adapted weights and moves it by gradient descent (TTT, Sun et al., 2024; Titans, Behrouz et al., 2025). Carrying instead a posterior over the code earns three things a point cannot. First, calibration: the posterior’s spread is an explicit statement of how much to trust the adaptation, usable to gate, abstain, or defer when the model is uncertain. Second, uncertainty-gated stability–plasticity: a precision-weighted update adapts fast where the posterior is unsure and protects what it is confident in, resisting catastrophic forgetting without a bolted-on regulariser (a derived analogue of elastic weight consolidation, Kirkpatrick et al., 2017). Third, principled forgetting: a process-noise term gives a controlled, optionally content-aware way to reopen plasticity when the input distribution shifts. These benefits are carried by the posterior’s spread, which is exactly the fragile part under amortization, so “Bayesian” here is an empirical claim about a calibrated posterior, not a free consequence of emitting a distribution, and validating it — exact-filter recovery, the amortization gap, and calibration of the posterior precision — is part of the continuous-Gaussian instantiation we leave to future work (§ 3.5).

The exact update, and why we amortize it.

The belief is updated by an exact recursive Bayesian filter, the same object at every cadence. At an update, the observation over the tokens since the last update is either the model’s own log-likelihood $ℒ\left(z\right)={\sum }_i\log p_{\theta }(x_i\mid x_{\phantom{}<i};z)$ (the self-supervised regime, always available) or an explicit feedback likelihood $p(y\mid \text{context};z)$: Boltzmann in a scalar reward, $p(y\mid z)\propto \exp \left(r_z/T\right)$, or Bradley–Terry for a pairwise preference (the feedback regime). The recursion is Bayes’ rule applied to the running posterior,

$P_t\left(z\right)\propto P_{t-1}\left(z\right)\cdot p({\text{obs}}_t\mid z),$

(2)

carried from step to step rather than recomputed from scratch. Throughout, we write $P$ for this exact recursive belief and $q$ for the amortized approximation to it that we actually run — the standard variational reading in which a learned $q$ is fit to a target $P$. Computing the likelihood term exactly requires a test-time backward pass to the code, impractical per token at deployment, so the exact belief $P$ is kept only as an offline reference (a distillation teacher, and a comparison baseline) and amortized: a trained forward map $F_{\varphi }$ emits the belief update in a single pass, its output $q$ distilled against $P$ (Putzky and Welling, 2017; Marino et al., 2018); the recognition-network instance of a state-space filter (Krishnan et al., 2015; Karl et al., 2017; Fraccaro et al., 2017). One property makes a single $F_{\varphi }$ serve both cadences below: the exact update over a window of tokens is the same function of (prior belief, accumulated observation) whatever the window’s length, so $F_{\varphi }$ reads the prior belief and a pooled summary of the window (with a length feature) and is distilled against the exact trajectory at both cadences. This is the per-token signal of Figure 3, drawn dashed because it exists only to amortise the posterior $q(z\mid x_{1:t})$ — it is not on the main data path, and switching it off returns the one-shot generator.

The three cadences.

The designs place this one machinery at three points on the belief-granularity axis, indexed by token $t$ or turn $\tau$; they are not three mechanisms but one belief updated more or less often.

• Design A — Contextual (implicit belief). No explicit belief is carried within a sequence; context is integrated by ordinary attention, and a router $R_{\ell }\left(h_{\ell ,t}\right)$ maps the contextual hidden state to the code. Per-token generation is then an amortized predictive inference, the forward pass approximating the Bayesian predictive in-context (Xie et al., 2022). This is the cheapest variant and the degenerate member of the family — Bayesian only in the weak sense that in-context learning implicitly approximates a posterior predictive, with none of the calibration, persistence, or controlled forgetting the explicit belief buys. We keep it as the baseline the explicit-belief designs must beat (§ 4.4).

• Design B — Session posterior (per-turn update). An explicit belief is maintained per layer and updated once per turn by $F_{\varphi }$, from the prior belief and a pooled encoding of the turn (and any feedback). Because it fires only per turn, B can equally run the exact filter online — one backward pass per turn is affordable — making amortization optional here. It gives persistent weight-space adaptation at turn granularity and carries no per-token state.

• Design C — Fast belief filter (per-token update). The belief is carried as a side state and updated every token by the same amortized map, $b_t=F_{\varphi }(b_{t-1},s_t)$, on a per-token signal $s_t$; the generator reads its summary. Here amortization is essential. This is a genuine per-token weight-space update realised as a benign, low-dimensional recurrence outside the attention/sequence path, the finest-grained and fully persistent variant, at the cost of a small carried belief and a cheap forward-only filter step per token.

All three instantiate the same idea — a generated, continuously-indexed expert space adapted by Bayesian inference over its latent code — and differ only in the granularity and persistence of that inference. The family also locates prior work within one frame: discrete MoE and $\infty$-MoE are the frozen limit; in-context learning is the contextual instance (A); and fast-weight/TTT methods are per-token updates placed in the sequence layer rather than, as in C, in a low-dimensional FFN-side belief. Because test-time gains tend to accrue with the number of updates rather than their size (Sun et al., 2024), we expect C to dominate B under fine-grained drift, with B the natural read-out when the phenomenon and its labels live at turn granularity; since C run over the whole conversation subsumes B, our accumulation study (§ 4.4) updates at C and reports at the turn level.

媒体内容 · 前往原文查看

	A — Contextual	B — Session (per-turn)	C — Fast filter (per-token)
Belief update	none (implicit in context)	once per turn $\tau$	every token $t$
Carried state	none	belief $P_{\tau }\left(z\right)$	belief $P_t\left(z\right)$
Update rule	router reads the code	$P_{\tau }\propto P_{\tau -1}\cdot p({\text{obs}}_{\tau }\mid z)$	$P_t\propto P_{t-1}\cdot p(x_t\mid z)$
Wins when	context suffices; short interactions	task shifts across turns; per-turn feedback	long single stream; fine-grained drift

3.4 The infinite-parameter view

We call the model an infinite-parameter LLM in a precise sense: the set of experts reachable at a generative layer is $\{W_0+G_{\varphi }\left(z\right):z\in 𝒵\}$, where $𝒵$ is the space of codes the encoder can materialise from data. The stored parameters — base, encoder, generator, and the small inference map — are finite and fixed; the reachable effective weights are not, because $𝒵$ is not a fixed finite index but a space populated by whatever data the model is given. Over an interaction the model instantiates a growing set of distinct weight configurations rather than reusing a fixed bank.

This is where the categorical instantiation of § 3.5 must be positioned carefully, because it looks like the finite selection the paper otherwise argues against. The distinction is the origin of the atoms. A classical MoE selects among a fixed, stored bank of experts; its reachable set is the convex hull of those atoms — bounded, a selection (Figure 4, left). Our categorical belief is a posterior over a pool of atoms that are themselves generated from data by $E_{\varphi }$: any new data materialises a new code, so the pool is unbounded and the atoms are drawn from a continuum, not enumerated in advance. A categorical belief over a data-materialised pool is thus the finite, tractable working-set representation of a belief over an unbounded generated space — the same relationship a Dirichlet-process mixture has to its infinite base measure, where any computation touches only a finite active set while the pool of possible components is unbounded (Rasmussen and Ghahramani, 2002). The unboundedness the name claims therefore does not require a continuous code at inference; it requires that codes be generated rather than stored, which the measurement channel (§ 3.2) guarantees. Selection over a stored bank is bounded; selection over a generated pool is not.

Two clarifications keep the claim honest. First, “infinite” is a statement about reachable weight configurations, not stored knowledge: knowledge remains bounded by the resident parameters (Allen-Zhu and Li, 2024), and “infinite” here never means a larger knowledge store. Second, adaptation adds no parameters; it re-allocates plasticity, since the belief’s uncertainty (§ 3.3) decides which latent directions stay plastic and which are protected, resolving the stability–plasticity trade (Dohare and others, 2024) at inference rather than freezing it. We are careful to claim only what is ours: that a layer’s weights can be made a data-dependent function rather than a stored constant is established (hypernetworks, Ha et al., 2017; dynamic layers, Babiloni et al., 2023), and MoE is itself a dynamic-weight layer with a finite index; our contribution is the specific coupling — a belief over a generated code space, written by live data and moved by online inference — not dynamic weights in the abstract.

This positioning also separates us from the neighbouring generated- and selected-expert methods along one axis, the origin of the atoms and whether the belief moves: $\mu$MoE (Oldfield et al., 2024) and discrete MoE select over a stored bank (bounded); DFC (Babiloni et al., 2023) and MoEGen (Zeng et al., 2026) generate an adapter and freeze it after one read; $\infty$-MoE (Takashiro et al., 2026) masks subsets of one fixed network; and HyperMoE (Zhao et al., 2024) generates a supplementary branch over a stored bank. None carries an online belief over a data-materialised pool, which is the coupling this paper adds.

媒体内容 · 前往原文查看

3.5 Architecture choice in this paper

The framework above is deliberately agnostic about the form of the belief and the shape of the generator. We now commit to the choices this paper evaluates: the form of the belief (categorical, § 3.5.1), and the generator and base it drives (§ 3.5.2). The alternative — a continuous-Gaussian belief with a nonlinear generator, the framework’s most expressive point — we note as a further direction at the end of this section rather than evaluate here.

3.5.1 A categorical belief over materialised codes

We instantiate the belief over $z$ as a categorical distribution over a pool of codes $\{m_1,\dots ,m_K\}$, each materialised from data by the encoder $E_{\varphi }$. The belief is $P_t\left(z\right)=\mathrm{Cat}\left({\pi }_t\right)$ with ${\pi }_t\in {\Delta }^{K-1}$, the code driving the generator is the posterior’s most probable atom (top-1) or its mean, and the online update of § 3.3 becomes recursive categorical Bayes,

${\pi }_{t,k}\propto {\pi }_{t-1,k}\cdot p({\text{obs}}_t\mid z=m_k),$

(3)

so that as the interaction proceeds the belief concentrates on the code that best explains the running evidence, and re-opens when the evidence shifts. This is the exact recursive filter of § 3.3 specialised to a categorical latent; its amortization is a learned selector that emits the posterior over the pool in a single forward pass. Concretely, the selector scores the running hidden state against each code and normalises: at generative layer $\ell$ with the layer-input activation $u_{\ell }$ as query and a learned key ${\kappa }_{\ell }\left(m_k\right)$ per code, $\pi \propto \exp ⟨u_{\ell },{\kappa }_{\ell }\left(m_k\right)⟩$. Selection is top-1 per layer, so a single generated expert is applied — not a top-$k$ mixture — which keeps the operator a genuine weight rather than an averaged one; the measurement channel of § 3.2 supplies the codes, and the selector supplies the cheap per-step inference over them. The two cadences of § 3.3 carry over directly: per-turn (B), the posterior is updated once per turn as questions accumulate over a fixed knowledge pool; per-token (C), it is updated as the sequence streams.

(The empirical study of this selector — how well the categorical posterior identifies the code that carries the answer, what signal drives it, and where in the network the routing signal lives — is the subject of § 4.3.)

3.5.2 The low-rank generator

For the generator and its base we adopt the concrete pipeline of SHINE (Liu et al., 2026) essentially unchanged, and materialise the categorical pool of § 3.5.1 by running it once per knowledge set. The base is a strong dense SwiGLU model (Shazeer, 2020, fixed in § 4.1), and the generated delta modulates its FFN projections $\{W_{\text{gate}},W_{\text{up}},W_{\text{down}}\}$ with a small rank ($r=8$) and latent dimension ($d_z=128$), the code-budget controls of the framework.

Reading data into a code.

The base is a frozen decoder-only transformer. To read context, its tokens are passed through the base with a set of $M$ learnable memory tokens appended to the sequence; these are input-independent probes, trained once and shared, that read information out of the evidence by ordinary attention. The memory tokens’ hidden states are collected from every layer, giving a memory grid $𝐦\in {ℝ}^{L\times M\times d}$. A memory-to-parameter (M2P) network then mixes this grid, and emits a flat latent code $z\in {ℝ}^P$, from which a trivial projection applies to give the LoRA parameters the base needs.

Reshaping the code into weight deltas.

For a weight $W_0\in {ℝ}^{\text{out}\times \text{in}}$ the low-rank (LoRA; Hu et al., 2022) factors $A\in {ℝ}^{\text{in}\times r}$, $B\in {ℝ}^{\text{out}\times r}$ and an optional bias $C\in {ℝ}^{\text{out}}$ are applied as

$W\left(z\right)x=W_{0}x+\left(\sqrt{s}B\right){\left(\sqrt{s}A\right)}^{\top }x+sC,$

(4)

with a fixed scale $s$ folded as $\sqrt{s}$ into each factor and $s$ into the bias. The rank is small ($r=8$), so each adapter is cheap; the memory-token count is set so the flat code $z$ has exactly the size the per-layer LoRA factors require. The delta is applied in factored form, $B\left(A^{\top }x\right)$, never materialised, so per-token cost is $𝒪\left(r\left(\text{in}+\text{out}\right)\right)$ per projection.

Why this generator, and what we change.

Two properties make this the right generator for our framework. First, it is a faithful, high-bandwidth reader: unlike a compact readout head, the memory-token/M2P path reads long evidence into a code that reconstructs per-layer adapters well enough to answer questions the base was never given (the § 4.2 result on which this paper’s data-to-weights claim rests). Second, it is deterministic and cacheable: one read per knowledge set yields a code, and that code is exactly the materialised atom $m_k$ of the categorical pool (§ 3.5.1). We take the generator, memory tokens, M2P network, and meta-LoRA frozen from a SHINE checkpoint and add only the categorical selector of § 3.5.1 on top; the sole trainable parameters introduced by this paper are the selector’s per-code key map and its query alignment, at a scale of well under a percent of the base. The code-to-weight reshape here is linear in the code, with the nonlinearity of the read concentrated in the encoder (the memory/M2P stack) rather than the code-to-weight step — a preliminary finding of ours is that a linear code-to-weight leg ties a nonlinear one at a fraction of the parameters, which is why we adopt it.

Cost.

Codes are computed once per knowledge set and cached, so at run time the only cost beyond a base forward pass is (i) the selector’s $K$ inner products per layer to update the categorical belief and (ii) applying the selected code’s factored deltas. Both are negligible relative to the base; in particular, nothing re-reads the evidence tokens at generation time. This is the concrete sense in which carrying data in weights, once compiled, is cheaper at run time than re-reading it from the prompt on every token (§ 4.2), and it is the property the dilution study of § 4.2 exploits when the evidence is too large to keep re-reading in-context.

The richer belief we do not evaluate.

The categorical form chooses among whole-code reads rather than moving within the code space, so a shift the pool does not already contain can be met only by materialising a new atom. The framework’s more expressive point (§ 3.2) instead carries a continuous-Gaussian belief over a zero-anchored code offset ${\psi }_{\ell }\sim 𝒩(0,{\Sigma }_0)$, $z_{\ell }=c_{\ell }+{\psi }_{\ell }$ (so ${\psi }_{\ell }=0$ recovers the un-adapted model), and turns the recursive update of § 3.3 into a Laplace / extended-Kalman filter whose posterior precision gates plasticity — adapting fast where it is uncertain, protecting what it is confident in, a derived analogue of elastic weight consolidation (Huszár, 2018; Chang et al., 2023; Kirkpatrick et al., 2017). This is the form in which “Bayesian” becomes load-bearing rather than decorative, but it demands a code-to-weight map smooth in $z$, the exact filter as a distillation teacher, and calibration of the amortized precision; we leave it to future work and evaluate the categorical belief here.

4 Experiments

Our experiments are set up to answer three questions, each resting on the one before and each the subject of one subsection, which together test the design promise that the infinite-parameter LLM can learn from its live interaction by writing that interaction into its weights, and go on adapting as the interaction grows. The first is whether run-time data can enter the weights and be used at all: does compiling a turn’s evidence into the generated weight let the model answer from it, with the evidence withheld from the prompt (§ 4.2)? The second arises once a session has written several pieces of data into a pool of codes — whether the model can infer which of them the current query needs, the single-step form of the belief over the code (§ 4.3). The third is whether that belief accumulates across the interaction, so the model routes better as the conversation lengthens than it would by treating each turn afresh (§ 4.4).

4.1 Setup

Base and generator.

We build on a frozen base (Qwen3-8B; Qwen Team, 2025) and a data-to-weights generator that compiles evidence into low-rank weight deltas (the pipeline of § 3.5.2); the generator is reused from prior work rather than retrained here. On top of this we add the categorical selector of § 3.5.1, which is trained lightly on a routing objective. Full training details are outside the scope of this paper.

Data and tasks.

We evaluate on five question-answering datasets spanning the axis that matters for weights-versus-prompt — how long, noisy, and multi-hop the evidence is. SQuAD (single short passage, clean) is the easy end, where the prompt is cheap and strong. MS MARCO v2.1 (a question with $\phantom{}\approx 10$ candidate passages, one marked answer-bearing) is the long, noisy, multi-passage end. Between them sit three multi-hop sets whose answers require combining several passages: HotpotQA (distractor setting: 2 gold + 8 distractor paragraphs), 2WikiMultihopQA, and MuSiQue (the hardest, built to resist single-hop shortcuts). The multi-passage sets carry per-passage gold relevance labels (is_selected in MS MARCO, supporting-fact annotations in the multi-hop sets), which give the selector experiment (§ 4.3) a routing target for free; SQuAD, having a single passage, is used only for the weights-versus-prompt comparison. Unless noted, results are over $n=150$ held-out groups, scored by answer F1 (generation) or top-1/top-3 routing accuracy (selection).

Baselines.

For weights-versus-prompt: closed-book (no evidence), in-context (evidence in the prompt), and the one-shot data-to-weights read. For selection over the code pool: random ($1/K$), BM25 and dense retrieval (bge-small, untrained) over the same candidate passages — the standard, strong way to pick the right passage — and an oracle that scores each code by the likelihood it assigns the true answer, which upper-bounds the routing signal.

4.2 Data-to-weights beats the prompt where evidence is long and multi-hop

We first reproduce the data-to-weights generator we build on (SHINE; Liu et al., 2026) on our own setup, to confirm on a validated base that run-time evidence compiled into the weights can actually be used. We then run a dilution study, new here, that probes where the single one-shot read breaks as evidence scales, and that motivates per-token dynamic adaptation.

Reproducing the base: data-to-weights versus the prompt.

Whether compiling a turn’s evidence into the code beats carrying it in the prompt depends entirely on the evidence (Table 4). On SQuAD — one short, clean passage — the prompt is the ceiling (in-context 85.3 vs data-to-weights 51.8): when the evidence is small and used once, nothing beats simply reading it. On MS MARCO — ten passages, mostly distractors — the picture inverts: data-to-weights reaches 48.0 F1 against the in-context 33.6, because the prompt now pays for length and noise while the compiled code does not. The three multi-hop sets sit on the weights-favoured side of the crossover, and are the datasets that most sharply test the claim: the answer spans several passages, so the prompt must hold them all while the code compiles them.

媒体内容 · 前往原文查看

Dataset (evidence)	Closed-book	In-context	Data-to-weights
SQuAD (1 short passage)	20.2	85.3	51.8
HotpotQA (2-hop, +distractors)	22.1	58.7	60.4
2WikiMultihopQA (multi-hop)	24.5	55.5	58.1
MuSiQue (hard multi-hop)	15.2	40.9	45.3
MS MARCO v2.1 (10 passages)	16.8	33.6	48.0

The dilution boundary.

Does a fixed-size code dilute as more evidence is packed into it? We hold the answer-bearing passage in the pool, add up to 64 distractor passages, and compare two placements: oracle (the answer passage kept at the front, so it survives) and realistic (passage order shuffled, so at inference — where the model does not know which passage carries the answer — it is as exposed as any other). We run this at two encoder context budgets, 1300 and 3000 tokens, to separate the effect from any one window size (Figure 5).

Two effects stand out, and the two budgets separate them. First, the code does saturate: even the oracle placement, with the answer passage fronted and nothing truncated, declines as the pool grows — at the 3000-token budget it falls $51.6\to 48.6\to 46.9$ F1 from 8 to 32 distractors with truncation held at 0%, so a fixed-size code genuinely loses fidelity as it is asked to carry more, independent of where the answer sits. Second, on top of saturation, the realistic placement falls further below the oracle, and why it falls further has two causes the budgets tease apart. At the small budget the answer passage is truncated out of the window as the pool overflows (at 1300 tokens, 100% of reads truncate by 32 passages and realistic F1 collapses to 27.8). Raising the budget to 3000 pushes that cliff back — but does not close the oracle–realistic gap: at 32 passages nothing is truncated (0% at 3000) and yet the realistic read still trails the oracle by $\approx$5 F1, because a buried answer passage is read less faithfully than a fronted one even when both fully fit. The three effects compound, but they divide into one about capacity and two about foregrounding. Saturation is a real cost of any single read, and bounds how much one code should be asked to hold. Truncation and burial are instead failures of which evidence the read spends its budget on, because at inference it does not know which passage carries the answer. The oracle–realistic gap — $\approx$8–20 F1 depending on budget — is the value on the table for a mechanism that can identify the right evidence rather than commit to one fixed read, and the saturation curve is the reason not to answer that by simply reading more into one code. This motivates carrying a belief over a pool of pre-encoded codes and sharpening it dynamically (§§ 4.3 and 4.4): each code reads one bounded passage in-window offline, small enough to stay clear of saturation. The question is then no longer what fits, or sits first, in one read but which code the belief selects and, across a session, how that selection improves as evidence accumulates.

媒体内容 · 前往原文查看

4.3 A trained belief over the code pool beats retrieval

Given one pre-encoded code per candidate passage, we ask whether a belief over the pool can route a question to the code carrying its answer. The routing signal is real but not free (Table 5): on MS MARCO, an oracle that scores each code by the likelihood it assigns the true answer routes almost perfectly (78.7 top-1, 96.7 top-3), confirming the codes are strongly separable — but a zero-shot proxy that scores each code by the model’s confidence in its own answer is near-random (22.7), so the belief must be trained, not read off for free.

Trained, the activation-routed selector (a query taken from the base’s own layer activations, scored against a learned key per code, § 3.5.1) routes far above random and lexical baselines and beats dense retrieval over the same candidates on every dataset, by 8 F1 on MS MARCO (53.3 vs 45.3) and 10–12 on the multi-hop sets (e.g. 70.1 vs 58.1 on 2Wiki, 53.0 vs 40.9 on MuSiQue). Retrieval is the honest bar here — it, too, picks the right passage — so beating it establishes that a belief over the generated codes, read from the base’s own activations, carries more single-question routing signal than a strong text retriever, while operating over compiled codes rather than re-read passages. Two findings from the MS MARCO runs explain where the signal comes from: it lives in the network’s later layers (early-layer activations route near-random, late-layer ones carry almost all of it), and taking the query from a single late-layer summary (token-0 of the code, § 3.5.2) outperforms pooling all memory tokens — the routing query is the model’s own settled representation of the question, which a text retriever does not have access to. That the margin widens on the multi-hop sets is notable given top-1 routing can name only a single code where the answer spans several; even so, identifying the most-relevant code more reliably than retrieval is enough to lead, and the multi-turn accumulation of § 4.4 is where a belief spanning several codes would extend it further.

媒体内容 · 前往原文查看

Router	MS MARCO	HotpotQA	2Wiki	MuSiQue
Oracle (code-likelihood of true answer)	78.7	80.9	82.8	70.1
Random ($K\approx 10$)	10.0	10.1	12.3	10.3
Zero-shot code confidence	22.7	24.0	23.8	20.4
BM25 (lexical)	20.7	30.5	34.3	22.5
Dense retrieval (bge-small)	45.3	52.2	58.1	40.9
Trained activation-routed selector (ours)	53.3	62.1	70.1	53.0

Selection sidesteps both limits of the single read.

The dilution study (§ 4.2) showed the one-shot read degrades at scale on two counts: the code saturates as it is asked to carry more, and the answer passage is truncated or buried as the pool overflows. Selection avoids both by construction: each code is compiled offline from one bounded passage — a small in-window read that never saturates and never truncates the answer — and at query time the selector picks among the pre-computed codes without ever concatenating the pool into one over-length read. Sweeping the pool size makes the divergence concrete (Table 6): the single big read answers well while the pool is small but decays as it grows ($48.8\to 27.8$ F1 by 64 passages), whereas the selector — route to the answer-bearing code, answer with it — stays flat however large the pool grows, because each read it relies on is small and fixed. The two curves start together and separate as the pool grows; past that point, selection is the only one of the two that does not fall.

媒体内容 · 前往原文查看

Knowledge-pool size	Single big read (F1)	Selector over per-passage codes (F1)
8 passages (fits window)	48.8	48.1
16 passages	45.6	48.0
32 passages (overflows)	35.0	47.8
64 passages	27.8	47.6

4.4 Cross-turn accumulation: the belief sharpens as the conversation grows

It is shown in § 4.2 that run-time data can enter the weights and be used, beating the prompt once evidence is long and noisy; in § 4.3, a trained belief over the resulting code pool identifies the right code better than strong retrieval. This section shows that when the belief accumulates across an interaction, the model routes better as a conversation grows than any single-question router.

Over a fixed knowledge pool of $K$ codes, we run conversations rather than isolated questions. Each conversation opens with a turn that names its topic explicitly, followed by a mix of two kinds of follow-up: self-contained turns that can still be placed from their own text, and context-dependent turns (“who designed it?”, “and its height?”) whose questions are answerable only given the earlier turns. We author the conversations from the § 4.1 datasets, so we know each turn’s gold code and construct this mix deliberately, and a pre-registered ambiguity audit (dense retrieval on each turn’s text in isolation) labels which turns actually fall in each class. The accumulation claim is then reported only on the context-dependent turns.

The belief is a single persistent state over the code pool, carried across the whole conversation and updated by recursive categorical Bayes, ${\pi }_t\propto {\pi }_{t-1}^{\gamma }\cdot \mathrm{softmax}\left({\ell }_t\right),$ where ${\ell }_t$ is the per-token belief evidence and $\gamma \in \left[0,1\right]$ controls forgetting. Nothing is retrained during the conversation, and the per-token cost stays at $K$ inner products per layer, flat in both token and turn index.

We consider per-question retrieval and per-question selector (our § 4.3 router, memoryless) as baselines, and the prompt-side way of accumulating, concat-history retrieval (the running query is turns $1\dots t$). Against these, the accumulated belief (the persistent posterior above). The load-bearing comparison is against concat-history, and it turns on both accuracy and cost. On accuracy (Figure 6, context-dependent turns): as the conversation establishes its topic the posterior concentrates, so later ambiguous turns route almost as well as unambiguous ones — the accumulated belief rises with turn index while the memoryless arms stay flat and collapse on turns that are ambiguous alone, and concat-history rises then sags as its growing query dilutes. On cost, the two accumulating routes differ in kind: concat-history’s per-turn cost grows with the turn index as the query lengthens, whereas the belief’s stays flat — $K$ inner products per layer, independent of turn (as above). Beating concat-history on accuracy while holding cost flat is the claim: the belief accumulates session state better and more cheaply than re-reading the growing history into the prompt.

媒体内容 · 前往原文查看

5 Limitations

The clearest limitation is a boundary the design lives within: a compact generator does not carry a large MoE’s stored knowledge, because knowledge is bounded by parameters (Allen-Zhu and Li, 2024) and generation does not move that bound — a generator the size of a small model can no more hold a large model’s facts than that small model could, and closed-book ability, unlike perplexity, is bounded by exactly this. The design answers this by compiling knowledge and behaviour from run-time data rather than storing it in weights, which shifts the burden onto the data being supplied: where the relevant facts or instructions are not provided, the model has only its base’s knowledge and default behaviour. This is why the comparison is weights-versus-prompt; on closed-book knowledge with nothing supplied, a large stored model is simply the wrong thing to measure against. The prompt is the sharpest competitor. Putting the data in the context is a strong, cheap baseline whenever the context is short and used once, so the advantage of compiling it into weights is specific to large or repeatedly-reused data and long horizons (§ 4.2), not universal. Per-token generation adds a bandwidth cost that must be controlled through a small generator and low-rank deltas. Weight generators risk memorising their training distribution rather than generalising to new data (Zeng et al., 2025); our codes are read from held-out evidence at test time, but a systematic generalisation study across unseen knowledge pools remains future work. Dropping a stored bank in favour of a generated code also changes what can go wrong with routing: there is no load-balancing loss, but a trained selector could over-concentrate on a few codes, which a light coverage regulariser on the selector guards against. Three assumptions in the adaptation model bear watching. The belief this paper evaluates is categorical over a pool of materialised codes, which chooses among reads rather than moving within the code space; a shift the pool does not contain can be met only by materialising a new code, and the richer continuous-Gaussian belief that would move within the space is left to future work (§ 3.5), where its added assumptions — a code-to-weight map smooth in $z$, and an amortized posterior whose precision stays calibrated out of distribution (Sun et al., 2024; Behrouz et al., 2025) — must be validated directly. The true posterior over which code a context implies may also be multimodal (Xie et al., 2022), which a single top-1 selection collapses. And while the central claim — that the belief accumulates usefully across a conversation (§ 4.4) — is now demonstrated on authored multi-turn conversations, it is shown at the categorical, top-1 point of the framework and over pools the conversations were built from; the forgetting control $\gamma$ (§ 3.3), longer horizons, and naturally-occurring rather than authored sessions are where it must be stress-tested next.

6 Conclusion

We have described an architecture in which a language model’s experts are neither stored nor selected from a fixed bank but generated from live data over a shared base, and a belief over the generating code that is carried and updated as the interaction proceeds. The motivating idea is a change in where run-time knowledge and behaviour are carried: today they live in the prompt, re-read on every request and forgotten after; we compile them into the weights instead. Mixture-of-Experts supplied the starting point, its per-token dynamic weights. We made a shared base FFN’s weights dynamic through a generated low-rank additive delta (§ 3.2), set out the belief-over-code framework and its cadences (§ 3.3), and positioned discrete MoE, $\infty$-MoE, $\mu$MoE, DFC, and the one-shot weight generators by the axis on which each departs from that structure. This paper realises the framework at its categorical point — a belief over a pool of data-materialised codes, selected top-1 and sharpened online (§ 3.5) — leaving the richer continuous-Gaussian belief to future work. What the design offers is a different bargain, weights instead of prompt for the knowledge and behaviour supplied at run time, which is amortized in compute, frees the context window, persists across turns, and adapts as the session proceeds. The sense in which the model has an unbounded, “infinite” space of parameters is precise and narrow: unbounded reachable effective weights and behaviours, compiled from live data, from a fixed footprint. Our experiments confirm that run-time data compiled into the weights can be used and, on long, noisy evidence, beats the prompt; that a trained belief over the code pool identifies the right code at least as well as strong retrieval; and that this belief, accumulated across a conversation, routes better as the session grows than any single read or a re-read of the growing history (§ 4.4). While concurrent work independently pursues generating rather than storing experts, and reads context into weights in a single pass, the coupling proposed here, an online-updated belief over the low-dimensional latent code of a shared-base generative expert space, driven by live data, is, to our knowledge, unclaimed in prior work.

References

• Abdalla et al. (2025) M. H. I. Abdalla, Z. Wang, C. Frey, S. Eger, and J. Grabocka Zhyper: factorized hypernetworks for conditioned LLM fine-tuning. Note: arXiv preprintarXiv:2510.19733 External Links: Link Cited by: §2.3, Table 1.
• Abnar et al. (2025) S. Abnar et al. Parameters vs. FLOPs: scaling laws for optimal sparsity for mixture-of-experts language models. Note: arXiv preprintarXiv:2501.12370 External Links: Link Cited by: §2.1.
• Aghajanyan et al. (2021) A. Aghajanyan, L. Zettlemoyer, and S. Gupta Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §1.
• Akyürek et al. (2024) E. Akyürek, M. Damani, A. Zweiger, L. Qiu, H. Guo, J. Pari, Y. Kim, and J. Andreas The surprising effectiveness of test-time training for few-shot learning. Note: arXiv preprintarXiv:2411.07279 External Links: Link Cited by: §2.4.
• Allen-Zhu and Li (2024) Z. Allen-Zhu and Y. Li Physics of language models: part 3.3, knowledge capacity scaling laws. In International Conference on Learning Representations (ICLR) 2025, External Links: Link Cited by: §1, §3.4, §5.
• Ba et al. (2016) J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.4.
• Babiloni et al. (2023) F. Babiloni, T. Tanay, J. Deng, M. Maggioni, and S. Zafeiriou Factorized dynamic fully-connected layers for neural networks. In IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Cited by: §2.3, Table 1, §3.4, §3.4.
• Behrouz et al. (2025) A. Behrouz, P. Zhong, and V. Mirrokni Titans: learning to memorize at test time. Note: arXiv preprintarXiv:2501.00663 External Links: Link Cited by: §2.4, Table 2, §3.3, §3.3, §5.
• Bengio et al. (2015) E. Bengio, P.-L. Bacon, J. Pineau, and D. Precup Conditional computation in neural networks for faster models. Note: arXiv preprintarXiv:1511.06297 External Links: Link Cited by: §2.1.
• Bengio et al. (2013) Y. Bengio, N. Léonard, and A. Courville Estimating or propagating gradients through stochastic neurons for conditional computation. Note: arXiv preprintarXiv:1308.3432 External Links: Link Cited by: §1, §2.1.
• Berges et al. (2024) V.-P. Berges, B. Oğuz, D. Haziza, W. Yih, L. Zettlemoyer, and G. Ghosh Memory layers at scale. In International Conference on Learning Representations (ICLR) 2025, External Links: Link Cited by: §2.1.
• Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.6.
• Buehler and Buehler (2024) E. L. Buehler and M. J. Buehler X-LoRA: mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning. External Links: Link Cited by: §2.2.
• Chang et al. (2023) P. G. Chang, G. Durán-Martín, A. Y. Shestopaloff, M. Jones, and K. Murphy Low-rank extended kalman filtering for online learning of neural networks from streaming data. In Conference on Lifelong Learning Agents (CoLLAs), PMLR 232, External Links: Link Cited by: §2.5, §3.5.2.
• Charakorn et al. (2025) R. Charakorn, E. Cetin, Y. Tang, and R. T. Lange Text-to-LoRA: instant transformer adaption. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §1, §1, §2.3, Table 1, §3.2.
• Charakorn et al. (2026) R. Charakorn, E. Cetin, S. Uesaka, and R. T. Lange Doc-to-LoRA: learning to instantly internalize contexts. Note: arXiv preprintarXiv:2602.15902 External Links: Link Cited by: §2.3, Table 1.
• Chen et al. (2025) I.-C. Chen, H.-S. Liu, W.-F. Sun, C.-H. Chao, Y.-C. Hsu, and C.-Y. Lee Retraining-free merging of sparse mixture-of-experts via hierarchical clustering. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.2.
• Clark et al. (2022) A. Clark et al. Unified scaling laws for routed language models. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.1.
• Dai et al. (2024) D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, et al. DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §2.1.
• DeepSeek-AI (2024) DeepSeek-AI DeepSeek-V3 technical report. Note: arXiv preprintarXiv:2412.19437 External Links: Link Cited by: §1.
• Dhankhar et al. (2026) N. Dhankhar, D. Baha, and A. Saparov Scaling laws for hypernetwork-based knowledge injection in large language models. Note: arXiv preprintarXiv:2607.19604 External Links: Link Cited by: §1, §2.3, Table 1, §3.2.
• Dialameh et al. (2025) M. Dialameh, H. Rajabzadeh, W. Zhang, W. Ahmed, and H. J. Kwon Bayesian mixture of experts for large language models. Note: arXiv preprintarXiv:2511.08968 External Links: Link Cited by: §2.5.
• Dohare et al. (2024) S. Dohare et al. Loss of plasticity in deep continual learning. Nature 632, pp. 768–774. Cited by: §2.5, §3.4.
• Dou et al. (2024) S. Dou, E. Zhou, Y. Liu, S. Gao, J. Zhao, W. Shen, et al. LoRAMoE: alleviate world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §2.2.
• Eliseev and Mazur (2023) A. Eliseev and D. Mazur Fast inference of mixture-of-experts language models with offloading. Note: arXiv preprintarXiv:2312.17238 External Links: Link Cited by: §2.2.
• Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch Transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research. External Links: Link Cited by: §2.1.
• Fraccaro et al. (2017) M. Fraccaro, S. Kamronn, U. Paquet, and O. Winther A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.5, §3.3.
• Frantar and Alistarh (2023) E. Frantar and D. Alistarh QMoE: practical sub-1-bit compression of trillion-parameter models. In Proceedings of Machine Learning and Systems (MLSys) 2024, External Links: Link Cited by: §2.2.
• Griffiths and Ghahramani (2011) T. L. Griffiths and Z. Ghahramani The indian buffet process: an introduction and review. Journal of Machine Learning Research 12, pp. 1185–1224. Cited by: §2.5.
• Ha et al. (2017) D. Ha, A. Dai, and Q. V. Le HyperNetworks. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.3, §3.4.
• He et al. (2023) S. He, R.-Z. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao Merging experts into one: improving computational efficiency of mixture of experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §2.2.
• He (2024) X. O. He Mixture of a million experts. Note: arXiv preprintarXiv:2407.04153 External Links: Link Cited by: §2.1.
• Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. Note: arXiv preprintarXiv:1503.02531 External Links: Link Cited by: §2.7.
• Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1, §2.1.
• Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §3.5.2.
• Huszár (2018) F. Huszár Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences (PNAS) 115 (11). Cited by: §3.5.2.
• Imam et al. (2026) R. Imam, D. Rashid, Y. Xie, D. Mahapatra, B. Lall, and M. Yaqub Can experts adapt without training? on test-time modality generalization in MVLMs. In Medical Image Computing and Computer Assisted Intervention (MICCAI), External Links: Link Cited by: §2.4, Table 2.
• Jacobs et al. (1991) R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton Adaptive mixtures of local experts. Neural Computation 3 (1), pp. 79–87. Cited by: §2.1, §2.5.
• Jiang et al. (2024) A. Q. Jiang et al. Mixtral of experts. Note: arXiv preprintarXiv:2401.04088 External Links: Link Cited by: §1.
• Jordan and Jacobs (1994) M. I. Jordan and R. A. Jacobs Hierarchical mixtures of experts and the EM algorithm. Neural Computation 6 (2), pp. 181–214. Cited by: §2.1, §2.5.
• Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, et al. Scaling laws for neural language models. Note: arXiv preprintarXiv:2001.08361 External Links: Link Cited by: §1, §2.1.
• Karl et al. (2017) M. Karl, M. Soelch, J. Bayer, and P. van der Smagt Deep variational bayes filters: unsupervised learning of state space models from raw data. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.5, §3.3.
• Khandelwal et al. (2020) U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis Generalization through memorization: nearest neighbor language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.6.
• Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS) 114 (13), pp. 3521–3526. External Links: Link Cited by: §2.5, §2.5, §3.3, §3.5.2.
• Komatsuzaki et al. (2023) A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. Riquelme Ruiz, B. Mustafa, J. Ainslie, et al. Sparse upcycling: training mixture-of-experts from dense checkpoints. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.1.
• Krajewski et al. (2024) J. Krajewski, J. Ludziejewski, K. Adamczewski, M. Pióro, M. Krutul, S. Antoniak, et al. Scaling laws for fine-grained mixture of experts. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.1.
• Krishnan et al. (2015) R. G. Krishnan, U. Shalit, and D. Sontag Deep kalman filters. Note: arXiv preprintarXiv:1511.05121 External Links: Link Cited by: §2.5, §3.3.
• Krishnan et al. (2017) R. G. Krishnan, U. Shalit, and D. Sontag Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), External Links: Link Cited by: §2.5.
• Lample et al. (2019) G. Lample, A. Sablayrolles, M. Ranzato, L. Denoyer, and H. Jégou Large memory layers with product keys. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.1.
• Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §2.6.
• Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.6.
• Li et al. (2024) D. Li, Y. Ma, N. Wang, Z. Ye, Z. Cheng, Y. Tang, et al. MixLoRA: enhancing large language models fine-tuning with lora-based mixture of experts. Note: arXiv preprintarXiv:2404.15159 External Links: Link Cited by: §2.2.
• Li and Hoiem (2017) Z. Li and D. Hoiem Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 40 (12), pp. 2935–2947. External Links: Link Cited by: §2.5.
• Liang et al. (2025) Z. Liang, D. Tang, Y. Zhou, X. Zhao, M. Shi, W. Zhao, et al. Drag-and-drop LLMs: zero-shot prompt-to-weights. Note: arXiv preprintarXiv:2506.16406 External Links: Link Cited by: §2.3.
• Liu et al. (2022) H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.3.
• Liu et al. (2024) J. Liu, P. Tang, W. Wang, Y. Ren, X. Hou, P.-A. Heng, M. Guo, and C. Li A survey on inference optimization techniques for mixture of experts models. ACM Computing Surveys. External Links: Link Cited by: §2.2.
• Liu et al. (2026) Y. Liu, X. Wang, Y. Mao, Y. Gelberg, H. Maron, et al. SHINE: a scalable in-context hypernetwork for mapping context to LoRA in a single pass. Note: arXiv preprintarXiv:2602.06358 External Links: Link Cited by: §2.3, Table 1, §3.2, §3.5.2, §4.2.
• Lopez-Paz and Ranzato (2017) D. Lopez-Paz and M. Ranzato Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.5.
• Lu et al. (2024) X. Lu, Q. Liu, Y. Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li Not all experts are equal: efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §2.2.
• Marino et al. (2018) J. Marino, Y. Yue, and S. Mandt Iterative amortized inference. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §3.3.
• McCloskey and Cohen (1989) M. McCloskey and N. J. Cohen Catastrophic interference in connectionist networks: the sequential learning problem. Psychology of Learning and Motivation 24, pp. 109–165. Cited by: §2.5.
• Muqeeth et al. (2024) M. Muqeeth, H. Liu, and C. Raffel Soft merging of experts with adaptive routing. Transactions on Machine Learning Research (TMLR). External Links: Link Cited by: §2.1.
• Nguyen et al. (2018) C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner Variational continual learning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.5.
• Nguyen et al. (2023) H. Nguyen, T. Nguyen, and N. Ho Demystifying softmax gating function in gaussian mixture of experts. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.5.
• Oldfield et al. (2024) J. Oldfield, M. Georgopoulos, G. G. Chrysos, C. Tzelepis, Y. Panagakis, M. A. Nicolaou, J. Deng, and I. Patras Multilinear mixture of experts: scalable expert specialization through factorization. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.1, §2.2, §3.4.
• Perez et al. (2018) E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville FiLM: visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), External Links: Link Cited by: §2.3.
• Phang et al. (2023) J. Phang, Y. Mao, P. He, and W. Chen HyperTuning: toward adapting large language models without back-propagation. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.3, Table 1.
• Putzky and Welling (2017) P. Putzky and M. Welling Recurrent inference machines for solving inverse problems. Note: arXiv preprintarXiv:1706.04008 External Links: Link Cited by: §3.3.
• Qu et al. (2022) J. Qu, T. Faney, Z. Wang, P. Gallinari, S. Yousef, and J.-C. de Hemptinne HMOE: hypernetwork-based mixture of experts for domain generalization. Note: arXiv preprintarXiv:2211.08253 External Links: Link Cited by: §2.3, Table 1.
• Qwen Team (2025) Qwen Team Qwen3 technical report. Note: arXiv preprintarXiv:2505.09388 External Links: Link Cited by: §1, §4.1.
• Rasmussen and Ghahramani (2002) C. E. Rasmussen and Z. Ghahramani Infinite mixtures of gaussian process experts. In Advances in Neural Information Processing Systems (NeurIPS), pp. 881–888. Cited by: §2.5, §3.4.
• Ritter et al. (2018) H. Ritter, A. Botev, and D. Barber Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.5.
• Rusu et al. (2016) A. A. Rusu, N. C. Rabinowitz, G. Desjardins, et al. Progressive neural networks. Note: arXiv preprintarXiv:1606.04671 External Links: Link Cited by: §2.5.
• Schlag et al. (2021) I. Schlag, K. Irie, and J. Schmidhuber Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.4.
• Schmidhuber (1992) J. Schmidhuber Learning to control fast-weight memories: an alternative to dynamic recurrent networks. Neural Computation 4 (1), pp. 131–139. Cited by: §2.4.
• Schwarz et al. (2018) J. Schwarz et al. Progress & compress: a scalable framework for continual learning. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.5.
• Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §2.1.
• Shazeer (2020) N. Shazeer GLU variants improve transformer. Note: arXiv preprintarXiv:2002.05202 External Links: Link Cited by: §3.5.2.
• Snell et al. (2024) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling LLM test-time compute optimally can be more effective than scaling model parameters. Note: arXiv preprintarXiv:2408.03314 External Links: Link Cited by: §2.4.
• Su et al. (2025) G. Su, Y. Yang, L. Shen, L. Yin, S. Liu, and J. Geiping Rewiring experts on the fly: continuous rerouting for better online adaptation in mixture-of-expert models. In International Conference on Machine Learning (ICML) 2026, External Links: Link Cited by: §2.4, Table 2.
• Sun et al. (2024) Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, et al. Learning to (learn at test time): RNNs with expressive hidden states. In International Conference on Machine Learning (ICML) 2025, External Links: Link Cited by: §2.4, Table 2, §3.3, §3.3, §3.3, §5.
• Takashiro et al. (2026) S. Takashiro, T. Kojima, S. Taniguchi, Y. Iwasawa, and Y. Matsuo $\infty$-MoE: generalizing mixture of experts to infinite experts. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), External Links: Link Cited by: §2.1, §3.4.
• van de Ven and Tolias (2019) G. M. van de Ven and A. S. Tolias Three scenarios for continual learning. Note: arXiv preprintarXiv:1904.07734 External Links: Link Cited by: §2.5.
• Villalobos et al. (2024) P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, and M. Hobbhahn Will we run out of data? limits of LLM scaling based on human-generated data. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §1.
• Wang et al. (2023) X. Wang, T. Chen, Q. Ge, et al. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, External Links: Link Cited by: §2.5.
• Wang et al. (2024) Y. Wang, H. Shi, L. Han, D. Metaxas, and H. Wang BLoB: bayesian low-rank adaptation by backpropagation for large language models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.5.
• Waterhouse et al. (1996) S. R. Waterhouse, D. J. C. MacKay, and A. J. Robinson Bayesian methods for mixtures of experts. In Advances in Neural Information Processing Systems (NeurIPS), pp. 351–357. Cited by: §2.5.
• Wu et al. (2024) T. Wu et al. Continual learning for large language models: a survey. Note: arXiv preprintarXiv:2402.01364 External Links: Link Cited by: §2.5.
• Wu et al. (2024) X. Wu, S. Huang, and F. Wei Mixture of LoRA experts. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.2.
• Xia et al. (2025) Z. Xia, A. Davtyan, and P. Favaro KOALA++: efficient kalman-based optimization with gradient-covariance products. Note: arXiv preprintarXiv:2506.04432 External Links: Link Cited by: §2.5.
• Xie et al. (2022) S. M. Xie, A. Raghunathan, P. Liang, and T. Ma An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.6, 1st item, §5.
• Xue et al. (2022) F. Xue, X. He, X. Ren, Y. Lou, and Y. You One student knows all experts know: from sparse to dense. Note: arXiv preprintarXiv:2201.10890 External Links: Link Cited by: §2.2.
• Yang et al. (2024) A. X. Yang, M. Robeyns, X. Wang, and L. Aitchison Bayesian low-rank adaptation for large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.5.
• Zadouri et al. (2023) T. Zadouri, A. Üstün, A. Ahmadian, B. Ermiş, A. Locatelli, and S. Hooker Pushing mixture of experts to the limit: extremely parameter efficient MoE for instruction tuning. Note: arXiv preprintarXiv:2309.05444 External Links: Link Cited by: §2.2.
• Zeng et al. (2025) B. Zeng, Y. Yin, Z. Xu, and Z. Liu Generative modeling of weights: generalization or memorization?. Note: arXiv preprintarXiv:2506.07998 External Links: Link Cited by: §2.3, §5.
• Zeng et al. (2026) Y. Zeng, L. Lu, Z. Li, Z. Li, S. Li, S. Liao, et al. MoEGen: mixture-of-experts for instance-adaptive LoRA generation. Note: arXiv preprintarXiv:2608.03275 External Links: Link Cited by: §2.3, §2.7, Table 1, §3.4.
• Zhao et al. (2024) H. Zhao, Z. Qiu, H. Wu, Z. Wang, Z. He, and J. Fu HyperMoE: towards better mixture of experts via transferring among experts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), External Links: Link Cited by: §2.3, Table 1, §3.4.
• Zhong et al. (2024) Z. Zhong, M. Xia, D. Chen, and M. Lewis Lory: fully differentiable mixture-of-experts for autoregressive language model pre-training. In Conference on Language Modeling (COLM), External Links: Link Cited by: §2.1.
• Zweiger et al. (2025) A. Zweiger, J. Pari, H. Guo, E. Akyürek, Y. Kim, and P. Agrawal Self-adapting language models. Note: arXiv preprintarXiv:2506.10943 External Links: Link Cited by: §2.4, Table 2.