~/shanegraffiti.com/research/dmoe Shane Graffiti Inc. Semantic Adversarial Research Division 2026

DECOUPLED MIXTURE OF EXPERTS FOR PARAMETRIC KNOWLEDGE

§ 1.0

The Architectural Bottleneck

LLM parametric knowledge is frozen at pre-training. Domain-specific facts go stale. Time-sensitive queries hallucinate. The two dominant responses to this both fail at the architecture level not the implementation level.

Paradigm 01

RAG

Keeps knowledge in an external corpus, retrieved at inference and appended as prompt context. Easy to update just modify the index. But knowledge never enters parameter space. It's exposed to the model as input tokens, not internalized as weights. Inference cost scales with retrieval and long-context attention recomputation on every query.

Shallow   prompt-level only

→

Paradigm 02

Post-Training

SFT or LoRA writes new knowledge directly into shared parameters. Deep integration knowledge becomes part of the model's weights. But those weights already encode diverse pre-trained knowledge. Continuous updates cause catastrophic forgetting, knowledge conflict, and require full retraining as the corpus changes.

Brittle   shared parameter space

→

Paradigm 03

DMoE

Decouples both experts and router from the base model. Each knowledge unit becomes an independent LoRA adapter stored externally. A lightweight router activates relevant experts only when uncertainty signals insufficient knowledge. Base model parameters never change. KV-cache remains valid throughout generation.

Modular   independently updatable

§ 2.0

Architecture Three Components

DMoE has three parts: a frozen base model, a decoupled router, and a bank of decoupled expert modules. None of the three are jointly trained. The router and experts are built after the base model is fixed, and they can be added, removed, or replaced without touching the backbone.

Frozen Base Model+

The backbone LLM is never modified. Its parameters θ remain fixed throughout expert construction, router training, and inference. This is the structural guarantee that prevents knowledge conflict and catastrophic forgetting.

Expert Modules+

Each knowledge unit K_i is converted into a LoRA adapter Δθ_i trained on instruction-style augmentations of that document. Rank 4, α = 16. One expert per passage. 27,613 experts in the full corpus bank. Each expert is ~481 KiB on disk, stored externally, loaded on demand.

Uncertainty-Aware Router+

At each decoding step t, Token Uncertainty (TU) = entropy of the softmax distribution over vocabulary. If TU > τ (default 2.0), routing fires. The router is a BM25 inverted index over expert text surrogates training-free, incrementally updatable, no neural encoder needed.

Final-Layer Attachment+

Experts attach only at the final transformer layer's FFN. This is the only placement that doesn't break KV-cache compatibility. Any earlier layer modifies hidden states that feed subsequent attention blocks, requiring KV recomputation. The last FFN output goes directly to logits it doesn't affect cached keys or values.

        Effective Parameters   Per-Step Composition
        θefft = θ + ΣEᵢ ∈ Esel Δθᵢ
        // θ: frozen base model   never changed
        // Δθᵢ: selected expert adapters   loaded transiently
        // θefft: temporary composition for this decoding state only
        // Esel = Top-k BM25(qt, Dᵢ)   k=3 by default
      

§ 3.0

Token Uncertainty as the Trigger Signal

The router solves two subproblems independently: when to activate experts, and which experts to activate. Token Uncertainty (TU) handles the first. BM25 handles the second. They are not jointly trained and ablations confirm both matter.

01 Compute Token Uncertainty

02 Triggering Decision

03 BM25 Expert Selection

04 Robustness Verified

Step 01 Compute Token Uncertainty

At each decoding step t: TU_t = −Σ_v p_t(v) log p_t(v). The entropy of the softmax distribution over vocabulary. High entropy = flat distribution = the model is uncertain about its next token = it likely lacks relevant knowledge. Low entropy = confident = no intervention needed.

Step 02 Triggering Decision

Trigger_t = 𝟙[TU_t > τ]. Default τ = 2.0. If triggered: deactivate current experts, run BM25 retrieval, load new experts. If not triggered: continue decoding with existing model state. The active expert set persists until the next triggering event avoiding redundant expert swaps at every token.

Step 03 BM25 Expert Selection

Routing query q_t is constructed from the task input plus the generated prefix up to (but excluding) the triggering position so retrieved experts are conditioned on exactly the context already in the KV cache. E_sel = Top-k BM25(q_t, D_i). Lexical, training-free, incrementally updatable: adding an expert just inserts D_i into the inverted index.

Step 04 Robustness Verified

Ablation Table 6 (paper): Full TU+BM25 achieves EM=0.1800, F1=0.2553. Random trigger + BM25 drops to 0.1667. TU + random router drops to 0.1767. Always trigger collapses to 0.1200. Both trigger precision and expert selection contribute and TU monotonically aligns with knowledge difficulty across entropy bins (Table 5).

§ 4.0

Why Final-Layer FFN Is Non-Negotiable

This is the key engineering insight. Placing experts at any intermediate layer breaks KV-cache compatibility. The proof is mechanical and the empirical table confirms it.

        The Cache Compatibility Constraint
        Standard: hℓ+1 = Fℓ(hℓ)
        // KV cache stores Kcachedj = WKhj for all j ≤ L
        With expert at layer k: h̃k+1 = hk+1 + Δ(hk)
        // This propagates forward: all h̃j ≠ hj for j > k
        // Cached KV for those layers is now invalid   must recompute
        Final layer only: ∀j ≤ L, h̃j = hj
        // Last FFN output → logits only, never feeds attention again
        // All cached KV states remain valid. Zero recomputation overhead.
      

Table 3 (paper) confirms this empirically. Inserting experts into the final 25%, 50%, 75%, or 100% of FFN layers consistently underperforms the only-last configuration. Only attaching to the single last layer achieves best overall accuracy while preserving cache compatibility. The performance gap isn't marginal earlier placement actively degrades quality because KV mismatch corrupts the attention context.

§ 5.0

Results Four Benchmarks, Two Models

Evaluated against Basic-RAG, FLARE, PRAG, and SFT-LoRA on HotpotQA (multi-hop), ComplexWebQuestions (open-domain compositional), Quasar-T (open-domain trivia), and StrategyQA (implicit multi-hop). All methods share the same 27,613-passage Wikipedia corpus, same BM25 retriever, same retrieval budget (k=3), greedy decoding.

11/14

Best or Tied-Best Metrics

3×

Faster Than FLARE

1.9×

Less GPU Memory vs FLARE

7.2×

Less Memory than OLMoE

7.5×

Faster Than OLMoE SFT-LoRA

EM Score   Llama-3.2-1B Base Model

Method	CWQ EM	CWQ F1	HotpotQA EM	Quasar-T EM	StrategyQA ACC
DMoE	0.2467	0.3479	0.1800	0.3133	0.5667
PRAG	0.2500	0.3284	0.0733	0.2200	0.5600
FLARE	0.2400	0.3154	0.0733	0.1867	0.5367
SFT-LoRA	0.2167	0.3092	0.0767	0.2133	0.5533
Basic-RAG	0.1633	0.2384	0.1700	0.2800	0.4333

Efficiency   Average Per-Sample Inference

Method	Time (s)	GPU Memory (GB)
FLARE	9.2643	13.97
DMoE	2.6656	7.24
SFT-LoRA	1.6700	4.82
PRAG	1.3600	4.83
Basic-RAG	1.8900	2.54

§ 6.0

The Expert Bank Storage, Not VRAM

The conventional MoE bottleneck is that all experts must reside in GPU memory simultaneously. DMoE inverts this: experts live on disk, only the top-k selected experts are loaded per triggering event. Bank size scales disk footprint not inference VRAM.

Per-Expert Footprint+

LoRA rank 4, α=16: 122,880 trainable parameters per expert. Approximately 481 KiB per expert on disk. The full 27,613-expert corpus bank occupies 13.08 GiB of disk storage no GPU memory pressure from inactive experts.

Training Cost+

One new passage-level expert: ~10 seconds on a single A100. Training is embarrassingly parallel because experts are independent. The 27,613-expert bank corresponds to ~76.7 A100 GPU-hours total, distributable across any number of GPUs.

Update Model+

Adding knowledge: train new expert, insert its text surrogate into the BM25 index. Removing or updating knowledge: delete the adapter directory, remove from index. The backbone and all unrelated experts are untouched. No re-training, no re-merging.

Bank Size Robustness+

Table 9 (paper): reducing the bank to 1/10 or 1/5 of full size shows near-identical downstream performance. HotpotQA slightly improves at 1/10. DMoE does not require a precisely tuned expert count the router compensates with BM25 precision over whatever bank size is available.

No re-training, no re-merging.

Update Model

        $ query: what does DMoE change
        // Knowledge doesn't have to live in prompt or in shared weights.
        // It can live in an external bank of isolated, updatable adapters.
        // The base model never changes. The KV-cache never breaks.
        $ query: what does this cost
        // 10 seconds per expert on one A100. Embarrassingly parallel.
        // 481 KiB per expert on disk. Active VRAM: top-k experts only.
        // Router is BM25. No neural encoder. Training-free. Updatable.
        $ query: what is the actual result
        // Best on 11 of 14 effectiveness metrics across 4 benchmarks.
        // 3× faster than FLARE. 7.5× faster than coupled MoE baseline.
        // Decoupled. Cache-safe. Modular. Independently updatable.
      