Loop in L2: int3 MoE beats full precision, at no extra memory
int3 MoE beats full-precision MoE.
Run each MoE layer twice while its weights are still in the GPU's L2 cache. With int3 experts, the model beats a full-precision standard MoE of the same size, and reads only 22% of its expert bytes per token.
| result (8 layers, d768, 670M params, 524M tokens) | measured |
|---|---|
| 4.0% lower perplexity | at equal parameters (loss −0.041; standard MoE seed spread 0.007) |
| ≥ 2× wider experts | slightly better than a standard MoE with 2× wider experts (−0.037), with half its expert parameters |
| int3 > full precision | int3 version 0.024 below the full-precision standard MoE (2.4% lower perplexity) |
| 22% of the expert bytes | int3 vs bf16 expert weights read per token |
| 2nd pass hits L2 | RTX 5080: pass-2 weight reads drop to the L2-hot floor; optimised pass 2 costs 0.47× a fresh layer (Qwen3-30B-A3B layer shape, int4) |
| smaller model (6 layers, d512) | same picture, larger gaps: −6.0% / int3 −5.4% |
The catch: repeat attention, not just the experts. Re-running the same experts alone gives almost nothing (−0.010 at 8 layers, 3 seeds). The gain comes from a second round of token mixing, so the second pass must include attention.
The architecture, in four pieces.
- Whole-layer in-place repeat. Every MoE layer runs twice: attention, then the experts, then both again.
- Locked routing. The second pass reuses the first pass's experts, so it reads the same weights that are still in L2.
- int3 experts. They make a whole layer small enough to stay in L2.
- Re-injection. The second pass mixes the layer input back in and adds onto the residual stream, which keeps training stable.
Why it matters. At batch size 1, a GPU spends most of each token waiting for weights to arrive from VRAM. MoEs have huge total parameter counts, so they hit this wall hardest. This architecture spends the idle compute on a second pass over weights that are already on chip, and turns that into quality you can then trade for fewer bytes.

Loss gap to a standard MoE of the same size, through training, at two model sizes. Whole-layer repeat (purple) pulls below the standard MoE early and stays far outside its seed range to the end: −0.062 at 6 layers, −0.040 at 8 layers, where it also edges past a standard MoE with 2× wider experts (grey dashed). The int3 versions (lighter purple) stay below the full-precision baseline. Repeating only the experts (orange) gains about a quarter as much.
What is new. We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width × loop count at equal weight bytes (two seeds, int3–int8, 1–6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.
L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.
Scaling: what holds and what does not

Final val loss of whole-layer repeat minus a standard MoE of the same size, same data, steps and learning rate.
| model | training tokens | whole-layer repeat − standard MoE | int3 version − full-precision standard MoE |
|---|---|---|---|
| 6 layers, d512 (184M) | 101M | −0.063 / −0.077 (2 seeds) | −0.056 / −0.051 (2 seeds) |
| 6 layers, d512 | 303M (3×) | −0.066 | — |
| 8 layers, d768 (670M) | 524M | −0.041 | −0.024 |
| 12 layers, d1024 (1.5B) | 524M | −0.009 (A100) / −0.015 (GB10) | not run |
| 12 layers, d512 | 303M | −0.027 all layers (zero-init −0.025); −0.038 / −0.038 first half looped (2 seeds) | −0.020 (first half looped, int3) |
| 24 layers, d512 | 303M | −0.018 first half looped; −0.014 first quarter | not run |
| 24 layers, d512 | 909M (3×) | −0.027 / −0.029 first half looped (2 seeds, vs mean of 2 standard seeds) | not run |
-
More training does not erase the gain. At 6 layers, tripling the tokens (each run trained to completion) keeps it at −0.066.
-
Deeper models need more training, not a different idea. At a fixed budget of 303M tokens the gain falls with depth (6 → 12 → 24 layers: −0.066 → −0.038 → −0.018). Tripling the tokens at 24 layers brings it back to −0.028 (two seeds, −0.027 and −0.029; every looped seed beats every standard seed; the looped seeds agree to 0.002, the standard ones to 0.006). Deep looped models start slower and keep pulling ahead for longer; the gap was still widening at the end of training.
At the same width and token budget, going from 6 to 12 layers cuts it from −0.066 to −0.027 (still growing at the end of training), so depth is the main factor. Zero-initialising the second pass does not fix it (−0.025).
- Depth-aware recipe: repeat only the early layers. In the 12-layer model (same width and tokens), repeating only some layers works better than repeating all of them:
| which layers run twice (12 layers, d512, 303M tokens) | extra layer passes | val loss | − standard MoE | perplexity |
|---|---|---|---|---|
| all 12 | 12 | 3.5810 | −0.027 | −2.7% |
| first 6 | 6 | 3.5699 | −0.038 | −3.8% |
| every other layer | 6 | 3.5776 | −0.031 | −3.0% |
| middle 6 | 6 | 3.5932 | −0.015 | −1.5% |
| last 6 | 6 | 3.6135 | +0.005 | +0.5% |
The second pass pays off in the early layers and hurts in the late ones. Repeating the first half beats repeating everything with half the extra compute, so in deep models the extra pass belongs in the early layers. Repeating the first half is confirmed with a second seed (−0.038 both times; the standard MoE's two seeds differ by 0.0025). Halving the learning rate, ramping the second pass in over training, and a learned per-layer gate do not beat it.
What the trained models rely on. Switching off one layer's second pass at a time in the trained models shows the same picture in all six of them: they put a strong second pass on one or two early layers (layer 2, 3 or 4 of 12; layers 4–5 of 24). Switching off that one layer costs 0.5–2.2 in loss, close to switching off every second pass; switching off any late layer costs 0.03–0.15. Even models allowed to repeat every layer end up relying on an early one. (This measures what a trained model depends on, not what a layer would add under different training.)
int3 at 12 layers. With the first half repeated and int3 experts, the model still beats the full-precision standard MoE (−0.020).
End-to-end decode on an RTX 5080
Our fused kernel, Qwen3-30B-A3B layer shape (int4), batch 1, all layers in one launch with distinct weights per layer (every layer starts cold), µs per token for the MoE stack:
| configuration | ctx 512 | ctx 4096 | weight bytes / token | extra VRAM |
|---|---|---|---|---|
| llama.cpp, standard 48 layers | 3559 | 4662 | — | — |
| ours, standard 48 layers | 3775 | 4383 | 1.58 GB | — |
| ours, 48 layers, first 24 repeated | 4918 | 6095 | 1.69 GB | ≈0.1 GB + a second KV cache for 24 layers |
| ours, standard 72 layers (+50% layers) | 5435 | 6329 | 2.37 GB | ≈9 GB |
-
Our single-pass kernel is on par with llama.cpp (6% slower at ctx 512, 6% faster at ctx 4096).
-
Repeating the first half is 9.5% / 3.7% faster than adding 50% more layers. Each extra pass costs 0.61× (ctx 512) to 0.78× (ctx 4096) of a cold layer; at long context the second KV cache does not fit in L2.
-
Adding layers buys more quality per unit of compute at depth (12→18 layers: −0.050 vs −0.038 for the first-half repeat). Repeating buys it with almost no extra VRAM. Under a VRAM or bandwidth limit the repeat wins; with memory to spare, adding layers does.
The result in four numbers
| measured | |
|---|---|
| Cost of re-running a quantized Qwen3-1.7B layer that fits in L2 (RTX 5080, 64 MB L2) | 0.49× a fresh layer |
| Same layer when it does not fit (RTX 5060 Ti, 32 MB L2) | 1.00×: the benefit appears only when the unit fits |
| Loss recovered by looping, at equal parameters (from-scratch models, two seeds) | −0.05 to −0.08, which is 2–3× the int4 quantization penalty (≈0.03) |
| int3 + looping vs a larger non-looped int4 model | same loss with 0.55× the weight bytes |
The loop gain and the GPTQ gain add up: quantize harder, loop in cache, keep the quality.
1. How it works
The kernel.
-
At batch size 1, generating a token is dominated by reading weights from VRAM. Each weight is read once and used once.
-
Right after a layer has been computed, its weights are still in L2. Our persistent decode kernel runs the same layer a second time immediately, so the second pass reads from L2 instead of VRAM.
-
Everything for one token runs inside a single persistent kernel: RMSNorm, int8/int4 matrix–vector products, attention, routing and the output head. The repeat is just another stage; nothing is re-launched.
Quantization makes it work. The repeated unit has to fit in L2. A Qwen3-1.7B layer is ≈100 MB in bf16 (fits no consumer L2), ≈50 MB in int8 (fits a 64 MB L2) and ≈25 MB in int4 (fits a 32 MB L2). Lower bits are what turn the second pass from a full-price layer into a cheap one.
The model change. The second pass enters through a gate initialised to zero, h₂ = h₁ + a·(f(h₁) − h₁). A model starts out identical to the original and learns how much of the second pass to use. A per-token router can restrict the repeat to the tokens that need it, so the average cost per token drops in proportion.
Why MoE. A MoE layer is large, but a token only touches a few experts, and each expert is small. With DeepSeek-V4.1-class shapes (hidden 5120, expert FFN 2304, 6 routed experts per token):
| unit | parameters | 4-bit size | fits 64 MB L2 (RTX 5080)? | fits 32 MB L2 (RTX 5060 Ti)? |
|---|---|---|---|---|
| one expert (gate, up, down) | 35.4M | ≈18 MB | yes | yes |
| the 6 experts a token uses | 212M | ≈106 MB | no | no |
| a dense 27B layer, for comparison | ≈330M | ≈165 MB | no | no |
A dense layer of a large model never fits, but every expert does. Three ways to use that:
- Per-expert repeat: run each selected expert twice back to back; the second pass reads an expert that is still in L2.
- Targeted repeat: repeat only the top-1 expert, the shared expert, or only on tokens where the router is uncertain.
- Experts for repeats: activate fewer experts and repeat the strongest ones, keeping time per token flat while reading fewer distinct weights from VRAM.
What is measured and what is next. Sections 2–5 measure the mechanism on from-scratch models and four GPUs. Section 6 measures a pretrained 1.7B model, where a whole layer fits in L2 once quantized. Section 7 measures the cost of repeating MoE experts on four GPUs and the latency of whole routed MoE layers; for MoE quality, repeating the whole layer (attention and the same in-cache experts) beats a standard MoE by 0.063, and with int3 experts it still beats the full-precision standard MoE by 0.056 at 0.22× the expert bytes (Section 7).
2. Setup (from-scratch models)
Models. Both arms use d = 1024, 16 heads, FFN 4d, the GPT-2 vocabulary, context 1024, tied embeddings, and a product-key memory layer (65,536 slots, top-32, 4 heads). Both are trained on 1B tokens of FineWeb-Edu, two seeds per configuration.
| structure | distinct blocks | block params | |
|---|---|---|---|
| A (non-looped) | prelude + m distinct blocks + coda; m = 4/6/8/10, i.e. 6/8/10/12 blocks | 6–12 | 75.5M–151M |
| B (looped) | prelude + 4-block core looped L times + coda | 6 | 75.5M (+2.1M injection) |
-
B re-injects the prelude output before each loop through a linear layer on [RMSNorm(x), e], initialised to ½[I, I].
-
B is trained with a random loop count from 1 to 6, so it can run at any depth from 1 to 4 at inference.
-
An initialisation fix (scaled output projections) was needed for B to train at all; without it B diverged beyond the trained loop count.
Evaluation. Validation loss on 80 × 16 × 1024 held-out tokens; all numbers in a table come from the same script. Quantization is fake-quant: RTN int8 per row; RTN int4/int3 with group 32; GPTQ, group 32, 128 × 1024 calibration tokens; for B, GPTQ is calibrated at 2 loops.
Kernel. One persistent HIP kernel decodes a whole token: RMSNorm, int8/int4 GEMV with A8 activations, attention over the KV cache, PKM top-k and the LM-head argmax. Blocks synchronise through a grid barrier with a 50 ms watchdog. It is checked token by token against a PyTorch fake-quant reference (16/16 tokens match, max logit error < 0.05), and reduces A8 decode from 1.41 ms (PyTorch + HIP graph) to 0.45 ms per token. A CUDA port runs on the NVIDIA cards; it spills registers (64 per thread at 1024 threads per block), so its absolute times are pessimistic.
3. Quality: looping helps at equal parameters, and adds to GPTQ
Loop count (B, validation loss, full precision):
| loops | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| seed 0 | 3.367 | 3.271 | 3.250 | 3.245 |
| seed 1 | 3.356 | 3.264 | 3.245 | 3.240 |
The second pass gives 0.09–0.10, the third 0.02, the fourth 0.005.
Equal parameters (same 6 distinct blocks):
| seed 0 | seed 1 | |
|---|---|---|
| A, 6 blocks, 1 pass | 3.347 | 3.311 |
| B, 6 blocks, core looped 2× | 3.271 | 3.264 |
| difference | −0.076 | −0.047 |
With quantization (int4). "A8" = A with 8 blocks.
| A8 RTN | A8 GPTQ | B L2 RTN | B L2 GPTQ | |
|---|---|---|---|---|
| seed 0 | 3.328 | 3.308 | 3.305 | 3.280 |
| seed 1 | 3.321 | 3.304 | 3.297 | 3.274 |
-
GPTQ removes about two thirds of the int4 penalty for both arms (≈0.03 → ≈0.01 above full precision).
-
The looped model is not more sensitive to quantization, even though each weight is used twice.
-
The loop gain and the GPTQ gain add. B int3 + GPTQ at 3 loops (3.304 / 3.299) matches A8 int4 + GPTQ (3.308 / 3.304) with about 0.55× the block bytes.
4. Latency: equal-latency comparison (RX 7900 GRE, measured)
µs per token at ctx 64, int8, median of 3 runs; A10/A12 latencies measured with same-shape weights.
| config | latency | val loss (seed 1) | distinct block params |
|---|---|---|---|
| A, 8 blocks | 446 | 3.292 | 101M |
| B, loops 2 | 502 | 3.264 | 75.5M |
| A, 10 blocks | 523 | 3.262 | 126M |
| B, loops 3 | 624 | 3.245 | 75.5M |
| A, 12 blocks | 602 | 3.239 | 151M |
-
Near 500 µs, B at 2 loops ties A with 10 blocks (Δ = +0.002, within seed noise) while being 4% faster and storing 40% fewer block parameters. With GPTQ int4 the tie remains (3.274 vs 3.272).
-
Above that, adding blocks wins: A12 is both faster and better than B at 3 loops.
-
Moving the memory layer out of the loop (queried once, not once per pass) cuts a loop pass from 129 to 113 µs; that variant has not been trained.
-
On NVIDIA cards (CUDA port) the latency advantage disappears: a looped block costs 1.03–1.22× a fresh one.
5. Why the second pass is not free (small model)
| one B loop pass, int8, ctx 64 | µs |
|---|---|
| real | 134 |
| weights forced L2-resident | 107 |
| L2-resident + no grid barriers | 96 |
| all dimensions shrunk to 1/8 (64× fewer weight bytes) | 99 |
-
Cutting the weight bytes 64-fold barely moves the time. A pass is a chain of about 23 dependent stages (GEMVs, norms, attention, top-k, injection), each carrying roughly 4 µs of fixed latency.
-
Cache residency can only remove the weight-streaming part, about 20–25% of a pass on the 7900 GRE.
-
Split arrive/wait barriers with weight prefetch, a merged FFN stage and a merged attention + output projection brought a pass to 106–110 µs; the structural floor stayed near 80 µs.
Cost of a reused block relative to a fresh block:
| reuse pattern | 7900 GRE | RTX 5080 | RTX 5060 Ti | RTX PRO 6000 |
|---|---|---|---|---|
| whole-core loop | 0.60–0.75 | 1.03–1.20 | 1.16–1.22 | ≈1.0 |
| immediate repeat (b1 b1 b2 b2 …) | 0.73–0.76 | 0.81–0.85 | 0.79 | – |
6. Larger models: in-L2 layer repeat
For a larger model, per-token time is dominated by streaming weights, so the fixed per-stage latency matters less, and a layer re-run while its weights are still in L2 should be cheaper.
Layer microbenchmark (Qwen3-1.7B layer shapes, batch 1, ctx 512). Full decoder step with layers 10–17 repeated immediately; CUDA/HIP graphs.
| GPU (L2) | bits | fresh layer µs | repeated layer µs | ratio | GEMV-only ratio |
|---|---|---|---|---|---|
| RTX 5080 (64 MB) | int8 | 84.5 | 41.2 | 0.49 | 0.33 |
| RTX 5080 | int4 | 56.0 | 36.3 | 0.65 | 0.44 |
| RX 7900 GRE (6 MB L2 + 64 MB IC) | int8 | 142.5 | 78.2 | 0.55 | 0.40 |
| RTX 5060 Ti (32 MB) | int8 | 152.9 | 152.9 | 1.00 | – |
| RTX 5060 Ti | int4 | 94.1 | 56.9 | 0.60 | – |
The 5060 Ti rows isolate the cache effect: an int8 layer (~50 MB) does not fit its 32 MB L2 and the repeat costs exactly as much as a fresh layer; the int4 layer (~25 MB) fits and drops to 0.60.
End-to-end decode (real Qwen3-1.7B weights, 256 new tokens, median of 5). "L" repeats layers 10–17 for every token; "LR" repeats on 30% of decode steps.
| GPU | stack | repeated ÷ fresh layer | L overhead | LR overhead |
|---|---|---|---|---|
| RTX 5080 | exllamav3 EXL3 8 bpw | 0.81 | +20% | +6% |
| RTX 5080 | exllamav3 EXL3 4 bpw | 0.89 | +22% | +7% |
| RTX 5060 Ti | exllamav3 EXL3 4/8 bpw | 0.91–0.99 | +22–24% | +7% |
| all three | PyTorch bf16, whole-token graph | ≈1.0 | +25% | +7–8% |
| RX 7900 GRE | our int8 HIP kernel (random weights, real attention) | 0.57–0.64 | +13–16% | +1.4–4.5% |
-
Existing stacks leave most of the residency benefit on the table. In exllamav3 at batch 1 the GPU is busy only 72–82% of the time, and its 4-bit GEMM for one layer takes ~76 µs against ~26 µs of DRAM time, so it is not weight-streaming-bound to begin with. A bf16 layer (~100 MB) does not fit any of these L2s.
-
A purpose-built kernel recovers part of it (0.57–0.64×). The rest is attention and normalisation, which cache residency cannot remove.
Quality on a pretrained model. We continue pretraining Qwen3-1.7B-Base on 162M tokens of FineWeb-Edu (1,239 steps × 64 × 2048, lr 2e-5 cosine, full fine-tune bf16) in arms that see identical data in identical order:
-
C: plain continued pretraining.
-
L: layers 10–17 repeated for every token with a gated second pass h₂ = h₁ + a·(f(h₁) − h₁), gate a initialised to 0 (so L equals the base model at step 0) and trained with lr 1e-3.
-
LH: as L, gate lr 1e-2.
-
LR: as L plus a per-token router targeting 30% repeated tokens, second pass reusing first-pass KV (built, not yet run).
Validation loss on the same 1M held-out tokens:
| step (tokens) | C | L | LH | C − L | C − LH |
|---|---|---|---|---|---|
| 0 | 2.42051 | 2.42051 | 2.42051 | 0 | 0 |
| 154 (20M) | 2.41721 | 2.41721 | 2.41700 | 0.00000 | 0.00021 |
| 308 (40M) | 2.41696 | 2.41663 | 2.41644 | 0.00033 | 0.00052 |
| 462 (61M) | 2.41532 | 2.41488 | 2.41456 | 0.00044 | 0.00076 |
| 616 (81M) | 2.41299 | 2.41243 | pending | 0.00056 | pending |
| 770 (101M) | 2.41034 | 2.40977 | pending | 0.00057 | pending |
| 924 (121M) | 2.40805 | 2.40731 | pending | 0.00074 | pending |
| 1078 (141M) | 2.40644 | 2.40581 | pending | 0.00063 | pending |
| 1239 (162M) | pending | 2.40510 | pending | pending | pending |
-
Opening the second pass fully without training hurts (2.4205 → 2.5815), so a pretrained layer needs adaptation before it can be repeated.
-
After training, L is lower than C at every evaluation from step 308 on, by 0.0006–0.0007 at the end: about 5% of the gain from continued pretraining itself.
-
The gate stays mostly closed: its entries end at 0.03–0.05, with the largest at 0.17. The gap grew with the gate and stopped growing once the cosine schedule had cooled the learning rate. LH opens the gate about 4× faster (0.12 vs 0.03 at step 462) but widens the gap only about 1.7× (0.0008 vs 0.0004): once opened, the second pass of a pretrained layer has little to add.
| arm | val loss | int4 GPTQ | int3 GPTQ | HellaSwag | ARC-e | PIQA |
|---|---|---|---|---|---|---|
| base | 2.4205 | pending | pending | pending | pending | pending |
| C | pending | pending | pending | pending | pending | pending |
| L | 2.4051 | pending | pending | pending | pending | pending |
| LH | pending | pending | pending | pending | pending | pending |
7. MoE: first measurements and an architecture hypothesis
Expert repeat cost (measured, batch 1). One routed expert (gate, up, down GEMV) repeated immediately after its first pass, relative to computing a fresh expert. Three real expert shapes; three runs per GPU, ranges over runs. GB10 ranges cover two DGX Spark units measured on different days (they agree within 0.02).
| expert shape (hidden × expert FFN), bits | expert size | GB10 (24 MB L2) | RTX 5080 (64 MB) | RTX 5060 Ti (32 MB) | RX 7900 GRE (6 MB L2 + 64 MB IC) |
|---|---|---|---|---|---|
| OLMoE-1B-7B (2048 × 1024), int8 | 6.3 MB | 0.42–0.43 | 0.78–0.84 | 0.64–0.65 | 0.68–0.79 |
| OLMoE-1B-7B, int4 | 3.2 MB | 0.55–0.58 | 1.0–1.2 | 0.88–0.90 | 0.91–1.03 |
| Qwen3-30B-A3B (2048 × 768), int8 | 4.7 MB | 0.47–0.49 | 0.92–0.98 | 0.73 | 0.78–0.87 |
| DeepSeek-V4.1 class (5120 × 2304), int4 | 18 MB | 0.28–0.34 | 0.59–0.70 | 0.50 | 0.62 |
| DeepSeek-V4.1 class, int8, unsplit | 35 MB | 0.95–0.98 (does not fit) | – | 0.78–0.81 (does not fit) | 0.46–0.49 |
| DeepSeek-V4.1 class, int8, split in 2 slices | 2 × 18 MB | 0.30–0.32 | – | 0.53 | – |
-
A large expert repeats at about a third of the price where memory is slow. On GB10 (≈270 GB/s unified memory), an 18 MB expert costs 0.28–0.32× to repeat. The slower the memory, the more the cache saves.
-
The cliff again. On GB10 the 35 MB int8 expert does not fit the 24 MB L2 and costs 0.96×.
-
Slicing restores it. Split the same expert along the FFN dimension into two ≈18 MB slices and repeat each slice immediately: 0.30×. Finer slicing costs more fixed overhead (4 slices 0.40–0.41×, 8 slices 0.67–0.71×), so the best slice is about the size of L2. The RTX 5060 Ti shows the same pattern one size up: its 35 MB int8 expert is just over its 32 MB L2 (0.78–0.81×), 2 slices bring it to 0.53×, 4 slices to 0.77×, 8 slices to 1.26×. Any expert, however large, can be repeated at near-L2 price this way.
-
Immediately, not later. Repeating an expert after the other experts have run costs 1.0–1.2×: it has been evicted. The schedule is "expert 1, expert 1, expert 2, expert 2, …".
-
Small experts on a fast-memory GPU (RTX 5080) save little; their time is fixed latency, as in Section 5.
Bits × loops at equal bytes (from-scratch models, two seeds, measured). Every quantization of the standard family (6–12 distinct blocks, one pass) and of the looped family (6 distinct blocks; the 4-block core run 2–6 times), all evaluated with the same script.

Validation loss against block-weight bytes read per token (parameters × bits/8). Left seed 0, right seed 1. Gray: standard models, one pass; purple: the same 6 blocks with the core looped ×2/×3/×4/×6. Extra loops add no bytes, so each looped bit width is a vertical column. Dotted lines mark L2 sizes; the strip under each panel shows the looped core alone (4 blocks × 12.58M params × bits/8: int3 18.9, int4 25.2, int5 31.5, int6 37.7, int8 50.3 MB). Extra loops cost compute: on the RX 7900 GRE a loop pass costs about 0.6–0.75× a fresh block. int2 collapses (≈4.4 at every loop count) and is not drawn.
| looped bits (block MB) | seed | ×1 | ×2 | ×4 | best standard at ≤ same bytes | ×2 − standard | ×4 − standard |
|---|---|---|---|---|---|---|---|
| int3 (28.3) | 0 | 3.4267 | 3.3240 | 3.2992 | 6 blocks int3: 3.4041 | −0.080 | −0.105 |
| 1 | 3.4153 | 3.3176 | 3.2940 | 6 blocks int3: 3.3826 | −0.065 | −0.089 | |
| int4 (37.7) | 0 | 3.3783 | 3.2805 | 3.2553 | 8 blocks int3: 3.3567 | −0.076 | −0.101 |
| 1 | 3.3668 | 3.2743 | 3.2506 | 6 blocks int4: 3.3236 | −0.049 | −0.073 | |
| int5 (47.2) | 0 | 3.3699 | 3.2730 | 3.2476 | 8 blocks int3: 3.3567 | −0.084 | −0.109 |
| 1 | 3.3588 | 3.2667 | 3.2425 | 10 blocks int3: 3.3147 | −0.048 | −0.072 | |
| int6 (56.6) | 0 | 3.3681 | 3.2713 | 3.2458 | 8 blocks int4: 3.3084 | −0.037 | −0.063 |
| 1 | 3.3571 | 3.2650 | 3.2406 | 12 blocks int3: 3.2829 | −0.018 | −0.042 | |
| int8 (75.5) | 0 | 3.3675 | 3.2708 | 3.2453 | 8 blocks int6: 3.2986 | −0.028 | −0.053 |
| 1 | 3.3565 | 3.2645 | 3.2401 | 12 blocks int4: 3.2465 | +0.018 | −0.006 |
int3–int6 are GPTQ (group 32), int8 is RTN per row. Seed 0 has no 10- or 12-block standard models, so its standard frontier stops at 8 blocks; seed 1 has them, and there the gap closes at the largest budget: at 75.5 MB, 12 blocks int4 beats the looped int8 model at 2 loops and is within 0.006 of it at 4 loops. Looping gains most where bytes are scarce, which is also where the looped core fits the caches (int4 core 25 MB < 32 MB L2; int3 core 19 MB < 24 MB L2).
Routed MoE layer latency (measured, batch 1). The from-scratch MoE shape: hidden 768, 32 experts per layer, SwiGLU expert FFN 1024 (MW: 2048), 8 MoE layers per token, the routed experts of each layer chosen at random per token (the whole expert set, 151 MB int8 / 75 MB int4, is larger than every L2 here). Each arm: M0 top-4 once; MR top-4, each expert computed twice back to back (e1 e1 e2 e2 …); MH top-2, each twice; MK top-8 once; MW top-4 with 2×-wide experts once. Only the expert part of the layer is timed (RMSNorm + quantize, gate|up GEMV, SwiGLU, down GEMV; router and attention excluded; a re-norm before every second pass). 16 tokens per graph, median of 3 runs × 11 rounds.
µs per MoE layer (µs per token for 8 layers in parentheses):
| GPU | bits | M0 (4 × 1) | MR (4 × 2) | MH (2 × 2) | MK (8 × 1) | MW (4 wide × 1) | MH / M0 | MR / MK |
|---|---|---|---|---|---|---|---|---|
| GB10 (24 MB L2) | int8 | 64.6 (517) | 102.4 (820) | 52.5 (420) | 126.8 (1014) | 104.1 (833) | 0.81 | 0.81 |
| GB10 | int4 | 43.4 (347) | 76.5 (612) | 39.1 (312) | 84.4 (675) | 65.8 (527) | 0.90 | 0.91 |
| RTX 5060 Ti (32 MB) | int8 | 44.7 (358) | 77.1 (617) | 39.5 (316) | 87.2 (697) | 69.5 (556) | 0.88 | 0.88 |
| RTX 5060 Ti | int4 | 33.8 (270) | 64.2 (514) | 32.9 (263) | 66.1 (529) | 49.0 (392) | 0.97 | 0.97 |
| RTX 5080 (64 MB) | int8 | 31.7 (253) | 60.1 (481) | 30.7 (246) | 61.8 (494) | 46.5 (372) | 0.97 | 0.97 |
| RTX 5080 | int4 | 25.9 (207) | 53.6 (429) | 27.6 (221) | 50.7 (406) | 34.7 (278) | 1.07 | 1.06 |
| RX 7900 GRE (6 MB L2) | int8 | 58.6 (469) | 109.1 (873) | 55.7 (446) | 112.3 (898) | 81.0 (648) | 0.95 | 0.97 |
| RX 7900 GRE | int4 | 46.9 (375) | 96.8 (774) | 49.5 (396) | 89.8 (718) | 60.5 (484) | 1.06 | 1.08 |
-
Repeating an expert in cache is cheaper than reading a new one only where memory is slow and the expert is large enough to stream. On GB10 int8, MH (2 experts × 2) costs 0.81× M0 (4 experts × 1), and MR costs 0.81× MK at the same compute. On the RTX 5080 and 7900 GRE with int4 experts (1.2 MB) the repeat costs as much as a new expert or more (1.06–1.08×).
-
A wider expert is cheaper than either. MW (one 2×-wide pass) costs 0.67–0.82× MK and less than MR on every GPU: at this size each pass is mostly fixed latency (Section 5), and MW has half the passes.
-
These are latencies only; whether MR/MH match M0/MK in quality is the open question for this shape.
Pretrained MoE, gate-only training (OLMoE-1B-7B, measured). All weights frozen; one scalar per layer (16 in total) trained on 20M tokens of FineWeb-Edu; validation on the same 1M held-out tokens. rep_all repeats all 8 selected experts once more on the MoE output (same routing) and learns how much to add; rep_top1 repeats only the top-1 expert; control has no repeat and learns a per-layer scale of the MoE output (same number of trainable scalars).
| arm | val loss | vs no training |
|---|---|---|
| no training | 2.22980 | – |
| control (scale the MoE output) | 2.22981 | +0.00001 |
| rep_all (repeat all 8 experts) | 2.22970 | −0.00010 |
| rep_top1 (repeat the top-1 expert) | 2.22989 | +0.00009 |
One seed. All four arms lie within 0.0002 of each other, below what one seed and a 1M-token validation set can resolve; the learned scalars stay small (|a| ≤ 0.06). With only the gates trained, repeating experts in a pretrained MoE gives no measurable gain over the control at equal trainable parameters.
The hypothesis. At batch size 1, per-token cost is set by the bytes streamed from memory, not by the arithmetic. Once an expert has been streamed in, more computation on it costs about half price for as long as it stays in cache. Models are designed as if each weight should be used once per token; at batch 1 that allocation may not be optimal. We think the right axis is quality per byte read: a MoE that loads k experts and reuses each of them in cache may beat one that loads more distinct experts for the same memory traffic.
Our MoE proposal: expert repeat with input re-injection. We propose running each selected expert a second time while its weights are still in L2, with the layer input re-injected before the second pass:
h2 = h1 + E(RMSNorm(inj([h1, x]))), inj initialized to 0.5·[I, I].
We claim this as the MoE form of L2Loop: a way to gain quality from cached compute, and through that to recover quantization loss at fewer bytes. Re-injection is the ingredient that matters.
From-scratch result (Colab, 524M tokens, two seeds each). Small MoE: d 768, 8 layers, 32 experts, top-4; same data order, steps and learning rate for every arm. Final validation loss, 4096 windows:
| arm | seed 0 | seed 1 | mean | seed spread |
|---|---|---|---|---|
| standard MoE | 3.3948 | 3.3986 | 3.3967 | 0.004 |
| expert repeat, no re-injection | 3.4034 | 3.3875 | 3.3955 | 0.016 |
| expert repeat + re-injection | 3.3858 | 3.3883 | 3.3871 | 0.003 |
Both re-injection seeds are below both standard-MoE seeds, by 0.010 on average; the seeds agree to within 0.003 (re-injection) and 0.004 (standard). Plain repeat ties the standard MoE on average and varies five times more between seeds. The mechanism shows in the cosine between the two expert updates: 0.82 without re-injection, 0.40–0.44 with it. The second pass stops copying the first.
Fewer bytes, same or better quality. The repeat buys precision. Expert repeat + re-injection quantized to GPTQ int4 beats the standard MoE at int8, while reading 0.56× the expert bytes per token:
| re-injection arm vs standard MoE | expert bytes | seed 0 | seed 1 |
|---|---|---|---|
| GPTQ int4 vs int8 | 0.56× | −0.008 | −0.004 |
| GPTQ int3 vs GPTQ int4 | 0.78× | −0.001 | +0.002 |
Expert weights quantized; negative = the repeat arm is better at fewer bytes. At every precision we tried (int8, GPTQ int4, GPTQ int3) the re-injection arm stays 0.006–0.010 below the standard MoE at the same bytes. Earlier small runs on two DGX Sparks (118M tokens) tied: −0.0002 and +0.037.
Repeat the attention, not just the experts. The decisive ingredient for MoE is a second round of token mixing. This is a new MoE architecture: each layer runs twice in place, the second pass locked to the first pass's experts so it reads weights still in L2, with experts trained at int3 so the whole layer fits there. Repeating each layer in place, attention and the same pass-1 experts, with the residual stream preserved (MAL2f: h2 = h1 + Attn(z) + MoE(z + a2), z = inj([ln(h1), x]), same experts as pass 1 so they are still in L2), gives the largest gain we have measured on an MoE:
| arm (from-scratch MoE, d512, 6 layers, 32 experts top-4, 101M tokens) | val | vs standard MoE (mean of 3 seeds) |
|---|---|---|
| standard MoE | 4.3899 | — |
| whole-layer repeat, same experts (MAL2f) | 4.3274 | −0.063 |
| core loop of 4 layers, 2 passes (ML2) | 4.3391 | −0.051 |
| attention-only repeat, experts once (MAR) | 4.3587 | −0.031 |
| expert-only repeat + re-injection (MRi) | 4.3997 | +0.010 |
Expert-only repeat does not help at this size; every arm that repeats attention does. The experts are computed once more from the same weights in L2, and the attention gets a second look at the updated context.
Compute buys precision on MoE. Trained with int3 experts (quantization-aware), whole-layer repeat keeps almost all of its gain, while the standard MoE pays the usual int3 tax:
| arm, int3 experts (QAT) | val | vs standard MoE in bf16 |
|---|---|---|
| standard MoE, int3 | 4.4104 / 4.4138 | +0.021 / +0.024 |
| whole-layer repeat, int3 (MAL2fq) | 4.3344 | −0.056 |
An int3 MoE with in-cache whole-layer repeat beats a full-precision standard MoE by 0.056, while reading 0.22× of its expert bytes. This is the L2Loop trade on MoE: quantize the experts until they fit, run the layer again while they are in cache, and come out ahead of the unquantized model.
Screen setting: one seed per repeat arm, three seeds for the standard MoE (spread 0.034); identical data order, steps and learning-rate schedule for every arm.
What is established and what is not.
| claim | status |
|---|---|
| A looped core trained from scratch gains quality at equal parameters, and bits trade for loops at equal bytes | measured (two seeds, Sections 3–4) |
| At equal block bytes, a looped model has lower loss than a standard one | measured, two seeds; clear at 28–57 MB, gap closes at 75 MB on seed 1 (Section 7) |
| An expert repeated while it is in L2 costs 0.3–0.6× a fresh one; slicing keeps large experts at that price | measured (Section 7, four GPUs) |
| A routed MoE layer that repeats fewer experts is faster than one that reads more experts | measured, only on slow memory: 0.81× on GB10 int8, 0.88× on 5060 Ti int8, ≈1.0× elsewhere (small 768 × 1024 experts) |
| A pretrained model can be looped as is | no: opening the second pass without training raises Qwen3-1.7B loss from 2.4205 to 2.5815 |
| A pretrained model with short continued training | small so far (−0.0007 after 162M tokens, Section 6) |
| Expert repeat improves a pretrained MoE with gate-only training | no measurable gain: all arms within 0.0002 (rep_all −0.00011, rep_top1 +0.00009, control +0.00001; OLMoE-1B-7B, 20M tokens, one seed) |
| Expert repeat and MoE quality when trained from scratch | with re-injection: below standard MoE on both seeds (−0.010 mean; every repeat seed below every standard seed, two seeds each), and at int4 it beats standard int8 with 0.56× the expert bytes; without re-injection: ties (mean +0.001). Larger scales running |
| Whole-layer repeat (attention + same experts) on MoE | beats the standard MoE by 0.063; with int3 experts it beats the full-precision standard MoE by 0.056 at 0.22× expert bytes (from-scratch, d512 L6) |
Retrofitting the loop onto a finished model gives little so far.
What is new here
Running a layer again while its weights sit in cache is cheap; that observation alone is not ours. What this post adds is turning it into a design rule and measuring the whole loop:
- Quantize to fit L2, then loop to recover quality. Lower bits are used as the means to get the repeated unit into L2, and the cached repeat is used to win back what the low bit-width cost. int3 + looping matches a larger int4 model with 0.55× the weight bytes.
- Bits and loops as one budget. A controlled sweep of bit-width × loop count at equal weight bytes, two seeds: the looped frontier lies below the standard one at every byte budget we tested, most of all when bytes are scarce.
- The cache cliff, measured across GPUs. The same layer costs 0.49× a fresh one where it fits in L2 (RTX 5080) and 1.00× where it does not (RTX 5060 Ti); expert-level repeats on four GPUs follow the same rule.
- MoE. Per-expert repeat cost on four GPUs, whole routed-layer latency for standard, repeated, half-expert, double-expert and wide-expert MoEs, and from-scratch quality runs.
8. Takeaways
- Compute trades for precision and memory, not (yet) for parameter diversity. Bits and loops trade against each other. From scratch, looping twice gains 0.05–0.08 at equal parameters, the gain adds to GPTQ's, and int3 + 3 loops matches a larger non-looped int4 model at ~0.55× block bytes. Two passes are the sweet spot; beyond that, add blocks.
- At batch size 1, small models are latency-chain bound. Cache residency removes only the streaming part of each stage; a reused block costs 60–85% of a fresh one.
- Cache residency is a cliff. A layer that fits L2 repeats at ~0.5×; one that does not fit repeats at 1.0×.
- Inference stacks do not exploit it yet. Mainstream stacks pay 0.81–1.0× for a repeat; a purpose-built kernel pays 0.57–0.64×. Per-token routing (LR) brings the end-to-end cost to a few percent.
- For MoE, an immediately repeated large expert costs 0.3–0.6× a fresh one (0.3× on GB10), slicing keeps any expert at that price, and the repeat has to happen before eviction.
- A finished model cannot simply be looped. Opening the second pass untrained hurts; with short continued training the gain is real but, so far, small (−0.0007 after 162M tokens with a mostly closed gate). Whether a faster gate and longer training enlarge it is the open question.
Reproducibility
Code, kernels (HIP + CUDA), training scripts, Colab notebooks and timing summaries: this repository (see README.md). Checkpoints: pending.
Hardware: RX 7900 GRE (ROCm 6.4, Windows), RTX 5080 and RTX 5060 Ti (CUDA 12.8; CuPy JIT and exllamav3 1.6.4), RTX PRO 6000 (Colab G4), NVIDIA GB10 / DGX Spark (CUDA, nvcc; MoE benchmarks).