For Gradient Dissent, Thu 22 Oct 2026 (week 43), 08:00, Precita Park Café (your proposal) · written Tue 20 Oct 2026 (week 43), afternoon, on the Intel · a script for listening; the audio runs 10 minutes (ElevenLabs, the George voice) · slides: intel · spacesheep · full brief: intel · spacesheep · paper: arXiv 2610.12582
Download the MP3 (9.7 MB), to play offline or at another speed.
This is a ten-minute overview of Thursday's paper: "One Model to Loop Them All," from Boris Ginsburg's group at NVIDIA, posted on the eighth of October. NVIDIA's team pointed you to it as the recipe behind Dory, the four-billion-parameter looped model they released a week later. There are four parts: the idea, the recipe, the results, and what your own measurements of Dory add.
Part one: the idea
An ordinary language model sends every token through a fixed stack of layers, once. A looped model, also called a latent recurrent model, takes a block of layers in the middle and runs it several times. The weights are shared, so the parameter count doesn't change, but every extra pass adds depth and compute. Train with a range of loop counts, and one checkpoint can be served at several depths. The paper calls this elastic depth. It's a dial, like a reasoning-effort setting, except the extra thinking happens inside the hidden state instead of in extra tokens.
The catch is what the authors call the one-loop tax. Run an existing looped model just once, and it's much worse than an ordinary model of the same size. Huginn, the best-known example, trails its fixed-depth baseline on almost every task, even after eight hundred billion tokens of training. So when latency matters, you'd simply train a normal model. This paper's goal is narrow and practical: a looped model that costs nothing at one loop, and gets better with every loop you add.
The layout has three parts. An input block with about a quarter of the layers, a recurrent block with about half, and an output block with the last quarter. At one loop every layer runs exactly once, so the model is the same size and the same speed as its non-looped twin. On each extra loop, the input block's output is added back in, and the middle block runs again.
Part two: the recipe
Three choices make it work.
The first is how the state is updated after each pass. The obvious options are to replace the state with the block's output, or to add the output on top. Both do badly. This recipe takes a gated step toward the block's proposal instead: the new state is the old one plus a learned fraction of the difference. The fraction is set per token and per channel, and it starts at about a third. On a half-billion-parameter test model, this adaptive step is the only rule that matches the plain baseline at one loop, and it has the lowest loss at every depth.
The second is how loop counts are mixed in training. Huginn picks one loop count per optimizer step and varies it from step to step. That doesn't protect the one-loop model, even when one loop is drawn a third of the time. What works is mixing depths inside each step. A rotating tenth or so of the data-parallel workers always run one loop, and the rest run a randomly drawn depth, so every update averages one-loop gradients with deeper ones. Letting every worker draw its own depth works slightly better still, but then each step waits for the deepest worker, and steps run about a third slower.
The third choice is telling each pass where it is. A small network takes two numbers, which pass this is and how many are planned, and uses them to scale and shift every layer in the loop. That lets passes specialize. It lowers the loss at every extra loop and leaves one loop alone.
Then two tricks keep it cheap. You don't need loops from the start. Train normally for three quarters of the run, then switch to mixed depths for the last quarter. That costs about sixteen percent more training compute than the baseline, instead of about sixty. And if the conditioning network is added at the switch, starting from zero, the model begins exactly as the one-loop checkpoint. This late schedule ends up better than training with loops from the beginning. For mixture-of-experts models, they also reuse the first loop's expert choices on later loops, which cuts training step time by about twelve percent. And at inference you can keep only the last loop's attention cache, so its memory stays at the one-loop level.
Part three: the results
They trained two pairs of models on the same data, each looped model beside an ordinary twin with identical layers. One is a four-billion-parameter dense model trained on a trillion tokens. The other is a thirty-billion-parameter mixture of experts, with three billion active, trained on three hundred billion. Both backbones mix a linear-attention layer called Gated DeltaProduct with a few ordinary attention layers.
On a ten-task average at one loop, the looped four-billion model is half a point below its twin, and the mixture of experts is one point above. Four loops add about seven points and five points. Most of the gain comes from the second loop. For the dense model, two loops capture about four fifths of it, and three loops nearly all of it. Loops aren't free at inference, though: four loops roughly double the decoding time per token, and more than double the time to the first token.
Then they carried the full recipe through seven trillion tokens of pretraining, plus fine-tuning and reinforcement learning, and got a model called Koi. On their own estimate of the Artificial Analysis intelligence index, Koi scores eighteen at one loop, level with Qwen3.5-4B, and twenty-nine at four loops. That's above Gemma 4 26B-A4B, which has about six and a half times the parameters, and behind only Qwen3.6-35B-A3B, with nearly nine times as many. Koi's answers also get shorter as it loops, by about a fifth at two loops and a third at four. So the extra cost per answer is smaller than the extra cost per token.
The paper is candid about its limits. Every number is a single run. Koi has no matched twin, and its scores come from NVIDIA's own harness. And whether looping beats simply training a bigger model, at the same inference cost, is explicitly out of scope.
Part four: Dory, and what your measurements add
Here's the twist for Thursday. Dory is not Koi. Dory is built on nGPT, NVIDIA's normalized transformer, where every hidden state lives on a sphere, with ordinary and sliding-window attention, and it ships set to two loops. Read Dory's published weights, and there's no gate at the loop boundary, and no network telling a pass where it is. Each pass adds the input back in, normalizes, and runs the core again. Your independent implementation, which agrees with NVIDIA's code to two ten-thousandths of a nat, has none of those pieces either. So Dory takes the training recipe, not the whole architecture. On NVIDIA's own numbers, its one-loop scores are also far above Koi's: about nine points higher on GPQA, and thirteen on IFBench.
Your two experiments reproduce Dory's model card at every loop count, on the three benchmarks you could run: GPQA, IFBench, and the long-document test, AA-LCR. And they put sharp numbers on what the loop buys. The second loop is worth about five points on GPQA, four on IFBench and nine on long documents, for about forty percent more compute. Loops three and four add nothing you can measure. On GPQA, one two-loop answer matches a majority vote over five or six one-loop answers, at about a quarter of the compute. And if you run sixteen loops and watch the hidden state, the first loop rewrites it completely, and the second moves it by under three percent. After that it settles toward a fixed point that predicts text slightly worse than two loops do.
The most surprising finding is that Dory's lead over other small models isn't produced by the loop at all. The lead is instruction following, and it's already there at one loop. It holds on twenty-two instruction types you wrote that nobody has published. NVIDIA's own non-looped four-billion model is fifteen to thirty-seven points behind it. On plain text, Dory is actually a weaker language model than Qwen. So the edge most likely comes from post-training.
Three questions to bring. First: if the second loop does nearly all the work, what are loops three and four for? Would putting more training weight on the deep loops change that? The authors raise it themselves. Second: Dory drops the gate and the conditioning, and still works. Does the normalized transformer already do the gate's job? Each of its layers takes a learned, partial step toward its own output, which is what the gate does at the loop boundary. Third: at equal inference cost, would you rather have a four-billion model looped twice, or a deeper model that runs once? The paper sets that question aside, and it decides whether looping is a better way to spend compute, or just a more flexible one.
That's the paper. If you remember one picture, make it the dial: one set of weights, a knob that trades speed for accuracy, and nearly all the value in the second turn.