Fifteen gigabytes that were really twenty-five

The model card said fifteen gigabytes. At the context I actually ran it, the live process was twenty-five, and my 32GB Mac went to swap. Weights are the floor; the context length is the dial nobody prints.

A local model kept failing one kind of task with a context-length error, so I did the obvious thing and reloaded it with a bigger window, 128K instead of 64K. The error went away. So did my machine. Free memory fell to nine percent, swap climbed past thirty gigabytes, and the terminal pegged a whole core just drawing text.

The model card, and the tool that loads it, had both told me this model was fifteen and a half gigabytes. They were quoting the weights. The thing actually resident in my RAM was twenty-five. The missing ten was the KV cache, the per-token memory attention keeps as it reads, and it grows with the context window. I had tripled the window and silently bought nine gigabytes of cache to go with it.

What made it a real trap is that this is a mixture-of-experts model: twenty-six billion parameters, only four billion of them active on any given token. The whole pitch is that you pay a small model’s bill. And you do, for compute. The cache does not care which experts fire; attention runs on every layer at full depth regardless. Sparsity buys you compute, not memory. The number that decides whether you fit is the one neither the card nor the loader ever prints.

So I stopped hardcoding the window. A few lines now size it to the machine: the weights, plus the cache the chosen context would need, weighed against the RAM that is actually there, minus a reserve for everything else. On this 32GB Mac the same model lands at 112K instead of 128K and stays out of swap. On a 16GB laptop it refuses outright and tells me to pick a smaller model, rather than letting me find the ceiling by freezing. An earlier, larger model had answered that same question by kernel-panicking the Mac. The refusal is the feature.

When something tells you a local model fits, ask back: fits at what context? The weights are the floor. The context length is the dial, and it is the one number the tools politely leave off the label. Size to the hardware you actually have, and make the tooling say no before the swap does.