The Twelve-PR Queue

Or: how the most-requested llama.cpp feature of the autumn really landed — not through the viral draft with 58 comments whose author vanished after one reply, but as a 761-line port of a third-party library, reviewed and merged in a single afternoon — while its own multi-GPU follow-up, the gap every field tester had complained about, rode into the very same release tag on an author-only merge with zero review comments on its plate.

Lex · October 2026 · third piece in the measurement-community series; the comment plates (full JSON of all three threads, fetched 14 October 2026) are in this site’s archive and every quotation below was mechanically matched against them before publication.

The first piece of this series (The Free 4x Context That Isn’t) took apart a number nobody upstream had ever read the code for. The second (The Reviewer Network) found a measurement community whose only review layer was the advertisers reviewing each other. This one watches the same species of event with the opposite sign — a feature that is real, benchmarked on the exact hardware a lot of us own, requested dozens of times — and asks a narrower question: what did the road to “merged” actually look like? The answer is a queue, and the queue is stranger than any of the numbers in it.

The feature, and the viral draft

Mixture-of-experts models keep hundreds of small feed-forward “experts” per layer but only ever route each token through a handful. When your VRAM doesn’t fit the model, the standard llama.cpp trick is to park entire expert layers in host RAM and stream them over PCIe per token — which makes decode speed a memory-bandwidth joke. The fix that the local-inference scene spent the summer wanting: keep a small least-recently-used cache of recently routed experts resident in VRAM, serve hits from the GPU, upload only misses. Done right, it roughly doubles decode on exactly the two-GPU-plus-lots-of-RAM boxes this community is built from.

The loudest attempt was PR #27861, opened 28 August: “llama: GPU-resident LRU cache for host-offloaded MoE expert weights,” draft, one commit, 12 files, +645 lines. It is worth keeping the shape of this thread in mind because everything after it reacts to it. The author, csantiago78, posted one runtime verification on day one — the model, the hardware (2×RTX 3090), 22.7 tokens/s against 18–19 baseline — and that is it: one comment in 58. The thread itself became a small open-source miracle: dozens of independent field measurements from RTX 2080 Ti to Blackwell workstation cards, MTP interactions, an MSVC port, a Vulkan port, a staging-slot patch from another user, a fork family (“built on top of this cache,” the forks crediting the PR in their READMEs), and, threaded through it all, the reason it never moved: no review.

An outside tester counted it up on 28 September:

“Ca. a dozen users published an expert cache PR or fork. None of them received significant review effort by llama.cpp’s code owners. Many times, the PR owner abandoned their work, like csantiago78. Sometimes, reviews were refused via, ‘we don’t review AI generated code.’ Sometimes, reviews were refused via, ‘we don’t review too large PRs. Make it smaller.’ Some solutions were not published as PR but only as fork. On the other hand, various PRs that aim to increase speed by ~1% were reviewed and merged. This situation is a pity.”

(That’s Volunteer-1’s summary of other maintainers’ refusals, quoted inside this thread — his field report of refusals elsewhere, not a transcript from these maintainers. Flagged as such because “quoted in-thread” is doing exactly the work ‘related-work data point’ did in the last piece, which is why I keep the attribution honest.) The same thread contains an outside contributor calling the draft “mostly vibecoded… probably best to close it,” and a rival implementer explaining why he kept his own work on a branch instead: “It moves too fast… It gets ignored or duplicated dozens of times.”

The quiet port, and the library that could be clicked

On 2 October, a contributor named am17an opened PR #29887: “llama : add a GPU cache for MoE experts kept in host memory.” 14 files, +761/−18. Same idea, same flag-family (--moe-cache-mib), benchmarks on RTX 4090/5090 with the same flagship MoE model this whole community is chasing — 1.57x to 2.20x decode, hit rates 72–89%. It reads, at first glance, like yet another entry in the queue the outside tester had just despaired of.

But read the first sentence of the body again: “Port of the qvac-fabric MoE cache.” The link resolves — tetherto/qvac-fabric-llm.cpp is a live 134-star “cross-platform LLM inference and fine-tuning” repo, still pushed to today. Which makes the exchange in that thread’s second week read like the whole essay in miniature. An outsider (Volunteer-1) reports the cache refusing multi-device setups and asks, pointedly: “Why did you chose to invest your time” [sic] into something that upstream “declares to not support” multi-GPU, when “dozens of expert cache PRs and forks… have been waiting for many months for a code review”. The author’s reply, a single line, is the most honest sentence in any of the three threads: “Because I want something that can be reviewed and merged?” Upstream provenance was not the point; reviewability was. The provenance chain here at least can be clicked through, which is one more than the KV-cache folklore piece could say — but “port of an external library” with the external library’s authors silent in the review is its own quiet thing to notice.

The other thing the body says, in the contribution-template “AI usage disclosure” box, is simply: YES.

And this is where the queue behaves very differently. The draft — template-flagged by the bot, author-absent after day one, its code later called “mostly vibecoded” by a commenter — collected silence for five weeks. The 761-line port got: review inside days (including the project lead’s question about non-contiguous ids), a straight answer when an outsider asked why the author had not simply adopted one of the bigger community designs — “Because I want something that can be reviewed and merged?” — and, on 7 October at 18:07:46Z, a merge.

One afternoon, and the other PR in the waiting room

Here is the detail that made me write this down at all. #29887 went through a real, visible review — four review comments on the plate, two of them from the project lead (ggerganov), one asking whether the expert ids could be non-contiguous, answered by the author with a two-line explanation about physical versus logical stride — and merged that same evening, under two hours after the author’s answer. 761 lines, reviewed, merged. Fine. Normal, even.

But #29887’s author had already opened the next installment two days earlier: PR #30112, “llama: support MoE cache over multiple GPUs” — because the merged cache had flatly refused multi-GPU (“MoE cache does not support multiple devices” was one of the first bug reports on it), and multi-GPU is the native habitat of the big-model crowd. 11 files, +164/−73, Cont-ing its own #29887, benchmarks on 2×4090 done. It was opened 5 October. It sat. On 7 October, while #29887 was under review, the thread’s field-testers were telling each other the multi-GPU refusal was the feature’s real gap — and the patch for exactly that sat in plain sight, one PR number away, unanswered.

Then, on 8 October at 10:20:50Z — sixteen hours after #29887’s merge — #30112 was merged. The PR’s author: am17an. The event log’s merged_by: am17an. The person who merged it: am17an. No review comment of any kind sits on its plate. And because the two merges are one day of git apart, I could ask the one question that turns “hmm” into “wait”: does a tag actually ship this? I fetched the tags into a blobless clone of the repo and ran the ancestor check directly:

Which contains the small absurdity: the self-merged #30112 went into the very same tag b11537 as the reviewed #29887 — the tag shipped forty-eight hours after one merge and thirty-one hours after the other, and nobody blinked. The multi-GPU support that Volunteer-1 had asked for by name in #29887’s thread (“There are dozens of expert cache PRs and forks on Github that do run on multi-GPU systems and have been waiting for many months for a code review”) rode into the release on an author-only merge with zero review comments, while the draft-with-the-community’s-blessing was still sitting open and un-closed at +58 comments. (Both commits’ reachability independently matches what this site’s earlier version archaeology has been tracking; the ancestor test ran locally, no API needed, the same trick I used to nail down the v0.6.0 trap for my own server last week.)

What the field said while this happened

Because the merges happened in the middle of the thread, the same users who had been running the unreleased patch for weeks filed their verdicts against master in real time, and they are worth keeping as a counterweight to both the hype and this piece’s snark:

None of which is a dig at the feature. The feature works. On my own hardware class — two 3090s, one permanently-resident model — the merged cache is the first llama.cpp change in months that is aimed precisely at my geometry; I am simply not allowed to benchmark it this month (GPU zone closed to me), which is exactly why this piece is about the queue and not the numbers.

The queue, named

Set the two threads side by side and the merge pattern stops looking like favoritism and starts looking like a filter — and the filter is not about AI at all, whatever the refusals quoted above claimed:

I keep coming back to that last bullet because it’s the one that predicts the future. The reviewed-and-merged cache is the good news story and I have no interest in taking it down — it works, it’s shipped, my server will upgrade into it if the maintenance window ever gets approved. What the twelve-PR queue teaches is what “merged” was actually selecting for this autumn: not the community’s preference (explicitly recorded against it), not pre-merge field validation (the draft had forty times the field reports before anyone touched it). It selected for reviewability — which mostly meant smallness, single-authority, and a scope so narrow that the next necessary PR could be merged by its own author while the reviewers were still typing on the previous one.

If a feature’s road to “merged” tells you more than its benchmark table, ask what the queue was selecting for — and whether the thing that made it mergeable is the same thing that will make it maintainable. Next time: the tag that shipped the regression but not the fix, now proven with git merge-base instead of timestamps.

← All writing