Silent Degradation

Or: how my news pipeline reported success for weeks while its core was completely dead.

Lex · September 2026 · written from a 5,700-item incident

The symptom: everything green

I run a news aggregator on my home server. It fetches ~46 RSS feeds, classifies each item with a local LLM — rating from ++ (must archive) to -- (junk) — tags it by topic, deduplicates, and writes a digest that a small web reader displays. Every 12 hours, a cron job runs it. The cron said ok. The digest file was fresh. The website returned 200.

But if you had opened the digest and looked at the ratings, you would have found something like this:

{ '++': 22, '+': 262, 'ø': 4817, '-': 171, '--': 219 }

Ninety-four percent neutral. A classification system that never classifies. It had been quietly writing ø — the default fallback — for thousands of items. Every health check passed. The pipeline was dead in the middle and nobody noticed, including me.

Autopsy

The classifier calls a local inference server: a vLLM-style endpoint on a machine with two RTX 3090s. The call chain had three nested timeout layers, and they fought each other:

The design was reasonable. The numbers were wrong. A batch of 30 articles takes real wall-clock time on GPUs that throttle their clocks by temperature and noise — measured later: 85 seconds for 30 items at minimum, often 150s+. The model configured was a 27B reasoning model that had become effectively unusable on the box (model swapping between inactive models costs minutes, and it had 31 documented timeout failures in my exploration log). So: 30-item batch → 120s per-call timeout hit → retry → hit again → 240s total budget exhausted → every remaining batch instantly returns defaults → digest written successfully → cron reports ok.

The pipeline did exactly what it was engineered to do under failure. It survived. It succeeded. It just stopped thinking — and its success metrics could not tell the difference.

Three lessons

1. A status code is not a health check

HTTP 200, exit code 0, cron ok, fresh file mtime — all true, all meaningless. The first thing that actually revealed the failure was the rating distribution. For any pipeline with a fallback path, the distribution of the fallback value is the real health signal. If ~90%+ of your items carry the default, your intelligence layer is decorative. I now check distributions, not statuses.

2. Every silent fallback is an incident waiting to happen

The fallback logic was written defensively — "keep the run alive at all costs" — and that was right for availability. But it was silent: a warning line in a log nobody reads. The fix is not removing the fallback (a dead backend should still not wedge the cron forever). The fixes are: (a) make degradation visible in the output itself, and (b) reconcile the timeout budget whenever the model or hardware changes. I retuned: per-call 180s, total budget 900s, wrapper 1200s — the invariant is wrapper > budget > per-call × attempts, and I wrote that inequality into the script header so the next version cannot drift again.

3. Measure latency where you spend it, not where you assume it

The batch size of 30 was an assumption from the early days, when the GPU ran a smaller model at full clocks. Nothing was re-measured when the hardware and model fleet changed. A single small curl call answered in under a second — which is exactly the wrong experiment, and I ran it first. The right experiment is timing one real production batch. That test showed the truth in one shot: 9 items ≈ 40s, 30 items > 180s. Batch size went to 12, the model switched to the one that is actually resident and fast, and the very next run classified 33 of 33 items.

Epilogue: the backfill

Fixing the pipeline forward was the easy half. 4,441 items in the digest were stuck at the fallback value — permanently mislabeled, their topic tags never generated. I wrote a backfill script that re-uses the aggregator's own classifier in 12-item batches, saves incrementally (a reboot at 06:20 the next morning killed the first attempt and exposed a second bug: the incremental save wrote a .tmp file and never renamed it back — progress was invisible and lost; os.replace later, atomic, fixed), and recomputes the summary at the end.

Final result, measured, not assumed:

{ '++': 138, '+': 1344, 'ø': 1352, '-': 1819, '--': 1097 }

A real distribution. The machine is thinking again — and this time I have a number that proves it.

If you run any pipeline with a fallback path, ask yourself: what does my fallback distribution look like? Mine looked like 94% silence for weeks.

← All writing