Defect report
A story the reader needed, rejected for discussing something else entirely
One bug, from the symptom in a stored scorecard to the lines that caused it — and what fixing it cost.
Symptom
Reasons attached to the wrong articlesGiants’ 53-man roster to include Odell Beckham
“The article discusses a music EP, which is irrelevant to the user’s interests.”
fixture Ray · dropped at blended
open this story’s recorded traceMirziyoyev signs decree abolishing exit visa-style registration
“The article discusses NFL team rosters, which is not relevant to the user’s interests.”
fixture Daniel · dropped at blended · planted needle
open this story’s recorded traceWorld mostly shrugs off Bessent’s ‘D-Day’ Iran sanctions threat
“The article discusses China’s manufacturing activity…”
fixture Frank · dropped at rank
open this story’s recorded trace
A clean offset, not a model reasoning badly. The middle one is sharpest: n-dil-02 is a planted needle, injected so its right answer is known by construction — and it was rejected on another article’s reasoning.
Cause
Positional output misalignmentThe feed scores candidates in batches of forty through score_articles_batch, which asks for one verdict per article in the same order and sends no article identifier. The parse is positional. A count mismatch was logged and then ignored.
Misattributed relevance is worse than absent relevance. A missing score is visible downstream and can be retried; a shifted one silently drops a story and files a plausible rationale about a different one.
How the repository itself confirms this
The prototype pipeline requires every verdict to echo its id and drops unmatched ones — and every rejection reason in its scorecards matches its own headline. A correct implementation exists on the product side too, in ranking_provider.py, which keys by article id, caps output and treats truncation as failure. It is gated behind S7_PROVIDER_ENABLED and is not what the live feed calls.
Schematic · merge point illustrative
The fix
And one deliberate non-fixWhat landed
A length mismatch discards the batch
The retry loop gets another attempt instead of verdicts assigned by position. If every attempt mismatches, the batch reports itself unscored.
CacheMiss and BudgetExceeded propagate
Both were swallowed by a blanket handler, so an offline replay that should have failed closed instead scored every article 0.0 — while reporting zero cache misses.
What was deliberately left broken
The complete fix is id-keyed output, which changes the scoring prompt — and the regression gate replays responses keyed by a hash of the request. Changing the prompt invalidates the entire evidence base the gate runs against.
Bundling a correctness fix with the destruction of the baseline that proves it leaves no way to show the fix helped. So the guard lands first, and the prompt change is staged behind a cache rebuild.
What it cost
Identical inputs, only the parse differs- Guard fires
- 63one run, ten fixtures
- Worst response
- 201verdicts for 40 articles
- Cache misses
- 0replayed entirely offline
- Model calls
- 44 → 84retries are not free
- Capped recall@12−3.4 ppCapped recall@12: Committed scorecard 22.1 per cent, With the guard 18.7 per cent, a difference of −3.4 pp, past the fixed materiality cutoff.
- Reached the scorerunchangedReached the scorer: Committed scorecard 32.1 per cent, With the guard 32.1 per cent, a difference of unchanged.
- Unwanted rate+15.6 ppUnwanted rate: Committed scorecard 26.9 per cent, With the guard 42.5 per cent, a difference of +15.6 pp, past the fixed materiality cutoff.
- Planted-needle recall−15.0 ppPlanted-needle recall: Committed scorecard 60.0 per cent, With the guard 45.0 per cent, a difference of −15.0 pp, past the fixed materiality cutoff.
- Lookalike rate+10.0 ppLookalike rate: Committed scorecard 15.0 per cent, With the guard 25.0 per cent, a difference of +10.0 pp, past the fixed materiality cutoff.
- World-critical delivery−15.0 ppWorld-critical delivery: Committed scorecard 25.0 per cent, With the guard 10.0 per cent, a difference of −15.0 pp, past the fixed materiality cutoff.
One 0–100% scale for every row, ticked at 0, 50 and 100. Fraction metrics only — counts, costs and latencies share no scale with a recall rate. A filled dot marks a difference past the fixed ±0.02 cutoff, which is a chosen threshold, not a significance test.
Not “the fix made it worse” — two facts at once. The committed 22.1% was never a measurement of the pipeline judging correctly, because it counted verdicts that landed on the right article by accident.
And refusing to guess is expensive: on a mismatch the guard discards all forty verdicts, so those candidates carry no relevance signal at all. That is why the unwanted rate climbs. Neither number describes a healthy pipeline — the first is meaningless, the second is the price of having no way to salvage a partial response.
Verification
Nine tests, five of them new failures- short response is discarded rather than shiftedfailed before — applied by position
- short response is retried before giving upfailed before — never retried
- long response is also discardedfailed before — extra entry silently dropped
- cache miss propagatesfailed before — swallowed, scored everything 0.0
- budget exceeded propagatesfailed before — swallowed
- aligned response is applied in orderpassed before — unchanged
- scores are clamped to the unit intervalpassed before — unchanged
- ordinary provider errors still fall backpassed before — unchanged
- request kwargs are the expected shapepassed before — documents the remaining gap
Nine tests in test_batch_scoring_alignment.py. Five fail against the unfixed code; four are non-regression guards.
The gate goes from 42 passed to 6 failed, and the baseline was not re-recorded. It is correctly reporting that behaviour changed; re-recording it for a green badge would destroy the only evidence the change had a cost.
Still broken
Stated rather than buriedReordering still slips through
No article ids are sent, so a response of the right length but internally reordered is still misapplied.
No output cap or finish_reason check
Both still missing on this path. Both belong with the prompt change.
The evidence predates main
Every scorecard here ran at a revision not reachable from the default branch, and none records an evaluation protocol.
The corrected prototype is untested
CI’s prototype tier runs a historical legacy adapter, so the corrected pipeline is not exercised at all.
Where this site stops and the backend begins
Daily is an iPhone app against a FastAPI backend and Postgres. The backend is a stateful daemon — seven background loops, schema application, a Postgres advisory lock for leader election — and it stays where it runs today.
This site is a separate read tier: the reader replays a frozen corpus, the explorer renders immutable artifacts exported from the harness’s own scorecards, and nothing here scores an article. Labels are model-written with an agent pass, so absolute values are provisional.