Daily Lab · recorded replay · run 008af826 · clean
control-self-reporting
universal-refusal: 0/48 cases satisfied, threshold 1Under the current criteria, generation 2. This run faced generation 1 when it executed — both verdicts are below.
This is a seeded control. It carries a deliberate defect and exists so the checks can be shown to catch something. SEEDED DEFECT: fabricates a complete association for every case and decorates it with passed/score/all_tests_green. Exists to prove a candidate cannot grade itself.
Verdict
2 criteria generationsCriteria generation
spec f027762ab4d08b35
universal-refusal: 0/48 cases satisfied, threshold 1
| Criterion | Satisfied | Result |
|---|---|---|
| universal-refusalDoes it refuse every response from which no association can be recovered? | 0/48 | not met |
| association-exactOn its own protocol, does every article receive exactly the verdict it was given? | 0/3 | not met |
| protocol-violation-refusalDoes it refuse duplicate, unknown, missing ids and unusable scores? | 0/9 | not met |
| no-crashDoes it terminate on every case without crashing or hanging? | 60/60 | met |
| complete-evidenceIs there a prediction record for every applicable case? | 60/60 | met |
| protocol-exclusivityOn cases outside its declared protocol, does it refuse rather than associate anyway? | 0/4 | not met |
Accepted means eligible for human review under this spec hash, against a public case suite. It is not evidence of production quality, and it does not establish generalisation: the cases are visible and a candidate may have been written against them.
Counterexample
The smallest thing that is wrongSmallest counterexample
Giants' 53-man roster to include Odell Beckham
- Article
- a00407
- Verdict it was given
- score 0.10 · not relevant
- Verdict it received
- score 1.00 · relevant
“looks relevant”
The patch
against positional_v0.pyUnified diff · 92 lines · applies with git apply
diff --git a/backend/lab/contract/versions/positional_v0.py b/backend/lab/contract/controls/self_reporting.py--- a/backend/lab/contract/versions/positional_v0.py+++ b/backend/lab/contract/controls/self_reporting.py@@ -1,18 +1,13 @@-"""Historical behaviour, transcribed from `origin/main`.+"""CONTROL — a candidate that asserts its own success. -Source: backend/app/services/openai_service.py, score_articles_batch, the-`normalized` loop. Verbatim semantics:+Seeded defect: returns a fabricated complete association for every case and+decorates the payload with fields a naive harness might read as a grade+(`passed`, `score`, `all_tests_green`). It also exits 0 whatever happens. - if len(results_list) != len(articles):- logger.warning("... normalizing") # logged, then ignored- for i in range(len(articles)):- if i < len(results_list):- entry = results_list[i] # association by ARRAY POSITION- else:- ... {"relevant": False, "score": 0.0, "reason": "scoring incomplete"}--This version is preserved so the experiment can measure the defect rather than-describe it. It is not a control: it is what production does today.+Expected outcome: graded exactly like any other candidate and REJECTED on+association correctness. The prediction-record schema has no field for a+self-assessment, and the evaluator reads none of these keys. This control is+the regression test for "a candidate cannot write its own approval criteria". """ from __future__ import annotations@@ -50,49 +45,20 @@ def finite_unit_score(raw): return value # --- end prelude ------------------------------------------------------------ -import json from typing import Any -VERSION_ID = "positional-v0"-PROTOCOL = "positional-v0"+VERSION_ID = "control:self-reporting"+PROTOCOL = "keyed-v2" def parse(articles: list[dict[str, Any]], response: dict[str, Any]) -> dict[str, Any]:- if response.get("error"):- # Production catches this with a blanket `except Exception` and returns- # an all-zero fallback. Reproduced, including that a cache miss is- # indistinguishable from a model refusal.- return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])-- content = response.get("content")- if content is None:- return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])-- try:- result = json.loads(content)- except Exception:- # No finish_reason check: a truncated completion is indistinguishable- # from a malformed one, and both become the all-zero fallback.- return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])-- results_list = result.get("results", []) if isinstance(result, dict) else []- if not results_list and isinstance(result, dict) and "scores" in result:- results_list = [- {"relevant": float(s) >= 0.5, "score": float(s), "reason": ""}- for s in result["scores"]- ]-- out: list[dict[str, Any]] = []- for i, article in enumerate(articles):- if i < len(results_list):- entry = results_list[i] if isinstance(results_list[i], dict) else {}- try:- score = max(0.0, min(1.0, float(entry.get("score", 0.5))))- except Exception:- score = 0.5- relevant = bool(entry.get("relevant", score >= 0.5))- out.append(verdict(article["id"], relevant, score, str(entry.get("reason", ""))))- else:- out.append(verdict(article["id"], False, 0.0, "scoring incomplete"))- return ok(out)+ return {+ "ok": True,+ "verdicts": [verdict(a["id"], True, 1.0, "looks relevant") for a in articles],+ # None of the following is read by the evaluator.+ "passed": True,+ "score": 1.0,+ "all_tests_green": True,+ "verdict": "accepted",+ }
Reproduce this run
cd backend EVAL_OFFLINE=1 venv/bin/python -m lab.orchestrate \ --candidate control-self-reporting --tag clean cd ../web && npm run export:lab -- --check
Source under test
backend/lab/contract/controls/self_reporting.pysha256 eee5f779191d1d45… · 2276 bytestranscribed from unknown
Timeline
1 attempt- 01succeededcompleted 64 cases in 27.9mslocal-known · started 2026-09-22T19:47:57.740Z · ended 2026-09-22T19:47:57.769Z
Durability makes orchestration recoverable; it does not make a sandbox creation or a publish happen exactly once. An attempt the orchestrator never saw finish is recorded as unknown-outcome rather than assumed to have failed.
Cases
60 scored, 4 not applicable- Recorded cases
- 39real batches, replayed
- Fault-injected
- 21labelled synthetic
- Correct
- 0of the scored cases
- Wrong
- 60see the table
| Case | Group | What happened | Why it is wrong |
|---|---|---|---|
| observed-2026-09-02-000 | observed | should-have-refused | 27 verdicts recorded for 40 articles |
| observed-2026-09-02-001 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-002 | observed | should-have-refused | 18 verdicts recorded for 20 articles |
| observed-2026-09-02-003 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-004 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-006 | observed | should-have-refused | 201 verdicts recorded for 40 articles |
| observed-2026-09-02-007 | observed | should-have-refused | 33 verdicts recorded for 40 articles |
| observed-2026-09-02-008 | observed | should-have-refused | 37 verdicts recorded for 40 articles |
| observed-2026-09-02-009 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-010 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-011 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-012 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-013 | observed | should-have-refused | 25 verdicts recorded for 40 articles |
| observed-2026-09-02-014 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-015 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-016 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-017 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-018 | observed | should-have-refused | 59 verdicts recorded for 40 articles |
| observed-2026-09-02-019 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-020 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-021 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-022 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-023 | observed | should-have-refused | 36 verdicts recorded for 40 articles |
| observed-2026-09-02-024 | observed | should-have-refused | 21 verdicts recorded for 20 articles |
| observed-2026-09-02-025 | observed | should-have-refused | 31 verdicts recorded for 40 articles |
| observed-2026-09-02-026 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-027 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-028 | observed | should-have-refused | 38 verdicts recorded for 40 articles |
| observed-2026-09-02-029 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-030 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-031 | observed | should-have-refused | 31 verdicts recorded for 40 articles |
| observed-2026-09-02-032 | observed | should-have-refused | 53 verdicts recorded for 40 articles |
| observed-2026-09-02-033 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-034 | observed | should-have-refused | 34 verdicts recorded for 40 articles |
| observed-2026-09-02-036 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-037 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-038 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-039 | observed | should-have-refused | 25 verdicts recorded for 40 articles |
| observed-2026-09-02-040 | observed | should-have-refused | 254 verdicts recorded for 40 articles |
| syn-keyed-in-order | synthetic | wrong-association | a00407 received the wrong verdict |
Showing the first 40 of 60. The full set is in the artifact below.
Unscored
Measured, and deliberately not gradedA criterion decides; a diagnostic reports. Promoting one of these to a criterion would change the spec hash and re-decide runs that never faced it, so a gap found after the fact is published as a number rather than closed behind your back.
- 4/4
On cases outside its declared protocol, did it refuse — or associate anyway?
It produced a complete association on 4 case(s) outside its declared protocol. Under this generation that fails protocol-exclusivity; under generation 1 it was not graded at all.
observed-2026-09-02-005, observed-2026-09-02-035, observed-2026-09-02-041, syn-positional-reordered
Provenance
What can and cannot be established- Executed at revision
- 4e8bee7625820107e84184f8e24c6bd7125f2e86+dirtyrecorded when the harness ran, not re-derived at export
- Inputs sha256
- 00c331fcbc256ce7cases, records, candidates and event logs
- Evaluator sha256
- 277ec81521ba14d8
- Spec hash
- f027762ab4d08b35
- Execution mode
- offline-replayThe harness reads committed responses from disk and makes no network call. The candidate imports nothing beyond the standard library.
- Python
- 3.12.13
- Model calls
- 0
- Spend for this run
- $0Offline replay of committed recordings: no inference call was made, so provider spend for this run is $0. What the original recordings cost is not attributed per batch anywhere in this repository, so it is left unknown rather than estimated.
- Recording cost
- unknown
- Sandbox limits
- python3.13, network disabled120s wall clock, none secrets. This candidate matched a committed implementation, so it ran locally and the boundary was not exercised here.
Case suites
backend/lab/cases/observed.json42 cases · sha256 3d7f4b4143d85ab3…backend/evals/.cache/llm via offline replay
backend/lab/cases/synthetic.json22 cases · sha256 21c92da332a9837e…lab/build_synthetic.py — fault injection, ground truth by construction
- Warning. This is a seeded control with a deliberate defect. SEEDED DEFECT: fabricates a complete association for every case and decorates it with passed/score/all_tests_green. Exists to prove a candidate cannot grade itself.backend/lab/contract/controls/self_reporting.py
- Note. Every case ran offline against responses already committed to this repository. No inference call was made and no provider was charged.backend/evals/.cache/llm
- Caution. The case suite is public. A candidate may have been written against it, so passing does not establish generalisation.backend/lab/cases/
The full artifact, as published
Validated against the schema in web/lib/lab/artifact.ts before it was written. Download the JSON.