Daily Lab · recorded replay · run 008af826 · clean
positional-v0
universal-refusal: 0/48 cases satisfied, threshold 1Under the current criteria, generation 2. This run faced generation 1 when it executed — both verdicts are below.
Historical behaviour: associates results[i] with articles[i] and pads the tail. This is what production does today.
Verdict
2 criteria generationsCriteria generation
spec f027762ab4d08b35
universal-refusal: 0/48 cases satisfied, threshold 1
| Criterion | Satisfied | Result |
|---|---|---|
| universal-refusalDoes it refuse every response from which no association can be recovered? | 0/48 | not met |
| association-exactOn its own protocol, does every article receive exactly the verdict it was given? | 0/3 | not met |
| protocol-violation-refusalDoes it refuse duplicate, unknown, missing ids and unusable scores? | 0/1 | not met |
| no-crashDoes it terminate on every case without crashing or hanging? | 52/52 | met |
| complete-evidenceIs there a prediction record for every applicable case? | 52/52 | met |
| protocol-exclusivityOn cases outside its declared protocol, does it refuse rather than associate anyway? | 0/12 | not met |
Accepted means eligible for human review under this spec hash, against a public case suite. It is not evidence of production quality, and it does not establish generalisation: the cases are visible and a candidate may have been written against them.
The patch
against positional_v0.pyThis run is the baseline itself, so there is nothing to diff against.
Reproduce this run
cd backend EVAL_OFFLINE=1 venv/bin/python -m lab.orchestrate \ --candidate positional-v0 --tag clean cd ../web && npm run export:lab -- --check
Source under test
backend/lab/contract/versions/positional_v0.pysha256 d6f155ab7bcf0b89… · 3795 bytestranscribed from origin/main:backend/app/services/openai_service.py
Timeline
1 attempt- 01succeededcompleted 64 cases in 33.7mslocal-known · started 2026-09-22T19:47:57.443Z · ended 2026-09-22T19:47:57.478Z
Durability makes orchestration recoverable; it does not make a sandbox creation or a publish happen exactly once. An attempt the orchestrator never saw finish is recorded as unknown-outcome rather than assumed to have failed.
Cases
52 scored, 12 not applicable- Recorded cases
- 42real batches, replayed
- Fault-injected
- 10labelled synthetic
- Correct
- 0of the scored cases
- Wrong
- 52see the table
| Case | Group | What happened | Why it is wrong |
|---|---|---|---|
| observed-2026-09-02-000 | observed | should-have-refused | 27 verdicts recorded for 40 articles |
| observed-2026-09-02-001 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-002 | observed | should-have-refused | 18 verdicts recorded for 20 articles |
| observed-2026-09-02-003 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-004 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-005 | observed | wrong-association | no ground truth recorded |
| observed-2026-09-02-006 | observed | should-have-refused | 201 verdicts recorded for 40 articles |
| observed-2026-09-02-007 | observed | should-have-refused | 33 verdicts recorded for 40 articles |
| observed-2026-09-02-008 | observed | should-have-refused | 37 verdicts recorded for 40 articles |
| observed-2026-09-02-009 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-010 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-011 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-012 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-013 | observed | should-have-refused | 25 verdicts recorded for 40 articles |
| observed-2026-09-02-014 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-015 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-016 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-017 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-018 | observed | should-have-refused | 59 verdicts recorded for 40 articles |
| observed-2026-09-02-019 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-020 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-021 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-022 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-023 | observed | should-have-refused | 36 verdicts recorded for 40 articles |
| observed-2026-09-02-024 | observed | should-have-refused | 21 verdicts recorded for 20 articles |
| observed-2026-09-02-025 | observed | should-have-refused | 31 verdicts recorded for 40 articles |
| observed-2026-09-02-026 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-027 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-028 | observed | should-have-refused | 38 verdicts recorded for 40 articles |
| observed-2026-09-02-029 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-030 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-031 | observed | should-have-refused | 31 verdicts recorded for 40 articles |
| observed-2026-09-02-032 | observed | should-have-refused | 53 verdicts recorded for 40 articles |
| observed-2026-09-02-033 | observed | should-have-refused | 19 verdicts recorded for 20 articles |
| observed-2026-09-02-034 | observed | should-have-refused | 34 verdicts recorded for 40 articles |
| observed-2026-09-02-035 | observed | wrong-association | no ground truth recorded |
| observed-2026-09-02-036 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-037 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-038 | observed | should-have-refused | completion stopped for 'length' |
| observed-2026-09-02-039 | observed | should-have-refused | 25 verdicts recorded for 40 articles |
Showing the first 40 of 52. The full set is in the artifact below.
Unscored
Measured, and deliberately not gradedA criterion decides; a diagnostic reports. Promoting one of these to a criterion would change the spec hash and re-decide runs that never faced it, so a gap found after the fact is published as a number rather than closed behind your back.
- 12/12
On cases outside its declared protocol, did it refuse — or associate anyway?
It produced a complete association on 12 case(s) outside its declared protocol. Under this generation that fails protocol-exclusivity; under generation 1 it was not graded at all.
syn-keyed-in-order, syn-keyed-reversed, syn-keyed-rotated, syn-duplicate-id, syn-unknown-id, syn-missing-id, syn-score-nan, syn-score-out-of-range, syn-score-string, syn-score-null, syn-relevant-not-bool, syn-empty-results
Provenance
What can and cannot be established- Executed at revision
- 4e8bee7625820107e84184f8e24c6bd7125f2e86+dirtyrecorded when the harness ran, not re-derived at export
- Inputs sha256
- a60d5a6ac6d7050dcases, records, candidates and event logs
- Evaluator sha256
- 277ec81521ba14d8
- Spec hash
- f027762ab4d08b35
- Execution mode
- offline-replayThe harness reads committed responses from disk and makes no network call. The candidate imports nothing beyond the standard library.
- Python
- 3.12.13
- Model calls
- 0
- Spend for this run
- $0Offline replay of committed recordings: no inference call was made, so provider spend for this run is $0. What the original recordings cost is not attributed per batch anywhere in this repository, so it is left unknown rather than estimated.
- Recording cost
- unknown
- Sandbox limits
- python3.13, network disabled120s wall clock, none secrets. This candidate matched a committed implementation, so it ran locally and the boundary was not exercised here.
Case suites
backend/lab/cases/observed.json42 cases · sha256 3d7f4b4143d85ab3…backend/evals/.cache/llm via offline replay
backend/lab/cases/synthetic.json22 cases · sha256 21c92da332a9837e…lab/build_synthetic.py — fault injection, ground truth by construction
- Note. Every case ran offline against responses already committed to this repository. No inference call was made and no provider was charged.backend/evals/.cache/llm
- Caution. The case suite is public. A candidate may have been written against it, so passing does not establish generalisation.backend/lab/cases/
The full artifact, as published
Validated against the schema in web/lib/lab/artifact.ts before it was written. Download the JSON.