# SAGEN-Bench - pre-registration (REGISTERED)

**Status: REGISTERED AND PUBLIC (approved 2026-07-11).**
OSF registration: **https://osf.io/s3gfm**, DOI
**10.17605/OSF.IO/S3GFM** (associated project: https://osf.io/72vjy). All instruments are built, versioned, and
drift-guarded in CI; the confirmatory corpus is generated and frozen under a
SHA256 manifest; the free deterministic arms were executed against the
frozen corpus and their results committed BEFORE submission, and are
disclosed in the registration as observed preliminary work. The registration is approved, so the PAID arms
(S1-live, S2) are formally unlocked, bounded by the declared spend ceiling
(USD 150). This document is the binding protocol; the
registration transcribes it (`sagen-bench-osf-registration.md`). Any
post-registration change to instruments or plan is a protocol amendment
filed on OSF, never a silent edit.

**Determinism disclosure, stated plainly.** The S3 lattice and S1-oracle arms
are deterministic: re-running them after registration reproduces the committed
bytes exactly, so for those arms "pre-registration" cannot do the work it does
for sampled data. The epistemic protections for the deterministic arms are
(a) the prediction matrix was authored before the lattice first executed and
is append-only with respect to results, and (b) the confirmatory corpus was
generated mechanically from a seeded PRNG, not authored by the prediction
author. Registration binds the PAID arms (S1-live judged scoring, S2
perception sampling), which do have sampling noise.

**No human subjects.** Every arm observes software: the SAGEN reference
engine, LLM APIs as perception or judge, and scripted scenario corpora. No
person is observed, recruited, or rated. No IRB trigger exists.

**Relationship to the papers.** This is the "systematic end-to-end
evaluation" the SAGEN paper (entry 02, `/research/sagen`) names as future
work, plus a systematic test of the Classification-as-Infrastructure position
paper's constitutiveness prediction (entry 01). It targets the corpus's two
least-defended figures: the 98.5% / 20.5% / 10.5% coverage table (ledger
claim `sagen-coverage`, currently `sourced`) and the 0.94 closed-loop
alignment (absent from the ledger).

## Arms

| Arm | Question | Cost | Status |
|---|---|---|---|
| S3 knockout lattice | Does each architectural commitment carry exactly the pre-predicted capabilities? | zero | **pilot AND confirmatory executed; 64/64 both** |
| S1 oracle coverage | Does the injection beat flat baselines on mechanical recoverability, at n = 72? | zero | **executed; results committed** |
| S1 live coverage | Does the result survive real LLM perception + judged prose scoring? | paid | **executed 2026-07-11/12; H1 HOLDS, H2 HOLDS, H3 FAILS (see results addendum)** |
| S2 perception stability | How aligned/stable is LLM perception across models, runs, depth? | paid | **executed 2026-07-11/12; H4 FAILS (see results addendum)** |

## Frozen instruments

All in `src/lib/research/sagen/bench/` (tests: `bench/__tests__/`), all
versioned; any change is a new version and (post-registration) a protocol
amendment:

| Instrument | Version | Freeze |
|---|---|---|
| Rubric (paper's 20 dimensions) | `RUBRIC_VERSION 1` | group sizes 5/4/3/4/4; structural gap re-derives to 16/20 in CI |
| External dimension set | `EXTERNAL_SET_VERSION 1` | 12 dimensions cited to DST / CoALA / ACT-R / BDI; 5 have no v1 analog (incl. procedural memory = the paper's own "no learning" limitation, found independently) |
| Falsifier set | `FALSIFIER_SET_VERSION 1` | 5 anti-capabilities with runnable probes; all absent on pilot + confirmatory samples (a capture publishes as a finding AGAINST the paper) |
| Event grammar + oracle | `EVENTS_VERSION 1` | oracle/engine baseline agreement is a hard precondition (576/576 checks green on the frozen corpus) |
| Confirmatory corpus | `corpusVersion 1` | 72 scenarios, seed 20260711, stratified 3 depths x 2 densities x 3 distractor counts; **sha256 `eba3a75ac9cf589fc3de98e618cfda46561492e6ed128610a14feb53e2d9aa8e`**; regeneration-equality in CI; fairness invariant (stated goals/questions appear verbatim in surface text) test-enforced |
| Knockouts / probes | v1 / v1 | 8 x 8 as piloted |
| Prediction matrix | `PREDICTIONS_VERSION 1` | authored before first lattice execution; append-only |
| Coverage detectors | `COVERAGE_VERSION 1` | mechanical recoverability rules; 16 scored + 4 unexercised-with-reasons |
| Power model | `POWER_VERSION 1` | paired design, analytic + seeded simulation |

## Executed results (free arms, committed under `bench/results/`)

### S3 - knockout lattice, confirmatory corpus

**64/64 prediction cells correct; 22 lost cells predicted, 22 observed; zero
misses** (`lattice-confirmatory.v1.json`). The pilot result generalized
unchanged to the machine-generated corpus. Descriptives: single-topic-type
collapse remains exactly as destructive as full entity collapse (topic typing
is the load-bearing type); goal tracking fully separable from entity typing.

### S1 - oracle-mode mechanical coverage, n = 72

(`coverage-oracle.v1.json`; scoring = mechanical recoverability, 16 scorable
dimensions, full/partial/none.)

| Condition | Mean coverage | ~Tokens |
|---|---|---|
| raw buffer | 0.160 | 245 |
| full transcript (long-context) | 0.195 | 296 |
| mechanical rolling summary | 0.268 | 320 |
| **SAGEN @ 300-token budget** | **0.937** | **266** |
| SAGEN @ 2000-token budget | 0.966 | 317 |

Findings the paper's single 4-turn manual evaluation could not see:

1. **The structural gap survives long context.** The full transcript captures
   ZERO typed-structure dimensions (priority, lifecycle, urgency, transition
   typing, decay, watchlist, parseable output all 0.00) while SAGEN carries
   them at fewer tokens. Confirmatory family hypothesis 2, already resolved
   mechanically.
2. **The paper's 98.5% is 93.7% under mechanical scoring at its own 300-token
   design point.** The fixed budget truncates the `<sagen>` envelope on deep
   scenarios (machine-parseable-output 0.78 at 300 vs 1.00 at 2000; pivot
   visibility 0.89 vs 1.00). Budget must scale with depth: a measured,
   publishable design result.
3. **Four dimensions are unexercisable** by the conversation adapter on
   scripted input (goal-hierarchy, sticky-retention, unknown-tracking,
   assumption-tracking): the paper's marks for them rest on schema
   affordance, not demonstrated capture. Reported as an audit finding.
4. Entity re-typing is unexercised by generator v1
   (entity-type-classification capped at 0.5): logged as corpus v2 work, not
   scored in SAGEN's favor.

### Power (S1-live judged arm)

Paired within-scenario design; grid over smallest-effect delta x judge noise;
**frozen N = 66** (alpha 0.05, two-sided, power 0.9 at delta 0.1, sigma_d
0.25), simulation-checked (`power.v1.json`). The 72-scenario corpus covers N
with margin.

## The paid arms (bound by this registration)

### S1-live

- Realization: a model that is neither judge nor system-under-test renders
  each frozen scenario's turns into naturalistic surface text; analysis dicts
  may NOT change (they are the frozen ground truth); mechanical fidelity
  checks re-run on realized text.
- Perception: the production perception prompt (`perception.mjs`) on a pinned
  model; the engine consumes LLM perception instead of oracle dicts.
- Scoring: the mechanical detectors re-run (structural claims), plus judged
  prose scoring of the four unexercised + partial-credit dimensions on the
  frozen 0/0.5/1 scale. Judge scores stored, never recomputed. Judged
  dimensions count as confirmatory only if two judge models reach kappa
  >= 0.70 on a 20-scenario stratified subsample (machine IRR gate).
- Confirmatory family (Holm-Bonferroni, alpha 0.05): (1) live-perception
  SAGEN coverage exceeds the strongest flat baseline at matched token budget;
  (2) the structural-gap dimension subset stays uncaptured under full
  transcript; (3) live coverage is within 0.10 of oracle coverage (the
  perception tax is bounded); (4) S2's composite alignment floor (below).
  Scenario-level bootstrap CIs.

### S2 - perception stability

K = 3 pinned models x R = 5 runs x 30 stratified scenarios. Per-field
metrics frozen: Jaccard (topics, goals, questions), exact (references_previous),
categorical (sentiment); composite weights frozen at registration. Confirmatory:
composite alignment >= 0.80 per flagship model; run-to-run within-model
composite Jaccard >= 0.90 at temperature 0; pre-registered trend test for
alignment decay with turn depth (the novel measurement). The paper's 0.94
enters the ledger as whatever S2 measures.

### Spend ceiling

One window, both paid arms, tagged `ai-spend`; ceiling declared in the
registration; window aborts (and logs) at ceiling.

## Stopping rules and freeze discipline

- Registration freezes: corpus hash, all instrument versions, judge prompts,
  model IDs, temperatures, the confirmatory family, N = 66, the spend ceiling.
- One confirmatory run per paid arm; reruns only for infrastructure failure,
  logged. Post-freeze data cannot mint new confirmatory claims (new
  registration required).
- Deterministic arms re-verify forever in CI (recompute-live); results publish
  regardless of direction.

## Public record (the repo is private)

The development repository is private; the public, citable record is the
self-verifying release bundle at
`https://jakelawrence.xyz/downloads/sagen-bench/v1/` (built by
`npm run sagen:release -- --write`, drift-guarded in CI): frozen corpus +
manifest, committed deterministic results, the pure instrument modules, this
protocol, and a dependency-free `verify.mjs` that re-derives every number
byte-for-byte. Key artifacts are also attached to the OSF registration,
which archives them independently.

Post-window, a second bundle at
`https://jakelawrence.xyz/downloads/sagen-bench/v2/` supersedes v1 by adding
the registered paid window's complete raw record (`window/`, verbatim
per-call JSONL), the paid-window instruments, and a `verify.mjs` that
re-derives the full registered analysis (H1-H4, seeded bootstrap CIs, the
Holm family, the judge kappa gate, the spend ceiling) from the raw record.
v1 stays published unchanged as the pre-window snapshot the registration
pointed at.

## Sequencing (updated)

1. (done) Phase 0 instruments + S3 pilot 64/64.
2. (done) Rubric v2: external dimension set + falsifiers (issue #2021).
3. (done) Generator, frozen corpus, power, S3 confirmatory 64/64, S1 oracle
   results (issues #2022, #2023 free half).
4. (done) Registration filed, approved, and public with DOI (2026-07-11).
5. (done) Paid window: S1-live + S2 under the ceiling (issue #2023 paid
   half; $84.75 of $150).
6. (done) Ledger upgrades + paper addendum + bundle v2 + this results
   section (issue #2024; the owner waived the earlier embargo idea, so
   results published immediately).

---

# POST-REGISTRATION RESULTS ADDENDUM (added 2026-07-12, after the window)

Everything above this line is the registered protocol, unchanged. This
section was appended AFTER the paid window executed and is the study's
results record. Nothing here mints a new confirmatory claim.

## Window execution

One window, dispatched by the owner via `sagen-paid-window.yml` on
2026-07-11 (UTC), completed 2026-07-12. Spend: **$84.75 of the registered
$150 ceiling, 8,488 calls**. Corpus hash verified before any call
(`eba3a75a...`); all 72 scenarios ran. Raw per-call record committed under
`bench/results/window/` and published verbatim in the v2 bundle.

One infrastructure interruption: an ACCOUNT-level API usage limit (not the
study ceiling) killed the first `go` dispatch mid-S2. Per the registered
rerun rule, completed units were persisted and the window resumed by key
after the owner raised the account limit; no unit was re-rolled. Logged as
infrastructure failure, not a protocol deviation.

Two deviations were declared in the committed window config
(`paid-config.mjs`) before the first paid call:

1. **no-temperature-control**: the API tier rejects an explicit temperature
   parameter, so all runs execute at default sampling and the R = 5 runs
   form one cell per model. Direction of bias: makes H4's stability floor
   STRICTER; H4 can fail from this but cannot spuriously pass.
2. **model-ids-pinned-at-window**: the registration named roles and counts
   but not vendor model ids; ids were pinned in the committed config before
   the first paid call (realizer `claude-sonnet-5`, perception
   `claude-sonnet-4-6`, judges `claude-opus-4-8` + `claude-haiku-4-5`).

One instrument gap found by the release verifier and disclosed: realization
fell back to frozen template content on 25 of 960 turns (logged, per the
registered rule), and on 7 of those the TEMPLATE text itself trips the
spurious-callback cue check (the phrase "since we talked" on a non-callback
turn). The affected turns carry the frozen ground-truth surface text
verbatim, so no analysis is affected; the cue list is the instrument at
fault, and the v2 verifier states the exact invariant (953/960 pass, 7
logged template fallbacks, 0 unexplained).

## Registered verdicts (confirmatory family, Holm-Bonferroni, alpha 0.05)

| Hypothesis | Registered criterion | Measured | Verdict |
|---|---|---|---|
| H1 live advantage | SAGEN-300 live beats strongest flat baseline | +0.230 vs rolling summary (95% CI 0.205 to 0.257, p = 0.0001) | **HOLDS** |
| H2 structural gap | 7 structural dims stay uncaptured by transcript | 0 violations across 72 scenarios | **HOLDS** |
| H3 perception tax | live within 0.10 of oracle coverage | tax 0.438 (95% CI 0.416 to 0.460); every bootstrap resample over the bound | **FAILS** |
| H4 perception quality | alignment >= 0.80 AND stability >= 0.90 | alignment 0.257; stability 0.719 | **FAILS** |
| Judge IRR gate | kappa >= 0.70 (two judges, 20-scenario subsample) | kappa 0.253 (n = 400 pairs) | **FAILS** (judged dims stay exploratory) |

Live coverage means (n = 72): buffer 0.140, transcript 0.174, summary
0.268, SAGEN-live-300 0.499, SAGEN-live-2000 0.828. S2 per-model composite
alignment (150 cells each): haiku 0.251, sonnet 0.257, opus 0.273.

**Labeling disclosure (H4).** The Holm table in `live-results.v1.json`
marks H4 "reject H0" because its two-sided bootstrap p only detects that
the alignment mean is far from the 0.80 floor. It is far BELOW the floor,
so the directional registered criterion fails: the registered verdict for
H4 is FAIL. The mechanical output's label is disclosed rather than
re-derived post hoc.

**Judged dimensions.** The kappa gate failed, so the four judged dimensions
(goal-hierarchy, sticky-retention, unknown-tracking, assumption-tracking)
report as exploratory only, exactly as registered. No judged number enters
any confirmatory claim.

## Reading

The architecture's claims held: structured state beats every flat baseline
even with weak perception (H1), the structural gap survives long context
(H2), and all 64 knockout predictions held (S3). The perception layer's
claims failed: live perception costs 0.438 of oracle coverage (H3) and the
paper's informal 0.94 closed-loop alignment measures 0.257 under the
registered composite metric (H4). Honest one-line summary: **the state
architecture is load-bearing and cheap; perception is now the measured,
binding constraint.**

Published surfaces: paper addendum (`/research/sagen/paper#sagen-bench`),
Claims Ledger entries (`sagen-bench-lattice`, `sagen-bench-live-advantage`,
`sagen-perception-tax`, `sagen-perception-measured`), and the v2 bundle
(`/downloads/sagen-bench/v2`).