# SAGEN-Bench public release v2

Data, results, instruments, the registered paid window's raw record, and a
standalone verifier for **SAGEN-Bench**, the pre-registered evaluation of
the SAGEN situational-awareness architecture
(paper: https://jakelawrence.xyz/research/sagen; registration:
https://doi.org/10.17605/OSF.IO/S3GFM).

v2 supersedes v1 by adding the registered paid window (executed 2026-07-11
to 2026-07-12) and its instruments. v1 stays published unchanged as the
pre-window snapshot the registration pointed at.

## Verify (Node >= 20, no dependencies)

    node verify.mjs

Re-derives everything v1 verified (corpus hash, corpus regeneration from
the frozen seed, the 64/64 S3 knockout lattice, S1 oracle coverage, the
power analysis) AND the full registered analysis from the raw per-call
record in window/: realization fidelity, per-scenario alignment, live
coverage means, H1 through H4 with their seeded bootstrap CIs, the Holm
family, the judge kappa gate, and the spend ceiling.

## Registered verdicts

- H1 HOLDS: live SAGEN at its 300-token budget beats the strongest flat
  baseline (rolling summary) by +0.230 coverage (95% CI 0.205 to 0.257,
  p = 0.0001, Holm-adjusted).
- H2 HOLDS: 0 structural-gap violations across 72 scenarios.
- S3 HOLDS: 64/64 knockout-by-probe cells predicted in advance and
  observed as predicted.
- H3 FAILS: perception tax 0.438 (95% CI 0.416 to 0.460) vs the registered
  0.10 bound; live 0.499 vs oracle 0.937 at the 300-token budget.
- H4 FAILS: flagship perception alignment 0.257 vs the 0.80 floor;
  run-to-run stability 0.719 vs the 0.90 floor.
- Judge IRR gate FAILS: kappa 0.253 < 0.70 (n = 400); judged dimensions
  stay exploratory, never confirmatory.
- Spend: $84.75 of the registered $150 ceiling, 8,488 calls.
- Disclosure: the Holm table marks H4 "reject H0" from a two-sided
  bootstrap p; the alignment mean sits far BELOW the floor, so the
  directional registered criterion fails and the verdict is FAIL.

## Contents

- corpus.v1.json + manifest.v1.json - the frozen confirmatory corpus
  (72 machine-generated scenarios; sha256 in the manifest)
- results/ - committed results for the deterministic arms
  (S3 lattice, S1 oracle coverage, power analysis)
- window/ - the paid window's raw record, VERBATIM: realized surface text,
  per-scenario live scores + perception dicts, the S2 stability sweep,
  the judge grid, the spend ledger, and the final analysis
  (live-results.v1.json). Model-written text is untouched raw data.
- code/ - the pure instrument modules, now including the paid-window
  instruments (alignment metrics + registered statistics, realization
  checks, live pipeline, the frozen window config)
- protocol/ - the binding pre-registration document, with the
  post-registration results section clearly marked as such
- SHA256SUMS - hashes of every file in this bundle

License: CC0 1.0 Universal. No human subjects: every arm observes software.