The Knockout Lattice, Recomputed Right Now
Before the confirmatory run, we filed 64 predictions: remove one SAGEN mechanism, and exactly these capability probes should fail. The pure instrument is small enough to run in a page load. This is not a screenshot of a past result. It just ran, in this tab.
Running the lattice...
Confirmatory Family (Holm-Bonferroni, alpha 0.05)
Every criterion below was written down before the paid window ran. Two hypotheses failed under their frozen bar; that is disclosed here exactly as filed, not softened.
| ID | Question | Registered criterion | Measured | Verdict |
|---|---|---|---|---|
| H1 | Live advantage | SAGEN-300 with live perception beats the strongest flat baseline | +0.230 coverage vs summary (95% CI 0.205 to 0.257, p = 0.0001) | HOLDS |
| H2 | Structural gap | 7 structural dimensions stay uncaptured by a full transcript | 0 violations across 72 scenarios | HOLDS |
| S3 | Knockout lattice | All 64 ablation-by-probe cells behave as predicted in advance | 64/64 on the frozen corpus | HOLDS |
| H3 | Perception tax | Live coverage stays within 0.10 of oracle coverage | tax 0.438 (95% CI 0.416 to 0.460); bound 0.1 | FAILS |
| H4 | Perception quality | Alignment >= 0.80 and run-to-run stability >= 0.90 | alignment 0.257; stability 0.719 | FAILS |
| IRR | Judge IRR gate | Two judge models reach kappa >= 0.7 | kappa 0.253 (n = 400) | FAILS |
Oracle-Mode vs Live Perception
Oracle mode hand-feeds the frozen ground-truth analysis (the ceiling). Live mode makes a real model do the perceiving (the realistic case). The gap between the two columns is the perception tax (H3).
n = 72 scenarios (live); n = 72 scenarios (oracle). Frozen power analysis: N = 66 (alpha 0.05, power 0.9 at delta 0.1).
Does the Blackboard Earn Its Keep?
The knockout lattice removes mechanisms but never removes persistence itself. This free, deterministic ablation does: it compares the persistent engine against a stateless-structured baseline — a fresh engine each turn that sees only that turn’s analysis and injects it, carrying no memory forward. The per-dimension delta separates the coverage that persistence buys from the coverage the typed schema emits for free with zero memory.
| Dimension | Persistence Δ | Reads as |
|---|---|---|
| explicit-goal-identification | +1.000 | needs memory |
| inferred-goals | +0.972 | needs memory |
| topic-pivot-detection | +0.893 | needs memory |
| active-topic-tracking | +0.743 | needs memory |
| goal-priority | +0.681 | needs memory |
| goal-lifecycle | +0.681 | needs memory |
| memory-decay-compression | +0.587 | needs memory |
| callback-detection | +0.000 | free from the schema |
| sentiment-tracking | +0.000 | free from the schema |
| sentiment-urgency | +0.000 | free from the schema |
| entity-type-classification | +0.000 | free from the schema |
| scan-pattern-watchlist | +0.000 | free from the schema |
| token-budget-rendering | +0.000 | free from the schema |
| temporal-transition-ordering | -0.028 | free from the schema |
| transition-type-classification | -0.056 | budget-truncation artifact |
| machine-parseable-output | -0.215 | budget-truncation artifact |
Persistence is load-bearing — it carries goal tracking, topic-pivot detection, cumulative topics, and memory decay. But roughly seven of sixteen dimensions come free with no memory at all (callback and sentiment ride in the per-turn analysis; machine-parseability and budget-fit are format properties). So part of SAGEN’s advantage over flat memory is schema affordance, not knowing more — the honest reading of a coverage number. This is oracle mode; the live-perception ablation is future work.
Alignment by Model
Three pinned models, five runs each, 30 stratified scenarios. Composite alignment against the frozen ground truth (Jaccard on topics/goals/questions, exact match on references and sentiment).
| Model | Composite alignment | Cells | Role |
|---|---|---|---|
| claude-haiku-4-5-20251001 | 0.251 | 150 | sweep |
| claude-sonnet-4-6 | 0.257 | 150 | flagship (H4 gate) |
| claude-opus-4-8 | 0.273 | 150 | sweep |
Deviations and Known Gaps
Registered: S2 temperature-0 and sampled cells kept separate; run-to-run stability measured at temperature 0.
Actual: Current Claude API tiers reject an explicit temperature parameter, so all runs execute at each model's default sampling and the R = 5 runs form ONE cell per model.
Bias: Makes H4's run-to-run stability floor STRICTER (default sampling includes sampling noise that temperature 0 would have suppressed); H4 can fail from this deviation but cannot spuriously pass because of it.
Registered: Model ids frozen at registration.
Actual: The registration text names roles and counts but not vendor model ids; ids were pinned in this committed file before the first paid call.
Bias: None on outcomes; a completeness gap in the registration transcription, disclosed.
The Holm table marks H4 “reject H0” from a two-sided bootstrap p that only detects distance from the 0.80 floor. The alignment mean sits far BELOW the floor, so the directional registered criterion fails: the verdict for H4 is FAIL, as shown above.
953 of 960 realized turns passed the frozen fidelity checks outright; 7 fell back to logged, frozen template text after 3 failed rewrite attempts, and on those 7 the template itself trips a spurious-callback cue. An instrument gap in the cue list, not a data problem: the affected turns carry ground-truth surface text verbatim.