SAGEN: Situational Awareness for Generative Agents

Introduction

The deployment of large language models as interactive agents has exposed a structural gap between model capability and operational coherence. A model that can reason about complex plans within a single context window routinely loses the thread of those same plans across sequential interactions. This is not a failure of intelligence. It is a failure of architecture.

The dominant paradigm treats agent memory as information retrieval: store facts, embed them, retrieve relevant chunks at query time. RAG and its derivatives address the question “What does the agent know?” They do not address: What is the agent trying to do? What just changed? What should it pay attention to? What has it tried before?

These are questions of situational awareness — a concept well-studied in human factors research but underexplored in LLM agents.

The core design principles are:

1. Modularity. Six cognitive modules that can be composed, extended, or replaced independently.
2. Domain agnosticism. A plugin-based adapter pattern that separates domain-specific perception from domain-general state management.
3. Compression over accumulation. Inspired by ACT-R’s memory decay, prioritize recent, salient, and failure-associated state.
4. Injection-native. State rendered as compact, structured text designed for LLM context windows.

Architecture

SAGEN’s awareness state is organized as a blackboard: a shared data structure that multiple cognitive modules read from and write to. The unified state object, AwarenessState, contains six modules and two coordination fields.

SAGEN architecture. Six cognitive modules reside on a shared blackboard. The Update Engine coordinates the Observe–Update–Inject loop, using a Domain Adapter to translate between raw observations and structured state updates.

Module 1: Goal Graph

The Goal Graph maintains a directed acyclic graph of agent objectives. Each goal has a lifecycle (active, completed, abandoned, blocked, deferred) and a provenance (explicit, inferred, emergent, or spawned).

Module 2: Trajectory

Records state transitions as a temporal sequence — the agent’s episodic memory. Each transition is typed: progress, reversal, pivot, discovery, external event, failure, or branch. The module implements ACT-R-inspired compression: recent transitions are detailed, routine progress fades, and failures/reversals/discoveries are sticky.

Module 3: World Model

An entity-relationship graph of the agent’s environment. Entities have types, mutable state, and affordances. The World Model also tracks assumptions (unverified beliefs) and unknowns (identified but unanswered questions) — giving the agent explicit access to its own epistemic boundaries.

Module 4: Self Model

The agent’s understanding of its own capabilities, limitations, resources, authority boundaries, and failure history. The can_i(action) method provides a pre-flight check across three dimensions: capability, authorization, and resourcing.

Module 5: Attention Priorities

A priority queue of threats, opportunities, anomalies, and transitions. Each item has an urgency score and optional TTL. The module also stores persistent scan patterns — watchlist templates defined by the domain adapter.

Module 6: Interaction Protocol

The operational contract: communication style, output format, collaboration mode (autonomous, supervised, collaborative, advisory), escalation rules, and hard constraints.

Unified State and the Tick

All six modules reside on the AwarenessState blackboard alongside a global step counter and timestamp. Formally, the awareness state at step $t$ is:

where $G$ is the Goal Graph, $T$ the Trajectory, $W$ the World Model, $M$ the Self Model, $A$ the Attention Priorities, and $P$ the Interaction Protocol.

The Update Engine

The Update Engine coordinates the three-phase Observe–Update–Inject (OUI) loop that transforms raw observations into structured state and then into LLM-consumable context.

Phase 1: Observe

The adapter’s parse_observation method receives raw input and the current state, returning a structured dictionary of entities, goals, attention items, assumptions, unknowns, and trajectory events. This phase is entirely domain-specific.

Phase 2: Update

The engine applies parsed observations using upsert semantics for entities and append semantics for goals, attention items, and lists. Expired attention items are removed based on TTL. This phase is domain-agnostic.

Phase 3: Inject

The adapter renders the current state as a compact text block sized to a token budget. The injection string is structured but natural-language-readable, wrapped in <sagen> tags.

<sagen>
ACTIVE GOALS:
  [explicit] Learn Python (p=0.7)
  [explicit] Build a web scraper (p=0.7)
  [inferred] Answer: What library for scraping? (p=0.6)

ATTENTION:
  [opportunity] Callback to earlier topic
  [transition] Topic shift: {'cooking'} -> {'Python'}

ACTIVE TOPICS: Python, web scraping

TRAJECTORY:
  [progress] Continuing: {'Python', 'web scraping'}
  [pivot] Pivoted from {'cooking'} to {'Python'}
</sagen>

Domain Adapters

Each adapter implements six methods that encapsulate all domain-specific logic:

Cross-Domain Comparison

The two adapters share zero entity types (except “concept”), zero scan patterns, and produce domain-appropriate injection formats — yet both run on the identical engine and schema without modification.

Evaluation

State Evolution (Conversation Domain)

Baseline Comparison

SAGEN captures 98.5% of evaluated information dimensions vs. 20.5% for rolling summaries and 10.5% for raw buffers. The key finding is not SAGEN’s absolute score but the structural gap: 16 of 20 dimensions are captured by none of the baselines. These are capabilities — goal hierarchy, typed transitions, epistemic boundary tracking, urgency scoring — that require explicit architectural support.

These percentages come from one scripted 4-turn demo. The pre-registered follow-up benchmark (see the addendum) re-measured the same design over 72 machine-generated scenarios: 93.7% in oracle mode and 49.9% under live perception at the same 300-token budget.

Qualitative Response Comparison

Turn 3 (Callback): Under rolling summary context, the LLM answers the factual question competently but treats it as standalone. Under SAGEN injection, it recognizes the callback, connects it to the original goal, and calibrates its response to the user’s stated skill level.

Turn 4 (Frustration): The summary-context response jumps directly to troubleshooting. The SAGEN-context response first acknowledges the user’s emotional state, calibrates its tone, and adopts a patient, step-by-step approach.

Closed-Loop Evaluation

LLM-generated perception achieves 0.94 alignment with hand-crafted analysis across all four turns. All key cognitive dynamics (pivot detection, callback recognition, frustration monitoring) are preserved regardless of whether analysis is hand-crafted or LLM-generated.

That 0.94 is an informal, single-scenario figure. Under the pre-registered composite metric, measured across 450 perception cells, flagship alignment came out at 0.257 (see the addendum). The registered measurement supersedes this paragraph.

State Serialization

Split-session experiment: Session 1 processes turns 1–2, state is serialized to JSON (4.7 KB), Session 2 processes turns 3–4 from restored state. The restored engine produces exactly identical state and injection output to continuous execution. The blackboard is the state, and the state is the JSON.

Discussion

Perception Is the Binding Constraint

Earlier drafts of this paper argued that the perception stage “does not need to be perfect,” on the strength of a 0.94 alignment score between LLM-generated and hand-crafted analysis. The pre-registered study contradicts that, and the contradiction is the most useful thing in the paper. Two registered perception hypotheses failed: H3 (the live-vs-oracle perception tax is 0.438 against a 0.10 ceiling) and H4 (measured alignment 0.257 against a 0.80 floor; run-to-run stability 0.719 against a 0.90 floor). Moving from ground-truth analysis dictionaries to a live perception model costs nearly half of SAGEN’s coverage, and the model does not produce the same analysis twice.

Three qualifications are owed, and none of them rescue the original claim. First, the measurement instrument itself failed its registered reliability gate — two independent judge models reached $\kappa = 0.253$ against a pre-registered floor of 0.70 ($n = 400$). The honest reading of the 0.257 figure is therefore not “perception is 25.7% accurate” but “perception quality is low and we do not yet have a reliable instrument for saying how low.” Second, part of the tax is a rendering artifact, not a perception failure: at a 2000-token budget live coverage recovers to 82.8% from 49.9% at 300 tokens, because over-extracted entities compete for a fixed injection budget and push high-salience items below the truncation threshold. Salience-weighted truncation and entity decay are therefore the first place to look for recovered coverage, and neither requires a better perception model. Third, the architecture is unaffected — the state layer’s registered claims (H1, H2, the full knockout lattice) held under exactly the degraded perception the other hypotheses failed on. SAGEN degrades gracefully in the specific sense that it keeps beating flat memory and keeps detecting pivots, callbacks, and sentiment escalation. What it does not do is degrade cheaply. Perception is where the next unit of engineering effort returns the most.

Classification as Infrastructure

SAGEN’s six-module decomposition is not a neutral description of cognition. It is a taxonomic commitment that determines what the agent can represent and therefore what it can reason about. An agent without a Trajectory module cannot distinguish a topic pivot from a topic abandonment. An agent without typed Attention cannot allocate urgency differentially. The taxonomy is not scaffolding; it is load-bearing structure.

Limitations

1. No learning. SAGEN maintains state but does not learn from it in the ML sense.
2. Perception quality and cost. The registered evaluation shows live LLM perception costs 0.438 of coverage relative to oracle analysis (H3, failed) and reaches only 0.257 alignment against a 0.80 floor (H4, failed), on top of the extra inference call each observation requires. Perception is the binding constraint on the system as deployed.
3. Single-domain assumption. One adapter at a time; multi-domain composition is not yet specified.
4. Limited evaluation scope. Single 4-turn scenario; the systematic end-to-end evaluation has since run as the pre-registered SAGEN-Bench (see the addendum below).
5. No confidence channel. The schema carries no per-item confidence; the p= and urgency= scores are fixed constants, not calibrated confidence, applied identically to correct and to hallucinated extractions. Given H4’s 0.257 alignment this is a material design gap: structured state renders low-alignment perception as confident typed assertions. Whether that amplifies perception error or degrades gracefully is an open, falsifiable hypothesis this study does not test — it needs a downstream-decision comparison, and H1 held under live perception, so the sign is unknown.
6. Measurement of perception is itself unresolved. The registered inter-rater-reliability gate on two judge models failed ($\kappa = 0.253$ against a 0.70 floor), so perception-alignment scores — including our own earlier 0.94 — should be treated with suspicion until an instrument with a demonstrated reliability floor exists.

SAGEN-Bench: the Pre-Registered Evaluation

Limitation 4 has been discharged. SAGEN-Bench is a pre-registered evaluation of this architecture (OSF registration DOI 10.17605/OSF.IO/S3GFM): 72 machine-generated, hash-frozen scenarios; a mechanical recoverability rubric; a knockout lattice with all predictions filed before execution; and one paid window (live LLM perception, $84.75 of a registered $150 ceiling, 8,488 calls) run after the registration went public. Everything below re-derives from the raw record in the public release bundle, and the Claims Ledger re-verifies the deterministic results live in your browser.

What the registered numbers correct in this paper

Two figures printed above do not survive the registered instrument. The 98.5% coverage (Section 5) was measured on one scripted 4-turn demo; on the 72-scenario frozen corpus the same design scores 93.7% in oracle mode at the 300-token budget, and 49.9% once a real model does the perceiving (82.8% at a 2000-token budget). The 0.94 closed-loop alignment was informal and single-scenario; the registered composite metric measures 0.257 for the flagship perception model, with haiku at 0.251 and opus at 0.273. The perception layer’s claims failed (H3, H4); the structural claims held (H1, H2, with the knockout lattice standing as a conformance and separability check rather than independent evidence of correctness).

Two cautions govern how to read H1. First, coverage measures whether a typed information slot is present and recoverable, not whether its content is correct; at 0.257 alignment much of the emitted state is wrong, so the advantage is a representational-affordance result, not evidence of superior understanding. Second, under live evaluation the rolling-summary baseline is scored from the frozen ground-truth analysis dicts while SAGEN runs on live perception, so the +0.230 margin is conservative and structure-driven — SAGEN wins by emitting typed fields the summary structurally lacks, not by knowing more. The honest summary: perception, not state architecture, is the binding constraint, and whether confidently-typed but low-alignment state helps or harms a downstream agent is an open question this study does not resolve.

Does persistence earn its keep? An oracle-mode ablation

The knockout lattice removes individual mechanisms but never removes persistence itself — every arm keeps accumulating state. To isolate what the blackboard actually contributes, we ran a free, deterministic oracle-mode ablation on the same 72 scenarios, comparing the persistent engine against a stateless-structured baseline: a fresh engine each turn that sees only that turn’s analysis and injects it, carrying no memory forward.

Persistence is load-bearing: it adds +0.338 mean coverage (0.937 vs 0.599), concentrated exactly where accumulation is required — goal tracking, topic-pivot detection (+0.89), cumulative topics (+0.74), and memory decay. But the split cuts both ways: roughly 7 of 16 dimensions are recovered with zero memory — callback and sentiment flags ride in the per-turn analysis, and machine-parseability and budget-fit are format properties (the memoryless injector even edges out the accumulated engine on machine-parseability at 300 tokens, because the fuller state overflows the budget and truncates). So part of SAGEN’s coverage advantage over flat memory is schema affordance — “having the slots” — not knowing more. This is oracle mode; the live-perception ablation is future work.

Disclosures

The registered Holm table marks H4 “reject H0” because its two-sided bootstrap p only detects that the alignment mean is far from the 0.80 floor; it is far below it, so the directional criterion fails and the registered verdict is FAIL. Two deviations were declared in the committed window config before the first paid call: no explicit temperature control (the API rejects the parameter; this makes the H4 stability floor stricter, so H4 can fail from it but not spuriously pass), and vendor model ids pinned at window-open rather than in the registration text. One infrastructure interruption (an account-level API usage limit, not the study ceiling) paused the window mid-S2; completed units were persisted and resumed by key, never re-rolled, per the registered rerun rule. Realization fell back to frozen template text on 25 of 960 turns (logged), and on 7 of those the template itself trips the spurious-callback cue check: an instrument gap in the cue list, disclosed in the protocol addendum, with no effect on any analysis.

Full protocol, frozen corpus, instruments, raw per-call JSONL, and a dependency-free verifier that re-derives every number: the results page (live recompute) · downloads/sagen-bench/v2 · OSF registration · Claims Ledger · runnable testbed