2026-06 · DevOps
Production went down with MIDDLEWARE_INVOCATION_FAILED. The bug was one setting. What made it expensive was everything around it, so every gap became a checked-in command, a CI gate, and a reproduction against a pre-fix checkout.
5
tools from four gaps
n/a
minutes to diagnose, before the harness: no metric captured
53
merges in DevOps, CI & Quality to date
Every route through the edge middleware returned a 500 while routes outside its matcher stayed healthy. The Sentry build plugin auto-injects a middleware wrapper into the built edge bundle, but the site's webpack config deliberately aliases the Sentry package away in that bundle, so the injected call was undefined at runtime. That path only runs when the Sentry token is present at build time: true on the host, never locally.
The failing code path was structurally invisible to local builds. Recovery had to be safe to run from an agent's own shell, which the first attempt, a kill by process name, had taken down with it. And the fix had to be proven, not asserted.
Do not write a postmortem nobody reruns. Each gap became a command: a version endpoint that returns the deployed sha and presence booleans for every build-time variable, excluded from the middleware matcher so it answers while the middleware is crashing; a declarative probe matrix with disciplined exit codes; a parity build that fills the env manifest with safe dummies so the host-only path runs locally, boots, and probes; PID-file server lifecycle; and a per-PR workflow that runs the parity build and probe with no secrets.
Rejected
A postmortem document
Nobody reruns a document. Each gap is a command in the repository, and the class of failure has a CI gate.
Kill the server by process name
It matched the agent's own shell and killed the operator mid-incident. Servers start in their own process group under a PID file and stop by signaling exactly that group.
The version endpoint with its matcher-exclusion test, the probe matrix, the parity build, PID-file lifecycle, and the per-PR guardrail, plus the proof: on a pre-fix checkout the gate reproduces the exact 500; on main it is green.
The guardrail then failed for its own reasons: the parity build ran out of memory on the CI runner and took six follow-up merges to settle (a heap raise, a warm cache, path gating, serialized static generation, a smaller heap ceiling, no source maps). A gate that is red for the runner's reasons trains people to ignore it, so none of those fixes was optional.
fix(auth): stop edge middleware from 500ing the whole site (MIDDLEWARE_INVOCATION_FAILED)
#1011 · 2026-06-11 · bugfix
🚨 hotfix: prod outage root cause — Sentry middleware wrapper (MIDDLEWARE_INVOCATION_FAILED)
#1015 · 2026-06-11 · feature
feat(ops): ops harness — version endpoint, prod-parity build, probe, serve/stop, CI guardrail
#1016 · 2026-06-11 · feature
ci: raise prod-parity build heap to 6 GB to stop build-and-probe OOMs
#1460 · 2026-06-20 · infra
ci: fix the prod-parity OOM by serializing static generation under CI
#2255 · 2026-07-27 · bugfix
ci(prod-parity): drop source-map work in parity CI so the build fits the runner
#2269 · 2026-07-28 · infra
Checked at build against the published corpus.
www.jakelawrence.xyz/research/case-study-library/ops-harness
Cite: Lawrence, J. (2026). From one outage to a standing ops harness. jakelawrence.xyz case study library.
I design and ship AI tools, full-stack apps, and data pipelines — end to end, to production. Tell me the problem in a sentence; I'll give you an honest read on fit within a day.
Work with me →