What exists that did not

Fifteen entries, counting today’s two. agent-harness: 952 lines of Python at version 0.3.4, 292 tests green in under a second, CI across CPython 3.10 through 3.13 on every push, two architecture decision records in docs/adr/, a golden trace pinning the executor’s canonical behavior, a 250-seed invariant sweep, and a trace inspector that reads any run back as a timeline. As of today, edge-bench: the benchmark protocol published on August 4 turned into code that enforces it: 232 lines, 12 tests, including one that fails if anyone ever adds a mean(). As of this audit, 113 commits across the site and agent-harness, each one small enough to read.

What the method bought

The compounding was the point, and it showed up on schedule. The golden trace, built on August 5 before anything needed it, is what made the August 9 refactor boring instead of risky. The failure taxonomy, argued on paper in the spine post, became the literal dispatch of the executor. The invariant sweep found no bugs and forced an ADR anyway. The best entries came from the worst weeks: a three-day GitHub outage sat on this site’s deploys and produced the CI post’s whole argument about handing guard duty to infrastructure that had just failed me.

What did not happen, and whose fault it is

The roadmap’s oldest open item is the live-model trace. The run kit has needed one key and one human since August 6, which is twenty-four days, and both belong to me. The Edge AI lane now has a protocol and a harness and still no numbers, because the phone they require is in my pocket and has stayed there. The pattern across both gaps is not subtle: the tooling half of this operation has out-shipped the half that holds the credentials. The audit records this the way the bench records an empty cell: reason attached, nothing silent.

One habit needs correcting too, and it is subtler than an errand. The roadmap struck “LLM-backed planner” as done on August 5 on the strength of tests that script the transport seam, while the live proof sat unstruck one line below. Those are different claims. Verified in the lab and proven in the field deserve different marks, and a roadmap that blurs them is borrowing credibility from its own future. When the live trace lands, the strike-through earns its ink; until then, the honest reading of that line is “built, not yet witnessed.”

The next four weeks

Live trace first; it unblocks the post this log has owed longest. Then the CPU column of the benchmark matrix, on the two devices the protocol promises to name. C++ enters where ADR-001 said it would: when a measurement points at a hot path, and not before. The cadence stays what it has been, entries when there is evidence, quiet when there is not.

Ledger

Twenty-seven days. Fifteen entries. Two code repos, 1,184 lines, 304 tests, every one green this morning. Two ADRs. One outage survived. Two items blocked, both on the same person, who keeps writing that down instead of doing something about it.

The measured indictment

Before proposing anything, the authors measure what existing serving systems did with that memory, and the number deserves to be quoted exactly: 61.8 to 79.6 percent of KV cache memory wasted. Only a fifth to a third of it held actual token state. The waste has three named sources, and anyone who has run a memory allocator will recognize all of them: slots reserved for tokens not yet generated, internal fragmentation from provisioning every request for its maximum possible length, and external fragmentation from the allocator itself. Serving systems were burning most of their scarcest resource on bookkeeping failures that operating systems classified, named, and solved decades ago.

The old answer, imported honestly

The fix is virtual memory, applied without embarrassment. Chop the KV cache into fixed blocks of sixteen tokens, keep a block table per sequence mapping logical to physical, allocate on demand, share read-only blocks across sequences, copy on write when a shared block diverges. Beam search stops copying whole caches and starts sharing prefixes. Every sentence of that description could have come from an OS textbook, and the paper is candid about the lineage.

What makes it architecture rather than borrowing is where the cost went. Contiguity was never a law of nature; it was a contract the attention kernel imposed on the storage layer. The authors moved that constraint out of the contract and paid for it inside a rewritten kernel that walks the block table. Utilization became near-total, and the price is indirection on every attention read: 20 to 26 percent higher attention-kernel latency by their own microbenchmark, diluted to little at system level because attention is one operator among many. Naming the price is what makes the trade honest, and the outcome at system level is 2 to 4 times the throughput of FasterTransformer and Orca at equal latency, widening with longer sequences and larger models.

What survives the trip to the edge

My lane is the device, so the honest question is which of these gains travel. The uncomfortable answer: the headline one mostly does not. vLLM’s throughput wins come from packing more concurrent requests into a fixed pool of GPU memory, and a phone has exactly one user. Batching gains that justify the indirection in a datacenter have nothing to batch on-device.

What does travel is the diagnosis. On-device KV cache competes with the entire OS for RAM, so the fragmentation and over-reservation the paper measured are, proportionally, a worse crime on an 8-gigabyte phone than on an 80-gigabyte GPU. Block-based allocation still belongs there, for different reasons: long contexts that grow without pre-reserving worst case, and a shared system-prompt prefix kept once across an app’s sessions rather than per conversation. Same mechanism, different justification. Importing the mechanism without re-deriving the justification is how edge runtimes end up carrying datacenter machinery they cannot afford, which is a mistake I intend to measure rather than assert when the benchmark protocol gets its numbers.

Ledger

This entry ships no code of its own; it is the analysis lane doing its job. Every number above is from the paper itself, checked against it today: 800 KB per token, 61.8 to 79.6 percent measured waste, blocks of sixteen, 20 to 26 percent kernel overhead, 2 to 4 times throughput at equal latency. The Edge AI lane, meanwhile, got its protocol turned into runnable code, so the only thing still standing between that lane and numbers is a phone in my pocket.

python -m agent_harness.inspect out/run.jsonl

What it prints

The run, as a timeline. This is real output from a run whose tool flaked once on its second step:

run.jsonl · 11 transitions · 3 checkpoints · final: done

   1  step 0  plan → act             step planned                 tool=echo
   2  step 0  act → verify           result produced
   3  step 0  verify → checkpoint    verified                   checkpoint
   4  step 1  plan → act             step planned                 tool=echo
   5  step 1  act → act              transient error, retrying  retry  error=network hiccup
   6  step 1  act → verify           result produced
   7  step 1  verify → checkpoint    verified                   checkpoint
   8  step 2  plan → act             step planned                 tool=echo
   9  step 2  act → verify           result produced
  10  step 2  verify → checkpoint    verified                   checkpoint
  11  step 3  plan → done            goal met

retries: 1 · escalations: 0 · checkpoints: 3
replay: clean · seq gapless · every checkpoint verified

The marks in the right column come from a small table keyed by state pair. An act → act edge is a retry because the state machine says it is, not because a string in the reason field looked retry-ish. The taxonomy classified the failure when it happened; the inspector only has to read the classification back.

Tool-free by construction

The inspector imports the store and replay(), and nothing else. It has no way to execute a tool, so pointing it at a trace can never change anything, and the verdict on the last line comes from the same replay() the test suite trusts, run over the same file. A trace that fails those invariants prints CORRUPT TRACE, names the violation, and exits 2, so a script or a CI job can gate on the story holding together.

What it refuses to show

Control flow, not payloads. Checkpoint results stay in the file; the timeline shows what the machine did, never what the tools returned. Payloads are where size and privacy live, and a viewer for them is a different tool with different problems. The story of a run — planned, acted, retried, verified, escalated — fits in eighty-four lines of reader precisely because it declines to be a browser.

Ledger

952 lines of harness, of which the inspector is 84. Four new tests bring the suite to 292, still green in half a second. Version 0.3.4. The reader was built one errand ahead of the first real trace it exists to read; the README’s count on that errand stands at fifteen days.

Status: accepted. Date: 2026-08-17. Decider: me, having already shipped the code on Friday and owing it a defense.

Context

The executor promises invariants: a terminal state always; a gapless, strictly increasing trace; a checkpoint one for one with a recorded verification; no tool called past its budget; every trace replays. Those needed testing against behavior I did not hand-pick, which means something has to generate planners, tools, and verifiers with conduct I never chose. Two candidate generators. The repo has zero runtime dependencies and one test dependency, and its whole purpose is to be read as a reference in an afternoon.

Options

A. Hypothesis. Strategies for planners, tools, and verifiers, @given over the executor, shrinking on failure. Battle-tested, and the shrinker is real engineering I would not want to write myself.

B. Seeded stdlib random. random.Random(seed) builds each world, @pytest.mark.parametrize("seed", range(N)) runs them all, and a failing seed number is the whole reproduction.

C. Both. Hypothesis for depth, seeds for the fast CI lane. Two mechanisms for one job.

Decision

Option B. The argument that settled it: the failure I most need to defend against is a reader not trusting the test, and every line of framework between the reader and the executor costs trust. The sweep is 169 lines a stranger can read top to bottom and see exactly what gets generated and exactly what gets asserted. Nothing is imported that a Python programmer has to go learn first.

Shrinking is the real thing given up, and I want to be precise about why that is bearable here rather than pretend it costs nothing. A failing seed in this codebase yields a trace of a few dozen transitions at most, each carrying its reason string. That is already a counterexample small enough to read unminimized. Shrinking earns its keep when the failing input is large and mostly irrelevant; the executor’s inputs are neither.

Consequences

Easier: widening the sweep is one number; the CI file stays at 29 lines; the test file doubles as the plainest statement of the invariants in the repo. Harder: no automatic minimization; the distributions are hand-written and exactly as hostile as I made them, no more; 250 draws cover what 250 draws cover, and a strategy-aware engine would explore differently.

Revisit when either of two things happens. A seed fails and its trace is too large to read, which is the day shrinking stops being a luxury. Or the generated worlds need structure the hand-rolled generators make ugly: nested plans, stateful tools, dependent draws. Either one justifies Hypothesis as an additional dev dependency, kept out of the default CI job so the badge keeps meaning what it means today.

Why the record moved into the repo

ADR-001 lived only on this site until today. A decision record that sits outside the codebase it governs is a blog post with a template; the next person to open the repo has no reason to know it exists. Both records now sit in docs/adr/, numbered, with an index that states the one rule for the folder: never rewrite. A superseded record keeps its place, changes its status, and points forward. Hindsight gets a new number.

Ledger

Two ADRs in the repo, 147 lines between them and their index. Version 0.3.3. The roadmap’s ADR item is struck; the live-run item is not, and I notice the README has said “it needs a key and a human” since August 6, which is eleven days. The key is mine. So is the human.

The world the seed builds

Each of 250 seeds constructs a small hostile world. One to four tools, some of which demand a scope the run was never granted. Tool bodies that flake with a TransientToolError, refuse with a permanent one, or reject their own work outright, at fixed odds per call. A verifier that is a weighted coin. A planner drawing random steps toward a random goal. And a human with finite patience: zero to three resumes, then abort.

The executor runs each world with a store attached, and then the invariants get their say. The run ends in DONE or FAILED, never anywhere else. The transition sequence is gapless and strictly increasing. A checkpoint exists exactly when a verification was recorded, one for one. No tool is ever called past its declared budget. And replay() reads the file the run left behind and reaches the same conclusions without executing anything. Until now replay had only ever been shown tidy runs and hand-forged corruptions; these 250 traces arrive full of denials, retries, escalations, and aborts.

Why a seed and not a framework

Hypothesis would do this with more sophistication, and its failure shrinking is real value I am walking away from. What a seed gives back is a perfect reproduction for free: if seed 217 fails, you run seed 217, forever, on any machine. It also costs nothing: no new dependency, no change to the CI file whose 29 lines I bragged about four days ago. For a reference implementation whose whole argument is a small dependency surface, that trade goes one way.

Randomness got a veto on everything except termination. A generated human that resumes forever would earn a run that runs forever, which is the design working as stated — the human is a state, and states can loop. So every generated human has finite patience, and a tripwire fails any seed whose planner is consulted ten thousand times. The sweep is allowed to be hostile. It is not allowed to be eternal.

What it found

No invariant violations. 250 seeds, zero counterexamples, and I will not pretend that proves the executor correct; it proves the executor survives 250 worlds drawn from the distributions I wrote, which is a smaller and more honest claim.

What writing it actually produced was precision. Stating invariants formally forced distinctions the prose versions blurred. One became a new targeted test on the spot: the permission gate sits before the call meter, so a denied call is never billed against the tool’s budget. True in the code all along, pinned by nothing until today. A second test sharpens an old pin: the suite already checked that a tireless planner triggers the step-budget escalation; it now also checks what such a run banks on the way out, exactly max_steps verified checkpoints and not one more.

Ledger

868 lines of harness, unchanged. One new 169-line test file, four test functions, one of which carries all 250 seeds. The suite pytest collects went from 35 to 288 and still finishes in half a second. Version 0.3.2, and the README’s rules list gained its eleventh entry: the invariants hold under behavior nobody chose.

Twenty-nine lines, nothing clever

The workflow is 29 lines: checkout, set up Python, pip install pytest, python -m pytest -q. The suite finishes in a sixth of a second, so the job’s wall clock is nearly all provisioning. A dependency cache could shave seconds off that. It would also add the one failure mode CI grows entirely on its own, staleness, to save time nobody is waiting on. Everything a workflow does beyond what the tests need is a place for the workflow itself to fail, and I had my fill of CI failing for its own reasons last week.

The matrix audits the metadata

pyproject.toml claims requires-python = ">=3.10". Until today nothing checked that claim: the suite ran on whichever single interpreter my machine had, so support down to 3.10 was an opinion extended on credit. The matrix now runs the full suite on CPython 3.10 through 3.13 — 140 test executions per push. If a future commit leans on syntax the older interpreter lacks, the 3.10 job fails and the metadata stops being aspirational. Declared support that no runner exercises is just a comment with better formatting.

One more deliberate choice: permissions: contents: read. The workflow can read the repo and do nothing else. A test runner that can push is one compromised dependency away from being an author. The two actions it uses are pinned to commit SHAs rather than version tags for the same reason the token is read-only: a tag can be moved by whoever controls it, and I would rather review an upgrade than inherit one.

About handing this job to Actions, of all systems

Four days ago GitHub Actions was still sitting on this site’s deploy queue, three days into an outage; the commit history of the site repo records the fight. Giving the same system guard duty over the tests calls for a sentence of justification, so here it is: the alternative is my laptop, and my laptop’s word is exactly what this post retires. When Actions fails, it fails in a log anyone can read. When my machine lies about what passed, nothing says anything.

What the badge does not prove

It proves the tests that exist ran and passed, on four interpreters, as of the last push. It says nothing about whether those tests pin anything that matters. That burden stays where it has been all along: on the golden trace that names any transition that moves, and on a suite where each test holds one rule from the design. The badge is evidence the evidence is alive. Its quality still has to be argued, which is what the rest of this log is for.

Ledger

868 lines of harness, 29 lines of workflow, 35 tests, four interpreters, 140 executions per push. The roadmap’s next unstruck item still needs a key and a human — the run kit has been sitting in examples/ since the lens post, and the machine half of that pair is not the one holding things up.

The lens post ended by admitting two taxonomy entries sat unemployed, with a note that I’d rather stay honest than invent work for them. What follows is not invented work. The refactor was sitting there waiting to be seen: _act_and_verify is now one try block, and every branch of control flow is an except clause on an exception from errors.py. The permission gate raises. The budget raises. So, at last, does the verifier. The dispatch is the taxonomy, arm for arm.

The part that makes this publishable

Rewiring the guts of a state machine is routine work. Proving that nothing observable moved is the part worth writing down, and the proof was already sitting in the repo: the golden trace from the replay post. The shipped happy_path.json pins every transition of the canonical run, in order, with reasons and fields. If this refactor had shifted a single edge of the state machine, that test would have failed and named the transition that moved.

It didn’t move. Thirty-one pre-existing tests, untouched, all green, the golden among them. Building the regression harness before I needed it felt close to process theater on August 5. Today it turned a risky internal rewrite into a boring one, which is the highest compliment a refactoring can receive.

One rule got wider, on purpose

Old behavior: the taxonomy applied only when the machinery raised it. New behavior: it applies no matter who raises it. A tool that throws VerificationFailure from its own body now gets exactly the treatment a failed verifier gets — retry within budget, then escalate. The new test has the tool announce “i checked my own work and it is bad,” and the executor answers with two retries and an escalation, per the contract. Self-rejecting tools were never in the design. The dispatch handles them anyway, because it no longer cares where an exception was born.

The call-budget path also picked up its first behavioral test. That path always worked; nothing pinned it. Now something does.

Ledger

868 lines, 35 tests, and every leaf of the taxonomy now has a raiser whose job it is: the budget raises BudgetExhausted, the verifier raises VerificationFailure, tools own the transient and permanent pair, and nothing’s fate is decided by a lonely if. The two-day gap before this entry was GitHub’s outage sitting on my deploy queue; the record of that fight lives in the site repo’s commit history, where records belong.

The design question that actually matters

Compaction sounds like a summarization problem. It isn’t, mostly. The hard question is what gets compacted, and the wrong answer ruins everything this harness stands for. If crossing the watermark rewrites the checkpoint history, then replay lies, the crash-resume file lies, and the append-only trace I spent a whole post defending becomes append-mostly.

So the rule, now enforced in code and tests: compaction is a lens, not an eraser. When ContextBudget trips, the compactor shrinks the view handed to the planner. The checkpoint list keeps every entry. The store keeps every byte. Replay still verifies the full history, and there’s a test that runs a compacted run against replay() to prove the record came through intact. The model’s context is negotiable; the record is not.

What the planner sees instead

FoldingCompactor folds everything but the last few checkpoints into one digest entry: which steps it swallowed, which tools ran, and a bounded scrap of each result. The planner still knows the shape of the past, just not every byte of it. The folding is deterministic because the core must never need a model just to function. A model-written summary can implement the same callable later, and the executor won’t know the difference — that’s the planner trick applied twice.

Known limit, stated plainly: the digest keeps a scrap per folded step, so it grows with history. Run long enough and even the compacted view outgrows the budget. That’s what the escalation path is for, and it’s the first thing a smarter compactor should fix.

The unit estimator is serialized-length-over-four, and the code labels it an approximation, because it is one. The budget needs a monotonic size signal, not tokenizer-exact counts. Set the watermark with headroom, well under the model’s hard limit: compaction should run on my schedule, not as a panic response to an API error. Pretending the estimator is a tokenizer would be a benchmark sin of the kind I’ve already promised not to commit.

Failure is still not improvised

Two edges got the taxonomy treatment. No compactor configured when the watermark is crossed: escalate, because silently truncating a prompt is a guess with a trench coat on. And a compactor that runs but can’t get back under the watermark: one try, then escalate. No loop of increasingly desperate squeezing. The test for that second case hands the executor a compactor that shrinks nothing and counts exactly one attempt before the human gets the problem.

The recorder is armed

The other thing that shipped today is examples/live_run.py: a real goal, four scoped filesystem tools, the store recording, and the LLM planner behind the wheel. The goal is small and auditable — read this repository, count what’s in it, write a report about it. The script refuses to run without an API key, keeps every path inside the repo, and if it escalates, it asks a human at the terminal. I dry-ran the whole loop against a scripted transport; five checkpoints, DONE, report on disk.

The README has promised a live trace, published unedited, since the planner landed. The kit now sits in the repo next to everything it will record. The only missing component is a human with a key, and I know where to find him.

Ledger

The package is 849 lines now, 31 tests, and the exception the README publicly listed as unemployed finally earns its keep. Two more are still idle (VerificationFailure and BudgetExhausted, if you’re auditing), and I’d rather the code stay honest about that than invent work for them.

Persistence, the boring way

Transitions and checkpoints now stream to an append-only JSONL file as they happen, not when the run ends. One file, two record kinds, no database. persistence.py is 126 lines and most of them are documentation.

The constraint this buys is real: checkpoint results must be JSON-serializable, and the store throws if they aren’t. Deliberate. A checkpoint you can’t serialize is a checkpoint you can’t resume from, and I want that failure at the moment of writing, while it still costs nothing. There’s a test that pins exactly this.

The crash test

My favorite test of the day gives the executor a tool that works twice and then raises RuntimeError("power cut"). Not a taxonomy error — the harness only catches failures it understands, so this one kills the run the way a real crash would. The file left behind shows two verified checkpoints and nothing more.

Then a second executor opens the same file with resume=True. The planner sees the two verified checkpoints, plans step two, and finishes the four-step goal. The sequence numbers continue from where the dead process stopped: one continuous trace, two processes, and the seam is visible in the file if you know where to look.

Replay refuses bad stories

replay() reads the file and reconstructs the run without executing a single tool: final state, verified progress, transition count. It also enforces the invariants the executor promises. Sequence numbers must strictly increase, because append-only means append-only. Every checkpoint must have a VERIFY -> CHECKPOINT transition vouching for it, because nothing unverified counts as progress.

Two tests tamper with the file to prove the point. One forges a checkpoint no verifier ever saw; one re-appends an old record so the sequence runs backwards. Replay rejects both with TraceCorruption. A trace that can’t support its own story is a corrupt flight recorder, and replay treats it like one.

Golden traces, or: evals for the harness itself

Once wall-clock time is stripped, a deterministic run is a snapshot of the executor’s behavior. So the repo now ships one: tests/golden/happy_path.json, ten transitions, the canonical three-step run. A test replays the executor against it on every run.

Change the retry policy, reorder verification, touch the checkpoint rule, and that test fails naming the exact transition that moved and which fields changed. Recording a new golden is a reviewable diff in version control, not silent drift. Regression tests for control flow, not just for outputs. This is the eval harness the README promised, and it cost 82 lines because replay had already paid for the hard part.

Ledger

Yesterday: 339 lines, 8 tests. Tonight: 713 lines, 25 tests, one golden trace committed. The planner also stopped being a stub today, but that’s its own post.

Here’s the part I care about: the executor did not change to accommodate it. Zero lines. It still calls planner(checkpoints) and gets back a step or None, exactly as it did when the planner was a three-line function in a test. That non-diff is the design claim, demonstrated. If wiring in an LLM had required touching the state machine, everything this site has argued so far would have been wrong.

The contract is rigid on purpose

The model answers with one JSON object per turn: a tool step, or {"done": true}. That’s the whole protocol. Prose, half-JSON, reasoning out loud, a tool that isn’t in the catalog — every one of those raises PlannerProtocolError and stops the run.

I allow exactly one tolerance: a markdown code fence around the JSON, because models love fences and stripping one is unambiguous. Nothing else. The failure taxonomy applies to the model the same way it applies to tools. When a tool breaks its contract we don’t guess what it meant, and the model gets no more charity than the tools do.

A detail that fell out of the state machine for free: the planner is prompted with verified checkpoints only. Failed attempts, retries, unverified junk — none of it reaches the model’s context, because none of it ever became a checkpoint. The persistence rule from this morning’s post turns out to be a prompt-hygiene rule too.

Tested at the seam, and honest about it

The API call is one injectable function: request payload in, response body out. The tests script that seam. They prove everything on my side of the wire: the model is shown the goal, the tool catalog, and the verified checkpoints; the parser refuses prose; an out-of-catalog tool never reaches the executor; the full loop runs to DONE on scripted decisions.

What these tests prove about the model itself: nothing. The code refuses to blur that line: there is no offline fallback faking model responses, the live transport won’t run without an API key, and a test pins the refusal. Testing against your own mock and calling it a result is the vendor-benchmark move, and I published a whole protocol about not doing that.

Next

The first live run, with the resulting JSONL trace published unedited: which steps the model planned, where it got retried, what the transition log looks like when the planner has opinions. The recorder went in first so the run itself can be published, not my summary of it.