Durable AI Agent Harness Design

A lot of people think the hard part of an agent harness is tool calling. It is not. The hard part is stopping the system from lying about completion.

Once an agent can edit files and run commands, the real engineering problem becomes operational control:

1. What exactly is the task?
2. What state survives the session?
3. What counts as done?
4. What evidence is required before the system says it is done?

If those answers are weak, the agent may look productive while quietly drifting.

The Failure Pattern I See Most

Most long-running agent failures are not model failures in the narrow sense. They are harness failures:

1. The task is underspecified, so the agent rewrites the goal into something easier.
2. Session context compresses away important requirements.
3. The agent edits a lot, verifies little, and reports confidently.
4. Progress is visible in chat but not durable in the repo.

That is why I prefer a plain, inspectable harness over a "smart" one. The job of the harness is not to look autonomous. The job is to make cheating difficult.

Use A Requirement Ledger, Not A Vibe

Keep a machine-readable feature list or requirement ledger in the repo:

[
  { "id": 1, "feature": "User can sign in", "passes": false },
  { "id": 2, "feature": "User can export CSV", "passes": false }
]

The important rule is strict: the agent may change passes: false to passes: true, but it must not delete requirements just because they are inconvenient.

That sounds obvious, but it blocks a real behavior: agents silently narrowing scope until the remaining task looks done.

Make Every Session Rebuild State

Every session should begin by rebuilding context from durable sources:

1. Check the working directory.
2. Read the progress note.
3. Inspect recent git history.
4. Read the feature list.
5. Run a baseline test.

This is the engineering equivalent of a shift handover. If a new session cannot reconstruct the work from repo state, the system is not durable yet.

Work One Verifiable Unit At A Time

Long sessions encourage fake progress. A better rhythm is:

1. Pick one unchecked feature.
2. Implement it.
3. Verify it.
4. Commit it.
5. Update progress.

This keeps rollback cheap, review understandable, and failure attribution possible.

Separate Claim From Proof

An agent saying "implemented" is not evidence. At minimum, the harness should require one of:

1. A passing test.
2. A build result.
3. A reproducible manual verification note.
4. A screenshot or trace for UI work.

This sounds procedural, but it is the line between automation and theater.

Verify Like A User

Unit tests are useful, but agents can pass unit tests while the product is unusable. For user-facing work, add end-to-end checks. A browser test that logs in, fills a form, and submits it is stronger evidence than a mocked function test.

Store Memory Outside The Model

Do not rely on conversation compression as the system of record. Compression is lossy, invisible, and impossible to review after the fact.

Use files, git commits, test output, and progress notes instead. The clean mental model is the same as distributed systems: treat model context like working memory, not durable storage.

A Better Harness Has Friction In The Right Places

The best harnesses are not "frictionless." They add friction exactly where agents are most dangerous:

1. Before changing task scope.
2. Before claiming success.
3. Before taking risky actions.
4. Before writing irreversible external state.

If a harness makes it easier to edit than to verify, it is biased toward hallucinated completion.

A Minimal Harness Shape

while work remains:
  read durable state
  choose one feature
  implement
  test
  commit
  update progress

This is not glamorous, but it is robust. The goal is not to impress someone with autonomy. The goal is to keep the work recoverable, inspectable, and difficult to fake.

That is the standard I use when evaluating an agent harness: if the current session vanished right now, would the next session know what happened and what still needs proof?