Reliable Agent Work Needs Tickets, Locks, and Recovery

Imagine a long-running agent task losing contact halfway through a workflow. The easiest response is also the most dangerous: launch it again.

The first worker might still have been running. It might have changed something before losing contact. A second attempt could repeat the same work, overwrite useful state, or send two answers.

The real problem was not that a worker failed. Workers will fail. The problem was that the system needed a clear way to answer basic questions: What request is this? Who owns it? What already happened? Is another turn still active? Where should the result return? What is safe to do next? A Practical Architecture for Reliable Multi-Agent Systems puts those controls around a larger ownership and delivery model.

Before launching another turn, inspect the ticket, the exact turn, its lock, callbacks, and any output that already exists. Recovery is an investigation before it is an action. A retry that skips that check can turn one uncertain result into duplicate work or an overwritten record.

For this kind of asynchronous work, I use a compact ticket, duplicate control, endpoint locks, explicit callbacks, and a governed recovery path. An endpoint is an agent context that keeps its own history, and a turn is one uninterrupted run in that context. Together, these controls let the system survive an interrupted process without pretending the interruption never happened.

For the shared public route and return pattern around those records, see Building One Agent from Coordinated Roles in Codex.

A ticket keeps the obligation visible

A ticket is a small record of one request that remains after the work starts. That detail matters because even a failed launch leaves something an operator can inspect.

In our case, the request runtime persisted the compact ticket, while a separate lifecycle tracker kept the open obligation, updates, residual posture, and close visible. The record kept the request identity, current state, lineage, and key receipts. It also kept pointers to any larger evidence or work product stored elsewhere.

A useful ticket might answer:

• Which endpoint owns this request?
• What stable key identifies duplicate attempts?
• Which parent request and trace led here?
• Is the request queued, running, waiting, complete, or in recovery?
• Which exact turn produced the response?
• Where should an approved result return?
• Was the callback accepted and the requester delivery attempted?

The ticket is transport and lifecycle truth. It is not the full truth about the work itself. The semantic owner still decides what the request means and whether the result is acceptable.

That boundary keeps the runtime useful without turning it into another domain manager.

A ticket is not a workspace

It is tempting to put everything beside the ticket. Source documents, transcripts, draft trees, logs, and attachments all seem helpful during debugging.

In this system, that approach became costly and unsafe. Large artifacts and callback output expanded the request ledger until a compact control record started acting like an archive.

I now treat the ticket like a claim check, not the luggage.

The ticket stores identity, status, compact results, receipts, and pointers. Substantial work stays in the owner’s project or a separate evidence bundle. Credentials, source corpora, full transcripts, and worker workspaces stay out of the ledger.

This is a state-placement decision, not just a storage preference. Where an Agent System Keeps Its State explains why a ticket, a task session, working evidence, and reusable knowledge need different homes.

This separation has practical benefits. Ticket scans remain fast. Recovery tools can inspect lifecycle state without opening sensitive source material. Callback activity cannot keep copying a large workspace through the request chain.

It also makes retention easier to reason about. A short transport record and a large evidence package often need different access and deletion rules.

Deduplication stops retries from becoming new work

Consider a weekly research brief. An automation submits the request, but its connection closes before it sees an acknowledgement. It tries again.

Without duplicate control, the system may launch two researchers. Both can produce reasonable briefs. Both may call back later. The public entry point now has to guess which answer is current.

A stable deduplication key lets the retry find the original ticket. In our implementation, duplicate identity joined the endpoint address with that key. The same pair returned the existing request whether it was queued, running, complete, or failed.

That is mechanical deduplication. It answers, “Have I already admitted this request for this endpoint?”

It does not answer a semantic question such as, “Should this week’s brief run again because the source changed?” Only the semantic owner has enough context to decide that. The runtime should suppress identical admissions without claiming that similar work is always the same work.

Compatibility routes create another duplicate risk. If an old entry point and a new one can both reach the same owner, they should converge on one owner-scoped key. Otherwise, a fallback can quietly become a second execution.

That same rule protects live migrations. How to Change an Agent System Without Breaking Live Work applies it when an old and new route must coexist.

Locks protect agent context, not correctness

Persistent endpoints carry context across turns. Two overlapping turns can both read the same starting state, make different decisions, and then write conflicting updates.

An endpoint lock prevents that overlap. In our case, one exclusive lock covered the endpoint from resume through exact completion or failure isolation. A separate request lock protected changes to the individual ticket.

The lock does not prove that the answer is correct. It only ensures that one turn at a time can change the endpoint’s continuing context.

The system also needs to know which turn produced a result. A response is acceptable only when it belongs to the registered endpoint task and the exact turn started for that request. “Some completion happened” is not enough evidence. A late result from another turn can be just as misleading as two overlapping turns.

This creates an important callback rule. Suppose an owner launches a child request whose result must return to that same owner. If the owner keeps its current turn open while waiting, it keeps the endpoint lock. The callback then cannot start because it needs that lock.

The safe pattern is simple:

1. Dispatch the child request.
2. Record where the work should continue.
3. End the current turn and release the lock.
4. Let the callback arrive as a new request.
5. Resume integration in that new turn.

A longer timeout does not fix this deadlock. The owner must release the resource that the callback needs.

Timeouts should limit work, not express impatience

A timeout is useful when it represents a real operating budget. It is harmful when it becomes a guess about normal model latency.

Small, context-complete work may fit a synchronous request with a finite deadline. Source review, decomposition, several workers, or callback shaping usually belongs in asynchronous mode. The caller can stop waiting while the ticket keeps the obligation visible.

If an exact turn exceeds its budget, the runtime should try to interrupt that exact turn. It should then wait for matching evidence that the turn stopped.

In our case, an endpoint entered quarantine—blocking new work—when the runtime could not prove the timed-out turn had ended. Later requests failed closed. Time passing or a different successful event did not clear the uncertainty.

Clearing that quarantine requires evidence tied to the uncertain turn and an accountable recovery decision. Otherwise the system could release a context while an unknown earlier turn can still write to it.

That may feel strict, but the alternative is worse. Releasing the lock while an unknown turn can still write creates overlapping execution by accident.

Explicit callbacks close the internal return path

When asynchronous work finishes, a result file does not move itself back to the responsible owner. A return address expresses intent, but it is not a return event.

Our runtime created an explicit callback request after a terminal result appeared. That request carried the original trace, the completed request identity, its response, and requester metadata. It targeted the component responsible for the next decision.

The receiving component then recorded its own callback result. Until that happened, the source work could be complete while owner integration remained pending.

Callbacks should also converge. A repeated completion notice should reuse the same callback rather than create another branch. Otherwise, one failure can turn into a callback storm.

A circuit breaker provides a final ceiling. It is a hard cap that stops a request chain from growing forever. It can warn as a trace grows and refuse further requests after a hard limit. The specific numbers will depend on the system.

Even a successful callback is still not delivery. It proves that the result returned inside the system. The public entry point must still get the owner-approved answer to the person or system that asked for it. I cover that distinction in Done Is Not Delivered.

Escalate multi-owner work carefully

A lifecycle tracker can keep a single owner’s obligation visible. When several owners and dependencies need follow-through, a workflow coordinator can also track the outcome, next update, unresolved dependencies, and requester posture.

That is a continuity job, not a second domain-owner job. The coordinator can make an overdue obligation visible, but it should not decide whether a release is safe or whether evidence is sufficient.

Recovery should inspect before it acts

Now return to the failed research brief. The worker process is gone, the ticket says running, and there is no final result. What should happen?

Blind replay is not a safe default. The missing worker may have completed part of the work or caused an external effect. Deleting a lock or editing the ticket by hand only removes evidence of that uncertainty.

The recovery path in this case had two phases:

Inspection comes first and should not change the evidence. An accountable owner then reviews the case. Only after that decision can the system finalize an orphan, authorize new work, or keep the endpoint quarantined.

Recovery must not invent a response that the worker never produced. It should not resume a stale callback or infer a successful outcome from a dead process. If substantive work must run again, it should receive a new identity after the owner allows it.

This makes recovery slower than deleting a lock and pressing retry. It also makes the result far easier to trust.

Recovery should leave a lesson

Recovery should not end with the ticket closing. The owner should ask what failed, what evidence made the diagnosis possible, and what would make the next run less likely to hit the same problem.

The answer may be a note for the next run, a clearer role guide, a better tool check, a test, or a new graph branch. Important changes still need the right review. How to Make an Agent System Learn shows how closeouts and manager reflection turn that evidence into approved change instead of a private memory.

The lifecycle is small, but each transition matters

The whole pattern can fit in a short flow:

ticket created
  -> dedupe check
  -> endpoint lock acquired
  -> exact turn runs within a set time limit
  -> terminal result recorded
  -> endpoint lock released
  -> explicit callback accepted
  -> requester delivery recorded

uncertain turn
  -> quarantine
  -> immutable inspection
  -> owner decision
  -> governed recovery

Each state answers a different question. Combining them into a single “done” flag saves fields but loses the evidence needed when something breaks.

Ticket creation and every lifecycle transition should also be crash-safe, atomic, and conditional on the expected prior state. Otherwise a process failure or competing update can corrupt the very record recovery depends on.

This design has an operating cost

Tickets, locks, callbacks, and recovery add state. They also add code, monitoring, storage rules, and operator work. Old and new endpoint generations can drift. A stuck lock can block useful work. A poor deduplication key can suppress a valid request.

This design is not a goal by itself. It does not make agent reasoning correct, and it does not replace owner review. It also does not belong around every prompt.

For one small task in one live conversation, the conversation may already provide enough identity and return path. A predictable batch may need only a simple workflow graph.

The heavier pattern earns its cost when work outlives the caller, crosses persistent owners, can cause side effects, or must survive process failure. In those cases, I want the system to preserve the obligation even when the worker disappears. When an Agent Needs to Be a Service, Not a Chat helps decide whether a standing service is warranted at all.

That is what reliable agent work means to me. Not that nothing fails, but that a failure leaves enough trustworthy evidence to decide what happens next.