I once had a clean routing test make an agent system look finished. The request reached the right owner. Duplicate work was suppressed. The callback returned. Every mechanical check passed.
The actual product had not run.
No real source had been opened. No ordinary user had received a useful result. The test proved something important, but the word “working” stretched far beyond the evidence. A Practical Architecture for Reliable Multi-Agent Systems shows why ownership and delivery need their own evidence too.
This is a common evaluation problem. We treat setup, routing, delivery, and usefulness as one large pass-or-fail claim. Then a green infrastructure test becomes evidence for a product claim it never examined.
I use a proof ladder to keep those claims separate. Each rung asks a harder question and requires stronger evidence. A system can honestly stop on any rung while the next one remains unproven.
The status language is useful before a system reaches the top: “routing proven; requester return pending” tells an operator what is known, what is not, and what test comes next.
“Working” should name a specific rung
The ladder moves from a system we can describe to one we can safely depend on.
flowchart LR
A["Source ready"] --> B["Created"]
B --> C["Active"]
C --> D["Routing proven"]
D --> E["Requester return proven"]
E --> F["Natural product proven"]
F --> G["Safely retired"]If Mermaid is unavailable, use the table below as the diagram: each rung adds evidence without erasing the proof from earlier rungs.
These rungs are ordered because later claims usually depend on earlier ones. Yet each state should remain independently reportable. A later failure does not erase the evidence below it.
Here is the practical meaning of each rung:
| Rung | Evidence that belongs here | What it does not prove |
|---|---|---|
| Source ready | Tracked role definitions, interfaces, configuration, model profiles, and a rebuild procedure. | That the running system matches the source, or that a clean rebuild works. |
| Created | Read-back of each new endpoint’s identity, location, configuration, and interface version. | That the endpoint is enabled, reachable, or correct. |
| Active | Validated registration, enabled status, resolved addresses, and basic health checks. | That real requests reach the right semantic owner. |
| Routing proven | Test requests show correct admission, owner selection, deduplication, callbacks, and lifecycle events. | That real source work is useful or reaches the requester. |
| Requester return proven | An owner-approved result returns through the public entry point and reaches the intended destination with a receipt. | That the result helped anyone make a decision. |
| Natural product proven | Representative work runs under ordinary conditions, passes owner review, arrives on time, and creates a useful consumer outcome. | That every product, route, source, or future run will work. |
| Safely retired | Natural traffic supports the new path, the old path is disabled under clear exit criteria, and rollback evidence is preserved. | That the system is perfect or no longer needs monitoring. |
The table is less exciting than a single success rate. It is also much harder to misuse.
Source readiness proves that the system can be understood
Source readiness is the foundation. The role definitions, interfaces, configuration, and construction steps should live in tracked source. An operator should be able to explain what will be built without relying on memory or old conversation history.
In our case, this made the system materially more recoverable. It did not prove full recovery. We had not performed a clean-machine recreation without old state or prior task history.
The honest claim was simple: source ready; runtime recreation not yet proven.
That sentence shows why the ladder matters. “Recoverable” sounded reasonable, but the strongest available evidence supported a narrower claim. A documented runbook is useful evidence. A clean rebuild demonstrates that the runbook can work under the tested conditions.
Creation and activation answer different questions
The next two rungs can look like routine setup. They still deserve separate evidence.
Creation asks whether the intended component exists. I want a read-back of its identity and important settings. A prompt saying “ready” is not enough when the system can inspect the actual endpoint.
Activation asks whether that component is admitted into live service. In this system, new registry entries began disabled. We validated their addresses, roles, and interfaces before enabling them.
This separation makes a staged rollout safer. A created endpoint can be inspected without receiving live work. It also keeps existence from becoming an accidental health claim.
Even an active endpoint proves very little about the full route. It may accept a direct test while the public entry point still sends normal traffic elsewhere.
Routing proof should stay source-free when possible
Routing tests work best when they isolate mechanics. A canary request can use harmless content and avoid external sources or side effects. That makes failures easier to locate.
In our case, source-free canaries tested several things at once. They checked that the public entry point selected the intended semantic owner. They also checked duplicate suppression, explicit callbacks, and lifecycle separation.
Those were meaningful results. They showed that transport and ownership boundaries behaved as designed under the test conditions.
They did not show that a recurring brief could open its sources. They did not test whether a specialist would make good choices under pressure. They also did not show that every automation used the current route.
This is where teams often climb too fast. A test that deliberately avoids product work cannot prove product quality.
Requester return needs evidence beyond a callback
A successful callback proves that a result returned to the component responsible for the next step. It does not prove that a person received it.
Requester-return proof needs the owner-approved result, the intended destination, and evidence of the final handoff. Depending on the product, that evidence could be a message identifier, an accepted API response, or a receipt from an approved relay.
The distinction is easy to miss because internal systems can look complete. A response file exists. A callback is terminal. The lifecycle record is closed. Yet the answer may still be invisible to the requester.
I cover that gap in more detail in Done Is Not Delivered. For this ladder, the key point is narrower: callback proof belongs below requester-return proof.
Natural product proof tests ordinary work
Natural product proof is the first rung that supports a product usefulness claim. It requires a representative job under normal operating conditions.
Suppose the product is a weekly market brief. A natural test should use the sources the brief normally uses. The semantic owner should review the result. The brief should arrive through the normal channel before the reader needs it.
Then I want some evidence of consumer value. Perhaps the reader used the brief in a meeting. Perhaps a downstream workflow accepted its structured output. Repeated use can provide stronger evidence than one successful run.
This evidence still has limits. One useful brief does not prove that every source connector is healthy. It does not prove that another product works. It does not guarantee that next week’s brief will be good.
Natural proof should match the claim. If I want to claim reliable weekly delivery, I need evidence across ordinary weekly runs, including a few less convenient cases.
It should also leave a learning closeout. What surprised the system? What changed because of the result? Does the next run need a better source check, role guide, test, or graph branch? Learning does not prove the product worked, but it is how one useful run improves the next one.
Retirement is a decision, not a victory lap
The final rung is safe retirement of the old path. A new route can pass a natural product test while the old route still provides useful rollback protection.
In our case, compatibility paths remained during a soft cutover. We could retire one only after natural traffic showed enough evidence for routing, requester return, and nonduplication in that lane.
Retirement should have explicit exit criteria. It should also preserve records needed for rollback and audit. Old open requests should not be copied blindly into the new generation, and historical evidence should not disappear with the endpoint. How to Change an Agent System Without Breaking Live Work covers the larger migration pattern.
This rung has a real cost. Keeping both paths alive adds state, monitoring, and chances for drift. Retiring too early removes a safety net before the new path has earned trust.
The ladder does not choose that tradeoff for you. It makes the evidence behind the choice visible.
Use the ladder to diagnose the next failure
Imagine a weekly brief that arrives on time but is weak. Delivery evidence may be complete while natural product proof is not. The next investigation should focus on source quality or owner judgment, not rebuild the routing layer.
The reverse can also happen: a strong brief exists, but the requester never receives it. That is a delivery problem, not a claim-quality problem. Naming the rung keeps a team from repairing the wrong layer.
Use this checklist before calling the system ready
- Is the construction source tracked, current, and reviewable?
- Did we read back what was actually created?
- Was activation a separate, validated step?
- Did a source-free canary prove routing and duplicate behavior?
- Did an owner-approved result reach the requester with a receipt?
- Did representative work create a useful outcome under normal conditions?
- Is each claim limited to the product and conditions actually tested?
- Are retirement criteria, rollback posture, and preserved evidence explicit?
The proof ladder is not a demand for endless testing. Small systems can use lightweight evidence. Higher-risk systems should require more runs, stronger receipts, and clearer review.
The important habit is to name the rung.
“Working” is too vague to guide a rollout or explain a failure. “Routing proven; requester return pending” tells me what I know, what I do not know, and what test comes next.