A refund agent has one rule in its prompt: never approve an amount above 200 without manager sign-off. In March the limit is raised to 500 for a VIP pilot. Nobody lowers it again. In June a 340 refund is approved with no sign-off, and the only record the system keeps is the final answer.
The postmortem usually produces a new sentence in the prompt. That makes the rule more emphatic, not more testable. Nothing in the run record says which version of the rule was in force. Nothing asserts that the rule was consulted on this input. And no test would have failed when someone changed 200 to 500, because the rule was prose and prose does not have a diff you can assert on.
This is a contract problem. The spec that governs a semantic node deserves the same treatment as an API schema: versioned, validated before use, and covered by cases that fail when it changes. The phrase for that here is a structured prompt contract: a versionable spec of role, hard constraints, I/O contract and fallback rules, testable and portable, rather than free-form prompting.
Four fields, each with a checkable criterion
A contract has four fields. Each one has a characteristic way of being wrong, and each way can be tested without running the whole agent.
Role sets the scope of the spec, not a character to perform. Written as a field, it says which node this contract governs: applies_to: classify_refund_request. A spec that quietly starts applying to the escalation node as well is a scope bug, and the criterion is simple: list the nodes a contract is allowed to bind, and assert that no other node loaded it.
Hard constraints are predicates over declared inputs, each with an id and a destination when it fails:
constraints:
- id: approve_limit
predicate: amount <= 200
on_fail: manager_review
reason: approve_limit_exceeded
- id: currency_known
predicate: currency in ["USD", "EUR"]
on_fail: blocked
reason: unsupported_currency
The predicate reads declared fields, not generated text. If approve_limit has to parse the amount out of a sentence the model wrote, the same component is producing the value and checking it, and the check will pass as often as the model happens to agree with itself.
I/O contract types the boundary. status is an enum, answer is null unless status is answered, and evidence_ids is a list of ids that came from retrieval. Every field names its source: declared input, declared constant, or model output. A model may only write fields marked as model output. That constraint is enforceable at the boundary, and it is what stops a fallback decision from leaking back through a free-text answer field.
Fallback rules map each constraint's on_fail to a reason and a state transition. They belong in the contract and in the orchestrator. Under hybrid orchestration the deterministic backbone owns the transition, while the contract declares which reason should accompany it. If a reason exists only in the spec, the model has to remember to emit it, which returns the system to a sentence in a prompt with better formatting. If it exists only in code, an operator cannot see which fallback the node is meant to have. Both copies are checked against each other at load time.
Version the spec, then diff the constraints
A contract that cannot be diffed is a document. Give each version an id and a digest, and write the digest into the run record rather than the full spec:
{
"run_id": "r-2041",
"node": "classify_refund_request",
"contract_id": "refund-v7",
"contract_digest": "9c1f...",
"input": { "amount": 340, "currency": "USD", "manager_signoff": false },
"constraint_evaluated": "approve_limit",
"state_transition": "pending -> manager_review",
"fallback_reason": "approve_limit_exceeded",
"model_called": false
}
Two consequences follow. First, the question "which rule was in force on this run" has a queryable answer even after the spec has moved on. Second, a deploy can be reviewed as a change to a known set of constraints. When a colleague asks what changed between v6 and v7, the answer is a four-line diff over predicates and reasons, not a recollection of a Slack thread.
This is where the vocabulary matters. persona and character describe role-play, and role-play is the part of a prompt that does not survive production. A structured prompt contract is a spec with a version and a test suite. Calling the same object a persona invites people to edit it for tone, and tone edits do not come with boundary cases.
Make the runner reject a bad contract
The strongest version of this idea is that an invalid contract never reaches a model. A validator at load time refuses a spec when:
- a predicate references a field the I/O contract does not declare;
- an
on_failvalue is not a state the orchestrator owns; - a
reasonis outside the declared enum; - a model-output field carries no acceptance criterion.
Each rejection is a test that fails before the agent does. That is the difference between a contract the runner reads and a contract the runner ignores politely. It is also cheap: validation is ordinary code over a small spec, and the failure tells you which line to fix.
There is a second check worth adding. Log the evaluation, not only the spec. constraint_evaluated in the run record above names the predicate that fired on this input. A spec that was present but never consulted looks identical to a spec that fired correctly if you record only the spec.
One experiment that proves the contract is live
Coverage counts are easy to inflate. A better proof is a perturbation that must flip a case.
Take one boundary case: amount = 340, no manager sign-off. With v7 the expected result is manager_review and fallback_reason: approve_limit_exceeded. Now change approve_limit from 200 to 500 in a new version and replay the same input. The expected result flips to answered. If both versions produce the same decision, the predicate was decoration and no amount of prompt rewording will help.
Keep the sets separate, as in white-box evaluation: positive cases where a grounded answer must pass and the fallback must stay quiet, boundary cases for empty retrieval, malformed tool output and out-of-scope questions, follow-up cases for multi-turn drift after a refusal. Three cases in each set, with two assertions each, cover this one constraint. The perturbation row belongs to the boundary set, and its expected decision is fixed before the version changes.
Portability is the last property that falls out. Compile the same spec to two different models and run the same sets. Wording may differ; the state transition and the reason must not. If they do, the difference is in the model, and you have located it at a node instead of guessing from a final answer.
Scope and limits
This argument is easy to oversell, so here is what it does not cover.
- It does not fix a semantic error at a node that genuinely requires judgement. A model that misreads an ambiguous email will still misread it inside a well-specified contract.
- It does not tell you whether 200 was the right threshold. It makes the threshold visible and diffable; choosing it is still a product decision.
- Deterministic delivery here means controllability, auditability and fallbacks at the architecture level. Rule logic is deterministic by construction over declared inputs. That is a statement about which parts of the system do not improvise, not a promise that every output is correct.
- A contract with too many fields becomes brittle. Every edit then demands a full replay, and teams start editing the prompt beside the spec to avoid it. Two or three constraints with real boundary cases beat twenty with none.
- For a throwaway prototype where a wrong answer costs nothing and nobody has to explain it later, a probabilistic black box is the simpler choice. The spec earns its keep when a rule change must be reviewed, a failure must be attributed, or the same behaviour has to hold across more than one model.
What to check first
Open the last change to a rule in your system. If the diff is a sentence in a prompt, ask three things: which version was live on the failing run, which case would have failed when it changed, and whether the runner could have rejected a malformed version of it. A structured prompt contract is worth building when those three questions have no answer today.
Top comments (0)