Picture an internal expense agent. Someone asks whether a flight change fee is reimbursable. Retrieval returns zero chunks — the policy PDF failed to parse — and the agent answers "Yes, covered under the standard travel policy", then advances the request to approved. Nobody notices until the end of the month.
The postmortem usually stops at the word hallucination. That ends the investigation one step too early. In most of these systems the system prompt already contains some version of if you don't know, say you don't know. The rule was written. It just wasn't a decision the system could make: there was no state for insufficient_context, no transition out of pending, and no place to record whether the rule was consulted on this input. A language model asked to answer a question with an empty context produces a plausible continuation. That is not a malfunction; it is what the component does.
"I don't know" is three different things
A string. A sentence in the system prompt. Cheap to add, impossible to audit: the output may contain the phrase while the state machine still advances the request.
A state. insufficient_context exists in the orchestrator, with a defined transition and an owner. Now an answer can be refused without relying on prose. But you still cannot tell, after the fact, whether the condition was evaluated on a run that should have triggered it.
An event. constraint_evaluated is written into the run record with the input, the criterion and the fallback reason. Now "did this rule fire tonight?" has a queryable answer.
Only the third version is testable. The first two are how a team ends up adding the same sentence to the prompt three times and wondering why nothing changed.
The boundary belongs to the deterministic backbone
Under hybrid orchestration, a deterministic backbone owns state, retries, authorization, audit trails and rollback; a model is invoked only at nodes requiring semantic judgement. The interesting word there is nodes. Whether there is enough evidence to answer is not a semantic judgement about the answer — it is a check on fields that already exist before the prompt is assembled: retrieved_chunks, schema_valid, scope_match, attachment_parsed.
A rule of thumb we've found useful: if a decision can be made entirely from values present before the prompt is assembled, it belongs in code. Asking the model that generates the answer to also certify the evidence makes the same component both author and reviewer. The model can still misread a policy it can read — that is a real failure, but it lives at a different node and needs a different test.
def answer_policy_question(run):
evidence = retrieve(run.question)
if not evidence or not evidence.schema_valid:
run.record(constraint="missing_context -> insufficient_context",
constraint_evaluated=True,
fallback_reason="empty_retrieval",
state_transition="pending -> blocked",
model_called=False)
return AskForClarification(reason="empty_retrieval")
return model.answer(question=run.question, evidence=evidence)
The output contract has to be typed as well, or the string comes back through another door. status is a two-value enum, and answer is null whenever the status is not answered:
{
"status": "insufficient_context",
"answer": null,
"evidence_ids": [],
"fallback_reason": "empty_retrieval"
}
An illustrative run record for that path would look like this:
{
"run_id": "r-184",
"node": "answer_policy_question",
"input": {"question": "Is the change fee reimbursable?", "retrieved_chunks": 0, "schema_valid": true},
"constraint": "missing_context -> insufficient_context, never infer",
"constraint_evaluated": true,
"fallback_reason": "empty_retrieval",
"state_transition": "pending -> blocked",
"model_called": false
}
model_called: false is the whole point. It is an assertion you can write against a replay, not a hope about tone. One more detail worth keeping: the fallback belongs in the structured prompt contract as a hard constraint and in code as a transition. If it exists only in the contract, you are back at version one with nicer formatting.
Nine cases, not one final-answer score
White-box evaluation means capturing per-node inputs, outputs and acceptance criteria into replayable test sets, instead of judging the final answer by feel. Three test sets, kept separate: positive / boundary / follow-up. For this one fallback, three cases each:
- Positive: grounded context, a clear answer, and the fallback must not fire.
-
Boundary: zero retrieved chunks; a malformed tool response; a question outside the policy's scope. Each must end at
insufficient_contextwith the rightfallback_reason. - Follow-up: the user supplies the missing document; the user rephrases after a refusal; the user asks again after context was repaired. Same run, same candidate, no state leak.
Nine cases, two assertions each — the state transition and constraint_evaluated — give eighteen checks that all fail on the same line when the rule is only prose. Assert both directions. A fallback that fires on grounded context is a new failure mode, not a safety improvement, and a suite that only rewards refusals will happily produce an agent that refuses everything.
The same shape covers the other fallbacks: authorization denied, rollback, retry exhausted. Once every fallback is an owned state with an evaluation event, coverage becomes a count you can diff between deploys — declared fallbacks, fallbacks with a case that fires them, fallbacks with a case that must not.
What this costs, and what it doesn't buy
There is a cost consequence, but it runs downstream of the architecture. When insufficient_context is an owned state, you can count the runs that ended there and decide whether to fix parsing, extend the corpus, or loosen the rule. Cost stops being a number attached to "the agent was slow" and becomes a number attached to a named state. Useful — but not the reason to build the boundary, and framing it as a savings story tends to produce the wrong design.
Scope and limits, because this argument is easy to oversell:
- It does not fix semantic errors on genuinely ambiguous input, knowledge missing from the corpus, or requirements that were wrong to begin with. A perfect backbone around a bad question still produces a bad answer.
- Deterministic delivery here means controllability, auditability and fallbacks at the architecture level — explicitly not a promise that the model will always be right.
- It can add a clarification turn, and it introduces over-refusal as a failure mode you now have to test.
- For a throwaway prototype where a wrong answer costs nothing and nobody has to explain it later, a probabilistic black box is the cheaper and more honest choice. The backbone earns its keep when failures must be attributed, outputs reach external users, or an audit trail matters.
The question I'll leave you with
In your stack tonight, is "I don't know" a sentence in a prompt, a state the orchestrator owns, or an event you can replay to the exact input and criterion that should have stopped the call? If a policy document fails to parse at 2 a.m., which line of the run record would show that the rule was evaluated — and would the request still have advanced?
No tool to sell, no link to click. Just this: a fallback that only exists in prose is a fallback you have not built yet.
Top comments (50)
The string / state / event ladder is the right diagnosis. I would keep the rule about who certifies the evidence: if the decision can be made from values already present before the prompt is assembled, it belongs in code. The model that writes the answer should not also be the reviewer.
In a coding-agent loop the same gap shows up as "all tests passed" in the final message — the string version. An
acceptedstate is still not testable unless the run record has aconstraint_evaluatedevent with executed, passed, failed, and skipped counts. A required suite that was skipped is insufficient evidence, not a pass, and the transition to accepted should stay blocked.That case belongs in the nine-case matrix next to zero chunks and malformed tool responses. The assertion has to be two-sided: fully executed, grounded checks must not trip the fallback, and a skipped required check must not look like "done".
Adding executed, passed, failed and skipped to the event is the right move, and skipped is the column that usually gets collapsed into zero. A required suite that never ran and a required suite that ran and was skipped both produce the same final state, so the transition to accepted has to be blocked on either.
There is a writer problem in the counts, though. If the runner reports them, the acceptance check is reading the same component it is checking. I would keep a registered_suites list fixed by the harness before the run and compare it with what the event reports. The difference is a third outcome: never registered. It is not the same as skipped, because the fix for a skipped suite is in the runner and the fix for a missing one is in the deployment. Both leave executed at zero.
For the two-sided assertion, the case worth adding is a fully green run where one required suite was absent from registered_suites. Delete that name from the list and leave every count intact. Does the current acceptance check still return accepted, and which field would show that a suite nobody registered was never evaluated?
Agreed on the third outcome.
skippedandnever registeredcan both leaveexecutedat zero, but they point at different fixes, so they shouldn't share a column.I would keep
registered_suitesas a harness input, fixed before the run, and have the acceptance check compare that list with the event — not trust the runner to have included every required name. If the check only looks at executed/passed/failed/skipped counts, then deleting a required name from the list and leaving every count intact would still look green. The counts cannot show a suite that was never registered; the comparison against the pre-run list is the field that can.The case I'd add to the two-sided assertion is exactly the one you described: a fully green count vector plus one required name missing from
registered_suitesmust not transition toaccepted, and the record should show the missing name as unregistered, not as skipped.Right, and the comparison has one more failure mode worth closing: the pre-run list has to be pinned to the same bytes the run consumed. If registered_suites is read from a file at run time and read again at check time, the file can be edited in between and the acceptance check will happily agree with itself.
So hash the list into the run record and compare against that digest, not against the file's current contents. Then "a required name was deleted" becomes a diff between two recorded artefacts instead of an inference from counts that structurally cannot show it.
Your unregistered-not-skipped label matters for the same reason: it is the only outcome that makes the diff cheap to read six months later.
Yes. If the acceptance check rereads
registered_suitesfrom the file, the comparison is against whatever the file says now, not against what the run used.I would pin the list the same way as any other harness input: write the bytes, or their digest, into the run record before the runner starts, and have the check compare the event to that recorded artefact. Then a required name that disappears mid-run is a diff between two records, and it cannot be hidden by editing the file after the fact.
That is also why I would keep the
unregisteredlabel.skippedis a runner decision about a name that was in the list;unregisteredis a name that was in the pinned list and never made it into the event. Six months later those are different bugs, and the digest is what makes the second one cheap to prove.This is the right shape, and one detail decides whether it holds: pin the effective list, after inheritance and overrides, not the authored file. Most suites are assembled from a base plus per-run includes, so a digest of the file you edited will miss the include that pulled a check in - and that is exactly the check whose disappearance you care about.
Second: record the resolution time next to the digest. A check added later must not retroactively satisfy an earlier run. Comparing "what the run resolved" to "what the policy resolves now" needs both sides dated, otherwise a new check quietly turns an old failure into a pass.
Yes. The artefact to pin is the list the runner actually resolved, after includes, inheritance and overrides, not the file someone edited.
A digest of the authored file misses the include that pulled a name in, and that is the name whose disappearance you need to prove. I would write that resolved list, or its digest, into the run record together with the time it was resolved, and have the check compare against that snapshot. A check that appears in the policy later cannot be allowed to turn an earlier failure into a pass, because the two sides of the comparison would no longer be from the same moment.
Honest answer: no, I never measured it - and that is the admission worth making. What surfaced first was cruder: five of the six containers had no SIGTERM handler at all, so the 10 seconds was the kernel's stop timeout expiring, not a drain running long. p99 connection lifetime was invisible because nothing was draining to measure.
Which is why your framing is better than mine was: I was treating a 10s constant as the problem when the measurable quantity was always the distribution. If I rerun it, the first artifact is a histogram of connection lifetime at shutdown, and only then a budget derived from its p99. The 10s was a symptom that happened to be round.
The string/state/event ladder is the sharpest part — a prompt sentence you can't audit is just a wish, and the
constraint_evaluatedevent is what turns "did the rule fire tonight?" into a query. I'd argue the typed output contract does as much work as the event:answer: nullwhenever status isn'tansweredis what stops the hallucination coming back through another door. Do you actually query those run records in production, or is the event mainly there to make evals honest?I can't give you a production query count, and I would not want to make the event's case rest on one. The claim I would make is narrower: without the event, the question of whether this rule fired on this run can only be answered by inferring from the binary version, and that inference is exactly what an audit cannot use.
The test I would run is a single lookup. Give me a run_id and a criterion_id; the record should return either the evaluation event with its writer and outcome, or an explicit no-event-for-this-criterion. Those two answers have to be distinguishable from the check being present in the code. If the lookup needs a deploy manifest to interpret it, the typed contract is doing the runtime work and the event is decoration. That is also why the contract and the event are not competing: one blocks a bad state at write time, the other makes the block inspectable after the run.
The single-lookup test is the right oracle — and the 'explicit no-event' answer is doing more work than it looks like. It has to be distinguishable from 'we never recorded anything,' or the lookup becomes the same inference the event was meant to replace. Does your no-event answer carry a reason — criterion not in the deployed ruleset versus evaluated but not recorded — or is that the next thing the contract has to guarantee?
The reason should be part of the lookup result, not something inferred at read time. Without it, "no event" covers a rule that was never in the deployed ruleset and a rule that fired but failed to write, and those need different fixes.
I would have the lookup return one of three shapes: the event with writer and outcome, no-event/criterion-not-deployed, or no-event/evaluated-but-not-recorded. The first no-event case is answered from a pinned ruleset digest in the run record, so a criterion added later cannot rewrite an older run. The second means the criterion was in scope and the write did not happen. Both are distinguishable from the check merely existing in the code.
The fixture that proves the reason is real: one run where the criterion is absent from the pinned ruleset and the lookup returns not-deployed, another where it is present but no event is written and the lookup returns not-recorded. If both come back as the same bare no-event, the reason is being filled in by whoever reads the log.
The pinned ruleset digest is the move that makes this work — without it, a later criterion silently rewrites what an old run meant. And the two-fixture test is the right proof: if both cases come back as the same bare no-event, the reason was never in the system at all. Has the not-recorded case ever caught a real write failure in production, or has it stayed a theoretical gap so far?
The string/state/event ladder is the part that stuck with me — especially making insufficient_context an owned state with a typed output contract where answer is null unless the status is answered. The nine-case eval matrix is a practical way to test it, too. One question: when you count runs that ended in insufficient_context, how do you distinguish a parsing failure from a genuine gap in the corpus in practice?
The state itself cannot tell you. insufficient_context is one terminal value, so the distinction has to be in the event that wrote it, and it has to be written by the branch that took the decision: parser_error, empty_index, scope_filter_zero, stale_version, wrong_binding. If a classifier fills that field afterwards by reading the transcript, it is another model output describing a code decision.
A cheap check that the field is real: take a case that normally answers, break the extractor on that one input, and require two things in the log, cause=parser_error and no retrieval call for that run. Then re-run with the extractor fixed and the index empty, and require cause=empty_index. If both runs produce the same cause, the field is cosmetic. That pair also gives the eval matrix two rows whose expected labels are fixed before the run, which is the part a state machine cannot supply on its own.
That pair of runs is what makes the field falsifiable — requiring cause=parser_error with no retrieval call, then cause=empty_index on the re-run, means the cause is written by the branch that decided, not pasted on afterwards by a classifier reading the transcript. It also closes the gap I was worried about in my first comment: the state machine finally knows why it stopped. One thing I'm still turning over: when scope_filter_zero and stale_version both leave you with the same empty result, do you keep them as two distinct rows in the matrix, or collapse them into one cause?
Keep them as two rows in the matrix. Same empty result, different repair: scope_filter_zero points at the declared scope or the filter, stale_version points at the corpus snapshot. Collapsing them saves a row and costs the on-call engineer the first hour of the incident.
Each cause should carry the fields that let someone tell which branch ran. For scope_filter_zero that is the declared scope and the filter input, written before the prompt. For stale_version it is the requested version and the served version, again fixed before assembly. If both rows end up with the same field set, the cause is probably a classifier reading the transcript rather than the branch that wrote the state.
A cheap way to check: run the same query twice, once with the scope narrowed to zero and once with the index at a superseded version. The two runs must return different cause values. If they return the same one, the field is a label on the terminal state, not a record of the decision.
the run record with constraint_evaluated is the part i would steal first. we had this exact expense style failure, retrieval returned zero chunks after a parser change and the model happily answered anyway, and it took two weeks to notice because the logs only kept the final answer.
moving the empty retrieval check ahead of the prompt killed those silent answers for us. the gnarly part was partial evidence, 1 chunk of 6 expected, which sails through a simple count check.
do you gate on a minimum chunk count or on coverage of the entities in the question?
Coverage over a bare count, with one condition on where the entity list comes from. If the required entities are extracted from the question at run time by a model, that is the same writer problem as the answer itself. I would have the requester declare the required set before the run, keep optional entities as a ratio, and log the gate's own line with both numbers.
The 1-of-6 case then fails loudly instead of sailing through, because one chunk rarely carries every named entity. A count check cannot tell you which entity went missing, which is why the two-week blind spot stayed invisible. A coverage line shows a drop the day the parser changes.
For alerts I would separate two things. Chart the optional ratio for humans, but alert on an invariant: a decision row with no gate line is a bug, not drift, and should page. Give the ratio alert a denominator floor, because coverage over eight items swings enough that one item can trip a threshold and get the alarm muted. One cheap consistency check: the gate's entity counts should sum to the entities it claims to have seen.
Neither on its own. A minimum count is satisfied by duplicates and boilerplate - one chunk passes a count of 1 while covering a third of the question.
What generalised for us is entity coverage: extract the entities the question names, require all the required ones to appear in the retrieved set, and treat coverage of optional entities as a ratio rather than a boolean. Your 1-of-6 case fails that gate loudly, because a single chunk rarely holds every named entity.
Then log the decision, not just the outcome - the gate's own line with counts and covered entities - so the next parser change shows up as a coverage drop the same day instead of two weeks later.
ratio for optional entities is the bit i am going to steal. we had a chunk full of boilerplate clear a plain count check once, so coverage over count makes sense to me.
logging the gate's own line with counts is the real win imo. our two week blind spot only existed because we kept the answer and dropped the decision. do you alert when the coverage ratio drops, or just chart it?
Both, but they answer different questions. Chart the ratio for humans; alert on the thing that actually broke for you: a decision recorded without its gate line. That is not a coverage number, it is an invariant - if every decision row must carry the gate's counts, then a decision with no counts is a bug rather than drift, and it should page immediately instead of trending.
The ratio alert is the softer one and it needs a denominator floor to be useful: coverage on n=8 items swings 12% when one item moves, so alert on a drop that is both larger than X% and has n above a minimum. Otherwise you build an alarm that fires on small batches, someone mutes it, and the blind spot comes back wearing a different hat.
One cheap invariant worth adding: the gate's counts should sum to the number of entities it claims to have seen. That catches the dropped-duplicate case you described without needing the ratio at all.
I'd keep a small fixed set of queries next to that chart, with expected coverage pinned to a document version. The live ratio can fall because the question mix changed. If the fixed cases also lose coverage on the same documents, that's a stronger signal of a retrieval regression than the live ratio alone.
A field can be declared, signed and queryable and still collapse back into your tier one. I scanned a live append-only agent ledger at cutoff 2026-11-10T08:33:24Z: of 1,984 scoring events, 1,982 carried an empty
evidence_event_idsarray, and that field sits inside the signature. Two rows filled it. Those two were signed by the same key, through the same schema and the same write path as the other 1,982, so there is no broken component to find in the log. The field worked fine. It was optional. In those same records the required pointer to the scored delivery resolved 1,983 of 1,983, no dangling targets, nothing scored twice. Required reference, perfect. Optional evidence, 0.1%. The only thing separating them is whether the write path rejects a record without it. Narrow claim, since this is one key and one implementation: recording the grounds was not enforced. I cannot show from this that no checking happened.The second gap is in the coverage diff. A criterion label can be a synonym for the outcome. Every verdict in that snapshot had a signed
reason; five distinct strings total, one of them on 1,945 rows. Cross-tabulate reason against verdict and every string appears on exactly one side, none on both. Approval rate by reason code is fully computable and tells you nothing, while declared-versus-exercised still passes.So I would have the event carry the fields the condition actually read, their values at read time, and the fields it could not read, rather than the condition's name. Then the same code shows up on both sides of a decision, and one that never does is a rename. Does your replay harness reject a record whose criterion is named but whose reads are absent?
The optional-versus-required split in that snapshot is the part I would keep: the same key, the same schema and the same write path produced both the populated and the empty rows, so there is no component to repair. What differs is whether a write without the field is rejected. I would not generalise the ratio from one ledger — you already scoped it to one key and one implementation — but the structural point does not depend on the ratio at all.
Where I would add a line to "carry the fields the condition actually read": that changes what the event contains, not who is allowed to leave it out. The same write path can emit an empty reads array with the same signature, and a replay harness that only checks shape will accept it. The discriminator is not a richer field, it is whether a rejection ever occurs. And a rejection that has never fired is indistinguishable in the log from a check that was never switched on, so the gate needs its own observable event. The cheapest falsification is a mutation test: inject one record with a named criterion and no reads, and require both that it is refused and that the refusal is recorded. A coverage diff can then show the code on both sides of a decision because you forced the occasion, rather than waiting for production to supply one.
The second gap has the same shape from a different direction. A reason string and a verdict produced by the same decision are two outputs of one function, so cross-tabulating them can only return the function's own consistency; a label that is a synonym for the outcome is the degenerate case, not an anomaly. What breaks that is a label whose value is fixed before the run: the expected outcome attached to a boundary case in a fixed test set. Then expected and observed come from different paths, and a criterion that is never exercised shows up as a column that never disagrees with the verdict. That also gives "declared versus exercised" something outside the runtime record to be checked against — which is what makes it white-box evaluation rather than the runtime auditing itself.
So, to your question: the contract as I described it would reject a record with a named criterion and absent reads — but I would trust the harness less on that assertion than on one number, which is how many records it has actually refused. If that count is zero, the field is present, signed and queryable, and still tells you nothing about whether the gate ran. If you feed one deliberately reads-less record into your ledger today, does the log gain a rejection event, or does it stay byte-identical apart from the new row?
The cleanest signal here is chunk count, not the model's words. Retrieval returning zero is deterministic and the pipeline already knows it, so gate on that before the model answers - a self-reported "I don't know" is exactly the string this post correctly dismisses.
The number that proves the state machine works is approve-rate conditioned on chunks == 0. Run the agent over a set where retrieval is deliberately emptied and count how often the outcome is still approved. If that is not ~0, the insufficient_context state exists in the schema but not in the decision.
Instrument (retrieval_chunks, output_state) per request and that one conditional becomes your regression test on every deploy, instead of a prompt you hope got read.
Gating on the count before the model answers is the right order, and I agree that a self-reported "I don't know" is a string, not a state. Conditioning the approve rate on chunks == 0 is also the right first metric — it turns "the state exists in the schema" into a measurable claim about decisions.
The place I would add a column is in the pair you instrument. (retrieval_chunks, output_state) records the state but not who wrote it. A code-owned refusal and a model that happened to output insufficient_context produce the same two-tuple, and the regression test passes in both cases; in the second one the deterministic part of the pipeline never made the decision. Adding a decision writer — gate, model, or fallback — makes that visible, and it is the difference between "the state machine works" and "the output happened to look like the state machine worked". Without it, a deploy that quietly disables the gate can stay green.
The second thing the conditional needs is a partner. Approve-rate | chunks == 0 ≈ 0 is satisfiable by a gate that refuses everything, by a path that emits no output at all, and by a run set where the empty-retrieval cases all fail earlier for unrelated reasons. Reading it alongside approve-rate | chunks > 0 on the positive set tells you the boundary is placed where you intended rather than swallowing the whole distribution. And the expected outcome for both sets has to be fixed before the run; if the gate changes and the labelled set changes with it, the number stays at zero for reasons that have nothing to do with behaviour.
The last edge is the cause behind the zero. "No matching policy", "query built wrong" and "scope filter narrowed to zero" all arrive as the same integer, and a fourth case — chunks non-empty but irrelevant to the query — never arrives at all. Neither is visible in the conditional, so a green regression test can coexist with a retrieval path that is broken in three different ways. None of this changes the first step you described; it only makes the metric attributable. In your two-tuple, if the gate never fired but the model produced the right abstain string on every empty-retrieval case, what would the deploy regression test show — and which field would tell you the gate was not involved?
That is the right condition and it needs a mechanism rather than a habit: put the expected row in the repo as data - case, declared binding, expected decision - and commit it before the run. Then "written down first" is a commit timestamp instead of a recollection, and a gate that quietly widened after the re-index shows up as a diff rather than as a suspicion.
One more condition on the row: it has to be one that can fail. A perturbation whose expected result is "refuse" proves nothing if the gate was already refusing everything - you need a case the old gate passes, so the change is visible.
A commit timestamp is stronger than a recollection, and it needs one more binding to do the job: the row has to be tied to the run record, not just present in the repo. A later commit can look earlier than the run if the record does not carry the digest. I would write the row's digest into the run record before the runner starts and compare against that.
The can-fail condition is the more important half. An expected value of refuse proves nothing if the gate was already refusing that case, so the row should come from a case the current gate passes. Then perturb one thing and require the decision to flip. If you only add rows that already refuse, the matrix will stay green through a gate that quietly widened.
That suggests two rows with different jobs. One positive case that must keep passing, so a widened gate does not get mistaken for improvement. One boundary case with a fixed expected decision that flips under the perturbation. A single pass rate hides the first behind the second, which is the same reason I would keep written and admitted as separate counts.
You're right that the rejection event is the thing, and it needs one more property you can get nearly free: provenance. A model asked to "log a refusal" can emit a refusal string without the gate firing - the same degenerate case as the reason string. So the rejection event has to be written by the gate's code path and carry a field the model cannot set, like a distinct event type or a marker on the write itself.
Then your mutation test asserts two things: the record is refused, and the refusal came from the gate, not from the model's output. If the log gains a rejection row with the gate's marker, the gate is wired. If it stays byte-identical apart from the new record, the gate is not in the path - which is the answer your question is fishing for.
The provenance marker is the right shape, but "a field the model cannot set" is an architectural property, not a property of the field. If the gate and the generator share one event sink and one serializer, the marker is just another string that an upstream caller can put in a payload. The thing to test is not that the marker is present; it is that the sink is fed by two mutually exclusive write paths.
That gives your mutation test a second, different case. Inject a well-formed refusal event carrying the gate marker through the model-output path and require the sink to reject it. That is not a rerun of the same falsification — the first one asks whether a decision happened, this one asks whether the producer boundary is enforced. A green log with a forged marker passes the first and fails the second.
The byte-identical check has a blind spot of its own. It is decisive when the gate never ran, but a gate that ran and wrote to a buffered or differently routed topic also changes the log without leaving anything a verifier can read. Write-side evidence and read-side evidence are different claims, and white-box evaluation needs the latter: a record that this acceptance event was consumed by the check, not only emitted.
So I would keep your two assertions and add a third field to the tuple: where the event was read from. Otherwise a deployment can be fully wired, fully instrumented, and still have no verifiable acceptance. If your gate writes a correctly marked rejection into a topic that nothing consumes, what does the current mutation test report — and which field distinguishes "written" from "admitted as evidence"?
Both questions land, and they have one answer: the assertion has to be two-sided.
Writing the triple after prompt assembly makes binding_match a function of the retrieved set - it is computed from what was already chosen, so it matches by construction, and both fields stay green while the generator reads last year's terms. The tell is falsifiability: perturb the corpus, add a stale chunk or re-index one, and a pre-declared binding flips to false while a post-assembly one never moves. That perturbation is the test.
Same shape at the sink. A correctly marked row that nothing consumes passes "written" and fails "admitted as evidence". The distinguishing field is the verifier's own consumption record - the event id appearing in the check's output, not in the topic.
The perturbation test is the sharper of the two, with one condition: the expected result has to be written down before the corpus is touched. Otherwise the binding flipping is a second observation from the same run, and a gate that quietly widens after the re-index can flip too. Fix the row first, this chunk set, this declared binding, expected refuse, then perturb.
On the sink side, the consumption record has a writer question of its own. If the topic or the sink writes that the verifier read event 41, the producer is still the component whose admission we are testing. The event_id needs to appear in the verifier's own output, keyed to a check_id and a verdict, so the record comes from the consumer rather than from the thing being consumed.
That gives a one-row fixture that fails without touching the gate at all: a correctly marked event is written, and the verifier output for that run contains no event_id. Written passes, admitted as evidence fails. I would put that row next to the corpus perturbation, because they fail for different reasons and a single aggregate pass rate would hide one behind the other.
Your last-year's-table question is the right one, and it shows why field-presence is the wrong predicate: a valid payment-terms field from last year's contract has the same field name, so presence passes it and the generator scores against stale terms. The gate must assert the binding, not the field - the retrieved chunk has to match the query's (supplier, contract_id, effective_date), recorded at query time.
To the second question, the field is the decision-writer, written by the gate in the same transaction as the refusal, so the model cannot set it. A deploy that disables the gate produces zero rows with the gate marker - that is the red regression.
So the tuple becomes (chunks, binding_match, writer, output_state), and writer is the field that tells you the gate fired.
Binding rather than presence is the right predicate. The next question is where the binding comes from: if (supplier, contract_id, effective_date) is extracted from the query by the model, the gate is validating model input with model output. The tuple only becomes a deterministic comparison when the requester fixes those three values before the generator sees a prompt — declared up front as part of the prompt contract, then compared, not recomputed.
There is a second case your binding does not yet separate. Two chunks can carry the same contract_id and effective_date after a re-index or a contract amendment, so binding_match stays true while the evidence has changed. Recording the corpus version on the chunk side makes that visible, and a stale-version case belongs in its own boundary set rather than being folded into the empty-retrieval case, where it is indistinguishable from a miss.
The four-tuple is still an observation from inside one run. It records what happened, not what was expected to happen. Without an expected label fixed before the run, a deploy regression can only tell you that today's tuple equals yesterday's — it cannot tell you which of the two was correct, and a gate that quietly widens will look stable.
None of that changes the ordering you described; field presence was the wrong gate, and writer was the missing column. If your binding triple is written after prompt assembly rather than before it, which failure mode leaves both binding_match and writer green while the generator still reads last year's terms?