DEV Community

Cover image for Retrieval Ran First. Was Its Result Actually Ready?

Retrieval Ran First. Was Its Result Actually Ready?

Distinguishing event order from data dependency

The trace starts with retrieval. Generation appears underneath it. The ordering test passes.

But generation began while retrieval was still running.

“A appears before B” can describe the order in which events were encountered. “A's result was available before B began” describes an execution dependency. “B actually consumed A's result” is a stronger data-flow claim. They are not three ways of saying the same thing.

I maintain AgentInspect. This article tests those distinctions with controlled promises and agent-inspect@6.31.35. The fixture does not invoke a model, search engine, or vector database.

Make the race deterministic

Avoid a demonstration that depends on a slow network. Hold retrieval open until the test explicitly releases it:

import { inspectRun, step } from "agent-inspect";

await inspectRun("overlapping-retrieval", async () => {
  let release;
  const ready = new Promise(resolve => { release = resolve; });

  const lookup = step.tool("retrieve_policy", () => ready);

  await step("generate_answer", async () => "synthetic answer", {
    type: "llm",
  });

  release({ policyId: "fixture-policy" });
  await lookup;
}, { traceDir: ".agent-inspect", silent: true });
Enter fullscreen mode Exit fullscreen mode

Here type: "llm" labels an instrumented boundary. It does not turn the callback into a model call or capture hidden reasoning. The callback returns a fixed string.

Retrieval is invoked before generation, but the result cannot become available until after generation completes. The fixture encodes that relationship without guessing a millisecond delay.

Two timelines compare retrieval starting first with retrieval finishing before generation starts.

Start order is insufficient when the second operation depends on the first result.

Use a typed relation across tool and model steps

After opening the generated file with openTraceFile from agent-inspect/readers, evaluate the relation:

import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";

const contract = defineTraceContract({
  run: { requireCompleted: true },
  steps: {
    orderRelations: [{
      before: { kind: "TOOL", name: "retrieve_policy" },
      after: { kind: "LLM", name: "generate_answer" },
      mode: "happens-before",
      requireEndpoints: true,
    }],
  },
});

const result = evaluateTraceContractRead(trace, contract);
Enter fullscreen mode Exit fullscreen mode

Use the typed relation because this crosses a TOOL-to-LLM boundary. tools.requiredOrder is a tool-only surface, not a generic list of arbitrary workflow nodes.

TraceContract is beta, and steps.orderRelations is explicitly experimental in this release. Keep the version pin and test the exact semantics you intend to rely on.

The measured results for the overlap fixture are:

Mode Result Question being tested
first-occurrence Pass Was the first matching retrieval encountered first?
happens-before Fail Did the first matching retrieval finish before generation started?
all-occurrences Fail Did all matching retrieval work finish before any matching generation started?

Overlap can produce diagnostic information even when first-occurrence ordering passes. Read findings and warnings, not only the top-level label.

Stronger is not always more correct

all-occurrences is a global boundary, not an automatic pairing algorithm.

Consider a legitimate two-turn interaction: retrieve, generate, retrieve again, generate again. A rule requiring all retrieval to precede all generation can reject that workflow even if each individual answer waited for its own retrieval.

That is not a reason to weaken the rule until it passes. It is a reason to model the intended scope. Use separate runs or explicit application-level links that associate each answer with its retrieval result. Don’t assume that matching names imply pairwise dependency.

Similarly, requireEndpoints: true matters. An ordering statement should not quietly satisfy your product requirement when one of the necessary endpoints is missing.

Fix the application dependency first

The corresponding positive control awaits retrieval before invoking generation:

const policy = await step.tool("retrieve_policy", async () => ({
  policyId: "fixture-policy",
}));

await step("generate_answer", async () => policy.policyId, {
  type: "llm",
});
Enter fullscreen mode Exit fullscreen mode

Inside a captured run, this fixture passes the tested all-occurrences relation. The change is the dependency in the application, not a delay inserted into the test.

For a real model call, completion still does not establish consumption. Record a bounded result reference at retrieval and the reference actually supplied to generation. An application assertion can compare those references without storing the retrieved document or prompt.

Even matching references prove wiring, not factual grounding. Whether the answer uses the evidence correctly requires a separate domain or semantic check.

Keep incomplete execution visible

The current implementation has a narrower boundary worth documenting. In the companion fixture, one generate_answer starts early and never finishes; another matching generation completes later. The isolated all-occurrences ordering rule passes, while the full contract with completion checking fails through run.status.

This was reproduced against 6.31.35. It is not a claim that the ordering rule completely diagnoses every unfinished occurrence. Do not ship an ordering-only gate or describe that diagnostic gap as fixed.

The test deliberately creates an unawaited pending operation to expose the boundary. That is a negative control, not an application pattern to copy. Keep a valid fixture, an overlap fixture, a repeated-turn fixture, and an incomplete fixture in your contract tests.

Ask for the smallest defensible conclusion

A trace can establish that a recorded step preceded another. With adequate timestamps and boundaries, it can establish that one finished before the other began. With application-owned references, it can help establish which result was supplied downstream.

None of those facts alone proves that an answer is correct.

The practical improvement is to name the obligation precisely. If generation needs retrieval output, test result availability. If it must use a particular retrieval result, test the data-flow link. If the answer must be grounded, test the answer against that evidence.

One green ordering check should not have to impersonate all three.

Implementation references

Top comments (7)

Collapse
 
max_quimby profile image
Max Quimby •

The three-way split — "A appears before B" vs "A's result was available before B began" vs "B actually consumed A's result" — is the distinction I wish more trace tooling forced you to make. The failure I keep seeing in RAG agents is the first one quietly standing in for the third: the ordering test is green, retrieval technically started first, and generation ran against an empty or stale result the whole time. Nobody notices until an answer is confidently wrong and the trace "proves" everything was in order.

Your happens-before contract nails the availability claim cleanly. The one I still find hard is the consumption claim — proving generation read retrieval's output, not just that it was free to. In practice that seems to need data provenance (tagging the value and checking it actually reached the consumer's inputs), not just timestamps, since a step can complete after another with zero data dependency between them. Does agent-inspect have a way to assert consumption, or is happens-before the strongest relation it models today?

Collapse
 
raju_dandigam profile image
Raju Dandigam Google Developer Experts •

@max_quimby You have the boundary right. The orderRelations check used here in agent-inspect establishes availability, not consumption; this fixture does not provide a built-in provenance assertion. I would tag the retrieved result with a bounded reference and compare it with the reference actually passed into generation. A missing or stale reference should fail even if happens-before passes. That proves wiring, while grounding the answer still needs a separate check.

Collapse
 
innokentyb profile image
Kent Bodrov •

The repeated-turn case also makes me want a stale-result negative control: hold retrieval A, start a newer request B, then release A after B completes. Even if both steps finish, the application should reject A when it belongs to the wrong request or an invalidated generation. A per-request correlation ID plus an explicit cancellation or freshness rule would make that failure observable without logging the retrieved content.

Collapse
 
raju_dandigam profile image
Raju Dandigam Google Developer Experts •

@innokentyb That is a strong negative control. I would let A and B both complete so a completion or ordering check can still be green, then assert that generation for B accepts only a retrieval reference carrying B’s request ID and current freshness epoch. Releasing A afterward should be recorded as stale and ignored, whether or not cancellation reached the underlying retrieval. That tests the application decision without storing retrieved content.

Collapse
 
aheadkit profile image
AheadKit •

Your three-way split — appeared first, finished first, actually consumed — is the bit I'll steal, because in our planner work the middle one is where things quietly go wrong. A planner acting on a half-ready retrieval feels like the same failure shape as acting on a step whose result is still UNKNOWN: the trace looks ordered, so nothing complains, but the decision was made on air. Do you check the bounded result reference at the point of consumption, or is that a check you'd put at the contract level instead?

Collapse
 
raju_dandigam profile image
Raju Dandigam Google Developer Experts •

@aheadkit I would check the bounded reference at the consumption boundary: record the retrieval result ID and the ID actually supplied to generation, then compare them in an application assertion. The current orderRelations contract checks timing; it cannot infer data flow from step names or timestamps. If those explicit links become part of a contract later, the same comparison could move there without claiming that ordering proves consumption.

Collapse
 
aheadkit profile image
AheadKit •

That's the check the contract is missing — the IDs, not the timestamps. Ordering can tell you the steps ran in sequence, it can't tell you generation actually ate what retrieval found. Going to try that assertion on a real pipeline and see what it catches that the timing check walks past. What would you do with a mismatch once it fires — fail the run, or just log it?