DEV Community

Cover image for An MCP Tool Call Can Resolve and Still Fail

An MCP Tool Call Can Resolve and Still Fail

Distinguishes API returns from real outcomes

An await finishes. The application returns a result. The trace contains a completed tool span. The test is green.

The tool also returned isError: true.

There is no contradiction. JavaScript promise completion and MCP tool success are different signals. Treating them as the same signal creates a surprisingly quiet failure: an agent can continue after a rejected operation while a completion-only test reports success.

I maintain AgentInspect. This article uses a synthetic test against agent-inspect@6.31.35 and @agent-inspect/mcp@6.31.35 to separate those signals. It makes no model calls and does not describe a production incident.

Three questions hiding inside “success”

The MCP tools specification distinguishes protocol-level errors from errors reported inside a tool result. A valid result can carry isError: true.

For an application, there is another boundary beyond either of those: whether the requested effect actually happened.

Question Evidence What it does not establish
Did the call return? Promise resolution or protocol response That the tool accepted the operation
Did the tool report failure? MCP result's isError The final state of an external system
Did the intended outcome occur? Independently observed state or receipt That every earlier execution step was correct

A booking tool might respond normally to explain that inventory is unavailable. An email tool might accept a message without proving delivery. A database timeout might leave the write outcome uncertain.

Those cases deserve different application behavior. They should not collapse into a single green check.

MCP response handling separates rejected promises, returned tool errors, and independently verified outcomes.

A returned error is evidence to evaluate, not an instruction to retry automatically.

Start with a deliberately small reproduction

Install matching package versions:

npm install --save-exact agent-inspect@6.31.35 @agent-inspect/mcp@6.31.35
Enter fullscreen mode Exit fullscreen mode

This JavaScript fixture is intentionally not a real MCP server. It isolates the returned-result branch of the wrapper:

import assert from "node:assert/strict";
import { mkdtemp, readdir } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { inspectRun } from "agent-inspect";
import { wrapMcpClient } from "@agent-inspect/mcp";

const traceDir = await mkdtemp(join(tmpdir(), "mcp-result-"));
const response = {
  isError: true,
  content: [{ type: "text", text: "Synthetic request rejected" }],
};

const client = wrapMcpClient(
  { async callTool() { return response; } },
  { serverName: "fixture" },
);

const returned = await inspectRun(
  "returned-tool-error",
  () => client.callTool({ name: "lookup_policy", arguments: {} }),
  { traceDir, silent: true },
);

assert.strictEqual(returned, response);
const [filename] = (await readdir(traceDir))
  .filter(name => name.endsWith(".jsonl"));
Enter fullscreen mode Exit fullscreen mode

The identity assertion matters. Instrumentation should not silently replace the caller's result with a different object or invent an exception. In this release, the wrapper preserves the result and records a failed outcome observation for the MCP tool error.

The application still owns the response: explain the rejection, choose an authorized fallback, request clarification, or stop. The tracing adapter records evidence; it does not choose the recovery policy.

Test the observation, not just completion

Append this to the same fixture:

import { openTraceFile } from "agent-inspect/readers";
import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";

const trace = await openTraceFile(join(traceDir, filename));

const completionOnly = evaluateTraceContractRead(
  trace,
  defineTraceContract({ run: { requireCompleted: true } }),
);

const withOutcomes = evaluateTraceContractRead(
  trace,
  defineTraceContract({
    run: { requireCompleted: true },
    observations: { failOn: ["failed"] },
  }),
);

assert.equal(completionOnly.status, "pass");
assert.equal(withOutcomes.status, "fail");
Enter fullscreen mode Exit fullscreen mode

These are measured results from the included test harness, not a hand-drawn CLI transcript. TraceContract is a beta surface; pin the release and keep the fixture beside the contract.

Notice what the second check says: a captured observation reported failure. It does not claim that AgentInspect independently queried a booking system or verified a database transaction. The observation comes from the MCP response. Its provenance determines its evidentiary strength.

Don’t turn every tool error into a retry

isError is not a retryability classification.

A malformed argument, denied permission, unavailable resource, and transient dependency failure can all produce unsuccessful tool results. Retrying the first two without changing anything adds work rather than recovery. Retrying a side-effecting action after an ambiguous response can duplicate an effect.

Prefer an application-owned, bounded policy. Separate read-only recovery from writes. Where a remote service supports idempotency, follow that service's actual contract. Neither the wrapper nor a trace field makes an operation exactly-once.

Also avoid parsing arbitrary error prose into policy decisions. If your server exposes reviewed structured error categories, test those categories. If it does not, preserve uncertainty instead of guessing from the word “timeout.”

Expand the fixture at the integration boundary

The fake client proves a narrow behavior. Before adopting a wrapper, add real SDK tests for normal results, thrown errors, isError, cancellation, method receivers, and lifecycle methods. Verify captured calls against an independently counted server handler.

The companion examples include an installed MCP SDK client/server test using an in-memory transport. That covers SDK method compatibility without implying that network reconnects, authentication, or every transport have been tested.

This distinction is the habit worth keeping: every green result should name the boundary it actually exercised.

If your current agent test only asserts that a promise resolved, add one fixture in which the tool explicitly refuses the request. Does the run still look successful? If so, the missing assertion is probably not about the final sentence.

Implementation references

Top comments (17)

Collapse
 
agentsearchhq profile image
AgentSearch •

The three-question table is the useful part. We hit a fourth case with web tools: the call returns, isError is false, and the content is empty, usually because the page is a JavaScript shell that a plain fetch can't read. Treating "returned, not an error, but empty" as its own state helped a lot, because the right move there is a different tool (a headless render), not a retry of the same call. Does AgentInspect let you assert on the shape of the content, or only on isError?

Collapse
 
mike-johnstone profile image
Mike Johnstone •

A genuine zero-result page would be a useful second fixture here. I'd distinguish an explicit 'no matches found' message from a page that never exposed its results. Then an empty list alone wouldn't trigger the headless fallback.

Collapse
 
kevinhall profile image
Kevin Hall •

same thing with data lookups, a clean 200 and an empty list because the filter was too tight or the source was behind. we compare the count to what the same query gave last time and fail the run when it falls off a cliff

Collapse
 
gautamtalksdev profile image
Gautam K •

This gets worse one step later, when the agent writes its own summary: a tool result with isError: true followed by "Done, I've updated the record" is exactly the gap between what an agent reports and what actually happened. Asserting on isError and then reading back the real state catches both halves. Does AgentInspect flag a final message that claims success after an errored tool call?

Collapse
 
raju_dandigam profile image
Raju Dandigam Google Developer Experts •

Good catch, @gautamtalksdev. The trace check can fail on the MCP isError observation, but it does not infer whether a final natural-language claim is true. I would test that at the application boundary: compare the final claim with the tool outcome and, for a write, an independent readback. That mismatch is a useful regression case.

Collapse
 
aheadkit profile image
AheadKit •

Your three-question table is the clearest version of this I have seen, and the third row is the one I am stuck on in practice. Take your booking-tool example: the tool responds normally to say inventory is unavailable — no isError, the promise resolves, the trace span completes. Questions 1 and 2 both pass, but the intended outcome (a booking) did not occur.

In your fixture, where does that case land? Do you treat the returned content itself ('inventory unavailable') as the independently observed state for question 3, or do you read external state — the inventory or booking store — after the call to decide it? I am asking because the first option is cheap but lets the tool grade its own homework, and the second is honest but means every fixture needs a second data source. Which side did you land on for the companion examples?

Collapse
 
jsbry profile image
Joshua Bryan •

Not the author, but we hit the same fork from the server side. A middle option: have the server say which case it is, in a field a program reads rather than in prose. In the MCP Write-Safety draft we wrote (MWS 0.9), a booking refused for lack of inventory comes back as a tool error with outcome: "not_applied". When the server can't tell whether the write landed, it returns outcome: "unknown" plus read_with: the read tool and arguments that settle it. Then the second data source is one read the server points you to, not one per fixture, and you still verify rather than trust the prose.

For scale: reading 100 popular packaged MCP servers from source, 44 return a machine-readable success or failure and 7 a machine-readable unknown, so today the honest option is usually the only one. https://isokron.ai/research/mcp-write-results-100-servers?utm_source=devto&utm_medium=social&utm_campaign=mws-launch&utm_content=comment

Collapse
 
aheadkit profile image
AheadKit •

The read_with pointer is the bit I like most — one read the server points you to instead of one per fixture is a real saving. The 44-of-100 number is the honest part though. What does MWS say the client does with the other 56 — still parse the prose and hope?

Thread Thread
 
jsbry profile image
Joshua Bryan •

Mostly it doesn't, on purpose. MWS is written for the server side, so for those 56 the fix it asks for is on the server: report a refused or failed write as a tool error (MWS-RES-2), and say "unknown" when it can't tell. It doesn't trust prose for its own verdicts either: a word list or a model reading the text never decides a result, only a declared field or a person does.

For a caller stuck with one of those servers today, what we'd do is treat any write result that isn't clearly an error as "unknown": read the object back before acting on it, and never retry a write that has no idempotency key until that read shows it didn't land. For a create there's no id to read by, so it helps to put something searchable in the write itself. That's one read per write, which is still cheaper than one data source per fixture.

Thread Thread
 
aheadkit profile image
AheadKit •

That closes the loop I was stuck on — one read per write is still cheaper than one data source per fixture, and the searchable-bit-in-the-create is a neat trick. I'm going to run that rule against a few of the prose-only servers and see how often the read actually settles it — does it ever break down for you, a create where the server won't take anything searchable at all?

Collapse
 
mp3sss31 profile image
mp3sss •

The three-level breakdown (returned, reported failure, actual outcome) maps onto how we assert IPC on the desktop side: a resolved promise from the main process is only an ack, not proof the file wrote or the side effect landed. A completion-only test goes green the same quiet way here. The test contract should check the tool's isError flag, then assert the externally observed state or receipt, rather than treating the completed span as success.

Collapse
 
srinivaspadmanabhuni profile image
Dr Srinivas Padmanabhuni •

Your three-question table is a useful reminder: a completed tool call does not prove the tool accepted the action, and neither proves the intended state occurred. In one recorded travel-planner trace we evaluated, a hotel hold using a price absent from search results was refused while the flight hold had already gone through, leaving a partial effect after the failed step. That is the kind of tool-trace behavior agentest checks: github.com/srpadmanabhuni/agentest

Collapse
 
hannie8 profile image
Hai Anh •

The third row is the one worth building tests around, and the readback step has a trap of its own. If the target system is eventually consistent (a read replica, a search index, most SaaS APIs behind a cache), an immediate readback can miss a write that actually landed, and the test flags a success as a failure. I'd make the readback bounded: poll for a short window, or read from the primary when the API lets you, and record "not yet visible" as its own outcome instead of folding it into "failed".

Collapse
 
paullovelace profile image
Paul Lovelace •

The line about not parsing error prose into policy decisions is the one I'll be stealing for code review. I've seen too many handlers that string-match their way to a retry, and then a server rewords a message and the whole policy quietly changes. Structured error kinds feel like the boring part of the design until they save you from exactly that.

Collapse
 
cianspider profile image
Cian •

The third row of your table is where my server sits, and there is one step before it that catches MCP clients out.

I run an MCP tool that is paid per call over x402. An unpaid tools/call never becomes a tool result at all. It comes back as HTTP 402 with the payment terms in the body, because the request did not reach the tool. So a client that maps any non-200 to "tool error" sees isError true and retries, and if it retries without paying it gets 402 again, forever. A client that treats 402 as a soft success is worse: it thinks the audit ran when nothing was audited.

The only fix I found was to put the price and the resource URL inside the 402 body, so the caller can choose to pay or stop instead of guessing from a status code. On the client side I ended up with three buckets, not two: transport status, then the tool result, then 402 on its own, which means "you have not asked yet" rather than "it failed".

Which leaves me with a question I cannot answer from my side: does AgentInspect record a call that is refused before dispatch as a tool span at all, or does it fall outside the contract because there was no result to evaluate?

My server, if the shape is useful to look at: mcptry.com/servers/io.github.cian-...

Collapse
 
onelifevibe profile image
OneLifeVibe •

The three-question table is the part I keep coming back to. In our own matching work, the same trap shows up as a green check: a payment call returns fine, the ledger updates, and only later does someone notice the intended outcome never happened. Your point about not turning every tool error into a retry is the one I would underline twice. A malformed argument retried ten times is still malformed, and a denied permission retried ten times just burns budget. Treating the returned error as evidence to evaluate rather than a signal to retry feels like the right default. Thanks for making the fixture small enough to actually copy.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.