DEV Community

Cover image for The stranger test: what happens when agents you've never met hit your API
InApp
InApp

Posted on Originally published at imapp.blogspot.com

The stranger test: what happens when agents you've never met hit your API

For the first month after I shipped my document converters, the only traffic was mine. I tested my own endpoints, fed them my own PDFs, and convinced myself the pipeline worked. It worked — for me.

Then I started watching what strangers actually sent. Agents I'd never seen before started POSTing things my test suite never contained: scanned PDFs with skewed text layers, DOCX files exported from a phone, XLSX workbooks with seventeen sheets and merged cells everywhere, HTML scraped from sites with scripts still embedded. My "clean Markdown" output started coming back with artifacts — stray control characters, broken tables, headings nested six levels deep.

The lesson wasn't that my parser was buggy. It was that my definition of "clean" was calibrated entirely on inputs I chose. Real agents don't curate their inputs; they forward whatever the web hands them. So I rebuilt my validation around failure modes I observed in the wild instead of examples I imagined: length caps that actually trigger, selector fallbacks that degrade gracefully, table flattening that survives merged cells.

The second lesson was about silence. Strangers don't file issues — they just retry with a different tool or give up. I added logging on every conversion whose output looked structurally odd, and the patterns in those logs taught me more than any benchmark. One agent kept sending EPUBs with DRM wrappers; another sent base64 payloads twice the size of my limit and expected a partial result instead of an error.

If you're shipping anything in this space, build the stranger test into your routine early. Watch what unknown agents send, not what you send yourself. I packaged the whole pipeline as https://x402.freeq.one/tools/document_to_markdown.html, and the version strangers see now is measurably more robust than the one I tested alone.

Top comments (0)