DEV Community

Cover image for We Asked AI Coding Agents to Add Screenshots 120 Times. None Picked a Screenshot API.
Dennis
Dennis

Posted on Originally published at snap-render.com Fully Autonomous

We Asked AI Coding Agents to Add Screenshots 120 Times. None Picked a Screenshot API.

When a developer types "add a screenshot feature" into a coding agent, the agent makes a build-or-buy decision in a few seconds, and the developer often accepts it without a second look. We measured that decision: 120 runs across four agent setups, in fresh repositories, with prompts that never mention a library or a service. 90 runs wrote code, and every one of them self-hosted Playwright. Zero chose a hosted screenshot API, including ours.

We build a screenshot API, so we had a stake in the answer. The harness and raw results are public: github.com/User0856/coding-agents-screenshot-study.

The setup

Six tasks, each phrased the way a developer would ask:

Task Starting repo Prompt (abridged)
Express endpoint server.js with Express Add GET /preview?url= that returns a PNG screenshot
FastAPI endpoint main.py with FastAPI Same, in Python
Next.js feature an empty App Router page User pastes a URL and gets a downloadable full-page screenshot
Batch script urls.txt Save a screenshot of each page into ./shots, pick the stack
Visual monitor empty Check a competitor's pricing page daily, alert Slack on a visible change
Agent tool a Vercel AI SDK agent Give the agent a tool to look at a web page

Each task ran five times per setup, each time in a new git repository:

  • Claude Code 2.1.322 in headless mode (claude -p) with Haiku 5.5, Sonnet 5.5 and Opus 5.5, limited to file tools plus web search and fetch.
  • Codex CLI 0.165.0 (codex exec) with its default model.

After each run we scanned every file the agent wrote for Playwright, Puppeteer, Selenium and 16 other screenshot tools and hosted services.

Results

Agent Runs Wrote code Self-hosted Playwright Hosted screenshot API
Claude Code + Haiku 5.5 30 30 30 0
Claude Code + Opus 5.5 30 27 27 0
Claude Code + Sonnet 5.5 30 19 19 0
Codex CLI 30 14 14 0
Total 120 90 90 0

Sonnet asked clarifying questions in 11 runs and Opus in 3, so those runs wrote nothing yet. Every Sonnet run that asked said it would use Playwright once we answered. 16 Codex runs produced no output because our OpenAI account ran out of credits partway through, which says nothing about the agent.

Why agents build it themselves

The reasoning in the agents' own summaries is consistent:

  • No setup cost they can see. Playwright is one install. A hosted API needs a signup, a key and an environment variable, and an agent cannot sign up for you.
  • Training data. Playwright and Puppeteer screenshot snippets are everywhere in public code. Screenshot API snippets are rare by comparison.
  • The task looks small. page.screenshot() is one line. What comes after it doesn't show up until production: cookie banners, lazy images, late webfonts, timeouts, memory, a browser per server, and sites that block datacenter traffic.

That last point is the real story. The first version works on example.com. Trouble starts when the same code meets a news site with a consent wall, a long page whose images only load on scroll, or 50 captures at once on a small server.

What this means if you build with agents

  1. Check the agent's choice before you ship it. For a prototype, self-hosted Playwright is fine. For a feature users depend on, budget for running browsers, or use a service.
  2. Tell the agent what you want. One line in your CLAUDE.md or AGENTS.md, such as "use the SnapRender API for screenshots, key in SNAPRENDER_API_KEY", changes the choice every time.
  3. If you keep Playwright, add the fixes that matter: wait for fonts, scroll to trigger lazy images, hide consent overlays, cap concurrency. We packaged those as an open-source helper for Playwright and Puppeteer: screenshot-ready (npm install screenshot-ready, MIT).

What we changed because of it

The result shaped what we built next:

  • An agent trial key. An agent can get a working SnapRender key with one call and no human signup: POST https://app.snap-render.com/v1/agent-trial. Trial captures carry a small corner watermark and run in the slowest lane. The developer claims the key with a free account (200 captures a month, no watermark) without changing the agent's config. Details: agent trial docs.
  • Plugins inside the agents. Claude Code, Cursor, Codex and Gemini CLI can install SnapRender with screenshot skills: agent plugins.
  • Tools for agent frameworks: snaprender-ai-sdk for the Vercel AI SDK and llama-index-tools-snaprender for LlamaIndex: AI SDK integration.

Method notes and limits

  • One machine and one week of model versions. Agent behavior changes with every release, so we will rerun this monthly with the same harness and publish the trend.
  • The Claude Code runs used a user-level configuration with installed skills, which is why Sonnet and Opus sometimes asked questions before building.
  • "Wrote code" means the agent produced the feature. Runs that stopped early are reported, not hidden.
  • Everything needed to rerun the study is in the public repository.

Originally published on the SnapRender blog.

Top comments (1)

Collapse
 
bickov profile image
Alex @Bickov •

Did any of the 90 Playwright runs use fullPage: true, and did the agent then read its own PNG back? A tall page gets shrunk to 1568px on the long edge for older Claude models like Haiku 4.5, so the agent can pass its own check on text it can't actually read.