When code and tests change together, a green build is not enough.
Imagine a coding agent optimizing a billing endpoint. The tests pass, yet a missing tenant check could let one customer read another company's invoice. What if the agent can also edit the tests used to approve its work? In Design for Churn, Ben Halpern argues that cheap rewrites make durable interfaces, schemas, and behavioral tests more important. I agree. Agents should improve development tests, but final security checks must remain outside that agent's control.
Green tests can mislead in two ways. The first is omission: the suite never checks an important requirement. UTBoost (ACL 2025) identified 36 SWE-bench tasks with insufficient tests, allowing incorrect patches to pass. The second is manipulation: an agent exploits the tests themselves. ImpossibleBench (ICLR 2026) deliberately made tests contradict specifications. Some agents modified tests despite instructions not to; others used shortcuts without editing them. Read-only tests stopped direct edits but not every shortcut. Stronger prompts helped in some settings, but instructions cannot enforce permissions. These are benchmark findings, not production failure rates.
Three practices make a green build more meaningful.
Keep acceptance independent. Agents can update development tests. Critical security checks should come from a protected baseline and run in separately controlled CI with limited privileges. Changes to those checks or the CI definition require independent review. An AGENTS.md file can guide agents, but it cannot enforce permissions. Keeping some security tests hidden from the agent can help, although it also reduces useful feedback.
Test the tests. In a throwaway copy, remove one authorization check at a time and rerun the security suite. If nothing fails, investigate missing coverage. Security-focused mutation testing exposes gaps; it cannot prove a system is secure.
Enforce critical boundaries separately. Database row-level security or a separately enforced authorization policy can limit damage if an application check disappears. That layer also needs restricted privileges and independent tests.
A small, deterministic Python demo shows both the missed check and how mutation testing exposes it. No LLM is involved. The original invoice lookup checks the caller's tenant. A deliberately flawed rewrite replaces a scan with a dictionary lookup but drops the tenant condition. The three existing tests cover only one tenant, so they still pass. Add a negative test that Alice at Acme cannot read a Globex invoice, and the rewrite passes only three of four. A separate mutation removes only the tenant condition from the original function: the basic suite misses it, while the security suite catches it. Try it yourself: github.com/xiaoleiren/green-tests-broken-isolation.
Frequent rewrites are not the problem. Before merging, ask who controls the final checks and whether they would turn red if a security boundary disappeared.
Xiaolei Ren works on software verification and the security of AI-driven systems.
Top comments (8)
This is one of the things that makes me nervous with agent-written code.
I'd let the agent add and update regular tests, but keep the really important checks — auth, tenant isolation, billing — outside its control or behind a separate review. Otherwise it can change the code and lower the bar it's supposed to pass at the same time.
Exactly! Agents should be free to improve everyday tests, but critical security checks need an independent gate. Otherwise, a green build might just mean we've lowered the bar. Thanks for sharing your thoughts! 😀
The line about instructions not enforcing permissions is the one I'd underline for anyone writing agent requirements. The same split works for agents that make business decisions, not just code: people who know the business write the golden set and its expected answers, the agent can't touch them, and the pass mark is set before the first run. If the agent, or the team tuning it, can edit the expected answers, a green run tells you nothing. Your mutation idea carries over too: delete one guardrail rule in a copy and make sure at least one golden case fails. I walk through that setup from dev to prod here: youtube.com/watch?v=IBteCgiXc6Q
Great point! I really like how you extend this idea beyond coding agents to business decisions. Keeping the golden set independent and using mutation checks to test guardrails makes a lot of sense. Thanks for sharing your experience and the video! 😀
Thanks. Next place I'd like to try the mutation idea is the prompt itself. Delete one guardrail line from the system prompt, rerun the golden set, and see whether anything fails. If nothing does, either that line wasn't doing anything or the golden set doesn't cover it. Both are worth knowing.
Thanks for reading!
If you'd like to try the example yourself, the Python demo is available here:
github.com/xiaoleiren/green-tests-...
It's small, dependency-free, and reproducible. Just run
python3 tenant_demo.py.I'd also love to hear how others handle this in practice: when AI agents can modify both code and tests, which security checks should remain outside their control?
Feel free to share your experiences or suggestions!
The mutation check is the part I started doing on auth paths: delete one tenant filter in a throwaway branch and see if CI actually goes red. A green suite that still passes after that is the real bug.
Exactly! That's the key point I wanted to highlight. A green CI build means little if removing a critical authorization check doesn't turn it red.
I really like your throwaway-branch approach. Simple, practical, and easy to reproduce.
Have you found any interesting gaps in your existing tests using this method? Would love to hear more!