Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
One useful measure for the panel might be the number of independently verified findings per unit of reviewer time, rather than the number of models agreeing. I’d also keep a small set of seeded failures and known false positives: that makes it possible to see whether adding a reviewer improves coverage or just increases the triage queue.
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I like the idea of seeded failures and known false positives as a way to measure whether adding reviewers actually improves the outcome. That's a much more useful measure than simply counting how many models agree.
One thing I'd add from my experiments is that reviewer time alone doesn't capture the full cost. Running the panel was relatively inexpensive compared to investigating and verifying its findings. A reviewer that produces twenty plausible findings, only two of which hold up, can create substantially more work than one that produces three findings with two independently verified.
I've also seen cases where smaller panels performed as well as, or better than, larger ones. More reviewers don't necessarily mean better coverage.
That raises an interesting measurement question: should we optimize for verified findings per reviewer hour, per verification hour, or per total investigation effort? I'm leaning toward the last one, since that's ultimately the cost of deciding whether a finding deserves action.
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
I’d use total investigation effort as the main measure, and keep reviewer time and model cost as separate components so the team can see where the cost moves. I’d also report serious defects missed on a held-out set: optimizing verified findings per hour alone could reward a quiet reviewer that finds almost nothing. How do you track the findings that weren’t discovered in the first place?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
That's an important limitation of the metric, and I agree that total investigation effort is the better primary cost measure. A reviewer that produces very few findings could look remarkably efficient while missing the problems we actually care about.
I think there are two different measurement problems here. With a controlled benchmark, we can seed known defects, retain known false positives, and measure which failures each reviewer discovers. That gives us at least a bounded measure of recall, particularly if we keep some cases out of the reviewer's tuning process.
In an actual production review, though, we don't know the complete population of defects. We can track findings discovered later through testing, incidents, or subsequent independent reviews, but those only tell us about misses that eventually became visible.
So I'd probably report verified findings per total investigation effort alongside detection rates on a held-out benchmark, while being explicit that neither establishes production completeness.
The unresolved question is how representative that benchmark is of the failures we're actually shipping. That's where I'd be cautious about claiming one panel configuration is definitively better than another.
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
That distinction helps. To check whether the held-out set stays representative, I would periodically sample later incidents and independent review findings, then compare their failure modes with the benchmark. Cases used to revise the benchmark should leave the held-out pool; otherwise the next score measures tuning as well as detection. The remaining unknowns still need to be reported, not treated as zero misses.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Good point, and I agree with the core: author and reviewer trained on similar data share blind spots. But I'd add a nuance I ran into while using models to audit code: the framing matters as much as the data.
Auditing a project of mine, with no context prompt the model handed me a list of vulnerabilities that were mostly false positives. As soon as I clarified what the code was for — that it's executed by the web AI, not the user — the list of 6 reds dropped to cosmetic fixes. An audit without intent is noise.
The corollary: that's why it's worth auditing in two stages. First clean code, no prompt, so the auditor doesn't inherit your framing and finds what you wouldn't. Then, with the threat model made explicit, to filter by real applicability. The risk in the second stage is inducing false negatives: if you tell it "the AI executes this," the auditor may accept it and stop flagging what happens when that assumption fails. That's why the order matters, and depending on what you're after, a third directed adversarial pass helps ("assume X fails, what happens?").
One practical note: what makes the contextual stage viable is persistence of the framing. In long threads (I use DeepSeek web for that reason, not raw power) the model keeps the threat model alive 30 replies later; if it loses it, it falls back to the default and the noise return
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I think there’s an important distinction here between giving a reviewer the system’s intended operating contract and giving it the author’s interpretation of whether the code satisfies that contract.
I like the idea of separating discovery from contextual evaluation, particularly the adversarial third pass. My concern with a completely context-free first pass is that it can produce findings against assumptions the system was never designed to satisfy. Those might be useful hypotheses, but they’re not necessarily defects.
The question I’ve been exploring is how to provide enough context for meaningful review without also supplying the conclusions we want the reviewer to reach. An explicit threat model helps, but as you point out, the reviewer also needs to challenge its assumptions rather than simply accept them.
That seems like another reason to separate identifying potential defects from independently verifying them.
Dev individual. Creador de WAIO 🐙 — motor local que da "manos y ojos" a IAs web (DeepSeek, ChatGPT, Gemini) sin APIs ni tokens. Python · Windows · MIT. Hecho en Argentina 🧉
Another point: steps 1–3 don't always give a convincing answer. When that happens, I escalate to a 4–5 step: I put the auditor thread against the thread that wrote the code. I hand the report to the producer and ask it to flag what's a false positive and why; then I take that answer back to the auditor. Only once they reach a consensus do I review the code on the agreed points.
I don't do this every time, because it has a cost: the producer tends to have the advantage since it knows the code, and the two sides can converge out of exhaustion rather than agreement. Right now I handle it by feel; it would probably be more thorough to require rules, for example that every dismissal comes with the assumption under which it changes ("it's a false positive if the executor is the AI; it stops being one if that assumption fails"), so the consensus is about applicability rather than who's right. That's the part I haven't systematized yet.
In practice it answers your open question: I give the context to the auditor as inputs, but I do the verification with two opposing framings, not just one.
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
I think your observation about convergence through exhaustion is particularly important. Two agents eventually agreeing doesn't necessarily mean either has established the correct answer. It may just mean one framing has prevailed.
Requiring each dismissal to identify the assumption that changes the finding's applicability seems like a useful improvement. It gives you something concrete to test rather than relying on which agent makes the more persuasive argument.
I'd be inclined to preserve the disagreement as well, even after reaching a decision. The original finding, the producer's objection, the assumptions involved, and the eventual disposition all have value as evidence of how that decision was reached.
That may be the distinction I'd draw between reaching consensus and establishing a verified result. Consensus can help prioritize an investigation, but it shouldn't substitute for the evidence that resolves it.
Two reviewers agreeing on the GITHUB_SHA claim wouldn't make it stand in my setup. For a blocker I ask Codex for three things: the input or steps that trigger it, the expected behaviour and the actual behaviour. A finding without a reproducible failure goes to follow-ups, and an unverifiable claim gets a request for evidence before anything is patched. That puts part of the verification back on the reviewer that made the claim. Did those two show a run where the workflow saw the wrong commit, or only assert it?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
That’s exactly the distinction I think gets lost when people focus on reviewer agreement. Two models independently making the same claim doesn’t establish that the claim is correct.
I like your requirement for triggering conditions, expected behavior, and actual behavior. It forces the reviewer to provide something testable rather than just a plausible explanation of what might go wrong.
In the GITHUB_SHA case, the important result was that the finding didn’t survive verification. The agreement between reviewers made it worth investigating, but it wasn’t sufficient evidence to justify changing the code.
One thing I’d add is that even a reviewer-supplied reproduction needs independent scrutiny. A model can construct a convincing test around an incorrect assumption just as easily as it can produce a convincing explanation.
That distinction between identifying, reproducing, and independently verifying a defect is becoming increasingly important in how I evaluate these tools.
In my loop the reviewer's reproduction arrives without ever having been run. The review is static: Codex reads the diff and the checkout and runs no tests, so its trigger steps are a claim on paper, and all the scrutiny you describe has to come after the review. A finding stands if it comes with a reproducible failure and goes to follow-ups if it doesn't. For one that stands, the fix goes back to Claude Code as a new commit, that commit gets a new review, and scope or anything the brief doesn't settle goes to a person. For the GITHUB_SHA finding, did you read the release event docs yourself, or did your agent check them and report back?
Developer educator and advocate who still ships code. 15 years teaching developers at MongoDB, Heroku, Cisco, and Treehouse. Now writing about AI-assisted development, agent memory, and provenance.
That's a useful clarification about your workflow. Having the reviewer supply reproduction steps without executing them makes the boundary between a proposed finding and a verified defect quite explicit.
I also like that a fix goes through another review rather than assuming that addressing the original finding necessarily produces correct code.
On the GITHUB_SHA question, I checked the documentation manually and also used an agent to check it.
That's an important distinction for the experiment itself. If an agent reports that a finding is invalid based on documentation it consulted, I still need to establish whether the documentation actually supports that conclusion. Otherwise, I've potentially replaced one unverified model claim with another.
It's one reason I'm increasingly interested in preserving the evidence and verification steps alongside the final disposition, rather than recording only whether a finding was accepted or dismissed.
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
Your note that one review had to leave files out to fit is the case I would build the gate around. A reviewer that never saw a file reports nothing about it, and nothing reads exactly like a clean pass; that is "absence does not raise" one level up, in the review itself rather than in the code.
So I would make coverage part of the verdict: for each reviewer, which files reached it and whether the run finished, recorded next to the findings. A finding from one reviewer is evidence about the lines it read. A pass is only evidence if every reviewer finished on the full diff, and a run that left files out should return no verdict rather than a green one.
We're a place where coders share, stay up-to-date and grow their careers.
One useful measure for the panel might be the number of independently verified findings per unit of reviewer time, rather than the number of models agreeing. I’d also keep a small set of seeded failures and known false positives: that makes it possible to see whether adding a reviewer improves coverage or just increases the triage queue.
I like the idea of seeded failures and known false positives as a way to measure whether adding reviewers actually improves the outcome. That's a much more useful measure than simply counting how many models agree.
One thing I'd add from my experiments is that reviewer time alone doesn't capture the full cost. Running the panel was relatively inexpensive compared to investigating and verifying its findings. A reviewer that produces twenty plausible findings, only two of which hold up, can create substantially more work than one that produces three findings with two independently verified.
I've also seen cases where smaller panels performed as well as, or better than, larger ones. More reviewers don't necessarily mean better coverage.
That raises an interesting measurement question: should we optimize for verified findings per reviewer hour, per verification hour, or per total investigation effort? I'm leaning toward the last one, since that's ultimately the cost of deciding whether a finding deserves action.
I’d use total investigation effort as the main measure, and keep reviewer time and model cost as separate components so the team can see where the cost moves. I’d also report serious defects missed on a held-out set: optimizing verified findings per hour alone could reward a quiet reviewer that finds almost nothing. How do you track the findings that weren’t discovered in the first place?
That's an important limitation of the metric, and I agree that total investigation effort is the better primary cost measure. A reviewer that produces very few findings could look remarkably efficient while missing the problems we actually care about.
I think there are two different measurement problems here. With a controlled benchmark, we can seed known defects, retain known false positives, and measure which failures each reviewer discovers. That gives us at least a bounded measure of recall, particularly if we keep some cases out of the reviewer's tuning process.
In an actual production review, though, we don't know the complete population of defects. We can track findings discovered later through testing, incidents, or subsequent independent reviews, but those only tell us about misses that eventually became visible.
So I'd probably report verified findings per total investigation effort alongside detection rates on a held-out benchmark, while being explicit that neither establishes production completeness.
The unresolved question is how representative that benchmark is of the failures we're actually shipping. That's where I'd be cautious about claiming one panel configuration is definitively better than another.
That distinction helps. To check whether the held-out set stays representative, I would periodically sample later incidents and independent review findings, then compare their failure modes with the benchmark. Cases used to revise the benchmark should leave the held-out pool; otherwise the next score measures tuning as well as detection. The remaining unknowns still need to be reported, not treated as zero misses.
Good point, and I agree with the core: author and reviewer trained on similar data share blind spots. But I'd add a nuance I ran into while using models to audit code: the framing matters as much as the data.
Auditing a project of mine, with no context prompt the model handed me a list of vulnerabilities that were mostly false positives. As soon as I clarified what the code was for — that it's executed by the web AI, not the user — the list of 6 reds dropped to cosmetic fixes. An audit without intent is noise.
The corollary: that's why it's worth auditing in two stages. First clean code, no prompt, so the auditor doesn't inherit your framing and finds what you wouldn't. Then, with the threat model made explicit, to filter by real applicability. The risk in the second stage is inducing false negatives: if you tell it "the AI executes this," the auditor may accept it and stop flagging what happens when that assumption fails. That's why the order matters, and depending on what you're after, a third directed adversarial pass helps ("assume X fails, what happens?").
One practical note: what makes the contextual stage viable is persistence of the framing. In long threads (I use DeepSeek web for that reason, not raw power) the model keeps the threat model alive 30 replies later; if it loses it, it falls back to the default and the noise return
I think there’s an important distinction here between giving a reviewer the system’s intended operating contract and giving it the author’s interpretation of whether the code satisfies that contract.
I like the idea of separating discovery from contextual evaluation, particularly the adversarial third pass. My concern with a completely context-free first pass is that it can produce findings against assumptions the system was never designed to satisfy. Those might be useful hypotheses, but they’re not necessarily defects.
The question I’ve been exploring is how to provide enough context for meaningful review without also supplying the conclusions we want the reviewer to reach. An explicit threat model helps, but as you point out, the reviewer also needs to challenge its assumptions rather than simply accept them.
That seems like another reason to separate identifying potential defects from independently verifying them.
Another point: steps 1–3 don't always give a convincing answer. When that happens, I escalate to a 4–5 step: I put the auditor thread against the thread that wrote the code. I hand the report to the producer and ask it to flag what's a false positive and why; then I take that answer back to the auditor. Only once they reach a consensus do I review the code on the agreed points.
I don't do this every time, because it has a cost: the producer tends to have the advantage since it knows the code, and the two sides can converge out of exhaustion rather than agreement. Right now I handle it by feel; it would probably be more thorough to require rules, for example that every dismissal comes with the assumption under which it changes ("it's a false positive if the executor is the AI; it stops being one if that assumption fails"), so the consensus is about applicability rather than who's right. That's the part I haven't systematized yet.
In practice it answers your open question: I give the context to the auditor as inputs, but I do the verification with two opposing framings, not just one.
I think your observation about convergence through exhaustion is particularly important. Two agents eventually agreeing doesn't necessarily mean either has established the correct answer. It may just mean one framing has prevailed.
Requiring each dismissal to identify the assumption that changes the finding's applicability seems like a useful improvement. It gives you something concrete to test rather than relying on which agent makes the more persuasive argument.
I'd be inclined to preserve the disagreement as well, even after reaching a decision. The original finding, the producer's objection, the assumptions involved, and the eventual disposition all have value as evidence of how that decision was reached.
That may be the distinction I'd draw between reaching consensus and establishing a verified result. Consensus can help prioritize an investigation, but it shouldn't substitute for the evidence that resolves it.
Two reviewers agreeing on the GITHUB_SHA claim wouldn't make it stand in my setup. For a blocker I ask Codex for three things: the input or steps that trigger it, the expected behaviour and the actual behaviour. A finding without a reproducible failure goes to follow-ups, and an unverifiable claim gets a request for evidence before anything is patched. That puts part of the verification back on the reviewer that made the claim. Did those two show a run where the workflow saw the wrong commit, or only assert it?
That’s exactly the distinction I think gets lost when people focus on reviewer agreement. Two models independently making the same claim doesn’t establish that the claim is correct.
I like your requirement for triggering conditions, expected behavior, and actual behavior. It forces the reviewer to provide something testable rather than just a plausible explanation of what might go wrong.
In the GITHUB_SHA case, the important result was that the finding didn’t survive verification. The agreement between reviewers made it worth investigating, but it wasn’t sufficient evidence to justify changing the code.
One thing I’d add is that even a reviewer-supplied reproduction needs independent scrutiny. A model can construct a convincing test around an incorrect assumption just as easily as it can produce a convincing explanation.
That distinction between identifying, reproducing, and independently verifying a defect is becoming increasingly important in how I evaluate these tools.
In my loop the reviewer's reproduction arrives without ever having been run. The review is static: Codex reads the diff and the checkout and runs no tests, so its trigger steps are a claim on paper, and all the scrutiny you describe has to come after the review. A finding stands if it comes with a reproducible failure and goes to follow-ups if it doesn't. For one that stands, the fix goes back to Claude Code as a new commit, that commit gets a new review, and scope or anything the brief doesn't settle goes to a person. For the GITHUB_SHA finding, did you read the release event docs yourself, or did your agent check them and report back?
That's a useful clarification about your workflow. Having the reviewer supply reproduction steps without executing them makes the boundary between a proposed finding and a verified defect quite explicit.
I also like that a fix goes through another review rather than assuming that addressing the original finding necessarily produces correct code.
On the GITHUB_SHA question, I checked the documentation manually and also used an agent to check it.
That's an important distinction for the experiment itself. If an agent reports that a finding is invalid based on documentation it consulted, I still need to establish whether the documentation actually supports that conclusion. Otherwise, I've potentially replaced one unverified model claim with another.
It's one reason I'm increasingly interested in preserving the evidence and verification steps alongside the final disposition, rather than recording only whether a finding was accepted or dismissed.
Your note that one review had to leave files out to fit is the case I would build the gate around. A reviewer that never saw a file reports nothing about it, and nothing reads exactly like a clean pass; that is "absence does not raise" one level up, in the review itself rather than in the code.
So I would make coverage part of the verdict: for each reviewer, which files reached it and whether the run finished, recorded next to the findings. A finding from one reviewer is evidence about the lines it read. A pass is only evidence if every reviewer finished on the full diff, and a run that left files out should return no verdict rather than a green one.