Eran Cohen

Security Researcher @ A Security

Eran Cohen is a Security Researcher at A Security, where he builds an autonomous red-team agent, weaponizing AI for the good guys. He is also part of the team behind ARENA, an open-source lab generator for security research. Since 2018, he has worked across security research, application security, and security architecture. His recent research into insecure AI-generated code was covered by The Hacker News and Cybernews. In his free time, he hunts bugs in AI infrastructure, from chatbots and app builders to MCP servers.

Beyond Coverage: How to Benchmark AI Web Pentest Agents and Frontier Models

Friday, 13 November 2026, 16:00 - 16:20

Evaluating an AI web red-teamer has no good options. Public CTFs are in every recent model's training data, so scores measure recall of a walkthrough. Real production targets are realistic but have no ground truth, so when the agent reports five findings and goes quiet, you cannot distinguish a clean app from a half-blind agent. "What it missed" is the whole question and neither option answers it.

We built ARENA, a lab generator that produces fresh, realistic web applications on demand with a known answer key: you specify vulnerability classes, counts, difficulty, and production friction such as hidden endpoints and WAF behavior. Seven frontier models ran as the exploitation engine inside one multi-agent harness (discovery through exploitation, this study benchmarking the exploitation half) against one generated ticketing app, three runs each. All seven got the same fixed 110-lead baseline, a subset of the leads discovery produced, holding every ground-truth bug plus extra leads. Allowing us to observe extra TP outside the answer key, FP and scope drifts.

Results. We start against the 8 seeded bugs: a heatmap of every model against every planted vulnerability, showing in how many of its three runs each one found it. Consistency shows up immediately. But the models kept reporting true positives that were never in our key, so we re-scored everything against a Validated Ground Truth built with LLM-as-a-judge and manual adjustment. The results then cover cost, full coverage, fallbacks, runtime, scope drift and advisor usage.