Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery

AI security research · arXiv cs.CR (AI security)

Where Do the Tokens Go? Understanding and Reducing Costs in LLM Agents for Vulnerability Discovery

REDCORE assessment

The research highlights that increasing the token spend for LLM agents does not linearly correlate with success in vulnerability discovery, often resulting in wasteful resource consumption. It identifies specific bottlenecks in code reasoning that lead to inefficiencies in agentic workflows. Defenders should conclude that agentic autonomy should be gated by evidence-based reasoning frameworks rather than simply increasing context window or token limits.

Impact for lean teams

Small teams using AI agents for code analysis or security tasks risk significant financial loss due to runaway token consumption without guaranteed improvements in code quality. Teams should monitor agent costs closely and avoid deploying autonomous agents in production environments without strict budget caps.

Source summary

LLM agents can spend millions of tokens during vulnerability discovery without producing a working proof of concept (PoC). What consumes that budget, and why does it fail to produce results? We diagnose these costs and failures through a multi-axis open-coding study of 200 CyberGym traces, spanning four agents (i.e., Codex, OpenCode, Cybench, and EnIGMA) under an unaided baseline and four existing efficiency methods. The study reveals three key findings. First, different agents vary substantially in success and cost, and higher spending does not consistently yield better outcomes. Second, code localization and understanding, together with vulnerability reasoning and trigger design, account for 60.4% of tokens and represent the two leading bottlenecks in failed runs. Third, only 24.4% of matched comparisons preserve success at lower total cost; unsuitable signals and auxiliary overhead li

Recommended actions

• Implement strict token usage limits and budget caps per task for all deployed LLM agents.
• Evaluate the necessity of agentic workflows versus manual or script-based code review to reduce operational overhead.
• Verify the cost-to-performance ratio of specific agent models before integration into standard operating procedures.
• Establish clear human-in-the-loop validation checkpoints to prevent agents from performing redundant or unnecessary reasoning steps.

This assessment was generated by REDCORE AI from the attributed source above and is provided for situational awareness. Validate against the original research and your own environment before acting.