Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools

AI security research · arXiv cs.CR (AI security)

Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools

REDCORE assessment

The LLM4SAST framework demonstrates that large language models can effectively automate the identification of logic and functional flaws in static analysis tools by generating diverse code variants. While this research proves SAST tools suffer from reliability gaps, it is an offensive testing advancement rather than a direct vulnerability in a typical production AI application. Defenders should conclude that reliance on single-tool automated security gates provides a false sense of security.

Impact for lean teams

This research highlights that SAST tools are not infallible and can produce inconsistent results when processing LLM-transformed code. Lean teams should be aware that automated security scanning is a component, not a replacement, for manual code review.

Source summary

Static application security testing (SAST) tools help detect potential software vulnerabilities, but they themselves contain faults undermining reliability and usability. Automated testing can address these issues, yet current techniques suffer from limited program transformation diversity and lack mechanisms to verify generated variants, leading to reduced fault detection and high manual effort. We present LLM4SAST, the first universal framework detecting faults in SAST tools using large language models and dynamic feedback. Our key insight is leveraging LLMs to understand and transform code in diverse programming languages. We design novel metamorphic relations with transformations to guide variant generation, using an LLM to produce multi-language variants that satisfy these relations. To reduce erroneous variants and identify unique faults, we propose two heuristics: dynamic-feedback

Recommended actions

• Verify critical security findings with secondary manual review or peer inspection.
• Implement defense-in-depth by using multiple scanning tools to cross-reference results.
• Monitor official vendor advisories for the SAST tools currently integrated into your CI/CD pipelines.
• Maintain a clear separation between LLM-generated code assistance and the final production-deployed codebase.

This assessment was generated by REDCORE AI from the attributed source above and is provided for situational awareness. Validate against the original research and your own environment before acting.