How to verify AI-discovered vulnerabilities aren't just training data echoes

The setup Last month a friend DM'd me a screenshot. An AI security agent had "discovered" a vulnerability in a popular open-source project. The agent walked through exploitation steps, suggested a patch, the whole nine yards. Looked legit. Then someone pointed out the CVE ID it kept almost-quoting was from years earlier. This is going to keep happening. As we wire LLMs into vulnerability research workflows, we run into a problem that doesn't have a clean analogue in traditional static analysis: