arXiv:2611.15737v1 Announce Type: cross Abstract: A vulnerability detector is only as trustworthy as the benchmark that scores it, and today's real-world datasets tend to flatter detectors. They are often built once and never refreshed, so they quickly miss new vulnerabilities, their labels are either noisy or heavily filtered, and their random splits let models train on near-copies of test code and on future vulnerabilities. Above all, when a fix only adds code, such as a missing NULL check, they do not point at where the vulnerability was, preventing detectors from learning to localize some of the most common C/C++ weaknesses. We present VulSpot, a living benchmark for detecting, classifying, and localizing vulnerabilities, with over 500k functions in 21 languages from 24k registered CVEs, including over 15k vulnerable C/C++ functions. By linking the code added by a patch to the existing statement it protects, VulSpot locates the vulnerability in 99.3\% of the functions that previous diff-based datasets used to leave without a location. Rather than discarding uncertain labels, it tells users how reliable each one is, and it comes with leak-free splits, complementary code context, and hierarchical CWE labels. An automated pipeline refreshes VulSpot every month, so that each release brings vulnerabilities no model has seen yet.
Continue to arXiv Software Engineering →You are being forwarded in 6s — stay on this page