arXiv:2610.33389v1 Announce Type: new Abstract: Code review requires finding functional bugs, memory-safety errors, security weaknesses, and performance problems across languages and software stacks. Tools can express bounded parts of this work as decisions about supplied code. We present OneCodeReviewBench, 1,968 synthetic cases across 12 programming languages and YAML, with Boolean, categorical, and ordered answers. Explicit contracts specify runtime assumptions, source scope, and answer criteria; separate evidence records support reference review. One author reviewed 150 cases; two additional engineers reviewed 100 of those cases. We compare ten model--adapter systems on the same cases and contracts: OpenAI Decisions reaches 81.3% Boolean balanced accuracy; Kev-27B reaches 76.8% Choice accuracy and Score mean absolute error of 0.367. Excluding two families with incomplete contracts preserves these leaders. Additional comparisons cover 586 cases with raw symbol search and 40 with a bounded Python import resolver; the resolver matches all 40 references within its supported profile. A versioned artifact reproduces these analyses and retains the 85 cases with matching engineer answers; the reported review subset excludes two incomplete-contract cases. Remaining reviewer disagreements and repaired inputs require resolution; repository-scale transfer and matched latency/cost evaluation remain future work.
Continue to arXiv Software Engineering →You are being forwarded in 6s — stay on this page