← ALB News · curated headline

We Fixed the Scorer First, and Cluster Correction Killed Our Own Planted-Defect Result in AI Code Review

arXiv Software Engineering · WORLD · Compilers · 11/18/2026, 12:00:00 AM

arXiv:2611.21635v1 Announce Type: new Abstract: Judging an AI code-review pipeline only by how much it catches hides an axis: what it dropped. Stronger selection raises precision for free; the cost never reaches the report. We ran four arms on the same eight cases, one scorer, asking whether planted-defect detection can rise without raising the claim error rate. Three repetitions are nested in eight cases, so p is reported per contrast and twice: under an exact permutation test over cases (2^8 arrangements) the two contrasts publishable under independence (Fisher p=0.017, 0.019) move to 0.0625 and 0.109, and after Holm correction no contrast separates. This is "not observable with this design," not "no effect": the sign test needs six discordant cases in one direction, its realized type-I rate under the null is 0.0078, and 80% power needs a per-case win probability near 0.80; closing it takes 12-25 cases, not repetitions. Within a run, a second pass never removes what the first found (0/24, 0/24 paired losses) while its gains (+3, +2) fall in one case; baselines on byte-identical prompts differed 15/24 vs 20/24, so run-to-run variance exceeds the effect. The claim error rate is 0.0% (0/178), with no upper bound on the rate claimed; scorer discrimination was measured with pre-registered positive controls (12/12 fabricated claims rejected, 0 false rejections). We retract our original causal explanation and report the instrument defects we hit. The contributions are not the numbers: (1) fixing the instrument before seeing results, then killing our own main effect with it; (2) the identity eps-hat = eps-hat(pi): a reported error rate is computed only on the subset a retention policy kept, silent about what was dropped, so it does not identify the quantity an irreversible gate needs; no improvement in the number promotes an automated verdict to publish, delete, or history-rewrite surfaces; a mechanized refusal enforces this.

Continue to arXiv Software Engineering →

You are being forwarded in 6s — stay on this page