arXiv:2611.10098v1 Announce Type: new Abstract: Can an autonomous cross-provider review step change whether code written for a real issue passes hidden tests? We report a retrospective benchmark of 12 historical GitHub issues (five C#, five JavaScript, and two Python) from five repositories. Two directed author--reviewer pairs, OpenAI GPT-6.1-Sol and Anthropic Claude Sonnet 5.5, each ran twice per issue, yielding 48 planned workflows. A bounded terminal agent produced an initial patch; a cross-provider model reviewed it; the author model adjudicated findings and could revise the patch. The same hidden evaluator assessed initial and final patches. Forty workflows completed, while eight terminated before a complete evaluator outcome. Four completed workflows passed, and all four already passed before review. No initial-to-final pass/fail transitions occurred among completed workflows.
Continue to arXiv Software Engineering →You are being forwarded in 6s — stay on this page