← ALB News · curated headline

Think Before You Code: A Benchmark for Testing LLMs on Software Engineering Deliberation

arXiv Software Engineering · WORLD · Compilers · 11/18/2026, 12:00:00 AM

arXiv:2611.22905v1 Announce Type: new Abstract: Think Before You Code (TBYC) is a benchmark pipeline for studying whether code-focused language models can reason about software issues in ways that resemble human engineering deliberation. The benchmark curates GitHub issue discussions into structured records, extracts human discussion artifacts such as problems, solutions, constraints, rationales, and trade-offs, and then prompts models to generate issue-level "thoughts" using retrieved code context and few-shot examples. It converts model responses back into the same artifact space and evaluates them against human annotations using complementary metrics for artifact type detection, metadata matching, tag overlap, and semantic summary similarity. Beyond scoring, this study includes visualization, leaderboard, and baseline tools that make it possible to compare models, prompting strategies, and extraction methods across repositories and issues. Think Before You Code is an end-to-end evaluation environment for measuring how closely LLM software reasoning aligns with human strategic judgment.

Continue to arXiv Software Engineering →

You are being forwarded in 6s — stay on this page