Code2Bench-2505 spun 880 recent Python projects into 1,163 benchmark tasks, and repository-level dynamic benchmarking stopped being a slide-deck phrase. The tasks came out of an automated pipeline, the test suites were synthesized with property-based testing. And the passing bar was execution rather than anyone’s opinion of correct code.
Here is the direct version.
Repository-level dynamic benchmarking evaluates LLMs on tasks generated fresh from real code repositories, with correct answers established by executing the code instead of consulting a fixed answer key a model could have memorized.
Three benchmarks define the approach right now: RepoReason tests whether models can verify assertions about how a program behaves, Code2Bench keeps pulling tasks from recently updated GitHub repositories so the questions never age. And FeatureBench grades agents on feature-level development work across 200 tasks and 3,825 executable environments built from 24 open-source repositories. If you’re the person who picks the coding model for your team or your clients, this is the evaluation layer your leaderboard habit has been missing.
Static benchmarks have a rot problem. When a task and its solution circulate widely enough, a score stops measuring reasoning and starts measuring recall. Dynamic, repository-level tests close that gap by regenerating the ground truth, which is the only version of a benchmark score I’d spend money against.
RepoReason Grades Reasoning, Not Recall
RepoReason is a repository-level code-reasoning benchmark built as a white-box diagnostic.
And its core task has a name worth remembering: Abductive Assertion Verification.
The model receives an assertion about a program and has to work out whether that assertion holds. That is far closer to real engineering than “complete this function,” because most of what we pay models for is judging claims about code we did not write.
The anti-memorization machinery is the interesting part.
RepoReason’s Execution-Driven Mutation framework uses the execution environment as a Semantic Oracle to regenerate ground-truth states, which the paper presents as a way to eliminate memorization while preserving authentic logical depth. Translation: the environment, not a frozen answer sheet, decides what is true.
Mutate the program, run it, record what actually happened. And a model that memorized the original repository is now confidently wrong.
Instead of one aggregate score, RepoReason applies dynamic program slicing and reports three orthogonal metrics: ESV (Reading Load), MCL (Simulation Depth), and DFI (Integration Width). I read those as three distinct ways to fail. A model can drown since it has to read too much, given that it cannot simulate execution deeply enough, or as integrating changes across too many modules exceeds what it can hold together. A single leaderboard number flattens all three into a blur, and a diagnostic splits them apart.
That split matters at the point of purchase.
When a client asks why the model failed on their repo, “it scored 71” is not an answer. “It collapsed once simulation depth went up” is an answer. And it points at a fix.
Code2Bench Builds Tests That Never Age
Contamination is the quiet scandal of benchmarking. If the tasks and solutions are public long enough, models can memorize answers instead of deriving them. And the score inflates without capability moving an inch.
That is the failure mode static leaderboards are structurally unable to admit.
Code2Bench attacks it with process rather than promises.
The framework is an automated, end-to-end pipeline for constructing contamination-resistant benchmark instances directly from recent real-world GitHub repositories.
And it continuously ingests code from active repositories with a focus on recent updates. You cannot memorize your way through a test that keeps getting rewritten.
The internals deserve a look even if you never run it. For each function, the pipeline uses language-agnostic scope-graph reference resolution to analyze external dependencies, then sorts tasks into Self-Contained and Weakly Self-Contained classes, with the weakly-contained ones permitting library usage. The test suites themselves are synthesized through property-based testing. The resulting dataset, Code2Bench-2505, was derived from 880 recent Python projects and comprises 1,163 code generation tasks with 100% average branch coverage on the ground-truth implementations.
That last number deserves more attention than it gets. Branch coverage is the difference between a test that catches subtly wrong code and a test that waves it through. Ground truth at 100% average branch coverage means the answer key actually exercises the paths where broken implementations hide.
This is what dynamic benchmarking means in practice: the benchmark is no longer an artifact, it is a process. The paper’s pipeline can run again next quarter against different repositories.
And the scores stay comparable since the generation method, not the specific questions, is the constant.
FeatureBench Tests the Job You Actually Have
Assertion checks and function tasks are still proxies for the work.
FeatureBench goes further and evaluates LLM-powered agents on feature-level, real-world software development tasks through an automated, execution-based pipeline. Version one packs 200 challenging evaluation tasks and 3,825 executable environments curated from 24 open-source repositories.
The task mining is the clever bit. FeatureBench traces from unit tests along a dependency graph to identify feature-level work, which means a single task can span multiple commits and pull requests scattered across a repository’s development timeline. Every environment gets initialized with Docker, so each task starts from a runnable repository rather than a bare prompt.
That is the shape of client work.
Nobody invoices for “write a function”; they invoice for “add this capability and do not break anything else,” which is exactly a multi-commit change judged by whether the surrounding features still function.
The paper even notes the approach ensures the proper functioning of other features after the task is separated out.
A benchmark that measures that, automatically, tells you where evaluation is heading: closer to your actual job and further away from trivia.
What to Steal If You Run a Small Shop
I have not run these three benchmarks against a client repository. And you will not either; they are research instruments. The design pattern underneath them is free, though, and it is worth copying wholesale.
– Make execution the judge. Your test suite already regenerates ground truth on every run. Pull recent real changes from your own repositories, have candidate models attempt them, and let the suite decide who passes. That is the Semantic Oracle trick with zero licensing fees.
– Stop buying on leaderboard deltas. A benchmark that cannot separate memorization from reasoning cannot inform a purchase decision for work on code the model has never seen. All client work is code the model has never seen.
– Diagnose failures in three shapes. When a model falls over on your repo, ask which failure it is: reading load, simulation depth, or integration width. You probably will not compute ESV, MCL, and DFI yourself, but that three-way split is a better debugging vocabulary than “the model is dumb.”
The uncomfortable summary is this.
Models you evaluated on static benchmarks were being graded on homework they had seen before. This generation of repository-level benchmarks grades them on homework generated after the syllabus is printed. And the scores from the second kind are the ones that survive contact with a production repo.
If you want that pattern built for real, that is my lane.
My agency builds execution-judged evaluation setups from a client’s own repositories before anyone commits to a model or an agent stack. Bring one repository with a working test suite. And you will leave knowing exactly what your model does when memorization stops being an option.
