Under 20%. That’s the best pass rate any system posted on LongCLI-Bench’s long-horizon command-line tasks. And most runs stall below 30% completion before they’ve meaningfully started (LongCLI-Bench). Claude Code running Claude Opus 4.6 hit the wall. Codex running GPT-5.3-Codex hit the same wall. Long-horizon planning. In agent terms, holding one objective across dozens or hundreds of steps and still knowing many steps later what the first one was for.
Isn’t an edge case here, it’s the default outcome.
And a stack of new benchmarks finally quantifies it.
So why do agents fail at long-horizon planning? Flat answer: individual steps go fine and the run loses the thread. LongCLI-Bench puts every state-of-the-art system below a 20% pass rate and names strategic planning and execution proficiency as what’s still holding agents back. Long-Horizon-Terminal-Bench ran 17 frontier models through 46 long terminal tasks and clocked them averaging 9.8M tokens and 88.9 minutes per run without reliably finishing (source). Raw model quality isn’t the constraint. Persistence and verification are. The strongest survival predictor in a simulated year-long startup run wasn’t the model at all. It was a scratchpad.
Why Long-Horizon Planning Fails in the First 30%
The most useful thing in LongCLI-Bench isn’t the 20% ceiling, it’s where the failures sit. Step-level analysis shows the majority of tasks stall at less than 30% completion, which the authors read as critical failures occurring in the early stages (source). Your agent isn’t running out of gas at the hard part near the end. It’s dying while the plan is still being formed.
That changes what you debug.
If failures clustered near the finish, you’d buy a smarter model for the endgame and call it fixed. They don’t. They cluster at the start, so retrying with a bigger model is the wrong move and an expensive one. Inspect the plan before any tool gets called. Because a bad plan discovered deep into a run has already burned everything behind it.
The paper’s own conclusion is that strategic planning and execution proficiency are what still limit current autonomous agents (source).
Not knowledge. Not tool use. Planning.
What Long-Horizon Planning Research Says About Memory
Long-Horizon-Terminal-Bench is the expensive version of the same lesson.
It’s 46 long-horizon terminal tasks across nine categories. Experiment reproduction, software engineering, multimodal analysis, interactive games, scientific computing. And 17 frontier models went through it, including Grok 4.5 and GPT-5.6 (source). What does a failed run actually cost you? 9.8M tokens on average, roughly 239 episodes, 88.9 minutes of execution time per run. And no finished task (source).
The benchmark’s analysis states the core problem plainly: agents “fail not given that every local step is wrong.
But as they cannot reliably sustain progress, verify completion.
And finish long-horizon tasks within budget.” The steps are mostly right. The run still dies. That’s the signature you get when nothing remembers what the steps were for.
One design choice there deserves copying into your own evals.
Tasks get decomposed into fine-grained graded subtasks rather than scored as binary pass/fail (source), which is how you see a run that’s most of the way correct. The realistic outcome for anything you’d actually deploy. Binary scoring can’t tell an agent that almost finished from one that never really started.
Then there’s YC-Bench, the closest thing in this batch to the job you actually hand an agent: a simulated startup operated over a one-year horizon spanning hundreds of turns (source). Scratchpad usage was the strongest predictor of success, described as “the sole mechanism for persisting information across context truncation.” Not which frontier model you picked — whether anything survived between turns. Context truncation eats plans, and an agent that loses its plan mid-run doesn’t know it lost it.
The second finding is less comfortable. Adversarial client detection was the primary failure mode, accounting for 47% of bankruptcies (source): counterparties lie and the agent pays. Translate that to the real world and an agent planning across hundreds of turns will eventually take input from something adversarial. And being right at each step won’t protect it from being wrong about the whole engagement. Over-parallelization showed up as its own distinct frontier-model failure mode too (source) — familiar to anyone whose agent spawned a fleet of parallel subtasks when one would’ve done. YC-Bench is open-source, reproducible. And configurable (GitHub), so you can run your stack through a simulated year before trusting it with a real quarter.
Single-Turn Defenses Don’t Survive Long Runs
AgentLAB is the benchmark that should kill one specific bad habit.
It’s the first benchmark dedicated to evaluating LLM agent susceptibility to adaptive, long-horizon attacks (source).
And its headline result is two sentences to pin above your orchestration layer: representative agents remain highly susceptible to long-horizon attacks. And defenses designed for single-turn interactions fail to reliably mitigate long-horizon threats.
A guardrail you configure at deployment is a single-turn defense.
It checks out early, you stop checking, and a patient attack just waits until nobody’s looking. That’s the operational risk sitting inside every set-and-forget pipeline, and it compounds with run length.
HORIZON adds the diagnostic layer the others skip.
A cross-domain benchmark built to study long-horizon failure behaviors, with GPT-5 variants and Claude models evaluated across more than 3,100 trajectories in four agentic domains (source). For failure attribution they run a trajectory-grounded LLM-as-a-Judge pipeline, validated against human annotation with inter-annotator agreement of kappa 0.61 and human-judge agreement of 0.84 (source). Humans only moderately agree with each other about why an agent failed. So be suspicious of any single confident postmortem. The judge pipeline itself is the part worth stealing: replay the trajectory, have a model attribute the failure, then check that attribution against your own read.
FAQ on Long-Horizon Planning
Why do AI agents fail at long-horizon planning? Individual steps pass while the run loses the thread — LongCLI-Bench’s step-level analysis puts most stalls below 30% completion, with critical failures in the early stages.
What is a scratchpad in agent planning? A place outside the context window where the agent writes state down and re-reads it. In YC-Bench it was “the sole mechanism for persisting information across context truncation” and the strongest predictor of success.
How many tokens does a long-horizon agent task burn? Agents on Long-Horizon-Terminal-Bench averaged 9.8M tokens, roughly 239 episodes, and 88.9 minutes per run.
Do bigger models fix long-horizon planning failures?
The evidence says no — every state-of-the-art system stayed under a 20% pass rate. And the strongest survival predictor was a scratchpad, not model choice.
Which benchmarks test long-horizon planning?
LongCLI-Bench, Long-Horizon-Terminal-Bench, YC-Bench, AgentLAB, and HORIZON, each covering a different slice of the failure surface.
What I’d Do With These Numbers
Four papers, one consistent conclusion: the constraint isn’t model quality, it’s persistence across a long run.
That converts into a deployment checklist that costs almost nothing.
– Persist state outside the context window. The strongest predictor of success in a year-long simulated run was a scratchpad, and a simple file the agent re-reads on a schedule counts.
– Grade subtasks, not just outcomes. Fine-grained subtask grading is what exposed the early-stall pattern that binary pass/fail hides.
– Kill stalls early. Most failures happen below 30% completion, so a run looping near the start is a failure you’re watching in real time. And a budget cap turns it into a bounded loss instead of a 9.8M-token one.
– Re-check invariants throughout the run. Single-turn defenses don’t survive long horizons, so whatever you validated at deployment, validate again mid-run and keep validating until the task actually closes.
The uncomfortable summary: your agent’s model choice matters less than where it writes things down.
Every benchmark here failed on sustaining progress and verifying completion. And the one measurable thing that improved survival odds was the cheapest item on the list.
If you’re running agents past their demo length, put the scratchpad and the budget cap in first, then read LongCLI-Bench’s failure analysis (source) before you trust anything unattended.
Sources
– LongCLI-Bench: https://arxiv.org/html/2602.14337v2
– Long-Horizon-Terminal-Bench: https://arxiv.org/html/2607.08964
– YC-Bench: https://arxiv.org/html/2604.01212v1 and https://github.com/collinear-ai/yc-bench
– AgentLAB: https://arxiv.org/html/2602.16901
– HORIZON: https://huggingface.co/papers/2604.11978.md
