The Evolution of Terminal-Bench - How an Agent Benchmark Became Software (1.0 to 4.1)

Figures were extracted and cross-checked from the full texts of the tbench.ai blog and the harbor-framework/terminal-bench GitHub release notes; reviewed by the human operator. Leaderboard numbers reflect each announcement date.

The most-cited name in agent evaluation right now is Terminal-Bench: give a model and an agent a Linux terminal and measure whether it can do real work. Launched on 2025-05-19 by the Laude Institute + Stanford team with 80 tasks, it went through five overhauls in 18 months (1.0 → 2.0 → 2.1 → 3.0 → 4.0 → 4.1) - and became the first large-scale demonstration that benchmarks can be maintained like software instead of being built once and abandoned.

Every figure in the charts below comes from the official tbench.ai announcements and GitHub release notes.

Version explorer

Click a version to see what it changed.

1.0 (2025-05): discovering the terminal axis

The founding observation: Cursor, Manus, Devin, Goose, OpenHands and Codex all execute actions through the terminal. It is text-based (matching LLM training), fine-grained, easy to sandbox, and still the standard tool of real engineers. Each task ships a dedicated Docker environment, a human-verified solution and a test suite; the team also built Terminus, a neutral reference agent whose only tool is a single tmux pane. Growth was fast: Anthropic's Claude 4 model card adopted the benchmark in May, terminal-native product Warp hit 52% SOTA in June, and the dataset registry landed in July.

Integrity saga (2025-09 → 2026-04)

Success attracts gaming. The 2026-04 integrity update disclosed: OpenBlock OB-1 (encrypted per-task solutions embedded in the agent binary - removed and apologized publicly), QuantFlow Pilot (uploaded each task's tests/ folder - removed, allowed to resubmit), and ForgeCode (its agent curled solutions from the internet into AGENTS.md - those trials rescored to 0). New policy: an agent judge runs over every passing trial, reward hacking scores 0, cheating means removal, and ATIF trajectories are mandatory for submissions. Much of the detection came from community auditors (DebugML's Adam Stein & Davis Brown, the Ante team).

2.0 + Harbor (2025-11): the hard reset

Terminal-Bench 2.0 rebuilt the dataset to 89 tasks with heavy manual + LM-assisted verification (the old download-youtube task broke whenever YouTube changed its anti-bot defenses). Alongside it, Harbor reinvented the harness: thousands of cloud containers in parallel, rollout interfaces for RL/SFT, and a simple "any agent that can be installed in a container" contract. The repo moved to the harbor-framework organization, split per version - the operating model that survives today.

2.1 (2026-05): fixing 28 of 89 tasks

A living dataset rots without maintenance. Using community reports and leaderboard trajectories, 28 tasks were repaired across three failure classes: 9 external-dependency drift, 8 resource mismatches (even the oracle solution could not finish), 11 specification mismatches (query-optimize asked for PostgreSQL but tested Spark SQL). Most agent-model pairs improved - Opus 4.6 + Claude Code +12.1%p - and after the fixes zero tasks remained unsolved. Z.AI's "TB 2.0 Verified" work contributed 11 of the 28 fixes.

Challenges (2026-06): the third axis

Terminal-Bench Challenges are long-horizon, token-intensive single tasks:

The first three: speed up the Rust compiler (scored by instruction count on a 52-crate slice of rustc-perf), serve Kimi 2.5 from a single C/CUDA file under 25 KB, and build a WASM software WebGL renderer. Early results were sobering: Opus 4.8 for 12 hours produced only regressions on Rust, and Devin + Fable 5 reached 96.4% WebGL 1.0 coverage but stalled at 20.5% on WebGL 2.0 conformance. Two reported failure modes: unwillingness to delete large chunks and explore large-scale rewrites, and overuse of the full 20-minute test suite instead of writing targeted tests.

3.0 (2026-07): re-measuring the frontier + "benchmarks are software"

By 2.1 the frontier was saturated near 80%, and Fable 5 vs Opus 4.8 differed by only 4.9%p. Terminal-Bench 3.0 shipped 74 tasks across 7 domains together with the Continuous Benchmarks manifesto: benchmarks are software and should be maintained like software. Structural changes: the agent container is now separated from the verifier container (artifacts are logged and regradable without re-running agents), semantic versioning powers cheap leaderboard migration, and CI gates (oracle full reward, no-op zero reward, cheating agent fails) guard every task. Best score: 34.4% - the reset worked, and discrimination returned (Fable 5 vs Opus 4.8 gap: 4.9 → 12.7%p). GPT-5.6 Sol matched Fable 5 while costing ~40% less and using ~50% fewer tokens; all three GPT-5.6 variants sit on the cost-pass-rate Pareto front. Repo trivia: PR links preserve a period when the project briefly adopted the name Frontier-Bench before reverting to Terminal-Bench.

4.0 / 4.1 (2026-08 → 11): maintenance in practice

Terminal-Bench 4.0 calibrated task resources (time, CPU, memory) using Anthropic's infrastructure-noise methodology, set a flat 8-hour agent timeout, removed 8 tasks (2 saturated - where every latest-generation model family solved 5/5 - 2 refusal-prone, 2 with public solutions, 2 with quality/platform issues) and fixed 19. Since version numbers are now semantic, an update containing agent-environment changes is 4.0, not 3.1. v4.1.0 (2026-11-04) followed as a pure minor/patch release: verifier isolation via RewardKit Workers, pinned images + lockfiles, no tasks added or removed, all 66 modified tasks migrated without re-running - plus a new GPU-free terminal-bench-cpu subset of 63 tasks.

harbor run -d terminal-bench/[email protected]        # full dataset
harbor run -d terminal-bench/[email protected]    # CPU-only 63 tasks

The pattern of these 18 months: the harness became a product (Harbor + Harbor Hub), operations became continuous (report → fix → regrade/rerun), difficulty resets are scheduled rather than fatal, integrity is enforced systematically, and the family is diversifying (Terminal-Bench-Science 0.1, Harbor-Index). Terminal-Bench 5.0 (new tasks) is already an open milestone.