Long‑Horizon‑Terminal‑Bench 2.0 (LHTB 2.0) expands the original 46‑task benchmark to 77 tasks across 13 domains, adding systematic reconstruction pipelines and a three‑hour evaluation horizon.
Core Updates in LHTB 2.0
- 77 tasks, 13 domains, 8 task types, with 31 tasks extended via a reconstruction‑task pipeline.
- Evaluation horizon extended from 90 minutes to 3 hours, with verifier checkpoints every 30 minutes.
- Dense progress ladders reward partial progress, mitigating memorization of public outputs.
- Across 1,145 model‑task runs, GPT‑6‑astra tops the leaderboard with a dense mean reward of 0.69 and 37 tasks achieving R ≥ 0.95.
- Binary verifier scores average 0.221, while dense ladder scores average 0.653 on 30 reconstruction tasks, highlighting the value of intermediate progress measurement.
From Short‑Horizon Checks to Process‑Aware Evaluation
The original LHTB focused on binary success after a single reward, limiting insights into sustained agent performance. LHTB 2.0 corrects infrastructure defects, preserves the 46 legacy tasks, and introduces a systematic pipeline that starts from a working artifact, generates multiple scenarios, and uses dense ladders to capture incremental progress. This shift aligns benchmarks with real‑world workflows where agents must maintain momentum over hours, not just hit a final checkpoint.
Implications for Agent Reliability and Research
LHTB 2.0 offers a hardened, process‑aware testbed that will shape future model training, prompting developers to design agents that can navigate extended workflows. Its dense reward structure and extended horizon are likely to become standard in evaluating long‑horizon capabilities, influencing both academic research and industry deployments that rely on sustained autonomous execution.