SpireBench, a new benchmark that pits large language models against the full deck‑builder game Slay the Spire, shows that while top models beat heuristic baselines, they still lag behind human experts, especially as runs lengthen.
Core Findings: Long‑Horizon Struggles and Skill Gaps
- 20 LLMs evaluated at the game’s highest difficulty.
- Frontier models outperform heuristic baselines and match average human players.
- They remain far behind experts, with the gap widening as the run lengthens.
- Strongest models win the first act as often as experts but falter later due to risk misjudgment.
Benchmark Roots: From Short‑Term Tests to Full‑Game Strategy
Agentic LLMs have begun tackling complex tasks, yet strategic reasoning over long horizons remains under‑studied. Most existing benchmarks are short and episodic, making it hard to dissect performance into knowledge, reasoning, and luck. SpireBench addresses this gap by giving models the complete original game of Slay the Spire, providing a self‑consistent renaming of every game term and a comparison with prior harnesses. The benchmark also reveals that models rely on prior game knowledge but fail to improve from the memories they write across runs, which are often wrong and overly cautious.
What Comes Next: Shaping AI Strategy and Evaluation
SpireBench offers an unsaturated long‑horizon benchmark with tools to pinpoint where and why agents fail. Its findings highlight the need for better risk assessment and memory utilization in agentic models. Future research can leverage this framework to develop training methods that improve long‑term strategic planning, ultimately narrowing the performance gap between AI and expert human players.