HomeResearch LibraryOdysseyBench: Evaluating LLM Agents on Long-Horizon Com…
Chapter 2 · 2025

OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows

Weixuan Wang, Dongge Han, Daniel Madrigal Diaz

Abstract

Existing benchmarks for LLM agents often focus on atomic tasks, failing to capture the complexities of real-world, long-horizon workflows. This paper introduces OdysseyBench, a comprehensive benchmark for evaluating LLM agents on long-horizon workflows across diverse office applications. It also proposes HomerAgents, a multi-agent framework for automated benchmark generation, demonstrating that OdysseyBench effectively challenges state-of-the-art LLM agents.

Eigenvector Insight — Zone III / PASF-PADE AnalysisNot part of the original paper
Eigenvector Research — Marco van Hurne
How this paper contributes to solving the Zone III problem (PASF-PADE)

This paper directly addresses one of the core structural challenges in Zone III deployments. The research on LLM agents, long-horizon tasks, benchmarks provides evidence-based foundations that enterprise architects cannot ignore when designing long-horizon autonomous workflows. The findings challenge the assumption that a base language model — however capable — can handle the complexity of durable, governed, multi-step execution without explicit architectural intervention. For Zone III practitioners, this paper belongs in the required reading list.

Why AI is not sufficient for Zone III without this

Zone III refers to high-complexity, high-risk, long-running agentic workflows — the class of enterprise AI deployments where a single failure can cascade across hundreds of steps. Standard AI models, trained to predict the next token, are not inherently designed for durable, governed, multi-step execution. This paper addresses one or more of the structural gaps that make Zone III deployments unsafe without explicit architectural intervention.

Topics

LLM agentslong-horizon tasksbenchmarksoffice applicationsmulti-agent systems