HomeResearch LibraryLong-Context Language Models: A Survey
surveyChapter 2arXiv · 2024

Long-Context Language Models: A Survey

Tianlong Chen (MIT), Xuxi Chen (UT Austin)

Abstract

We survey methods for extending the context length of language models, covering positional encoding extensions, efficient attention mechanisms, and memory-augmented architectures.

Eigenvector Insight — Zone III / PASF-PADE AnalysisNot part of the original paper
Eigenvector Research — Marco van Hurne
How this paper contributes to solving the Zone III problem (PASF-PADE)

Context length is the primary technical constraint on Zone III agents. This survey provides the comprehensive map of approaches for extending context — essential for enterprise architects designing long-horizon agent memory systems.

Why AI is not sufficient for Zone III without this

Zone III refers to high-complexity, high-risk, long-running agentic workflows — the class of enterprise AI deployments where a single failure can cascade across hundreds of steps. Standard AI models, trained to predict the next token, are not inherently designed for durable, governed, multi-step execution. This paper addresses one or more of the structural gaps that make Zone III deployments unsafe without explicit architectural intervention.

Key Contributions

  • →Long-context methods survey
  • →Positional encoding extensions
  • →Efficient attention mechanisms

Topics

long contextcontext lengthefficient attentionmemory