Long-Context Language Models: A Survey

Abstract

We survey methods for extending the context length of language models, covering positional encoding extensions, efficient attention mechanisms, and memory-augmented architectures.

Key Contributions

• →Long-context methods survey
• →Positional encoding extensions
• →Efficient attention mechanisms

Topics