Long-Context Large Language Models: Efficient Attention, Positional Extrapolation, Streaming Inference, and Memory Optimization

Abstract

The context window determines how much information a large language model can process during an individual inference sequence. Expanding this window became a major research priority as language models were increasingly applied to long documents, books, source-code repositories, multi-turn conversations, and retrieval-intensive workflows. Standard Transformer self-attention incurs quadratic memory and computational costs with sequence length, while models trained on shorter sequences frequently fail to generalize to unseen positional ranges. This review examines methods for extending and efficiently processing long contexts through 2024. Transformer-XL introduced segment-level recurrence, while sparse-attention systems including Longformer and BigBird reduced attention complexity. Rotary positional embeddings and attention with linear biases offered alternative positional representations. FlashAttention and FlashAttention-2 improved exact attention efficiency through IO-aware GPU implementations. Positional interpolation, YaRN, and LongRoPE extended pretrained models beyond their original context lengths through modifications to positional encodings. StreamingLLM demonstrated that retaining initial attention-sink tokens alongside a rolling cache could support stable streaming inference. LongLLMLingua addressed another dimension of the problem by compressing prompts before inference. The review also examines key-value cache growth, positional degradation, retrieval within long contexts, attention sinks, memory bandwidth, long-context evaluation, and the difference between nominal context-window size and effective information utilization. Evidence available by 2024 showed that context length had become a systems, architecture, representation, and evaluation problem rather than a single model hyperparameter.