Memory-Access Patterns in LLM Attention vs. Host Virtual Memory Management

Overview

This research focuses on the differences between the KV Cache access patterns of LLM attention mechanisms—which scale linearly with sequence length [C000, C007]—and the generic page-replacement logic of host virtual memory managers (VMM). While traditional VMMs treat memory as agnostic pages, the attention mechanism requires specific tensor retrieval that, when mismatched with host paging, leads to "cache thrashing" and unpredictable latency ^[C001]. A "Tensor-Aware" page-replacement algorithm aims to align the VMM with these specific patterns, utilizing strategies such as mapping tensors to contiguous logical block address (LBA) regions to bypass the filesystem and kernel page cache entirely ^[C001].

This optimization is critical for long-horizon autonomous agents because as context windows expand toward 1M tokens, the KV Cache inevitably exceeds GPU capacity, forcing offloading to CPU or NVMe ^[C008]. Standard offloading methods are insufficient for agentic stability due to two primary failure modes:
1. VMM Thrashing: File-based offloading relies on the kernel page cache, which introduces high software overhead and latency spikes under memory pressure ^[C001].
2. Pruning Collapse: While semantic pruning (e.g., IceCache) maintains high accuracy for general tasks ^[C000], it causes catastrophic accuracy collapse in context-intensive tasks like Text2JSON, where structured knowledge extraction is required ^[C004].

The following table compares current memory management strategies and their structural trade-offs:

Landscape

Current efforts to align LLM memory-access patterns with host virtual memory managers split into four primary architectural paths: paging-based allocation, direct-storage offloading, semantic pruning, and near-memory processing.

The baseline for modern serving is PagedAttention (implemented in vLLM), which applies OS-style paging to KV caches to eliminate fragmentation and enable flexible sharing ^[C002]. However, this approach introduces a conflict: by allocating physical memory at runtime, it transforms the KV cache from a contiguous tensor into a non-contiguous virtual memory layout, creating non-trivial programming and performance overheads ^[C003].

To prevent the "thrashing" associated with traditional host memory management, DUAL-BLADE implements a dual-path residency framework [C001, C005]. It identifies that file-based offloading relies too heavily on the kernel page cache, which triggers unpredictable latency under memory pressure ^[C001]. By mapping KV tensors to contiguous logical block address (LBA) regions via an NVMe-direct path, it bypasses the filesystem entirely, reducing decode latency by up to 42.4% [C001, C005].

A different approach focuses on reducing the tensor volume itself. IceCache utilizes semantic token clustering to organize related tokens into contiguous memory regions ^[C000]. This allows the system to maintain 99% accuracy on LongBench while using only 25% of the KV cache token budget ^[C000]. However, this "semantic" approach is task-dependent; benchmarks like Text2JSON demonstrate that pruning often causes catastrophic accuracy collapse during structured extraction tasks, where the model must retrieve specific, non-semantic fragments of the context ^[C004].

At the hardware level, CXL (compute Express Link) is being used to move the "selection" logic closer to the data. Rather than recalling non-resident KV tokens to the GPU—which creates a PCIe bottleneck—Processing-Near-Memory (PNM) accelerators perform token page selection within the CXL memory itself ^[C008]. This PNM-KV architecture has demonstrated up to 21.9x throughput improvement for 1M-token contexts by eliminating the need for costly data recalls to the GPU ^[C008].

Key Findings

The conflict between LLM attention patterns and traditional virtual memory managers (VMM) manifests as a tension between memory fragmentation and access latency. While PagedAttention reduces memory waste to near-zero by treating KV caches as non-contiguous pages ^[C002], this transformation alters the virtual memory layout from contiguous to non-contiguous, introducing significant programming and performance overheads ^[C003]. Traditional kernel page caches further exacerbate this by causing cache thrashing and unpredictable latency under memory pressure ^[C001].

Evidence shows that the efficacy of KV cache optimization is highly task-dependent, creating a divide between general long-sequence generation and structured extraction:

• Semantic Pruning: IceCache utilizes semantic token clustering to maintain high accuracy on LongBench while significantly reducing the KV cache token budget ^[C000].
• Extraction Failure: These same offloading and compression techniques suffer significant performance degradation on context-intensive tasks, such as those in the Text2JSON benchmark, where the model must extract structured knowledge from raw text ^[C004]. This failure is attributed to unreliable landmarks and the use of low-rank projections for keys ^[C004].

To bypass VMM bottlenecks, research has shifted toward hardware-direct paths and Processing-Near-Memory (PNM). DUAL-BLADE mitigates I/O bottlenecks by mapping KV tensors to contiguous logical block address (LBA) regions, bypassing the filesystem to reduce decode latency ^[C001]. For extreme context lengths (1M+ tokens), CXL-enabled PNM architectures eliminate the need to recall tokens to GPU memory by offloading token page selection to a PNM accelerator, resulting in substantial throughput improvement and lower energy per token ^[C008].

Ultimately, no single strategy dominates across all deployment scenarios ^[C007]. The optimal configuration depends on the tension between hardware constraints (e.g., edge device memory vs. datacenter CXL availability) and the specific nature of the workload (e.g., chat vs. accuracy-critical reasoning) ^[C007].

Tensions and Tradeoffs

Practitioners implementing long-context memory management must navigate three primary structural tensions: memory layout overhead, the stability of pruned contexts, and the latency cost of off-device residency.

Memory Layout vs. Access Efficiency
The adoption of PagedAttention eliminates memory fragmentation and enables flexible KV cache sharing ^[C002]. However, this introduces a fundamental tradeoff: by allocating physical memory at runtime, PagedAttention transforms the KV cache virtual memory layout from contiguous to non-contiguous ^[C003]. This shift creates non-trivial programming overhead and performance penalties compared to contiguous allocations ^[C003].

Accuracy vs. Memory Footprint
Reducing the KV cache footprint through pruning or semantic clustering creates a task-dependent risk profile. While IceCache maintains high accuracy on the LongBench benchmark using a reduced KV cache token budget ^[C000], these optimizations are not universal. In context-intensive tasks—specifically structured knowledge extraction like Text2JSON—standard KV offloading and compression strategies cause significant performance degradation ^[C004].

Offloading Path: Kernel Cache vs. Direct Access
When scaling beyond GPU memory, the choice of offloading path determines system stability under pressure:
* File-based designs: Relying on the kernel page cache for NVMe offloading leads to cache thrashing and unpredictable latency [C001, C005].
* Direct-path designs: Bypassing the filesystem via contiguous logical block address (LBA) mapping, as seen in DUAL-BLADE, reduces prefill and decode latency [C001, C005].

These tensions indicate that no single optimization dominates; the optimal configuration depends on whether the workload prioritizes high-throughput datacenter serving, edge device constraints, or accuracy-critical reasoning ^[C007].

Opportunities

To align LLM attention patterns with host memory management, development should focus on bypassing the kernel page cache and aligning tensor layouts with physical storage.

Systems to Build

• Semantic-Direct Offloading Engine: Integrate semantic token clustering IceCache ^[C000] with an NVMe-direct path DUAL-BLADE ^[C001]. By mapping semantically related token clusters directly to contiguous logical block address (LBA) regions, the system can eliminate the kernel page cache overhead that currently causes thrashing and unpredictable latency during memory pressure [C001, C005].
• Layout-Aware Memory Orchestrator: Build a manager that resolves the performance overhead created by PagedAttention's non-contiguous virtual memory layout ^[C003]. This orchestrator should dynamically switch between non-contiguous paging for fragmentation control ^[C002] and contiguous mapping for high-throughput I/O during large-scale KV-cache recalls.
• Task-Specific Pruning Guardrails: Develop a validation layer using the Text2JSON benchmark ^[C004] to detect "silent failures" in KV cache pruning. This tool would identify when semantic pruning—which maintains accuracy in chat tasks ^[C000]—causes catastrophic collapse in structured extraction tasks ^[C004].

Critical Research Questions

• The Recomputation Threshold: At what specific context length and TFLOPS-to-bandwidth ratio does GPU recomputation become more efficient than recalling non-resident tokens via CXL-enabled Processing-Near-Memory (PNM) ^[C008]?
• Interconnect Bottlenecks: How does the transition from PCIe to CXL-PNM architectures specifically impact the throughput of "steady-token" selection mechanisms in 1M-token contexts ^[C008]?
• Adaptive Pipeline Optimization: Which combination of cache eviction, compression, and hybrid memory solutions yields the lowest latency for agentic loops that oscillate between low-context reasoning and high-context information retrieval ^[C007]?

References

• [C000] IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs — https://arxiv.org/abs/2604.10539
• [C001] DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference — https://openalex.org/W7159734508
• [C002] Efficient Memory Management for Large Language Model Serving with PagedAttention — https://doi.org/10.1145/3600006.3613165
• [C003] vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — https://doi.org/10.1145/3669940.3707256
• [C004] KV Cache Offloading for Context-Intensive Tasks — https://arxiv.org/abs/2604.08426
• [C005] DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference — https://arxiv.org/abs/2604.26557
• [C007] KV Cache Optimization Strategies for Scalable and Efficient LLM Inference — https://arxiv.org/abs/2603.20397
• [C008] Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits — https://arxiv.org/abs/2511.00321