GPU-Resident Autoregressive Inference

Overview

This research focuses on eliminating the "dispatch tax"—the structural latency incurred when a host CPU orchestrates the token-level control plane of an Autoregressive model [C000, C005]. In standard inference runtimes, the CPU remains on the critical path for every token generated, managing kernel launches, scheduling, and KV-cache updates [C000, C005]. For large-scale models, this creates a bottleneck where the GPU's compute capacity is underutilized while it waits for the host to signal the next operation in the sequence [C002, C006].

A GPU-resident autoregressive loop migrates the entire serving stack—including batching and scheduling—into a persistent GPU kernel [C000, C005]. By removing the host CPU from the steady-state inference path, the system avoids performance degradation caused by CPU interference, which can otherwise reduce system stability by up to two orders of magnitude [C000, C005]. This architectural shift is critical for reducing synchronization overhead, lowering the Time Per Output Token (TPOT) and energy consumption [C000, C005].

Implementing this requires a first-principles approach to memory management, as the architecture must handle the autoregressive loop without returning control to the host until the stopping criteria (e.g., an <end> token) is met ^[C006]. This approach mirrors the efficiency gains seen in non-LLM high-compute tasks, where migrating the entire pipeline to the device achieves near-zero host-device communication and significant algorithmic speedups ^[C001].

Landscape

Current efforts to eliminate the "dispatch tax" in autoregressive inference focus on three primary architectural shifts: removing the host CPU from the critical path, parallelizing the sequential nature of token generation, and implementing hardware-native memory hierarchies.

CPU-Bypass and Device-Resident Orchestration

The dominant approach to reducing per-token synchronization overhead is the migration of the control plane—scheduling, batching, and KV-cache management—directly onto the accelerator. Blink implements this by delegating request handling to a SmartNIC, which uses rdma to deliver inputs directly to GPU memory [C000, C005]. By replacing host-driven scheduling with a persistent GPU kernel, Blink improves decode throughput compared to vLLM and TensorRT-LLM ^[C005]. This mirrors strategies in high-performance computing, such as the lock-free, fully GPU-resident architecture used for Goldbach's conjecture verification, which utilizes L1 shared-memory tiling to achieve near-zero host-device communication ^[C001].

Algorithmic Parallelization of the AR Loop

To break the $O(N)$ linear dependency of next-token prediction, researchers are introducing non-sequential prediction heads. FlashAR utilizes a two-way next-token prediction framework, adding a lightweight "vertical head" for column-wise prediction to complement the standard "horizontal head" ^[C003]. This adaptation allows for highly parallel generation, achieving up to a 22.9$\times$ speedup for 512x512 image generation ^[C003]. Similarly, Token-Shuffle reduces the total token count in Transformers by merging spatially local tokens along the channel dimension, reducing the total number of autoregressive steps required for high-resolution synthesis ^[C004].

hardware-Software Co-Design

Alternative hardware architectures are addressing the "memory wall" by replacing monolithic HBM with tiered memory systems. The SambaNova SN40L Reconfigurable Dataflow Unit (RDU) employs a three-tier hierarchy (on-chip SRAM $\rightarrow$ on-package HBM $\rightarrow$ off-package DDR DRAM) to facilitate the execution of trillion-parameter models ^[C008]. This dataflow approach reduces machine footprint by up to 19$\times$ and accelerates model switching by 15$\times$ to 31$\times$ compared to conventional GPU baselines ^[C008]. Additionally, the move toward 1-bit binarization, as seen in FBI-LLM, seeks to shift the computational burden from floating-point GEMM to native bitwise arithmetic, though this requires specialized hardware integration to realize wall-clock gains ^[C009].

Key Findings and Trade-offs

The primary bottleneck in autoregressive LLM inference is the "dispatch tax," where the host CPU remains on the critical path for orchestration, scheduling, and token-level control ^[C000]. This dependency makes TPOT (Time Per Output Token) and TTFT (Time to first Token) highly sensitive to CPU interference, often leading to performance degradation of up to two orders of magnitude when the host is under load ^[C005].

Evidence demonstrates that migrating the control plane to a GPU-resident architecture—using a persistent GPU kernel to handle batching, scheduling, and KV-cache management—eliminates this synchronization overhead ^[C005]. A similar transition in non-LLM computational pipelines, utilizing L1 shared-memory tiling to achieve near-zero host-device communication, resulted in a 45.6$\times$ algorithmic speedup over host-coupled predecessors ^[C001].

The quantitative impact of removing the CPU from the steady-state inference path (as demonstrated by Blink) is significant:
* Latency: P99 TTFT is reduced by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$ ^[C005].
* Throughput: Decode throughput increases by up to 2.1$\times$ ^[C005].
* Efficiency: energy consumption per token is reduced by up to 48.6% ^[C005].

Practitioners face a direct tradeoff between the flexibility of host-driven orchestration and the efficiency of a fully device-resident loop. While GPU residency solves the dispatch tax, it shifts the critical bottleneck to the "memory wall" ^[C008]. Moving the autoregressive loop entirely to the GPU requires that all orchestration logic and model state reside in VRAM, exacerbating memory pressure. To maintain viability for large-scale models, this necessitates a tradeoff between the high operational intensity of fused, on-chip operations and the capacity of off-package memory (DDR), or the adoption of fully binarized weights (1-bit) via FBI-LLM to reduce the memory footprint and computational intensity [C008, C009].

Opportunities

Implementation Targets

To eliminate the "dispatch tax," development should focus on migrating the autoregressive control plane from the host CPU to a persistent GPU kernel. This involves building:

• A Persistent Orchestration Kernel: Based on the Blink architecture, a kernel that remains resident on the GPU to handle batching, scheduling, and KV-cache management ^[C005].
• Lock-Free Work-Stealing Pools: Implementing an asynchronous, lock-free mechanism for token generation and segment claiming ^[C001]. This approach can achieve near-zero host-device communication and has demonstrated up to a 45.6$\times$ algorithmic speedup over host-coupled architectures in non-LLM computational pipelines ^[C001].
• Parallel Prediction Heads: Integrating a "vertical head" for column-wise prediction alongside the standard horizontal head to enable parallel generation ^[C003]. This post-training adaptation can achieve up to a 22.9$\times$ speedup by breaking the strictly sequential next-token prediction bottleneck ^[C003].
• Binarized Weight Integration: Implementing 1-bit weights via FBI-LLM to reduce memory bandwidth pressure ^[C009].

Research Questions

The following technical gaps must be addressed to validate a GPU-resident loop for large-scale models:

1. hardware-Specific Dispatch: Can the performance gains of a CPU-free stack—which Blink achieved using a SmartNIC for rdma ^[C005]—be replicated on hardware with different memory controller architectures?
2. Binarization Efficiency: Does the theoretical perplexity parity of fully binarized LLMs ^[C009] translate to wall-clock speedups, or does the lack of native bitwise arithmetic kernels create a new "DX Gap"?
3. Memory Tiering: How does the transition to a GPU-resident loop affect the efficiency of a three-tier memory system (SRAM $\rightarrow$ HBM/LPDDR $\rightarrow$ DDR) when managing trillion-parameter scale experts ^[C008]?

Architectural Trade-offs: Host-Driven vs. GPU-Resident

References

• [C000] Breaking LLM Bottlenecks with DecodeIR: GPU-Resident Decoding — https://www.linkedin.com/posts/dharun-narayan-venkatesh-a7b478255_systems-gpu-cuda-activity-7421901724396994560-ZGDm
• [C001] Blink:CPU-Free LLM Inference by Delegating the Serving Stack to... — https://arxiv.org/html/2604.07609v1
• [C002] A Lock-Free, Fully GPU-Resident Architecture for the Verification of Goldbach's Conjecture — https://arxiv.org/abs/2603.07850
• [C003] LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference — https://arxiv.org/abs/2602.00426
• [C004] FlashAR: Efficient Post-Training Acceleration for Autoregressive Image Generation — https://arxiv.org/abs/2605.09430
• [C005] Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models — https://arxiv.org/abs/2504.17789
• [C006] Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU ... — https://arxiv.org/pdf/2604.07609
• [C008] Mastering LLM Techniques: Inference Optimization - NVIDIA Developer — https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
• [C009] LLM Inference Handbook — https://bentoml.com/llm/