Technical Analysis & Systems Synthesis by DistributedApps.ai | Synthesizing frontier research from DeepSeek, Moonshot AI, Zhipu AI, Alibaba, NVIDIA, and Google DeepMind into production engineering blueprints.
This is Chapter 6 of the 10-part series The Physics & Engineering of Frontier LLM Inference. I announced the full curriculum, including every later chapter, in Announcing the 10-Part Substack Series.
Read the earlier parts first: Chapter 1: The Physics of LLM Inference, Chapter 2: The KV Cache Frontier, Chapter 3: Next-Gen Speculative Decoding, Chapter 4: Extreme Quantization, Chapter 5: Hardware-Aware Attention Kernels.
In monolithic serving architectures, long-context prefill operations monopolize GPU compute and memory bandwidth, causing severe queuing delays and latency spikes for concurrent decode streams. The diagram below contrasts monolithic colocation with physical prefill-decode disaggregation, showing how dedicated prefill clusters and independent decode pools eliminate Head-of-Line blocking:
In the early architecture of modern Large Language Model (LLM) serving systems—exemplified by first-generation engines such as Orca (OSDI '22), initial iterations of vLLM (SOSP '23), and early TensorRT-LLM runtimes—the entire inference lifecycle of a request was executed inside a single, colocated GPU instance or a tightly coupled Tensor Parallel (TP) GPU group. Under this monolithic colocation paradigm, every GPU device is forced to simultaneously execute two fundamentally antagonistic computational phases:
**The Prefill Phase (Prompt Evaluation):**Highly compute-bound matrix multiplications (GEMMs) operating across long input sequence lengths (N_in in [1,000, 128,000] tokens). The prefill phase exhibits high arithmetic intensity (typically 100 to 250 FLOPs per byte moved from High-Bandwidth Memory), heavily saturating Tensor Core processing pipelines and driving Model FLOPs Utilization (MFU) toward hardware ceilings (65%–80%).**The Decode Phase (Autoregressive Token Generation):**Memory-bandwidth-bound matrix-vector multiplications (GEMVs) operating token-by-token (N_step = 1 token per active sequence). The decode phase exhibits extremely low arithmetic intensity (typically 0.5 to 4 FLOPs per byte moved), where the GPU execution pipelines sit largely idle, waiting for tens to hundreds of gigabytes of model weights and distributed Key-Value (KV) cache tensors to stream across the memory bus for every single generated token.
When these two phases are scheduled together onto the same hardware execution context, catastrophic multi-tenant interference occurs. When a single user submits a large 32,000-token document or agentic system prompt, the prefill kernel seizes the Streaming Multiprocessors (SMs) for hundreds of continuous milliseconds. Consequently, dozens of concurrently active decode streams on that GPU are subjected to severe Head-of-Line (HoL) blocking. The Inter-Token Latency (ITL, also referred to as Time Per Output Token or TPOT) explodes from an optimal 15–20 milliseconds to 400–600 milliseconds, instantly shattering strict 99th-percentile (P99) enterprise Service Level Objectives (SLOs).
Engineers attempted to mitigate this interference by introducing Chunked Prefill (e.g., Sarathi-Serve and vLLM v0.6), which artificially slices long prompts into smaller token chunks (e.g., 512 tokens) and interleaves them with decode passes in continuous batching iterations. However, chunked prefill is a compromise that introduces severe systemic inefficiencies:
**Arithmetic Intensity Collapse:**Slicing a 32k prefill into 64 chunks of 512 tokens drastically degrades the operational intensity of the prefill GEMMs, reducing Tensor Core MFU by 35% to 50% and prolonging the overall Time to First Token (TTFT).**Scheduling Bubbles and Memory Jitter:**Interleaving chunked prefill GEMMs with decode GEMVs causes fine-grained cache line evictions, memory bus contention, and frequent warp scheduling pipeline stalls, preventing decode streams from ever achieving deterministic, jitter-free generation latencies.**Asymmetric Parallelism Constraints:**Monolithic serving forces both prefill and decode to share identical parallelism dimensions (e.g., Tensor Parallelism TP=8). Yet prefill latency scales efficiently with high TP (distributing large GEMMs across SMs), whereas single-token decode latency is dominated by cross-GPU all-reduce collective communication overheads, which severely degrade efficiency at high TP counts.
The inescapable conclusion of 2025–2026 systems research is that monolithic colocation is an architectural dead end. To achieve true hardware saturation, sub-20ms P99 decode latency, and sub-200ms TTFT under massive concurrent traffic, modern AI infrastructure must embrace Physical Prefill-Decode Disaggregation (PD Disaggregation).
Below is the complete architectural roadmap covered in this chapter:
**The Physics of Monolithic Interference & Resource Contention:**Mathematical queueing models proving why co-locating prefill and decode creates Head-of-Line blocking.**Moonshot AI Mooncake Architecture:**Hierarchical KVCache storage engine (VRAM, Host DRAM, Local SSD, Distributed Pool) and transfer coordination in the 2.8T Kimi K3 stack.**High-Speed Zero-Copy RDMA Transport:**GPUDirect RDMA over InfiniBand and RoCEv2 fabrics for microsecond-scale KV tensor migration.**vLLM V1 C++ Core Architecture:**Async request scheduling, lock-free queues, and zero-overhead worker coordination.**Cluster Sizing, Goodput Benchmarks & TCO Analysis:**Sizing prefill-to-decode cluster ratios and measuring goodput under strict P99 latency SLOs.
To access the complete technical treatise, production code repositories, Triton/CUDA kernels, and infrastructure sizing templates for this chapter, upgrade to a paid subscription today.
Special Offer: Get 50% OFF the annual subscription to the 2026 Foundation Model Inference Series using the link below:
