Serving Optimization: Inference service performance optimization

LLM Serving Engine requires simultaneous optimization of request scheduling, KV Cache memory, GPU execution efficiency, and multi-GPU distributed inference.

1. Core technology classification

LLM Serving Engine
│
├── Scheduler
│   ├── Continuous Batching
│   └── Chunked Prefill
│
├── Memory / KV Cache
│   ├── PagedAttention
│   ├── KV Cache Management
│   └── Prefix Caching
│
├── Execution Optimization
│   ├── FlashAttention
│   ├── CUDA Graph
│   └── Speculative Decoding
│
└── Distributed Runtime
    ├── Tensor Parallel（TP）
    ├── Pipeline Parallel（PP）
    ├── Data Parallel（DP）
    ├── Expert Parallel（EP）
    └── Prefill-Decode Disaggregation

2. Continuous Batching

Continuous Batching allows different requests to join or exit GPU Batch at different times.

Traditional Static Batching:

Batch:
A B C D

A → 20 tokens
B → 100 tokens
C → 300 tokens
D → 500 tokens

If fixed batch processing cannot dynamically remove or supplement requests, it is easy to produce invalid calculations and waste resources.

Continuous Batching：

Step 1: A B C D
Step 2: E B C D   ← A Complete，E Join
Step 3: E F C D   ← B Complete，F Join

Main benefits:

• Raise GPU Utilization.
• Raise Throughput.
• Reduce idle resources in batch processing.

Cost:

• More complex request scheduling.
• More concurrent requests may occupy more KV Cache.
• Queuing delays may still be increased under high load.

Continuous Batching usually needs to be paired with the dynamic KV Cache Management.

3. Chunked Prefill

Prefill with a long Prompt may occupy GPU for a longer period of time, affecting requests already at Decode.

For example:

Request A: Prefill 100K tokens
Request B/C/D: Decode

GPU:
AAAAAAAAAAAAAAAAAAAA
B/C/D wait

Chunked Prefill splits the long Prompt into multiple Chunk:

A1: 2048 tokens
A2: 2048 tokens
A3: 2048 tokens
...

The scheduler can combine Prefill and Decode in different scheduling steps:

Step 1: A1 + B/C/D Decode
Step 2: A2 + B/C/D Decode
Step 3: A3 + B/C/D Decode

The core purpose is to improve the scheduling relationship between Prefill and Decode.

Possible benefits:

• Reduce the risk of Decode being blocked by long Prefill.
• Improved TPOT and Tail Latency.
• Improve scheduling efficiency under mixed loads.

The trade-off is that Prefill with long Prompt may require more scheduling rounds, and single request TTFT may not necessarily be reduced.

4. PagedAttention

PagedAttention divides KV Cache into fixed-size Block, managed through logical and physical address mapping.

Traditional reservation method:

Request A: [Used=====................]
Request B: [Used===========..........]
Request C: [Used==...................]

If you reserve video memory according to the maximum Context Length, it will cause a lot of waste.

PagedAttention：

Request A:
Logical Block 0 → Physical Block 7
Logical Block 1 → Physical Block 2
Logical Block 2 → Physical Block 9

Request B:
Logical Block 0 → Physical Block 4
Logical Block 1 → Physical Block 1

KV Block is not required to be contiguous in physical video memory.

Main values:

• Reduce KV Cache memory fragmentation.
• Avoid reserving the full Context Length for each request.
• Allocate and recycle KV Block on demand.
• Support higher concurrency.
• Provides the basis for Prefix Sharing at Block level.

PagedAttention borrows ideas from the operating system's paging memory management, but it handles GPU KV Cache.

5. Prefix Cache / Prompt Caching

If multiple requests have the same Token Prefix, the already calculated KV Cache can be reused.

For example:

Request A:
[System Prompt] + [Question A]

Request B:
[System Prompt] + [Question B]

Common parts:

System Prompt
      ↓
Prefill
      ↓
Cached KV Blocks
      ├── Request A
      └── Request B

Main benefits:

• Reduce duplication Prefill.
• Lower TTFT.
• Reduce GPU computational overhead.

Important limitations:

• The corresponding prefix matching rules must be met.
• The same text in the middle of Prompt may not be directly reusable as Prefix Cache.
• Chat Template, Token IDs and model configurations need to be compatible.
• Cache hit rate depends on the request pattern.

For example, vLLM can implement Automatic Prefix Caching by calculating the cache ID based on the Token content of KV Block and its previous complete prefix.

6. KV Cache Management

KV Cache Management is the core memory management module of Serving Engine.

Need to solve:

KV Cache Manager
├── Allocation
├── Growth
├── Sharing
├── Reuse
├── Eviction
├── Reclamation
└── Memory Pressure Handling

Major issues include:

• Which requests get KV Cache?
• How many Block are allocated per request?
• When is it released after the request is completed?
• Which Prefix can be kept?
• Which caches should be evicted when there is insufficient video memory?
• Do some requests need to be paused or recalculated?

KV Cache is an important resource for limiting long context and high concurrency inference.

7. FlashAttention

FlashAttention optimizes the calculation process of Attention and the memory access of GPU.

Normal Attention:

$$\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$

If you directly write the complete Attention Matrix to HBM, a large number of memory reads and writes will occur.

FlashAttention uses methods such as Tiling and Online Softmax:

Q Block
K Block
V Block
   ↓
Compute Attention Tile
   ↓
Online Softmax
   ↓
Accumulate Output

As much as possible, let the intermediate calculations be completed in faster on-chip storage such as GPU's SRAM / Registers, reducing HBM access.

FlashAttention and PagedAttention

The three can cooperate with each other rather than replacing each other.

8. Speculative Decoding

Ordinary autoregressive generation typically requires executing Target Model one by one Token.

Speculative Decoding first generates multiple candidates Token, and then allows Target Model to be verified in parallel.

Draft Model（Small）
        ↓
Generate 5 Candidate Tokens
        ↓
Target Model（Large）
        ↓
Verify Candidates
        ↓
Accept Valid Prefix
        ↓
Continue Generation

For example:

Draft:
The capital of France is Paris

Target Verification:
capital ✓
of      ✓
France  ✓
is      ✓
Paris   ×

Continuous prefixes that meet the verification conditions can be accepted before continuing to generate.

Why is acceleration possible?

Target Model can process multiple candidate locations in one validation Forward, rather than performing a complete autoregressive step for each candidate Token individually.

Common solutions:

• Draft Model。
• N-gram Speculation。
• EAGLE and other specialized speculation generation mechanisms.

Prices and Limitations

• Draft generation itself requires computation.
• Gains may be small when candidate acceptance rates are low.
• Batch may not necessarily accelerate when it is larger.
• Additional validation and KV Cache management required.

When using the correct acceptance and correction algorithms, Speculative Decoding can maintain the original sampling distribution of the target model; however, different implementations and approximation schemes do not necessarily have exactly the same guarantees.

The core indicators are Acceptance Rate and the actual obtained Decode Speedup.

9. Tensor Parallel（TP）

Tensor Parallel splits the calculation of a certain layer of the model into multiple GPU.

For example:

Transformer Layer
        ↓
Matrix Multiplication
        ├── GPU 0
        ├── GPU 1
        ├── GPU 2
        └── GPU 3
        ↓
Communication / Synchronization

Main uses:

• A single sheet GPU cannot accommodate the model.
• Utilizing multiple GPU to improve single-model computing power.

The price is frequent communication between GPU.

Therefore TP has higher performance requirements on the GPU interconnect bandwidth and communication library, such as NVLink / NVSwitch and NCCL.

TP does not guarantee that doubling the number of GPU will result in twice the performance.

10. Pipeline Parallel（PP）

Pipeline Parallel is calculated by model layer.

For example, 80-layer model:

GPU 0: Layers 0–19
GPU 1: Layers 20–39
GPU 2: Layers 40–59
GPU 3: Layers 60–79

Execution process:

Input
  ↓
GPU 0
  ↓
GPU 1
  ↓
GPU 2
  ↓
GPU 3
  ↓
Output

The main value is spreading the model across multiple devices.

Pipeline Bubble

If there is only one request, different GPU waits may occur:

Time →
GPU 0: A ----
GPU 1:      A ----
GPU 2:           A ----

Pipeline utilization can be improved by interleaving multiple Micro-batch executions:

Time →
GPU 0: A → B → C
GPU 1:     A → B → C
GPU 2:         A → B → C

However, the pipeline method is more suitable for specific multiple requests or multiple Micro-batch loads, which does not mean that the single request delay will necessarily be reduced.

11. Data Parallel（DP）

Data Parallel handles different requests through multiple copies of the model.

Request Queue
      ↓
Load Balancer
      ├── Model Replica 1
      ├── Model Replica 2
      └── Model Replica 3

If the model can fit into a single sheet GPU:

GPU 0 → Full Model
GPU 1 → Full Model
GPU 2 → Full Model

If the model must use multiple GPU, you can also copy multiple TP/PP model instances.

For example:

TP = 4
DP = 2

Replica A: GPU 0–3
Replica B: GPU 4–7

Total GPUs = 8

The main purpose is to improve the total system throughput and request processing capacity.

DP generally cannot reduce the parameter size of a single model copy.

12. Expert Parallel（EP）

Expert Parallel is mainly used for Mixture-of-Experts (MoE) models.

In the MoE model, different Token can participate in the calculation by selecting part of Expert through Router.

Token Hidden State
        ↓
      Router
        ↓
Select Top-k Experts
        ↓
Expert Computation
        ↓
Combine Outputs

Expert Parallel assigns different Expert to different GPU.

GPU 0 → Expert 0, 1
GPU 1 → Expert 2, 3
GPU 2 → Expert 4, 5
GPU 3 → Expert 6, 7

Main benefits:

• Spread large amounts of Expert weights.
• Supports larger MoE model.
• Implement multiple GPU Expert calculations.

Main challenges:

• Token Dispatch。
• All-to-All Communication。
• Expert Load Balancing。

If a large number of Token select the same Expert, local load imbalance may occur.

EP is an important technique to know when deploying large MoE models such as DeepSeek.

13. Prefill-Decode Disaggregation

Prefill and Decode generally have different resource requirements:

Prefill → Compute-intensive
Decode  → Memory-bandwidth-intensive

PD Disaggregation deploys two stages to different Worker.

HTTP Request
      ↓
Prefill Worker
      ↓
Transfer KV Cache
      ↓
Decode Worker
      ↓
Generated Tokens

Main advantages:

• Configure Prefill and Decode resources independently.
• Reduce the interference of long Prefill on Decode.
• There is an opportunity to optimize TTFT, TPOT and high load Tail Latency.

Main cost:

• KV Cache is transmitted across Worker.
• More complex routing and resource management.
• Higher deployment and operation and maintenance costs.

SGLang provides related architecture and transport backend support, actual performance depends on interconnect speed, workload and scheduling configuration.

For single GPU inference, PD separation is usually not necessary.

14. CUDA Graph

Under normal circumstances, CPU needs to submit GPU Kernel one by one:

CPU:
Launch Kernel A
Launch Kernel B
Launch Kernel C
Launch Kernel D

Decode repeats a large number of similar computational steps, and Kernel Launch Overhead may become apparent.

CUDA Graph can capture a set of GPU operations:

Capture:
A → B → C → D
       ↓
   CUDA Graph
       ↓
Repeated Replay

This can reduce repeated CPU Kernel Launch overhead in the future.

Limit

CUDA Graph usually has certain requirements on execution structure, memory address and shape stability.

Batch Size, sequence length, etc. in LLM Serving may change dynamically, so the inference engine often needs to perform Graph Capture or use other adaptation methods for different shapes.

CUDA Graph optimizes the scheduling overhead from CPU to GPU. It will not directly reduce the model weight, nor will it eliminate the memory requirements of KV Cache.

15. Correspondence between technology and performance indicators

Actual results depend on model, hardware, request length, and concurrency, and not all optimizations will improve TTFT, TPOT, and Throughput simultaneously.

Summary

The optimization of LLM Serving can be understood as four questions:

Core relationships:

Requests
    ↓
Scheduler
    ├── Continuous Batching
    └── Chunked Prefill
    ↓
KV Cache Manager
    ├── PagedAttention
    └── Prefix Caching
    ↓
Execution Engine
    ├── FlashAttention
    ├── CUDA Graph
    └── Speculative Decoding
    ↓
Distributed Runtime（on demand）
    ├── TP / PP / DP / EP
    └── PD Disaggregation
    ↓
GPU

For preliminary learning of vLLM/SGLang, priority should be given to understanding:

1. Continuous Batching and Chunked Prefill.
2. PagedAttention and Prefix Caching.
3. FlashAttention and CUDA Graph.
4. Speculative Decoding。
5. The difference between TP, PP and DP.

Expert Parallel and PD Disaggregation are important techniques for large-scale distributed inference, but can be further studied when multiple GPU or MoE deployments are actually required.

---

Reference documentation

• vLLM Documentation
• vLLM Automatic Prefix Caching
• SGLang Documentation
• SGLang Expert Parallelism
• SGLang PD Disaggregation
• FlashAttention
• NVIDIA CUDA Graphs