Inference Engines & Model Serving

The LLM inference framework is responsible for loading the model, performing calculations and generating Token.

On this basis, Model Serving also needs to handle API requests, scheduling, batching, concurrency, caching, and performance monitoring.

1. Classification of inference tools

These tools have overlapping capabilities and are not strictly mutually exclusive.

For example, llama.cpp also provides HTTP Server and Continuous Batching; vLLM and SGLang can also be used for single-user local testing.

2. Inference Runtime and Serving Engine

Inference Runtime

Mainly responsible for:

Model Weights
      ↓
Load Model
      ↓
Execute GPU / CPU Kernels
      ↓
Prefill / Decode
      ↓
Generated Tokens

Serving Engine

Adding to the model execution capabilities:

HTTP / OpenAI-compatible API
               ↓
          Request Queue
               ↓
            Scheduler
               ↓
       Continuous Batching
               ↓
       Prefill / Decode
               ↓
        KV Cache Manager
               ↓
          GPU Kernels
               ↓
           CUDA / ROCm
               ↓
              GPU

Modern Serving Engine typically also require support for:

• Prefix Caching。
• Quantization。
• Streaming。
• Structured Outputs。
• LoRA Adapters。
• Multi-GPU。
• Metrics / Observability。

3. Hugging Face Transformers

Transformers is a general model library and execution framework, not specifically designed for high concurrency Serving.

Mainly provides:

• Model Architecture implementation.
• Pretrained Weights loaded.
• Tokenizer。
• Generation。
• Training / Fine-tuning supported.

Basic example:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "your-model-id"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
)

inputs = tokenizer("Hello", return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Suitable for:

• Understand the model architecture.
• Debugging the model.
• Study generative behavior.
• Fine-tune and experiment.
• Establish performance and quality baselines.

But if a large number of concurrent requests are required, specialized Serving Engine is usually considered.

4. llama.cpp

llama.cpp is an efficient inference project mainly implemented by C/C++.

It is especially suitable for local, consumer-grade hardware and quantized models.

Support:

• CPU Inference。
• CUDA, Metal and other hardware backends.
• GGUF。
• Quantization。
• GPU Offloading。
• HTTP Server。
• Continuous Batching。
• Speculative Decoding。
• Embedding / Reranking。
• Structured Outputs。

Official llama-server already provides OpenAI-compatible API, multi-request inference and Continuous Batching.

basic commands

llama-cli -m model.gguf

Start HTTP Server:

llama-server \
  -m model.gguf \
  --host 127.0.0.1 \
  --port 8080

Suitable for:

Apple Silicon
CPU-only
Consumer GPU
Limited VRAM
Local Quantized Models
Edge Devices

5. Ollama

Ollama encapsulates local model downloading, management, running, and API services into easy-to-use tools.

ollama run qwen3

Main abilities:

• Download and manage models.
• Run local model.
• Manage model parameters and templates.
• Create a custom model configuration via Modelfile.
• Provide HTTP API.
• Supports common quantization models.

The local API default address is usually:

http://localhost:11434

Ollama provides both its native API and OpenAI-compatible interfaces.

For learning local model deployment, Ollama is often simpler than directly manipulating the underlying inference engine.

But if you need to study the underlying KV Cache management, Kernel, and complex GPU scheduling strategies, you should further study vLLM, SGLang, or llama.cpp itself.

6. LM Studio

LM Studio is a tool for local model use and development.

Main abilities:

• GUI model search and download.
• Load and run local models.
• Model parameter adjustment.
• Local chat.
• HTTP API Server。
• CLI tools.

llama.cpp Runtime is supported, and the MLX model is supported on Apple Silicon.

The main differences from Ollama are:

The positioning of the two is close, and it is usually enough to master one.

7. vLLM

vLLM is an open source inference engine for high-throughput, low-latency LLM Serving.

Main optimizations:

• PagedAttention / Paged KV Cache。
• Continuous Batching。
• Chunked Prefill。
• Automatic Prefix Caching。
• Speculative Decoding。
• Quantization。
• Multi-GPU / Distributed Inference。
• LoRA Serving。
• OpenAI-compatible API。

The key capability of vLLM is to improve GPU utilization and overall Serving Throughput through efficient scheduling and KV Cache management.

Start model service

vllm serve your-model-id \
  --host 127.0.0.1 \
  --port 8000

The model, GPU environment and dependencies need to be configured correctly.

Common parameters:

Actual parameter available values vary with vLLM version, model, and hardware.

Typical uses

• High concurrency LLM API.
• RAG service.
• Agent backend.
• Multi-user chat system.
• GPU inference performance test.

8. SGLang

SGLang is also a high-performance LLM Serving Framework.

It has large functional overlap with vLLM, including:

• Continuous Batching。
• Paged Attention。
• Quantization。
• Tensor / Pipeline / Expert Parallelism。
• Speculative Decoding。
• Structured Outputs。
• Multi-LoRA。
• Prefill-Decode Disaggregation。

Key features of SGLang include RadixAttention, Prefix Caching and the high-performance Runtime.

RadixAttention / Radix Cache

The shared Prompt Prefix is organized through structures such as Radix Tree and the corresponding KV Cache is reused.

For example:

Request A:
System Prompt + User Question A

Request B:
System Prompt + User Question B

              Shared Prefix
                   ↓
             Cached KV Blocks
                   ↓
         ┌─────────┴─────────┐
      Request A           Request B

For requests with a large number of shared System Prompt, Few-shot Examples, or duplicate prefixes, significant gains may be achieved.

But vLLM also supports Prefix Caching, so this is not a capability unique to SGLang.

vLLM and SGLang

You cannot generally assume that one engine is always faster.

Benchmark should be performed for the same model, hardware, input and output lengths, and concurrency conditions.

9. TensorRT-LLM

TensorRT-LLM is the technology stack of NVIDIA optimized for LLM inference.

The focus is on fully utilizing the hardware capabilities of NVIDIA GPU.

Related technologies:

• Tensor Cores。
• CUDA Optimized Kernels。
• FP8 / FP4 Quantization。
• Kernel Fusion。
• In-flight Batching。
• Paged KV Cache。
• Chunked Prefill。
• Speculative Decoding。
• Multi-GPU / Multi-node Parallelism。

The official NVIDIA documentation lists the above main optimization features, but low-precision format support also depends on the GPU architecture. For example, high-performance support for FP4 is related to hardware such as Blackwell.

Suitable for:

• NVIDIA GPU production environment.
• High-performance inference optimization.
• Deep tuning for specific GPU architecture.

It emphasizes more on the NVIDIA hardware and software ecosystem, and is not a universal solution across all hardware platforms.

10. TGI：Text Generation Inference

TGI is LLM Serving Engine developed by Hugging Face.

Once an important open source model production deployment solution, it supports:

• Continuous Batching。
• Streaming。
• Quantization。
• Paged Attention。
• Prometheus Metrics。
• OpenAI-compatible API。

But TGI has entered Maintenance Mode.

Hugging Face It is currently recommended that new deployments prioritize alternatives such as vLLM and SGLang.

Therefore, it is enough to understand the historical positioning and basic mechanism of TGI. It is not recommended to use it as the first choice for learning and producing Serving.

11. OpenAI-compatible API

Modern inference engines often provide a HTTP interface compatible with the OpenAI API style.

For example:

Application
     ↓
OpenAI-compatible HTTP API
     ↓
Inference Server
     ↓
Model Runtime
     ↓
GPU

Common Endpoint:

POST /v1/chat/completions
POST /v1/completions
GET  /v1/models

Python example

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local-key",
)

response = client.chat.completions.create(
    model="your-model-id",
    messages=[
        {"role": "user", "content": "Hello"}
    ],
)

print(response.choices[0].message.content)

By modifying base_url, applications can connect to different compatible inference engines.

However, OpenAI-compatible does not mean that all functions are completely consistent. There may still be compatibility differences in tool calls, structured output, Reasoning, Streaming, etc.

In addition, the externally exposed inference service must correctly set up authentication, rate limiting and access control; API Key in the local example does not mean that the production authentication has been configured.

12. Prefill-Decode Disaggregation

Prefill and Decode have different resource requirements:

Prefill
   ↓
Compute-intensive

Decode
   ↓
Memory-bandwidth-intensive

Prefill-Decode Disaggregation executes the two phases on different Worker or GPU resources.

Incoming Request
       ↓
  Prefill Worker
       ↓
  Transfer KV Cache
       ↓
  Decode Worker
       ↓
  Output Tokens

Main purpose:

• Optimize Prefill and Decode independently.
• Reduce the interference of long Prompt on Decode.
• Improved tail ITL / TPOT.
• Allocate different computing resources to different stages.

Cost:

• Increase system complexity.
• Efficient transmission of KV Cache is required.
• Introduces additional communication and scheduling overhead.

Both vLLM and SGLang already provide related capabilities, the specific maturity and deployment method depend on the version.

It is mainly used for large-scale production deployment, and the single GPU learning stage does not require focused research.

13. How to choose an inference engine

Key points in selection

When actually choosing an inference engine, the main checks are:

1. Model Compatibility: Whether this model architecture, Tokenizer and Chat Template are supported.
2. Hardware Compatibility: CUDA, ROCm, Metal and GPU models.
3. Quantization Support: Whether to support the current model quantization format and optimization Kernel.
4. Performance: TTFT, TPOT, Throughput and video memory usage.
5. Serving Features: Streaming, Tool Calling, Structured Outputs, LoRA, Metrics, etc.

Function support cannot be judged based on the engine name alone. The specific version, model and deployment backend should be checked.

Summary

Model Framework
   └── Transformers

Local Inference
   ├── llama.cpp
   ├── Ollama
   └── LM Studio

Production GPU Serving
   ├── vLLM
   ├── SGLang
   └── TensorRT-LLM

Legacy / Maintenance
   └── TGI

The most important differences:

• Transformers: Model loading and general execution framework.
• llama.cpp: Lightweight, cross-hardware efficient inference Runtime.
• Ollama/LM Studio: Simplify local model running and management.
• vLLM / SGLang: Emphasis on high throughput and high concurrency Serving.
• TensorRT-LLM: Deeply optimized for NVIDIA GPU.
• TGI: Hugging Face Serving Engine entering maintenance mode.

For inference deployment learning, mastering the two routes llama.cpp/Ollama and vLLM can cover most of the core knowledge of local quantized inference and GPU production Serving.

---

Reference documentation

• Transformers
• llama.cpp
• Ollama
• LM Studio
• vLLM
• SGLang
• TensorRT-LLM
• TGI