What it is

vLLM is a fast, easy-to-use open-source library for LLM inference and serving, with the motto “easy, fast, and cheap LLM serving for everyone”. It began in the Sky Computing Lab at UC Berkeley and has become one of the most active open-source AI projects, maintained by a community spanning dozens of universities and companies and more than 2,000 contributors.

If tools like Ollama solve “get it running on my laptop”, vLLM solves “make it fast and cheap on a server”. When one model has to handle hundreds or thousands of concurrent requests, how memory is allocated, how requests are scheduled and how busy the GPU stays decide both cost and latency. vLLM optimises heavily for these, which is why it is widely used as a production inference backend.

Its core technique, PagedAttention, was published at SOSP 2023, a top systems conference. vLLM supports more than 200 model architectures on Hugging Face, including dense models such as Llama, Qwen and Gemma, mixture-of-experts models such as DeepSeek-V3 and Mixtral, and multimodal, embedding and reward models.

Key features

Performance:

  • State-of-the-art serving throughput;
  • Efficient management of attention key-value memory with PagedAttention;
  • Continuous batching of incoming requests, plus chunked prefill and prefix caching;
  • Quantisation including FP8, INT8, INT4, GPTQ/AWQ and GGUF;
  • Optimised attention kernels such as FlashAttention and FlashInfer;
  • Speculative decoding methods including n-gram and EAGLE;
  • Disaggregated prefill, decode and encode.

Flexibility:

  • Seamless integration with Hugging Face models;
  • Decoding algorithms such as parallel sampling and beam search;
  • Tensor, pipeline, data, expert and context parallelism for distributed inference;
  • Streaming, structured outputs, tool calling and reasoning parsers;
  • An OpenAI-compatible API server, plus the Anthropic Messages API and gRPC;
  • Efficient multi-LoRA support.

Hardware: NVIDIA and AMD GPUs and x86, ARM and PowerPC CPUs, with plugins for Google TPU, Intel Gaudi, Huawei Ascend, Apple Silicon and more.

How it works

PagedAttention. While generating text, a model keeps the attention keys and values (the KV cache) for every earlier token of each request. The traditional approach reserves one contiguous block of GPU memory per request sized for the maximum length. Since the real length is unknown in advance, much of that memory sits reserved but unused, and fragmentation is high.

PagedAttention borrows the idea of paged virtual memory from operating systems. The KV cache is split into fixed-size blocks allocated on demand, the blocks need not be contiguous in physical memory, and a table maps logical positions to physical ones. Far less memory is wasted, so the same GPU holds more concurrent requests, and requests with a shared prefix can share the same blocks.

Continuous batching. Static batching waits for a whole batch to finish before starting the next, so short requests sit idle. Continuous batching schedules at the granularity of each generation step: a finished request frees its slot immediately and new ones join at any time, keeping GPU utilisation high.

On top of these, vLLM layers prefix caching, chunked prefill, CUDA graphs, torch.compile and other optimisations to deliver its performance.

Distribution and parallelism. When a model is too large for one GPU, or one GPU is too slow, vLLM offers several kinds of parallelism. Tensor parallelism splits each layer’s matrix operations across GPUs; pipeline parallelism assigns different layers to different GPUs; data parallelism runs several replicas to share requests; expert parallelism places the experts of an MoE model on different devices; context parallelism handles very long inputs. They can be combined and scaled across nodes with tools such as Ray and Kubernetes.

Disaggregated serving. Inference has a prefill phase, which processes the whole input at once and is compute-heavy, and a decode phase, which generates output token by token and is memory-bandwidth-heavy. vLLM can run prefill, decode and even multimodal encoding on separate instances so each phase gets the resources it suits best.

How to use it

Install. The docs recommend Linux with Python 3.10 to 3.13 and uv for environments. For NVIDIA GPUs:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

AMD, Intel, TPU, Ascend and Apple Silicon each have their own install path; see the installation guide.

Offline batched inference:

from vllm import LLM, SamplingParams

prompts = ["Hello, my name is", "The capital of France is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(output.outputs[0].text)

Online serving: one command starts an OpenAI-compatible server, by default at http://localhost:8000:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Query it in the OpenAI format, for example curl http://localhost:8000/v1/models to list models. Before exposing it publicly, enable key checks with --api-key or the VLLM_API_KEY environment variable.

Models download from Hugging Face by default. Set VLLM_USE_MODELSCOPE=True to use ModelScope instead.

Use cases

  • Online LLM services: high-concurrency backends for chatbots, support agents and API platforms.
  • Private deployment: running open models on your own GPUs to replace or supplement commercial APIs.
  • Offline batch jobs: large-scale labelling, generation and evaluation.
  • Reinforcement learning: a fast sampling engine in RLHF and similar training loops.
  • Backend for higher-level tools: integrations with LangChain, LlamaIndex, Dify, Open WebUI and others.

FAQ

How does vLLM differ from Ollama? Ollama targets personal computers and puts simple setup first. vLLM targets servers and production, putting throughput and memory efficiency under load first, with multi-GPU and multi-node inference.

Can I use it without an NVIDIA GPU? Yes. It supports AMD GPUs and several CPU types, with plugins for TPU, Ascend, Apple Silicon and more, though feature coverage and performance vary by platform.

Why do my sampling settings differ from what I expected? By default vLLM reads the model repository’s generation_config.json and applies the creator’s recommended sampling parameters. Pass --generation-config vllm when starting the server to turn that off.

Can one server host several models? According to the quickstart, a vLLM server hosts one model at a time. For several models, run several servers or use a routing layer on top.