vLLM quickstart for an OpenAI-compatible server

vLLM serves an open language model through the OpenAI API protocol on your own hardware. This tutorial follows the vLLM quickstart and ends with a running, key-protected endpoint that internal applications can call.

Niklas Schult4 min read

Illustration: A plug in a wall socket with a small padlock hanging from its cable, symbolising a secured self-hosted server.

What you will have at the end and what you need

At the end you will have a vLLM server on a Linux machine that hosts the model Qwen/Qwen2.5-1.5B-Instruct and answers requests in the OpenAI format. Internal tools can use it in place of an external provider, so prompts stay on a server you operate. The steps follow the vLLM quickstart documentation.

The documentation lists Linux as the operating system and Python 3.11 to 3.14. The commands below are for NVIDIA GPUs, while the documentation has separate instructions for AMD ROCm, Intel GPUs, Google TPUs, Ascend NPUs and Apple Silicon. You also need the Python environment manager uv. By default vLLM downloads models from Hugging Face, so the server needs access to it for the first start. If you are planning a wider rollout, the overview of running models locally may help.

Install vLLM in its own Python environment

The documentation recommends uv for creating and managing the environment. The first command creates a virtual environment with Python 3.12, the second activates it, and the third installs vLLM. The flag --torch-backend=auto lets uv pick the matching PyTorch index from the installed CUDA driver version.

Shell
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

The last command, vllm download-kernels, fetches precompiled kernels. Without it vLLM downloads and compiles them at startup, which the documentation says can add several minutes on Hopper and newer GPUs.

If you prefer conda, the documentation shows how to create a conda environment and install uv inside it with pip. The vLLM installation then uses the same uv pip command.

Start the server and query the endpoint

The command below starts the server with the Qwen model and requires an API key. According to the documentation, --api-key or the environment variable VLLM_API_KEY makes the server check for a key in the request header. You can pass several keys after the flag and the server accepts any of them, which helps with key rotation. Replace the example value with a secret of your own.

Shell
vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key ersetzen-sie-diesen-wert

By default the server listens at http://localhost:8000, and you can change the address with --host and --port. It hosts one model at a time and offers endpoints such as list models, create chat completion and create completion. Because the protocol is the OpenAI one, existing applications mostly need only a new base URL. Our developer API follows the same OpenAI-compatible idea for applications that should reach models and company knowledge.

How do I check that the server responds?

The simplest test lists the hosted models. If the server runs, the response names Qwen/Qwen2.5-1.5B-Instruct. When you started the server with --api-key, the request needs the key in its header, otherwise the server rejects it. The documentation does not show the header format for this case, so check the OpenAI API conventions for your client.

Shell
curl http://localhost:8000/v1/models

A second test sends a chat request with the openai Python package. The script points base_url at the local server and sets the model name to the one that was started. The documentation uses the value EMPTY for api_key on a server without a key, and with a key set you would pass that key instead.

Python
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a joke."},
    ],
)
print(response)

Typical pitfalls from the documentation

Three points from the documentation tend to cause surprises. Each affects start time or answer quality and leaves the server running.

  • Without vllm download-kernels, kernels are downloaded and compiled at startup, which can take several minutes on Hopper and newer GPUs.
  • The server applies generation_config.json from the Hugging Face model repository if it exists, so sampling defaults may come from the model creator. Passing --generation-config vllm when launching disables this.
  • The server uses a predefined chat template stored in the tokenizer, and the documentation points to a separate page on overriding it.

The documentation also notes that vLLM selects an attention backend automatically. You can set one with --attention-backend, for example FLASH_ATTN on NVIDIA CUDA, but the automatic choice is meant to be the most performant for your system.

Where this fits in a company

A self-hosted endpoint keeps prompts on infrastructure you control, which matters for GDPR questions and for professional secrecy under § 203 StGB, the German criminal law provision for lawyers, tax advisers and similar professions. The documentation does not cover operation, so IT needs to name an owner for several questions: who patches the server, who issues API keys, how the port is shielded from the wider network and what gets logged. Pages on GDPR and AI at work and security and hosting cover what companies commonly check.

A bare model server is only one building block. Staff still need search, source citations and permissions on top. souveraen.ai offers an on-premise option called Private Spark, where the platform runs on an NVIDIA DGX Spark in the company's own network. Its model catalogue includes local models, models hosted in Germany and external EU models.

Sources

More articles

All articles

How safe are AI API routes via resellers and routers?

Security firm Dragos sent about two million requests with fake credentials to AI providers and found compromised services and resellers reading customer prompts. Companies that use AI through third parties need to know who sits on the path of their data.

Data protection4 min

Run Text Generation Inference with Docker on Your Server

Text Generation Inference (TGI) serves open language models from a single Docker container on your own hardware. This tutorial starts it, sends requests over HTTP and keeps prompts inside your network.

Tutorial4 min

Cloud outage liability: who answers when Microsoft fails

On 12 November 2026 a software fault took Exchange Online and Outlook offline for about 36 hours. A legal analysis published shortly afterwards argues that responsibility for such outages often lies with the operator rather than the provider.

Regulation4 min

Test it with your own documents.

Free for 3 days, then from €24.90 per user per month. Cancellable monthly.

Free for 3 days, then from €24.90 per user per month.Free for 3 days
Try free for 3 days