vLLM quickstart for an OpenAI-compatible server
vLLM serves an open language model through the OpenAI API protocol on your own hardware. This tutorial follows the vLLM quickstart and ends with a running, key-protected endpoint that internal applications can call.

What you will have at the end and what you need
At the end you will have a vLLM server on a Linux machine that hosts the model Qwen/Qwen2.5-1.5B-Instruct and answers requests in the OpenAI format. Internal tools can use it in place of an external provider, so prompts stay on a server you operate. The steps follow the vLLM quickstart documentation.
The documentation lists Linux as the operating system and Python 3.11 to 3.14. The commands below are for NVIDIA GPUs, while the documentation has separate instructions for AMD ROCm, Intel GPUs, Google TPUs, Ascend NPUs and Apple Silicon. You also need the Python environment manager uv. By default vLLM downloads models from Hugging Face, so the server needs access to it for the first start. If you are planning a wider rollout, the overview of running models locally may help.
Install vLLM in its own Python environment
The documentation recommends uv for creating and managing the environment. The first command creates a virtual environment with Python 3.12, the second activates it, and the third installs vLLM. The flag --torch-backend=auto lets uv pick the matching PyTorch index from the installed CUDA driver version.
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernelsThe last command, vllm download-kernels, fetches precompiled kernels. Without it vLLM downloads and compiles them at startup, which the documentation says can add several minutes on Hopper and newer GPUs.
If you prefer conda, the documentation shows how to create a conda environment and install uv inside it with pip. The vLLM installation then uses the same uv pip command.
Start the server and query the endpoint
The command below starts the server with the Qwen model and requires an API key. According to the documentation, --api-key or the environment variable VLLM_API_KEY makes the server check for a key in the request header. You can pass several keys after the flag and the server accepts any of them, which helps with key rotation. Replace the example value with a secret of your own.
vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key ersetzen-sie-diesen-wertBy default the server listens at http://localhost:8000, and you can change the address with --host and --port. It hosts one model at a time and offers endpoints such as list models, create chat completion and create completion. Because the protocol is the OpenAI one, existing applications mostly need only a new base URL. Our developer API follows the same OpenAI-compatible idea for applications that should reach models and company knowledge.
How do I check that the server responds?
The simplest test lists the hosted models. If the server runs, the response names Qwen/Qwen2.5-1.5B-Instruct. When you started the server with --api-key, the request needs the key in its header, otherwise the server rejects it. The documentation does not show the header format for this case, so check the OpenAI API conventions for your client.
curl http://localhost:8000/v1/modelsA second test sends a chat request with the openai Python package. The script points base_url at the local server and sets the model name to the one that was started. The documentation uses the value EMPTY for api_key on a server without a key, and with a key set you would pass that key instead.
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Tell me a joke."},
],
)
print(response)Typical pitfalls from the documentation
Three points from the documentation tend to cause surprises. Each affects start time or answer quality and leaves the server running.
- Without vllm download-kernels, kernels are downloaded and compiled at startup, which can take several minutes on Hopper and newer GPUs.
- The server applies generation_config.json from the Hugging Face model repository if it exists, so sampling defaults may come from the model creator. Passing --generation-config vllm when launching disables this.
- The server uses a predefined chat template stored in the tokenizer, and the documentation points to a separate page on overriding it.
The documentation also notes that vLLM selects an attention backend automatically. You can set one with --attention-backend, for example FLASH_ATTN on NVIDIA CUDA, but the automatic choice is meant to be the most performant for your system.
Where this fits in a company
A self-hosted endpoint keeps prompts on infrastructure you control, which matters for GDPR questions and for professional secrecy under § 203 StGB, the German criminal law provision for lawyers, tax advisers and similar professions. The documentation does not cover operation, so IT needs to name an owner for several questions: who patches the server, who issues API keys, how the port is shielded from the wider network and what gets logged. Pages on GDPR and AI at work and security and hosting cover what companies commonly check.
A bare model server is only one building block. Staff still need search, source citations and permissions on top. souveraen.ai offers an on-premise option called Private Spark, where the platform runs on an NVIDIA DGX Spark in the company's own network. Its model catalogue includes local models, models hosted in Germany and external EU models.
Sources
- vLLM quickstart, vLLM documentation


