Run Text Generation Inference with Docker on Your Server
Text Generation Inference (TGI) serves open language models from a single Docker container on your own hardware. This tutorial starts it, sends requests over HTTP and keeps prompts inside your network.

What you will have and what you need first
At the end you will have a container that serves an open language model on port 8080 of your server. Applications in your network can query it over HTTP. Prompts go to your own machine, and the only outside traffic is the first download of the model weights from Hugging Face.
The project README from Hugging Face lists the prerequisites. You need Docker and, for NVIDIA GPUs, the NVIDIA Container Toolkit. The README recommends NVIDIA drivers with CUDA version 12.2 or higher. AMD Instinct MI210 and MI250 GPUs are supported through a separate image tag ending in -rocm.
One caveat belongs at the start. The README states that TGI is in maintenance mode and accepts only minor bug fixes and documentation improvements. It names vllm, SGLang, llama.cpp and MLX as engines to use going forward. TGI still works for a first internal test, but factor this in when you choose a long-term engine. Our page on running local AI covers the wider options.
Start the container and load a model
The command below sets the model name and a local folder for the weights. It then starts the official image, publishes container port 80 as port 8080 on the host and mounts the folder at /data. Because of the mount, the weights are downloaded only once. The flag --shm-size 1g gives the container shared memory, which the README explains is used by NCCL for communication between GPUs.
model=HuggingFaceH4/zephyr-7b-beta
# share a volume with the container to avoid downloading weights every run
volume=$PWD/data
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id $modelThe first start takes a while because the model is downloaded. Later starts reuse the data folder. A server without a GPU can drop --gpus all and add --disable-custom-kernels, although the README warns that CPU is not the intended platform and performance may be poor.
Query the model over HTTP
The native endpoint is generate_stream. The following request sends a prompt as JSON and limits the answer to 20 new tokens with the parameter max_new_tokens. The reply arrives as a stream of events.
curl 127.0.0.1:8080/generate_stream \
-X POST \
-d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' \
-H 'Content-Type: application/json'TGI also offers a Messages API that is compatible with the OpenAI Chat Completion API. The next request uses the path /v1/chat/completions with a chat message, the model name tgi and streaming switched on. Existing tools that speak the OpenAI format can therefore point at your server instead of a cloud service.
curl localhost:8080/v1/chat/completions \
-X POST \
-d '{"model":"tgi","stream":true,"max_tokens":20,\
"messages":[{"role":"user","content":"What is deep learning?"}]}' \
-H 'Content-Type: application/json'Check that everything works
A working setup returns streamed text for both requests, and the container log shows the model loading without errors. The README says the OpenAPI documentation of the REST API is served under the /docs route, so opening that path on port 8080 confirms the server is up. The same Swagger documentation is published on the Hugging Face project site.
text-generation-launcher --helpTo see every launch option, run the launcher help. The README names this command for listing all settings, including quantization and tracing.
Typical pitfalls from the documentation
The README names several problems, and each has a documented remedy. The list below summarises them so you can check your setup against it before you start searching forums.
- Without the NVIDIA Container Toolkit the GPU flag does not work in Docker.
- Gated models such as Llama variants need a Hugging Face read token, passed to the container with the environment variable HF_TOKEN.
- Too little shared memory affects multi-GPU setups. Under Kubernetes, mount a memory-backed volume at /dev/shm. Setting NCCL_SHM_DISABLE=1 turns sharing off but costs performance.
- Large models can exceed GPU memory. The README describes quantization with the --quantize option, including bitsandbytes-nf4 and bitsandbytes-fp4, to reduce the VRAM requirement.
Where this fits in a company
Someone has to own it, so pick an administrator, restrict port 8080 to the internal network and plan updates, since the project is in maintenance mode. For monitoring, the README mentions Prometheus metrics and OpenTelemetry tracing through the --otlp-endpoint argument.
Data protection questions remain with you. Prompts stay on your server, but the works council (Betriebsrat) and your data protection officer usually want to know what is logged and who has access. For GDPR and AI at work the model itself is only one part of the picture.
souveraen.ai is an AI platform for company knowledge that runs in two modes with the same software, hosted in Germany or on-premise as Private Spark. It offers an OpenAI-compatible developer API for your own applications. Teams that need answers from documents with sources, rather than a bare model endpoint, can compare it with a self-built TGI setup.
Sources
- Text Generation Inference README, Hugging Face


