Run Text Generation Inference with Docker on Your Server

Text Generation Inference (TGI) serves open language models from a single Docker container on your own hardware. This tutorial starts it, sends requests over HTTP and keeps prompts inside your network.

Niklas Schult4 min read

Illustration: A small wooden crate holding a brass engine connects by a pipe to a latched wooden hut, suggesting self-hosted, private operation.

What you will have and what you need first

At the end you will have a container that serves an open language model on port 8080 of your server. Applications in your network can query it over HTTP. Prompts go to your own machine, and the only outside traffic is the first download of the model weights from Hugging Face.

The project README from Hugging Face lists the prerequisites. You need Docker and, for NVIDIA GPUs, the NVIDIA Container Toolkit. The README recommends NVIDIA drivers with CUDA version 12.2 or higher. AMD Instinct MI210 and MI250 GPUs are supported through a separate image tag ending in -rocm.

One caveat belongs at the start. The README states that TGI is in maintenance mode and accepts only minor bug fixes and documentation improvements. It names vllm, SGLang, llama.cpp and MLX as engines to use going forward. TGI still works for a first internal test, but factor this in when you choose a long-term engine. Our page on running local AI covers the wider options.

Start the container and load a model

The command below sets the model name and a local folder for the weights. It then starts the official image, publishes container port 80 as port 8080 on the host and mounts the folder at /data. Because of the mount, the weights are downloaded only once. The flag --shm-size 1g gives the container shared memory, which the README explains is used by NCCL for communication between GPUs.

Shell
model=HuggingFaceH4/zephyr-7b-beta
# share a volume with the container to avoid downloading weights every run
volume=$PWD/data

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id $model

The first start takes a while because the model is downloaded. Later starts reuse the data folder. A server without a GPU can drop --gpus all and add --disable-custom-kernels, although the README warns that CPU is not the intended platform and performance may be poor.

Query the model over HTTP

The native endpoint is generate_stream. The following request sends a prompt as JSON and limits the answer to 20 new tokens with the parameter max_new_tokens. The reply arrives as a stream of events.

Shell
curl 127.0.0.1:8080/generate_stream \
    -X POST \
    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' \
    -H 'Content-Type: application/json'

TGI also offers a Messages API that is compatible with the OpenAI Chat Completion API. The next request uses the path /v1/chat/completions with a chat message, the model name tgi and streaming switched on. Existing tools that speak the OpenAI format can therefore point at your server instead of a cloud service.

Shell
curl localhost:8080/v1/chat/completions \
    -X POST \
    -d '{"model":"tgi","stream":true,"max_tokens":20,\
    "messages":[{"role":"user","content":"What is deep learning?"}]}' \
    -H 'Content-Type: application/json'

Check that everything works

A working setup returns streamed text for both requests, and the container log shows the model loading without errors. The README says the OpenAPI documentation of the REST API is served under the /docs route, so opening that path on port 8080 confirms the server is up. The same Swagger documentation is published on the Hugging Face project site.

Shell
text-generation-launcher --help

To see every launch option, run the launcher help. The README names this command for listing all settings, including quantization and tracing.

Typical pitfalls from the documentation

The README names several problems, and each has a documented remedy. The list below summarises them so you can check your setup against it before you start searching forums.

  • Without the NVIDIA Container Toolkit the GPU flag does not work in Docker.
  • Gated models such as Llama variants need a Hugging Face read token, passed to the container with the environment variable HF_TOKEN.
  • Too little shared memory affects multi-GPU setups. Under Kubernetes, mount a memory-backed volume at /dev/shm. Setting NCCL_SHM_DISABLE=1 turns sharing off but costs performance.
  • Large models can exceed GPU memory. The README describes quantization with the --quantize option, including bitsandbytes-nf4 and bitsandbytes-fp4, to reduce the VRAM requirement.

Where this fits in a company

Someone has to own it, so pick an administrator, restrict port 8080 to the internal network and plan updates, since the project is in maintenance mode. For monitoring, the README mentions Prometheus metrics and OpenTelemetry tracing through the --otlp-endpoint argument.

Data protection questions remain with you. Prompts stay on your server, but the works council (Betriebsrat) and your data protection officer usually want to know what is logged and who has access. For GDPR and AI at work the model itself is only one part of the picture.

souveraen.ai is an AI platform for company knowledge that runs in two modes with the same software, hosted in Germany or on-premise as Private Spark. It offers an OpenAI-compatible developer API for your own applications. Teams that need answers from documents with sources, rather than a bare model endpoint, can compare it with a self-built TGI setup.

Sources

More articles

All articles

How safe are AI API routes via resellers and routers?

Security firm Dragos sent about two million requests with fake credentials to AI providers and found compromised services and resellers reading customer prompts. Companies that use AI through third parties need to know who sits on the path of their data.

Data protection4 min

Cloud outage liability: who answers when Microsoft fails

On 12 November 2026 a software fault took Exchange Online and Outlook offline for about 36 hours. A legal analysis published shortly afterwards argues that responsibility for such outages often lies with the operator rather than the provider.

Regulation4 min

GitLab 19.6 adds an audit log for AI agents

GitLab 19.6 records what AI agents do through its MCP server. A Ruby flaw is patched, but several NGINX flaws remain open, so admins who run GitLab themselves have settings to enable and a workaround to apply.

Models4 min

Test it with your own documents.

Free for 3 days, then from €24.90 per user per month. Cancellable monthly.

Free for 3 days, then from €24.90 per user per month.Free for 3 days
Try free for 3 days