Run llama-server as an OpenAI-compatible server

llama-server from llama.cpp serves a GGUF model over an OpenAI-compatible HTTP interface and runs on CPU as well as GPU. Existing tools can then talk to a model that stays on your own server.

Niklas Schult4 min read

Illustration: A small wooden house with an open door and a light bulb in a jar, with a bell on a cord beside it that answers when pulled.

What you will have at the end and what you need

At the end you have a llama-server process that loads one GGUF model and answers chat requests on an OpenAI-compatible route. The llama.cpp server README states that the server runs inference for F16 and quantized models on GPU and CPU. A GPU cluster is therefore not a precondition.

You need a llama.cpp build that contains the llama-server binary, or Docker if you prefer containers. You also need a GGUF model file on the server and a shell with curl for the test. The documentation leaves open how much memory a given model needs, so compare the size of your file with the RAM of the machine.

Start the server with a GGUF model

The first command loads the model with -m, sets the prompt context to 4096 tokens with -c and gives the model the name firmenmodell through --alias. Clients use this alias in their requests. Without further options the server listens on 127.0.0.1, so only the machine itself can reach it.

Shell
llama-server -m /srv/models/mein-modell.gguf -c 4096 --alias firmenmodell

The README also describes a built-in web UI, which is enabled by default and can be switched off with --no-ui. The options -hf and -mu exist for fetching a model instead of reading it from disk. A model file on local storage keeps the setup independent of outside downloads.

The option -np sets the number of server slots, and the README lists continuous batching as enabled by default. Several colleagues can therefore send requests at the same time, and the server decodes them in parallel. The context size is shared between the slots, so a larger number of slots leaves less room per conversation.

Make it reachable in the company network and secure it

To serve colleagues and other systems, the server needs a different listening address. The next command sets --host to a network address of your server and --port to 8080, and adds --api-key-file, which points to a file with one API key per line. Lines starting with a hash count as comments. With several keys, each application can have its own.

Shell
llama-server -m /srv/models/mein-modell.gguf -c 4096 \
  --alias firmenmodell \
  --host 0.0.0.0 --port 8080 \
  --api-key-file /etc/llama/keys.txt

For transport encryption the README lists --ssl-key-file and --ssl-cert-file, which take PEM files. The documentation does not show a complete TLS setup, so many teams place the server behind a reverse proxy they already operate. Make the key file readable only for the service account.

Run it as a container with Docker Compose

The README offers an image and shows a Compose file that configures the server through environment variables. The file below mounts a models directory, sets the model path, context size and number of parallel slots, and publishes port 8080. An argument on the command line takes precedence over the matching environment variable.

YAML
services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_PORT: 8080

The environment variable for the API key file is LLAMA_ARG_API_KEY_FILE, and the one for the alias is LLAMA_ARG_ALIAS. Both appear in the parameter tables of the README. Without an API key the container answers every client that can reach the published port.

Check that the interface answers

The request below sends one chat message to the route /v1/chat/completions. It passes the key as a bearer token and uses the alias as model name. Replace IHR_SCHLUESSEL with a line from your key file. A JSON reply with a choices entry shows that model, alias and authentication work together.

Shell
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer IHR_SCHLUESSEL" \
  -d '{"model":"firmenmodell","messages":[{"role":"user","content":"Hallo"}]}'

Tools with an OpenAI client usually need only a base URL pointing at your server and the key. The developer API of souveraen.ai follows the same OpenAI-compatible idea, so code written for one often needs few changes for the other.

Pitfalls and how this fits into operation

The README notes several defaults that matter. The port default is 9931 in this version, so set --port explicitly if other systems expect 8080. The default for --host is 127.0.0.1, which explains why remote clients fail at first. The experimental options --tools, --agent and the MCP settings are marked as unsuitable for untrusted environments. The server refuses them on a non-loopback address without an API key unless you add --allow-unauthenticated-tools.

In operation, someone has to own the service: updating the binary or image, replacing models, rotating keys and watching load. Prompts can contain personal data, so the topics in GDPR and AI at work apply even when the model is local. Background on the approach is on the page about local AI. souveraen.ai offers an on-premise mode called Private Spark, which runs the platform on an NVIDIA DGX Spark in the company network, described under on-premise operation.

Sources

More articles

All articles

How safe are AI API routes via resellers and routers?

Security firm Dragos sent about two million requests with fake credentials to AI providers and found compromised services and resellers reading customer prompts. Companies that use AI through third parties need to know who sits on the path of their data.

Data protection4 min

Run Text Generation Inference with Docker on Your Server

Text Generation Inference (TGI) serves open language models from a single Docker container on your own hardware. This tutorial starts it, sends requests over HTTP and keeps prompts inside your network.

Tutorial4 min

Cloud outage liability: who answers when Microsoft fails

On 12 November 2026 a software fault took Exchange Online and Outlook offline for about 36 hours. A legal analysis published shortly afterwards argues that responsibility for such outages often lies with the operator rather than the provider.

Regulation4 min

Test it with your own documents.

Free for 3 days, then from €24.90 per user per month. Cancellable monthly.

Free for 3 days, then from €24.90 per user per month.Free for 3 days
Try free for 3 days