LiteLLM Proxy as an Internal Gateway for AI Models

LiteLLM can sit between your applications and several model providers and offer them through one OpenAI-compatible interface. This tutorial sets up a proxy with one local and one cloud model and calls both from Python.

Niklas Schult4 min read

Illustration: One inlet pipe splits into two, leading to a small stone hut and a small cloud.

What you will have at the end and what you need

At the end you run a LiteLLM proxy on port 4000 that exposes two model names. One points to a local model served by Ollama, the other to an OpenAI model. Your applications talk only to the proxy, so a change of provider happens in one configuration file instead of in every application.

The LiteLLM README describes the AI Gateway as an open source, self-hosted service that calls more than 100 LLM providers in the OpenAI format. You need Python with the uv package manager, a running Ollama instance on its default address, a local model pulled there, and an OpenAI API key. The README does not cover installing Ollama, so follow its own documentation for that part.

Install the proxy and create the configuration file

The README installs the proxy as a uv tool with the proxy extra. The following command does that and makes the litellm command available.

Shell
uv tool install 'litellm[proxy]'
litellm --model gpt-6.1-sol

The configuration file lists each model under a name that your applications will use. The README lists ollama_chat as a provider and openai as another, and the OpenAI model name below is the one its examples use. The documentation excerpt does not show a config file format, so treat the next file as a sketch and check the proxy documentation for the exact keys. Save it as config.yaml.

Start the proxy and call it from Python

Set the key in the shell and start the proxy. The README shows the litellm command with a model argument for its quick start. How to hand over a configuration file is left open by the excerpt, so look that up in the proxy documentation before you start.

Python
import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")

response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

The proxy speaks the OpenAI protocol, so the official openai Python package works against it.

Check that the proxy reaches both models

If the script prints one answer for each name, the proxy reaches both the local and the external model. When only one name fails, look at that entry in config.yaml, at the Ollama address or at the API key. You can also test a single model without Python.

The README shows a curl call against the chat completions endpoint for its auto router example. The version below uses the same pattern with the local model name. It expects your virtual key in the variable LITELLM_API_KEY, which the README names in that example. The documentation excerpt does not say how to create virtual keys, so the proxy documentation is the place to look.

Shell
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-6.1-sol","messages":[{"role":"user","content":"Hello!"}]}'

Typical pitfalls according to the README

The README is short on troubleshooting, so the notes below come from its wording. The provider prefix in the model entry matters, because the README lists ollama and ollama_chat as separate providers and the chat variant supports chat completions. The key for the cloud model belongs in an environment variable, and the proxy must be started from a shell where that variable is set.

  • For production use the README recommends the Docker images with the -stable tag, which it says went through 12 hour load tests before publication.
  • Docker images on GHCR are signed with cosign, and the README shows how to verify them against a pinned commit hash.
  • Install only one SDK distribution per environment, because litellm and litellm-core own overlapping Python files.

The README also lists Terraform modules for AWS and GCP and a Helm chart. These cover a larger deployment than this single-host setup.

Where this fits in a company

A central gateway gives IT one place for model access. The README names authentication, virtual keys, spend tracking per project and user, guardrails and an admin dashboard as gateway features. Someone has to own the configuration file, the provider keys and the updates, and the proxy then becomes part of your normal operations.

For data protection the routing decision matters. Companies commonly check which data may go to which provider and document that, often with the works council (Betriebsrat) involved. Our pages on local AI and GDPR and AI cover those questions.

souveraen.ai offers an OpenAI-compatible developer API and a model catalogue with local models, models hosted in Germany and external models in the EU. If you prefer a ready platform to a self-built gateway, see the on-premise option.

Sources

More articles

All articles

How safe are AI API routes via resellers and routers?

Security firm Dragos sent about two million requests with fake credentials to AI providers and found compromised services and resellers reading customer prompts. Companies that use AI through third parties need to know who sits on the path of their data.

Data protection4 min

Run Text Generation Inference with Docker on Your Server

Text Generation Inference (TGI) serves open language models from a single Docker container on your own hardware. This tutorial starts it, sends requests over HTTP and keeps prompts inside your network.

Tutorial4 min

Cloud outage liability: who answers when Microsoft fails

On 12 November 2026 a software fault took Exchange Online and Outlook offline for about 36 hours. A legal analysis published shortly afterwards argues that responsibility for such outages often lies with the operator rather than the provider.

Regulation4 min

Test it with your own documents.

Free for 3 days, then from €24.90 per user per month. Cancellable monthly.

Free for 3 days, then from €24.90 per user per month.Free for 3 days
Try free for 3 days