LiteLLM Proxy as an Internal Gateway for AI Models
LiteLLM can sit between your applications and several model providers and offer them through one OpenAI-compatible interface. This tutorial sets up a proxy with one local and one cloud model and calls both from Python.

What you will have at the end and what you need
At the end you run a LiteLLM proxy on port 4000 that exposes two model names. One points to a local model served by Ollama, the other to an OpenAI model. Your applications talk only to the proxy, so a change of provider happens in one configuration file instead of in every application.
The LiteLLM README describes the AI Gateway as an open source, self-hosted service that calls more than 100 LLM providers in the OpenAI format. You need Python with the uv package manager, a running Ollama instance on its default address, a local model pulled there, and an OpenAI API key. The README does not cover installing Ollama, so follow its own documentation for that part.
Install the proxy and create the configuration file
The README installs the proxy as a uv tool with the proxy extra. The following command does that and makes the litellm command available.
uv tool install 'litellm[proxy]'
litellm --model gpt-6.1-solThe configuration file lists each model under a name that your applications will use. The README lists ollama_chat as a provider and openai as another, and the OpenAI model name below is the one its examples use. The documentation excerpt does not show a config file format, so treat the next file as a sketch and check the proxy documentation for the exact keys. Save it as config.yaml.
Start the proxy and call it from Python
Set the key in the shell and start the proxy. The README shows the litellm command with a model argument for its quick start. How to hand over a configuration file is left open by the excerpt, so look that up in the proxy documentation before you start.
import openai
client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gpt-6.1-sol",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)The proxy speaks the OpenAI protocol, so the official openai Python package works against it.
Check that the proxy reaches both models
If the script prints one answer for each name, the proxy reaches both the local and the external model. When only one name fails, look at that entry in config.yaml, at the Ollama address or at the API key. You can also test a single model without Python.
The README shows a curl call against the chat completions endpoint for its auto router example. The version below uses the same pattern with the local model name. It expects your virtual key in the variable LITELLM_API_KEY, which the README names in that example. The documentation excerpt does not say how to create virtual keys, so the proxy documentation is the place to look.
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-6.1-sol","messages":[{"role":"user","content":"Hello!"}]}'Typical pitfalls according to the README
The README is short on troubleshooting, so the notes below come from its wording. The provider prefix in the model entry matters, because the README lists ollama and ollama_chat as separate providers and the chat variant supports chat completions. The key for the cloud model belongs in an environment variable, and the proxy must be started from a shell where that variable is set.
- For production use the README recommends the Docker images with the -stable tag, which it says went through 12 hour load tests before publication.
- Docker images on GHCR are signed with cosign, and the README shows how to verify them against a pinned commit hash.
- Install only one SDK distribution per environment, because litellm and litellm-core own overlapping Python files.
The README also lists Terraform modules for AWS and GCP and a Helm chart. These cover a larger deployment than this single-host setup.
Where this fits in a company
A central gateway gives IT one place for model access. The README names authentication, virtual keys, spend tracking per project and user, guardrails and an admin dashboard as gateway features. Someone has to own the configuration file, the provider keys and the updates, and the proxy then becomes part of your normal operations.
For data protection the routing decision matters. Companies commonly check which data may go to which provider and document that, often with the works council (Betriebsrat) involved. Our pages on local AI and GDPR and AI cover those questions.
souveraen.ai offers an OpenAI-compatible developer API and a model catalogue with local models, models hosted in Germany and external models in the EU. If you prefer a ready platform to a self-built gateway, see the on-premise option.
Sources
- LiteLLM README, LiteLLM


