Inference Manager: Ubuntu manages local AI models

Canonical has presented the Inference Manager, a tool that installs and serves locally running AI models on Ubuntu. Companies with their own servers get a single entry point for models, though several checks remain before productive use.

Niklas Schult4 min read

Illustration: A reception bell on a counter in front of a drawer stand holding three pebbles, with a small lamp above.

What Canonical has presented

Canonical introduced the Inference Manager at its own conference, the Ubuntu Summit, as reported by heise online. The tool provides a command line program called inference, which installs models on the local machine and makes them available. Other programs then reach the models through an HTTP interface.

According to Canonical, this interface is compatible with the OpenAI API, so existing clients can connect directly. All installed models answer on TCP port 8400. Desktop chat assistants, coding agents in an editor and speech-to-text programs can each use the model they need.

To demonstrate the idea, Canonical built a slot car track at the summit in which AI models steered the cars. A web application let visitors pick a model, and the drivers reacted to spoken commands such as driving faster. The demonstration shows that one local interface can serve several models with different tasks.

Background: inference snaps and open-weight models

Snap is Ubuntu's own package format. The Snap Store already offers several AI models as inference snaps, which run entirely on the device and need no internet connection. The current release, Ubuntu 26.10, ships Myna, a speech-to-text program for dictation into any text field, and it builds on these snaps.

The Inference Manager adds a uniform layer on top. Canonical says it wants to rely on locally running open-weight models, meaning models whose trained weights are published and can be run on your own hardware. Planned uses include context-aware help with troubleshooting, better speech recognition for accessibility and automation through agentic workflows.

Canonical states that the AI functions will arrive as opt-in and can be uninstalled at any time. The source does not say which models beyond the example will be supported or how they are licensed. It also notes that parts of the open-source community discuss the use of AI controversially, which is the context for the opt-in approach.

How to install and manage models

The Inference Manager ships as a snap and is available in the Snap Store now. You install it through the App Center or on the command line. After that, models are added with the inference command.

  1. Install the tool with sudo snap install inference, or search for it in the App Center.
  2. Install a model, for example with inference install smollm2.
  3. Point your application at the local HTTP interface on port 8400 and configure it like an OpenAI-compatible endpoint.
  4. Check which models are installed and remove those you no longer need.

The sources describe only the install command and the interface. Commands for updating or listing models are not named there, so check the official documentation for those. A small model on a single workstation is enough for a first test of how existing clients behave. Test one client at a time, so that you can see which applications work with the interface unchanged.

What companies should check before using it

A local model keeps requests on your own hardware, which matters for personal data under the GDPR and for holders of professional secrets under § 203 StGB. That still leaves open who may use the endpoint. The sources do not mention authentication on port 8400, so restrict access through firewall rules and network segmentation until you have verified it.

  • Licence of each model: open-weight does not automatically allow commercial use.
  • Hardware: model size decides how much memory and GPU capacity a server needs.
  • Access: who can reach the interface, and is it logged?
  • Updates: who tests a new model version before it replaces the old one?
  • Works council: in German companies the Betriebsrat, the elected employee representation, may have a say if AI use can monitor staff.

Small models suit simple tasks such as dictation or short summaries. For demanding work, compare quality against the models you use today, and read more on running models locally before you size hardware. Smaller companies can find a general starting point in the page on AI in SMEs.

Where a platform complements the operation

The Inference Manager serves models, but it does not connect them to company documents, permissions or approval steps. That layer needs separate work, whether built in-house or bought. souveraen.ai, from clouditiv GmbH in Auerbach, is one such platform for company knowledge.

It answers questions from connected sources such as SharePoint or file servers and cites the document, version and page. It runs hosted in Germany or on-premise on an NVIDIA DGX Spark, and its model catalogue includes local models. An OpenAI-compatible API lets your own applications use models and company knowledge.

Sources

More articles

All articles

How safe are AI API routes via resellers and routers?

Security firm Dragos sent about two million requests with fake credentials to AI providers and found compromised services and resellers reading customer prompts. Companies that use AI through third parties need to know who sits on the path of their data.

Data protection4 min

Run Text Generation Inference with Docker on Your Server

Text Generation Inference (TGI) serves open language models from a single Docker container on your own hardware. This tutorial starts it, sends requests over HTTP and keeps prompts inside your network.

Tutorial4 min

Cloud outage liability: who answers when Microsoft fails

On 12 November 2026 a software fault took Exchange Online and Outlook offline for about 36 hours. A legal analysis published shortly afterwards argues that responsibility for such outages often lies with the operator rather than the provider.

Regulation4 min

Test it with your own documents.

Free for 3 days, then from €24.90 per user per month. Cancellable monthly.

Free for 3 days, then from €24.90 per user per month.Free for 3 days
Try free for 3 days