This guide walks you through installing, configuring, and maintaining LLM Systems Manager on your servers.
Agents can run on:
The script installer needs Python 3.10+ on the agent host. The native packages and the binary tarball bundle a self-contained agent and need no Python at all.
Each agent host needs network access to ports 8081 (alarm engine) and 5000 or 5443 (manager).
The following (configurable) ports must be reachable between the components listed.
| Port | What it is | Who needs to reach it |
|---|---|---|
| 5000 | Manager web interface (HTTP) | Browser |
| 5443 | Manager web interface (HTTPS, optional) | Browser |
| 5444 | Alarm event WebSocket proxy | Browser |
| 8081 | Alarm Engine API — receives metrics from agents | Agents and Manager |
| 8082 | Agent API — manager contacts the agent here | Manager |
| 8086 | InfluxDB time-series database | Alarm Engine and Manager |
For a quick installation install on one host, choose the full install option:
bash <(curl -fsSL https://raw.githubusercontent.com/llmsyscore/llm-systems-manager/main/tools/installer/install.sh)
You will see a menu like this:
Select the deployment option:
1) Full system manager + alarm engine + agent + InfluxDB
2) Manager + alarm manager + alarm engine (existing InfluxDB)
3) Manager only Flask manager + dashboard
4) Alarm engine only standalone FastAPI alarm engine
5) Agent only Linux + macOS host agent
6) InfluxDB only InfluxDB v2 + scoped tokens (DB host)
7) Update installed detect, diff, backup, sync-only-changed, restart
8) Uninstall remove all services and files, with confirmation prompts
9) Quit exit with no changes
Mode [1-9]:
Choose option 1 for a first-time setup on a single server. This installs everything you need in one step.
The installer will ask a few questions:
If you are unsure about any optional setting, you can accept the default and change it later in the configuration file.
The installer enables the services but does not start them — you decide when to bring them up. Start all three services:
sudo systemctl start llm-systems-manager
sudo systemctl start llm-systems-alarm-engine
sudo systemctl start llm-systems-agent
Verify each one is running:
sudo systemctl status llm-systems-manager
sudo systemctl status llm-systems-alarm-engine
sudo systemctl status llm-systems-agent
Each should show active (running). If a service failed to start, check the logs (see Viewing Logs below).
Open a browser and go to:
http://<your-server-address>:5000
Log in with the default credentials:
llmadminllmadminImportant: Change this password immediately after your first login. Go to the account menu in the top navigation bar and choose Change Password.
Each GitHub Release also ships .deb and .rpm packages: llm-systems-manager installs the manager + alarm engine (the mode-2 layout: InfluxDB stays external, agents install separately), and per-arch llm-systems-agent packages install the self-contained agent binary. The script installer remains the preferred, fully automated path — packages exist for hosts managed through apt/dnf tooling. Download the package for your distro from the Releases page, then:
Debian / Ubuntu:
sudo apt install ./llm-systems-manager_<version>_all.deb
The install prompts (via debconf) for the dashboard admin login and SMTP settings; press ENTER to accept defaults. Non-interactive installs (DEBIAN_FRONTEND=noninteractive) take the defaults silently.
RHEL / Rocky / Alma / Fedora:
sudo dnf install ./llm-systems-manager-<version>-1.noarch.rpm
RPM installs are non-interactive: config is generated with detected defaults at /opt/llm-systems-manager/config/llm-systems.toml — edit it and sudo systemctl restart llm-systems-manager afterwards. EL9’s default python3 is 3.9; install python3.11 (sudo dnf install python3.11 python3.11-pip) first — the package picks the newest Python ≥ 3.10 automatically.
Both manager packages create the llmsys runtime user, install and start the two systemd units, and build the Python venvs at configure time (network access to PyPI is required during install). On upgrades the live config is preserved (new keys are merged in). apt purge llm-systems-manager removes everything — config, data, logs, and the runtime user — when the package created the tree; state it didn’t create is kept (see Mixing install methods). dnf remove always keeps config/data behind with a notice.
InfluxDB: the package declares influxdb2 only as a Recommends — it lives in InfluxData’s third-party repo (not distro repos) and may legitimately run on another host, so a hard dependency would break both cases. If InfluxDB isn’t reachable after install, the postinst prints a notice pointing at tools/installer/install-influxdb.sh (local install) or the [influxdb] config section (external server). Metric history and alarms need it; the dashboard runs without it in the meantime. While [influxdb.tokens] still holds its REPLACE_ME placeholders, the alarm engine is enabled but not started — the postinst prints the steps (running install-influxdb.sh prints the tokens to paste); after setting the tokens, systemctl start llm-systems-alarm-engine.
Agent package:
sudo apt install ./llm-systems-agent_<version>_amd64.deb # or _arm64
sudo dnf install ./llm-systems-agent-<version>-1.x86_64.rpm # or .aarch64
The deb prompts (debconf) for the manager URL; the rpm takes defaults — set MANAGER_URL in /opt/llm-systems-agent/agent_config.yaml and restart if left blank. The binary is installed owned by llmsys so manager-driven self-update (Admin → Agents → Update) keeps working; after a self-update the on-disk binary is newer than the package until the next apt/dnf upgrade re-syncs it. Provider toggles (llama.cpp/LM Studio/vLLM control, sudo wrappers) are what the script installer automates — enable them in agent_config.yaml per its inline docs.
Packages are built by tools/packaging/build-packages.sh and tools/packaging/build-agent-package.sh (fpm) — see those scripts for the build-from-source path.
Install methods do not mix on one host — the script installer, the native packages, Docker, and the agent binary tarball each own the install tree, the systemd units, and the llmsys user differently, and mixing them shadows units or desyncs the package database. Both sides now guard against it:
/etc/systemd/system, a config the package didn’t create, a venv in the agent tree) and over busy service ports (a Docker control plane or script install still running). Override: LLMSYS_PACKAGE_FORCE=1 in the environment — the tree is then marked adopted and a later apt purge keeps config/data instead of deleting them.apt/dnf instead (the updater skips just the agent when only the agent is packaged). Override: LLMSYS_IGNORE_NATIVE_PACKAGE=1.apt purge only deletes config/data/logs when the package created the tree and no script-installer state appeared since; otherwise it keeps them and says so.Supported migrations:
tools/installer/uninstall.sh first, then install the package (fresh config), or LLMSYS_PACKAGE_FORCE=1 to adopt in place (config preserved).apt purge / dnf remove first, then run the script installer.Multi-arch images for the manager and alarm engine are published to ghcr.io on every release. No repo checkout is needed: download docker-compose.yml and .env.example, fill in the secrets, and docker compose up -d brings up the manager + alarm engine + InfluxDB together. See docker/README.md for the full walkthrough. Agents still install natively on each monitored host (they need sensor/GPU/systemd access).
The tap also carries control-plane formulas for macOS (Apple Silicon) and Linux:
brew tap llmsyscore/tap && brew trust llmsyscore/tap
brew install llm-systems-manager llm-systems-alarm-engine influxdb@2 influxdb-cli
$(brew --prefix)/etc/llm-systems-manager/llm-systems.toml, seeded on first install with generated alarm-engine ingest/management tokens (equivalent to a co-located script install). It is kept across upgrades.$(brew --prefix)/var/llm-systems-manager/ and also survives upgrades — the kegs hold only code + venvs.llm-systems-influx-setup (installed by the manager formula) — it starts the influxdb@2 service, onboards it on first boot, creates the buckets + scoped tokens, and writes them into [influxdb.tokens]. It needs both influxdb@2 (the v2 server — Homebrew’s plain influxdb formula is InfluxDB 3.x, whose API this stack does not speak) and influxdb-cli (the influx command ships in the separate influxdb-cli formula). Manual alternative: brew services start influxdb@2, influx setup, then create the buckets/tokens per Configuration and fill [influxdb] + [influxdb.tokens] yourself. The manager and alarm engine run without it, but history and alert evaluation stay degraded until then.brew services start llm-systems-manager first (first boot creates the internal CA and issues ae-tls.{crt,key} for the alarm engine), then brew services start llm-systems-alarm-engine.If you already have a manager running and want to start monitoring an additional server, install only the agent on that remote machine. The script installer below is the preferred path; two alternatives exist for hosts where it doesn’t fit:
apt/dnf install of the per-arch llm-systems-agent package — see Installing from Native Packages.llm-systems-agent-<platform>.tar.gz bundling the self-contained binary, agent_config.yaml.example, and the service unit template — extract to /opt/llm-systems-agent, set MANAGER_URL in a copied agent_config.yaml, install the unit, and systemctl enable --now llm-systems-agent. Full steps in the README’s Agent binary section.brew tap llmsyscore/tap && brew trust llmsyscore/tap && brew install llm-systems-agent, then set MANAGER_URL in $(brew --prefix)/etc/llm-systems-agent/agent_config.yaml and brew services start llm-systems-agent (launchd on macOS, systemd user unit on Linux). brew upgrade tracks new releases automatically. Full steps in the README’s Homebrew section.Copy just tools/installer/install.sh from an existing manager installation using scp or another file-transfer method, then run bash install.sh from the directory you copied them into.
The agent installer works on both Linux and macOS. It will ask for the manager server address so the agent knows where to register.
Or you can optionally download and run the installer from github
bash <(curl -fsSL https://raw.githubusercontent.com/llmsyscore/llm-systems-manager/main/tools/installer/install.sh)
On Linux:
sudo systemctl start llm-systems-agent
NOTE: On macOS, the installer registers a launchd service. Start it with:
launchctl start com.llm-systems-agent
New agents must be approved before the manager will accept their data. The agent should appear in the dashboard within about 30 seconds of starting.
Once approved, the agent begins sending metrics and the manager can communicate with it.
All runtime settings for the manager and alarm engine live in a single configuration file. The installer creates this file for you. Most settings have defaults and do not need to be changed.
The configuration file is located at:
config/llm-systems.toml
It is readable only by the service user (file permission 0600).
A fully documented template showing every available setting is located at:
config/llm-systems.toml.example
Refer to that file when you need to understand what a setting does or when adding a new key.
| Setting path | What it controls | Default |
|---|---|---|
[manager].port |
Port the manager web interface listens on | 5000 |
[manager].tls_port |
Port for HTTPS access (set to 0 to disable) |
5443 |
[manager.auth].mode |
Login requirement: required, trusted_cidr, or disabled |
required |
[manager].alarm_engine_url |
Network address where the Manager can reach the Alarm Engine | http://localhost:8081 |
[alarm_engine].tls_enabled |
Whether the alarm engine uses HTTPS | true |
[alarm_engine].ingest_token |
Shared token agents use to send metrics; blank means open | (set by installer) |
[notifications.smtp].server |
SMTP server hostname for email alarm notifications | (not set) |
[notifications.smtp].user |
Account / sender address used to send alarm emails | (not set) |
[influxdb].host |
InfluxDB server address | localhost |
[influxdb].port |
InfluxDB port | 8086 |
[manager.gateway].enabled |
OpenAI-compatible inference gateway | true |
[manager.gateway].api_keys |
Bearer keys for external clients; empty means dashboard-session only | [] |
[manager.reportcard].price_kwh |
Electricity price used for the Report Card’s $/Mtok estimate | 0.15 |
[manager.energy].cloud_price_in_per_mtok |
Hosted-API input price the savings card compares against | 0.15 |
[manager.energy].cloud_price_out_per_mtok |
Hosted-API output price the savings card compares against | 0.60 |
[manager.discord].enabled |
Interactive Discord bot | false |
[manager.discord].allowed_user_ids |
Discord user IDs permitted to use the bot; empty refuses everyone | [] |
[manager.discord].allow_model_control |
Whether the bot may load/unload models | false |
After editing config/llm-systems.toml, restart the affected service for the changes to take effect.
For changes that affect the manager:
sudo systemctl restart llm-systems-manager
For changes that affect the alarm engine:
sudo systemctl restart llm-systems-alarm-engine
If you changed a setting used by both (such as InfluxDB credentials), restart both.
Everything below works once the manager and at least one approved agent are running. None of it is required — each feature is independent.
One OpenAI-compatible endpoint fronts every provider you run (llama.cpp, LM Studio, vLLM), so applications target the manager instead of an individual backend.
Base URL: http://<manager-host>:5000/api/gateway/v1
GET /v1/models returns the merged catalog from every pool, each entry tagged with its provider and deduplicated by id.POST /v1/chat/completions and POST /v1/completions accept the usual OpenAI request bodies, streaming or not./api/gateway/llama/v1/*, /api/gateway/lms/v1/*, /api/gateway/vllm/v1/*.The manager resolves which provider owns a model from the model id in the request, then picks a host by per-model pin, explicit ?agent=, pool round-robin, and finally the system default. If that host can’t be reached before the first token, it fails over to another live host that serves the same model. The response carries an X-Proxied-To header naming the host that answered.
By default the gateway only accepts a logged-in dashboard session. To let external clients in, add bearer keys:
[manager.gateway]
enabled = true
api_keys = ["sk-your-secret-key"]
read_timeout_s = 600.0
Restart the manager afterwards. These keys are accepted only on /api/gateway/*.
Admin → Routing → Model Autopilot. You describe which models should be resident and where; Autopilot compares that against what is actually loaded and proposes the difference.
Each entry names a model, its provider (llama, vllm, or lms), and a placement — either a specific host id or auto. Optional per-entry settings:
| Field | Meaning | Default |
|---|---|---|
failover |
semi proposes a move for you to approve; auto performs it |
semi |
min_replicas / max_replicas |
How many copies to keep resident | 1 / same as min |
priority |
Lower wins when hosts are contended | 100 |
size_mb |
Size override for models the manager can’t measure | (unset) |
autoscale |
target_saturation, up_window_s, down_window_s |
0.75, 120, 900 |
Autopilot is disabled by default. While disabled it still evaluates continuously and shows what it would do, so you can watch it for a while before handing it control. Enable it with the toggle on the same card once the proposals look right.
Placement is gated on memory: a host is only offered a model if it has the VRAM to hold it, or the RAM if the host has no GPU. That is why a size override matters for models the manager can’t measure — without a size, an entry can be skipped rather than placed badly.
LLM Control → Report Card. Pick a provider and host, choose the standard preset, and run. The card reports time-to-first-token, prefill and generation throughput, tokens/joule, and $/Mtok, along with the GPU and VRAM it ran on.
The same preset runs against every provider, so cards are comparable between machines and between backends. Results are stored, so the Trends view plots them over time. Set the electricity price used for the cost figure with:
[manager.reportcard]
price_kwh = 0.15
It can also be overridden per run from the sub-tab. Leaderboard submission is present but disabled in v1.1.0.
Dashboard → Energy. Shows measured power draw converted to a $/Mtok figure, a monthly-savings comparison against hosted-API list pricing, and idle-power accounting so hardware that is powered but unused is attributed rather than ignored.
[manager.energy]
# price_kwh = 0.15 # unset inherits [manager.reportcard].price_kwh
cloud_price_in_per_mtok = 0.15
cloud_price_out_per_mtok = 0.60
cloud_price_label = "budget cloud API tier"
Cost is only computed across hosts that report both power and token telemetry. A host that reports one but not the other is left out rather than being allowed to skew the total. LM Studio reports no token counts of its own, so the gateway counts them as requests pass through — LM Studio traffic that bypasses the gateway is not counted.
Slash commands for host queries, model load/unload, and alarm acknowledgement.
[manager.discord]
enabled = true
bot_token = "<your-bot-token>"
guild_id = "<your-server-id>"
allowed_user_ids = ["111111111111111111", "222222222222222222"]
allow_model_control = false
Three things to get right:
allowed_user_ids refuses everyone. Each ID must be its own quoted string.allow_model_control = false; loading and unloading models requires setting it to true.bot and applications.commands scopes, otherwise the slash commands never register.Restart the manager after editing this section — the bot reads its allowlist at startup.
To update the manager, alarm engine, and any locally installed agent to the latest version, run the installer in update mode:
cd /opt/llm-systems-manager
bash tools/installer/install.sh --update
The update process:
You do not need to stop services first — the updater handles restarts.
To update an agent running on a remote machine without logging into that machine:
The agent downloads and applies the latest version of itself, then restarts.
Check whether each service is running:
sudo systemctl status llm-systems-manager
sudo systemctl status llm-systems-alarm-engine
sudo systemctl status llm-systems-agent
A healthy service shows active (running). A failed service shows failed and usually includes the last few log lines explaining why.
Manager — log file updated continuously:
tail -f /var/log/llm-systems-manager/llm-systems-manager.log
Or via journald:
journalctl -u llm-systems-manager -f
Alarm Engine:
journalctl -u llm-systems-alarm-engine -f
Agent:
journalctl -u llm-systems-agent -f
Add --since "1 hour ago" to any journalctl command to limit output to recent entries.
The Admin tab in the dashboard includes a System Health card. It shows:
The Admin tab button in the navigation bar turns red when any component reports a problem — you do not need to check manually.
To remove LLM Systems Manager from a server:
bash tools/installer/install.sh --uninstall
The uninstaller:
After your deployment is up and running, refer to these documents for deeper reference: