LLM Systems Manager — Deployment Guide

This guide walks you through installing, configuring, and maintaining LLM Systems Manager on your servers.

Prerequisites

Manager Server

• Operating system: Ubuntu 22.04 or later (other modern Linux distributions should work)
• Python: 3.11 or later (the installer refuses to proceed below this)
• RAM: 2 GB minimum; 4 GB recommended
• Disk: At least 10 GB free for logs, metrics history, and model benchmark data

Remote Agent Hosts

Agents can run on:

• Linux — Ubuntu 22.04+ or equivalent
• macOS — macOS 13 (Ventura) or later

The script installer needs Python 3.10+ on the agent host. The native packages and the binary tarball bundle a self-contained agent and need no Python at all.

Each agent host needs network access to ports 8081 (alarm engine) and 5000 or 5443 (manager).

Required Ports

The following (configurable) ports must be reachable between the components listed.

Installing the Full Stack

For a quick installation install on one host, choose the full install option:

Step 1:

bash <(curl -fsSL https://raw.githubusercontent.com/llmsyscore/llm-systems-manager/main/tools/installer/install.sh)

Step 2: Choose an Installation Mode

You will see a menu like this:

  Select the deployment option:

    1)  Full system          manager + alarm engine + agent + InfluxDB
    2)  Manager + alarm      manager + alarm engine (existing InfluxDB)
    3)  Manager only         Flask manager + dashboard
    4)  Alarm engine only    standalone FastAPI alarm engine
    5)  Agent only           Linux + macOS host agent
    6)  InfluxDB only        InfluxDB v2 + scoped tokens (DB host)
    7)  Update installed     detect, diff, backup, sync-only-changed, restart
    8)  Uninstall            remove all services and files, with confirmation prompts
    9)  Quit                 exit with no changes

Mode [1-9]:

Choose option 1 for a first-time setup on a single server. This installs everything you need in one step.

Step 3: Answer Configuration Prompts

The installer will ask a few questions:

• Manager address — the IP address or hostname other machines will use to reach this server
• Admin password — the password for the initial admin account. It is stored as a secure hash, never in plain text.
• SMTP settings (optional) — email address and credentials for alarm notifications
• InfluxDB details — the installer can provision InfluxDB automatically in mode 1

On a manager-only (mode 3) install, the installer also asks for the alarm engine's management_token. This prompt cannot be skipped: paste the value mode 4 printed on the alarm-engine host, type new to mint one here and copy it over, or press Enter to keep a value carried over from a previous config. Non-interactive installs can supply it via LLMSYS_CFG_AE_MGMT_TOKEN.

If you are unsure about any optional setting, you can accept the default and change it later in the configuration file.

Step 4: Start the Services

The installer enables the services but does not start them — you decide when to bring them up. Start all three services:

sudo systemctl start llm-systems-manager
sudo systemctl start llm-systems-alarm-engine
sudo systemctl start llm-systems-agent

Verify each one is running:

sudo systemctl status llm-systems-manager
sudo systemctl status llm-systems-alarm-engine
sudo systemctl status llm-systems-agent

Each should show active (running). If a service failed to start, check the logs (see Viewing Logs below).

Step 5: Open the Dashboard

Open a browser and go to:

http://<your-server-address>:5000

Log in with the default credentials:

• Username: llmadmin
• Password: llmadmin

The first sign-in with the shipped password lands on a mandatory change-password form — the dashboard is unreachable until you set a new one (8 characters minimum). Afterwards, change it any time from the settings cog → Change my password.

Installing from Native Packages (.deb / .rpm)

Each GitHub Release also ships .deb and .rpm packages: llm-systems-manager installs the manager + alarm engine (the mode-2 layout: InfluxDB stays external, agents install separately), and per-arch llm-systems-agent packages install the self-contained agent binary. The script installer remains the preferred, fully automated path — packages exist for hosts managed through apt/dnf tooling. Download the package for your distro from the Releases page, then:

Debian / Ubuntu:

sudo apt install ./llm-systems-manager_<version>_all.deb

The install prompts (via debconf) for the dashboard admin login and SMTP settings; press ENTER to accept defaults. Non-interactive installs (DEBIAN_FRONTEND=noninteractive) take the defaults silently.

RHEL / Rocky / Alma / Fedora:

sudo dnf install ./llm-systems-manager-<version>-1.noarch.rpm

RPM installs are non-interactive: config is generated with detected defaults at /opt/llm-systems-manager/config/llm-systems.toml — edit it and sudo systemctl restart llm-systems-manager afterwards. EL9's default python3 is 3.9; install python3.11 (sudo dnf install python3.11 python3.11-pip) first — the package picks the newest Python ≥ 3.11 automatically.

Both manager packages create the llmsys runtime user, install and start the two systemd units, and build the Python venvs at configure time (network access to PyPI is required during install). On upgrades the live config is preserved (new keys are merged in). apt purge llm-systems-manager removes everything — config, data, logs, and the runtime user — when the package created the tree; state it didn't create is kept (see Mixing install methods). dnf remove always keeps config/data behind with a notice.

InfluxDB: the package declares influxdb2 only as a Recommends — it lives in InfluxData's third-party repo (not distro repos) and may legitimately run on another host, so a hard dependency would break both cases. If InfluxDB isn't reachable after install, the postinst prints a notice pointing at tools/installer/install-influxdb.sh (local install) or the [influxdb] config section (external server). Metric history and alarms need it; the dashboard runs without it in the meantime. While [influxdb.tokens] still holds its REPLACE_ME placeholders, the alarm engine is enabled but not started — the postinst prints the steps (running install-influxdb.sh prints the tokens to paste); after setting the tokens, systemctl start llm-systems-alarm-engine.

Agent package:

sudo apt install ./llm-systems-agent_<version>_amd64.deb        # or _arm64
sudo dnf install ./llm-systems-agent-<version>-1.x86_64.rpm     # or .aarch64

The deb prompts (debconf) for the manager URL; the rpm takes defaults — set MANAGER_URL in /opt/llm-systems-agent/agent_config.yaml and restart if left blank. The binary is installed owned by llmsys so manager-driven self-update (Admin → Agents → Update) keeps working; after a self-update the on-disk binary is newer than the package until the next apt/dnf upgrade re-syncs it. Provider toggles (llama.cpp/LM Studio/vLLM control, sudo wrappers) are what the script installer automates — enable them in agent_config.yaml per its inline docs.

Packages are built by tools/packaging/build-packages.sh and tools/packaging/build-agent-package.sh (fpm) — see those scripts for the build-from-source path.

Mixing install methods

Install methods do not mix on one host — the script installer, the native packages, Docker, and the agent binary tarball each own the install tree, the systemd units, and the llmsys user differently, and mixing them shadows units or desyncs the package database. Both sides now guard against it:

• The packages refuse a fresh install over script-installer state (units in /etc/systemd/system, a config the package didn't create, a venv in the agent tree) and over busy service ports (a Docker control plane or script install still running). Override: LLMSYS_PACKAGE_FORCE=1 in the environment — the tree is then marked adopted and a later apt purge keeps config/data instead of deleting them.
• The script installer, updater, and uninstaller refuse a package-managed host and point at apt/dnf instead (the updater skips just the agent when only the agent is packaged). Override: LLMSYS_IGNORE_NATIVE_PACKAGE=1.
• apt purge only deletes config/data/logs when the package created the tree and no script-installer state appeared since; otherwise it keeps them and says so.

Supported migrations:

• Script install → package: tools/installer/uninstall.sh first, then install the package (fresh config), or LLMSYS_PACKAGE_FORCE=1 to adopt in place (config preserved).
• Binary tarball → agent package: supported directly — a bare binary + config with no hand-made unit is adopted automatically (config/token preserved, binary replaced by the packaged one).
• Package → script install: apt purge / dnf remove first, then run the script installer.
• A self-updated agent binary (Admin → Agents → Update) is newer than what the package database recorded; a later package upgrade replaces it and warns if that was a downgrade — re-update from the dashboard or install a newer package.

Installing with Docker (control plane)

Multi-arch images for the manager and alarm engine are published to ghcr.io on every release. No repo checkout is needed: download docker-compose.yml and .env.example, fill in the secrets, and docker compose up -d brings up the manager + alarm engine + InfluxDB together. See docker/README.md for the full walkthrough. Agents still install natively on each monitored host (they need sensor/GPU/systemd access).

Installing with Homebrew (control plane)

The tap also carries control-plane formulas for macOS (Apple Silicon) and Linux:

brew tap llmsyscore/tap && brew trust llmsyscore/tap
brew install llm-systems-manager llm-systems-alarm-engine influxdb@2 influxdb-cli

• Config is shared at $(brew --prefix)/etc/llm-systems-manager/llm-systems.toml, seeded on first install with generated alarm-engine ingest/management tokens (equivalent to a co-located script install). It is kept across upgrades.
• State (internal CA, agent registry, TLS certs, SQLite DBs) lives under $(brew --prefix)/var/llm-systems-manager/ and also survives upgrades — the kegs hold only code + venvs.
• InfluxDB: run llm-systems-influx-setup (installed by the manager formula) — it starts the influxdb@2 service, onboards it on first boot, creates the buckets + scoped tokens, and writes them into [influxdb.tokens]. It needs both influxdb@2 (the v2 server — Homebrew's plain influxdb formula is InfluxDB 3.x, whose API this stack does not speak) and influxdb-cli (the influx command ships in the separate influxdb-cli formula). Manual alternative: brew services start influxdb@2, influx setup, then create the buckets/tokens per Configuration and fill [influxdb] + [influxdb.tokens] yourself. The manager and alarm engine run without it, but history and alert evaluation stay degraded until then.
• Start order: brew services start llm-systems-manager first (first boot creates the internal CA and issues ae-tls.{crt,key} for the alarm engine), then brew services start llm-systems-alarm-engine.
• Don't mix with a script/package install on the same host — both would fight over ports 5000/8081.

Installing Agents on Remote Computers

If you already have a manager running and want to start monitoring an additional server, install only the agent on that remote machine. The script installer below is the preferred path; two alternatives exist for hosts where it doesn't fit:

• Native package (Linux, no Python needed): apt/dnf install of the per-arch llm-systems-agent package — see Installing from Native Packages.
• Binary tarball (Linux or macOS, no Python needed): each release ships llm-systems-agent-<platform>.tar.gz bundling the self-contained binary, agent_config.yaml.example, and the service unit template — extract to /opt/llm-systems-agent, set MANAGER_URL in a copied agent_config.yaml, install the unit, and systemctl enable --now llm-systems-agent. Full steps in the README's Agent binary section.
• Homebrew (macOS/Apple Silicon, or Linux x86_64/arm64 with Homebrew): brew tap llmsyscore/tap && brew trust llmsyscore/tap && brew install llm-systems-agent, then set MANAGER_URL in $(brew --prefix)/etc/llm-systems-agent/agent_config.yaml and brew services start llm-systems-agent (launchd on macOS, systemd user unit on Linux). brew upgrade tracks new releases automatically. Full steps in the README's Homebrew section.

Step 1: Get the Installer on the Remote Host

Copy just tools/installer/install.sh from an existing manager installation using scp or another file-transfer method, then run bash install.sh from the directory you copied them into.

The agent installer works on both Linux and macOS. It will ask for the manager server address so the agent knows where to register.

Or you can optionally download and run the installer from github

bash <(curl -fsSL https://raw.githubusercontent.com/llmsyscore/llm-systems-manager/main/tools/installer/install.sh)

Step 2: Start the Agent (if not started during the installation)

On Linux:

sudo systemctl start llm-systems-agent

NOTE: On macOS, the installer registers a launchd service. Start it with:

launchctl start com.llm-systems-agent

Step 3: Approve the Agent

New agents must be approved before the manager will accept their data. The agent should appear in the dashboard within about 30 seconds of starting.

1. Open the dashboard in your browser
2. Go to the Admin tab
3. Click Agents
4. Find the new agent in the list and click Approve

Once approved, the agent begins sending metrics and the manager can communicate with it. Fleet-wide actions — Approve all pending, Update all, Push CA, and the agent-auth slider — live under the Manage ▾ menu on the same tab.

Configuration

All runtime settings for the manager and alarm engine live in a single configuration file. The installer creates this file for you. Most settings have defaults and do not need to be changed.

Configuration File Location

The configuration file is located at:

config/llm-systems.toml

It is readable only by the service user (file permission 0600).

A fully documented template showing every available setting is located at:

config/llm-systems.toml.example

Refer to that file when you need to understand what a setting does or when adding a new key.

Key Settings

Applying Changes

After editing config/llm-systems.toml, restart the affected service for the changes to take effect.

For changes that affect the manager:

sudo systemctl restart llm-systems-manager

For changes that affect the alarm engine:

sudo systemctl restart llm-systems-alarm-engine

If you changed a setting used by both (such as InfluxDB credentials), restart both.

InfluxDB Memory Settings

When the installer sets up InfluxDB (modes 1 and 6), it also protects it on hosts where memory is tight. Running install.sh --update applies the same settings to existing installs.

• Out-of-memory protection. A systemd drop-in, /etc/systemd/system/influxdb.service.d/llm-systems-manager.conf, sets OOMScoreAdjust=-500. If the host runs out of memory, the kernel then stops other processes before InfluxDB. Without it, InfluxDB is usually the largest process, so it is the one that gets stopped, and because it needs 2–3 GB while it starts back up, it can get stuck being stopped and restarted over and over. Check it with systemctl show -p OOMScoreAdjust influxdb (expect -500).
• Memory limit (GOMEMLIMIT). When InfluxDB shares the host with the manager, alarm engine or an agent, the installer writes GOMEMLIMIT into /etc/default/influxdb2, the file the service reads its environment from. This makes InfluxDB free unused memory sooner instead of holding gigabytes it no longer needs. The value is 35% of the host's RAM, at least 1024 MiB, and at least 1600 MiB on hosts with 4 GiB or more (6 GiB → 2150 MiB, 16 GiB → 5734 MiB). It is a target, not a hard cap: InfluxDB can go above it when it really needs the memory. The installer summary shows the value. An InfluxDB-only host (mode 6) gets no limit.
• Your own value wins. The installer only edits the lines between its # === llm-systems-manager memory (managed) === markers. To pick your own limit, add a GOMEMLIMIT=... line outside the markers; the installer then removes its block and leaves your line alone.
• Don't add MemoryMax. A hard systemd memory cap below what InfluxDB needs while starting up gets it killed on every start, which is the same restart loop the protection above prevents.
• Applying changes. A fresh install restarts InfluxDB itself. When an update changes either setting, InfluxDB is added to the updater's restart prompt. After editing /etc/default/influxdb2 by hand, run sudo systemctl restart influxdb.
• Uninstalling InfluxDB through the uninstaller removes the drop-in and the managed block.
• Homebrew (macOS) installs don't get these settings: macOS has no Linux out-of-memory killer to adjust, and Homebrew generates the influxdb@2 service file itself.

Setting Up the Main Features

Everything below works once the manager and at least one approved agent are running. None of it is required — each feature is independent.

Inference Gateway

One OpenAI-compatible endpoint fronts every provider you run (llama.cpp, LM Studio, vLLM), so applications target the manager instead of an individual backend.

Base URL: http://<manager-host>:5000/api/gateway/v1

• GET /v1/models returns the merged catalog from every pool, each entry tagged with its provider and deduplicated by id.
• POST /v1/chat/completions and POST /v1/completions accept the usual OpenAI request bodies, streaming or not.
• Provider-scoped twins exist when you want to force one backend family: /api/gateway/llama/v1/*, /api/gateway/lms/v1/*, /api/gateway/vllm/v1/*.

The manager resolves which provider owns a model from the model id in the request, then picks a host by per-model pin, explicit ?agent=, pool round-robin, and finally the system default. If that host can't be reached before the first token, it fails over to another live host that serves the same model. The response carries an X-Proxied-To header naming the host that answered.

By default the gateway only accepts a logged-in dashboard session. To let external clients in, add bearer keys:

[manager.gateway]
enabled = true
api_keys = ["sk-your-secret-key"]
read_timeout_s = 600.0

Restart the manager afterwards. These keys are accepted only on /api/gateway/*.

Model Autopilot

Admin → Gateway → Model Autopilot. You describe which models should be resident and where; Autopilot compares that against what is actually loaded and proposes the difference.

Each entry names a model, its provider (llama, vllm, or lms), and a placement — either a specific host id or auto. Optional per-entry settings:

A separate opt-in toggle, Protect other models (protect_unmanaged, off by default), sits beside Autopilot on the Gateway card: when on, a host whose single resident slot holds a model Autopilot doesn't manage is blocked from displacement, and freed-memory credit from displacing a model is never counted toward it. Route sync also skips busy hosts.

Autopilot is disabled by default. While disabled it still evaluates continuously and shows what it would do, so you can watch it for a while before handing it control. Enable it with the toggle on the same card once the proposals look right.

Placement is gated on memory: a host is only offered a model if it has the VRAM to hold it, or the RAM if the host has no GPU. That is why a size override matters for models the manager can't measure — without a size, an entry can be skipped rather than placed badly.

GPU Report Card

LLM Control → Tools → Report Card. Pick a provider and host, choose the standard preset, and run. The card reports time-to-first-token, prefill and generation throughput, tokens/joule, and $/Mtok, along with the GPU and VRAM it ran on.

Report Card shares the Tools launcher with two sibling modules, Benchmark and Autotune, run in-tab from the same app-style workspace. All three record to a cross-tool run ledger, so a run started in one browser is visible in every other session and survives a closed tab.

The same preset runs against every provider, so cards are comparable between machines and between backends. Results are stored, so the Trends view plots them over time. Set the electricity price used for the cost figure with:

[manager.reportcard]
price_kwh = 0.15

It can also be overridden per run from the sub-tab. Leaderboard submission is present but disabled.

Energy & Cost Intelligence

Dashboards → Energy. Shows measured power draw converted to a $/Mtok figure, a monthly-savings comparison against hosted-API list pricing, and idle-power accounting so hardware that is powered but unused is attributed rather than ignored. The window selector covers trailing spans, Today (local midnight to now), Year to date, and a Custom range of up to 366 days; the hourly chart itself is capped at 45 days and says so in its label when a wider window truncates it.

[manager.energy]
# price_kwh = 0.15                # unset inherits [manager.reportcard].price_kwh
cloud_price_in_per_mtok = 0.15
cloud_price_out_per_mtok = 0.60
cloud_price_label = "budget cloud API tier"

Cost is only computed across hosts that report both power and token telemetry. A host that reports one but not the other is left out rather than being allowed to skew the total. LM Studio reports no token counts of its own, so the gateway counts them as requests pass through — LM Studio traffic that bypasses the gateway is not counted.

Discord Bot

Slash commands for host queries, model load/unload, and alarm acknowledgement.

[manager.discord]
enabled = true
bot_token = "<your-bot-token>"
guild_id = "<your-server-id>"
allowed_user_ids = ["111111111111111111", "222222222222222222"]
allow_model_control = false

Three things to get right:

• The allowlist is fail-closed. An empty allowed_user_ids refuses everyone. Each ID must be its own quoted string.
• Model control is separate. Queries work with allow_model_control = false; loading and unloading models requires setting it to true.
• The bot invite needs both the bot and applications.commands scopes, otherwise the slash commands never register.

Restart the manager after editing this section — the bot reads its allowlist at startup.

Tower assistant

An in-dashboard drawer that answers questions about hosts, models, alerts, energy, and saved model profiles over the inference gateway. Enable it in Admin › Settings › Tower assistant: turn Enabled on and, optionally, pick a Primary model from the loaded ones — the default auto lets Tower use any loaded chat model, since Tower never loads one itself.

Choosing a model. Tower needs a chat model that can call tools. Any resident chat model works with Primary model set to auto; a pinned model must already be loaded (Tower never loads one). Recommendations:

• Size: 7B parameters or more. Under 7B (the drawer marks these small model) expect tool calls written as text now and then; Tower corrects one per question and falls back to a canned line after the second, so answers get slower and shorter.
• Family: an instruction-tuned model with tool-call training — Qwen3 8B/14B/27B, Gemma 3 12B/27B, Llama 3.1 8B+ or their fine-tunes. Base, embedding, reranker, speech and image models are skipped automatically.
• Thinking models (Qwen3 in think mode, reasoning fine-tunes) spend part of every reply thinking. The Thinking setting under Temperature decides how much: Off asks the model to skip it, Low, Medium (the default) and High allow 1k, 2k or 6k reasoning tokens on top of Answer length, and the budget is a hard stop everywhere: llama.cpp stops the model itself, and on LM Studio and vLLM Tower stops the thinking at the budget and asks once more with thinking off, handing the model its notes so far. A model that loops in thought is cut off and made to answer. Expect the first token to take longer with thinking on; Tower retries a reply that finished thinking without writing an answer once, then reports the last step it completed.
• Server: built-in function calling needs llama.cpp with jinja templates on (the default in current builds), LM Studio, or vLLM; otherwise Tower uses its text format, which works everywhere but depends more on the model. Leave Tool calls on Auto: the Tool response check row in Settings › Tower assistant probes the primary and fallback models once and shows Tools OK (filled = built-in, outlined = text format), No tool support or small model; Verify re-runs it after a server or model change.
• Fallback: with a second chat model loaded, switch Fallback model on so a timeout, a failed check or a model that keeps writing tool calls as text hands the question to the other model.
• Measure it: the Model evaluation row in Settings › Tower assistant runs a handful of canned questions through Tower against the primary model (plain reads, a host named by alias, a two-step read, a timer, an action proposal, and a question card when two hostnames share a prefix) and scores each: the right tool with the right arguments, the right kind of reply, a real answer. More calls per question, corrections and retries mean a weaker model. Nothing changes on any host (every action card is denied, every question card takes its first choice); results are kept per model with its quant and server, listed in the Tools tab ledger, and Export downloads the full JSON.
• Download a Tower model: the row below it lists recommended models by the VRAM they need (Qwen3.5 4B / 9B, Gemma 4 E4B / 12B / 26B-A4B, Llama 3.1 8B), capped at 14B dense plus one mixture-of-experts: bigger models cost first-token time and VRAM on every question without a better score. Q4_K_M is the floor (lower quants start dropping tool calls); the 12 GB and 16 GB tiers offer the 9B and 12B at Q6 instead, since small models gain more from precision than from parameters. Download fetches one to any llama.cpp or LM Studio host (primaries listed first), loads it, runs the tool check and the eval, and then offers Pin as primary; the fallback stays as configured. On a llama.cpp host the model gets a config profile named tower (32k context, 99 GPU layers, flash attention, jinja, ubatch 1024, temperature 0.2, reasoning on with a 2048-token budget) that LLM Control shows as its active profile, and on either host the alias Tower Model unless it already has one. A model kept for Tower takes VRAM away from what the host serves, and on a llama.cpp host the server restarts after the download, so what it was serving is unloaded until it is loaded again; the Download button says so before it starts. Stop on a llama.cpp download also removes the half-finished files from the host's cache; on LM Studio it ends the job here only, and the download has to be cancelled in LM Studio as well (the row says so, and a download cancelled in LM Studio ends the job here). LM Studio loads the model with a 32k context and a 2048 batch; temperature is per request, and Tower sends its own. The Primary model list shows each model with its server and hosts, and marks a pinned model the gateway no longer lists as not available.

It starts scoped down: "Capabilities" defaults to read (answer only), and off-topic questions are refused. Both can be relaxed later in the same settings group, as can the tool list (Available tools).

Set "Capabilities" to Answer and act to let it load or unload a model, wake llama-server, or acknowledge / close an alert; incl. admin actions adds restarting a provider server (admins only). Every action pauses on a card in the drawer — what it will do, where, what it will not do, and who may approve — and runs only after Approve. Approvals expire after 10 minutes, the caller's role and the tier are re-checked at that moment, and each decision lands in Admin › Audit as tower via <user>. Any single tool can still be switched off under Available tools.

The phone companion carries the same assistant as a Tower tab: ask and read the answer, approve or deny action cards, answer question cards, switch between conversations, and read the Insights list with Dismiss, an Alert link into the Alerts screen, and Apply where the role and tier allow it. Apply and Approve confirm in the bottom sheet first. The tab is hidden while Tower is off.

Tower never produces code: a language-tagged code fence in a reply is withheld and replaced with a single "Code withheld" line, though configuration Tower reads for diagnosis — TOML, JSON, YAML, plain text and the like — still shows. "Report rule-bypass attempts" (Settings › Tower assistant, on by default) covers messages that try to make Tower ignore its rules, reveal its instructions, skip approvals or take on another persona: these are refused outright, and with reporting on the refusal is logged to Admin › Audit as a critical tower.violation and raises a critical alert; with reporting off, Tower still refuses but nothing is logged or alerted. Questions that probe the manager's security internals (vulnerabilities, exploits, security holes, attack surface, breaking into it, getting around the login) get the same treatment whatever the off-topic setting: Tower answers with a fixed block that points at the developer's support contacts and what to include in a report, the model is never asked, and with reporting on the message is logged as a warning tower.violation and raises a warning alert named "Tower security probe".

Request timeout (s) bounds how long Tower waits for the model to start answering. With Fallback model on, a question that misses it is asked of the next loaded chat model instead (another host first), and the reply says so; the next question goes back to the primary model.

Debug logging (Settings › Tower assistant, and the same switch under Inference Gateway) writes a per-step trace to the manager log with names, sizes and timings only.

"Diagnose new alarms" (Settings › Tower assistant, off by default) makes Tower look at every alert that opens at or above "Lowest severity". Each look is a read-only pass over the same tools, at most five reads, stopped after "Diagnosis time limit" (180 s by default; raise it for a slow model), and its result lands on the drawer's Insights tab as a compact card: a one-line summary, Details (what it checked), and a button for the playbook when one of Tower's playbooks fits. While a conversation is open, a one-line notice above it links to a new insight. The playbooks wake llama-server, reload an LM Studio model pinned to that host, acknowledge an alert once it has recovered (a threshold alert whose value is back past its threshold, or a non-engine alert that reports recovery), and restart llama-server (admin-only). Before anything runs, Tower re-reads the alert and refuses when it has closed or changed. The header badge counts insights not yet seen, including ones a safe playbook already applied, a toast announces one while the drawer is closed, opening Tower with unseen insights lands on the Insights tab, and insights show even when no chat model is loaded. "Apply safe playbooks" lets the three safe ones run without asking when "Capabilities" is at least Answer and act; each run is written to Admin › Audit as tower via alarm <id>. Alerts already open when the switch turns on are not diagnosed.

Timers let Tower poll something and report later: ask for "the RAM on box every minute for 10 minutes" or "check the alarms again in 5 minutes" and Tower schedules a timer instead of waiting. Each tick reads one live host metric (CPU, RAM, GPU, GPU temperature or watts) or any read tool, at most once every 30 s, for up to 60 minutes or 120 ticks, with three timers per user and eight per manager; the strip under the drawer's tabs shows each live timer with its next tick, the time left and a Cancel. When a timer finishes, its samples come back as a new turn in the same conversation with a small graph, and Tower reports the values, the range and the trend. Ticks count against the same per-minute limit as questions, a manager restart ends every live timer, and every schedule, cancel and report is written to Admin › Audit. The schedule tool can be switched off under Available tools.

llama.cpp serves native tool calls only when its server args include --jinja; otherwise Tower falls back to a fenced JSON tool format automatically. Nothing Tower reads or does leaves the lab.

Forecast

A scheduled, read-only check that looks for trends in up to 30 days of stored history — alarm history, per-host hardware and load, energy and cost, benchmark and report-card results, model and gateway activity, the audit log — and reports what is heading toward trouble before it becomes an alarm. It runs on its own schedule, reads only, and never needs Tower to be on. Off by default.

Turn it on in Admin › Settings › Forecast: switch Run forecasts on, pick how often it runs and at what local time (hourly through weekly, or Custom… for any number of hours or days and a time to the minute) and how far back it looks, and optionally let findings also raise alerts through your normal alert channels once they reach a chosen severity. Any check can be switched off individually from the same page. Run now starts an out-of-schedule pass at any time.

Each finding is written by plain code — the trend, the projected date, the confidence, and a suggested next step all come from measured numbers, never from a language model. When Tower is enabled, it can additionally add a short plain-language gloss on top of the code's own findings: a Tower effort setting controls how much of that Tower does, and Auto (the default) picks a level for you based on how well the current Tower model scores, its size and its speed, so a small or slow model does less and a strong one does more:

• Off — no Tower involvement; every finding shows only the code's own summary and next step.
• Light — one short digest tying the run's findings together at the top of the page.
• Standard — the digest, plus a one-line "Likely cause" under each finding.
• Full — the digest and likely causes, plus Tower looking into each flagged check with its own read tools for a closer explanation.

Tower is never allowed to invent or restate a number, date or rate — those figures always come from the code that measured them, and a finding's chart, facts and suggested step never change because of what Tower adds. A weekly digest summarising the run also appears on its own day and time, independent of how often forecasts run.

The Forecast page lists open findings with search, severity/host/check filters, sortable columns and pages; a 30-day line shows every dated prediction. Cleared and dismissed findings have their own tabs. Each finding offers Ask Tower (opens Tower with the finding as the question) and, where there is somewhere useful to go, a button that opens that part of the dashboard. The Overall page shows only critical and warning findings.

Two things worth knowing before you rely on it. Dismissing a finding is not permanent: dismissed and cleared findings are swept after 90 days, so a trend that is still running when its row is swept comes back as a new finding. Tower conversations are kept by default (tower_history) — at the Full effort level each check's investigation runs in its own private conversation, and an admin can open it read-only from the finding ("Open conversation"). Nobody can continue or rename it, it never appears in anyone's History, and it is swept with the rest of Tower's history. Turn the setting off to delete each conversation as soon as its check finishes.

Backups

Admin → Backups. A scheduled run writes an encrypted manager archive (config, agent registry, CA, users, model profiles, and the three SQLite files manager.db, audit.db, energy.db; archives made before the split, carrying a single metrics.db, still restore) and, when [alarm_engine].management_token is set, the alarm engine's own export in the same run — without a management token the run is recorded as manager only. Retention counts runs, not archives, so keep_last = 7 can retain up to 14 files.

[manager.backup]
enabled = true
interval_hours = 24.0
keep_last = 7
mirror_dir = "/mnt/backup-share"   # optional second copy of every archive

mirror_dir is validated on save — it must be an absolute path that exists or can be created, and writable by the service user — and applies without a restart. Any retained archive can be downloaded from the Backups card (backup.download is recorded in the audit log); a run where the alarm engine is configured but unreachable is marked partial rather than silently dropping that half.

OpenClaw Proxy Origin

If you run the OpenClaw tool through Tools → OpenClaw, the OpenClaw host's own gateway checks the browser's Origin header and rejects pages it doesn't recognize. Add the manager dashboard's origin to the allowlist on the OpenClaw host:

gateway.controlUi.allowedOrigins = ["https://<manager-host>:5443"]

Without this, the embedded page fails with "Browser origin not allowed" even though the proxy itself is reachable.

OpenClaw Telemetry (OTLP)

OpenClaw's diagnostics-otel plugin can push metrics, traces, and logs to the alarm engine's OTLP receiver (/v1/metrics, /v1/traces, /v1/logs on the ingest port, bearer = [alarm_engine].ingest_token). The receiver is served over TLS with a certificate signed by the manager's internal CA, and the OpenClaw gateway (a Node process) verifies it. Node does not read that CA from the OS trust store even with NODE_USE_SYSTEM_CA=1, so give it the CA file directly through NODE_EXTRA_CA_CERTS. The agent on the OpenClaw host already keeps a current copy at <agent install dir>/data/tls-ca.pem, and Node adds it on top of its built-in public roots.

The gateway service reads its environment from a file OpenClaw generates at ~/.openclaw/service-env/ai.openclaw.gateway.env (macOS) or ~/.openclaw/service-env/openclaw-gateway.env (Linux). Either regenerate it with the variable set:

NODE_EXTRA_CA_CERTS=<agent install dir>/data/tls-ca.pem openclaw gateway install --force

or change the existing NODE_EXTRA_CA_CERTS= line in that file to the CA path and run openclaw gateway restart. Both only restart the gateway process; OpenClaw's config, sessions and plugins are untouched (a later openclaw gateway install regenerates the file, so set the variable again then).

Then point the plugin at the engine in openclaw.json:

diagnostics.otel.endpoint = "https://<alarm-engine-host>:8081"
diagnostics.otel.headers.Authorization = "Bearer <ingest_token>"

Without the CA the exporter fails silently: the OpenClaw log shows unable to verify the first certificate and the engine's heartbeat otlp journal line stays at metrics+0 traces+0 logs+0; the Admin System Health card notes it once the engine has run 15 minutes with no OTLP data (the Alarm Engine node's detail shows OTLP received). The ingest token is not involved in that failure. Note this feed is separate from the Dashboards → OpenClaw tab, which the agent fills from OpenClaw's own session store.

Which attributes become tags. The receiver keeps telemetry attributes as InfluxDB tags only when they are bounded dimensions. Keys that name identifiers, network endpoints or free text (*_id, *Id, *hash, token, session, trace/span/frame ids, pid, port, addr, endpoint, peer, url, path, userAgent, message(s), content, value, command, prompt, description, …) are dropped; numeric attributes and keys ending in a unit (_ms, _bytes, _chars, _count, _tokens, _ratio, …) are stored as fields on the point instead of tags; the standard gen_ai.*, http.*, rpc.*, error.type and service.version dimensions and OpenClaw's bounded security.policy_id / security.control_id are always kept. Every other string attribute is a tag until it has shown [alarm_engine.otlp].tag_value_cap distinct values (default 100), after which new values are stored as other; at most max_tags attribute tags (default 32) are kept per point. Add keys to tag_allow / tag_deny under [alarm_engine.otlp] to override the pattern rules (dotted OpenTelemetry names or their underscored tag form both work). The heartbeat otlp journal line reports tags dropped+N capped+N fields+N, and the same counters are written as source=otlp-receiver metrics (otlp.tags_dropped, otlp.tags_capped, otlp.attr_fields, otlp.tag_keys_tracked) so a rule can watch them.

Existing installs. Series minted by the earlier receiver (one per event id, timing value or connection) stop growing as soon as the engine restarts on this version and age out with the raw bucket's retention; the rollup bucket keeps its copies for its own, longer retention. No manual cleanup is needed. To reclaim the index sooner, delete the OpenClaw source's history from both buckets with the InfluxDB CLI — note this removes all history for that source, and predicate deletes leave tombstones that slow queries until the next compaction:

influx delete --org <org> --bucket alarm_engine_metrics        --start 1970-01-01T00:00:00Z --stop $(date -u +%Y-%m-%dT%H:%M:%SZ) --predicate 'source="openclaw-gateway"'
influx delete --org <org> --bucket alarm_engine_metrics_rollup --start 1970-01-01T00:00:00Z --stop $(date -u +%Y-%m-%dT%H:%M:%SZ) --predicate 'source="openclaw-gateway"'

Operator-Provided TLS Certificate

By default the HTTPS port ([manager].tls_port, 5443) serves a certificate from the manager's internal CA, which browsers on other devices do not trust. To serve a certificate they do trust — a Let's Encrypt cert for your domain, or one from a corporate CA:

[manager]
tls_cert_file = "/etc/ssl/private/lab.example.com.fullchain.pem"
tls_key_file  = "/etc/ssl/private/lab.example.com.key.pem"

How it behaves:

• Selection is by SNI. The operator cert is served only for hostnames its DNS SANs cover (a wildcard matches one label). Requests by IP or by names outside the SANs — including every agent, which pins the internal CA — still get the internal-CA cert, so configuring this never breaks the agent control channel.
• Both keys must be set and readable by the service user. A half-set pair or unreadable files log a warning and fall back to the internal CA. Relative paths resolve against the install root, and a group/world-readable key file is warned about.
• Renewals are your job. The internal-CA cert auto-rotates; an operator cert does not — the Admin system-health card tracks the expiry of whichever cert is actually served and says so. Point the config at the live paths your ACME client maintains and restart the manager after renewal.
• Set [manager].ws_proxy_tls_port (default 5446) so the Events/alerts WebSocket has a wss endpoint — an HTTPS page cannot open a plain ws:// connection.
• Baseline security headers are always on. Every manager response carries X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy: strict-origin-when-cross-origin and Content-Security-Policy: frame-ancestors 'self'. The dashboard embeds the alarm-engine UI and the tool proxies in same-origin iframes, which these allow; embedding the manager from another origin is refused. A route that sets its own header (the proxies' [manager.security].proxy_html_csp) keeps it.
• HSTS is opt-in. [manager].hsts_max_age_s (default 0) adds a Strict-Transport-Security header to HTTPS responses. Leave it at 0 unless the plain-HTTP port is closed — HSTS preserves the port, so one HTTPS visit makes the browser rewrite http://host:5000 to https://host:5000 and fail.

This is also the prerequisite for installing the phone companion below.

Phone Companion (PWA)

The companion at /companion installs to a phone's home screen and receives alarm push notifications. It needs a trusted HTTPS origin (previous section) — service workers and web push do not work from an untrusted certificate.

[manager.companion]
push_contact = "mailto:you@example.com"   # VAPID contact browser push services see
release_check = false                     # opt-in GitHub release check (Settings toggle)
push_notify_token = ""                    # blank = fall back to the alarm-engine tokens

Setup:

1. On the phone, open https://<your-domain>:5443/companion, sign in, and use the browser's Add to Home Screen / Install prompt.
2. Enable notifications from the companion's Settings screen, then use Send test notification to confirm delivery end-to-end.
3. Alerts are pushed by the alarm engine through the manager's /api/companion/push/notify bridge. A co-located install needs no token configuration; a split install presents [alarm_engine].management_token (or push_notify_token if you set one).

Home, Alerts, Tower, Energy, and Models screens work for every role (the Tower tab shows only while the Tower assistant is enabled); control actions (model swap/pin, autopilot approval, restarts) and the Admin screen require an admin session, and every action confirms in a sheet before it runs. pywebpush must be installed in the manager venv for push delivery — the installer includes it.

Updating

Updating All Components

To update the manager, alarm engine, and any locally installed agent to the latest version, run the installer in update mode:

cd /opt/llm-systems-manager
bash tools/installer/install.sh --update

The update process:

• Detects which components are currently installed
• Compares installed files against the latest version
• Backs up any files that will change
• Syncs only the changed files
• Reloads systemd units and restarts affected services
• Refreshes the InfluxDB memory settings on hosts that run InfluxDB
• Runs the smoke test to confirm the update succeeded

Every update also re-stamps the install root's RELEASE marker, so an install that could not previously name its release — and therefore never reported an available update — self-heals on its next update.

You do not need to stop services first — the updater handles restarts.

Updating a Remote Agent

To update an agent running on a remote machine without logging into that machine:

1. Open the dashboard
2. Go to the Admin tab
3. Click Agents
4. Find the agent's row and open its overflow (⋯) menu
5. Click Update

The agent downloads and applies the latest version of itself, then restarts. To update every agent at once, use Update all under the Manage ▾ menu, which also carries a pending-update badge.

Monitoring Service Health

Checking Service Status

Check whether each service is running:

sudo systemctl status llm-systems-manager
sudo systemctl status llm-systems-alarm-engine
sudo systemctl status llm-systems-agent

A healthy service shows active (running). A failed service shows failed and usually includes the last few log lines explaining why.

Viewing Logs

Manager — log file updated continuously:

tail -f /var/log/llm-systems-manager/llm-systems-manager.log

Or via journald:

journalctl -u llm-systems-manager -f

Alarm Engine:

journalctl -u llm-systems-alarm-engine -f

Agent:

journalctl -u llm-systems-agent -f

Add --since "1 hour ago" to any journalctl command to limit output to recent entries.

Dashboard Health Page

The Admin tab in the dashboard includes a System Health card. It shows:

• Slim service rows for each component, with version and uptime
• A live data-flow diagram whose edges carry current rates, with a click on any node for the underlying numbers
• An overall Healthy / Attention / Down pill
• A restart-pending chip per service, and an alarm-engine auth chip (auth open, no token, or token mismatch)
• Release and agent-update status rows
• A restart control that always offers the alarm engine alongside the manager

The Admin tab button in the navigation bar turns red when any component reports a problem — you do not need to check manually.

Uninstalling

To remove LLM Systems Manager from a server:

bash tools/installer/install.sh --uninstall

The uninstaller:

• Stops and disables the systemd services
• Removes the installed files
• Prompts before deleting the runtime user account and InfluxDB data, so you can preserve your data if needed

Next Steps

After your deployment is up and running, refer to these documents for deeper reference:

• ARCHITECTURE.md — How the components fit together, data flow from agent to dashboard, and the multi-agent model
• COMPONENTS.md — Detailed description of each component: manager, alarm engine, and agent
• API_REFERENCE.md — Full reference for the manager and alarm engine HTTP APIs, including request/response formats