LLM Systems Manager — Architecture

What Is This System?

LLM Systems Manager is a monitoring and control dashboard for an AI lab. It watches over multiple servers running AI models — tracking temperatures, memory usage, processing speeds, and whether models are responding — and displays everything in a live web dashboard accessible from a browser on the local network.

When something goes wrong, the system can automatically send an alert by email, a chat webhook, or Discord. Operators can also use the dashboard to load and unload AI models, run benchmarks, and review recent performance history.

The system is built around four core services: a central control server (the Manager), lightweight monitoring programs installed on each machine being watched (Agents), a dedicated alert-processing service (the Alarm Engine), and a time-series database that stores all the historical numbers (InfluxDB). Everything communicates over the local network using encrypted connections.

The Manager itself is more than a proxy in front of the other services — it also hosts a growing set of feature subsystems in its own right: an OpenAI-compatible inference gateway, a Model Autopilot placement engine, energy/cost accounting, an interactive Discord bot, and a standardized cross-provider GPU Report Card. See "Manager Feature Subsystems" below.

System Overview

Network Topology

  ┌─────────┐                ┌───────────────────────────────────────────────────────┐
  │         │  :5000 / 5443  │                    Manager Server                     │
  │         │───────────────►│  ┌──────────────┐  :8081   ┌──────────────┐           │
  │ Browser │                │  │   Manager    │─────────►│ Alarm Engine │           │
  │         │                │  │  (Flask)     │◄─────────│  (FastAPI)   │           │
  │         │  :5444 (WS)    │  └──────┬───────┘  proxy/  └──────┬───────┘           │
  │         │───────────────►│  WS proxy thread    push          │ :8086             │
  └─────────┘                │  (standalone)                     ▼                   │
                             │         │                 ┌────────────────┐          │
                             │         │                 │    InfluxDB    │          │
                             │         │                 │   (port 8086)  │          │
                             └─────────┼─────────────────┴────────────────┴──────────┘
                                       │
                     ┌─────────────────┴──────────────────┐
                     │  :8082 (control calls)             │
                     ▼                                    ▼
           ┌──────────────────┐               ┌──────────────────┐
           │   Agent Host A   │               │   Agent Host B   │
           │   (FastAPI)      │               │   (FastAPI)      │
           └────────┬─────────┘               └────────┬─────────┘
                    │                                  │
                    │  :8081 metric batches            │  :8081 metric batches
                    └──────────────────┐  ┌────────────┘
                                       ▼  ▼
                                  Alarm Engine (above)

  Note: all Agent ↔ Manager and Agent ↔ Alarm Engine connections use TLS. The WS proxy on
  :5444 is a separate `websockets` server running in its own daemon thread inside the Manager
  process — Cheroot (serving :5000/:5443) cannot speak WebSocket, so alert events reach the
  browser through this dedicated port instead, which bridges to the Alarm Engine's own /ws.

How Data Flows

1. Collection — On a configurable interval, an Agent reads the hardware sensors on its host machine: CPU load, RAM usage, GPU temperature and memory, fan speeds, power draw, and network activity. If an AI model server is running on that machine, the Agent also reads its current state (which model is loaded, how fast it is generating tokens, how much context is in use).

2. Buffered forwarding to the Alarm Engine — The Agent queues those readings in memory. Every 15 seconds it sends the accumulated batch to the Alarm Engine over an encrypted connection. If the Alarm Engine is temporarily unreachable, the readings are saved to disk so nothing is lost.

3. Live state push to the Manager — Simultaneously, the Agent sends the most recent snapshot directly to the Manager every 5 seconds. The Manager holds this in memory (not written to disk) so the dashboard always shows the freshest possible numbers.

4. Storage — The Alarm Engine writes each incoming batch to InfluxDB. A background task also compresses older data down to one-minute averages to keep storage manageable over longer periods.

5. Alert evaluation — As each reading arrives, the Alarm Engine checks it against every active alarm rule. If a value crosses a threshold (e.g., GPU temperature above 85 °C), the engine opens an alert, writes it to its local database, and immediately sends notifications to all configured channels (email, Discord, webhook).

6. Dashboard display — The web dashboard polls the Manager every 2 to 30 seconds (faster when a model is actively running). For live numbers it reads from the Manager's in-memory snapshot; for history charts it asks the Manager, which forwards the query to the Alarm Engine, which reads from InfluxDB. Alert events are also pushed to the dashboard in real time over a persistent WebSocket connection — a standalone bridge running inside the Manager process (its main HTTP/HTTPS server can't speak WebSocket) that pipes frames between the browser and the Alarm Engine's own WS endpoint.

Data Flow Diagram

 ┌─────────────────────────────────────────────────────────────────────┐
 │  AGENT (runs on each monitored machine)                             │
 │                                                                     │
 │  Hardware sensors → collector_loop (every 5s)                       │
 │         │                                                           │
 │         ├─► BufferedMetricClient ──► POST /api/alarm/metrics/batch  │
 │         │        (flush every 15s)          │                       │
 │         │                                   │                       │
 │         └─► POST /api/remote/provider-state (every 5s, in-memory)   │
 │                        │                    │                       │
 └────────────────────────┼────────────────────┼───────────────────────┘
                          │                    │
                          ▼                    ▼
              ┌───────────────┐   ┌────────────────────────────────────┐
              │    Manager    │   │         Alarm Engine               │
              │  (in-memory   │   │                                    │
              │   snapshot)   │   │  write to InfluxDB                 │
              │               │   │       │                            │
              │  /api/metrics │   │  evaluate alarm rules              │
              │  /api/history─┼───┤       │                            │
              │  (proxy)      │   │  if threshold crossed:             │
              └───────┬───────┘   │    ├─ open alert (SQLite)          │
                      │           │    ├─ send email / webhook /       │
                      │           │    │  Discord notification         │
                      │           │    └─ push WebSocket event         │
                      │           └──────────────────┬─────────────────┘
                      │                              │
                      ▼                              ▼
              ┌───────────────┐           ┌──────────────────┐
              │    Browser    │◄──────────│  Live WS stream  │
              │  (Dashboard)  │  alerts   │  to dashboard    │
              │               │           └──────────────────┘
              │  polls every  │
              │  2–30 seconds │
              └───────────────┘

Manager Feature Subsystems

Beyond forwarding metrics and proxying model control, the Manager backend hosts several self-contained subsystems, each in its own module:

How the manager talks to LM Studio

The manager never dials LM Studio itself: every call goes to the agent on the LM Studio host, which forwards it to the LM Studio server on that host (LMS_API_URL, port 1235 by default). Two LM Studio APIs are in use:

The native chat API takes one input turn plus a system_prompt, chains history only through a server-stored previous_response_id, has no client-declared tools (only server-run plugin integrations), and validates reasoning against the model's own allowed_options — off/on for the Qwen, Gemma and Nemotron class models, the effort levels only for models whose chat template takes one. Tower's tool loop therefore stays on the OpenAI-compatible path. The Thinking levels are enforced there by the manager: Tower counts the reasoning tokens as they stream, stops at the level's budget (1k / 2k / 6k) when no answer has started, and makes one more call with thinking off that carries the notes so far; llama.cpp gets the same budget as a server-side stop instead. An LM Studio without the native API (404 on /lms/native/models) keeps every caller on the OpenAI-compatible path; the capability list is cached for five minutes per agent.

Security Model

Service-to-service traffic — Manager ↔ Agent and Agent ↔ Alarm Engine — is encrypted with TLS. The Manager generates its own internal Certificate Authority on first startup — essentially acting as its own trusted signing authority for the private network. It uses this CA to issue certificates for each approved Agent and for the Alarm Engine, so those connections can be verified end-to-end without relying on public internet certificate authorities.

Agents must be explicitly approved by an administrator before they can send data or receive commands. Approval issues the Agent a signed certificate and a bearer token; both must be present for the Manager and Alarm Engine to accept requests from that Agent. Short-lived tokens are used for live dashboard streams (SSE) because browser APIs for those connections cannot send custom headers. Current agents additionally present a hardware fingerprint on the approval-status poll and on re-registration; records last written by an older agent keep the previous behaviour until that agent upgrades.

Two exceptions worth knowing about

The dashboard itself is served over plain HTTP on port 5000 unless you enable [manager].tls_port (5443), and the browser is what you point at it. The HTTPS port serves an internal-CA certificate by default; setting [manager].tls_cert_file/tls_key_file adds an operator-provided certificate selected by SNI for the hostnames it covers, giving browsers a publicly trusted origin (required for the PWA companion and web push) without changing what CA-pinned agents see. [manager].hsts_max_age_s (default 0) can emit Strict-Transport-Security on the TLS listener; it is off by default because HSTS preserves the port and the plain-HTTP listener shares the same hostname. Every response also carries the baseline X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy: strict-origin-when-cross-origin and Content-Security-Policy: frame-ancestors 'self' headers (a route-set header wins, so the proxies' proxy_html_csp is untouched).

The alert WebSocket bridge on [manager].ws_proxy_port (5444) is served as plain ws://, and requires a short-lived ticket on every handshake. The browser first calls GET /api/alarm-ws-ticket — an ordinary dashboard route, so it is covered by whatever login mode you have configured — and receives an HMAC-signed ticket valid for [manager.security] stream_token_ttl_s (300 seconds by default). It then connects to ws://<host>:5444/ws/alarm?ticket=…. The bridge verifies the signature and expiry before opening anything upstream; a handshake with a missing, expired, tampered, or already-used ticket is closed with code 1008 and never reaches the Alarm Engine. Each ticket carries a signed random nonce and is spent on its first accepted handshake, so a captured ticket cannot be replayed within its TTL. A fresh ticket is fetched on every connect and reconnect.

The same bridge also serves /ws/openclaw, ticketed by a separate GET /api/openclaw-ws-ticket route. Tickets are path-bound — one issued for /ws/alarm cannot be replayed against /ws/openclaw or vice versa — so the two ticket routes cannot be swapped. On the OpenClaw path the bridge forwards the browser's real Origin header upstream, so the OpenClaw gateway's own allowed-origins check still applies.

The plaintext part is deliberate: the Manager's certificate is signed by the internal CA, so terminating wss:// there would just move the trust problem into the browser, which is the thing the bridge exists to avoid. Its upstream hop to the Alarm Engine is verified TLS when Alarm Engine TLS is on. Because the ticket travels in the URL of a plaintext connection, treat port 5444 as a trusted-network service: on an untrusted network, front it with a reverse proxy holding a real CA certificate (nginx, Caddy, Traefik).

The Alarm Engine's own /ws endpoint is a separate surface with two checks on the handshake. First the Origin header is validated (CORS middleware does not apply to WebSocket handshakes), so a browser page on another origin is rejected before the connection is accepted. Second, the handshake must carry Authorization: Bearer <token> satisfying the same gate as the Alarm Engine's management routes — [alarm_engine].management_token, falling back to ingest_token; when neither is configured the stream is open, matching the fail-open convention of every other token gate on port 8081. The Manager's WS bridge presents this bearer automatically on its upstream hop, so the dashboard path is unaffected. Two consequences of enabling a token: a non-browser client (curl, a script) must now send the bearer to subscribe, and browsers can no longer dial port 8081 directly (they cannot attach an Authorization header to a WebSocket handshake) — the Manager stops advertising a direct-dial URL in that configuration, so live toasts require the bridge. The Alarm Engine's own standalone dashboard then has no live stream either: its Alerts tab still refreshes on its 15-second poll (and the Metrics sub-tab on its 60-second poll), but the overview and rules views only update on page load. A rejected handshake surfaces to non-browser clients as a plain HTTP 403 on the upgrade request. Two operational notes: the Manager reads the bearer at startup, so restart it after provisioning a token, and on a fleet with no tokens configured behaviour is unchanged — keep port 8081 on a trusted network.

Setting [manager].ws_proxy_port = 0 disables the bridge entirely. The dashboard keeps working — the Events and Admin status dots refresh on their own 30-second poll — but live alert toasts stop arriving, since those are pushed over this connection and have no polling fallback.

The web dashboard requires a username and password. Passwords are stored as one-way hashes (scrypt) and are never written in plain text anywhere. There are two access levels: Admin users can manage agents, users, alarm rules, and system configuration; Operator users can monitor the lab and control AI models but cannot change security settings or manage other users. The system also enforces automatic lockout after repeated failed login attempts to resist brute-force attacks.

A session created over the HTTPS listener is stored in a separate __Secure-session cookie with its own signing salt; plain-HTTP sessions keep the session cookie. The two are independent — a cookie minted on one scheme is not accepted on the other, and upgrading from an earlier release that served HTTPS invalidates any existing HTTPS sessions. Separately, an account still holding the shipped default password is held on a mandatory change-password form — derived server-side on every request — until it sets a new one; every other API returns 403 in the meantime.

Storage