Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ It also supports **non-Spark units**: any Linux machine with an NVIDIA GPU (e.g.
- [ComfyUI monitoring](#comfyui-monitoring)
- [Hermes Agent monitoring](#hermes-agent-monitoring)
- [Tailnet monitoring](#tailnet-monitoring)
- [Clock control](#clock-control)
- [Full changelog](./CHANGELOG.md)
- [Quick start](#quick-start)
- [Architecture](#architecture)
Expand Down Expand Up @@ -79,6 +80,7 @@ Full history: [CHANGELOG.md](./CHANGELOG.md)
| **GPU processes** | See the top GPU processes by VRAM usage directly in the GPU panel, including process name and memory allocation |
| **Spark uptime** | System uptime displayed inline on each Spark header for at-a-glance availability |
| **Power controls** | Graceful shutdown (SSH host script) and Wake-on-LAN; batch actions on Overview |
| **Clock control** | Opt-in per unit: click the Clock Cap row in the GPU/CPU panels to set a CPU (X925 / A725) or GPU cap within the discovered hardware range; apply live or persist to the boot unit. One-time helper install, no password in the UI |
| **Spark roles** | **Head** / **Worker** / **Standalone** — worker label + head link; standalone can disable LLM monitoring; optional hide workers from Overview and tabs |
| **Unified memory** | GB10 128 GB LPDDR5X pool (~273 GB/s), GPU/CPU split, bandwidth via `nvidia-smi dmon`. Non-Spark hosts show discrete **VRAM** (nvidia-smi) and system **RAM** separately |
| **Themes** | Dark, light, cool white, OLED — neutral palettes, persisted in `localStorage` |
Expand Down Expand Up @@ -233,6 +235,59 @@ Env (optional): `POLL_INTERVAL_TAILSCALE` (default `30000`), `TAILSCALE_PROBE_TI

---

## Clock control (opt-in add-on)

**Not core.** Lives under `scripts/sparkdash-clock-addon/`. sparkDash stays a
dashboard until you install the host helper, keep the sudoers drop-in, and
enable **Allow clock control** on that Spark.

The **Clock Cap** rows in the GPU and CPU panels become clickable: pick a value inside the hardware-legal range (slider + number input, presets included), then either apply it for this boot only or save it to the boot unit so it survives reboot.

### What is supported

| Capability | Details |
|------------|---------|
| **Opt-in per Spark** | `clockControlEnabled` (default **off**) in **Edit Spark**; the routes answer 403 while off |
| **Hardware-bounded** | Bounds are discovered at runtime — CPU domains from `cpuinfo_min_freq`/`cpuinfo_max_freq` (grouped by domain: X925 big cores / A725 little cores), GPU ceiling from the `nvidia-smi -q -d CLOCK` *Default Applications Clock → Graphics* value. Nothing is hardcoded |
| **Two CPU domains** | The big (X925) and little (A725) core groups can be capped independently — each core in a domain gets the same `max_perf` value |
| **Remove the cap** | A "No cap" preset (or clearing the value) restores each core's own hardware maximum (CPU) or removes the `nvidia-smi` lock with `-rgc` (GPU) |
| **Apply vs Save** | **Apply (this boot only)** writes the cap now and warns that it reverts on reboot; **Save** also rewrites the boot systemd unit (`cpu-clock-cap.service` / `gpu-clock-lock.service`) and runs `daemon-reload`, so it persists. Both CPU domains share one unit, so a single-domain Save regenerates the sibling domain's lines too (from its current live cap) rather than wiping them — byte-identical whether the host helper or the container path does the writing |
| **Presets** | "No cap" (= the domain's hardware maximum) per domain; a boot-default revert is just saving the values the boot unit currently installs |
| **Honest state** | A live-only apply is reported as such (with a "reverts on reboot" warning) and is shown in the UI until a fresh read converges — the dashboard never claims a persisted value it did not write |
| **Rate limited** | Applies go through the same destructive-action rate limiter as job cancellation |

### Privilege model (no password in the UI)

sparkDash never asks for a password or sudo prompt. Privileged work happens through a tiny root helper on the Spark host, invoked over SSH with a **scoped** passwordless-sudo rule — the same shape as the shutdown feature:

```bash
# One-time, on the Spark host (as a sudo-capable user):
sudo ./scripts/sparkdash-clock-addon/install-clock-helper.sh
```

The installer copies `scripts/sparkdash-clock-addon/sparkdash-set-clock` to `/usr/local/bin/` and adds `/etc/sudoers.d/sparkdash-clock`, validated with `visudo -c`. That rule is **this binary only**. In sudoers, listing a command with no argv means **any argv** for that binary — not “argumentless only.” The helper still parses a tight flag set. It never touches `/etc/sudoers` and grants no other sudo rights.

Availability is probed with an argumentless `sudo -n <helper>` so a check cannot apply a cap: the helper prints usage and exits 1. A sudo refusal exits 1 **without** the usage line (HTTP 423 + installer hint). The API always names the real cause.

On a **local** Spark (dashboard container running privileged on the same machine), sparkDash falls back to the container's own root path: live caps go through the container's rw `/sys` and `nvidia-smi`, and persistence goes through `nsenter` into the host mount namespace. Which path was used is reported in the API response (`source`: `helper` or `container`).

### Config fields (persisted on the Spark)

| Field | Default | Description |
|-------|---------|-------------|
| `clockControlEnabled` | `false` | Make the Clock Cap rows editable and enable the clock API routes for this unit |

Env (optional): `SPARKDASH_CLOCK_BIN` (default `/usr/local/bin/sparkdash-set-clock`), `GPU_CLOCK_MAX_MHZ` (last-resort GPU ceiling used only when `-q -d CLOCK` is unparseable; the parsed value always wins), `GPU_CLOCK_LOCK_UNIT`.

### Related API

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/api/sparks/:id/clocks/bounds` | Discovered per-domain ranges, current caps, presets, and helper availability |
| POST | `/api/sparks/:id/clocks` | Apply/persist a cap: `{ "domain": "gpu", "maxMHz": 2200, "persist": true }` — `maxMHz: null` removes the cap |

---

## Quick start

```bash
Expand Down Expand Up @@ -356,6 +411,8 @@ sparkDash/
| POST | `/api/sparks/test` | Ephemeral SSH + LLM (+ Comfy if enabled) test (no persist) |
| POST | `/api/sparks/:id/test` | Connectivity test (can save password) |
| POST | `/api/sparks/:id/comfy/cancel` | Cancel ComfyUI job by `promptId` |
| GET | `/api/sparks/:id/clocks/bounds` | Clock-cap domains with discovered hardware ranges (opt-in) |
| POST | `/api/sparks/:id/clocks` | Apply / persist a CPU or GPU clock cap (opt-in) |
| PUT | `/api/sparks/:id/password` | Save SSH password (works offline) |
| PUT | `/api/sparks/:id/disabled-devices` | Hide storage devices (hot) |
| PUT | `/api/sparks/:id/disabled-interfaces` | Hide network interfaces (hot) |
Expand Down Expand Up @@ -499,6 +556,7 @@ Choice is stored in `localStorage`.
| `npm run docker:dev` | Dev Compose |
| `npm run docker:dev:build` | Dev Compose with rebuild |
| `./deploy.sh` | Recreate container; `--build`, `--frontend` flags |
| `sudo ./scripts/sparkdash-clock-addon/install-clock-helper.sh` | One-time install of the clock helper + scoped sudoers rule on a Spark host |

---

Expand Down
30 changes: 30 additions & 0 deletions scripts/sparkdash-clock-addon/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# sparkDash clock-cap add-on

**Not core sparkDash.** Core stays read-only collectors. This package is the
privileged actuator: CPU/GPU clock caps from the device page.

## Arming (all three required)

1. Install this helper **on the Spark host** (once).
2. Sudoers drop-in present (`/etc/sudoers.d/sparkdash-clock`).
3. Edit Spark → **Allow clock control** (`clockControlEnabled`, default off).

Missing any of those: Clock Cap rows stay read-only, API 403, no slider.

```bash
# On the Spark, as a sudo-capable user:
sudo ./scripts/sparkdash-clock-addon/install-clock-helper.sh
```

## Privilege facts

`NOPASSWD: /usr/local/bin/sparkdash-set-clock` with **no argv list** means
sudo allows **that binary with any arguments**. It is not “argumentless only.”
The helper itself still accepts only `--domain` / `--max-mhz` / `--unlock` /
`--persist` | `--no-persist`.

Live apply is the default UI action. **Save (survive reboot)** writes
`cpu-clock-cap.service` / `gpu-clock-lock.service` — a second, labeled step.

Revoke: delete `/etc/sudoers.d/sparkdash-clock` and/or the helper binary,
and turn the Edit Spark flag off.
48 changes: 48 additions & 0 deletions scripts/sparkdash-clock-addon/install-clock-helper.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/sh
# install-clock-helper.sh — one-time provisioning for sparkDash clock control.
#
# Run ON the DGX Spark host (as a sudo-capable user):
# bash install-clock-helper.sh
# or from the dashboard host over SSH:
# ssh dgx@<host> 'sudo sh -s' < install-clock-helper.sh
#
# Installs:
# 1. /usr/local/bin/sparkdash-set-clock (root-owned, 0755)
# 2. /etc/sudoers.d/sparkdash-clock (scoped NOPASSWD for that binary only)
#
# The sudoers drop-in is validated with visudo -c BEFORE installation.
# A command listed with no argv matches ANY argv for that binary (sudoers
# semantics). Scope is “this helper only”, not “argumentless only”.
set -eu

HELLO_SRC="$(dirname "$0")/sparkdash-set-clock"
HELLO_DST=/usr/local/bin/sparkdash-set-clock
SUDOERS_DST=/etc/sudoers.d/sparkdash-clock

[ "$(id -u)" -eq 0 ] || { echo "run as root (sudo)" >&2; exit 1; }

# 1. Helper binary.
if [ -f "$HELLO_SRC" ]; then
install -m 0755 "$HELLO_SRC" "$HELLO_DST"
else
echo "helper source not found next to installer ($HELLO_SRC); aborting" >&2
exit 1
fi

# 2. Scoped sudoers drop-in: this binary only (any argv). Not blanket sudo.
TMP=$(mktemp /tmp/sparkdash-clock.XXXXXX)
trap 'rm -f "$TMP"' EXIT
cat > "$TMP" <<SUDOERS
# Managed by sparkDash install-clock-helper.sh — scoped clock-control grant.
# Passwordless sudo for THIS binary only (any argv — sudoers default).
# Remove this file to revoke clock control.
%sudo ALL=(root) NOPASSWD: $HELLO_DST
SUDOERS

# Validate before installing; never leave a broken sudoers file behind.
visudo -c -f "$TMP" >/dev/null
install -m 0440 "$TMP" "$SUDOERS_DST"

echo "installed $HELLO_DST and $SUDOERS_DST"
echo "verify from the dashboard host: ssh <user>@<host> 'sudo -n $HELLO_DST'"
echo " expected output: the helper usage line (sudo allowed the argumentless run)"
Loading