Each phase ends with green tests and updated docs. Phase N+1 doesn't start while phase N has failing mandatory tests.
Goal: vexor start app.js --name api -i 1, stop, restart, delete, list --json, logs, and vexor daemon with crash restart + backoff. No PM2 config yet.
| Step | Deliverable | Acceptance |
|---|---|---|
| 2.1 ✅ | Workspace skeleton (crates/vexor bin only; other crates are added in the step that first needs them), rust-toolchain.toml (1.99.1), CI: fmt, clippy -D warnings, test on linux x64/arm64 + macOS, cargo-deny; actions pinned by commit SHA |
all checks green locally; CI not yet run (no GitHub remote) |
| 2.2 ✅ | vexor-core: state machine + restart policy (pure) |
property tests: no transition sequence yields unbounded restarts within the budget window; every event in every state is handled |
| 2.3 ✅ | vexor shim: spawn (argv exec, own process group, env/cwd), pipes, NODE_CHANNEL_FD, identity capture, restart loop |
integration: env/args/cwd round-trip; metachar args not shell-split |
| 2.4 ✅ | Stop: SIGTERM → kill_timeout → SIGKILL on pgid; shim subreaper orphan reap | test app ignoring SIGTERM is killed at timeout; grandchild cleaned up |
| 2.4b ✅ | Shim ↔ daemon protocol, buffered events, reconnect, daemon reconcile | zero-downtime proof: HTTP app under constant load (wrk2), kill -9 daemon × 100 in a loop → 0 failed requests, 0 app restarts, log lines contiguous; app crash while daemon is down → still restarted by shim |
| 2.5 ✅ | CLI ↔ daemon socket (daemon.sock, 0600, peer UID check, JSON lines), single-daemon flock, state.json desired state (pulled forward from 2.6), CLI start/stop/restart/delete/list with daemon auto-start |
stop survives daemon kill -9; 8 concurrent identical starts → 1 app; corrupt state refused and kept; mutant without persisted stop fails tests. Wrong-UID rejection is implemented but not tested (needs a second user) |
| 2.6 (mostly done) | State store (atomic state.json) + resurrect on daemon start (policy R1/replace) |
kill -9 daemon in a loop × 100 → no duplicate app instances, state consistent |
| 2.6b ✅ | events.db (SQLite, WAL): writer thread with bounded queue, every shim event stored with its resume cursor in one transaction, retention (30 days / 1M rows), corrupt-file recovery, vexor events |
daemon kill -9 mid crash loop → exactly max_restarts restarts, then errored; stored history contiguous by sequence number including events from the outage; corrupt db kept as events.db.corrupt-<ts>, app not restarted, warning shown. Crash budgets live in the shim (R2), so they survive daemon restarts without the database |
| 2.7 ✅ | Log capture + size rotation + drop policy (writer thread per log file, bounded queue, drop/block overflow, reopen after external move, write errors reported once) |
1 GB flood: ≤ 3 files of 1 MiB on disk, shim peak RSS ~6 MB, app exit 0 (macOS + Linux); block mode keeps all 200,000 lines across rotation; drop mode accounts for every byte |
| 2.8 ✅ | Legacy runtime checks in CI (legacy-node job): static Linux x64 musl binary, tests/legacy/node-smoke.sh in node:<v>-alpine for Node 8, 10, 12, 14, 16, 18, 20, 22, 24 |
6 checks per version (ready via IPC, natural exit with IPC open, graceful SIGTERM, crash restart + crash-loop limit, args/env/cwd/instances, readiness timeout); all pass locally on linux-arm64; CI result below |
Done:
vexordlauncher (ES5, no dependencies, no install scripts) + platform packagesvexord-{linux,darwin}-{x64,arm64}withos/cpufiltering;npm/scripts/stage.jsstamps the Cargo version and checks the package list against the launcher.install.sh: direct binary install, mandatory SHA-256 check, no sudo.- CI on every push: static linux-x64 binary → launcher checks on Node 10–24 (npm 6, 8, 10, 11) and install.sh checksum checks; macOS launcher check.
release.ymlonv*.*.*tags: 4 targets,SHA256SUMS, CycloneDX SBOM, build-provenance attestations, GitHub Release (prerelease), npm publish with provenance only when theNPM_TOKENsecret is set.
Open:
- Reserve the npm names (needs
npm login), addNPM_TOKEN, cutv0.1.0. The release workflow has not run yet; its first run is the real test. - Static musl builds need a C cross compiler for the bundled SQLite (
musl-gcc; cargo-zigbuild if cross-compiling); release binary is ~3.7 MB. - pnpm/yarn and
--ignore-scriptsinstall checks. - darwin-x64 is cross-built on an arm64 runner and not run at all yet (the release smoke test skips it); it needs an Intel runner before it can be advertised.
4.1 ✅ Ecosystem files (crates/vexor-pm2): ES5 loader run on the app's own Node (.js/.cjs/.mjs; .json without Node), every PM2 field classified (supported / partial / ignored / fatal; unknown fields reported), vexor start <ecosystem> [--only] [--env NAME] [--allow-unsupported], vexor check, autostart: false. Tested: 21 translator/loader unit tests, 4 CLI tests, loading an ES5 config on Node 8–24.
4.2 ✅ Cluster mode: ES5 cluster primary (crates/vexor-shim/src/cluster.js) on the app's Node holds the shared listening sockets; the shim keeps one state machine per worker and drives workers through it (spawn / signal / events). Primary death → its group is killed and every worker restarts under its own policy. exec_mode: cluster in ecosystem files; vexor start app.js -i N is cluster mode (as in PM2), --fork opts out. Tested: workers share a port and are replaced individually (others keep serving), primary death restarts all, readiness per worker, graceful stop; cluster check on Node 8–24.
4.3 ✅ Health-gated vexor reload: reload is a phase of the core state machine (replacement spawned next to the current process; promoted when healthy; abandoned, keeping the current process, on exit, spawn failure or timeout; interrupted by stop/restart/crash). Cluster mode is start-first (healthy = ready with wait_ready, else listening); fork mode is stop-first unless reload_strategy: start_first (healthy = ready, else up for min_uptime). Old cluster workers retire gracefully (worker.disconnect(), then SIGTERM, SIGKILL after kill_timeout). Tested: 3 reloads of a 1-instance cluster app under load with 0 failed requests and exactly one worker left; failed reload keeps the old process with 0 failed requests and is reported within a second; fork default = restart; fork start-first on a fixed port fails safely (EADDRINUSE); 6 core unit tests + property tests; cluster reload on Node 8–24.
4.4 ✅ vexor logs [name] [--lines N] [--follow] [--out|--err] [--raw] (tail reads backwards in chunks; rotated .1 fills in short files; follow survives rotation and truncation) and vexor describe <name> [--json] [--show-env] (config, live status with uptime, policy in words, log settings, recent events; env values hidden by default). vexor list gains an UPTIME column; event history is ordered by event time. Tested: unit tests for tail/follow/rotation/truncation; CLI tests for describe redaction and JSON, per-stream tails, and following 1,500+ lines across 3+ rotations with no gap or duplicate. A mutant follower that ignores rotation fails both.
4.5 ✅ vexor save / resurrect (explicit dump.json snapshot, 0600; the daemon already restores apps from its own state at every start), vexor startup [systemd|launchd] / unstartup (systemd system unit vexor-<user>.service with KillMode=process, so stopping the service never stops apps; per-user LaunchAgent with AbandonProcessGroup; non-root on systemd prints the exact sudo command). Tested: unit tests for the generated files (quoting, escaping, plutil -lint on macOS); CLI tests for save/resurrect (desired state kept, idempotent) and startup --print; a real systemd run in a container: verify, install, stop keeps the app, start reconnects to the same app, killed daemon restarted, unstartup leaves apps running. launchd install not run live (it would add a login item on the dev machine).
4.6 ✅ PM2 conformance harness (tests/conformance): 11 scenarios run under PM2 7.1.0 and Vexor in isolated homes, comparing what a probe app saw (runs, argv, cwd, env, instance numbers, stop signal) and what each tool reported. It found two real incompatibilities, fixed for 0.1.1: max_restarts off by one (and 0 = PM2 default 16), and SIGTERM instead of SIGINT on stop (new kill_signal setting). Result: 10 identical, 1 intentional difference (wait_ready without a ready signal is a failed start, PM2 says online). Report: docs/conformance.md.
dry-run, backup, journaled apply, rollback (docs/migration.md).
5.1 ✅ vexor migrate --from-pm2 --dry-run: reads dump.pm2 (no PM2 command, no Vexor state, no daemon), groups per-instance entries into apps, drops PM2 bookkeeping and defaults, keeps PM2's log files and the saved environment, and runs the result through the ecosystem translator. Report per app: command line, cwd, logs, env variable names (never values), findings, downtime class (none / brief / blocked); global warnings for a world-readable dump, a PM2 boot service and captured shell environment. --json; exit 4 when some apps are blocked. Tested: 9 unit tests on a real PM2 7.1.0 dump (fixture), 2 CLI tests (redaction, no daemon or state created, exit codes); end to end against PM2 7.1.0 in a container (cluster, fork, interpreter: none, stopped app): all apps planned, dump unchanged, PM2's apps still running.
5.2 ✅ vexor migrate --from-pm2 --backup: copies PM2's state files and boot service, the plan (plan.json, redacted) and a manifest with SHA-256 hashes to <home>/migrations/<id>/ (0700 dirs, 0600 files), via a temporary directory renamed when complete; never overwrites; a --dump outside the PM2 home is backed up alone. Tested: CLI test (two backups get distinct ids, permissions, byte-identical dump copy, manifest hashes, redacted plan, no leftovers, PM2 files unchanged, no daemon, nothing kept when the dump is missing); cargo deny with the new sha2 dependency.
5.3 ✅ vexor migrate --from-pm2 --apply: applies the newest (or --backup-id) backup's plan only if PM2's dump is unchanged since (SHA-256) and the plan intact; confirmation (--yes); per app pm2 stop → Vexor start → health check (all instances online, no restart, same live processes for min_uptime), else vexor delete + pm2 start and stop; blocked apps stay on PM2; refuses names Vexor already has; journal journal.jsonl in the backup (fsync per step) for resume; lock per backup. Tested: 3 CLI tests with a recording stand-in for pm2 (happy path + journal + idempotent rerun; unhealthy app put back under PM2; no backup / no --yes / changed dump / existing name / failing pm2 stop), journal unit test; end to end against PM2 7.1.0 (3 apps in 3 s, 2 of 456 probe requests failed during a ~240 ms cutover gap).
5.4 ✅ vexor migrate --rollback [ID] (newest first: delete from Vexor, pm2 start and wait for online in pm2 jlist; per-app confirmation or --yes; journaled; repeatable; --only) and vexor migrate --status [ID] [--json]. Tested: rollback-scope unit test (order, never-begun and already-rolled-back apps, re-migration), CLI test (status before/after, refused without --yes, rollback, second rollback no-op, migrate again); end to end against PM2 7.1.0 (apply then rollback, PM2 back to its original states, 3 of 273 probe requests failed). Found while testing: on a heavily loaded macOS host an instantly crashing app's crash can be reported after a 1 s health window (no exit is lost; the report lags), documented in docs/migration.md.
Phase 5 done: dry run, backup, apply and rollback.
HTTP/TCP health checks, readiness-gated reload, max_memory_restart, watch, cron_restart, metrics endpoint, doctor, fault-injection suite ✅ (docs/faults.md), uid/gid review, Windows feasibility study.
Phase 6 resilience ✅ (0.13): adopting a dead shim's instances, in-place shim upgrade (fork mode), and the fault-injection suite. Details in docs/architecture.md §7 and docs/faults.md. Tests: daemon tests for adoption (fork: same PIDs, one new shim, adopted process restarted on exit, restart restores output capture; cluster: leftovers stopped, app started again, nothing listening twice), tests/upgrade.rs (same shim PID and incarnation, same app PIDs, output with no line lost or repeated, event replay continuous, inherited child's exit signal seen; HTTP app with Node IPC under load: 0 failed requests, one process, IPC still connected; missing program skipped; failed exec resumes in the same program), unit tests (survivor assignment, handover file and kept descriptors, version comparison, Instance::inherited, Proc waits), and fault_* daemon tests (hung daemon, hung shim, corrupted event database, unwritable log file). Found and fixed on the way: CLI questions to a hung daemon waited 5 minutes; now 10 s with what to do.
First the three settings that block migrations (and vexor start of ecosystem files) today:
6.1 ✅ max_memory_restart: the shim reads each instance's resident memory (Linux /proc/<pid>/statm, macOS proc_pidinfo) every 5 s (memory_check_interval_ms) while it is online and not reloading, after checking the PID is still the process it started; above the limit, a graceful restart (counted as a restart, not a crash) and a memory_restart event with the measured size. Ecosystem max_memory_restart (bytes or K/M/G) is now supported; vexor start --max-memory-restart; shown in describe and events. Tested: RSS reader unit tests, size parsing and translation tests, shim tests (restarts over the limit repeatedly without a crash-loop error; leaves an app under the limit alone; cluster mode restarts only the worker over the limit).
6.2 ✅ cron_restart: the shim schedules restarts of every instance with croner (the Rust port of the library PM2 7 uses; 5 or 6 fields, local time), re-reading the clock at least every 30 s so clock changes delay a run by at most that; a job that exited (autorestart: false) runs again, and a standalone shim stays up for it. Invalid patterns are rejected by vexor check/start/the shim. vexor start --cron-restart; describe, events. Checked against PM2 7.1.0: running app restarted, exited job re-run, and PM2 also restarts an app stopped with pm2 stop (Vexor keeps it stopped: documented difference). Tested: pattern parsing, translation/validation, shim tests (restart on schedule across 2 instances, job re-run, invalid pattern rejected), CLI test through the daemon.
6.3 ✅ watch / ignore_watch / watch_delay: the shim watches the paths (inotify through rustix on Linux, adding directories as they appear; polling every second elsewhere) and restarts every instance once per burst of changes (at least 100 ms, or watch_delay); PM2's default ignore list, glob patterns, the app's own logs always ignored, symlinks not followed. vexor start --watch; describe, events. No new license: notify was considered and dropped because it is CC0-1.0 (and inotify ISC), which deny.toml doesn't allow; globset is MIT. Tested: matcher unit tests (PM2 default, globs, own logs, invalid patterns), translation tests, shim tests on macOS and Linux (ignored changes don't restart; a change in a directory created after the start does; one restart per instance per change; watch_delay), CLI test.
6.4 ✅ HTTP/TCP health checks (health_check, a Vexor extension): minimal HTTP/1.1 client on loopback only; fork mode per instance ({instance} substitution, readiness on first pass, unhealthy/restart thresholds through the core state machine, stale results for a replaced process ignored); cluster mode app-wide, reloading after restart_after failures once armed by a pass, and stopping when the app stops. vexor start --health-check; describe, events. Tested: target parsing, probe (statuses, junk, refused, silent server), translation, shim tests on macOS and Linux (readiness gated, failing instance restarted while the other is untouched, never-passing check fails the start, cluster reload exactly once), CLI test. Found while testing: the cluster check kept running during shutdown and triggered a second reload (fixed).
The three settings that blocked migrations are supported now; vexor doctor (0.10): home permissions and network filesystem (statfs), daemon reachability and version, warnings the daemon reports (event database, alerts.json, settings.json, delivery), apps without a connected supervisor, crash loops, missing executables, free space for logs (statvfs), open-file limit (getrlimit), start at boot; --json; exit 1 on a problem. Since 0.10: uid/gid/username (resolved by the shim with getpwnam_r/getgrnam_r, numbers accepted, the user's primary group by default; root's supplementary groups are dropped; refused unless Vexor runs as root). Since 0.8: increment_var (fork mode sets it per instance, cluster mode through the primary) and shutdown_with_message (a graceful stop, restart or reload sends 'shutdown' over the Node IPC channel, to the worker through the primary in cluster mode, instead of the stop signal; SIGKILL still follows after kill_timeout; fatal on a non-Node app).
Goal: an advanced, self-hosted pm2 monit. Admins get warning, danger, and
recovered ("green") alerts, plus a timeline that helps with root-cause
analysis. PM2 offers alerting only in the paid PM2 Plus. Everything runs
locally; no SaaS.
| Step | Deliverable | Acceptance |
|---|---|---|
| 6b.1 ✅ | Per-process metrics: CPU, RSS, open FDs, restarts, uptime, health results. Sampled by the shim (default every 5 s), stored by the daemon | Sampling 100 processes adds < 1% of one core; counted as supervisor overhead in benchmark B3 |
| 6b.2 ✅ | Time-series storage in events.db (SQLite) with rollups: raw 5 s for 24 h, 1 min for 7 days, 1 h for 90 days; size cap |
100 processes × 30 days stays under a documented size; a 24 h chart query for one process returns in < 100 ms |
| 6b.3 ✅ | Alert rules (vexor.alerts.yaml): thresholds with warning / danger levels and a for: duration; built-in rules for crash loop, restart rate, memory, CPU, unhealthy, log disk usage |
Fires once per incident, sends "recovered" when it clears; dedup, cooldown and flap damping covered by tests |
| 6b.4 ✅ | Channels: email (SMTP with STARTTLS) and webhook (JSON, HMAC-signed); Slack after that | Channel credentials are never logged or shown in describe/doctor (security.md T3); a failed delivery is retried with backoff and reported |
| 6b.5 ✅ | Per-route attribution from the app's existing structured JSON logs (pino/winston: route, status, duration). Field mapping is configurable; apps need no code changes | Error rate and p95 latency per route on the timeline; logs that aren't JSON are reported as "no route data", never guessed |
| 6b.6 ✅ (terminal, not web) | Local dashboard: one timeline per process (CPU/memory, restarts, deploys/reloads, health, error bursts per route, alerts) | Binds 127.0.0.1 by default and requires login; non-loopback needs explicit config (security.md T11) |
| 6b.7 ✅ (Prometheus; OTLP dropped) | Export: Prometheus /metrics, and OpenTelemetry (OTLP) ingestion for apps that already emit traces |
Works with an existing Grafana/Prometheus setup |
6b.1 ✅ Per-process metrics: every metrics_interval_ms (default 5 s, 0 = off) each shim measures its running instances (CPU over the interval in % of one core, RSS, open files; Linux /proc/<pid>/stat, statm, fd/; macOS proc_pidinfo task info and fd list), identity-checked like the memory check, and attaches the latest health result. The latest sample goes to the daemon as a metrics message (shim protocol 2): not sequenced, buffered, logged or stored, each replacing the last; the daemon keeps it in memory and drops it on disconnect. vexor ls takes CPU and memory from it (CPU is now the current load, not ps's lifetime average on Linux) and falls back to ps for a process without a full sample yet; vexor describe gains a Resources section (CPU, memory, open files, health and since when, sample age); list --json and describe --json carry metrics. Restarts and uptime were already in the state snapshot. Only the instance leader is measured (as PM2), not processes it spawned. Shims accept protocols 1 and 2, and a daemon refused by an older shim (one that survived an upgrade) says hello again with 1 and gets no metrics from it, so upgrades and downgrades keep working. Measured overhead, 100 apps of one process each, 60 s windows, supervisor CPU (shims + daemon) with sampling off vs every 5 s: macOS arm64 0.22–0.35% → 0.88–1.02% of one core (+0.67%, twice); Linux arm64 container (colima, 4 CPUs) 2.0–2.35% → 2.45–2.55% (+0.1 to +0.55%). Reading one sample costs ~3 µs; most of the cost is waking the shim and the daemon for each message. Tested: CPU % math, own CPU and RSS, exact open-file counts of child processes, /proc stat parsing, health tracking per instance and app-wide (reset when a new process starts), metrics accepting fields added later; daemon tests: samples with CPU, memory, open files and a passing TCP health check, the restarted process's sample after a crash, and a stand-in protocol-1 shim followed after one refusal and without metrics.
6b.2 ✅ Resource history: the daemon stores every sample in events.db (schema 2, migrated forward from 1): metrics_raw (one row per sample) and metrics_rollup (1-minute and 1-hour buckets with sums, maxima and a count, updated as each sample is written, so no separate rollup pass). Compact encoding: integer app ids (metric_apps), CPU in tenths of a percent, memory in KiB, rollup times in seconds; raw samples keyed by the millisecond (schema 3: with second keys, a replacement process sampled in the same second overwrote the dead one's last sample; found by the arm64 CI runner). Kept: raw 24 h, minutes 7 days, hours 90 days; above 3M raw rows, raw samples are kept proportionally less than a day. Pruning goes through each (app, instance) key prefix, no table scan. Samples wait up to 2 s in the writer and go into one transaction (and always before a prune). vexor metrics <app> [--since 1h] [--json]: raw points for short ranges, else minutes or hours (about 1,500 points at most); per instance a CPU and a memory sparkline sized to the sample spacing, averages, peaks, most open files and restarts in the window (PID changes). Measured: 100 processes with full retention = 2.95M rows, 85 MiB (about 0.85 MiB per process; first schema was 206 MiB); a 24 h query for one process (1,440 minute points) well under the 100 ms target (asserted in a test). Overhead with storage, 100 apps, 60 s, macOS arm64: 0.12% off → 0.93% on (+0.82% of one core), daemon 0.25% of it (first, one transaction per message: 0.55%). Tested: rollup sums/maxima/counts with a missing CPU value and a restart, resolution choice by range and retention, schema-1 database migrated with its data kept, expired rows pruned per resolution, 24 h query time; daemon test: vexor metrics --json shows the process before and after a crash, and the text view counts the restart; size benchmark: cargo test --release -p vexor-daemon metrics_storage -- --ignored --nocapture.
6b.3 ✅ Alert rules (docs/alerts.md): a pure incident engine in the daemon (crates/vexor-daemon/src/alerts.rs), evaluated every 5 s from what the daemon already holds: state snapshots (crash loop), restart counters over a sliding hour (restarts; counter resets from a new shim handled), health results and unhealthy states, the latest resource samples (CPU, memory, open files; ignored when older than 3 sampling intervals), and log sizes on disk refreshed every minute off the runtime thread. One incident per (rule, app, instance): opens or escalates only after its level held for for, recovers after staying clear as long, notifies once per level; 3 openings within an hour make it flapping (one notice, then recorded quietly until clear for the cooldown, then one recovered). Every transition is an alert event in events.db and a daemon log line, with notify for 6b.4. Built-in rules work without configuration (crash loop, restarts 3/10 per hour, unhealthy for 30 s, log disk 1/5 GiB); CPU, memory and open files are off until given thresholds. Config: $VEXOR_HOME/alerts.json (JSON rather than the planned YAML: no new dependency, and ecosystem files are already JSON/JS), re-read on change; an invalid file keeps the previous rules and every response warns. vexor alerts [--json]: open incidents, the last 7 days of alerts and the rules in words; vexor events shows alerts too. Tested: 6 engine unit tests (for-duration, short spikes, ramp across levels, escalation and recovery once each, flapping with quiet changes and the cooldown recovery, built-in defaults, config parsing and validation, restart rate over a sliding hour with counter resets), CLI formatting tests, and a daemon test: a crash-looping app opens a danger incident, a broken alerts.json warns while the rules stay, fixing the app and restarting it recovers the incident.
6b.4 ✅ Alert delivery (docs/alerts.md, crates/vexor-daemon/src/notify.rs): a delivery thread with blocking I/O and 15 s timeouts, fed by the alert loop for every change marked notify, filtered per channel by level; retries after 5 s, 30 s, 2 min, 10 min, 30 min, then an alert_delivery_failed event; each channel's last failure shows as a warning on every command. Email: a small SMTP client (EHLO, STARTTLS required when configured or implicit TLS, AUTH PLAIN or LOGIN, dot-stuffing, address validation against header and command injection). Webhooks: HTTP/1.1 POST, JSON or Slack format, HMAC-SHA256 over <timestamp>.<body> in X-Vexor-Signature. TLS: rustls with the ring provider and the system's trusted CAs (no OpenSSL, static musl builds unchanged), optional ca_file, accept_invalid_certs for testing (warned on every command). Secrets inline or from files, redacted in serialization, debug output and errors; a group/world-readable alerts.json with channels is warned about. vexor alerts test delivers a test alert to every channel and reports each result. Cost: release binary 3.39 → 4.18 MB; new licenses ISC (ring, rustls-webpki, untrusted) and BSD-3-Clause (subtle) allowed in deny.toml. Tested: base64 (RFC 4648), HMAC-SHA256 (RFC 4231 cases 1, 2, 6), URL parsing, dot-stuffing, redaction, channel config validation (including a CRLF injection attempt), SMTP conversations against a scripted server (AUTH PLAIN; fallback to LOGIN; missing STARTTLS refused without leaking the password), webhook signature and non-2xx handling, retry scheduling with level filtering; daemon test: a crash loop delivers a signed webhook and the secret appears only as <redacted>. End to end against Stalwart 0.16.27 in Docker (submission on 587, STARTTLS, AUTH, self-signed certificate with accept_invalid_certs): vexor alerts test and a real crash-loop incident both arrived in the recipient's mailbox (read back over IMAPS).
6b.5 ✅ Per-route stats (docs/routes.md, crates/vexor-shim/src/routes.rs): the log writer threads hand each output chunk to a tap (off the shim's runtime thread; chunks are tagged by pipe so lines from different processes never join). A line is a request when it is a JSON object with a route or URL, a status 100–599 and a duration, under configured dotted paths or the usual names (pino-http, express-winston, morgan JSON, OpenTelemetry); Fastify's two lines are joined by reqId per process (output pipe, plus pino's pid for cluster workers sharing the primary's pipe). URLs are normalized (no scheme/host/query, id-like segments → :id, control characters replaced, 200 bytes). Per route and window: count, 4xx, 5xx, total and longest duration, and a sparse log-scale histogram (buckets 2^¼ wide from 0.1 ms, 100 buckets, quantiles within ~9%; vexor_proto::hist). The shim sends one routes message per metrics interval (shim protocol 4; the daemon steps down to older shims as before, and they send none). The daemon stores windows straight into minute and hour buckets in routes_rollup (schema 4: route_names + rollups with histograms as varint blobs; kept 7 days / 90 days; at most 1,000 routes per app, then (other)), keeps line counts from the connected shim to explain missing data ("aren't JSON", "no request fields"), and counts requests and 5xx per route for Prometheus. Plain-text lines are only counted, never guessed at. vexor routes <app> [--since] [--all] [--json]: per route requests, 4xx/5xx share, p50/p95/max, a requests sparkline. Ecosystem key routes (false or a field mapping). Cost, Apple M5 release build: pino-http request line 2.3 µs, other JSON line 0.2 µs (pre-filtered by key names before parsing), plain text 26 ns: 1,000 req/s ≈ 0.25% of one core. Tested: URL normalization, pino-http/express-winston/morgan lines, Fastify joining across two processes with equal ids (separate pipes, and one shared pipe told apart by pid), lines split across chunks and overlong lines, non-requests not counted, configured paths/units and the route cap; histogram buckets, quantiles, encoding and merging; storage: windows merged per minute with quantiles and series, per-app route limit, pruning per resolution; daemon test: a shell app writing pino-http lines shows /users/:id with the right 5xx share and p95, a plain-text app is reported as not JSON; Prometheus counters rendered.
6b.6 ✅ Timeline, in the terminal (decision 2026-10-25: no web dashboard, so no HTTP server, login or T11 exposure): vexor timeline <app> [--since 1h] [--json] draws one app's history on one time axis, one row each: CPU and memory (per column, the sum over instances of each instance's peak), requests and 5xx (route buckets spread over the columns they cover, the current bucket only up to now; 5xx columns without errors stay blank so one error doesn't look like an outage), p95 (worst per column), and events: ✗ crash or crash loop, ▲ alert, ! health failing, ↻ reload, ↺ restart (memory, watch, cron, or any process start, unlisted), ● started or scaled, the most serious per column. Columns follow the resource samples (5 s when raw, at most 48); the last 10 notable events are listed under the chart. Data: the existing metrics, routes (new totals series: all routes per bucket, with merged histograms for p95) and events requests; nothing new is stored. Tested: per-app sums across instances, bucket spreading, markers and priorities, axis labels; daemon test: the timeline of an app writing JSON request logs shows every row and its start, --json has the request series.
6b.7 Prometheus ✅ (docs/prometheus.md): settings.json {"prometheus": {"listen": "127.0.0.1:9615"}} turns on GET /metrics (text format 0.0.4) served by the daemon; re-read every 5 s and rebound on change, a failed bind retried; any non-loopback address warned about on every command (security.md T11); only GET /metrics answers, with a 5 s / 8 KiB request limit. Families: per instance up, state (label), restarts (counter), uptime, CPU, memory, open files, health; open alerts by rule and level; build info. Label values escaped. Tested: rendering (escaping, missing samples omitted, alert counts), settings validation, loopback check; daemon test: a 2-instance app shows up on the endpoint, other paths 404. OTLP ingestion: dropped (decision 2026-10-25). Apps that use the OpenTelemetry SDK already send traces to a collector or backend that handles them far better; ingesting them would make the supervisor a small APM backend (an HTTP/gRPC server, protobuf, trace storage: size, dependencies, attack surface), per-route stats already come from JSON request logs (6b.5), and Prometheus export covers Grafana setups. Reconsider only on request.
PM2 parity (0.8–0.9): vexor flush, signal (send-signal), kill, monit (live view, single frame without a terminal) and scale (shim protocol 3: per-instance stop channels, instance_removed events, an instance scaled back up while stopping is restarted; the count is persisted; a start that differs only in instance count scales to it instead of conflicting). Tested: CLI tests for signal/flush/kill and monit; daemon tests for scale (PIDs kept on scale-up, only instance 0 left after scaling to 1, count survives a daemon restart, re-starting the original spec scales back without restarting instance 0, 0 refused) and protocol step-down (a protocol-1 shim is reached after refusals at 3 and 2; scaling it is refused with advice).
Decisions and limits:
- Storage: SQLite, not DuckDB, inside the supervisor. The workload is frequent small, crash-safe writes, plus range queries per process. DuckDB would add tens of MB of C++ to the binary, defaults to using up to 80% of RAM, and complicates the static builds. Heavy analytics go through export (Parquet, Prometheus, OTLP), or an optional separate tool that queries the data with DuckDB.
- Vexor sees processes, not requests. Without the app's structured logs or OpenTelemetry data, it can't say which API endpoint is slow or failing, and it says so instead of guessing.
- "Root cause" means a correlated timeline (for example: deploy at 14:02 →
memory climb → crash loop → errors on
/api/orders). Vexor does not claim automatic root-cause detection. - Alerting and the dashboard run in the daemon, not the shims, so app supervision never depends on them. Both can be turned off.
Daemon-crash recoveryDecided: per-app shim (R2), so daemon crash or upgrade causes zero app downtime and zero log loss (architecture.md §7, which also lists where zero downtime is impossible).State formatDecided:state.jsonfor desired state + SQLiteevents.dbfor history, restart budgets and migration journal (§8).- Cluster mode via ES5 cluster-shim on app's Node (§6).
LicenseDecided:MIT OR Apache-2.0(LICENSE-MIT, LICENSE-APACHE). PM2 is AGPL-3.0, so Vexor is a clean-room implementation built from docs and observed behaviour; no PM2 code is copied or translated.npm nameDecided:vexord(npm i -g vexord), commandvexor, platform packagesvexord-<os>-<arch>. Names to reserve before the public release.Rust toolchainDone: rustup, stable 1.99.1, minimal profile + rustfmt + clippy, pinned inrust-toolchain.toml. Linux musl cross targets (cargo-zigbuild) are added in Phase 3.