Skip to content

feat(controller): serve Prometheus fleet metrics - #180

Open
devmittal02 wants to merge 1 commit into
agentdesktop-dev:mainfrom
devmittal02:feat/controller-prometheus-metrics
Open

devmittal02 wants to merge 1 commit into
agentdesktop-dev:mainfrom
devmittal02:feat/controller-prometheus-metrics

Conversation

@devmittal02

@devmittal02 devmittal02 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #183.

What

Adds an opt-in metricsListen controller setting. When set, the controller serves GET /metrics (Prometheus text format) on that address. Unset (the default) changes nothing.

metricsListen: 0.0.0.0:9090

Why

Fleet health is only visible through the management UI today. adminListen must stay on loopback, so a scraper cannot reach it. Operators running a metrics stack (Prometheus, Grafana Alloy, and similar) have no way to alert on stale devices, failed configuration applies, or devices lagging behind the active configuration revision. The usage reporting from #160 covers tokens and cost, not fleet health.

Metrics

Metric Type
agentdesktop_build_info{version} gauge
agentdesktop_devices gauge
agentdesktop_devices_online gauge (seen in last 90s, same threshold as the UI overview)
agentdesktop_devices_connected gauge (open streams on this replica)
agentdesktop_device_oldest_seen_age_seconds gauge
agentdesktop_devices_by_version{os,agent_version} gauge
agentdesktop_devices_by_config_state{state} gauge (applied, failed, unreported)
agentdesktop_devices_config_behind gauge
agentdesktop_config_active_revision gauge
agentdesktop_inventory_mcp_servers gauge
agentdesktop_inventory_skills gauge
agentdesktop_telemetry_events_total{type} counter
agentdesktop_device_message_errors_total counter

Fleet state is read from the database at scrape time, so it is correct after a restart. The connected gauge and the two counters are in memory and start from zero on restart, as Prometheus counters do. With several replicas those are per replica and should be summed.

Privacy

Label values come from closed sets, or from the OS and agent version of enrolled devices. Hostnames, users, device IDs, MCP server names, and tool inputs are never exported. scrape_reports_fleet_state_without_identifying_labels asserts that.

Design notes

  • A separate listener instead of a route on the admin server, because the admin listener is required to be loopback and has no authentication of its own.
  • The endpoint is unauthenticated, like most exporters. The config comment and README say to expose it only to the scraper.
  • insert_telemetry now returns the stored event type so the counter uses the same value that is persisted.
  • No new dependencies. The text format is written by hand, since the output is small and fixed.

Testing

  • cargo test -p agentdesktop-core -p agentdesktop-controller passes, including a scrape test against a SQLite database and a config test for the new field.
  • cargo clippy -p agentdesktop-controller -p agentdesktop-core --all-targets -- -D warnings and cargo fmt --all --check are clean.
  • I did not build the desktop app crate, which needs its own frontend build and is not touched here.
  • Not tested: an end-to-end scrape against a running controller with real devices.

Draft while I get feedback on the metric names and on whether you would rather expose this through an OpenTelemetry exporter.

🤖 Generated with Claude Code

Add an opt-in `metricsListen` address on which the controller serves
`GET /metrics` in the Prometheus text format. The management UI listener
must stay on loopback, so metrics get their own listener that a scraper can
reach.

Fleet state (devices, online devices, versions, configuration state and
revision drift, reported MCP servers and skills) is read from the database
at scrape time. Connected streams and stored telemetry events are counted
in memory.

Labels come from closed sets or from the OS and agent version of enrolled
devices. Hostnames, users, MCP server names, and tool inputs are never
exported, and a test checks that.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: devmittal02 <94045165+devmittal02@users.noreply.github.com>
@devmittal02

Copy link
Copy Markdown
Contributor Author

@peterj could you take a look when you have a moment? This is the implementation for #183. It is still a draft, mainly to get your view on the metric names, the separate metricsListen listener, and the open questions listed in the issue. Thanks!

@devmittal02
devmittal02 marked this pull request as ready for review October 21, 2026 04:13
@peterj
peterj requested a balanced review from Copilot October 22, 2026 18:11
@peterj

peterj commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Ideally I'd like Krisztian Fekete to chime in here too -- I did ping him and he'll take a look once he's back.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Listener security and replica semantics still await maintainer review, and the connection gauge has an unresolved lifecycle defect.

1 open issue
What changed in this PR

Adds opt-in Prometheus fleet-health metrics to the controller, complementing the existing token and cost reporting.

Changes:

  • Adds a separate, unauthenticated /metrics listener configured through metricsListen.
  • Exposes database-backed fleet gauges and in-memory connection, telemetry, and error metrics.
  • Documents endpoint security and replica behavior, with configuration and metrics tests.
File Description
schema/​controller-config.md Documents the optional metrics listener.
schema/​controller-config.json Adds the metrics listener to the configuration schema.
README.md Documents metrics, access restrictions, and replica behavior.
crates/​core/​src/​config.rs Adds metricsListen and configuration tests.
crates/​controller/​src/​service.rs Instruments connections, telemetry storage, and message errors.
crates/​controller/​src/​metrics.rs Implements the exporter, counters, gauges, and tests.
crates/​controller/​src/​main.rs Wires shared metrics and starts the optional listener.
crates/​controller/​src/​lib.rs Exports the metrics module.
crates/​controller/​src/​database.rs Adds inventory totals and returns stored telemetry types.

Review effort: Balanced

let stream_guard = metrics.stream_connected();
let auth_sender = sender.clone();
tokio::spawn(async move {
let _stream_guard = stream_guard;

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose controller fleet-health metrics in Prometheus format

3 participants