Skip to content

fix(service): use the unit's installed scope and report restart failures - #2092

Open
BradPita wants to merge 2 commits into
TencentCloud:developfrom
BradPita:fix/service-scope-from-installed-unit
Open

BradPita wants to merge 2 commits into
TencentCloud:developfrom
BradPita:fix/service-scope-from-installed-unit

Conversation

@BradPita

Copy link
Copy Markdown

Problem

When a deployment's unit lives in the other scope than the euid heuristic guesses, scope inference resolved the wrong one. Concretely: a SYSTEM unit (/etc/systemd/system/octop.service, WantedBy=multi-user.target) whose service runs as User=<owner> was managed with systemctl --user …. On a headless host there is no user D-Bus/session, so the command failed with:

Failed to connect to bus: No medium found

POST /api/update/restart had already answered {"status": "restarting"}, and the background task's exception only reached the server log. The dashboard therefore kept its dialog spinning until its own 2-minute poll budget expired and then reported a generic timeout — the operator had no way to see what actually failed.

Changes

1. Scope inference follows the installed unit — octop/infra/setup/service.py

  • New _unit_scope_candidates() / _installed_scope(); build_runtime() adopts the scope whose standard unit file exists (provisional scope first) and keeps the euid heuristic when neither exists, so fresh installs and an explicit --scope are unchanged.
  • is_service_installed() probes both standard locations through the same helper, so a legitimately installed SYSTEM unit is recognised instead of reporting octop system service is not installed (expected unit for mode=systemd).

2. Restart failures are reported instead of hidden — octop/api/routers/update.py + dashboard

  • POST /restart bounces the service in a worker thread and waits a short window (3 s) for an immediate failure before answering restarting.
  • Privilege failures map to a new RESTART_PRIVILEGE_REQUIRED (409) whose localized message tells the operator what to do; other early failures return the raw reason.
  • The dashboard shows the server's message for a failed restart instead of the generic toast, so users no longer have to inspect logs to understand a failure.

Test plan

  • pytest tests/unit/api/test_update_router.py tests/unit/i18n tests/unit/infra/setup/test_service.py tests/unit/cli/test_service_cmd.py
    • new cases: scope follows the installed unit (plus the regression), the dual-location probe, restart fail-fast, privilege-vs-other error mapping, and a slow restart that must not hold the request open
    • the existing "not installed" tests were made hermetic (they previously depended on whether the host running the suite has an octop unit)
  • ruff format --check and ruff check on the touched files

Notes

  • Default install paths are unaffected: sudo octop service install runs the service as root (no elevation needed at restart time), while a non-root install creates a user-scope unit that is still managed with systemctl --user.
  • Managing a system-scope unit from a non-root process requires passwordless sudo for the systemctl verbs. That is an operator-level decision and deliberately not enforced here — the API now simply explains the reason instead of failing silently behind a spinner.

…e euid

`resolve_service_scope()` derives the install scope from the caller's euid
(root → system, non-root → user).  That is right for a fresh install but wrong
when managing an existing deployment: a system unit installed under
`/etc/systemd/system` (or `/Library/LaunchDaemons`) by root and later run as
`User=<owner>` was managed with `systemctl --user ...`, which fails on headless
hosts with:

    Failed to connect to bus: No medium found

The dashboard restart button then spun until its 2-minute health-poll budget
expired and reported a generic timeout, while the real error only reached the
server log ("background service restart failed").

- `build_runtime()` now prefers the scope whose standard unit file actually
  exists (provisional scope first, so `--scope` and the euid heuristic still
  win when they match); fresh installs are unchanged.
- `is_service_installed()` probes both standard locations through the same
  helper, so a SYSTEM unit is recognised even when the inferred scope points at
  the user location (no more spurious "octop system service is not installed").
- Tests: add hermetic scope-inference and probe cases; make two existing
  "not installed" tests independent of the host's own installation.

Note: managing a system-scope unit as a non-root user still requires a
passwordless sudo rule for the systemctl verbs used by the restart path.
`POST /api/update/restart` answered "restarting" and bounced the service from a
fire-and-forget background task, so a failure — e.g. a non-root process that
needs a password to run `systemctl` against a system-scope unit — never reached
the client: the dashboard spun until its 2-minute budget expired and then
reported a generic timeout, leaving the operator to dig through logs.

- Start the bounce in a worker thread and wait a short window for an immediate
  failure, so the real error is returned while the request is still open. A
  successful restart still replaces this process and the client keeps polling
  health exactly as before.
- Privilege failures (`sudo: a password is required`, missing TTY, denials) map
  to a new localized `RESTART_PRIVILEGE_REQUIRED` error that tells the operator
  what to do; other early failures keep their original message.
- The dashboard shows the server-provided reason instead of the generic
  "restart failed" toast, so no log digging is required.

Tests cover the early-failure paths, a slow restart, and the privilege
classification. The new error code is registered in both the backend and the
dashboard locales (the i18n parity test enforces the key sets).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant