Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ flowchart LR
end

subgraph etc["/etc/hispec"]
ENV["instances/&lt;name&gt;.env<br/>HISPEC_DAEMON, HISPEC_CONFIG"]
ENV["instances/&lt;name&gt;.env<br/>HISPEC_DAEMON, HISPEC_HOST"]
YML["&lt;name&gt;.yaml<br/>deployed config"]
end

Expand All @@ -231,8 +231,9 @@ flowchart LR

One systemd template unit, `hispec@.service`, runs every daemon. An
instance `hispec@<name>` reads `/etc/hispec/instances/<name>.env`, which
names the daemon script (`HISPEC_DAEMON`, relative to `daemons/`) and its config
(`HISPEC_CONFIG`). Running a second unit of the same hardware model — two
names the daemon script (`HISPEC_DAEMON`, relative to `daemons/`). Its config
is `/etc/hispec/<name>.yaml`, derived from the instance name rather than
declared. Running a second unit of the same hardware model, two
Lakeshores, five filter wheels — is two env files pointing at the same script
with different configs.

Expand Down
44 changes: 25 additions & 19 deletions docs/operations/systemd.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ An instance gets its identity from two files:

| File | What it says |
|---|---|
| `/etc/hispec/instances/<name>.env` | which daemon script to run (`HISPEC_DAEMON`) and which config to hand it (`HISPEC_CONFIG`) |
| `/etc/hispec/instances/<name>.env` | which daemon script to run (`HISPEC_DAEMON`) and which host should run it (`HISPEC_HOST`) |
| `/etc/hispec/<name>.yaml` | the config itself: serial ports, addresses, limits, keyword names |

So running a second Lakeshore is not new code and not a new unit. It is a
Expand Down Expand Up @@ -239,7 +239,6 @@ matching `systemd/instances/<name>.env`:

```sh
HISPEC_DAEMON=<path relative to daemons/, e.g. generic/inficon or hsfei/adc>
HISPEC_CONFIG=/etc/hispec/<name>.yaml
HISPEC_HOST=<the host that runs it, e.g. hispecserver or fei>
```

Expand Down Expand Up @@ -368,10 +367,11 @@ Common causes, in the order they bite:
- **`Failed to load environment files`**: `/etc/hispec/instances/<name>.env`
is missing or unreadable. The instance name in the command must match the
file name exactly.
- **`No such file or directory`** on the config: `HISPEC_CONFIG` points
somewhere that does not exist. The config has to be deployed to `/etc/hispec/`
as well as the `.env`. `hispec deploy <name>` copies both; copying one by
hand and forgetting the other is the usual mistake.
- **`No such file or directory`** on the config: the unit passes
`/etc/hispec/<name>.yaml`, derived from the instance name, and nothing is
there. The config has to be deployed as well as the `.env`. `hispec deploy
<name>` copies both; copying one by hand and forgetting the other is the
usual mistake.
- **A serial port or USB error**: the device is unplugged, powered off, or
claimed by another process. Two instances pointing at the same port will do
this to each other, and so will a daemon left running from a manual test.
Expand Down Expand Up @@ -505,30 +505,35 @@ sudo ./systemd/install.sh alice bob # operator usernames
`install.sh` is idempotent. Re-run it:

- to enrol more operators (`sudo ./systemd/install.sh carol`);
- after a pull that changed dependencies in `pyproject.toml`, or
`hispec@.service`, the polkit rule or `hispec-enable` under `systemd/`;
- after a pull that changed dependencies in `pyproject.toml`, or the unit or
polkit rule in `instrumentctl`;
- to repair a host `hispec doctor` says is broken.

You do **not** need it to add a daemon or to pick up code changes;
[`hispec deploy`](#deploying-an-instance) and a `git pull` cover those. It:
[`hispec deploy`](#deploying-an-instance) and a `git pull` cover those.

The script itself only bootstraps a pinned `uv` and builds the venv at
`/opt/hispec/venv`, which is also where the `hispec` command comes from. It
then hands over to `hispec install`, which lives in `instrumentctl` and does
the rest:

- creates the `hispec-ops` group and the unprivileged `hispec` system user the
daemons run as;
- adds each named operator to `hispec-ops` and `systemd-journal`;
- creates `/etc/hispec/{,instances/}` and `/var/log/hispec/`, group-owned by
`hispec-ops` and setgid so files created by one operator stay readable by the
rest;
- creates `/etc/hispec/secrets.env`, writable by `hispec-ops`;
- creates the venv at `/opt/hispec/venv` with the repo `pip install -e`'d into
it, which also provides the `hispec` CLI;
- installs `hispec@.service` (then runs `daemon-reload`), the polkit rule, the
sudoers drop-in, `/usr/local/bin/hispec` and `/usr/local/sbin/hispec-enable`;
- removes the retired `hispec-fei-start`, `hispec-fei-stop` and `hispec-doctor`
scripts (now `hispec start hsfei`, `hispec stop hsfei` and `hispec doctor`);
- migrates any instance still running under the old `hispec-daemon@` name,
preserving whether it was enabled and whether it was up;
- creates `/etc/hispec/secrets.env`, writable by `hispec-ops`, and records this
host's role in `/etc/hispec/host`;
- renders `hispec@.service` (then runs `daemon-reload`), the polkit rule, the
sudoers drop-in, `/usr/local/bin/hispec` and `/usr/local/sbin/hispec-enable`
from `src/hispec/cli/instrument.toml`, reporting only what it changes;
- lists instances that are deployed but not set to start at boot.

Because those five are rendered rather than committed, `hispec doctor`
compares what is installed against what `install` would write now, which is
what catches a host whose unit predates a template change.

`HISPEC_REPO_DIR` and `HISPEC_VENV_DIR` override the paths.

`/usr/local/bin/hispec` is a tiny wrapper that runs the CLI from the venv. The
Expand All @@ -552,7 +557,8 @@ and `/var/log/hispec`; device nodes under `/dev` are unaffected by both.

Starting and stopping a unit is the polkit action
`org.freedesktop.systemd1.manage-units`, and systemd tells polkit which unit is
involved, so `systemd/polkit/49-hispec.rules` can allow it for `hispec@*` and
involved, so `/etc/polkit-1/rules.d/49-hispec.rules` can allow it for
`hispec@*` and
nothing else. Enabling and disabling is
`org.freedesktop.systemd1.manage-unit-files`, and systemd does **not** pass a
unit name with it. A polkit rule for it would grant `hispec-ops` enable/disable
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ dependencies = [
# influxdb extra: needed by daemons/generic/keygrabber
"libby[influxdb]@git+https://github.com/CaltechOpticalObservatories/libby.git",
"hardware_device_base@git+https://github.com/COO-Utilities/hardware_device_base",
"instrumentctl@git+https://github.com/CaltechOpticalObservatories/instrumentctl.git",
"instrumentctl@git+https://github.com/CaltechOpticalObservatories/instrumentctl.git@v0.2.0",
"camera_daemon@git+https://github.com/COO-Utilities/camera-daemon.git"
]

Expand Down
3 changes: 2 additions & 1 deletion src/hispec/cli/instrument.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@

[instrument]
name = "hispec"
display_name = "HISPEC"
user = "hispec"
ops_group = "hispec-ops"
# Instance files set HISPEC_DAEMON, HISPEC_CONFIG and HISPEC_HOST
# Instance files set HISPEC_DAEMON and HISPEC_HOST
env_prefix = "HISPEC"
docs_url = "https://caltechopticalobservatories.github.io/hispec/operations/systemd.html"

Expand Down
25 changes: 19 additions & 6 deletions systemd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,25 @@ directory.

| Path | What it is |
| --- | --- |
| `hispec@.service` | The template unit. One file runs every daemon; `hispec@<name>` is an instance of it. Installed to `/etc/systemd/system/`. |
| `instances/<name>.env` | Per-instance settings: `HISPEC_DAEMON` (script, relative to `daemons/`) and `HISPEC_CONFIG` (deployed config path). Deployed to `/etc/hispec/instances/`. |
| `polkit/49-hispec.rules` | Lets `hispec-ops` members start/stop/restart `hispec@*` without sudo. Installed to `/etc/polkit-1/rules.d/`. |
| `bin/hispec` | Wrapper that runs the `hispec` CLI from the venv. Installed to `/usr/local/bin/`. |
| `bin/hispec-enable` | Root helper behind `hispec enable` / `hispec deploy`: enables/disables instances at boot, which `systemctl enable` cannot be granted per-unit. Installed to `/usr/local/sbin/`, invoked via `sudo -n` by a NOPASSWD drop-in. |
| `install.sh` | Host setup for admins / IT: users, groups, directories, venv, and all of the above. Idempotent, run as root. Not needed to add a daemon. |
| `instances/<name>.env` | Per-instance settings: `HISPEC_DAEMON` (script, relative to `daemons/`) and `HISPEC_HOST` (which host runs it). Deployed to `/etc/hispec/instances/`; the config path is derived as `/etc/hispec/<name>.yaml`. |
| `install.sh` | Host setup for admins / IT. Bootstraps uv, builds the venv, then hands over to `hispec install`. Idempotent, run as root. Not needed to add a daemon. |

`hispec install` writes the rest, rendered from
`src/hispec/cli/instrument.toml`:

| Installed | What it is |
| --- | --- |
| `/etc/systemd/system/hispec@.service` | The template unit. One file runs every daemon; `hispec@<name>` is an instance of it. |
| `/etc/polkit-1/rules.d/49-hispec.rules` | Lets `hispec-ops` members start/stop/restart `hispec@*` without sudo. |
| `/usr/local/bin/hispec` | Wrapper that runs the CLI from the venv. |
| `/usr/local/sbin/hispec-enable` | Root helper behind `hispec enable` / `hispec deploy`: enables/disables instances at boot, which `systemctl enable` cannot be granted per-unit. Invoked via `sudo -n` by a NOPASSWD drop-in. |
| `/etc/sudoers.d/hispec-ops` | That drop-in. |

The templates live in
[instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl),
so a change to the unit or the polkit rule is made there and reaches a host
through `install.sh`. `hispec doctor` compares what is installed against what
`install` would write now.

## Quick reference

Expand Down
17 changes: 0 additions & 17 deletions systemd/bin/hispec

This file was deleted.

60 changes: 0 additions & 60 deletions systemd/bin/hispec-enable

This file was deleted.

63 changes: 0 additions & 63 deletions systemd/hispec@.service

This file was deleted.

Loading
Loading