Skip to content

StatusWriter: detect two processes writing one status file - #110

Open
ampledata wants to merge 1 commit into
mainfrom
feat/status-single-writer
Open

StatusWriter: detect two processes writing one status file#110
ampledata wants to merge 1 commit into
mainfrom
feat/status-single-writer

Conversation

@ampledata

Copy link
Copy Markdown
Collaborator

Three separate gateway integrations hit the same hazard independently today — dronecot (6 capture workers), aprscot (APRSWorker + SensorWorker) and sikw00fcot (three entry points, only two of which Conflicts= each other). Each integrator worked it out for themselves, which is a good sign the library should say it rather than leaving it to be rediscovered a fourth time.

The hazard

Every write serialises a whole document, so two processes sharing an app_name do not merge. The file alternates between two disjoint sets of counters once a second, and every reader sees whichever wrote last.

A UI shows figures flickering between two gateways' worth of traffic, and nothing anywhere reports a fault. It isn't a crash — which is exactly what makes it nasty.

What this adds

The rule is now stated in the module docstring: give the writer to the single choke point the workers feed, not to each worker.

And because a docstring only helps someone who reads it, StatusWriter now detects the case. Before replacing the file it checks whether the document there carries a different pid that is still alive; if so it warns once and records contended_with in its own output, so a UI can report that the figures may not be this gateway's.

Status file /run/sikw00fcot/status.json is also being written by PID 1443.
Each write replaces the whole document, so the two will overwrite one another
and readers will see only whichever wrote last. Give the StatusWriter to a
single choke point rather than to each worker.

Deliberately narrow, to avoid crying wolf

  • a dead pid is not contention — that's our own previous run, and taking the file over is correct after a restart
  • our own pid is not contention
  • an absent or half-written file concludes nothing
  • rate-limited to every 30 s, and stops entirely once it has something to say, so the common correct case costs nothing

Liveness uses os.kill(pid, 0), which sends no signal. A pid that exists but isn't ours to signal still counts as live.

Tests

6 new (29 in the module, 131 in the suite). All five mutations tried were caught:

Mutation Caught
never detects contention
warns per-write instead of once
treats a dead pid as live
flags our own pid
drops the field from the document

🤖 Generated with Claude Code

https://claude.ai/code/session_0197da7dhvcPoHxYKamrYqyM

Three separate gateway integrations hit the same hazard independently today --
dronecot (6 capture workers), aprscot (APRSWorker + SensorWorker) and
sikw00fcot (three entry points, only two of which Conflicts= each other). Each
integrator worked it out for themselves, which is a good sign the library
should say it rather than leaving it to be rediscovered a fourth time.

The hazard: every write serialises a WHOLE document, so two processes sharing
an app_name do not merge. The file alternates between two disjoint sets of
counters once a second, and every reader sees whichever wrote last. A UI shows
figures flickering between two gateways' worth of traffic and nothing anywhere
reports a fault. It is not a crash, which is exactly what makes it nasty.

So the rule is now stated in the module docstring -- give the writer to the
single choke point the workers feed, not to each worker -- and, because a
docstring only helps someone who reads it, StatusWriter now detects the case:
before replacing the file it checks whether the document there carries a
different pid that is still alive, and if so warns ONCE and records
`contended_with` in its own output so a UI can report that the figures may not
be this gateway's.

Deliberately narrow, to avoid crying wolf:

  * a dead pid is NOT contention -- that is our own previous run, and taking
    the file over is the correct behaviour after a restart;
  * our own pid is not contention;
  * an absent or half-written file concludes nothing;
  * the check is rate-limited to every 30s and stops entirely once it has
    something to say, so the common correct case costs nothing.

Liveness uses os.kill(pid, 0), which sends no signal. A pid that exists but is
not ours to signal still counts as live.

6 new tests, 29 in the module, 131 in the suite. All five mutations tried were
caught: never detecting, warning per-write instead of once, treating a dead pid
as live, flagging our own pid, and dropping the field from the document.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197da7dhvcPoHxYKamrYqyM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant