Skip to content

Document the trade-offs of --workers - #59

Merged
tokibito merged 1 commit into
mainfrom
workers-trade-offs
Oct 2, 2026
Merged

tokibito merged 1 commit into
mainfrom
workers-trade-offs

Conversation

@tokibito

@tokibito tokibito commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Documentation only, following #58. The Running several workers section had a short list of what N costs; this replaces it with a Trade-offs subsection that says what `--workers N` buys and what it costs, so N can be picked deliberately.

  • Isolation: one crashed, leaking or hung task costs one worker, and nothing is asked of the task code.
  • Memory: N full Django processes with no copy-on-write sharing, since the children are spawned rather than forked; `--max-tasks` as the leak bound.
  • CPU: one interpreter and one GIL per worker, so CPU-bound tasks scale across cores and more than the core count only helps I/O-bound ones.
  • Polling load and connections, as before, with the LISTEN/NOTIFY broadcast note.
  • Stability of the supervisor itself: what happens to the workers if it is killed outright (they are in their own process group and keep running), why that is a non-issue under systemd's default `KillMode` and in a container, and that `requeue_stale_database_tasks` covers the rows either way.
  • What a process manager still does better, and that running `--workers N` under one is a reasonable middle.
  • Threads inside a worker are a later step tracked in Built-in parallel workers: a supervisor with --workers N, and --threads M inside a worker #25, and on this backend would save memory but not connections.

The CHANGELOG entry for `--workers` under Unreleased gains a sentence pointing at the subsection. No code changes.

Refs #25

Isolation, memory, CPU, polling load and connections, what happens to the
workers when the supervisor itself is killed, where a process manager still
does better, and that threads are a later step. Follows #58.

Refs #25
@tokibito
tokibito merged commit 1be8625 into main Oct 2, 2026
10 checks passed
@tokibito
tokibito deleted the workers-trade-offs branch October 2, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant