Repository navigation
Add pid and worker_index to the worker's log records - #71
Merged
Merged
Conversation
Under --workers the supervisor's records carry a worker's index and pid, and the worker's own records carry the worker id it picked for itself, so one host's logs could not be joined. The supervisor now passes each child its slot's index in DJANGO_DATABASE_TASK_WORKER_INDEX, and the worker adds the index and its own pid to Worker started, Worker finished, the failed broker receive and every task record, under the names the supervisor already uses. A worker started without the supervisor records its pid and no index. Added fields only; no logger name, message or existing field changes. Refs #65
The index was read from the environment on every record and parsed with int(), so task code could change what later records said, and values the supervisor never sets (-1, +3, 1_0, non-ASCII digits) were recorded. The worker now reads it once at startup, reports and ignores anything that is not a whole number of 1 or more, and leaves the variable in the environment for what the tasks start. The README documents the variable for process managers that start the workers themselves. Refs #65
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First half of #65. Under
--workersthe supervisor's records carry a worker'sworker_indexandpid, while the worker's own records carry the worker id it picked for itself, so one host's logs could not be joined. This addspidandworker_indexto the worker's records, under the names the supervisor already uses, so the two sides join onpid.What it does
DJANGO_DATABASE_TASK_WORKER_INDEXin each child's environment to the slot's index. A restarted worker keeps its slot's index and gets a new pid.worker_indexand its ownpidtoWorker started,Worker finished, the failed broker receive, every task record (throughtask_log_fields()) and the executor-thread failure record. With-v 1and up, the banner showsWorker index: NafterWorker ID:when there is one.int()alone would take-1,+3,1_0or non-ASCII digits, none of which the supervisor ever sets.os.environafterwards does not change what the records say. The variable is left in the environment, so a task, and anything it starts, can read which worker it runs in.pidand noworker_index.Things to look at
%iand "worker 2" names the same unit across restarts. The README has a Worker index subsection under Structured logging, with a template unit example.Ignoring the worker indexWARNING, writes the same line on stdout, and runs without an index. The index only labels the records, and a worker that will not start over a label seemed worse than one that runs without it.test_fields_describe_the_taskcompares the wholetask_log_fields()dict, so it now includespid. That is the behaviour change this PR makes: task records gainpideverywhere, including tasks run from the admin actions.Compatibility
Added fields, one added WARNING record for an invalid value, and one added banner line when the variable is set. No logger name, message or existing field changes, so filters written against the current records keep working. Existing projects need no changes. The CHANGELOG entry is under Unreleased.
Tests
tests/test_logging.py: the fields on the worker and task records, no index without the variable, an index the environment changes after startup is not picked up, a task changing the variable does not change later records, a task can read the variable, an invalid value is reported and ignored, and the banner line.tests/test_supervisor.py:read_worker_index()accepting and refusing values, each child seeing its own index matched to the pid inWorker process started, a restarted worker keeping its slot's index, and the supervisor's own environment left alone.Run on Python 3.13 / Django 6.0 (SQLite), Python 3.14 / Django 6.1 with
-W error::DeprecationWarning, and the whole suite against PostgreSQL 16.Refs #65