feat: Add opt-in upgrade_drain_timeout to drain control nodes before an upgrade

[Aureliolo]

SUMMARY

termination_grace_period_seconds does not keep jobs alive across a version upgrade, which is the case its own documentation says it covers ("AWX upgrade by the operator").

As soon as the first pod running the new version registers as an instance, every control node still on the old version stops its own services, because cluster_node_heartbeat shuts a node down when it sees a peer reporting a higher version:

# awx/main/tasks/system.py
if Version(other_inst.version...) > Version(awx_application_version...) and not settings.DEBUG:
    logger.error("Host {} reports version {}, but this node {} is at {}, shutting down")
    stop_local_services(communicate=False)
    raise RuntimeError("Shutting down.")

That shutdown comes from inside the container, so it preempts the PreStop hook, which gates only kubelet's SIGTERM. Observed on a production cluster during a 25.4.0 -> 25.5.1 upgrade, from one pod's logs: the drain hook was already waiting on the right job when the node killed it seven seconds later.

08:37:27  [preStop.exec] [master] [handler] Instance [...] has been disabled
08:37:27  [preStop.exec] [master] [handler] 1/inf: Waiting 30s before the next attempt to
                                   see if the following instance' managed jobs have finished: [...]
08:37:34  ERROR awx.main.tasks.system Host [...] reports version 25.5.1, but this node [...] is at 25.4.0
08:37:34  WARNING awx.main.utils.reload Stopping services on this node
08:37:39  WARNING awx.main.dispatch job NNNN (failed) encountered an error (rc=1)
08:37:57  [preStop.exec] [master] [handler] Done waiting for instance' managed jobs to finish!

The drain machinery works, it just runs too late. This adds upgrade_drain_timeout, which drains ahead of the rollout instead: when the application image changes, each control node is disabled so that newly submitted jobs stay pending rather than being scheduled onto a pod about to be replaced, and the operator then waits for the jobs already running there.

The drain is placed immediately before Apply deployment resources, which is the last point where the cluster is still wholly on the old version. migrate_schema.yml runs later in install.yml, so draining nodes never run old code against a migrated database. That ordering is deliberate and is why this lives in the operator rather than in cluster_node_heartbeat.

Reaching the timeout does not abort the upgrade. It proceeds and any jobs still running fail as they do today, so the worst case is current behaviour.

Opt-in: upgrade_drain_timeout defaults to 0, which changes nothing for existing users. An alternative is to key this off termination_grace_period_seconds, since anyone who set that has already asked for jobs to be respected on termination. Happy to switch if you prefer one knob to two.

ISSUE TYPE

• New or Enhanced Feature

COMPONENT NAME

• Installer role
• CRD

ADDITIONAL INFORMATION

Skipped unless all of the following hold, so ordinary reconciles are unaffected:

• upgrade_drain_timeout > 0
• the task Deployment already exists (not a fresh install)
• its running application image differs from the resolved one
• a web pod is available to run awx-manage in

All control nodes are disabled before any of them is waited on: an instance that is still enabled keeps accepting work, so draining them one at a time would not converge. The wait uses disable_instance --wait --retry=1 as a probe and retries from Ansible, rather than holding a single k8s_exec stream open for the whole drain.

Known trade-off: if the operator dies between the drain and the apply, the instances stay disabled until a reconcile completes. The next reconcile drains and applies, and the rollout replaces those pods with freshly registered, enabled ones. Note also there is no awx-manage enable_instance; recovery in that window is a PATCH on the instance.

Verified against a live operator built from this branch on a throwaway cluster:

• the CRD accepts the field (upgrade_drain_timeout=60 on the CR)
• the operator reconciles normally with it set
• the drain is correctly skipped when the image is unchanged (false_condition: "_deployed_image != _image"), with zero disable_instance calls
• yamllint -c .yamllint clean; ansible-lint reports only rules that existing files in the repo also report

molecule verify did not complete in my environment: the deployment never became ready there for reasons unrelated to this change (a pod networking fault in my local cluster), so the added test's final assertion, that an ordinary reconcile leaves every control node enabled, did not execute. The scenario also never changes the application image, so it does not exercise the drain body itself. Flagging both rather than implying coverage I did not get.

[Copilot]

Pull request overview

Adds an opt-in upgrade_drain_timeout mechanism to proactively drain AWX control nodes before applying new deployment resources during an upgrade, preventing running jobs from being cut short by AWX’s version-skew self-shutdown behavior. This fits into the installer/operator reconcile flow by inserting a drain step immediately before “Apply deployment resources”, and extends the CRD + docs + Molecule coverage to support the new field.

Changes:

• Introduces upgrade_drain_timeout (default 0) and wires it into the installer reconcile to conditionally run a new drain task before upgrading deployments.
• Adds drain_control_nodes.yml to disable task pods (instances) and wait for their running jobs to finish prior to rollout (without aborting the upgrade on timeout).
• Updates CRD schema, user documentation, and Molecule scenario/spec + verification to include/assert the new field and ensure ordinary reconciles don’t disable instances.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
roles/installer/tasks/resources_configuration.yml	Calls the new drain task before applying deployment resources when upgrade_drain_timeout > 0.
roles/installer/tasks/drain_control_nodes.yml	Implements the pre-upgrade control-node disable + wait logic.
roles/installer/defaults/main.yml	Adds the new upgrade_drain_timeout default setting (opt-in).
molecule/default/verify.yml	Includes the new Molecule verification task file.
molecule/default/templates/awx_cr_molecule.yml.j2	Sets upgrade_drain_timeout in the Molecule AWX CR template.
molecule/default/tasks/upgrade_drain_test.yml	Verifies that a reconcile with unchanged image does not disable instances even when upgrade_drain_timeout is set.
docs/user-guide/advanced-configuration/pods-termination-grace-period.md	Documents upgrade behavior and the new upgrade_drain_timeout option.
config/crd/bases/awx.ansible.com_awxs.yaml	Adds upgrade_drain_timeout to the CRD schema with default 0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Aureliolo]

until retries per loop item, and during those retries the registered variable holds
that item's own result. The .results aggregation only exists once the loop has
finished, which is after until has been evaluated, so return_code is the right
reference here.

Checked on ansible-core 2.18.18 (the version in the operator image) and on 2.21.1,
using the same condition shape as this task, including the | default(1) fallback
that would have masked the problem if it were real:

- ansible.builtin.command: /bin/true
  loop: [one, two]
  register: r
  until: r.rc | default(1) == 0
  retries: 5
  delay: 1

item=one rc=0 attempts=1
item=two rc=0 attempts=1

Both items satisfy the condition on the first attempt. If it were reading the
aggregate, r.rc would be undefined, default(1) would evaluate it to false, and
each item would show attempts=5 followed by a "Failed until condition" failure.

Leaving the task as is.