Skip to content

CNTRLPLANE-3691: Decouple NodePool condition handling from CAPI reconciliation - #9875

Open
redhat-chai-bot wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:fix-CNTRLPLANE-3691-v2
Open

redhat-chai-bot wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:fix-CNTRLPLANE-3691-v2

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adapt the three commits from CNTRLPLANE-3691: Decouple NodePool condition handling from CAPI reconciliation #9032 to current main.
  • Return reconcile-local condition results from CAPI reconciliation and apply them in the NodePool controller before error handling.
  • Preserve condition fields and ordering across success, early returns, later errors, and MHC readiness overrides, while leaving replica, version, and annotation mutations in their existing locations.

Tests

  • Extend CAPI condition tests, including repeated reconciliation and nonzero observed generation.
  • Add controller-level coverage for persisting returned conditions through the outer patch-on-error path.
  • Focused race-enabled NodePool tests passed.
  • Required make pre-commit passed, including builds, E2E compilation, verification, lint, and repository-wide race tests.

Live E2E execution and final-revision CI results remain to be verified on this PR.


AI-generated. Review for accuracy.

@bryan-cox requested from Slack

Summary by CodeRabbit

  • Bug Fixes
    • NodePool status now reflects readiness conditions reported by managed machines, with clearer fallback details when a readiness reason is unavailable.
    • Conditions collected during reconciliation are retained in NodePool status even when a later reconciliation step fails.
    • MachineHealthCheck creation now waits until the NodePool reaches the ignition endpoint, preventing checks from being created too early.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 2, 2026
@openshift-ci-robot

openshift-ci-robot commented Oct 2, 2026 •

Copy link
Copy Markdown

@redhat-chai-bot: This pull request references CNTRLPLANE-3691 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.1.0" version, but no target version was set.

Details

In response to this:

Summary

  • Adapt the three commits from CNTRLPLANE-3691: Decouple NodePool condition handling from CAPI reconciliation #9032 to current main.
  • Return reconcile-local condition results from CAPI reconciliation and apply them in the NodePool controller before error handling.
  • Preserve condition fields and ordering across success, early returns, later errors, and MHC readiness overrides, while leaving replica, version, and annotation mutations in their existing locations.

Tests

  • Extend CAPI condition tests, including repeated reconciliation and nonzero observed generation.
  • Add controller-level coverage for persisting returned conditions through the outer patch-on-error path.
  • Focused race-enabled NodePool tests passed.
  • Required make pre-commit passed, including builds, E2E compilation, verification, lint, and repository-wide race tests.

Live E2E execution and final-revision CI results remain to be verified on this PR.


AI-generated. Review for accuracy.

@bryan-cox requested from Slack

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs, /pipeline remaining to request missing second-stage tests, or /pipeline required to rerun the selected second-stage set.

@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: openshift/hypershift
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: 43bbac6
  • Layers: Repository YAML (base), Central YAML (inherited)
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: eee95fde-1da0-4aa6-ab0c-0eec5d4c52e0
Commits

Reviewing files that changed from the base of the PR and between 3198cc0 and b4df383.

Files selected for processing: 6
  • hack/tools/hypershiftlinter/analyzers/e2eutilallowlist/allowlist.go
  • hack/tools/hypershiftlinter/analyzers/e2eutilallowlist/e2eutilallowlist_test.go
  • hypershift-operator/controllers/nodepool/capi.go
  • hypershift-operator/controllers/nodepool/capi_test.go
  • hypershift-operator/controllers/nodepool/nodepool_controller.go
  • hypershift-operator/controllers/nodepool/nodepool_controller_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 2 remain after this review.

Walkthrough

CAPI reconciliation now returns accumulated NodePool conditions alongside errors. MachineSet and MachineDeployment reconciliation return readiness conditions derived from their status. The NodePool controller applies returned conditions before handling errors. Auto-repair MachineHealthCheck creation remains gated on the ignition endpoint. Upgrade-scope allowlist labels and checks are also updated.

Sequence Diagram(s)

sequenceDiagram
  participant CAPI as CAPI.Reconcile
  participant NodePool as NodePoolReconciler
  participant Status as NodePool.Status
  CAPI->>NodePool: return conditions and error
  NodePool->>Status: apply returned conditions
Loading

Suggested reviewers: bryan-cox

Priority: Normal

Merge Risk: Minimal · previous review b4df3

This change moves NodePool condition handling out of CAPI reconciliation and applies the returned conditions in the controller before errors are handled. No concrete merge-blocking risk was identified. Live end-to-end results and final-revision CI remain unverified.

Pre-merge checks: Passed 11
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: moving NodePool condition handling out of CAPI reconciliation.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names Passed All added subtest titles use literal, stable names. The ignition cases use two literal table names, and the status value remains in the test body. No changed test title uses generated identifiers, tim…
Test Structure And Quality Passed The prior assertion-message concern is addressed. Changed assertions in capi_test.go and nodepool_controller_test.go include diagnostic messages for reconciliation errors, results, conditions, res…
Topology-Aware Scheduling Compatibility Passed PASS: The pull request changes condition propagation and reconciliation return values. The changed-file inventory contains no deployment manifests. Added production lines in capi.go and `nodepool_co…
Ipv6 And Disconnected Network Test Compatibility Passed No new Ginkgo e2e test is added in the reviewed diff. The changed tests are standard Go unit tests, and the allowlist test only parses existing Ginkgo source. The diff introduces no e2e network connec…
No-Weak-Crypto Passed The pull request changes NodePool reconciliation, condition conversion, tests, and an allowlist. The added and modified lines contain no MD5, SHA-1, DES, RC4, 3DES, Blowfish, ECB, crypto API, custom c…
Container-Privileges Passed The pull request changes only Go source and test files. No container or Kubernetes manifest files changed. Added-line inspection found no privileged: true, hostPID, hostNetwork, hostIPC, `SYS_…
No-Sensitive-Data-In-Logs Passed No sensitive-data logging was introduced. The changed capi.go lines only rename the existing result variable to operationResult and preserve the same reconciliation log messages and fields. The …
Full details: Stable And Deterministic Test Names

Explanation

All added subtest titles use literal, stable names. The ignition cases use two literal table names, and the status value remains in the test body. No changed test title uses generated identifiers, timestamps, namespaces, node names, IP addresses, UUIDs, or legacy metadata tokens. Prior failure addressed: capi_test.go now defines literal False and Unknown case names at lines 2007-2012 and passes the status only through the test body; the prior formatted ignitionStatus title is absent.

Full details: Test Structure And Quality

Explanation

The prior assertion-message concern is addressed. Changed assertions in capi_test.go and nodepool_controller_test.go include diagnostic messages for reconciliation errors, results, conditions, resource counts, and persisted fields. The new allowlist test uses contextual Fatalf and Errorf messages. The diff adds no Ginkgo It, Eventually, or Consistently blocks, and it uses fake clients rather than cluster resources requiring cleanup. Prior failure addressed: Changed test assertions now include specific messages throughout capi_test.go and nodepool_controller_test.go, including condition persistence and error-path checks. The previously cited bare assertions are remediated.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The pull request changes condition propagation and reconciliation return values. The changed-file inventory contains no deployment manifests. Added production lines in capi.go and nodepool_controller.go introduce no affinity, topology spread, toleration, node selector, replica-count, arbiter, or scheduling constraints. The MaxUnavailable references remain existing rollout configuration and are not new scheduling constraints.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

No new Ginkgo e2e test is added in the reviewed diff. The changed tests are standard Go unit tests, and the allowlist test only parses existing Ginkgo source. The diff introduces no e2e network connection or IPv4-only behavior requiring this check.

Full details: No-Weak-Crypto

Explanation

The pull request changes NodePool reconciliation, condition conversion, tests, and an allowlist. The added and modified lines contain no MD5, SHA-1, DES, RC4, 3DES, Blowfish, ECB, crypto API, custom crypto implementation, or secret/token comparison. The added equality checks compare conditions and allowlist symbols only.

Full details: Container-Privileges

Explanation

The pull request changes only Go source and test files. No container or Kubernetes manifest files changed. Added-line inspection found no privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation: true, or root execution setting. The CPOCapabilities test field is unrelated to container privileges.

Full details: No-Sensitive-Data-In-Logs

Explanation

No sensitive-data logging was introduced. The changed capi.go lines only rename the existing result variable to operationResult and preserve the same reconciliation log messages and fields. The new log value is a controller-runtime operation result enum. No added logs contain passwords, tokens, API keys, PII, session IDs, hostnames, or customer data.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release label Oct 2, 2026
@openshift-ci
openshift-ci Bot requested review from bryan-cox and cblecker October 2, 2026 14:53
@codecov

codecov Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.41558% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.43%. Comparing base (364a78f) to head (1efae11).

Files with missing lines Patch % Lines
hypershift-operator/controllers/nodepool/capi.go 83.09% 11 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9875      +/-   ##
==========================================
+ Coverage   52.31%   52.43%   +0.11%     
==========================================
  Files         826      826              
  Lines      105380   105368      -12     
==========================================
+ Hits        55130    55249     +119     
+ Misses      46728    46606     -122     
+ Partials     3522     3513       -9     
Files with missing lines Coverage Δ
...erator/controllers/nodepool/nodepool_controller.go 59.21% <100.00%> (+1.29%) ⬆️
hypershift-operator/controllers/nodepool/capi.go 81.74% <83.09%> (+4.74%) ⬆️

... and 4 files with indirect coverage changes

Flag Coverage Δ
cmd-support 46.67% <ø> (ø)
cpo-hostedcontrolplane 54.69% <ø> (ø)
cpo-other 56.94% <ø> (ø)
hypershift-operator 60.93% <84.41%> (+0.45%) ⬆️
other 39.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

reason = c.Reason
if condition.Reason != "" {
reason = condition.Reason
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you extract the MachinesReadyCondition to NodePoolReadyCondition conversion into one unexported helper and use it from both reconcileMachineDeploymentStatus and reconcileMachineSet? Both paths currently repeat the status/message mapping, empty-reason fallback to AsExpectedReason, and observed-generation assignment. A shared helper would keep those semantics aligned while preserving the current behavior, including returning no condition when MachinesReadyCondition is absent.

This is a nonblocking cleanup suggestion; the duplication predates this PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in b4df383. Extracted the unexported nodePoolReadyConditions helper and used it from both reconcileMachineDeploymentStatus and reconcileMachineSet. It preserves the absent-condition nil return, status/message mapping, empty-reason fallback to AsExpectedReason, and NodePool observed generation, while leaving transition-time handling with the caller.

Added focused conversion tests and callsite coverage, including repeated MachineSet reconciliation and absent readiness. Full make pre-commit and recursive race-enabled NodePool tests pass on the pushed revision. Final-revision CI and live cloud E2E results remain unverified.


AI-generated. Review for accuracy.

@openshift-ci openshift-ci Bot added the area/ci-tooling Indicates the PR includes changes for CI or tooling label Oct 27, 2026
@openshift-ci

openshift-ci Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ironcladlou for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Addressed the test-name and assertion-diagnostic feedback in #9875 (comment).

  • Replaced formatted ignition subtest titles with literal False/Unknown table-case names.
  • Added specific diagnostics to assertions introduced or modified by this PR in capi_test.go and nodepool_controller_test.go.
  • Extracted the shared readiness conversion helper requested in the inline review.
  • Resolved conflicts with current main. Repaired stale callback-scoped utility allowlist entries that blocked lint, with source-derived regression tests; no new allowed symbols or blanket suppressions.

Published revision: b4df383. Complete make pre-commit, recursive race-enabled NodePool tests, and focused race-enabled allowlist analyzer tests pass. Final diff review and generated-file checks pass, and the committed working tree is clean. Fresh external CI and live cloud E2E results are not claimed as passing.


AI-generated. Review for accuracy.

Return reconcile-local condition results from CAPI so resource reconciliation no longer mutates NodePool conditions directly.

Apply accumulated results in the NodePool controller before error handling.

Preserve condition order and status updates across early returns and failures.

Assisted-by: LLM
Signed-off-by: Chai Bot <chai-bot@redhat.com>
@openshift-ci

openshift-ci Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

r.reachedIgnitionEndpointCondition,
r.machineAndNodeConditions,
r.validPlatformConfigCondition,
// TODO(alberto): consider moving here:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this removed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes only the TODO comment, not either condition. The cleanup accompanied moving condition application into the NodePool controller: CAPI still computes NodePoolUpdatingPlatformMachineTemplateConditionType and NodePoolAutorepairEnabledConditionType, returns them, and the controller applies them immediately after capi.Reconcile, before handling its error.

However, the TODO specifically suggested moving them into this signalConditions list. This PR does not do that; it preserves their later reconciliation order. Removing the comment therefore overstates how much of that TODO is addressed. The location-specific TODO can remain for a separate reorganization.


AI-generated. Review for accuracy.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok so are you putting the todo comment back?

@bryan-cox bryan-cox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec review found a readiness rollback regression on failed workload writes. The standards review found an unrelated test addition. Both findings are inline.

Validation: focused NodePool tests and the added linter test passed with race detection. A controller-level regression check passed on base 364a78f and failed on this head. Live E2E tests were not run. The spec review used this PR and #9032 because Jira was inaccessible.

}
}

func TestAllowlistOSImageStreamUpgradeScopes(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3][Standards] Move the unrelated OSImageStream test to a separate PR.

TestAllowlistOSImageStreamUpgradeScopes has no dependency on CAPI condition collection or application. CONTRIBUTING.md, under "Prior to Submitting a Pull Request", requires contributors to "Scope commits to one thing" and "save additional improvements for separate PRs." Please separate this test so the controller refactor does not also include unrelated linter-test maintenance.

return fmt.Errorf("failed to reconcile MachineDeployment %q: %w",
return reconcileErr
})
result.Conditions = append(result.Conditions, conditions...)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2][Spec] Preserve the Ready rollback when workload persistence fails.

reconcileWorkload restores the previous NodePool snapshot when CreateOrPatch fails. Previously, that also rolled back the Ready update computed inside the mutation. This append returns that condition despite the rollback, and the controller then persists it through its error path. The MachineSet branch at line 155 has the same issue.

I reproduced this through the outer NodePool reconciler by requesting five replicas from an existing three-replica MachineDeployment and injecting a workload patch failure. The stored MachineDeployment still requested three replicas, but this revision persisted Ready=True with ObservedGeneration=17. The same regression check passed on the base revision, which preserved the previous readiness state.

This contradicts #9032's promise that "the same conditions are set under the same circumstances" and this PR's requirement to preserve conditions across later errors. Please append workload conditions only after reconcileWorkload succeeds. Conditions computed before that operation should still survive its failure, and successful workload conditions should still survive later MHC errors. Update the new failure tests, which currently assert the changed behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci-tooling Indicates the PR includes changes for CI or tooling area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants