fix(icms): Throw 503 on config reload failure

[dmikhaylovnv]

Problem

A live configuration update can introduce an invalid value, such as a misspelled enum. Spring Cloud applies the new environment and destroys refresh-scoped beans, but normally recreates those beans lazily.

As a result, the refresh can appear successful while the application context is unusable. Requests later fail with misleading responses, while Kubernetes readiness remains healthy and continues routing traffic to the affected pod.

What changed

• Added src/control-plane-services/instance-cluster-management/icms-core/src/main/java/com/nvidia/icms/configuration/refresh/ConfigRefreshConfiguration.java:

  • Provides a Spring Cloud Kubernetes ConfigurationUpdateStrategy.
  • Runs ContextRefresher.refresh().
  • Publishes REFUSING_TRAFFIC when refresh fails.
  • Publishes ACCEPTING_TRAFFIC after a subsequent successful refresh.
  • Does not override readiness refusals published by another component.
  • Backs off when the configured reload strategy is not refresh.

• Added src/control-plane-services/instance-cluster-management/icms-core/src/main/java/com/nvidia/icms/configuration/refresh/RefreshScopeBeanValidator.java:

  • Recreates refresh-scoped beans immediately after refresh.
  • Converts latent property-binding failures, such as an invalid enum, into refresh failures.

• Added unit tests covering failed refreshes, recovery, readiness ownership, Spring event wiring, and non-refresh strategy behavior.

• Added src/control-plane-services/instance-cluster-management/icms-service/src/test/java/com/nvidia/icms/ConfigRefreshFailureIntegrationTest.java:

  • Replays the invalid telemetry enum scenario.
  • Verifies readiness becomes unavailable while liveness remains healthy.
  • Verifies readiness recovers after fixing the configuration.

• Updated Bazel dependencies for Spring Cloud Kubernetes commons and Awaitility.

@sanjay-saxena I have a feeling this should go to parent lib.

Closes NVCF-11367

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 64c8a165-7be8-4c92-befe-949093fdaa5f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🛡️ CodeQL Analysis

🚨 Found 11 issue(s)

Severity Breakdown:

• 🔴 Errors: 0
• 🟡 Warnings: 0
• 🔵 Notes: 0

📋 Top Issues

🔗 View full details in Security tab

_{🕐 Last updated: 2026-08-14 19:15:26 UTC | Commit: 9516ace}

[sanjay-saxena]

The goal is to make K8s take the pods out of rotation via its readiness check whenever remote config refresh fails. This will automatically result in 503s to be reported by the infra. This would be similar to when Cassandra or any other dependency is not ready, K8s takes the pods out of rotations and reports 503.

There is a far simpler and Spring native approach to accomplish this by publishing an AvailabilityChangeEvent with ReadinessState.REFUSING_TRAFFIC. This can be used in conjunction with Spring Cloud K8s(SCK)'s ConfigurationUpdateStrategy. Using ConfigurationUpdateStrategy, we can raise the event with ReadinessState.REFUSING_TRAFFIC state when the config validation fails or with ReadinessState.ACCEPTING_TRAFFIC state when the config validation succeeds.

[dmikhaylovnv]

Right, we need to mark app health as DOWN and while K8s is removing pod from rotation reject all traffic, except health endpoint.