Feature Request: ClusterPromotion — Rollback on Failure

[gianlucam76]

Is your feature request related to a problem? Please describe.

ClusterPromotion orchestrates a multi-stage deployment pipeline across cluster groups. Each stage creates a ClusterProfile targeting a subset of clusters, waits for all matching ClusterSummary instances to reach Provisioned, runs optional post-delay health checks, and then advances to the next stage.

A limit its usefulness in production pipelines:

1. When a stage fails there is no automatic recovery path — the pipeline stalls and the operator must intervene manually.

Describe the solution you'd like
When PostDelayHealthChecks fail after a stage has been deployed, the controller keeps requeuing but has no path forward. The only recovery is manual operator intervention: inspect the stage, revert the ProfileSpec manually, or delete the ClusterPromotion and start over. In a production pipeline this is unacceptable — a bad rollout to staging should not require human remediation before the pipeline can continue.

Add a OnFailure field to the Stage struct that controls what the controller does when the stage cannot reach Provisioned (or health checks fail) after a configurable number of retries:

Halt    — stop the pipeline at the failed stage (current behaviour, made explicit)
Rollback — re-apply the last known-good ProfileSpec snapshot to the stage's clusters

For Rollback to work the controller must store a snapshot of the ProfileSpec that was last successfully applied for each stage. The natural place is a new field in StageStatus:

// LastSuccessfulProfileSpec is a snapshot of the ProfileSpec that was
// successfully applied the last time this stage reached Provisioned.
// Used as the rollback target when OnFailure is set to Rollback.
// +optional
LastSuccessfulProfileSpec *ProfileSpec `json:"lastSuccessfulProfileSpec,omitempty"`

Rollback mechanics:

1. When a stage transitions to Provisioned and all health checks pass, the controller writes the current ProfileSpec into StageStatus.LastSuccessfulProfileSpec.
2. When a failure is detected (health checks failing for longer than a configurable FailureThreshold duration, or MaxConsecutiveFailures exceeded), the controller checks Stage.OnFailure.
3. If OnFailure: Rollback and a snapshot exists, the controller applies LastSuccessfulProfileSpec to the stage's ClusterProfile, resets the stage status, and emits an event.
4. If no snapshot exists (first-ever run of this stage), the controller falls back to Halt and records a message explaining why rollback was not possible.

The OnFailure field only applies to non-final stages if the intent is to protect downstream stages from a bad promotion. It also applies to the final (production) stage to protect the cluster fleet from a broken rollout.

API changes

type FailureBehavior string

const (
    FailureBehaviorHalt     FailureBehavior = "Halt"
    FailureBehaviorRollback FailureBehavior = "Rollback"
)

type Stage struct {
    Name            string                        `json:"name"`
    ClusterSelector libsveltosv1beta1.Selector     `json:"clusterSelector,omitempty"`
    Trigger         *Trigger                       `json:"trigger,omitempty"`

    // OnFailure controls what happens when this stage cannot reach Provisioned.
    // Defaults to Halt.
    // +kubebuilder:default:=Halt
    // +optional
    OnFailure FailureBehavior `json:"onFailure,omitempty"`

    // FailureThreshold is the duration the controller waits after detecting a
    // failure before acting on OnFailure. Gives transient failures time to self-heal.
    // +optional
    FailureThreshold *metav1.Duration `json:"failureThreshold,omitempty"`
}

type StageStatus struct {
    // ... existing fields ...

    // LastSuccessfulProfileSpec is a snapshot of the ProfileSpec at the time
    // this stage last reached Provisioned with all health checks passing.
    // +optional
    LastSuccessfulProfileSpec *ProfileSpec `json:"lastSuccessfulProfileSpec,omitempty"`

    // RolledBack is true if the controller applied a rollback for this stage
    // in the current promotion run.
    // +optional
    RolledBack bool `json:"rolledBack,omitempty"`
}

[gianlucam76]

Thats a good question @hiSandog

In my view, a ClusterPromotion is a single unit. So if a ClusterPromotion is changing N applications, all N should be reverted back (when rollback feature is implemented).

User can still create N different ClusterPromotion (one per application) if the goal is to treat the applications independently