Root OCIRepository verification does not survive onto the live cluster

🤖 Generated by the Agentic Engineer

Evidence

The root Flux source on prod has no signature verification, and this survives a full successful deploy against a correctly-configured repo.

flux-system/flux-system   url=oci://ghcr.io/devantler-tech/platform/manifests
                          ref.tag=latest   interval=1m0s   spec.verify = null

Every tenant OCIRepository in the same cluster (ascoachingogvaner, aws, doggy-countdown, github-config, wedding-app) does carry a cosign verify block, so verification works here — the root source specifically lacks it.

This is not a pending rollout:

#2919 moved the config to spec.workload.flux.verify on 2026-08-03T03:26Z, and yq confirms it resolves there today with all three intended identities. spec.cluster.verify is null.
The 🚀 Deploy to Prod job succeeded at 2026-08-03T23:17:36Z (run 30861750366) — about 20 hours later.
The root source was still spec.verify: null when re-read on 2026-08-04T00:2xZ.

Where the value is lost

The live FluxInstance (flux-system/flux) has no verify key under spec.sync; its sync block is exactly interval, kind, path, provider, pullSecret, ref, url. flux-operator derives the root OCIRepository from that instance, so a sync without verify yields a root source without spec.verify.

Both KSail halves exist, which is why this reads as a defect rather than a gap:

input modelled — Workload.Flux.Verify (FluxVerifySpec), pkg/apis/cluster/v1alpha1/types.go:285, documented as "rendered onto the generated flux-system OCIRepository"
renderer present — pkg/svc/installer/flux/ocirepository_patcher.go (buildVerifyPatch / applyVerify / ensureVerify)
platform CI pins KSail 7.178.13

Hypotheses to discriminate (not yet proven)

Create-time only. ensureVerify runs on a path the prod deploy does not take, so an existing cluster never receives it. Same class as the floatingIPEnabled no-op in feat: enable the Hetzner floating-IP stable API endpoint on prod #2524: static validation green, production effect nil.
Applied then reverted. flux-operator owns that OCIRepository; a direct patch with FluxInstance.spec.sync.verify unset is liable to be reconciled back to null. Relevant: KSail's installer-side Sync struct (pkg/svc/installer/flux/fluxinstance_types.go:132) models eight fields and has no Verify, so KSail cannot currently express sync.verify — the same typed-model-narrower-than-the-CRD class as ksail#6455.

Acceptance criteria

The mechanism is identified with evidence — a deploy-log read showing whether ensureVerify executed, plus a re-read of spec.verify immediately after a deploy to detect an apply-then-revert.
The live root OCIRepository has a non-null cosign verify carrying only the three intended production identities.
The value survives a subsequent flux-operator reconcile and a subsequent prod deploy — verified by re-reading after both, not by observing it once.
A regression fails if root verification is absent, asserted against live or rendered shape, not against the source YAML — the whole point of this issue is that a correct source file did not produce a verified root.
If the root cause is in KSail, the fix lands there and platform pins the version carrying it.

Out of scope

Publication atomicity (#2930) and the DR signer path (#2840) — both already merged.

Rough size: small-to-medium — likely one KSail fix plus a version bump, but the discrimination step comes first and may redirect it.

Part of #2627.

⚠️ Possible Duplicate Issue(s)

Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922

🔗 Related PRs

#2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
#2625 - fix(deploy): synchronize production GHCR pull auth [merged]
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]
#2635 - fix(deploy): reboot stale nodes so containerd adopts refreshed GHCR auth [merged]
#2919 - fix(security): move Flux cosign verify to the path KSail reads [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Both hypotheses are refuted — it is a config gap here, not a KSail defect

Discriminated read-only on admin@prod via SSA field ownership, which settles it without touching anything:

$ kubectl -n flux-system get ocirepository flux-system -o json --show-managed-fields
spec.verify: ABSENT     spec.ref: {"tag":"latest"}

manager         | operation | time                 | owns spec.verify
flux-operator   | Apply     | 2026-07-13T21:22:56Z | -
source-controller | Update  | 2026-05-23T00:41:54Z | -
ksail           | Update   | 2026-08-03T23:21:35Z | -
source-controller | Update  | 2026-08-03T23:21:37Z | -

Hypothesis 1 — create-time-only no-op (the #2524 floating-IP class): FALSE. ksail is a field manager on this object, with an Update at 23:21:35Z — during the 🚀 Deploy to Prod that succeeded at 23:17:36Z. It ran.

Hypothesis 2 — applied, then reverted by flux-operator: FALSE. flux-operator's only Apply is 2026-07-13, three weeks before ksail's update. Nothing overwrote anything, and the source-controller update 2s later is the artifact/status write.

What ksail actually owns is the whole story:

{"f:metadata":{"f:annotations":{"f:reconcile.fluxcd.io/requestedAt":{}}}}

It owns only the reconcile-trigger annotation. It never touched spec at all — so it did not set spec.verify and did not fail to; it was never asked to.

Root cause: spec.workload.verify is absent from this repo's ksail.yaml. ensureVerifyIfConfigured (pkg/svc/installer/flux/resources.go:368) patches spec.verify only "when the user has configured it", so with no verify: key the call short-circuits before ensureVerify. grep -n verify ksail.yaml → no match; the workload: block carries only sourceDirectory, kustomizationFile, validation.

KSail is not at fault: it models the input (Workload.Flux.Verify), renders the patch (ocirepository_patcher.go), and CI pins a version that has both.

⚠️ This is NOT a safe one-line config addition — sequencing matters

Setting verify: makes Flux verify signatures at pull time on the root source. If root publication is not already producing signatures the cluster can verify, the root OCIRepository stops reconciling — a cluster-wide GitOps outage, not a degraded control.

So the ordering is: prove root artifacts are signed and verifiable first, then enable verify: — which is what #2922 ("configured but not enforced") is about, and why this belongs behind it rather than beside it. I have deliberately not shipped the config change on this evidence alone.

Suggested next step: re-scope this issue from "why is the renderer a no-op" (answered: it is not) to "add spec.workload.verify to ksail.yaml, gated on root signing being verified", and mark it blocked by #2922.

🤖 Generated by the Agentic Engineer

Mechanism identified — it is neither hypothesis

ksail cluster update never runs on prod. The step that invokes it is conditionally skipped, and has been on every prod deploy for at least four days. No KSail version can fix this, because the code that would apply the verify block is never executed.

Evidence

.github/actions/deploy-prod/action.yml gates the step that runs ./scripts/run-ksail-prod-with-pull-auth.sh cluster update:

if: >-
  steps.cilium_rollout_gate.outputs.active != 'true' &&
  steps.wait_flux_revision.outcome == 'success'

In the most recent prod deploy (merge_group run 30927795575, 2026-08-04T16:09Z) the log shows which arm failed:

Step	Outcome
`wait_flux_revision`	`outcome=success` ✅ (second arm satisfied)
`cilium_rollout_gate`	`Cilium homogeneous-device rollout gate active: Cluster Autoscaler is suspended.` ⇒ `active == 'true'` ❌
`__self.cluster_update`	`outcome=skipped; duration_ms=0`

Not a one-off — cluster_update outcome across prod deploys:

Deploy	`cluster_update`
2026-08-04T16:09Z (`30927795575`)	skipped
2026-08-04T15:33Z (`30924710728`)	skipped
2026-08-03T07:58Z (`30795581205`)	skipped
2026-08-01T11:04Z (`30696960482`)	skipped

The gate is still active right now: kube-system/cluster-autoscaler-hetzner-cluster-autoscaler reads 0/0 on prod.

Why the version pin was a red herring

ksail #6454 ("re-assert Flux artifact verification on cluster update") merged 2026-08-03T19:29Z. I verified containment rather than inferring it from dates — compare/v7.178.13...56bb93ba returns identical, so the fix shipped in v7.178.13, and live main already pins 7.178.14 across all four call sites. The fix is present and pinned; the command that carries it is skipped.

Both hypotheses in the body are now closed

Create-time only — not the mechanism. The path exists and is wired; it is gated off.
Applied then reverted — already disproven separately: SSA field ownership shows flux-operator applies the root OCIRepository without f:verify, so a patch is never reconciled out. (Also worth correcting the body's premise: the FluxInstance CRD has no spec.sync.verify property, so the "KSail cannot express sync.verify" framing describes the CRD, not a KSail typed-model gap — KSail patches the OCIRepository directly.)

The larger problem this uncovered

The gate's own comment calls it temporary and says a "reviewed rollout-completion or rollback artifact removes the gate". It has instead been suppressing ksail cluster update continuously for ≥4 days, and that step is the platform's only Talos machine-config sync. So the exposure is much wider than this issue: machine config has not been reconciled on prod since at least 2026-08-01, silently, with a green deploy each time.

Filing that separately rather than widening this issue — a temporary safety gate that indefinitely disables an unrelated reconcile loop, with no signal, is its own defect.

What this changes for the acceptance criteria

The first criterion (mechanism identified with evidence) is met by the above. The remaining criteria cannot be satisfied while the gate is active, because the applying command does not run — so the fix here is not a KSail change. It is either releasing the Cilium gate, or decoupling machine-config sync from it.

Leaving this claimed while I file the companion issue.

🤖 Generated by the Agentic Engineer

Still true two days later — and it has now survived ~109 successful merge-queue runs

Re-measured live on 2026-08-06 ~06:15Z. Nothing has changed, and the control still holds cleanly:

flux-system/flux-system   url=oci://ghcr.io/devantler-tech/platform/manifests
                          ref.tag=latest   interval=1m0s   spec.verify = NULL   ready=True

Every tenant source in the same cluster still carries a full cosign block — 5 of 5:
ascoachingogvaner, aws, doggy-countdown, github-config, wedding-app, each with
provider: cosign + matchOIDCIdentity. So the mechanism demonstrably works in this cluster; the
root source is the single one without it.

What is new since this was filed: there have been 109 successful merge_group CI runs between
2026-08-04T00:26Z and now. The original evidence rested on the value not surviving one deploy; it
has now not survived a hundred. (Stated precisely: those are successful merge-queue CI runs — the
🚀 Deploy to Prod job is path-gated, so not every one of the 109 reconciled prod. The point is that
no amount of ordinary successful delivery is converging on the declared state.)

One correction for whoever picks this up

spec.verify on the root is not observable from --branch main run history. The deploy job runs
on the merge_group event, so a gh run list --branch main sweep shows zero deploys and reads as
"nothing has shipped" — it hasn't; 128 main runs in the window are CodeQL/Release/TODOs/
Validate Main only. Query --event merge_group instead.

flux-system/aws-package is also verify=NULL, but that is a third-party upstream
(ghcr.io/flux-iac/aws-primitive-modules) pinned to an immutable tag — a different risk shape, and
out of scope here.

The security asymmetry is the part worth holding onto: the one source that is not verified is the
root of trust that deploys everything else, including all five verified tenants.

🤖 Generated by the Agentic Engineer

Consolidating into #2922 — same defect, and this issue had the better mechanism

#2960 is now decided,
and settling it made clear that this issue and #2922 are the same defect described from two angles:
the root flux-system OCIRepository has no spec.verify, and a fully successful prod deploy does not
change that.

Credit where it is due: this issue's diagnosis was the more accurate of the two. Its "Where the
value is lost" section already identified that flux-operator derives the root OCIRepository from
FluxInstance.spec.sync, and that a sync without verify therefore yields a source without
spec.verify. That is exactly right, and it is confirmed: spec.sync has 8 properties and the string
verify appears zero times in the whole v1 CRD schema. #2922 instead attributed it to a KSail
patch on the install-only path, which live field ownership shows is not what happens (KSail's only
field-manager entry on that resource is the reconcile.fluxcd.io/requestedAt annotation — no spec key).
That correction has been carried into #2922, along with this issue's evidence that the gap survives a
successful deploy.

Keeping #2922 as the single tracking issue because it is older, is typed Security (correct for a
supply-chain control, where this one is typed Bug), and carries the acceptance criteria and the now
amended mechanism. Nothing from here is lost:

the derivation mechanism → now the basis of Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922's chosen route;
"survives a full successful deploy" (🚀 Deploy to Prod green at 2026-08-03T23:17:36Z, root source
still spec.verify: null on 2026-08-04) → the durability evidence on Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922;
the tenant-vs-root contrast (5 tenant OCIRepositories carry verify and are Ready=True) → cited on
Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922 and in Spike: can a non-operator field manager hold spec.verify on the root OCIRepository? #2960 as evidence the mechanism functions in this cluster.

Closing as a duplicate of #2922. Reopen if you would rather track the observed symptom separately from
the fix.

Navigation Menu

Repository navigation

Root OCIRepository verification does not survive onto the live cluster #2938

Evidence

Where the value is lost

Hypotheses to discriminate (not yet proven)

Acceptance criteria

Out of scope

Both hypotheses are refuted — it is a config gap here, not a KSail defect

⚠️ This is NOT a safe one-line config addition — sequencing matters

Mechanism identified — it is neither hypothesis

Evidence

Why the version pin was a red herring

Both hypotheses in the body are now closed

The larger problem this uncovered

What this changes for the acceptance criteria

Still true two days later — and it has now survived ~109 successful merge-queue runs

One correction for whoever picks this up

Consolidating into #2922 — same defect, and this issue had the better mechanism

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Root OCIRepository verification does not survive onto the live cluster #2938

Description

Evidence

Where the value is lost

Hypotheses to discriminate (not yet proven)

Acceptance criteria

Out of scope

Activity

coderabbitai commented on Aug 4, 2026

devantler commented on Aug 4, 2026

Both hypotheses are refuted — it is a config gap here, not a KSail defect

⚠️ This is NOT a safe one-line config addition — sequencing matters

devantler commented on Aug 4, 2026

Mechanism identified — it is neither hypothesis

Evidence

Why the version pin was a red herring

Both hypotheses in the body are now closed

The larger problem this uncovered

What this changes for the acceptance criteria

devantler commented on Aug 6, 2026

Still true two days later — and it has now survived ~109 successful merge-queue runs

One correction for whoever picks this up

devantler commented on Aug 6, 2026

Consolidating into #2922 — same defect, and this issue had the better mechanism

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions