Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository

Evidence

#2919 merged the cosign policy onto the key KSail actually reads
(spec.workload.flux.verify → consumed at pkg/svc/installer/flux/resources.go:378). The
merge-queue deploy for it succeeded (🚀 Deploy to Prod green) and published a new artifact.

The live resource is still unverified afterwards. Read from the prod cluster after that deploy:

ocirepository/flux-system in flux-system — spec.verify absent, Ready=True,
serving a new revision.
Resource generation: 3, created 2026-05-23 — it is long-lived, not recreated by the deploy.

So the config half is now correct and the enforcement half is still off. The gap #2627 tracks is
not closed by #2919 alone.

Why it does not take effect

Two contributing facts, both checked:

KSail applies verification by patching the OCIRepository, from
ensureVerifyIfConfigured — which sits on the Flux install / SetupInstance path
("creates the FluxInstance CR and configures OCIRepository settings"), not on the path a
routine workload push + cluster update deploy exercises against an existing cluster.
The flux-system OCIRepository is generated by the flux-operator FluxInstance, whose
spec.sync (kind: OCIRepository, ref: latest, url: …/platform/manifests) carries no
verify field. A resource the operator reconciles from a spec without verify is liable to have a
direct patch reconciled back out.

This is the create-time-only class already seen in #2524: a field that validates green, is read by
the consumer, and never reaches the running system because the update path does not apply it.

Expected behaviour

After a normal prod deploy, ocirepository/flux-system carries the configured
spec.verify and Flux enforces signatures on the infrastructure source.

Acceptance criteria

A routine deploy (not a bootstrap) results in spec.verify present on the live
flux-system OCIRepository.
The setting survives a flux-operator reconcile — i.e. it is expressed where the operator will
keep it (most likely FluxInstance.spec.sync) rather than as a patch it can revert.
Verified live, by reading the resource back after a deploy, not by a green validate.
A green ksail validate is explicitly not sufficient evidence here.
A negative control: an artifact signed by a non-pinned ref is rejected, so the control is
shown to be doing something rather than merely present.

Size

Small-to-medium — most likely moving the verify block into the FluxInstance sync spec (KSail-side
support may be needed), plus the live read-back proof.

Part of #2627. Follow-up to #2919 / #2850 — #2850 asked to move the key and activate; the move
landed, the activation did not.

🔗 Related PRs

#2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
#2399 - feat: accept devantler-tech/actions cosign subjects during the publish-workflow repoint [merged]
#2457 - fix(talos): accept the actions publish-app identity at node image verification [merged]
#2625 - fix(deploy): synchronize production GHCR pull auth [merged]
#2633 - fix(deploy): verify incoming image on Talos nodes [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

The proposed direction is not achievable — the fix is in KSail

Acceptance criterion 2 suggests expressing verify "most likely in FluxInstance.spec.sync". That path does not exist:

The installed flux-operator is v0.50.0, and its FluxInstance v1 CRD schema lists spec.sync properties as exactly interval, kind, name, path, provider, pullSecret, ref, url — no verify. Read from the live CRD, not from docs.
spec.kustomize.patches is not a way around it either: the operator's documentation states the patches apply to the Flux controllers, and all six patches this repo already uses target Deployment. Patching an OCIRepository through it is undocumented, and shipping an undocumented mechanism for a control that already failed silently once is precisely the wrong move here.

So the block cannot live where the operator would keep it, and criterion 2 needs rewording.

Root cause, one level deeper than filed

The issue reasons that ensureVerifyIfConfigured "sits on the install path, not the path a routine deploy exercises". That is nearly right, and the precise version matters. SetupInstance is reachable from ksail cluster update — but only through reconcileWorkloadTag and reconcileFluxVersion, which are change-triggered handlers. And KSail's diff engine emits changes for cluster.workload.tag and cluster.workload.flux.distributionVersion and nothing for verify.

That explains the exact observed history: #2919 changed only the verify block, so the diff engine saw no change, no handler ran, nothing was patched — and the deploy was green because nothing failed. It also explains why verification would have appeared to work had any other field changed in the same deploy, which is the kind of coincidence that makes this class hard to catch.

Fix

devantler-tech/ksail#6454 (draft) — KSail detects that a configured verify is missing from the live OCIRepository and re-asserts it, the same way it already handles a rotated registry credential the structural diff cannot see.

Confirmed still live at 08:35Z: root flux-system/flux-system has spec.verify null while all five tenant artifacts carry cosign.

This issue stays open

Criterion 3 requires the live read-back after a routine deploy, which needs #6454 merged, released, and the pinned KSAIL_VERSION (currently 7.178.11) bumped here. Criterion 4's negative control is likewise still outstanding. A green validate is explicitly not sufficient — that is the whole point of this issue — so it should not be closed on the KSail merge alone.

Separately filed while in there: devantler-tech/ksail#6455, on SetupInstance replacing the whole FluxInstance spec from a type that models only two of its eight fields. Relevant to this repo because the FluxInstance here carries six load-bearing kustomize.patches.

Correction — hypothesis 2 is false, and acceptance criterion 2 is unachievable as written

I went to build the platform half of this and found the diagnosis above is half wrong. Recording it
before anyone spends a run chasing the fix it points at.

`FluxInstance.spec.sync` has no verify field

Read from the live CRD schema (fluxinstances.fluxcd.controlplane.io), spec.sync accepts exactly:
interval, kind, name, path, provider, pullSecret, ref, url. There is no signature or
cosign field. The upstream CRD reference agrees. So "express it in FluxInstance.spec.sync" — the
remediation criterion 2 names — cannot be done at all, at this operator version.

spec.kustomize.patches is not a way round it either: upstream documents it as patches "to apply to
the Flux controllers", i.e. the component manifests, not the sync-generated source. (Docs-derived,
not tested — but moot given the next section.)

The operator does not revert a direct patch — two independent reasons

Hypothesis 2 says a direct patch is "liable to be reconciled back out". The live resource says
otherwise:

It carries kustomize.toolkit.fluxcd.io/ssa: Ignore (set by flux-operator itself, alongside
prune: Disabled). Upstream defines Ignore as instructing the controller to skip applying the
resource — it will not create, update, or manage it. A resource under Ignore is not fought over.
Field ownership leaves verify unclaimed. flux-operator owns spec via server-side
Apply, and the fields it owns are interval, provider, ref, secretRef, url. It does
not own verify. An SSA apply removes only fields that that manager previously owned and has
stopped setting, so a spec.verify written by another manager is not pruned by it.

What the evidence does support — hypothesis 1, and more sharply than stated

There is a ksail field manager on the live resource, from today. It owns no spec fields at all
— the only thing it set is the reconcile.fluxcd.io/requestedAt annotation. So on a routine deploy
KSail's sole interaction with this source is asking Flux to reconcile it; it never attempts to
write spec.verify. That is exactly the create-time-only gap hypothesis 1 describes, and it is the
whole of the defect.

Consequence

The fix is the KSail-side one already in flight (devantler-tech/ksail#6454): detect the drift and
re-assert verification on cluster update. A platform-side change is probably not needed at all
— there is nowhere better to put this, and the patch is durable where it lands.

Proposed criteria replacing #2:

~~expressed in FluxInstance.spec.sync~~ — not a supported field; the setting survives a
flux-operator reconcile, evidenced by a live read-back some time after a deploy rather than
immediately after it.

Criteria 1, 3 and 4 stand unchanged, and criterion 3's live read-back is what will actually settle
this: the reasoning above is derived from field ownership and the applier annotation, not from
an applied-and-observed patch, because this lane holds read-only cluster access. Treat it as a
corrected diagnosis, not as proof of the fix.

fix(cluster): re-assert Flux artifact verification on cluster update (#…

The KSail-side half of this landed — this issue is not resolved by it

devantler-tech/ksail#6454 merged at 19:29:30Z (56bb93ba): "fix(cluster): re-assert Flux artifact verification on cluster update". It makes reconcileFluxVerify re-assert the FluxInstance so a configured spec.workload.flux.verify reaches an already-bootstrapped cluster, which is the mechanism this issue says is missing. Its code comment cites this issue by number.

Why this does not close it. That fix lives in KSail, and this issue is about this repository's deploy path. It becomes real here only once a KSail release carrying 56bb93ba exists and the platform adopts it. Neither has happened, so the live OCIRepository is still unverified today. Closing on the strength of an upstream merge would be exactly the "validated green ≠ works" mistake — the enacting path has not been traced end-to-end on this side.

I verified the KSail behaviour behaviourally rather than by reading it: the reassertion is pinned by a test whose ablation I ran (guard removed → the second handler recomputes and attempts a real SetupInstance upsert; guard present → memoized no-op), so the re-assert genuinely happens once per update pass.

Claim state: the assignment here dates to 08:34:42Z — about 11 hours ago — and branch claude/root-oci-verify-2922 (tip 10f47989) carries no pull request. Under the claim protocol that lease is long expired, so this is fair game for takeover. Anyone picking it up should start a fresh branch rather than reuse that one, and re-assign to reset the lease clock.

Suggested next step: confirm which KSail version the platform pins, and whether 56bb93ba is in a released tag yet; the follow-up here is the version bump plus an end-to-end check that the live OCIRepository actually gains its spec.verify block.

The proposed direction is not achievable — measured, then verified against upstream docs

The issue proposes expressing the policy at FluxInstance.spec.sync ("most likely") so the operator keeps it. That field does not exist.

Measured on the live prod cluster today (read-only):

kubectl explain fluxinstance.spec.sync returns exactly: interval, kind, name, path, provider, ref, url. No verify.
The string "verify" does not appear anywhere in the installed fluxinstances.fluxcd.controlplane.io CRD schema.
ocirepository/flux-system still has spec.verify absent at generation: 3 — the defect this issue reports is confirmed live.
OCIRepository.spec.verify does exist in the source-controller CRD, so the gap is purely in how the resource is produced.

Verified against upstream documentation (docs only, no repository accessed): spec.sync is documented with kind, url, ref, path, pullSecret, provider, interval, name — no verify, and no mention of Cosign verification for the sync source.

Why `spec.kustomize.patches` is not the workaround

It was the obvious candidate, so I checked it rather than assuming. Upstream documents it as "the Kustomize patches to apply to the Flux controllers". Consistent with that, all 8 patches configured on our live FluxInstance target Deployment kinds only — there is no live instance of a patch reaching a sync-generated resource, so there is no evidence it is even a supported target.

What this means for the acceptance criteria

Criterion 2 as written — "expressed where the operator will keep it (most likely FluxInstance.spec.sync)" — is unsatisfiable. The remaining directions, neither yet chosen:

Take the root source out of operator ownership — stop using spec.sync for it and declare the OCIRepository + Kustomization ourselves, where spec.verify is ours to set and the operator will not reconcile it away.
Upstream support — ask flux-operator to carry verify through spec.sync. Cross-org contribution, so it needs maintainer approval and an AI-contribution-policy check first, and it does not unblock us on our own timeline.

I have deliberately not started either. Option 1 changes how the cluster syncs itself: getting it wrong is a full GitOps outage on prod, and the operator would stop managing a resource it currently owns. That is a decision to record before code, not a change to slip into a tick — and this issue is now correctly a decision, not an implementation.

The rest of the issue stands: the defect is real, still live, and criteria 1, 3 and 4 (routine deploy, live read-back, negative control) remain exactly right whichever direction is chosen.

fix(cluster): detect and repair Flux artifact-verification drift on u…

Live re-verification (2026-08-05 ~05:10Z) — the defect is current, and one of the two stated causes is wrong

Re-read from prod before picking this up. The gap is real and unchanged:

ocirepository/flux-system  generation: 3  spec.verify: null
url: oci://ghcr.io/devantler-tech/platform/manifests   ref.tag: latest   Ready: True

Two new pieces of evidence materially change the fix design.

1. `FluxInstance.spec.sync` CANNOT express verification — the field does not exist

The served CRD schema admits exactly these keys under spec.sync:

interval, kind, name, path, provider, pullSecret, ref, url

There is no verify property. So "have the flux-operator declare verify in spec.sync" is not a
fix that can be written — any design that assumed it would fail at apply time against a nonexistent
field. That closes the operator-declares-it direction outright.

2. Contributing fact #2 is refuted — flux-operator will NOT prune a foreign `verify`

The concern was that "a resource the operator reconciles from a spec without verify is liable to have
a direct patch reconciled back out". Server-side-apply field ownership says otherwise:

Manager	Operation	`spec` subfields owned
`flux-operator`	Apply	`interval`, `provider`, `ref`, `secretRef`, `url`
`source-controller`	Update	—
`ksail`	Update	—

An Apply manager prunes only fields it previously owned and has since dropped. flux-operator
has never owned f:verify, so a verify written by another manager is outside its ownership set and
does not get reconciled away. A direct patch is therefore both the only available mechanism and a
durable one.

What that leaves

Contributing fact #1 stands alone as the cause: nothing writes verify on the path a routine deploy
exercises. That mechanism is now tracked precisely in
ksail#6480 — checkFluxVerifyDrift is wired
into the Updater diff path but not into computeSpecOnlyDiff, so cluster diff never previews the
repair and non-Updater provisioners never detect it at all.

Note that platform CI pins KSAIL_VERSION 7.178.14, which post-dates ksail#6454 (merged
2026-08-03T19:29Z), so the deployed KSail does carry the update-path re-assertion. The remaining
question for this issue is why the deploy's cluster update leg is not landing it on prod — not
whether the capability exists.

I am picking up ksail#6480 now; this issue stays open for the prod-side enforcement proof.

Blocked on #2963 — re-verified live today, and two of this issue's own premises are now settled

Full evidence posted on #2963; the parts that change this issue:

Contributing fact 2 is refuted. The worry that a direct patch would be "reconciled back out" by the operator does not hold: the operator owns only interval/provider/ref/secretRef/url on that resource under Server-Side Apply, and SSA prunes only fields an applier previously owned — so a verify written by a different manager survives. Separately, the proposed remedy is unimplementable as written: the FluxInstance CRD has no verify key under spec.sync, so the operator cannot declare it at all.

That leaves contributing fact 1 as the sole cause — nothing writes the field on the deploy path. Specifically, the step that would apply it (cluster_update) is skipped on every prod deploy by the Cilium rollout gate, including the most recent one. Live prod still reads spec.verify: null.

Acceptance criterion 2 needs rewording. As written it requires the setting to live in FluxInstance.spec.sync, which that CRD cannot express. The intent — "survives an operator reconcile" — is already satisfied by the SSA ownership above, so the criterion should ask for that property rather than that location.

The risk argument has also flipped in this issue's favour. The reason to be cautious about enabling verification was that verifying an unsigned artifact would stop the root source reconciling. The publication chain is now signed and attested before latest moves (verify_evidence precedes promote_latest, all green on the latest deploy), so that is no longer the barrier.

Not proposing a workflow change to unblock this: the gate is deliberate and guards a real scale-up race, so removing it would trade one control for another. The unblock is the operator action named in #2963.

I could not independently re-verify the artifact signature from the agent host — the available credential lacks package-read scope, so that check is deferred rather than claimed.

Re-verified live: fact 1 confirmed, fact 2 refuted, and acceptance criterion 2 is unsatisfiable

Checked against the live prod cluster and the released KSail this platform actually deploys with
(KSAIL_VERSION: 7.178.14, pinned in .github/actions/deploy-prod/action.yml).

Still reproducing

ocirepository/flux-system — spec.verify absent, generation: 3. Unchanged since this was
filed, so the gap is open.

Contributing fact 1 — CONFIRMED

At tag v7.178.14, ensureVerifyIfConfigured has exactly one call site
(pkg/svc/installer/flux/resources.go:152), inside setupFluxCore (declared line 85) — the
install / SetupInstance path. Nothing on a routine workload push + cluster update deploy
calls it. That is the whole defect.

Contributing fact 2 — REFUTED

The concern was that a direct patch would "lose the reconcile fight" with flux-operator. It would
not. Server-side-apply field ownership on the live resource shows flux-operator's Apply entry
owning only:

f:interval, f:provider, f:ref, f:secretRef, f:url

f:verify is not among them. An operator only prunes fields it owns, so a patch written to
spec.verify by another manager survives its reconcile. This removes the reason the fix looked
risky.

Acceptance criterion 2 cannot be met as written

FluxInstance.spec.sync has no verify key. Read from the installed CRD
(fluxinstances.fluxcd.controlplane.io), the complete property set is:

interval, kind, name, path, provider, pullSecret, ref, url

So flux-operator cannot be made to declare verification, and any work that starts from this
criterion is building something the CRD does not support. Combined with the refutation above, the
criterion is also unnecessary: durability does not require operator ownership, because the operator
will not prune a field it does not own.

Suggested replacement for criterion 2

The routine deploy path (not bootstrap) applies spec.verify to the live
flux-system OCIRepository.
The applied value survives a subsequent flux-operator reconcile — verified by reading the
resource back after one, rather than assumed from ownership.

The other three criteria stand as written, including the negative control.

Not the fix

ksail#6491 is adjacent but does not close this: it makes cluster diff preview verification
drift that the apply path would perform. It changes what is shown, not what is applied on the
install-only path above.

Root cause: the fix already shipped and is being suppressed, not missing

Following up on my previous comment. The remaining question was what fails to apply verification on
a routine deploy. It is not a missing capability — the repair exists, ships in the KSail prod runs,
and is skipped every deploy.

The chain, each link checked

The repair exists. reconcileFluxVerify (pkg/cli/cmd/cluster/reconciler.go) "re-asserts the
OCIRepository's spec.verify block so a configured spec.workload.flux.verify reaches a cluster
that is already installed" — added by ksail#6454, and its comment names this issue.
Prod already has it. It first shipped in v7.178.13; prod pins v7.178.14
(deploy-prod/action.yml). So the deployed CLI has carried the fix since the 08-04 bump (chore(deps): update dependency devantler-tech/ksail to v7.178.14 #2943).
It only runs via ksail cluster update. That is the sole deploy-path entry point to the
reconciler.
That step is skipped whenever the Cilium rollout gate is active. The 🔄 Update cluster step's
condition is steps.cilium_rollout_gate.outputs.active != 'true' && ….
The gate is active right now. Both legs are true on main: the component is listed in the
controllers kustomization, and the component still sets type: OnDelete.

Empirical confirmation, not inference

From today's green merge-group prod deploy (run 30987841587, the deploy of #2964):

end-action id=__self.cilium_rollout_gate;outcome=success
end-action id=__self.cluster_update;outcome=skipped

Cilium homogeneous-device rollout gate active: Cluster Autoscaler is suspended.

The gate has been active since 2026-07-28 (#2813 introduced OnDelete). 81 successful
merge-group prod deploys have completed since that date, every one of them skipping this step.
The count is complete rather than a lower bound — the sampled window reaches back to 2026-07-04.

What this means for this issue

There is no code to write here. spec.verify will be applied by the already-deployed KSail on
the first prod deploy where cluster update is not gated off. This issue is therefore blocked on the
Cilium homogeneous-device rollout completing (or being rolled back), which is the suppression #2951
tracks — it is not blocked on the verification work itself.

Two consequences worth recording:

The apparent silence is one gate, not several bugs. Any other repair that only cluster update
performs has been equally suppressed for those 81 deploys. That is worth checking as its own
question rather than diagnosing each symptom separately.
The negative control in criterion 4 cannot be exercised until the gate releases, so this issue
should not be picked up as implementable work before then.

Suggested handling

Mark this blocked by #2951, leave the criteria as amended in my previous comment, and re-verify live
(spec.verify non-null on ocirepository/flux-system) on the first ungated prod deploy. If it is
still null then, the diagnosis above is wrong and the issue becomes implementable again — that read
is the cheap decisive test.

Correction to the two dates/counts in my previous comment. I derived the activation commit with
git log -1 -S, which returns the most recent commit touching that string, not the one that
introduced it — so I named a later edit (#2813, 2026-07-28) as the activation.

The gate actually went active with dc51cf6 "feat(cilium): activate homogeneous device selection"
(#2793), committed 2026-07-25T22:56:37Z — which agrees with #2963 rather than contradicting it.
Recounting from that timestamp: 92 successful merge-group prod deploys have skipped
cluster_update, not 81. Still a complete count, not a lower bound (sample reaches back to
2026-07-04).

Everything else in that comment stands — the causal chain, the end-action evidence from run
30987841587, and the conclusion that this issue is blocked rather than unimplemented. The
already-recorded blocked by #2963 is the accurate link; #2951 is closed and covered reporting only.

Unblocked — #2960 is decided. The route is `FluxInstance.spec.kustomize.patches`.

The spike this issue was gated on is closed with a recorded
decision. Summary of
what it changes here, because it touches this issue's mechanism and one of its acceptance criteria.

AC-2 is falsified as written — please read it as a deviation, not as pending work

FluxInstance.spec.sync has no verify field, and cannot grow one here: the CRD's spec.sync
has exactly 8 properties (interval, kind, name, path, provider, pullSecret, ref, url) and the string
verify appears zero times anywhere in the v1 schema at flux-operator v0.50.0. So the
parenthetical guess is not merely unimplemented — it is unimplementable without an upstream change,
which #2960 keeps out of scope.

The intent of AC-2 is still exactly right, and is satisfied a different way: the block goes in
spec.kustomize.patches, so flux-operator itself writes spec.verify onto the OCIRepository
it generates. That is the operator's own documented mechanism for this — its reference example is
verbatim target: {kind: OCIRepository, name: flux-system} with op: add, path: /spec/verify — and
the operator already owns 5 spec keys on that resource via Apply, so verify joins fields it owns.
There is no patch for it to revert and no field contention. Anyone draining AC-2 as written would go
looking for a sync.verify field that does not exist.

This issue's stated cause is right in conclusion, sharper in mechanism

The body says KSail "applies verification by patching the OCIRepository" on the install path only.
Live field ownership refines that: KSail's only field-manager entry on ocirepository/flux-system
is metadata.annotations["reconcile.fluxcd.io/requestedAt"]. It writes no spec key at all on the
deploy path — so this is not a patch that lands and gets reverted, and not a patch that is skipped;
there is no spec write attempted. Consequence for scoping: "make KSail patch on the update path too"
is not a sufficient fix — it would add a new writer that then contends with the operator forever.
Nothing needs to change in KSail for this.

Relatedly, the comment above the verify block in ksail.prod.yaml is now known to be wrong and
should be corrected as part of this work — it states the OCIRepository is one "that KSail generates
and owns" and that "a hand-written override would lose the reconcile fight". flux-operator generates
and owns it, and the way to win that fight is to go through the operator. That comment is what makes
the ksail.prod.yaml route look correct, so leaving it in place invites the same wrong turn again.

New, mandatory requirement: assert `spec.verify` by reading the live resource back

🔴 A mistargeted kustomize.patches entry is a silent no-op — proven in #2960 with two ablations
that both exited 0 with no error and no warning, rendering no verify at all. So a green
validate, a clean FluxInstance reconcile, and a successful deploy are all consistent with the root
source still being unverified. This issue's AC-3 already demands a live read-back; #2960's finding is
why that is load-bearing rather than belt-and-braces. The general detection gap is filed separately as
#2996.

Suggested amended acceptance criteria

spec.verify is expressed as a FluxInstance.spec.kustomize.patches entry in
k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml, carrying the
three identities already maintained in ksail.prod.yaml.
After the deploy, kubectl get ocirepository flux-system -n flux-system shows spec.verify
present and flux-operator listed as its field manager for that key — read back live, not
inferred from a green validate.
The root source is Ready=True after the change (i.e. the current artifact passes the block).
The misleading ksail.prod.yaml comment is corrected in the same PR.
Negative control (unchanged, still required): an artifact signed by a non-pinned ref is rejected.

Preconditions before implementing — the blast radius has not changed

A wrong verify block stops the root GitOps source, which halts all delivery to prod and cannot be
repaired through GitOps. Two things should be true before this merges:

The three signer identities are re-confirmed against the current artifact. I could not do this
from the agent host: the artifact is in a private registry and the host credential is scoped without
registry read, and my first attempt returned identical DENIED output for the positive and
the negative control — which measures nothing, so I am not reporting it as a pass. The identities
have a prior in-repo confirmation note, but that is not a current measurement.
A rollback path is written down before the change, not during an incident — the repair for a bad
block is a manual edit of the live FluxInstance (or ksail cluster update), because GitOps
itself will be the thing that is stuck.

Given (1), the safest shape is to land the patch only once someone with registry read has
re-verified the current artifact against all three identities. That is a small, concrete, delegable
step — not another spike.

I am not claiming this issue in this run; it stays unassigned and available. Recording the decision
and the corrected mechanism so whoever picks it up does not re-derive any of it.

Root cause measured — and the recorded hypothesis is refuted

The explanation currently written into ksail.prod.yaml (and inherited by #2627) is that the deploy's
ksail cluster update is being skipped by #2951, so "once #2951 releases that gate this may simply
start working". #2951 closed on 2026-08-04T23:48Z and this did not start working. Measured today:

Live root source still unverified — ocirepository/flux-system spec.verify is null
(control on the same read: ref.tag=latest, interval=1m0s, so the object reads fine).
12 successful merge_group CI runs have deployed prod since Cilium rollout gate silently suppresses Talos machine-config sync indefinitely #2951 closed, the most recent at
2026-08-06T10:54Z.

So the absence is not a delivery lag. The reason is more specific:

`cluster_update` is still skipped — #2951 made the suppression visible, it did not lift it

From the latest prod deploy (run 31095082643):

##[start-action display=🔄 Update cluster (Talos machine config);id=__self.cluster_update]
##[end-action   id=__self.cluster_update;outcome=skipped;conclusion=skipped;duration_ms=0]

Its condition is steps.cilium_rollout_gate.outputs.active != 'true' && …, and the same run logs
Cilium homogeneous-device rollout gate active: Cluster Autoscaler is suspended with
CILIUM_ROLLOUT_GATE_ACTIVE: true. The reporting step #2951 added then warns:

Talos machine-config sync has been suppressed by the Cilium rollout gate for 11 days (threshold 7).

cluster update is the only thing in the pipeline that would push spec.workload.flux.verify onto the
live OCIRepository. It has not run for 11 days, so the config is correct, CI-validated, and simply
never applied.

The real blocker is therefore the Cilium homogeneous-device rollout gate, not the KSail config path
and not #2951. Per the step's own comment the gate clears on "a reviewed rollout-completion or
rollback artifact" — that artifact is what unblocks this issue, and it is now 4 days past its own
7-day warning threshold.

Consequence for #2627

Two of that issue's acceptance criteria look already satisfied and should not be rebuilt. The deploy
composite's publication sequence is now genuinely staged-then-promoted:

staging_reference → push_staging → cosign_sign → generate_sbom → attest_sbom →
attest_provenance → verify_evidence → promote_latest

latest moves only after signing, attestation and an evidence check — which is what
"Flux cannot observe a new artifact until signature and required attestations are complete" and
"any failed push/sign/attest leaves the last verified artifact active" ask for. Its static
config-nesting criterion is also done: scripts/validate-flux-verify exists with unit tests and is
CI-wired (go test ./scripts/validate-flux-verify + go run … ksail.prod.yaml), and it explicitly
rejects the spec.cluster.verify mis-nesting that caused this.

What remains genuinely open on #2627 is this issue (live enforcement, gated as above) and the DR
signed-artifact path.

Note on state

I did not claim this issue. There is a stale claude/root-oci-verify-2922 branch from 2026-08-05
carrying only an empty chore: claim #2922 commit (ahead 1, 0 files changed, 45 behind main)
with no PR and no current assignee — abandoned work, not work in progress. Left in place rather than
deleted, per the report-only rule for no-PR branches.

Fix shipped as #3001 — and the framing in this issue's own body was half wrong

AC-2 guessed FluxInstance.spec.sync. That field does not exist: the v1 CRD's spec.sync has
exactly 8 properties and the string verify appears zero times anywhere in the schema
(flux-operator v0.50.0). The route is spec.kustomize.patches, which patches the resources the
operator generates — flux-operator's docs carry a verbatim example titled "OCI sync with Cosign
verification" targeting kind: OCIRepository, name: flux-system with op: add, path: /spec/verify.
Resolved on the #2960 spike; this issue is the delivery.

Why that matters beyond the field name: because the operator applies the patch, the field lands
under the manager that already owns the resource. There is no second writer, no field contention and
no SSA-pruning question — which is what makes AC-2 ("survives a flux-operator reconcile") true by
construction rather than by hope.

It also means this was never blocked on the Cilium rollout gate. The gate suppresses
ksail cluster update; the FluxInstance is reconciled by kustomize-controller as ordinary GitOps.
The memory note that recorded this as gate-blocked was reasoning from the KSail path only.

AC status

AC	State
1 — routine deploy leaves `spec.verify` present	⏳ post-merge; lands on the next prod deploy
2 — survives an operator reconcile	✅ by construction (operator is the writer)
3 — verified live, not by a green validate	⏳ post-merge read-back
4 — the control is shown to be doing something	✅ see below

AC-4 as written asked for a wrongly-signed artifact. That is not producible without publishing a bad
artifact at the real root source, so it is answered instead by ablating the control itself: the
new check fires on a 2-line target change and passes again on revert, and separately a mistargeted
patch was shown to render nothing at exit 0 with no warning. That is the property AC-4 was after —
evidence the control is not inert.

Two things #3001 adds beyond the field

A mistargeted patch is a silent no-op. Wrong kind or wrong name ⇒ nothing rendered, exit 0, no
warning, FluxInstance reconciles cleanly, deploy succeeds. A security control delivered this way is
unfalsifiable from the file alone, so the pre-merge check now asserts the target, not just the
presence. The general case across all 7 patches is A FluxInstance kustomize patch that matches no target is silently discarded #2996.
The policy exists in two places and nothing compared them. The cluster config covers bootstrap,
the patch covers the running cluster, and neither covers the other's window — so both are needed.
But tightening one would have silently left the other trusting a publisher it should not. The check
now fails the build when the two signer lists differ.

What the next run must do

Do not close this on merge — that is why #3001 says Part of, not Fixes. After the next prod
deploy:

kubectl get ocirepository/flux-system -n flux-system -o yaml | grep -A5 verify

spec.verify present ⇒ AC-1 and AC-3 met, close. Absent ⇒ the patch did not take effect and the
FluxInstance should be read back for its applied patch count (expect 7, was 6).

Blocking constraint for this issue: a multi-subject cosign allow-list is NOT expressible in Flux

The approach #3001 took for this issue — three pinned matchOIDCIdentity entries — is rejected by the
verifier and has been failing on production since 14:29:01Z today (it reached prod through a
speculative merge-group deploy, not a merge; see #3005).

Flux documents matchOIDCIdentity as an OR-evaluated list, but sigstore/cosign supports a single
identity and returns unsupported: multiple identities are not supported at this time for a set. The
list fails closed for every entry, so the control verifies nothing while looking correct.

Any fix for this issue must therefore collapse the allow-list into one matcher with an alternation
in the subject regexp, and must be demonstrated verifying a real signed artifact — static and
rendered-shape validation provably cannot catch this class. Full evidence and the candidate regex:
#3001 (comment)

Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922

Description

Evidence

Why it does not take effect

Expected behaviour

Acceptance criteria

Size

Activity

coderabbitai commented on Aug 3, 2026

devantler commented on Aug 3, 2026

The proposed direction is not achievable — the fix is in KSail

Root cause, one level deeper than filed

Fix

This issue stays open

devantler commented on Aug 3, 2026

Correction — hypothesis 2 is false, and acceptance criterion 2 is unachievable as written

FluxInstance.spec.sync has no verify field

The operator does not revert a direct patch — two independent reasons

What the evidence does support — hypothesis 1, and more sharply than stated

Consequence

devantler commented on Aug 3, 2026

The KSail-side half of this landed — this issue is not resolved by it

devantler commented on Aug 4, 2026

The proposed direction is not achievable — measured, then verified against upstream docs

Why spec.kustomize.patches is not the workaround

What this means for the acceptance criteria

18 remaining items

devantler commented on Aug 5, 2026

Live re-verification (2026-08-05 ~05:10Z) — the defect is current, and one of the two stated causes is wrong

1. FluxInstance.spec.sync CANNOT express verification — the field does not exist

2. Contributing fact #2 is refuted — flux-operator will NOT prune a foreign verify

What that leaves

devantler commented on Aug 5, 2026

Blocked on #2963 — re-verified live today, and two of this issue's own premises are now settled

devantler commented on Aug 5, 2026

Re-verified live: fact 1 confirmed, fact 2 refuted, and acceptance criterion 2 is unsatisfiable

Still reproducing

Contributing fact 1 — CONFIRMED

Contributing fact 2 — REFUTED

Acceptance criterion 2 cannot be met as written

Suggested replacement for criterion 2

Not the fix

devantler commented on Aug 5, 2026

Root cause: the fix already shipped and is being suppressed, not missing

The chain, each link checked

Empirical confirmation, not inference

What this means for this issue

Suggested handling

devantler commented on Aug 5, 2026

devantler commented on Aug 6, 2026

Unblocked — #2960 is decided. The route is FluxInstance.spec.kustomize.patches.

AC-2 is falsified as written — please read it as a deviation, not as pending work

This issue's stated cause is right in conclusion, sharper in mechanism

New, mandatory requirement: assert spec.verify by reading the live resource back

Suggested amended acceptance criteria

Preconditions before implementing — the blast radius has not changed

devantler commented on Aug 6, 2026

Root cause measured — and the recorded hypothesis is refuted

cluster_update is still skipped — #2951 made the suppression visible, it did not lift it

Consequence for #2627

Note on state

devantler commented on Aug 6, 2026

Fix shipped as #3001 — and the framing in this issue's own body was half wrong

AC status

Two things #3001 adds beyond the field

What the next run must do

devantler commented on Aug 6, 2026

Blocking constraint for this issue: a multi-subject cosign allow-list is NOT expressible in Flux

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

`FluxInstance.spec.sync` has no verify field

Why `spec.kustomize.patches` is not the workaround

1. `FluxInstance.spec.sync` CANNOT express verification — the field does not exist

2. Contributing fact #2 is refuted — flux-operator will NOT prune a foreign `verify`

Unblocked — #2960 is decided. The route is `FluxInstance.spec.kustomize.patches`.

New, mandatory requirement: assert `spec.verify` by reading the live resource back

`cluster_update` is still skipped — #2951 made the suppression visible, it did not lift it