Spike: can a non-operator field manager hold spec.verify on the root OCIRepository?

🤖 Generated by the Agentic Engineer

Evidence

#2922 established, by Server-Side Apply field ownership on the live cluster, that flux-operator
owns exactly five fields under ocirepository/flux-system's spec — interval, provider, ref,
secretRef, url — and has never owned verify.

SSA prunes only the fields an applier previously owned. That implies a spec.verify written by a
different field manager should survive the operator's reconciles indefinitely, which would close
the enforcement gap #2922 describes without needing an upstream FluxInstance.spec.sync.verify
field (that field does not exist in flux-operator v0.50.0, checked against the live CRD).

That implication is reasoned from the ownership readings, not observed. It is the single
assumption the whole remediation rests on, and #2922 must not be implemented until it is tested.

Why this needs its own timeboxed investigation

The object in question is the root GitOps source. If the assumption is wrong — if the operator
re-applies in a mode that strips unowned fields, or if adding verify triggers a conflict that
flaps url/ref — the failure mode is that all GitOps delivery to prod stops. That is the one
place where "try it live and watch" is not an acceptable first move.

It also cannot be tested from the engineer's normal footing: the role has read-only cluster access
by design, and no throwaway cluster exists on the agent host.

What to determine

On a throwaway cluster running the same flux-operator version, does a spec.verify
written by a non-operator field manager survive repeated operator reconciles (forced, and
across an operator restart)?
Which mechanism sets only verify without claiming url/interval? A Kustomize resource
cannot — OCIRepository requires both, so kustomize-controller would contend for fields the
operator owns. Evaluate a Kyverno mutation as the leading candidate.
Does the resulting configuration actually reject an unsigned or wrongly-signed artifact?
A verify block that is present but not enforcing is the same class of dead control this
whole thread is about — so the negative control is part of the deliverable, not optional.
Confirm the operator does not log or surface the foreign field as drift.

Definition of done

A recorded decision on this issue — route chosen or ruled out, with the observations behind it —
plus the follow-up implementation issue if it is viable. No delivery PR from this issue.

Out of scope

The upstream route (asking flux-operator to add sync.verify) is a third-party contribution and
is a maintainer decision, not an agent one. Note it, do not open it.

Part of #2922, under #2627.

🔗 Related PRs

#2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
#2919 - fix(security): move Flux cosign verify to the path KSail reads [merged]
#2930 - feat(ci): verify published evidence before promoting the prod digest [merged]
#2944 - fix(ci): assert Flux verification is in effect, not merely configured [merged]
#2952 - fix(ci): require the Flux verify block to constrain a signer [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

docs(ksail): state both reasons the verify block does not reach prod

docs(ksail): keep the do-not-hand-write-an-override warning

🤖 Generated by the Agentic Engineer

Cheaper first move found — do this before building a throwaway cluster

This spike was filed on the premise that the SSA-coexistence question needed an isolated cluster
because the root source is too high-blast-radius to experiment on. That premise still holds for a
new mechanism — but a cheaper experiment now exists that answers the same question on the real
cluster for free.

#2951 shows the deploy's ksail cluster update has been skipped for days by the Cilium rollout
gate (verified on two prod deploys: outcome=skipped duration_ms=0). KSail is itself a non-operator
field manager. So once that gate is released and cluster update runs once, reading
ocirepository/flux-system back answers this spike's central question directly:

if spec.verify is present and survives the next flux-operator reconcile → the coexistence
assumption is confirmed on the real operator version, and no separate mechanism is needed;
if it is applied and then disappears → the assumption is wrong, and that is when this spike's
throwaway-cluster work and the Kyverno-mutation route become the right next step.

So: do not build the throwaway cluster until #2951 is resolved and that read has been taken.
Running it first would spend real effort re-deriving something the production path is about to
demonstrate.

The rest of the spike's checklist is unchanged and still needed either way — in particular the
negative control (does it actually reject a wrongly-signed artifact?), which the read above does not
answer.

Blocked by #2951.

🤖 Generated by the Agentic Engineer

Premise re-confirmed today, plus an existence proof this spike does not yet have

Re-measured ocirepository/flux-system in flux-system on 2026-08-06 ~06:30Z.

1. The ownership premise still holds, unchanged. flux-operator (op=Apply) owns exactly:

interval, provider, ref, secretRef, url

and does not own verify. Its last Apply was 2026-07-13T21:22:56Z — over three weeks ago,
across ~109 successful merge-queue runs since. No field manager owns spec.verify, consistent with
the field being absent.

2. New: a non-operator field manager is ALREADY writing to this exact object. The full manager
list is four entries, not one:

manager	operation	time	spec keys owned
`flux-operator`	Apply	2026-07-13T21:22:56Z	interval, provider, ref, secretRef, url
`source-controller`	Update	2026-05-23T00:41:54Z	(none)
`ksail`	Update	2026-08-06T03:06:37Z	(none)
`source-controller`	Update	2026-08-06T03:06:39Z	(none)

ksail touched the root source three hours ago and coexists with flux-operator without
conflict. That is the closest thing to a live existence proof available without mutating the object:
a non-operator manager demonstrably holds a foothold here already.

3. This sharpens the diagnosis in #2922, and possibly contradicts it. #2922 frames the problem as
KSail's verify patch not taking effect. But ksail owns zero spec keys — so on this evidence the
patch is not being applied-then-reverted; it appears not to be landing at all. Those two failure
modes need opposite fixes, and only one of them depends on this spike's answer. Worth resolving which
before implementing #2922.

⚠️ Reasoned, not observed (same caveat this issue already makes): I have not written spec.verify
and watched it survive a reconcile. The decisive experiment still requires mutating the root GitOps
source, which is outside my grant — this is corroboration of the premise, not the spike's answer.

Methodological trap worth recording on this issue

kubectl get … -o json strips managedFields by default. My first read returned an empty
managedFields and empty ownerReferences, which reads exactly like "nothing manages this object" —
the precise opposite of the truth. A control against a tenant OCIRepository returned 0 as well,
confirming the read path rather than the object. Only --show-managed-fields reveals the four
managers above. Any future SSA-ownership check on this issue must pass that flag, or it will
confidently conclude the reverse.

🤖 Generated by the Agentic Engineer

Decision: use `FluxInstance.spec.kustomize.patches`. The spike's own question is moot — and Kyverno is not needed.

This spike asked whether a non-operator field manager can hold spec.verify on the root
OCIRepository, because FluxInstance.spec.sync has no verify field. Both halves of that framing
turn out to be answerable without a throwaway cluster, and the second half dissolves: flux-operator
has its own supported patch mechanism for the resources it generates, so nothing foreign ever
needs to write that field.

What was measured (live, read-only, `admin@prod`)

#	Observation	Consequence
1	`FluxInstance` CRD v1 `spec.sync` has exactly 8 properties — `interval, kind, name, path, provider, pullSecret, ref, url`. The string `verify` appears zero times anywhere in the CRD schema.	`spec.sync.verify` cannot be the answer. Confirms the spike's premise.
2	`spec.kustomize.patches` exists and is described as patching "the Flux installation"; the reference for it says Deployments, ServiceAccounts, or other resources created by the operator.	The operator itself can write the field.
3	flux-operator's own documentation carries a verbatim example titled "OCI sync with Cosign verification", whose target is literally `kind: OCIRepository, name: flux-system` and whose patch is `op: add, path: /spec/verify`.	This is the documented, supported route for exactly this problem.
4	On `ocirepository/flux-system`, `flux-operator` is the `Apply` manager owning exactly 5 spec keys (`interval, provider, ref, secretRef, url`) — and has never owned `verify`.	`verify` written by the operator joins keys it already owns ⇒ no field contention, no SSA pruning question at all.
5	This platform already uses `spec.kustomize.patches` 6×, live, on this same operator version (`v0.50.0`): kustomize-controller args, three `replicas: 2` HA patches, `topologySpreadConstraints`, `dnsConfig`. All are in effect right now.	The mechanism is proven working in this cluster, not just documented.
6	The live `FluxInstance/flux` has exactly 6 patches, matching `k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml`, and its `Apply` manager is `kustomize-controller`.	The repo manifest is the reconciled source of truth ⇒ the fix ships as an ordinary GitOps change.

Two beliefs recorded in this repo are falsified by (4)

ksail.prod.yaml documents the verify block as being for "the flux-system OCIRepository that KSail
generates and owns (rendered onto spec.verify; a hand-written override would lose the reconcile
fight, so it is configured here per ksail#4987)".

KSail does not own that resource. flux-operator does (Apply, 5 spec keys). KSail's only live
field-manager entry on it is metadata.annotations["reconcile.fluxcd.io/requestedAt"] — a
reconcile trigger. It writes no spec key.
The reconcile fight that matters is therefore with flux-operator, and the way to win it is to
go through the operator — which is precisely what spec.kustomize.patches does.

This also corrects an inference I recorded earlier from partial evidence: I had written that KSail's
verify patch "never lands". More precisely — on this path KSail never attempts a spec write at
all. The distinction matters, because it rules out "also patch on the update path" as a sufficient
fix: that would be adding a new capability that then has to fight the operator forever, not
restoring a dropped one.

Answering the spike's checkboxes

Does a spec.verify from a non-operator manager survive operator reconciles? — Not
determined, and no longer load-bearing. No natural experiment exists to settle it: across every
operator-applied resource in flux-system, every foreign manager owns zero spec keys, so
there is nothing to observe. The chosen route makes the question unnecessary.
Which mechanism sets only verify without claiming url/interval? —
FluxInstance.spec.kustomize.patches. The spike reasoned a Kustomize resource cannot do this
(true — OCIRepository requires url, so kustomize-controller would contend). A Kustomize
patch carried by the FluxInstance is a different thing: it adds one field and the operator
applies it. Kyverno is not needed and should not be used here.
Does the configuration actually reject an unsigned/wrongly-signed artifact? — Still
unproven. Carried to Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922 (its AC-4). See "not proven" below.
Does the operator surface the foreign field as drift? — Moot: there is no foreign field.

Offline proof that the patch document is correct

Because the realistic failure mode here is my own JSON6902 path/target syntax, that was proven
offline with kubectl kustomize against a minimal OCIRepository shaped exactly like the live one —
with a true baseline and two ablations, so the pass is not vacuous:

Arm	`verify:` in rendered output
A — true baseline, no `patches` block	0
B — the patch, target `kind: OCIRepository, name: flux-system`	1 (all three identities intact, regex escaping preserved)
C — ablation: target `name: NOT-THE-ROOT-SOURCE`	0
D — ablation: target `kind: GitRepository`	0
E — post-revert control	1

🔴 Sub-finding that changes the implementation's requirements: a mistargeted patch is a SILENT
no-op. Arms C and D exited 0 with no error and no warning — the verify block simply never
appears. So a wrong target selector produces a root source that is still unverified while the PR is
green and the FluxInstance reconciles cleanly. That is the same dead control class this whole
#2627 thread exists to eliminate, so #2922 must assert spec.verify by reading the live resource
back after deploy — a green validate, and even a successful reconcile, prove nothing.

What is NOT proven (carried, not hidden)

Enforcement. That a present verify block actually rejects a bad signature is untested here.
Supporting evidence: 5 tenant OCIRepositories in this cluster carry cosign verify blocks and are
Ready=True/Succeeded, so the mechanism demonstrably functions — but "functions" is not
"rejects". Remains Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository #2922 AC-4.
A patch landing on a non-Deployment target in this cluster. All 6 live patches target
Deployments. Reaching sync resources is documented (2)(3) and consistent with the observed
ownership in (4), but has not been observed here. The implementation PR is what observes it.
I could not re-confirm the three signer identities with cosign verify from the agent host: the
artifact is in a private registry and the host credential is scoped without registry read. First
attempt returned identical output for the positive and the negative control
(DENIED), which measures nothing — recording that rather than reporting a pass. The identities
already have a prior in-repo confirmation note; that is someone else's measurement, not mine.

Blast radius — unchanged, and still the reason to be careful

The spike is right that this is the one object where "try it and watch" is not acceptable: a wrong
verify block stops the root source, which halts all GitOps delivery to prod and cannot be repaired
through GitOps. That risk is not removed by this decision — it is why (1) and (3) above are
preconditions on #2922 rather than details, and why the read-back assertion is mandatory.

Disposition

Route chosen: FluxInstance.spec.kustomize.patches in
k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml.
Route ruled out: Kyverno mutation, and any non-operator field manager.
Upstream (sync.verify) stays out of scope per this issue, and is no longer needed.

Implementation lives in #2922, which this unblocks — no new implementation issue required, so no
delivery PR from here (per this issue's DoD). One new follow-up filed for the silent-no-op gap.

Closing as decided.

Navigation Menu

Repository navigation

Spike: can a non-operator field manager hold spec.verify on the root OCIRepository? #2960

Evidence

Why this needs its own timeboxed investigation

What to determine

Definition of done

Out of scope

Cheaper first move found — do this before building a throwaway cluster

Premise re-confirmed today, plus an existence proof this spike does not yet have

Methodological trap worth recording on this issue

Decision: use `FluxInstance.spec.kustomize.patches`. The spike's own question is moot — and Kyverno is not needed.

What was measured (live, read-only, `admin@prod`)

Two beliefs recorded in this repo are falsified by (4)

Answering the spike's checkboxes

Offline proof that the patch document is correct

What is NOT proven (carried, not hidden)

Blast radius — unchanged, and still the reason to be careful

Disposition

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Spike: can a non-operator field manager hold spec.verify on the root OCIRepository? #2960

Description

Evidence

Why this needs its own timeboxed investigation

What to determine

Definition of done

Out of scope

Activity

coderabbitai commented on Aug 4, 2026

devantler commented on Aug 4, 2026

Cheaper first move found — do this before building a throwaway cluster

devantler commented on Aug 6, 2026

Premise re-confirmed today, plus an existence proof this spike does not yet have

Methodological trap worth recording on this issue

devantler commented on Aug 6, 2026

Decision: use FluxInstance.spec.kustomize.patches. The spike's own question is moot — and Kyverno is not needed.

What was measured (live, read-only, admin@prod)

Two beliefs recorded in this repo are falsified by (4)

Answering the spike's checkboxes

Offline proof that the patch document is correct

What is NOT proven (carried, not hidden)

Blast radius — unchanged, and still the reason to be careful

Disposition

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Decision: use `FluxInstance.spec.kustomize.patches`. The spike's own question is moot — and Kyverno is not needed.

What was measured (live, read-only, `admin@prod`)