Make root OCI publication verified and atomic before moving latest

🤖 Generated by the Codex Daily AI Engineer instance.

Problem

Production's root Flux OCIRepository tracks ghcr.io/devantler-tech/platform/manifests:latest every minute, but live spec.verify is null. The intended keyless Cosign policy is nested at spec.cluster.verify in ksail.prod.yaml; KSail v7.167 reads spec.workload.flux.verify, so the unknown field is silently ignored.

The production deploy currently moves mutable latest before signing and attesting it. With verification disabled, Flux can fetch an unsigned artifact immediately after push. Even after the verify config is corrected, Flux can fetch after signing but before provenance/SBOM attestations complete. The DR rebuild is coupled to the same defect: it pushes an unsigned artifact, lacks id-token: write, and its workflow identity is not in the intended allow-list, so simply moving verify would make a clean DR reconcile fail.

Live evidence on 2026-07-13: root source ref.tag=latest, interval=1m0s, spec.verify=null, Ready on the current main artifact. This is pre-existing and separate from the GHCR credential propagation fix in #2625.

Proposed direction

Move the Cosign configuration to the KSail-supported spec.workload.flux.verify path and add a rendered/live-shape regression that fails if root verification is absent.
Make publication atomic from Flux's perspective: publish to an immutable staging reference, sign and attest it, then promote/repoint latest; or suspend/repoint the root source with explicit failure recovery until the complete supply-chain bundle exists.
Give DR a valid signed-artifact path (id-token: write plus a narrowly allowed DR identity), or have DR consume an already signed immutable artifact instead of publishing unsigned bytes.
Prove normal merge-group, manual CD, failed delivery, and from-zero DR behavior.

Acceptance criteria

The live root OCIRepository has non-null Cosign verification with only the intended production identities.
Flux cannot observe a new artifact until signature and required attestations are complete.
Any failed push/sign/attest leaves the last verified artifact active.
DR can rebuild from zero without weakening verification and finishes on a signed, allowed artifact.
Static tests validate config nesting, workflow permissions/identity, and publication ordering.

Rough size: medium; coordinate the KSail config correction, publication transaction, and DR signer path in one rollout so no intermediate state breaks production or recovery.

Related: #2613, #2625

🔗 Related PRs

#2317 - ci: migrate deploy-prod Flux diagnostics to shared diagnose-flux composite [merged]
#2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
#2399 - feat: accept devantler-tech/actions cosign subjects during the publish-workflow repoint [merged]
#2404 - fix: tenant artifact verification rejects the dual signing identities from #2399 [merged]
#2457 - fix(talos): accept the actions publish-app identity at node image verification [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

feat(security): sign the DR-published root artifact and allow its ide…

🤖 Generated by the Agentic Engineer

Premise re-verified (2026-07-28), and it is sharper than filed

This issue was written 2026-07-13 against KSail v7.167; the tap is now v7.177.3, so I re-checked rather than inheriting it. All of it still holds, and the live contrast makes the severity clearer than the original text:

Source	`spec.verify`
`flux-system/flux-system` (root — delivers the whole platform)	null
`ascoachingogvaner`, `aws`, `doggy-countdown`, `github-config`, `wedding-app`	`cosign`

Every tenant artifact is signature-verified; the infrastructure root is not. The posture is inverted exactly where blast radius is largest.

Confirmations:

Misnesting is real and current. yq '.spec.cluster.verify' ksail.prod.yaml resolves — the block sits at spec.cluster.verify. KSail reads Workload.Flux.Verify (pkg/apis/cluster/v1alpha1/types.go, pinned by that package's envvar_drift_test.go as Workload.Flux.Verify.*). So the config is inert, silently.
Publication is non-atomic. In deploy-prod/action.yml the order is ksail workload push → install cosign → sign → attest, so latest moves before a signature exists. Flux polls the root at interval: 1m0s.
DR cannot sign at all. dr-rebuild.yaml job rebuild declares only contents: read + packages: write; the file contains no cosign invocation.

The ordering is load-bearing — this is why the issue has not moved

The four parts are not independently shippable, and the dependency runs opposite to the order they are listed in. Moving verify to the path KSail reads is the LAST step, not the first — doing it first switches verification on while two writers still publish unsignable or unsigned artifacts, converting a latent gap into a production and recovery outage.

Sequencing:

DR signer path — safe now, because signing an artifact nothing verifies is inert. → Give the DR rebuild a signed-artifact publication path #2839, PR feat(security): give the DR rebuild a signed-artifact publication path #2840
Atomic publication — sign/attest before latest moves, so Flux cannot observe an unsigned artifact in the push→sign window.
Attestation parity for DR — SBOM/provenance, if the acceptance criterion "required attestations complete" is to hold for a recovery too. Note Flux spec.verify gates on signatures only, so this is a criteria question, not a gate question.
Flip the config nesting — move the block to spec.workload.flux.verify and add the rendered/live-shape regression. Only safe once 1–3 hold.

Step 1 is shipped as #2839 / PR #2840 (verification deliberately left off). Steps 2–4 remain here; I have not filed them as children yet, since 2 and 3 want their own measurement of the push→sign window before being specified.

feat(security): give the DR rebuild a signed-artifact publication path (

test(ci): pin the prod publication ordering so evidence precedes release

🤖 Generated by the Agentic Engineer

Revalidation against current evidence (2026-08-05) — scope has moved, issue still valid

Picked this up as the oldest open Security issue and re-checked its premises against live prod
rather than implementing the 2026-07-13 plan as written. Two of the four proposed directions have
changed state.

Direction 1 (move Cosign config to spec.workload.flux.verify) — DONE. #2919 landed it, and
platform CI now pins KSAIL_VERSION 7.178.14, which post-dates ksail#6454's update-path
re-assertion.

Acceptance criterion 1 (live root OCIRepository has non-null verification) — STILL FAILING, and
that half is now tracked in sharper form by #2922. Read from prod today:

ocirepository/flux-system  generation: 3  spec.verify: null   ref.tag: latest   Ready: True

I posted the decisive mechanism evidence on #2922: the FluxInstance CRD has no verify key
under spec.sync (so the operator cannot declare it), and server-side-apply ownership shows
flux-operator owns only interval/provider/ref/secretRef/url — never f:verify — so a direct
patch is both the only available mechanism and safe from pruning. The remaining defect is that
nothing writes it on the deploy path, now tracked as
ksail#6480.

Directions 2–4 (atomic publish-then-promote, DR signer path) remain open and are the substance of
this issue. Note a scripts/validate-dr-signing gate already exists in cd.yaml, so the DR half
has moved too and should be re-scoped against it rather than built from the original description.

Suggested reframe: narrow this issue to the atomic publication guarantee (criteria 2 and 3 —
Flux cannot observe an artifact before signature and attestations complete, and a failed
push/sign/attest leaves the last verified artifact active), and let #2922 + ksail#6480 carry
criterion 1. Criterion 4 (DR from zero) is worth its own child now that the signing validator exists.

Not implementing the enforcement half here — I am shipping ksail#6480 this run.

🤖 Generated by the Agentic Engineer

Triage: one criterion is already met, one is blocked, two remain implementable

I picked this up as the oldest actionable Security issue, investigated, and am releasing it rather
than holding a claim I am not delivering against this run. What I established:

Criterion 5 (static tests validate config nesting) is already satisfied. scripts/validate-flux-verify
exists and is wired into CI (go test ./scripts/validate-flux-verify). It checks the read path
spec.workload.flux.verify, reports stray verify keys at any other path, and asserts the block
actually constrains a signer rather than merely being present. Its header comment cites this issue.
No work needed.

Criterion 1 (live root OCIRepository has non-null verification) is blocked, not unimplemented.
The repair already ships in the KSail prod deploys with (reconcileFluxVerify, ksail#6454, first in
v7.178.13; prod pins v7.178.14), but it is reachable only through ksail cluster update, and that
step is skipped on every prod deploy while the Cilium rollout gate is active. Full evidence and the
end-action markers are in my comments on #2922; the live blocker is #2963.

Criteria 2–4 (atomic publication, failed-push safety, DR signing path) are unaffected by that gate
and remain implementable now. They are the genuinely startable part of this issue: publish to an
immutable staging reference, sign and attest it, then repoint latest, so Flux cannot observe an
artifact before its supply-chain bundle is complete.

Suggest whoever picks this up next starts at criteria 2–4 and leaves criterion 1 to close on its own
once #2963 releases the gate — and re-reads the live resource then rather than assuming it applied.

🤖 Generated by the Agentic Engineer

Revalidated against current `main` (`2c2d4c09`)

This issue was written on 2026-07-13. Four of its five acceptance criteria have since been delivered by other work, so the remaining scope is much narrower than the body suggests. Evidence for each, gathered by running the guards rather than reading them:

Criterion	State	Evidence
1. Live root OCIRepository has non-null verification	❌ UNMET — the only real remaining work	`kubectl --context admin@prod get ocirepository/flux-system -n flux-system` → `spec.verify` empty, `ref.tag=latest`
2. Flux cannot observe an artifact before signature + attestations	✅ met	publication is staging-ref → sign → SBOM → attest → verify → promote; the ordering chain is asserted in `validatePublicationAction`, and `test-verify-published-evidence.sh` pins the gate before promotion and after both attestations
3. A failed push/sign/attest leaves the last verified artifact active	✅ met (one gap found, being fixed)	promotion is last and unconditional; see #2967 below
4. DR rebuilds from zero without weakening verification	✅ met	`validate-dr-signing` requires the DR job to grant `id-token: write` + `attestations: write`, to publish through the shared action, and its Fulcio identity to be in the cluster allow-list
5. Static tests validate nesting, permissions/identity, ordering	✅ met	`validate-flux-verify` (nesting), `validate-dr-signing` (permissions/identity/ordering), `test-verify-published-evidence.sh` (gate wiring)

Both guards pass at this head: DR publication contract passed. and passed=30 failed=0.

Criterion 1 is not a config defect. ksail.prod.yaml already nests verification at the KSail-supported spec.workload.flux.verify. The reason it does not reach the cluster is that ksail cluster update — the only code path that applies it — is skipped on every production deploy while the Cilium rollout gate is active. That is #2963's subject, so I have marked this issue blocked by it rather than restating the analysis here.

One genuine gap surfaced while checking criterion 3, filed as #2967 and fixed in #2968: the contract proved each step existed in the right order but never that a step's failure stops the run, so a one-line continue-on-error: true on verify_evidence passed every guard while discarding the evidence check. GitHub documents that key on composite steps as "allow[ing] the action to pass when this step fails", so the promotion would then have proceeded.

Suggested disposition

Keep this issue open for criterion 1 only, or close it in favour of #2963 + #2922, which already track exactly that. No further work here is startable until the gate clears.

🤖 Generated by the Agentic Engineer

Scope check before anyone implements this — two ACs are already met

Measured against main today. Do not rebuild these:

Atomic publication (ACs 2 and 3) is live. The deploy-prod composite already publishes to an
immutable staging reference, signs, attests and verifies before moving the mutable tag:
staging_reference → push_staging → cosign_sign → generate_sbom → attest_sbom → attest_provenance → verify_evidence → promote_latest. So Flux cannot observe an artifact before its signature and
attestations exist, and a failed step leaves the previously promoted artifact active.
The static config-nesting test (AC 5, first half) exists and is CI-wired.
scripts/validate-flux-verify has unit tests and runs on every PR
(go test ./scripts/validate-flux-verify + go run … ksail.prod.yaml); it explicitly rejects the
spec.cluster.verify mis-nesting described in the Problem section. The config in ksail.prod.yaml
is already at the supported spec.workload.flux.verify path.

AC 1 (live root OCIRepository verified) is still unmet, and its blocker is now precisely known —
it is not the KSail config path and not #2951. ksail cluster update is the only step that would
apply the config to the live resource, and it has been skipped for 11 days by the still-active
Cilium homogeneous-device rollout gate. Full evidence, including the skipped-step markers and the live
spec.verify=null read, is in #2922:
#2922 (comment)

Remaining genuine scope here: live enforcement (blocked on the rollout gate's completion/rollback
artifact) and the DR signed-artifact path.

🤖 Generated by the Agentic Engineer

Re-measured against `main` today — this has been delivered, criterion by criterion

Picked up as the oldest open Security issue. Before implementing anything I checked whether its
evidence still holds, and it does not: the publication path was rebuilt after this was filed on
2026-07-13, and it now does what this issue asked for. Evidence per acceptance criterion, all read
from main:

2. "Flux cannot observe a new artifact until signature and required attestations are complete" — met.
.github/actions/deploy-prod/publish-platform-manifests/action.yml no longer moves latest first. The
order is now: push to a unique staging tag (staging-<sha>-<run>-<attempt>) → resolve the immutable
digest → cosign sign that digest → attest SBOM and SLSA provenance (push-to-registry: true) →
verify the published evidence → and only then 🚀 Promote the evidenced digest to latest. latest
— the only reference production follows — is moved in the last step, after the bundle exists.

3. "Any failed push/sign/attest leaves the last verified artifact active" — met.
Promotion is the final step and every step runs under set -euo pipefail, so any earlier failure
aborts before latest moves; the previously promoted digest stays in place. The promote step also
re-reads latest and fails if it does not resolve to the digest just evidenced, so a partial
promotion cannot pass silently.

The verification step is enforcing by default (enforce-evidence-verification, default: "true") and
neither caller overrides it — checked both deploy-prod/action.yml and dr-rebuild.yaml. It also
asks the registry what it actually holds rather than inferring success from the steps having run,
which closes the "green step, no usable evidence" gap this issue worried about.

4. "DR can rebuild from zero without publishing unsigned bytes" — met.
dr-rebuild.yaml now declares id-token: write and attestations: write, and calls the same
evidenced publish composite rather than pushing raw. Its identity is also in the intended allow-list:
dr-rebuild.yaml@refs/heads/main is one of the three pinned verify subjects, so a DR-produced
artifact verifies instead of being refused — the coupling this issue flagged is resolved on both ends.

1. "The live root OCIRepository has non-null Cosign verification with only the intended identities" —
met on the cluster, but not yet durable in Git.
Measured live just now, ocirepository/flux-system carries exactly the three intended subjects. It is
there via a speculative merge-group deploy of #3001 rather than from main, so the next successful
deploy from main would remove it again — see the correction on
#3001 and the
mechanism filed as #3005.

Disposition

Closing as delivered. The one residual — making criterion 1 durable in Git — is not a separate piece
of work: it is exactly the scope of #2922, which is open with #3001 ready and parked on the
#2963 deploy blocker. Keeping this open alongside them tracks the same remaining change twice and
keeps a three-week-stale description of the publication path at the head of the security queue.

Nothing here was implemented by this run; this is a measurement of what already shipped.

Navigation Menu

Repository navigation

Make root OCI publication verified and atomic before moving latest #2627

Problem

Proposed direction

Acceptance criteria

Premise re-verified (2026-07-28), and it is sharper than filed

The ordering is load-bearing — this is why the issue has not moved

32 remaining items

Revalidation against current evidence (2026-08-05) — scope has moved, issue still valid

Triage: one criterion is already met, one is blocked, two remain implementable

Revalidated against current `main` (`2c2d4c09`)

Suggested disposition

Scope check before anyone implements this — two ACs are already met

Re-measured against `main` today — this has been delivered, criterion by criterion

Disposition

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Make root OCI publication verified and atomic before moving latest #2627

Description

Problem

Proposed direction

Acceptance criteria

Activity

coderabbitai commented on Jul 13, 2026

devantler commented on Jul 28, 2026

Premise re-verified (2026-07-28), and it is sharper than filed

The ordering is load-bearing — this is why the issue has not moved

32 remaining items

devantler commented on Aug 5, 2026

Revalidation against current evidence (2026-08-05) — scope has moved, issue still valid

devantler commented on Aug 5, 2026

Triage: one criterion is already met, one is blocked, two remain implementable

devantler commented on Aug 5, 2026

Revalidated against current main (2c2d4c09)

Suggested disposition

devantler commented on Aug 6, 2026

Scope check before anyone implements this — two ACs are already met

devantler commented on Aug 6, 2026

Re-measured against main today — this has been delivered, criterion by criterion

Disposition

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Revalidated against current `main` (`2c2d4c09`)

Re-measured against `main` today — this has been delivered, criterion by criterion