Skip to content

feat(omni): single source of truth for versions + CI drift gate - #27

Merged
syscod3 merged 3 commits into
mainfrom
chore/versions-single-source
Aug 3, 2026
Merged

syscod3 merged 3 commits into
mainfrom
chore/versions-single-source

Conversation

@syscod3

@syscod3 syscod3 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Single hand-owned home for desired Talos/Kubernetes versions, plus a CI gate that makes drift unmergeable.

What

  • omni/versions.yaml — global pin + per-cluster overrides. Hand-owned.
    omni/talos-image.yaml stays the machine-written build ledger.
  • scripts/check-version-drift.py — fails CI when any derived file disagrees
    with the pin. Carries --selftest for the resolver logic.
  • Deleted .github/workflows/talos-image-update.yml — an orphaned second PR
    path (no sender in any repo) that would bump talos_version with no
    schematic_id or oci_image_ocid, i.e. a ledger pointing at an unbuilt
    image. That is root cause chore: Talos upgrade to v1.12.7 #4 of the 2026-08-03 incident.
  • Deleted omni/machine-classes/unraid-control-plane.yaml — hand-written
    duplicate of the omni-on-unraid renderer output, already drifted
    (memory: 4096 vs the renderer's 6144, no installImage).
  • pluto now checks deprecations against the desired k8s version, not the
    build ledger. It was checking v1.37.0-beta.0 while clusters run v1.36.3.

Drift repaired

The new gate failed on its first run: both OCI machine classes installed
v1.14.0-beta.1 while oci-lab targets v1.13.7, so any node provisioned
into oci-lab would have installed the beta. Repaired to v1.13.7.

Merge this first — omni-on-unraid, talos-images and oci-free-tier-manager all
read versions.yaml from main.

Plan: openspec/changes/automate-talos-upgrade-pipeline in syscode-ai-internal-plans.

syscod3 added 3 commits August 3, 2026 23:26
…ine class

Desired Talos/Kubernetes versions get one hand-owned home. talos-image.yaml
stays the machine-written build ledger — it is rewritten wholesale by the
talos-images workflow and records what was built, not what should run.

unraid-control-plane.yaml was a hand-written duplicate of what
omni-on-unraid's renderer generates for unraid-cp, and had already drifted:
memory 4096 against the renderer's 6144, and no installImage at all.

Refs: openspec/changes/automate-talos-upgrade-pipeline (tasks 1.4, 2.1)
… workflow

The pluto job read kubernetes_version from omni/talos-image.yaml, which is the
build ledger — it records what was last built, not what clusters run. With the
ledger at v1.37.0-beta.0 and clusters on v1.36.3, the deprecation gate was
checking the wrong target entirely. It now reads the desired pin from
omni/versions.yaml and fails if that pin is missing.

talos-image-update.yml was a second, competing PR-opening path: a
repository_dispatch receiver that sed-bumps talos_version in the ledger and
opens its own PR on a different branch from the one talos-images/build.yml
opens. No sender exists in any repo (grepped all of ~/syscode/git), so it is
orphaned — but if it ever fired it would bump talos_version without
schematic_id or oci_image_ocid, producing a ledger pointing at a version whose
image was never built. That is root cause #4 of the 2026-08-03 incident.

Refs: openspec/changes/automate-talos-upgrade-pipeline
Nothing reconciled the derived files when versions.yaml changed without
needing a new image — the build workflow only opens a PR when it has
something to build, so flipping a cluster back to an already-built version
left machine classes and templates stale with no failure anywhere.

The checker caught live drift on its first run: both OCI machine classes
still installed v1.14.0-beta.1 while oci-lab targets v1.13.7, so any node
provisioned into oci-lab would have installed the beta. Repaired to v1.13.7.

Kubernetes is never inherited across a diverging Talos target: a cluster that
overrides talos without kubernetes is an error, not an inherit, because the
global kubernetes pin belongs to a different Talos minor.

Refs: openspec/changes/automate-talos-upgrade-pipeline
@syscod3
syscod3 merged commit 060857e into main Aug 3, 2026
5 checks passed
@syscod3
syscod3 deleted the chore/versions-single-source branch August 3, 2026 23:01
syscod3 added a commit that referenced this pull request Aug 8, 2026
feat(omni): single source of truth for versions + CI drift gate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant