feat(oci): centralize GitOps bootstrap

README.md

@@ -8,8 +8,10 @@
 > [docs/architecture/gitops-layering.md](docs/architecture/gitops-layering.md#current-state--todo).
 
 The apps running on the Syscode homelab Kubernetes clusters, and their config.
-Argo CD (self-managed, one instance per cluster) watches this repo and applies
-changes automatically — no manual `kubectl apply`.
+Argo CD (self-managed, hosted in `unraid-lab`) watches this repo and applies
+changes automatically — no manual `kubectl apply`. It also reconciles
+`oci-lab` remotely through a registered external cluster; `oci-lab` runs no
+Argo CD of its own.
 
 ## Apps
 

appset.yaml

@@ -1,9 +1,11 @@
 # Layering ApplicationSet — the "chart-app" engine.
 #
-# SEEDED PER CLUSTER by the Talos inline manifests. The two `list` elements below
-# (clusterName / clusterType) are the ONLY per-cluster difference — each cluster's
-# inline manifest embeds this file with its own identity. This copy is the
-# canonical reference; the identities here are placeholders.
+# ONE instance, hosted in unraid-lab's Argo CD (embedded via unraid-lab's Talos
+# inline manifests). It covers BOTH clusters: oci-lab is reconciled remotely
+# through the registered external cluster — oci-lab deliberately runs no local
+# Argo CD. Application destinations differ per element (see the conditional in
+# `destination` below): in-cluster server for unraid-lab, registered cluster
+# name `oci-lab` for OCI.
 #
 # Selection (which apps run): three git generators —
 #   apps/*/app.yaml                        shared (every cluster)
@@ -26,8 +28,14 @@ spec:
         generators:
           - list:
               elements:
-                - clusterName: CLUSTER_NAME
-                  clusterType: CLUSTER_TYPE
+                - clusterName: unraid-lab
+                  clusterType: homelab-kvm
+                  destinationName: ""
+                  destinationServer: https://kubernetes.default.svc
+                - clusterName: oci-lab
+                  clusterType: cloud
+                  destinationName: oci-lab
+                  destinationServer: ""
           - git:
               repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps
               revision: HEAD
@@ -37,8 +45,14 @@ spec:
         generators:
           - list:
               elements:
-                - clusterName: CLUSTER_NAME
-                  clusterType: CLUSTER_TYPE
+                - clusterName: unraid-lab
+                  clusterType: homelab-kvm
+                  destinationName: ""
+                  destinationServer: https://kubernetes.default.svc
+                - clusterName: oci-lab
+                  clusterType: cloud
+                  destinationName: oci-lab
+                  destinationServer: ""
           - git:
               repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps
               revision: HEAD
@@ -48,8 +62,14 @@ spec:
         generators:
           - list:
               elements:
-                - clusterName: CLUSTER_NAME
-                  clusterType: CLUSTER_TYPE
+                - clusterName: unraid-lab
+                  clusterType: homelab-kvm
+                  destinationName: ""
+                  destinationServer: https://kubernetes.default.svc
+                - clusterName: oci-lab
+                  clusterType: cloud
+                  destinationName: oci-lab
+                  destinationServer: ""
           - git:
               repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps
               revision: HEAD
@@ -74,7 +94,15 @@ spec:
           targetRevision: HEAD
           ref: v
       destination:
-        server: https://kubernetes.default.svc
+        # oci-lab has no in-cluster Argo: target the registered external
+        # cluster by name. Its Secret lives in unraid-lab's argocd namespace
+        # (clusters/unraid-lab/apps/oci-cluster-registration). Exactly one of
+        # name/server is set per element — Argo resolves by name when server
+        # is empty, by server when name is empty. The fields stay pure YAML:
+        # ApplicationSet templates render AFTER YAML parsing, so conditional
+        # KEYS are not possible here; per-element params keep it parseable.
+        name: "{{ .destinationName }}"
+        server: "{{ .destinationServer }}"
         namespace: "{{.namespace}}"
       syncPolicy:
         automated:

clusters/oci-lab/apps/oci-pivot-secrets/application.yaml

@@ -3,6 +3,9 @@
 # until the owner delivers the bw-auth-token Secret + fills in
 # organizationId/projectId/bwSecretId — see add-external-secrets-bitwarden
 # (syscode-ai-internal-plans).
+#
+# Applied by unraid-lab's Argo (oci-raw lane) to the registered oci-lab
+# cluster — oci-lab runs no local Argo CD.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
@@ -15,7 +18,7 @@ spec:
     targetRevision: HEAD
     path: clusters/oci-lab/apps/oci-pivot-secrets/manifests
   destination:
-    server: https://kubernetes.default.svc
+    name: oci-lab
     namespace: oci-pivot-system
   syncPolicy:
     automated:

clusters/unraid-lab/apps/oci-bootstrap/application.yaml

@@ -0,0 +1,26 @@
+# unraid-lab-only. Syncs bootstrap/ to the oci-lab EXTERNAL cluster —
+# but ONLY the Tailscale bits. bootstrap/argocd-cm.yaml and
+# bootstrap/argocd-rbac-cm.yaml configure unraid's LOCAL Argo CD
+# (PocketID URL, RBAC) and must not land on oci-lab, which runs no Argo.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oci-bootstrap
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps.git
+    targetRevision: HEAD
+    path: bootstrap
+    directory:
+      include: "tailscale-*.yaml"
+  destination:
+    name: oci-lab
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

clusters/unraid-lab/apps/oci-cluster-registration/application.yaml

@@ -0,0 +1,24 @@
+# unraid-lab-only. Argo CD on unraid-lab reconciles the oci-lab external
+# cluster (see oci-cluster-registration). This Application ships the
+# cluster-registration ExternalSecret into unraid's Argo CD — it is a
+# plain raw manifest, not a chart-app.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oci-cluster-registration
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps.git
+    targetRevision: HEAD
+    path: clusters/unraid-lab/apps/oci-cluster-registration/manifests
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

clusters/unraid-lab/apps/oci-cluster-registration/manifests/external-secret.yaml

@@ -0,0 +1,50 @@
+# Registers the oci-lab external cluster with unraid-lab's Argo CD.
+#
+# oci-lab runs NO local Argo CD; this Secret is what lets unraid's Argo
+# reconcile it (ApplicationSet destination `name: oci-lab`, oci-raw /
+# oci-bootstrap lanes). Endpoint is the PRIVATE API address —
+# https://<oci-cp-vpn-subnet-ip>:6443 — reachable only over
+# tailnet → wrt-london → IPSec → the OCI vpn subnet.
+#
+# The bearer token authenticates as ServiceAccount argocd-manager
+# (kube-system on oci-lab, created by the argocd-manager inline manifest).
+# It is a long-lived service-account token (Secret of type
+# kubernetes.io/service-account-token annotated with that SA), generated
+# POST-bootstrap and stored in Bitwarden — never committed. Stays unhealthy
+# until Bitwarden contains the separate server, CA (base64 TLS CA of the
+# oci-lab API server), and token secrets referenced below.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: oci-lab-cluster
+  namespace: argocd
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: bitwarden
+    kind: ClusterSecretStore
+  target:
+    name: oci-lab-cluster
+    creationPolicy: Owner
+    deletionPolicy: Retain
+    template:
+      metadata:
+        labels:
+          # Makes unraid-lab's Argo CD treat this Secret as a cluster registration.
+          argocd.argoproj.io/secret-type: cluster
+      type: Opaque
+      data:
+        name: oci-lab
+        server: "{{ .server }}"
+        config: |
+          {"tlsClientConfig": {"insecure": false, "caData": "{{ .ca }}", "bearerToken": "{{ .token }}"}}
+  data:
+    - secretKey: server
+      remoteRef:
+        key: "817bc3d5-0e06-4bea-abfb-b4b600a39165" # gitleaks:allow -- Bitwarden secret UUID, not a credential
+    - secretKey: ca
+      remoteRef:
+        key: "979fe9de-2c24-4806-8402-b4b600a3919d" # gitleaks:allow -- Bitwarden secret UUID, not a credential
+    - secretKey: token
+      remoteRef:
+        key: "f42aff26-639b-454a-a6d6-b4b600a391d8" # gitleaks:allow -- Bitwarden secret UUID, not a credential

clusters/unraid-lab/apps/oci-raw/application.yaml

@@ -0,0 +1,27 @@
+# unraid-lab-only. Raw-manifest lane for the oci-lab EXTERNAL cluster:
+# plain-manifest apps that aren't a chart-app (e.g. cert-manager
+# Issuer/Certificate, oci-pivot-secrets) applied to oci-lab by unraid's
+# Argo CD. Mirrors argocd-unraid-raw — oci-lab runs no local Argo CD.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oci-raw
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps.git
+    targetRevision: HEAD
+    path: clusters/oci-lab/apps
+    directory:
+      recurse: true
+      include: "*/application.yaml"
+  destination:
+    name: oci-lab
+    namespace: argocd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

clusters/unraid-lab/omni/inline-manifests.yaml

@@ -34186,10 +34186,12 @@ cluster:
       contents: |
         # Layering ApplicationSet — the "chart-app" engine.
         #
-        # SEEDED PER CLUSTER by the Talos inline manifests. The two `list` elements below
-        # (clusterName / clusterType) are the ONLY per-cluster difference — each cluster's
-        # inline manifest embeds this file with its own identity. This copy is the
-        # canonical reference; the identities here are placeholders.
+        # ONE instance, hosted in unraid-lab's Argo CD (embedded via unraid-lab's Talos
+        # inline manifests). It covers BOTH clusters: oci-lab is reconciled remotely
+        # through the registered external cluster — oci-lab deliberately runs no local
+        # Argo CD. Application destinations differ per element (see the conditional in
+        # `destination` below): in-cluster server for unraid-lab, registered cluster
+        # name `oci-lab` for OCI.
         #
         # Selection (which apps run): three git generators —
         #   apps/*/app.yaml                        shared (every cluster)
@@ -34214,6 +34216,12 @@ cluster:
                       elements:
                         - clusterName: unraid-lab
                           clusterType: homelab-kvm
+                          destinationName: ""
+                          destinationServer: https://kubernetes.default.svc
+                        - clusterName: oci-lab
+                          clusterType: cloud
+                          destinationName: oci-lab
+                          destinationServer: ""
                   - git:
                       repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps
                       revision: HEAD
@@ -34225,6 +34233,12 @@ cluster:
                       elements:
                         - clusterName: unraid-lab
                           clusterType: homelab-kvm
+                          destinationName: ""
+                          destinationServer: https://kubernetes.default.svc
+                        - clusterName: oci-lab
+                          clusterType: cloud
+                          destinationName: oci-lab
+                          destinationServer: ""
                   - git:
                       repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps
                       revision: HEAD
@@ -34236,6 +34250,12 @@ cluster:
                       elements:
                         - clusterName: unraid-lab
                           clusterType: homelab-kvm
+                          destinationName: ""
+                          destinationServer: https://kubernetes.default.svc
+                        - clusterName: oci-lab
+                          clusterType: cloud
+                          destinationName: oci-lab
+                          destinationServer: ""
                   - git:
                       repoURL: https://github.com/syscode-labs/syscode-homelab-gitops-apps
                       revision: HEAD
@@ -34260,7 +34280,15 @@ cluster:
                   targetRevision: HEAD
                   ref: v
               destination:
-                server: https://kubernetes.default.svc
+                # oci-lab has no in-cluster Argo: target the registered external
+                # cluster by name. Its Secret lives in unraid-lab's argocd namespace
+                # (clusters/unraid-lab/apps/oci-cluster-registration). Exactly one of
+                # name/server is set per element — Argo resolves by name when server
+                # is empty, by server when name is empty. The fields stay pure YAML:
+                # ApplicationSet templates render AFTER YAML parsing, so conditional
+                # KEYS are not possible here; per-element params keep it parseable.
+                name: "{{ .destinationName }}"
+                server: "{{ .destinationServer }}"
                 namespace: "{{.namespace}}"
               syncPolicy:
                 automated:

docs/architecture/gitops-layering.md

@@ -1,21 +1,27 @@
 # GitOps layering: base → type → cluster
 
-How apps and config are layered across clusters. Self-managed Argo CD (each
-cluster runs its own), no central hub.
+How apps and config are layered across clusters. One Argo CD, hosted in
+`unraid-lab`; `oci-lab` is reconciled remotely through a registered external
+cluster (see Current state / TODO below).
 
 ## Bootstrap
 
-Talos `inlineManifests` (per cluster) install Argo CD, then seed **three roots**.
-The only per-cluster difference is the cluster's identity (`clusterName`,
-`clusterType`), baked into the seeded `appset`.
+Talos `inlineManifests` on unraid-lab install Argo CD and seed **three roots**:
 
 - **`apps`** (ApplicationSet) — syncs `apps/*`, `types/<type>/apps/*`,
-  `clusters/<name>/apps/*`. The **chart-apps**, values layered.
+  `clusters/<name>/apps/*`. The **chart-apps**, values layered. Covers BOTH
+  clusters; oci-lab Applications target the registered `oci-lab` cluster.
 - **`bootstrap`** (Application) — syncs `bootstrap/`. Shared raw manifests
   (tailscale operator).
-- **`<cluster>-raw`** (Application) — syncs
-  `clusters/<name>/apps/*/application.yaml`. **Custom ApplicationSets** that
-  aren't chart-apps (arc-runners).
+- **`unraid-raw`** (Application) — syncs
+  `clusters/unraid-lab/apps/*/application.yaml`. **Custom ApplicationSets**
+  and raw apps, including the oci-* lanes below.
+
+On oci-lab, `inlineManifests` bootstrap **Cilium only** (plus the
+`argocd-manager` service account unraid's Argo authenticates as) — no local
+Argo CD. Three unraid-hosted Applications reconcile it:
+`oci-cluster-registration`, `oci-raw` (`clusters/oci-lab/apps/*/application.yaml`),
+`oci-bootstrap` (`bootstrap/` tailscale manifests).
 
 Chart-app lane keys on **`app.yaml`**; raw lane keys on **`application.yaml`** —
 so the two never double-manage an app in the same directory.
@@ -70,12 +76,22 @@ Argo Application per app, with the layered `valueFiles`.
 
 ## Current state / TODO
 
-- `appset.yaml` (repo root) is the canonical ApplicationSet. `omni/scripts/generate-manifests.sh`
-  already injects it into each cluster's `inlineManifests` along with the Argo CD
-  install manifest.
+- `appset.yaml` (repo root) is the canonical ApplicationSet. It is hosted ONCE,
+  in unraid-lab's Argo CD (injected via unraid-lab's `inlineManifests`), and
+  covers BOTH clusters: static `unraid-lab`/`oci-lab` list elements; the
+  `destination` template targets the in-cluster server for unraid-lab and the
+  registered external cluster `name: oci-lab` for OCI.
+- `oci-lab` runs NO local Argo CD. Its inline manifests bootstrap Cilium plus
+  the `argocd-manager` service account only. unraid-lab's Argo reconciles it
+  through the registered external cluster:
+  `clusters/unraid-lab/apps/oci-cluster-registration` (ExternalSecret →
+  argocd cluster Secret; endpoint is the private VPN-subnet API address),
+  `oci-raw` (raw lane for `clusters/oci-lab/apps/*/application.yaml`),
+  `oci-bootstrap` (tailscale-* manifests from `bootstrap/`).
 - `unraid-lab` is validated: all Argo CD Applications are **Synced** and **Healthy**.
-- **OCI remains to be validated.** Run a no-op re-sync check on `oci-lab` before
-  the GitOps layering fully replaces the live setup there.
+- **OCI not yet bootstrapped.** First bootstrap happens after these changes
+  land; validate app sync after cluster registration (see
+  centralize-oci-gitops-and-node-dns in syscode-ai-internal-plans).
 - Migrated: cert-manager, cilium (shared, `apps/`); harbor, arc-controller
   (unraid-only, `clusters/unraid-lab/apps/`). arc-runners stays a custom
   ApplicationSet on the raw lane.