Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 22 additions & 12 deletions docs/deploy/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,23 +118,33 @@ Detail: `syscode-ai-internal-plans/projects/image-factory-registry/handoffs/2026

## Secrets bootstrap (unraid-lab, ESO + Bitwarden)

DR order for the secrets pipeline: Argo installs ESO + `bitwarden-sdk-server`
(app `external-secrets`); cert-manager issues the sdk-server serving cert.
The single provider token is SOPS-encrypted in the private repo
`syscod3/homelab-secrets` at `unraid-lab/bws-token.enc.yaml`. Restore it once
with:
The provider token is secret zero. It must exist before External Secrets can
read any other Bitwarden value. The automatic path is
`omni/scripts/apply-inline-manifests.sh unraid-lab --apply`: it decrypts the
SOPS file from the sibling private `homelab-secrets` checkout, validates that it
is the expected `external-secrets/bws-token` Secret, and adds it to the same
Omni inline-manifest ConfigPatch as the Argo bootstrap. The decrypted value is
written only to a temporary file and the ConfigPatch is stored by Omni.

The default private-repository location is `../homelab-secrets` relative to
this repository. Override it with `HOMELAB_SECRETS_ROOT` when the checkouts are
elsewhere. The encrypted source remains:
`homelab-secrets/unraid-lab/bws-token.enc.yaml`.

After the cluster starts, Argo installs ESO and the Bitwarden SDK server. ESO
then uses `bws-token` to create the remaining secrets, including the Argo
cluster-registration Secret and Tailscale OAuth Secret. Re-running the command
is safe because the ConfigPatch and Kubernetes Secret are applied
idempotently.

For recovery or verification only, the equivalent manual command is:

```sh
sops -d /path/to/homelab-secrets/unraid-lab/bws-token.enc.yaml | \
kubectl --context omni-unraid-lab apply -f -
kubectl --context omni-unraid-lab -n external-secrets apply -f -
```

Everything else (grafana-cloud, argocd-pocketid-oidc, and
tailscale/operator-oauth) then syncs via ExternalSecrets against
ClusterSecretStore `bitwarden`. The Tailscale Bitwarden value has the client
ID on line one and client secret on line two; ESO templates those into the
keys the operator requires. Never create per-namespace provider tokens; never
paste the token into any agent session.
Never commit the decrypted file or paste the token into a terminal transcript.

## Secrets bootstrap (oci-lab, ESO + Bitwarden)

Expand Down
43 changes: 39 additions & 4 deletions omni/scripts/apply-inline-manifests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ case "$CLUSTER" in
;;
unraid-lab)
MANIFESTS="$REPO_ROOT/clusters/$CLUSTER/omni/inline-manifests.yaml"
PATCH_ID="202-cluster-unraid-lab-omni/patches/inline-manifests.yaml"
PATCH_ID="200-cluster-unraid-lab-omni/patches/inline-manifests.yaml"
ARGOCD_CONFIG="$REPO_ROOT/bootstrap/argocd-cm.yaml"
;;
esac
Expand All @@ -35,6 +35,41 @@ esac
temporary="$(mktemp -d)"
trap 'rm -rf "$temporary"' EXIT

EFFECTIVE_MANIFESTS="$temporary/inline-manifests.yaml"
cp "$MANIFESTS" "$EFFECTIVE_MANIFESTS"

if [[ "$CLUSTER" == "unraid-lab" ]]; then
HOMELAB_SECRETS_ROOT="${HOMELAB_SECRETS_ROOT:-$REPO_ROOT/../homelab-secrets}"
BWS_TOKEN_FILE="$HOMELAB_SECRETS_ROOT/unraid-lab/bws-token.enc.yaml"
[[ -f "$BWS_TOKEN_FILE" ]] || {
printf 'missing encrypted bootstrap secret: %s\n' "$BWS_TOKEN_FILE" >&2
exit 1
}
command -v sops >/dev/null || {
printf 'sops is required to decrypt the unraid-lab bootstrap secret\n' >&2
exit 1
}
sops -d "$BWS_TOKEN_FILE" >"$temporary/bws-token.yaml"
python3 - "$temporary/bws-token.yaml" <<'PY'
import sys

import yaml

secret = yaml.safe_load(open(sys.argv[1]))
if (
secret.get("apiVersion") != "v1"
or secret.get("kind") != "Secret"
or secret.get("metadata", {}).get("name") != "bws-token"
or secret.get("metadata", {}).get("namespace") != "external-secrets"
or "token" not in secret.get("stringData", {}) | secret.get("data", {})
):
raise SystemExit("unexpected bws-token bootstrap Secret shape")
PY
BWS_TOKEN="$temporary/bws-token.yaml" yq -i \
'.cluster.inlineManifests += [{"name":"bootstrap-bws-token","contents": loadstr(strenv(BWS_TOKEN))}] |
(.cluster.inlineManifests[-1].contents) style="literal"' "$EFFECTIVE_MANIFESTS"
fi

cat >"$temporary/configpatch.yaml" <<YAML
metadata:
namespace: default
Expand All @@ -46,10 +81,10 @@ spec:
data: ""
YAML

P="$MANIFESTS" yq -i \
P="$EFFECTIVE_MANIFESTS" yq -i \
'.spec.data = loadstr(strenv(P)) | .spec.data style="literal"' "$temporary/configpatch.yaml"

python3 - "$MANIFESTS" <<'PY'
python3 - "$EFFECTIVE_MANIFESTS" <<'PY'
import sys

import yaml
Expand All @@ -60,7 +95,7 @@ for inline_manifest in inline_manifests:
print("inline manifests: valid")
PY

python3 - "$MANIFESTS" "$temporary/argocd.yaml" <<'PY'
python3 - "$EFFECTIVE_MANIFESTS" "$temporary/argocd.yaml" <<'PY'
import sys
from pathlib import Path

Expand Down
Loading