diff --git a/cos-ca-cert-mitigation/README.md b/cos-ca-cert-mitigation/README.md new file mode 100644 index 0000000..637903b --- /dev/null +++ b/cos-ca-cert-mitigation/README.md @@ -0,0 +1,83 @@ +# Container-Optimized OS (COS) Live CA-Certificate Mitigation + +This directory provides live mitigation tools to distrust the compromised eMudhra (`emSign`) root CAs and the cross-signing `AAA Certificate Services` (`Comodo_AAA_Services_root.crt`) root CA on running **Container-Optimized OS (COS)** instances and **Google Kubernetes Engine (GKE)** Standard nodes before upgrading to patched OS images. + +## What This Mitigation Does + +1. **System CA Store (`/etc/ca-certificates.conf` & `/etc/ssl/certs`)**: + - Deselects all `mozilla/emSign_*.crt` roots and `mozilla/Comodo_AAA_Services_root.crt` in `/etc/ca-certificates.conf` and runs `update-ca-certificates --fresh`. + - Mitigates host OpenSSL/C/C++ consumers (`curl`, `wget`, `git`, `oslogin`, `systemd`, `chrony`), host Python (`certifi`, `urllib3`, `requests`, `cloud-init`), and Go `crypto/x509` CLIs (`kubectl`, `crictl`, `ctr`, `docker`). +2. **Read-Only `/usr` Trust Stores (`dm-verity`)**: + - Masks `/usr/share/chromeos-ca-certificates/Comodo_AAA_Services_root.pem` via a bind mount from `/dev/null` (mitigating `update_engine`, `libbrillo`, and `crash_sender`). + - Bind-mounts the regenerated `/etc/ssl/certs/ca-certificates.crt` over `/usr/share/grpc/roots.pem` if present. +3. **Built-In Container Utilities (`toolbox` & `cos-extensions`)**: + - Updates `TOOLBOX_DOCKER_TAG="v20261020"` in `/etc/default/toolbox`. + - Wraps `/usr/bin/cos-extensions` via `/run/cos-mitigation` so `cos-extensions install gpu` defaults to `gcr.io/cos-cloud/cos-gpu-installer:v2.7.11`. +4. **Active Host Daemons**: + - Restarts active host services (`containerd`, `docker`, `kubelet`, `fluent-bit`, `update-engine`, `node-problem-detector`, `google-guest-agent*`, `google-osconfig-agent`, `device_policy_manager`) so long-running Go and C/C++ processes reload `/etc/ssl/certs/ca-certificates.crt` without disrupting running pods. + +--- + +## 1. GKE Standard Nodes (`COS_CONTAINERD`) + +Deploy the `cos-ca-cert-mitigator` DaemonSet to automatically apply the mitigation across all existing and newly autoscaled COS nodes in a GKE Standard cluster. + +### Prerequisites & Compatibility +* **GKE Standard only:** This DaemonSet requires privileged host access (`hostPID: true`, `privileged: true`) to update host trust stores and restart node services. +* **No Node Reboot or Pod Eviction:** Restarting `containerd` and `kubelet` reloads the host trust store in-place while running pods remain managed by their `containerd-shim` processes. + +### Usage + +Apply the DaemonSet to your cluster: + +```bash +kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-node-tools/master/cos-ca-cert-mitigation/gke/cos-ca-cert-mitigation.yaml +``` + +Or apply the local file: + +```bash +kubectl apply -f gke/cos-ca-cert-mitigation.yaml +``` + +Monitor the rollout status and inspect the init container logs: + +```bash +kubectl -n kube-system rollout status daemonset/cos-ca-cert-mitigator +kubectl -n kube-system logs -l k8s-app=cos-ca-cert-mitigator -c apply-cos-ca-cert-mitigation +``` + +### Removal + +Once your node pools have been upgraded to a patched COS image release, remove the DaemonSet: + +```bash +kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-node-tools/master/cos-ca-cert-mitigation/gke/cos-ca-cert-mitigation.yaml +``` + +--- + +## 2. Standalone Container-Optimized OS (COS) GCE VMs + +On standalone COS GCE VMs, copy `cos/mitigate_cos_ca_certs.sh` to the instance and execute it as `root`: + +```bash +sudo bash mitigate_cos_ca_certs.sh +``` + +> **Note:** Because `/tmp` and `/home` on COS are mounted `noexec`, invoke the script with `sudo bash mitigate_cos_ca_certs.sh` rather than `./mitigate_cos_ca_certs.sh`. + +--- + +## Important Notes + +* **Persistence Across Reboots:** Changes made by `mitigate_cos_ca_certs.sh` are **not** persistent across VM reboots. + * On **GKE**, keep the `cos-ca-cert-mitigator` DaemonSet deployed so the mitigation is automatically re-applied if a node reboots or new nodes are added by the cluster autoscaler. + * On **standalone COS GCE VMs**, configure a startup script or `cloud-init` config to run `mitigate_cos_ca_certs.sh` on each boot (see [Creating and configuring instances](https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance)). +* **Running or Cached Containers:** This script does **not** update running or cached containers that bundle their own root certificates. Pull fixed container images and recreate any containers created from unpatched images. +* **Patched COS Container Utilities:** + * `gcr.io/cos-cloud/cos-gpu-installer:v2.7.11` (default updated in `cos-extensions`) + * `gcr.io/cos-cloud/toolbox:v20261020` (default updated in `/etc/default/toolbox`; if you customized `~/.toolboxrc`, update `TOOLBOX_DOCKER_TAG` there as well) + * `gcr.io/cos-cloud/cos-dkms:v0.3.19` + * `gcr.io/cos-cloud/cos-customizer:v20261020` +* **`ensurepip`:** This script does not apply mitigations to the bundled `ensurepip` wheel (do not use `python3 -m ensurepip`). diff --git a/cos-ca-cert-mitigation/cos/mitigate_cos_ca_certs.sh b/cos-ca-cert-mitigation/cos/mitigate_cos_ca_certs.sh new file mode 100755 index 0000000..6a5b014 --- /dev/null +++ b/cos-ca-cert-mitigation/cos/mitigate_cos_ca_certs.sh @@ -0,0 +1,129 @@ +#!/bin/bash +# +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Mitigation script for eMudhra certificate compromise on Container-Optimized OS (COS). + +set -euo pipefail + +if [[ $EUID -ne 0 ]]; then + echo "This script must be run as root in order to apply mitigations." + exit 1 +fi + +# Match all 5 emSign root CAs and Comodo_AAA_Services_root.crt (AAA Certificate +# Services), which cross-signed "emSign ECC Root CA - G3". +BAD_CERT_REGEX='^mozilla/(emSign_.*|Comodo_AAA_Services_root)\.crt' + +# ============================================================================== +# Step 1: System CA Store (app-misc/ca-certificates) +# ============================================================================== +# Prefix matching entries in /etc/ca-certificates.conf with "!" to deselect them, +# then regenerate /etc/ssl/certs/ca-certificates.crt and hash symlinks. +# This mitigates all host OpenSSL/C/C++ consumers (curl, wget, git, oslogin, +# systemd, chrony), host Python packages (dev-python/certifi-10001 is a shim +# pointing to /etc/ssl/certs/ca-certificates.crt), and Go crypto/x509 CLIs. +echo "[1/4] Updating system CA store (/etc/ca-certificates.conf & /etc/ssl/certs)..." +sed -i -E "s#${BAD_CERT_REGEX}#!&#" /etc/ca-certificates.conf +update-ca-certificates --fresh + +# ============================================================================== +# Step 2: Standalone Trust Stores on Read-Only /usr (dm-verity) +# ============================================================================== +# Because /usr on COS is a read-only dm-verity filesystem, packages that ship +# their own CA files under /usr are mitigated via bind mounts. +echo "[2/4] Mitigating read-only /usr trust stores (chromeos-ca-certificates, grpc)..." + +# 2a. chromeos-base/chromeos-ca-certificates: +# update_engine, libbrillo-http, and crash_sender hardcode +# /usr/share/chromeos-ca-certificates instead of /etc/ssl/certs. Mask the +# cross-signing Comodo_AAA_Services_root.pem with /dev/null. +CROS_COMODO="/usr/share/chromeos-ca-certificates/Comodo_AAA_Services_root.pem" +[[ -s "${CROS_COMODO}" ]] && mount --bind /dev/null "${CROS_COMODO}" + +# 2b. net-libs/grpc: +# If /usr/share/grpc/roots.pem is present, bind-mount the mitigated system bundle. +if [[ -f /usr/share/grpc/roots.pem ]]; then + umount -l /usr/share/grpc/roots.pem 2>/dev/null || true + mount --bind /etc/ssl/certs/ca-certificates.crt /usr/share/grpc/roots.pem +fi + +# ============================================================================== +# Step 3: Point Built-In Container Utilities (toolbox & cos-extensions) to Fixed Images +# ============================================================================== +echo "[3/4] Updating default container tags for toolbox and cos-extensions..." + +# 3a. app-admin/toolbox: +# Update system default tag in /etc/default/toolbox to the patched release. +sed -i 's/^TOOLBOX_DOCKER_TAG=.*/TOOLBOX_DOCKER_TAG="v20261020"/' /etc/default/toolbox + +# 3b. app-admin/extensions-manager (/usr/bin/cos-extensions): +# Wrap /usr/bin/cos-extensions so `cos-extensions install gpu` defaults to the +# patched cos-gpu-installer:v2.7.11 image via COS_GPU_INSTALLER. +# Because /run is mounted with noexec on COS, bind-mount and remount +# /run/cos-mitigation with exec so both .wrapper and .orig can be executed. +WRAP_DIR="/run/cos-mitigation" +mkdir -p -m 0755 "${WRAP_DIR}" +mountpoint -q "${WRAP_DIR}" || mount --bind "${WRAP_DIR}" "${WRAP_DIR}" +mount -o remount,bind,exec "${WRAP_DIR}" + +umount -l /usr/bin/cos-extensions 2>/dev/null || true +cp -fp /usr/bin/cos-extensions "${WRAP_DIR}/cos-extensions.orig" + +cat <<'EOF' > "${WRAP_DIR}/cos-extensions.wrapper" +#!/bin/bash +export COS_GPU_INSTALLER="${COS_GPU_INSTALLER:-gcr.io/cos-cloud/cos-gpu-installer:v2.7.11}" +exec /run/cos-mitigation/cos-extensions.orig "$@" +EOF + +chmod 0755 "${WRAP_DIR}" "${WRAP_DIR}/cos-extensions.orig" "${WRAP_DIR}/cos-extensions.wrapper" +mount --bind "${WRAP_DIR}/cos-extensions.wrapper" /usr/bin/cos-extensions + +# ============================================================================== +# Step 4: Restart Active Daemons +# ============================================================================== +# Go's crypto/x509 loads /etc/ssl/certs/ca-certificates.crt into memory once per +# process via sync.Once. Restarting active Go and C/C++ daemons forces them to +# reload the mitigated system trust store. +echo "[4/4] Restarting active daemons to reload root certificates..." +for svc in containerd docker kubelet fluent-bit update-engine node-problem-detector \ + google-guest-agent google-guest-agent-manager google-guest-compat-manager \ + google-osconfig-agent device_policy_manager; do + systemctl try-restart "${svc}.service" 2>/dev/null || true +done + +echo "Automatic mitigation complete." + +# ============================ IMPORTANT ======================================= +# +# These changes are not persistent. You will need to run this script again to +# reapply the mitigations if this instance reboots. Refer to +# https://docs.cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance +# to configure your instance to run this script on each boot. +# +# This has NOT updated any running or cached containers which may be impacted. +# You will need to install fixed container images and recreate any containers +# created from unpatched images. +# +# The following container images supported by COS have fixes available: +# - gcr.io/cos-cloud/cos-gpu-installer:v2.7.11 (default updated in cos-extensions) +# - gcr.io/cos-cloud/toolbox:v20261020 (default updated in /etc/default/toolbox; +# if you customized ~/.toolboxrc, update TOOLBOX_DOCKER_TAG there as well) +# - gcr.io/cos-cloud/cos-dkms:v0.3.19 +# - gcr.io/cos-cloud/cos-customizer:v20261020 +# +# This script has not applied mitigations to ensurepip (do not use `python3 -m ensurepip`). +# +# ============================================================================== diff --git a/cos-ca-cert-mitigation/gke/cos-ca-cert-mitigation.yaml b/cos-ca-cert-mitigation/gke/cos-ca-cert-mitigation.yaml new file mode 100644 index 0000000..38c0c29 --- /dev/null +++ b/cos-ca-cert-mitigation/gke/cos-ca-cert-mitigation.yaml @@ -0,0 +1,205 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Deploy this DaemonSet to apply live CA-certificate mitigations for the +# eMudhra (emSign) root/intermediate CA compromise across all existing and +# newly autoscaled Container-Optimized OS (COS) nodes in a GKE Standard +# cluster without rebooting nodes or disrupting running pods. + +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: cos-ca-cert-mitigator + namespace: kube-system + labels: + k8s-app: cos-ca-cert-mitigator +spec: + selector: + matchLabels: + k8s-app: cos-ca-cert-mitigator + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + k8s-app: cos-ca-cert-mitigator + spec: + hostPID: true + nodeSelector: + kubernetes.io/os: linux + cloud.google.com/gke-os-distribution: cos + tolerations: + - operator: Exists + initContainers: + - name: apply-cos-ca-cert-mitigation + image: gke.gcr.io/debian-base:bookworm-v1.0.0-gke.1 + imagePullPolicy: IfNotPresent + securityContext: + privileged: true + resources: + requests: + memory: 15Mi + cpu: 5m + command: + - nsenter + - -t + - "1" + - -m + - -u + - -i + - -n + - -p + - -- + - /bin/bash + - -c + - | + #!/bin/bash + # + # Copyright 2026 Google LLC + # + # Licensed under the Apache License, Version 2.0 (the "License"); + # you may not use this file except in compliance with the License. + # You may obtain a copy of the License at + # + # https://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. + + # Mitigation script for eMudhra certificate compromise on Container-Optimized OS (COS). + + set -euo pipefail + + if [[ -f /tmp/cos_ca_cert_mitigated ]]; then + echo "Mitigations already applied." + exit 0 + fi + + if [[ $EUID -ne 0 ]]; then + echo "This script must be run as root in order to apply mitigations." + exit 1 + fi + + # Match all 5 emSign root CAs and Comodo_AAA_Services_root.crt (AAA Certificate + # Services), which cross-signed "emSign ECC Root CA - G3". + BAD_CERT_REGEX='^mozilla/(emSign_.*|Comodo_AAA_Services_root)\.crt' + + # ============================================================================== + # Step 1: System CA Store (app-misc/ca-certificates) + # ============================================================================== + # Prefix matching entries in /etc/ca-certificates.conf with "!" to deselect them, + # then regenerate /etc/ssl/certs/ca-certificates.crt and hash symlinks. + # This mitigates all host OpenSSL/C/C++ consumers (curl, wget, git, oslogin, + # systemd, chrony), host Python packages (dev-python/certifi-10001 is a shim + # pointing to /etc/ssl/certs/ca-certificates.crt), and Go crypto/x509 CLIs. + echo "[1/4] Updating system CA store (/etc/ca-certificates.conf & /etc/ssl/certs)..." + sed -i -E "s#${BAD_CERT_REGEX}#!&#" /etc/ca-certificates.conf + update-ca-certificates --fresh + + # ============================================================================== + # Step 2: Standalone Trust Stores on Read-Only /usr (dm-verity) + # ============================================================================== + # Because /usr on COS is a read-only dm-verity filesystem, packages that ship + # their own CA files under /usr are mitigated via bind mounts. + echo "[2/4] Mitigating read-only /usr trust stores (chromeos-ca-certificates, grpc)..." + + # 2a. chromeos-base/chromeos-ca-certificates: + # update_engine, libbrillo-http, and crash_sender hardcode + # /usr/share/chromeos-ca-certificates instead of /etc/ssl/certs. Mask the + # cross-signing Comodo_AAA_Services_root.pem with /dev/null. + CROS_COMODO="/usr/share/chromeos-ca-certificates/Comodo_AAA_Services_root.pem" + [[ -s "${CROS_COMODO}" ]] && mount --bind /dev/null "${CROS_COMODO}" + + # 2b. net-libs/grpc: + # If /usr/share/grpc/roots.pem is present, bind-mount the mitigated system bundle. + if [[ -f /usr/share/grpc/roots.pem ]]; then + umount -l /usr/share/grpc/roots.pem 2>/dev/null || true + mount --bind /etc/ssl/certs/ca-certificates.crt /usr/share/grpc/roots.pem + fi + + # ============================================================================== + # Step 3: Point Built-In Container Utilities (toolbox & cos-extensions) to Fixed Images + # ============================================================================== + echo "[3/4] Updating default container tags for toolbox and cos-extensions..." + + # 3a. app-admin/toolbox: + # Update system default tag in /etc/default/toolbox to the patched release. + sed -i 's/^TOOLBOX_DOCKER_TAG=.*/TOOLBOX_DOCKER_TAG="v20261020"/' /etc/default/toolbox + + # 3b. app-admin/extensions-manager (/usr/bin/cos-extensions): + # Wrap /usr/bin/cos-extensions so `cos-extensions install gpu` defaults to the + # patched cos-gpu-installer:v2.7.11 image via COS_GPU_INSTALLER. + # Because /run is mounted with noexec on COS, bind-mount and remount + # /run/cos-mitigation with exec so both .wrapper and .orig can be executed. + WRAP_DIR="/run/cos-mitigation" + mkdir -p -m 0755 "${WRAP_DIR}" + mountpoint -q "${WRAP_DIR}" || mount --bind "${WRAP_DIR}" "${WRAP_DIR}" + mount -o remount,bind,exec "${WRAP_DIR}" + + umount -l /usr/bin/cos-extensions 2>/dev/null || true + cp -fp /usr/bin/cos-extensions "${WRAP_DIR}/cos-extensions.orig" + + cat <<'EOF' > "${WRAP_DIR}/cos-extensions.wrapper" + #!/bin/bash + export COS_GPU_INSTALLER="${COS_GPU_INSTALLER:-gcr.io/cos-cloud/cos-gpu-installer:v2.7.11}" + exec /run/cos-mitigation/cos-extensions.orig "$@" + EOF + + chmod 0755 "${WRAP_DIR}" "${WRAP_DIR}/cos-extensions.orig" "${WRAP_DIR}/cos-extensions.wrapper" + mount --bind "${WRAP_DIR}/cos-extensions.wrapper" /usr/bin/cos-extensions + + # ============================================================================== + # Step 4: Restart Active Daemons + # ============================================================================== + # Go's crypto/x509 loads /etc/ssl/certs/ca-certificates.crt into memory once per + # process via sync.Once. Restarting active Go and C/C++ daemons forces them to + # reload the mitigated system trust store. + echo "[4/4] Restarting active daemons to reload root certificates..." + for svc in containerd docker kubelet fluent-bit update-engine node-problem-detector \ + google-guest-agent google-guest-agent-manager google-guest-compat-manager \ + google-osconfig-agent device_policy_manager; do + systemctl try-restart "${svc}.service" 2>/dev/null || true + done + + touch /tmp/cos_ca_cert_mitigated + + echo "Automatic mitigation complete." + + # ============================ IMPORTANT ======================================= + # + # These changes are not persistent. You will need to run this script again to + # reapply the mitigations if this instance reboots. Refer to + # https://docs.cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance + # to configure your instance to run this script on each boot. + # + # This has NOT updated any running or cached containers which may be impacted. + # You will need to install fixed container images and recreate any containers + # created from unpatched images. + # + # The following container images supported by COS have fixes available: + # - gcr.io/cos-cloud/cos-gpu-installer:v2.7.11 (default updated in cos-extensions) + # - gcr.io/cos-cloud/toolbox:v20261020 (default updated in /etc/default/toolbox; + # if you customized ~/.toolboxrc, update TOOLBOX_DOCKER_TAG there as well) + # - gcr.io/cos-cloud/cos-dkms:v0.3.19 + # - gcr.io/cos-cloud/cos-customizer:v20261020 + # + # This script has not applied mitigations to ensurepip (do not use `python3 -m ensurepip`). + # + # ============================================================================== + containers: + - name: pause + image: gke.gcr.io/pause:3.8