Skip to content

Distrust compromised emSign/Comodo AAA roots, bump pip/uv/certifi (NVP-512) - #24

Merged
jelitox merged 3 commits into
mainfrom
fix/emsign-ca-incident
Oct 21, 2026
Merged

jelitox merged 3 commits into
mainfrom
fix/emsign-ca-incident

Conversation

@Oslan17

@Oslan17 Oslan17 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Why

eMudhra/emSign CA incident (disclosed 2026-10-17, https://emudhra.we-have-a-problem.com/): the private keys of eight eMudhra intermediates are recoverable from the certificates. Browsers block the chains via CT, but Python/OpenSSL clients only stop trusting them when their trust store is updated.

All Navigator Python images (navigator-api, navigator-api-ai, dataintegrator, dataintegrator-worker*, navigator-fieldsync) inherit from this image. Current flowcore:main (2026-10-19) has certifi 2026.10.18 (OK) but Debian ca-certificates 20250419~deb12u1, so the system store used by aiohttp, curl and psql still trusts emSign ECC Root CA - G3 and AAA Certificate Services. pip 25.0.1 and uv 0.13.1 also predate the fix.

Jira: NVP-512

What

  • Dockerfile: upgrade ca-certificates (bookworm-security now ships 20250419~deb12u2) and blocklist both roots in /etc/ca-certificates.conf as defence in depth, then update-ca-certificates --fresh.
  • Dockerfile: pip>=26.2.2, uv>=0.13.4 (both carry their own CA bundle).
  • pyproject.toml: certifi>=2026.10.19 floor. Downstream images inherit this uv.lock, so the floor must live here.

Local build verification (docker build on 2026-10-20, exit 0)

Check Result
ca-certificates 20250419~deb12u2
/etc/ssl/certs G3 / AAA absent; the 3 unaffected emSign roots remain
ssl.create_default_context() (aiohttp path) 138 CAs, no G3 / AAA
pip / uv 26.2.2 / 0.13.4
certifi (venv) 2026.10.19, no G3 label
gunicorn 26.2.0 (smoke)

Rollout

Merge → Docker Hub publishes flowcore:main → rebuild navigator-api, navigator-dataintegrator, navigator-dataintegrator-worker and fieldsync → verify staging pods.

Rollback

Revert this PR and republish main; no data or infra changes.

🤖 Generated with Claude Code

Oslan17 and others added 2 commits October 20, 2026 19:59
eMudhra intermediate CA private keys are recoverable (disclosed 2026-10-17).
Their chains end in "emSign ECC Root CA - G3" and "AAA Certificate Services".
certifi already dropped them, but the Debian system trust store used by
OpenSSL (aiohttp default context, curl, psql, git) did not until
ca-certificates 20250419~deb12u2.

- Upgrade ca-certificates and blocklist both roots in /etc/ca-certificates.conf
  (defence in depth), then regenerate /etc/ssl/certs.
- pip >= 26.2.2 and uv >= 0.13.4: both carry their own CA bundle.

Ref: https://emudhra.we-have-a-problem.com/ (NVP-512)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First certifi release that removes "emSign ECC Root CA - G3" while keeping the
three unaffected emSign roots. Downstream images (navigator-api, dataintegrator,
worker, fieldsync) inherit this lock, so the floor has to live here. (NVP-512)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses the eMudhra/emSign CA security incident by upgrading ca-certificates in the Dockerfile, blocklisting the compromised roots, upgrading pip and uv to versions with safe CA bundles, and enforcing a minimum version of certifi in pyproject.toml. The review feedback highlights two critical issues: first, the certificate name for the Comodo AAA root in the sed command is incorrect, which would prevent the blocklist from being applied; second, upgrading pip globally as a non-root user without the --user flag will likely result in a permission error.

Comment thread Dockerfile
Comment on lines +36 to +40
RUN apt-get update -y && apt-get install -y --only-upgrade ca-certificates && \
sed -i -e 's#^mozilla/emSign_ECC_Root_CA_-_G3.crt#!&#' \
-e 's#^mozilla/Comodo_AAA_Services_root.crt#!&#' /etc/ca-certificates.conf && \
update-ca-certificates --fresh && \
apt-get clean && rm -rf /var/lib/apt/lists/*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

The certificate name for the Comodo AAA root in the Debian ca-certificates package is mozilla/AAA_Certificate_Services.crt, not mozilla/Comodo_AAA_Services_root.crt. Because of this, the sed command will not match the entry in /etc/ca-certificates.conf, and the defense-in-depth blocklist will fail to disable the Comodo AAA root if the package upgrade is not available (e.g., due to a stale mirror).

RUN apt-get update -y && apt-get install -y --only-upgrade ca-certificates && \
    sed -i -e 's#^mozilla/emSign_ECC_Root_CA_-_G3.crt#!&#' \
           -e 's#^mozilla/AAA_Certificate_Services.crt#!&#' /etc/ca-certificates.conf && \
    update-ca-certificates --fresh && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

Comment thread Dockerfile
# Install UV using pip (no-cache so pip doesn't bloat the layer).
# pip >= 26.2.2 and uv >= 0.13.4 ship CA bundles without the compromised emSign
# roots (both carry their own trust store and ignore the system one).
RUN pip install --no-cache-dir --upgrade "pip>=26.2.2" "uv>=0.13.4"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Since this command is run as the non-root user troc, attempting to upgrade pip globally without the --user flag can result in a PermissionError when pip tries to uninstall the existing system-wide pip installation. Adding the --user flag ensures that the upgraded pip and uv are installed in the user's home directory (/home/troc/.local/bin), which is already prepended to the PATH in the next line.

RUN pip install --no-cache-dir --user --upgrade "pip>=26.2.2" "uv>=0.13.4"

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jelitox
jelitox merged commit 7914121 into main Oct 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants