Repository navigation
fix(policies): add-default-deny has generated an invalid CiliumNetworkPolicy into every namespace for ~101 days #3501
Description
🤖 Generated by the Agentic Engineer
Evidence
The add-default-deny ClusterPolicy's generate-default-deny rule has never produced a valid resource. Every CiliumNetworkPolicy/default-deny it generates is rejected by Cilium:
status.conditions[Valid] = False
message: "rule must have at least one of Ingress, IngressDeny, Egress, EgressDeny"
Measured live in prod (2026-09-01): 39 invalid default-deny CiliumNetworkPolicies — one in every namespace the rule targets, and every single one invalid. Unfiltered control: of all CNPs cluster-wide, the only ones with Valid=False are these 39; every allow-* CNP is Valid=True. Oldest such object dates to 2026-05-23, so the rule has been generating a rejected resource for roughly 101 days.
The cause is stated in the policy's own comment, which is mistaken about the CRD:
Cilium's CRD also requires at least one policy direction to be present, so retain empty allow lists for both directions.
ingress: [] / egress: [] are empty arrays. Cilium requires at least one direction to be present with a rule, and an empty list does not satisfy that — so the whole object is rejected, enableDefaultDeny included.
Impact — no open hole today, but one brace of a two-brace design is missing
Enforcement is currently intact, and I want to be precise about that rather than overstate this. The sibling rule generate-default-deny-networkpolicy generates a standard Kubernetes NetworkPolicy (podSelector: {}, policyTypes: [Ingress, Egress], no allow rules), which Cilium does enforce. Verified: 39 of 43 namespaces carry it; the 4 without are kube-system, kube-public, kube-node-lease and kyverno, which the policy deliberately excludes. So namespaces are isolated today.
What is actually broken:
- The documented design is half-missing. The policy describes a deliberate belt-and-braces arrangement — the Cilium policy activates whitelist mode, the standard NetworkPolicy makes segmentation visible to scanners and CIS (C-0030/C-0260/C-0206). The Cilium brace has never existed. Anyone who trims the "redundant" standard NetworkPolicy on the strength of the comment would open a real gap.
- 39 permanently-invalid objects in prod. Any check asking "are there invalid network policies?" is permanently dirty, which trains readers to ignore it — a guard that always fires is one nobody reads.
- It has propagated. The identical file exists in
devantler-tech/platform-templateat the same path, so every platform created from the template inherits the broken rule.
Expected
generate-default-deny produces a CNP that Cilium accepts (Valid=True) and that activates default-deny by endpoint selection, without introducing an explicit ingressDeny/egressDeny — the policy comment is right that an explicit deny would take precedence over each namespace's allow policies.
⚠️ Hazard — this change can take prod down, and has before
A deny-shaped Cilium policy change caused a live prod egress outage in this cluster (CoreDNS to host DNS was cut in under 10 seconds, and a render-only test passed beforehand). enableDefaultDeny defaults to true for any direction that carries rules, so getting this subtly wrong revokes egress cluster-wide across all 39 namespaces at once. This is explicitly not a quick fix to push:
- prove the corrected shape on a throwaway namespace on a local/dev cluster first, asserting
Valid=Trueand that an existing allow policy still admits its traffic; - a render/schema test is not sufficient evidence — the previous outage passed exactly that;
- stage the rollout rather than flipping all namespaces at once.
Acceptance criteria
-
generate-default-denyemits a CNP that Cilium reportsValid=True. - Existing per-namespace
allow-*CNPs still admit their traffic (DNS egress specifically verified). - Zero
Valid=FalseCiliumNetworkPolicies remain in prod. - The misleading "retain empty allow lists" comment is corrected to describe what the CRD actually requires.
- The same fix lands in
devantler-tech/platform-template. - A check fails if any generated CNP is
Valid=False, so this cannot silently regress for another 101 days.
Rough size: S–M for the policy change; the validation and staged rollout are the bulk of the work.
Activity
devantler commentedon Oct 2, 2026
Re-verified 2026-10-02. The fix shape is confirmed and it cannot narrow traffic, but the hazard gate above still blocks a push from this agent host.
Still broken. A read-only prod read today finds 167 CiliumNetworkPolicies. 39 are
Valid=False, and all 39 aredefault-deny, with the same spec as when this issue was filed (ingress: [],egress: []).The corrected shape. Cilium's Layer 3 docs ("Ingress/Egress Default Deny") put an endpoint into default deny for a direction with a rule section holding one empty rule:
ingress: [{}]/egress: [{}]. That section allows no peers. It also satisfies the validator ("at least one of Ingress, IngressDeny, Egress, EgressDeny"), and it needs noingressDenyoregressDeny.Why it cannot narrow traffic in prod. All 39 namespaces that carry the invalid CNP also carry the standard
default-denyNetworkPolicy (podSelector: {},Ingress+Egress), measured today, 39 of 39. Cilium already enforces default deny in both directions there, and allows are a union across policy types. A valid CNP that allows nothing therefore leaves the allowed set and the enforcement mode as they are. If the empty rule were ever read as allow-all, the failure would be widening, not an outage. That is still why the live proof matters.Blocker. The hazard section requires proving the shape on a throwaway cluster before rollout. This host has no container runtime. Neither platform's CI nor platform-template's runs an ephemeral cluster: the template's System Test is skipped in the template repository. So nothing reachable from here can run the proof.
Cheapest ways to unblock:
Once unblocked, the change is the two lines above, a corrected comment, and a static check in
validatethat fails on any generated CiliumNetworkPolicy with no non-empty direction.chore: agent-claim #3501 nonce=61765de10d1efe386f79431a0b806cd8
devantler commentedon Oct 3, 2026
The explicitly approved disposable proof has now cleared the issue’s pre-production hazard gate for candidate head 01e40ec using reviewed main workflow head 42d5010: https://github.com/devantler-tech/platform/actions/runs/37102114892
The finite receipt reports verdict PASS and cleanup PASS on Cilium 1.20.2. Cilium accepted the generated default-deny and both allow policies as valid. DNS and explicitly allowed HTTP remained reachable; unallowed ingress, egress, and target HTTP were denied. The pre-floor control confirmed the HTTP path was reachable before the deny floor was applied.
This removes the throwaway-cluster blocker, but does not claim completion. Remaining acceptance is the same fix in devantler-tech/platform-template, current-main integration of platform PR #4381, staged production rollout, and a live readback showing zero invalid CiliumNetworkPolicies.
devantler commentedon Oct 3, 2026
Post-merge production readback for #4406 (
fc2cca1a197adee3cc5b9ae113009c0cd9f93ab5):add-default-denypolicy reportsReady=Trueand contains the expected generated Cilium shape;default-denyCiliumNetworkPolicies useingress: [{}]andegress: [{}]with both default-deny flags enabled;The disposable traffic proof remains independently green from run 37102114892, including cleanup.