From 353ddd88e156c31e69f57c30d1861a34c6ca28d8 Mon Sep 17 00:00:00 2001 From: stitrace Date: Fri, 5 Jun 2026 23:52:37 +0700 Subject: [PATCH 1/4] flannel migration: document removing leftover flannel iptables rules The live-migration controller removes the flannel daemonset and deletes the flannel network devices, but it does not remove the iptables chains flannel programs (FLANNEL-POSTRTG in nat, FLANNEL-FWD in filter). These survive the migration and the FLANNEL-POSTRTG masquerade rule keeps SNAT-ing cross-node pod-to-pod traffic to the node tunnel IP, which silently breaks NetworkPolicy after migration. Add a cleanup step (with a reboot alternative) so operators can remove the leftover rules. --- .../flannel/migration-from-flannel.mdx | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 7bb7540983..6a9c9abfa9 100644 --- a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -92,6 +92,39 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me kubectl delete -f $[manifestsUrl]/manifests/flannel-migration/migration-job.yaml ``` +1. Remove leftover flannel `iptables` rules from each node. + + The migration controller removes the flannel daemonset and deletes the flannel + network devices (`flannel.` and `cni0`), but it does not remove the + `iptables` chains that flannel programs: `FLANNEL-POSTRTG` in the `nat` table and + `FLANNEL-FWD` in the `filter` table. These chains survive the migration. + + This matters once you start using Kubernetes `NetworkPolicy`. The masquerade rule + in `FLANNEL-POSTRTG` keeps SNAT-ing cross-node pod-to-pod traffic to the node's + tunnel IP, so the source address no longer matches pod-selector rules and Calico + drops the traffic. The symptom is that cross-node connections to policy-selected + pods time out after an otherwise successful migration, while same-node traffic + keeps working. Remove the leftover chains so that Calico is the only owner of pod + masquerading. + + Run the following on every node. It covers both the legacy and nft `iptables` + backends and is safe to re-run: + + ```bash + for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do + command -v "$ipt" >/dev/null 2>&1 || continue + "$ipt" -w -t nat -D POSTROUTING -j FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -F FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -X FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t filter -D FORWARD -j FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -F FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -X FLANNEL-FWD 2>/dev/null + done + ``` + + Alternatively, reboot each node during a drained maintenance window to clear the + leftover rules. + ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require From 199d7d2ac04769d9afa820e6644e41afab3d4992 Mon Sep 17 00:00:00 2001 From: stitrace Date: Sat, 25 Jul 2026 08:21:13 +0700 Subject: [PATCH 2/4] flannel migration: recommend rolling node reboot as primary cleanup path --- .../flannel/migration-from-flannel.mdx | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 6a9c9abfa9..a004deab0d 100644 --- a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -107,8 +107,22 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me keeps working. Remove the leftover chains so that Calico is the only owner of pod masquerading. - Run the following on every node. It covers both the legacy and nft `iptables` - backends and is safe to re-run: + Because flannel is no longer running, nothing re-creates these chains, so you only + need to clear them once. Choose whichever of the following approaches fits your + cluster. + + **If you can reboot or replace nodes** + + Perform a rolling reboot of the cluster nodes (drain, reboot, uncordon each node + in turn), or replace the nodes entirely if you use immutable or managed node + pools. The leftover rules do not persist across a reboot, so no manual work on + the nodes is needed. + + **If rebooting nodes is undesirable** + + Remove the chains in place by running the following on every node. It takes + effect immediately without disrupting workloads, covers both the legacy and nft + `iptables` backends, and is safe to re-run: ```bash for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do @@ -122,9 +136,6 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me done ``` - Alternatively, reboot each node during a drained maintenance window to clear the - leftover rules. - ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require From 3bb5864e7e21334fefd8b338bf23323c0f416ac1 Mon Sep 17 00:00:00 2001 From: Andrew Chubarov Date: Wed, 12 Aug 2026 17:19:49 +0700 Subject: [PATCH 3/4] docs: tighten leftover flannel iptables cleanup step --- .../flannel/migration-from-flannel.mdx | 38 +++++-------------- 1 file changed, 9 insertions(+), 29 deletions(-) diff --git a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx index a004deab0d..3db282ad7b 100644 --- a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -94,35 +94,15 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me 1. Remove leftover flannel `iptables` rules from each node. - The migration controller removes the flannel daemonset and deletes the flannel - network devices (`flannel.` and `cni0`), but it does not remove the - `iptables` chains that flannel programs: `FLANNEL-POSTRTG` in the `nat` table and - `FLANNEL-FWD` in the `filter` table. These chains survive the migration. - - This matters once you start using Kubernetes `NetworkPolicy`. The masquerade rule - in `FLANNEL-POSTRTG` keeps SNAT-ing cross-node pod-to-pod traffic to the node's - tunnel IP, so the source address no longer matches pod-selector rules and Calico - drops the traffic. The symptom is that cross-node connections to policy-selected - pods time out after an otherwise successful migration, while same-node traffic - keeps working. Remove the leftover chains so that Calico is the only owner of pod - masquerading. - - Because flannel is no longer running, nothing re-creates these chains, so you only - need to clear them once. Choose whichever of the following approaches fits your - cluster. - - **If you can reboot or replace nodes** - - Perform a rolling reboot of the cluster nodes (drain, reboot, uncordon each node - in turn), or replace the nodes entirely if you use immutable or managed node - pools. The leftover rules do not persist across a reboot, so no manual work on - the nodes is needed. - - **If rebooting nodes is undesirable** - - Remove the chains in place by running the following on every node. It takes - effect immediately without disrupting workloads, covers both the legacy and nft - `iptables` backends, and is safe to re-run: + The migration controller deletes the flannel network devices but leaves flannel's `iptables` chains (`FLANNEL-POSTRTG` in `nat`, `FLANNEL-FWD` in `filter`) in place. Their masquerade rule SNATs cross-node pod traffic, which breaks Kubernetes `NetworkPolicy` for policy-selected pods. Because flannel is no longer running, you only need to clear the chains once. Use either method: + + **Method 1: Reboot or replace nodes** + + Perform a rolling reboot (drain, reboot, uncordon each node in turn), or replace the nodes if you use immutable or managed node pools. The rules do not persist across a reboot, so no work on the nodes is needed. + + **Method 2: Remove the chains in place** + + Run the following on every node. It takes effect immediately without disrupting workloads, covers both the legacy and nft backends, and is safe to re-run: ```bash for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do From 1019f4ba695fc7a3721d0c23a401665821a15703 Mon Sep 17 00:00:00 2001 From: Andrey Chubarov Date: Wed, 12 Aug 2026 17:29:44 +0700 Subject: [PATCH 4/4] docs: move flannel iptables cleanup step to versioned docs only The controller-side fix lands in 3.33, so the manual step is only needed for clusters migrated with 3.29-3.32. --- .../flannel/migration-from-flannel.mdx | 24 ------------------- .../flannel/migration-from-flannel.mdx | 24 +++++++++++++++++++ .../flannel/migration-from-flannel.mdx | 24 +++++++++++++++++++ .../flannel/migration-from-flannel.mdx | 24 +++++++++++++++++++ .../flannel/migration-from-flannel.mdx | 24 +++++++++++++++++++ 5 files changed, 96 insertions(+), 24 deletions(-) diff --git a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 3db282ad7b..7bb7540983 100644 --- a/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -92,30 +92,6 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me kubectl delete -f $[manifestsUrl]/manifests/flannel-migration/migration-job.yaml ``` -1. Remove leftover flannel `iptables` rules from each node. - - The migration controller deletes the flannel network devices but leaves flannel's `iptables` chains (`FLANNEL-POSTRTG` in `nat`, `FLANNEL-FWD` in `filter`) in place. Their masquerade rule SNATs cross-node pod traffic, which breaks Kubernetes `NetworkPolicy` for policy-selected pods. Because flannel is no longer running, you only need to clear the chains once. Use either method: - - **Method 1: Reboot or replace nodes** - - Perform a rolling reboot (drain, reboot, uncordon each node in turn), or replace the nodes if you use immutable or managed node pools. The rules do not persist across a reboot, so no work on the nodes is needed. - - **Method 2: Remove the chains in place** - - Run the following on every node. It takes effect immediately without disrupting workloads, covers both the legacy and nft backends, and is safe to re-run: - - ```bash - for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do - command -v "$ipt" >/dev/null 2>&1 || continue - "$ipt" -w -t nat -D POSTROUTING -j FLANNEL-POSTRTG 2>/dev/null - "$ipt" -w -t nat -F FLANNEL-POSTRTG 2>/dev/null - "$ipt" -w -t nat -X FLANNEL-POSTRTG 2>/dev/null - "$ipt" -w -t filter -D FORWARD -j FLANNEL-FWD 2>/dev/null - "$ipt" -w -t filter -F FLANNEL-FWD 2>/dev/null - "$ipt" -w -t filter -X FLANNEL-FWD 2>/dev/null - done - ``` - ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require diff --git a/calico_versioned_docs/version-3.29/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico_versioned_docs/version-3.29/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 994dc5f1a5..1e90be7edc 100644 --- a/calico_versioned_docs/version-3.29/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico_versioned_docs/version-3.29/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -92,6 +92,30 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me kubectl delete -f $[manifestsUrl]/manifests/flannel-migration/migration-job.yaml ``` +1. Remove leftover flannel `iptables` rules from each node. + + The migration controller deletes the flannel network devices but leaves flannel's `iptables` chains (`FLANNEL-POSTRTG` in `nat`, `FLANNEL-FWD` in `filter`) in place. Their masquerade rule SNATs cross-node pod traffic, which breaks Kubernetes `NetworkPolicy` for policy-selected pods. Because flannel is no longer running, you only need to clear the chains once. Use either method: + + **Method 1: Reboot or replace nodes** + + Perform a rolling reboot (drain, reboot, uncordon each node in turn), or replace the nodes if you use immutable or managed node pools. The rules do not persist across a reboot, so no work on the nodes is needed. + + **Method 2: Remove the chains in place** + + Run the following on every node. It takes effect immediately without disrupting workloads, covers both the legacy and nft backends, and is safe to re-run: + + ```bash + for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do + command -v "$ipt" >/dev/null 2>&1 || continue + "$ipt" -w -t nat -D POSTROUTING -j FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -F FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -X FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t filter -D FORWARD -j FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -F FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -X FLANNEL-FWD 2>/dev/null + done + ``` + ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require diff --git a/calico_versioned_docs/version-3.30/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico_versioned_docs/version-3.30/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 994dc5f1a5..1e90be7edc 100644 --- a/calico_versioned_docs/version-3.30/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico_versioned_docs/version-3.30/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -92,6 +92,30 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me kubectl delete -f $[manifestsUrl]/manifests/flannel-migration/migration-job.yaml ``` +1. Remove leftover flannel `iptables` rules from each node. + + The migration controller deletes the flannel network devices but leaves flannel's `iptables` chains (`FLANNEL-POSTRTG` in `nat`, `FLANNEL-FWD` in `filter`) in place. Their masquerade rule SNATs cross-node pod traffic, which breaks Kubernetes `NetworkPolicy` for policy-selected pods. Because flannel is no longer running, you only need to clear the chains once. Use either method: + + **Method 1: Reboot or replace nodes** + + Perform a rolling reboot (drain, reboot, uncordon each node in turn), or replace the nodes if you use immutable or managed node pools. The rules do not persist across a reboot, so no work on the nodes is needed. + + **Method 2: Remove the chains in place** + + Run the following on every node. It takes effect immediately without disrupting workloads, covers both the legacy and nft backends, and is safe to re-run: + + ```bash + for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do + command -v "$ipt" >/dev/null 2>&1 || continue + "$ipt" -w -t nat -D POSTROUTING -j FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -F FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -X FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t filter -D FORWARD -j FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -F FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -X FLANNEL-FWD 2>/dev/null + done + ``` + ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require diff --git a/calico_versioned_docs/version-3.31/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico_versioned_docs/version-3.31/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 994dc5f1a5..1e90be7edc 100644 --- a/calico_versioned_docs/version-3.31/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico_versioned_docs/version-3.31/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -92,6 +92,30 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me kubectl delete -f $[manifestsUrl]/manifests/flannel-migration/migration-job.yaml ``` +1. Remove leftover flannel `iptables` rules from each node. + + The migration controller deletes the flannel network devices but leaves flannel's `iptables` chains (`FLANNEL-POSTRTG` in `nat`, `FLANNEL-FWD` in `filter`) in place. Their masquerade rule SNATs cross-node pod traffic, which breaks Kubernetes `NetworkPolicy` for policy-selected pods. Because flannel is no longer running, you only need to clear the chains once. Use either method: + + **Method 1: Reboot or replace nodes** + + Perform a rolling reboot (drain, reboot, uncordon each node in turn), or replace the nodes if you use immutable or managed node pools. The rules do not persist across a reboot, so no work on the nodes is needed. + + **Method 2: Remove the chains in place** + + Run the following on every node. It takes effect immediately without disrupting workloads, covers both the legacy and nft backends, and is safe to re-run: + + ```bash + for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do + command -v "$ipt" >/dev/null 2>&1 || continue + "$ipt" -w -t nat -D POSTROUTING -j FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -F FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -X FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t filter -D FORWARD -j FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -F FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -X FLANNEL-FWD 2>/dev/null + done + ``` + ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require diff --git a/calico_versioned_docs/version-3.32/getting-started/kubernetes/flannel/migration-from-flannel.mdx b/calico_versioned_docs/version-3.32/getting-started/kubernetes/flannel/migration-from-flannel.mdx index 7bb7540983..3db282ad7b 100644 --- a/calico_versioned_docs/version-3.32/getting-started/kubernetes/flannel/migration-from-flannel.mdx +++ b/calico_versioned_docs/version-3.32/getting-started/kubernetes/flannel/migration-from-flannel.mdx @@ -92,6 +92,30 @@ There are two ways to switch your cluster to use $[prodname] networking. Both me kubectl delete -f $[manifestsUrl]/manifests/flannel-migration/migration-job.yaml ``` +1. Remove leftover flannel `iptables` rules from each node. + + The migration controller deletes the flannel network devices but leaves flannel's `iptables` chains (`FLANNEL-POSTRTG` in `nat`, `FLANNEL-FWD` in `filter`) in place. Their masquerade rule SNATs cross-node pod traffic, which breaks Kubernetes `NetworkPolicy` for policy-selected pods. Because flannel is no longer running, you only need to clear the chains once. Use either method: + + **Method 1: Reboot or replace nodes** + + Perform a rolling reboot (drain, reboot, uncordon each node in turn), or replace the nodes if you use immutable or managed node pools. The rules do not persist across a reboot, so no work on the nodes is needed. + + **Method 2: Remove the chains in place** + + Run the following on every node. It takes effect immediately without disrupting workloads, covers both the legacy and nft backends, and is safe to re-run: + + ```bash + for ipt in iptables-legacy iptables-nft ip6tables-legacy ip6tables-nft; do + command -v "$ipt" >/dev/null 2>&1 || continue + "$ipt" -w -t nat -D POSTROUTING -j FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -F FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t nat -X FLANNEL-POSTRTG 2>/dev/null + "$ipt" -w -t filter -D FORWARD -j FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -F FLANNEL-FWD 2>/dev/null + "$ipt" -w -t filter -X FLANNEL-FWD 2>/dev/null + done + ``` + ### Modify flannel configuration The migration controller autodetects your flannel configuration, and in most cases, does not require