From 2af46d46df20b86d3dd2b6885c1f0f8e26f0f912 Mon Sep 17 00:00:00 2001 From: James Nesbitt Date: Fri, 6 Nov 2026 11:30:28 +0200 Subject: [PATCH 1/2] [PRODENG-3929] Admit the NodePort range the aws-simple example installs Second PR on PRODENG-3929. #689 replaced aws-simple's blanket `protocol = "-1"` ingress rule with four scoped ones -- 22 and 5986 from var.admin_cidrs, 443 and 6443 from admin_cidrs plus the VPC CIDR. The scoping is right and stays. What went with the blanket rule, and was not replaced, is the Kubernetes NodePort range: launchpad.tf installs MKE with --nodeport-range=32768-35535 and no rule admitted it at any CIDR, so a re-apply produced a cluster declaring a NodePort range nothing could reach, with no tfvars value that restored it. Measured rather than assumed. The pre-#689 external rule was from_port 0 / to_port 0 / protocol "-1" from 0.0.0.0/0, which in AWS means every port of every protocol, so the literal lost set is everything outside the four new rules. Of the ports this repo actually names, only the NodePort range needs an external rule: - 2377/tcp is node-to-node; SwarmAddress is built from the host's InternalAddress (pkg/product/mke/config/host.go:388) - 7946 tcp+udp, 4789/udp and 10250/tcp appear only in the host-firewall user_data, all node-to-node - 2376/tcp is referenced nowhere in the repo at all All of those are already carried by the self=true rule, which #689 left in place at launchpad.tf:83-90 and which covers every nodegroup, so re-opening them externally would be a widening with no consumer. The range is now stated once, as local.nodeport_range, and feeds both the security group rule and the --nodeport-range install flag. The two previously could not disagree only because one of them did not exist; sharing the local means they still cannot. Source is a new var.nodeport_cidrs rather than admin_cidrs. NodePort services are reached by application clients, who are not the administrators admin_cidrs names: with one list, admitting app clients also hands them SSH and the MKE and Kubernetes APIs. It defaults to null and falls back to admin_cidrs through a local, so an existing tfvars is unchanged and a default run restores exactly the pre-#689 exposure of that range. An empty list is rejected, because a cluster installed with --nodeport-range and no way in is the failure the variable exists to prevent. The rule covers TCP and UDP -- both worked before #689 and a UDP NodePort service is legal -- and takes no VPC CIDR, because nothing in local.launchpad_ingresses load-balances the range, so there is no in-VPC health check to admit. Verified without AWS, with AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE and AWS_EC2_METADATA_DISABLED pinned off throughout. terraform init, validate and fmt -check pass on aws-simple and on untouched aws-complete. A plan against a tfvars reconstructed from the template resolves every variable, plans the three non-AWS resources and stops at "No valid credential sources found", which is as far as it can go. terraform console renders the rule list as tcp 22, tcp 5986, tcp 443, tcp 6443, then tcp 32768-35535 and udp 32768-35535; with admin_cidrs=["198.51.100.17/32"] and nodeport_cidrs=["203.0.113.0/24"] the NodePort rules carry 203.0.113.0/24 while the rest carry 198.51.100.17/32; nodeport_cidrs=[] fails at plan with the validation message; and the flag renders "--nodeport-range=32768-35535", byte-identical to the literal it replaces. No apply, no AWS call of any kind. Two findings left out of this change deliberately, both recorded on the PR. The tfvars user_data opens 2377, 7946, 4789 and 10250 but not the NodePort range, so on rhel_9 and sles_15 firewall-cmd still blocks it whatever the security group says -- that predates #689 and is noted in the README rather than fixed here. And aws-complete declares the same --nodeport-range with no rule for it, before and after #689; it lost nothing, so it is not this regression. Refs: PRODENG-3929 Written by AI: claude-opus-5 Signed-off-by: James Nesbitt --- examples/terraform/aws-simple/README.md | 29 ++++++++++++++ examples/terraform/aws-simple/launchpad.tf | 38 ++++++++++++++++++- .../aws-simple/terraform.tfvars.template | 13 +++++++ examples/terraform/aws-simple/variables.tf | 33 ++++++++++++++++ 4 files changed, 112 insertions(+), 1 deletion(-) diff --git a/examples/terraform/aws-simple/README.md b/examples/terraform/aws-simple/README.md index 42c70103..9c573e3e 100644 --- a/examples/terraform/aws-simple/README.md +++ b/examples/terraform/aws-simple/README.md @@ -25,6 +25,35 @@ admin_cidrs = ["198.51.100.17/32", "203.0.113.0/24"] Node-to-node traffic does not pass through these rules, so narrowing this does not affect the cluster talking to itself. +#### NodePort exposure — `nodeport_cidrs` + +This example installs MKE with `--nodeport-range=32768-35535`, so Kubernetes +NodePort services land in that range on every node. `nodeport_cidrs` lists the +CIDRs allowed to reach it. The range itself is stated once, as +`local.nodeport_range` in `launchpad.tf`, and feeds both the install flag and +the security group rule so the two cannot disagree. + +Leaving it unset means "use `admin_cidrs`", which keeps an existing +`terraform.tfvars` working and carries `admin_cidrs`'s exposure with it: at +both defaults the NodePort range is reachable from the whole internet. Set it +when the people who reach your applications are not the people who administer +the cluster — widening `admin_cidrs` to let application clients in would also +hand them SSH and the MKE and Kubernetes APIs: + +```hcl +nodeport_cidrs = ["203.0.113.0/24"] +``` + +The rule covers TCP and UDP, and does not include the VPC CIDR: nothing in +this example load-balances the NodePort range, and node-to-node traffic is +already covered by the cluster-internal rule. + +On `rhel_9` and `sles_15` nodes the host firewall is a second gate. The +`user_data` in `terraform.tfvars.template` opens 2377, 7946, 4789 and 10250 +but not the NodePort range, so `firewall-cmd` still blocks it there whatever +the security group says. Add the range to that `user_data` if you need +NodePort services on those platforms. + #### Credentials The MKE admin password (`launchpad.mke_connect.password`) and diff --git a/examples/terraform/aws-simple/launchpad.tf b/examples/terraform/aws-simple/launchpad.tf index 2e9a671b..93739644 100644 --- a/examples/terraform/aws-simple/launchpad.tf +++ b/examples/terraform/aws-simple/launchpad.tf @@ -62,6 +62,21 @@ locals { } } + // The Kubernetes NodePort range. Stated once here and used twice: for the + // ingress rule below and for the --nodeport-range install flag in the + // generated launchpad.yaml. Declaring a range the security group does not + // admit produces a cluster whose NodePort services are unreachable, so the + // two must not be able to drift apart. + nodeport_range = { + from = 32768 + to = 35535 + } + + // NodePort services are reached by application clients from outside the + // cluster, who are not necessarily the administrators var.admin_cidrs + // names. Unset means inherit admin_cidrs; see variables.tf. + nodeport_cidrs = var.nodeport_cidrs != null ? var.nodeport_cidrs : var.admin_cidrs + // MCR/MKE/MSR firewall rules. // // Node-to-node traffic inside the group stays unrestricted: MKE, swarm and @@ -75,6 +90,11 @@ locals { // local.launchpad_ingresses above reach the nodes from inside the VPC // rather than from an administrator's address, and their health checks // would fail if only var.admin_cidrs were allowed. + // + // The NodePort range takes its own CIDR list and no VPC CIDR: nothing in + // local.launchpad_ingresses load-balances it, so there is no in-VPC health + // check to admit, and node-to-node NodePort traffic is already covered by + // the self rule. launchpad_securitygroups = { "permissive" = { description = "Common SG for all cluster machines" @@ -119,6 +139,22 @@ locals { protocol : "tcp" self : false cidr_blocks : concat(var.admin_cidrs, [var.network.cidr]) + }, + { + description : "Kubernetes NodePort services (TCP), from application clients" + from_port : local.nodeport_range.from + to_port : local.nodeport_range.to + protocol : "tcp" + self : false + cidr_blocks : local.nodeport_cidrs + }, + { + description : "Kubernetes NodePort services (UDP), from application clients" + from_port : local.nodeport_range.from + to_port : local.nodeport_range.to + protocol : "udp" + self : false + cidr_blocks : local.nodeport_cidrs } ] egress_ipv4 = [ @@ -230,7 +266,7 @@ spec: installFlags: - "--san=${local.MKE_URL}" - "--default-node-orchestrator=kubernetes" - - "--nodeport-range=32768-35535" + - "--nodeport-range=${local.nodeport_range.from}-${local.nodeport_range.to}" upgradeFlags: - "--force-recent-backup" - "--force-minimums" diff --git a/examples/terraform/aws-simple/terraform.tfvars.template b/examples/terraform/aws-simple/terraform.tfvars.template index 70163550..3699d465 100644 --- a/examples/terraform/aws-simple/terraform.tfvars.template +++ b/examples/terraform/aws-simple/terraform.tfvars.template @@ -12,6 +12,19 @@ // // admin_cidrs = ["198.51.100.17/32", "203.0.113.0/24"] +// CIDRs allowed to reach the Kubernetes NodePort range (32768-35535, the +// range this example installs MKE with). +// +// Unset means "the same as admin_cidrs", so with both left alone the +// NodePort range is reachable from the whole internet -- the same real risk +// as above, and the exposure this example had before its ingress rules were +// scoped. Set this when the people who reach your applications are not the +// people who administer the cluster: widening admin_cidrs to admit +// application clients would also hand them SSH and the MKE and Kubernetes +// APIs. Uncomment and edit: +// +// nodeport_cidrs = ["203.0.113.0/24"] + aws = { region = "us-east-2" } diff --git a/examples/terraform/aws-simple/variables.tf b/examples/terraform/aws-simple/variables.tf index 4c86f486..f1c981eb 100644 --- a/examples/terraform/aws-simple/variables.tf +++ b/examples/terraform/aws-simple/variables.tf @@ -96,6 +96,39 @@ variable "admin_cidrs" { } } +variable "nodeport_cidrs" { + description = <<-EOT + CIDRs allowed to reach the Kubernetes NodePort range on the cluster + nodes. The range itself is local.nodeport_range in launchpad.tf, which + also builds the --nodeport-range install flag, so the rule and the flag + cannot drift apart. + + Unset (null) means "use admin_cidrs", which is the default and keeps an + existing tfvars working unchanged. Set it when the people who reach + your applications are not the people who administer the cluster: + admin_cidrs admits SSH, WinRM and the MKE and Kubernetes APIs, and + widening it to admit application clients would hand them those too. + + nodeport_cidrs = ["203.0.113.0/24"] + + Inheriting admin_cidrs inherits its exposure as well: with both left at + their defaults the NodePort range is reachable from the whole internet. + That is a real risk and not a recommendation -- it is the exposure this + example had before the ingress rules were scoped. + + An empty list is rejected rather than treated as "no rule", because a + cluster installed with --nodeport-range and no way in is the failure + this variable exists to prevent. Node-to-node traffic does not come + through here. + EOT + type = list(string) + default = null + validation { + condition = var.nodeport_cidrs == null || length(coalesce(var.nodeport_cidrs, [])) > 0 + error_message = "nodeport_cidrs must list at least one CIDR, or be left unset to inherit admin_cidrs." + } +} + variable "windows_password" { description = <<-EOT Password set on the local Administrator account of Windows nodes and used From d7342013b12cb06e30fa4569bc220e2df41c1c5c Mon Sep 17 00:00:00 2001 From: James Nesbitt Date: Fri, 6 Nov 2026 13:39:43 +0200 Subject: [PATCH 2/2] PRODENG-3929: note the ubuntu_22.04 host-firewall gap too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The NodePort host-firewall caveat singled out rhel_9 and sles_15. Both ubuntu_22.04 user_data entries in terraform.tfvars.template carry the identical omission: `sudo ufw allow 2377,7946,10250/tcp; sudo ufw allow 7946,4789/udp`, with no 32768-35535. Say so, and mark the ufw-active precondition as the inference it is — it cannot be tested without AWS. README only; no terraform changed. Written by AI: claude-opus-5 Signed-off-by: James Nesbitt --- examples/terraform/aws-simple/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/examples/terraform/aws-simple/README.md b/examples/terraform/aws-simple/README.md index 9c573e3e..4eb4b75c 100644 --- a/examples/terraform/aws-simple/README.md +++ b/examples/terraform/aws-simple/README.md @@ -53,6 +53,10 @@ On `rhel_9` and `sles_15` nodes the host firewall is a second gate. The but not the NodePort range, so `firewall-cmd` still blocks it there whatever the security group says. Add the range to that `user_data` if you need NodePort services on those platforms. +The `ubuntu_22.04` `user_data` omits the range in exactly the same way — it +opens 2377, 7946, 10250/tcp and 7946, 4789/udp through `ufw` and nothing else — +so the same gate applies on Ubuntu whenever `ufw` is active in that AMI, which +is unverified here because confirming it needs a running AWS cluster. #### Credentials