Skip to content

Derive the load balancer scheme from what the controller decides it with #247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stxkxs merged 1 commit into from
Sep 5, 2026
Merged

Derive the load balancer scheme from what the controller decides it with #247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/chart-provenance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,30 @@ jobs:
# moved upstream underneath it.
- name: Re-measure every pinned directory source
run: ./scripts/check-directory-manifest-size.py --live

scheme-inputs:
name: the controller still decides schemes the way the policy reads them
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false

- name: Install Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5

- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version-file: .python-version
- name: Install Python dependencies
run: pip install --require-hashes -r requirements.txt

# The blocking half compares the record against the chart pin, so it fails
# the moment somebody moves the controller. What it cannot see is the
# controller growing a new way to decide a scheme between bumps, or the
# chart changing which flags it renders from the same values. Both need the
# source and a render, and both are read here.
- name: Re-derive the scheme inputs from the pinned controller
run: ./scripts/check-lb-scheme-inputs.py --live
17 changes: 17 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,23 @@ jobs:
- name: Run policy unit tests
run: ./scripts/kyverno-test.sh policies/kyverno/tests

# The subnet-injection policy decides private-or-public from a scheme it
# believes a load balancer will have, and that belief is only as wide as the
# set of inputs it reads. The AWS Load Balancer Controller decides the same
# thing from four annotations, an IngressClassParams field, an Ingress
# group, and a load balancer that already exists in AWS. This derives that
# set from the controller source at the pinned chart version and fails when
# one of them is neither read by the policy nor recorded with the reason it
# is not — so a spelling nobody has heard of yet lands red rather than
# quietly taking the default. Offline: the derivation is keyed on the chart
# pin, so moving the pin fails here, and upstream/pins re-derives on a
# schedule.
- name: Every input the controller decides a scheme on is one the policy reads
run: |
pip install --require-hashes -r requirements.txt
./scripts/check-lb-scheme-inputs.py
./scripts/check-lb-scheme-inputs.py --self-test

# The unit tests above load BASE policies and pin verify-images' match and
# exclude scoping only. This structural gate guards the signing-IDENTITY
# contract they are blind to — required signature, GitHub OIDC issuer,
Expand Down
27 changes: 24 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ task validate:policy-admission # Prove no addon is denied by the Enforce-tie
task validate:externalsecret-keys # Each ExternalSecret names its remote secret once, and the delivering appset patches it
task validate:secret-store-refs # Every secret-store reference names the one store this catalog declares, and the published contract states it
task validate:directory-manifest-size # Every directory-source Application fits the repo-server's combined-manifest ceiling
task validate:lb-scheme-inputs # Every input the AWS LB Controller decides a scheme on is one the injection policy reads
task validate:dashboards # grafana.com dashboard ids exist and are AMG-saveable
task validate:athena-panel-columns # Every column a CUR panel names is one the export delivers
task validate:fork-safety # No hardcoded catalog repoURL in applied ApplicationSets (report-only locally)
Expand All @@ -114,7 +115,8 @@ task validate:image-vulnerabilities # Every fixed CRITICAL in a rendered image i

`task validate` runs the structural gates (lint, kustomize build, helm-render,
ApplicationSet schema, sync-wave ordering, appset render, policy-admission,
secret-store references, directory-source sizes, dashboards, fork-safety). CI runs those plus several gates that have **no local
secret-store references, directory-source sizes, load-balancer scheme inputs,
dashboards, fork-safety). CI runs those plus several gates that have **no local
`task` target**, and one that has a target but is deliberately outside the
aggregate, so a clean `task validate` is necessary but not sufficient:

Expand Down Expand Up @@ -179,6 +181,23 @@ aggregate, so a clean `task validate` is necessary but not sufficient:
delivered panel measures is a finding, because a figure compared to nothing is
how the last wrong one survived. The figure has no independent existence:
there is no constant to correct and none in the summary that anything trusts
- **Load-balancer scheme inputs** — `scripts/check-lb-scheme-inputs.py`, in the
`kyverno` job. `inject-adopt-lb-subnets` injects private or public subnet ids
according to the scheme it believes a load balancer will have, and reading one
annotation per object kind made that belief a pattern: a second spelling of the
same thing, `aws-load-balancer-internal`, is still honoured and still ahead of
the controller's default, so a Service setting only that one was internet-facing
to the controller and internal to the policy. Adding the second annotation fixes
the instance and not the next one, so the population is derived instead: the
gate reads `buildLoadBalancerScheme` (Service and Ingress) and
`IsServiceSupported` in the controller source at the version the chart pin
installs, follows the functions they call, and requires every symbol those
bodies consult to be recorded in `scripts/lb-scheme-inputs.json` as READ with a
string the policy must contain, UNREAD with the reason it cannot be consulted,
or PLUMBING with the reason it decides nothing. A symbol that is none of those
fails `--sync`. The record is keyed on the chart pin, so moving the controller
fails the blocking half; `--live` re-derives from source and re-renders the
chart on the schedule, because a controller can grow an input between bumps
- **Directory-source manifest sizes** — `scripts/check-directory-manifest-size.py`,
in the `appsets` job beside the offline chart-provenance half. The repo-server
refuses to generate a directory-type Application whose combined manifest files
Expand Down Expand Up @@ -232,8 +251,10 @@ documents: `task validate` runs it report-only, CI runs it `--blocking`.
`scripts/check-directory-manifest-size.py --live`, which re-measures each
pinned directory source: the blocking half already fails when a pin moves away
from its measurement, and what only a clone can answer is whether a tag nobody
moved here was moved upstream underneath it. Both need the network, which is
why neither is on the merge path
moved here was moved upstream underneath it. A third runs
`scripts/check-lb-scheme-inputs.py --live`, which re-derives the inputs the AWS
Load Balancer Controller decides a load balancer's scheme from. All three need
the network, which is why none is on the merge path
- Manual diff rendering available via `.github/workflows/diff.yml`

## Claude Code Tooling
Expand Down
9 changes: 8 additions & 1 deletion Taskfile.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,13 +206,19 @@ tasks:
- ./scripts/check-directory-manifest-size.py
- ./scripts/check-directory-manifest-size.py --self-test

validate:lb-scheme-inputs:
desc: "Load-balancer scheme gate — every input the controller decides a scheme on is one the injection policy reads"
cmds:
- ./scripts/check-lb-scheme-inputs.py
- ./scripts/check-lb-scheme-inputs.py --self-test

validate:athena-panel-columns:
desc: "Athena panel gate — every column a CUR panel names is one the export delivers"
cmds:
- ./scripts/check-athena-panel-columns.py

validate:
desc: Run all validations (lint, build, helm-render, appset schema, sync waves, label values, appset render, policy admission, secret refs, directory sizes, athena panels, dashboard/fork-safety)
desc: Run every gate this repo can run offline — lint, build, helm-render, schema, sync waves, label values, appset render, policy admission, and the reference/derivation gates
deps:
- lint:yaml
- lint:python
Expand All @@ -226,6 +232,7 @@ tasks:
- validate:externalsecret-keys
- validate:secret-store-refs
- validate:directory-manifest-size
- validate:lb-scheme-inputs
- validate:athena-panel-columns
- validate:dashboards
- validate:fork-safety
Expand Down
141 changes: 121 additions & 20 deletions policies/kyverno/networking/base/inject-adopt-lb-subnets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ metadata:
auto-discovery finds nothing. This policy supplies the subnets explicitly.
It reads the cluster-local network-config ConfigMap (written by
cluster-bootstrap in every mode) and, only when that record says the cluster
is in adopt mode, injects the subnet-id annotation the controller reads. It
is scheme-aware: an internal load balancer (the controller's default when no
scheme is set) lands on the private subnets, an internet-facing one on the
public subnets. The subnet annotation is added only if absent, so a tenant
that pins its own subnets is never overridden. A create-mode cluster owns its
VPC and its own ELB role tags, so the ConfigMap reports create, every rule's
precondition fails, and nothing is mutated.
is in adopt mode, injects the subnet-id annotation the controller reads. An
internal load balancer lands on the private subnets, an internet-facing one
on the public subnets, and the scheme is resolved the way the controller
resolves it rather than from a single annotation. The subnet annotation is
added only if absent, so a tenant that pins its own subnets is never
overridden. A create-mode cluster owns its VPC and its own ELB role tags, so
the ConfigMap reports create, every rule's precondition fails, and nothing
is mutated.
spec:
admission: true
emitWarning: false
Expand All @@ -41,7 +42,7 @@ spec:
webhookConfiguration:
failurePolicy: Ignore
rules:
# ── Ingress, internal scheme (the default) → private subnets ──────────────
# ── Ingress, internal scheme → private subnets ────────────────────────────
- name: ingress-internal-subnets
skipBackgroundRequests: true
match:
Expand All @@ -54,17 +55,36 @@ spec:
configMap:
name: network-config
namespace: kube-system
# The object's own scheme, defaulted to internal to match the AWS Load
# Balancer Controller's default when the annotation is absent.
- name: scheme
# An Ingress that names a group shares one load balancer, and its scheme
# -- one scheme for the whole group -- is decided across every member. No
# single object carries the answer, and a member that disagrees with the
# group is rejected by the controller rather than given its own subnets.
# Injecting from this object alone would be a guess about the others.
- name: groupname
variable:
jmesPath: 'request.object.metadata.annotations."alb.ingress.kubernetes.io/group.name"'
default: ""
- name: schemeannotation
variable:
jmesPath: 'request.object.metadata.annotations."alb.ingress.kubernetes.io/scheme"'
default: internal
default: ""
# The controller's order for a single-member group: an IngressClassParams
# scheme, else this annotation, else its --default-load-balancer-scheme.
# IngressClassParams is not read here and that is a decision, not an
# oversight -- scripts/check-lb-scheme-inputs.py records which inputs are
# read and which are knowingly not, and fails when the controller grows
# one that is neither.
- name: scheme
variable:
value: "{{ schemeannotation || 'internal' }}"
preconditions:
all:
- key: "{{ networkconfig.data.network_mode }}"
operator: Equals
value: adopt
- key: "{{ groupname }}"
operator: Equals
value: ""
- key: "{{ scheme }}"
operator: Equals
value: internal
Expand All @@ -90,15 +110,25 @@ spec:
configMap:
name: network-config
namespace: kube-system
- name: scheme
- name: groupname
variable:
jmesPath: 'request.object.metadata.annotations."alb.ingress.kubernetes.io/group.name"'
default: ""
- name: schemeannotation
variable:
jmesPath: 'request.object.metadata.annotations."alb.ingress.kubernetes.io/scheme"'
default: internal
default: ""
- name: scheme
variable:
value: "{{ schemeannotation || 'internal' }}"
preconditions:
all:
- key: "{{ networkconfig.data.network_mode }}"
operator: Equals
value: adopt
- key: "{{ groupname }}"
operator: Equals
value: ""
- key: "{{ scheme }}"
operator: Equals
value: internet-facing
Expand All @@ -110,7 +140,7 @@ spec:
metadata:
annotations:
+(alb.ingress.kubernetes.io/subnets): "{{ networkconfig.data.public_subnet_ids }}"
# ── Service type=LoadBalancer, internal scheme (the default) → private ────
# ── Service the controller provisions, internal scheme → private subnets ──
- name: service-internal-subnets
skipBackgroundRequests: true
match:
Expand All @@ -123,15 +153,61 @@ spec:
configMap:
name: network-config
namespace: kube-system
- name: scheme
# WHICH SERVICES THE CONTROLLER PROVISIONS. A type=LoadBalancer Service is
# not automatically the AWS Load Balancer Controller's. It claims one whose
# spec.loadBalancerClass is its own class, or that carries
# aws-load-balancer-type: nlb-ip, or aws-load-balancer-type: external with
# an nlb-target-type. Every other type=LoadBalancer Service goes to the
# in-tree cloud provider, which builds a Classic Load Balancer and reads
# neither the scheme annotation nor the subnets annotation this policy
# writes. The chart sets enableServiceMutatorWebhook: false, so nothing
# stamps loadBalancerClass on a Service that does not ask for it, and the
# plain shape stays the in-tree provider's.
- name: lbclass
variable:
jmesPath: 'request.object.spec.loadBalancerClass'
default: ""
- name: lbtype
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-type"'
default: ""
- name: nlbtargettype
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-nlb-target-type"'
default: ""
- name: controllermanaged
variable:
value: "{{ lbclass == 'service.k8s.aws/nlb' || lbtype == 'nlb-ip' || (lbtype == 'external' && (nlbtargettype == 'ip' || nlbtargettype == 'instance')) }}"
- name: schemeannotation
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-scheme"'
default: internal
default: ""
# The legacy spelling, still honoured and still ahead of the default.
# aws-load-balancer-internal: "false" means internet-facing; a Service
# setting it without the newer scheme annotation is internet-facing to the
# controller and was internal to this policy, which is the private-subnet
# CSV on a load balancer the controller wants on public ones.
- name: internalannotation
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-internal"'
default: ""
# The controller's order: the scheme annotation, else the legacy internal
# flag, else the scheme of a load balancer that already exists, else its
# --default-load-balancer-scheme. The existing load balancer is not
# readable from an admission request at all; the default is asserted
# against the rendered chart by scripts/check-lb-scheme-inputs.py rather
# than assumed here.
- name: scheme
variable:
value: "{{ schemeannotation || (internalannotation == 'false' && 'internet-facing' || 'internal') }}"
preconditions:
all:
- key: "{{ request.object.spec.type }}"
operator: Equals
value: LoadBalancer
- key: "{{ controllermanaged }}"
operator: Equals
value: true
- key: "{{ networkconfig.data.network_mode }}"
operator: Equals
value: adopt
Expand All @@ -146,7 +222,7 @@ spec:
metadata:
annotations:
+(service.beta.kubernetes.io/aws-load-balancer-subnets): "{{ networkconfig.data.private_subnet_ids }}"
# ── Service type=LoadBalancer, internet-facing scheme → public subnets ────
# ── Service the controller provisions, internet-facing → public subnets ───
- name: service-internet-facing-subnets
skipBackgroundRequests: true
match:
Expand All @@ -159,15 +235,40 @@ spec:
configMap:
name: network-config
namespace: kube-system
- name: scheme
- name: lbclass
variable:
jmesPath: 'request.object.spec.loadBalancerClass'
default: ""
- name: lbtype
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-type"'
default: ""
- name: nlbtargettype
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-nlb-target-type"'
default: ""
- name: controllermanaged
variable:
value: "{{ lbclass == 'service.k8s.aws/nlb' || lbtype == 'nlb-ip' || (lbtype == 'external' && (nlbtargettype == 'ip' || nlbtargettype == 'instance')) }}"
- name: schemeannotation
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-scheme"'
default: internal
default: ""
- name: internalannotation
variable:
jmesPath: 'request.object.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-internal"'
default: ""
- name: scheme
variable:
value: "{{ schemeannotation || (internalannotation == 'false' && 'internet-facing' || 'internal') }}"
preconditions:
all:
- key: "{{ request.object.spec.type }}"
operator: Equals
value: LoadBalancer
- key: "{{ controllermanaged }}"
operator: Equals
value: true
- key: "{{ networkconfig.data.network_mode }}"
operator: Equals
value: adopt
Expand Down
Loading