Skip to content

Implement endpoint steering for Kafka seed server traffic - #2018

Merged
andrewstucki merged 2 commits into
mainfrom
as/kafka-traffic-steering
Oct 27, 2026
Merged

andrewstucki merged 2 commits into
mainfrom
as/kafka-traffic-steering

Conversation

@andrewstucki

@andrewstucki andrewstucki commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

This adds opt-in traffic steering for brokers for the Kafka protocol port. It does so by taking any brokers that fail a simple TCP check across its damped window out of the backing endpoints for a steered service.

NOTE: This only affects initially bootstrapping clients connecting to the seed brokers for the first time. Pretty much all Kafka clients out their hold their own list of brokers internally after initially seeding. So, the only possible window this affects is newly connecting clients during a partial broker outage. It makes it so that newly connecting (read, not reconnecting) clients no longer run a small percentage risk of having to retry to connect to a seed broker, potentially delaying the connection startup for less smart clients (things like franz-go handle this with no real issues already) by somewhere from 10s to a minute depending on the client library.

For a system that proxies via cloud load balancers via externalTrafficPolicy: Local and requires this:

AWS/Azure: Requires adding per-target TCP health checks to exercise the kube-proxy path that the backing endpointslices manipulate. Without them NLBs and the like ignore the endpoint slices.
GCP: You can't do per-port TCP checks, so any ports targeted that need to be split out require their own separate LoadBalancer service, which can be backed by the same logical LB IP.

@secpanda

secpanda commented Oct 23, 2026 •

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The integration test’s Kafka listeners need to accept and close connections to prevent backlog exhaustion and false endpoint exclusions.

1 open issue
What changed in this PR

Extends opt-in endpoint steering to Kafka seed traffic, excluding brokers whose Kafka listeners fail damped TCP checks while leaving other ports independently controlled.

Changes:

  • Discovers internal and external Kafka listener ports for V1 and V2 clusters.
  • Adds per-listener TCP probes using existing damping mechanisms.
  • Expands unit, integration, and acceptance coverage and updates documentation.
File Description
operator/​pkg/​client/​cluster_brokers.go Exposes Kafka listener ports for both cluster versions.
operator/​pkg/​client/​cluster_brokers_test.go Tests listener port discovery and defaults.
operator/​internal/​controller/​endpointsteering/​resolver.go Adds Kafka ports to the cluster interface.
operator/​internal/​controller/​endpointsteering/​integration_test.go Tests Kafka endpoint removal and recovery.
operator/​internal/​controller/​endpointsteering/​endpointsteering.go Documents Kafka steering behavior.
operator/​internal/​controller/​endpointsteering/​checker.go Adds damped TCP checks for Kafka listeners.
operator/​internal/​controller/​endpointsteering/​checker_test.go Covers Kafka probing and shared damping behavior.
acceptance/​features/​endpoint-steering.feature Verifies non-serving pods are excluded from Kafka endpoints.
.changes/​unreleased/​operator-Added-20260915-120000.yaml Updates the release note to include Kafka steering.

Review effort: Balanced

Comment thread operator/internal/controller/endpointsteering/integration_test.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes consistently extend existing steering behavior, with coverage for listener discovery, exclusion, recovery, and damping, and no blocking issues identified.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@andrewstucki
andrewstucki merged commit 3a92fb7 into main Oct 27, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants