[Bug]: collector.ignore_path_prefixes cannot suppress /_gateway-health probes

[Tharsanan1]

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/Logging (Log formats, instrumentation, improvements), Aspect/Configuration (Config files, settings, env vars, defaults)

Description

Now that the gateway health endpoints are served on the same port as API traffic, every readiness/liveness probe appears in the traffic log and the access log. Adding the reserved prefix to collector.ignore_path_prefixes does not suppress them:

[collector]
ignore_path_prefixes = ["/_gateway-health"]

Both probes are still logged:

[pol] {"timestamp":"2026-08-11T09:56:50.172Z","correlationId":"50837bc9-...","status":200,"operation":{"method":"GET"},"target":{"statusCode":200,"destination":"localhost:8080/_gateway-health/ready"},"client":{"ip":"192.168.128.1","userAgent":"curl/8.4.0"}}
[pol] {"timestamp":"2026-08-11T09:56:50.186Z","correlationId":"16aced0b-...","status":200,"operation":{"method":"GET"},"target":{"statusCode":200,"destination":"localhost:8080/_gateway-health/healthy"},"client":{"ip":"192.168.128.1","userAgent":"curl/8.4.0"}}

Root cause — the config is honoured; the filter structurally cannot match these routes. The prefix list is compiled into an Envoy AccessLogFilter attached to the ALS sink, and it is present in the live config dump exactly as configured:

{"or_filter": {"filters": [
  {"header_filter": {"header": {"name": "x-envoy-original-path", "present_match": false}}},
  {"header_filter": {"header": {"name": "x-envoy-original-path",
                                "invert_match": true, "prefix_match": "/_gateway-health"}}}]}}

notEffectivelyMatchesPrefix (gateway-controller/pkg/xds/translator.go) matches on x-envoy-original-path, and its first OR branch is present_match: false — i.e. "if the header is absent, log the entry." That is a deliberate choice, documented in the code: without the original-path header it cannot distinguish a route that genuinely never rewrites from an upstream path that merely shares the prefix, so it errs toward logging.

But Envoy only sets x-envoy-original-path when a route rewrites the path, and the gateway health endpoints are DirectResponse routes (buildGatewayHealthRoutes) matching the exact paths /_gateway-health/ready and /_gateway-health/healthy with no upstream and no rewrite. The header therefore never exists on these requests, the OR short-circuits to true, and the entry is always logged. No value of ignore_path_prefixes can ever suppress them.

The same blind spot applies to the other direct-response route, no-api-found — a 404 for an unmatched path is logged for the same reason.

Additional scope

Two related parts of the same complaint are worth tracking, as a filter on the ALS sink alone does not cover them:

1. The stdout access log still logs the probes. ignore_path_prefixes is wired only to the gRPC ALS sink; the file sink is created with no filter at all ("filter": null in the config dump).
2. The probes are still sent to Datadog. Datadog is fed by Envoy's OpenTelemetry tracer configured on the HCM (createTracingConfig) with only RandomSampling and no path-based exclusion, so health probes generate spans regardless of any access-log filtering.

Steps to Reproduce

1. Configure traffic logging and the ignore prefix in gateway/configs/config.toml:

   [traffic_logging]
   enabled = true
   
   [collector]
   ignore_path_prefixes = ["/_gateway-health"]
   
   [router.access_logs]
   enabled = true

2. Start the gateway.

3. Hit the health endpoints:

   curl -s -o /dev/null http://localhost:8080/_gateway-health/ready
   curl -s -o /dev/null http://localhost:8080/_gateway-health/healthy

4. Inspect the runtime log:

   docker compose logs --tail 100 gateway-runtime | grep correlationId | grep _gateway-health

   Both probes appear, despite the configured prefix.

Proposed fix

A fix for the ALS/collector part is proposed in #3192, which attaches a separate filter matching the :path pseudo-header against the reserved health prefix and AND-s it with the user's prefix list. Matching :path is safe for these routes precisely because they never rewrite the path. Note that it suppresses the reserved health namespace unconditionally rather than making ignore_path_prefixes honour it, and it does not address the two additional scope items above.

Severity Level of the Issue

Severity/Major (Important functionality is broken. Should be prioritized. Doesn't need immediate attention)

Environment Details (with versions)

API Platform 1.2.0 (reproduced on the 1.2.0 gateway images built from source)