feat: add Ingress Availability and Latency burn-rate alerts (ARO-25969) by ArrisLee · Pull Request #7097 · Azure/ARO-HCP

Arris Li (ArrisLee) · 2026-09-23T00:52:01Z

https://redhat.atlassian.net/browse/ARO-25969

What

Adds multi-window multi-burn-rate (MWMBR) alerts for the Ingress customer user journey, covering both baseline SLIs established in the SLI/SLO PR #6516 (merged):

Availability (arohcp_ingress_availability_slo_alerts):

userJourneyIngressAvailabilityErrors1h5m — Fast burn (14.4x, >7.2%, Sev 4)
userJourneyIngressAvailabilityErrors6h30m — Medium burn (6x, >3.0%, Sev 4)
userJourneyIngressAvailabilityErrors3d6h — Slow burn (1x, >0.5%, Sev 4)

Latency (arohcp_ingress_latency_slo_alerts):

userJourneyIngressLatencyP99Errors1h5m / 6h30m / 3d6h — same burn-rate tiers (P99 since the 99.5% SLO target is closer to a P99 objective than P95), all Sev 4 (see follow-ups below for why the fast/medium tiers aren't Sev 3)

Canary liveness (arohcp_ingress_canary_liveness_alerts):

userJourneyIngressCanaryAbsent — fires when an HCP's canary series stops reporting entirely, a case none of the burn-rate tiers above can see (a missing series doesn't register as a high error rate, it registers as no data)

Each burn-rate tier is a true MWMBR pairing: it ANDs an independently-computed short-window condition (5m/30m/6h) with the long-window one (1h/6h/3d), each with its own volume floor on the window's total observation/check count — not an instantaneous ratio gated only by for:. The short window confirms the burn is still active right now (so a recovered incident stops paging); the long window confirms it has consumed enough of the error budget to matter. This mirrors UserJourneyKubeApiserverAvailabilityMonitor-prometheusRule-KSM.yaml. 6h is reused as both the medium tier's long window and the slow tier's short window.

All alerts are per-HCP, link to aka.ms/arohcp-runbook-ingress, and are registered in the RP per-HCP lane (alerts-rp-hcps.yaml → generatedRPHCPPrometheusAlertingRules.bicep → icm-action-group-rp). Summary annotations omit {{ $labels.cluster }} (the RP bicep template prepends it). correlationId includes the alert family (e.g. userJourneyIngressAvailabilityErrors/{{ $labels._id }}) so the three burn tiers within a family correlate into one IcM incident per HCP, while Availability, Latency, and the absence alert stay distinct from each other.

Why

ARO-25969 requires MWMBR alerting for each applicable baseline SLI of the Ingress user journey, tied to the SLIs/SLOs defined in #6516 and following the ARO HCP Alerting Recommendation ADR. Without these alerts the ingress canary SLI series exist but page no one when the ingress data path degrades, slows, or goes dark end to end.

Testing

Promtool unit tests (ingress-availability-slo-prometheusRule_test.yaml), 9 cases:

Availability: all reachable (no alerts); 50% failing across 10 routes (fast + medium burn); ~5.9% failing across 17 routes (slow only, fast suppressed); multi-cluster isolation; volume floor — 100% failing but only 2 total observations, below the floor, alert suppressed until real history accumulates.
Latency: all checks within 200ms (no alerts); 50% slow (fast + medium burn); ~5.9% slow (slow only, fast suppressed).
Canary absence: a series that goes silent partway through fires userJourneyIngressCanaryAbsent; a continuously-reporting series never does.

Recording-rule tests (ingress-availability-slo-recordingRule_test.yaml) additionally verify the windowed records' arithmetic directly.

make -C observability alerts / recording-rules regenerates the bicep; the committed bicep is az bicep format-ed.

Review history

Simon Wiencki (@swiencki) (first round) flagged that the original design gated the instantaneous error rate with only for:, so the named windows were never actually computed. Fixed by adding windowed recording rules + volume floors, per the discussion on the diff.
Tejas Kesharwani (@tkesharw-RH) and Simon Wiencki (@swiencki) (second round) flagged that even after that fix, each tier only evaluated its long window — the 5m/30m/6h values were for: persistence durations, not independently-calculated short-window burn rates, so a recovered incident could keep paging on stale long-window data. Fixed by pairing a real short-window condition with the long-window one in every tier (true MWMBR), per UserJourneyKubeApiserverAvailabilityMonitor-prometheusRule-KSM.yaml.
Simon Wiencki (@swiencki) also flagged: alert names not matching the (unpublished) ingress-tsg.md runbook routing, correlationId collapsing all 6 tiers into one IcM per HCP, a missing canary-absence alert, and bare Latency naming instead of a percentile. All fixed — see inline replies for details.
Rebased onto main twice as it advanced; the recording-rule files duplicated into this branch pre-observability: add Ingress Availability and Latency SLI/SLO definitions (ARO-22170) #6516-merge have been dropped, and a stale etcd-alert registration line (added upstream then reverted) picked up in an earlier rebase was removed.

Follow-ups (outside this PR)

Create the aka.ms/arohcp-runbook-ingress short-link and publish the Ingress TSG/runbook (needs a lead per the ADR); alert names here are my best guess at what ingress-tsg.md expects per Simon Wiencki (@swiencki)'s comments and may need adjustment once it's published.
Validate the windowed expressions and volume floors (currently 3/15/10/30/60 across the 5m/30m/1h/6h/3d windows) against real canary cardinality using alert-tester — flagged by Simon Wiencki (@swiencki).
No Grafana dashboard exists yet for this SLO; tracked separately in ARO-22171 (dashboards for all user journeys, not just ingress).
These alerts don't exclude internal/test subscriptions from their SLO error budgets — internalSubscriptionFilter is only wired up for the services lane today, and neither this group nor the pre-existing KAS alert group in alerts-rp-hcps.yaml opts in. Tracked in ARO-30437 as a follow-up covering both.
The fast/medium availability and latency tiers are Sev 4, not Sev 3, as an interim measure: the canary's router pods and load balancer run on the customer's own workers and network, so a customer scaling to zero workers or changing their own NSG rules fails the canary the same way a platform-side problem would, with no way today to tell them apart — flagged by Simon Wiencki (@swiencki). Tracked in ARO-30462 to restore Sev 3 for genuinely platform-caused failures once that distinction exists.

🤖 Generated with Claude Code

openshift-merge-bot · 2026-09-23T00:52:03Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

Copilot

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

openshift-ci · 2026-09-23T00:52:14Z

Hi Arris Li (@ArrisLee). Thanks for your PR.

I'm waiting for a Azure member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@swiencki

… alerts Addresses review feedback from @swiencki on PR Azure#7097: the burn-rate tiers were gating the instantaneous errors:ingress_canary:error_rate with only `for:`, so the named 1h/6h/3d windows were never actually computed, and a single failed observation among few redundant canary routes (e.g. 1 of 2, a 50% instantaneous ratio) could already exceed every threshold outright. Adds windowed recording rules (errors:ingress_canary(_latency):{failed, total,error_rate}_{1h,6h,3d}) that accumulate real observation/check counts over each window via sum_over_time/count_over_time (availability, a gauge) and increase() (latency, a cumulative histogram), following the pattern already established in cluster-provision-slo-prometheusRule.yaml. Each alert tier now requires both a volume floor on the window's total observation/check count and the windowed error rate threshold, so a sparse or freshly-started window cannot page. Also rebases onto main now that PR Azure#6516 (SLI/SLO foundation) has merged, dropping the recording-rule files that were duplicated into this branch only so promtool tests could run standalone before that PR landed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

Tejas Kesharwani (tkesharw-RH) · 2026-09-28T01:04:55Z

+        errors:ingress_canary:total_1h >= 10
+        and
+        errors:ingress_canary:error_rate_1h > 0.072
+      for: 5m


The PR describes these as MWMBR alerts, but each expression evaluates only its long window (error_rate_1h or error_rate_6h). The 5m/30m values are persistence durations via for:, not independently calculated short-window burn rates. Would it be worth using paired conditions such as error_rate_1h > 0.072 AND error_rate_5m > 0.072 and error_rate_6h > 0.03 AND error_rate_30m > 0.03? That would ensure the long-window budget impact is significant and the burn is still active, preventing a recovered incident from paging while old failures remain in the long window. If the current single-window-with-persistence behavior is intentional, could the description and short_window labels clarify that distinction?

Correct, and thanks for catching this -- fixed in dda099f. Each tier now ANDs an independently-computed short-window condition with the long-window one, e.g. for the fast tier:

(errors:ingress_canary:total_5m >= 3 and errors:ingress_canary:error_rate_5m > 0.072) and (errors:ingress_canary:total_1h >= 10 and errors:ingress_canary:error_rate_1h > 0.072)

Added 5m/30m windowed recording rules alongside the existing 1h/6h/3d ones (6h is reused as both the medium tier's long window and the slow tier's short window, mirroring how the KAS journey alert reuses sli_count_6h). for: is now a short debounce (10m/30m/3h) rather than doing the window's job. This exactly fixes the "recovered incident keeps paging" case you described -- once the 5m window clears, the AND fails and the alert resolves, even if stale failures are still inside the 1h window.

Simon Wiencki (swiencki) · 2026-09-28T22:21:03Z

Do you have the results of the tester tool? more details in the ARO HCP repo and slack regarding the tool

@swiencki

… alerts Addresses review feedback from @swiencki on PR Azure#7097: the burn-rate tiers were gating the instantaneous errors:ingress_canary:error_rate with only `for:`, so the named 1h/6h/3d windows were never actually computed, and a single failed observation among few redundant canary routes (e.g. 1 of 2, a 50% instantaneous ratio) could already exceed every threshold outright. Adds windowed recording rules (errors:ingress_canary(_latency):{failed, total,error_rate}_{1h,6h,3d}) that accumulate real observation/check counts over each window via sum_over_time/count_over_time (availability, a gauge) and increase() (latency, a cumulative histogram), following the pattern already established in cluster-provision-slo-prometheusRule.yaml. Each alert tier now requires both a volume floor on the window's total observation/check count and the windowed error rate threshold, so a sparse or freshly-started window cannot page. Also rebases onto main now that PR Azure#6516 (SLI/SLO foundation) has merged, dropping the recording-rule files that were duplicated into this branch only so promtool tests could run standalone before that PR landed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@tkesharw-RH

…review Addresses review feedback from @tkesharw-RH and @swiencki on PR Azure#7097: - Each burn-rate tier now ANDs an independently-computed short-window condition with the long-window one (e.g. error_rate_5m > threshold AND error_rate_1h > threshold), each with its own volume floor, instead of gating only the long window with a `for:` duration. This matches UserJourneyKubeApiserverAvailabilityMonitor-prometheusRule-KSM.yaml: the short window confirms the burn is still active (so a recovered incident stops paging), the long window confirms it has consumed enough of the error budget to matter. Adds 5m/30m windowed recording rules alongside the existing 1h/6h/3d ones (6h is reused as both the medium tier's long window and the slow tier's short window). - Renames alerts to match the (unpublished) ingress-tsg.md runbook's expected routing: userJourneyIngressAvailabilityErrors{1h5m,6h30m,3d6h} and userJourneyIngressLatencyP99Errors{1h5m,6h30m,3d6h} (P99 since the 99.5% SLO target is closer to a P99 latency objective than P95). Tier suffixes now include the real short window (3d6h, not 3d). - correlationId now includes the alert family (userJourneyIngress{Availability,LatencyP99}Errors/{{ $labels._id }}) so the three burn tiers within a family still correlate into one IcM incident per HCP, while Availability and Latency (different SLIs, different problems) no longer collapse into the same incident. - Adds userJourneyIngressCanaryAbsent: none of the burn-rate tiers can see an HCP whose canary series disappears entirely (a missing series doesn't register as a high error rate, it registers as no data at all). Compares a 24h presence window against a 10m one so the alert stays firing for the whole outage instead of self-clearing once a narrower historical reference window ages out. All promtool tests pass; bicep regenerated and az bicep format-ed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

Arris Li (ArrisLee) · 2026-09-30T03:22:11Z

Do you have the results of the tester tool? more details in the ARO HCP repo and slack regarding the tool

will have a try with https://github.com/mmazur/alert-tester

Simon Wiencki (swiencki) · 2026-10-02T19:36:12Z

/ok-to-test

@tkesharw-RH

…review Addresses review feedback from @tkesharw-RH and @swiencki on PR Azure#7097: - Each burn-rate tier now ANDs an independently-computed short-window condition with the long-window one (e.g. error_rate_5m > threshold AND error_rate_1h > threshold), each with its own volume floor, instead of gating only the long window with a `for:` duration. This matches UserJourneyKubeApiserverAvailabilityMonitor-prometheusRule-KSM.yaml: the short window confirms the burn is still active (so a recovered incident stops paging), the long window confirms it has consumed enough of the error budget to matter. Adds 5m/30m windowed recording rules alongside the existing 1h/6h/3d ones (6h is reused as both the medium tier's long window and the slow tier's short window). - Renames alerts to match the (unpublished) ingress-tsg.md runbook's expected routing: userJourneyIngressAvailabilityErrors{1h5m,6h30m,3d6h} and userJourneyIngressLatencyP99Errors{1h5m,6h30m,3d6h} (P99 since the 99.5% SLO target is closer to a P99 latency objective than P95). Tier suffixes now include the real short window (3d6h, not 3d). - correlationId now includes the alert family (userJourneyIngress{Availability,LatencyP99}Errors/{{ $labels._id }}) so the three burn tiers within a family still correlate into one IcM incident per HCP, while Availability and Latency (different SLIs, different problems) no longer collapse into the same incident. - Adds userJourneyIngressCanaryAbsent: none of the burn-rate tiers can see an HCP whose canary series disappears entirely (a missing series doesn't register as a high error rate, it registers as no data at all). Compares a 24h presence window against a 10m one so the alert stays firing for the whole outage instead of self-clearing once a narrower historical reference window ages out. All promtool tests pass; bicep regenerated and az bicep format-ed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot

🔵 Needs a closer look

A critical HA deduplication issue in the recording rules requires correction and regression coverage.

3 open issues

_{Review effort: Lite}

+        sum by (_id, cluster, region) (max without (prometheus_replica) (sum_over_time(ingress_canary_route_reachable[5m])))
+    - record: errors:ingress_canary:total_5m
+      expr: |
+        sum by (_id, cluster, region) (max without (prometheus_replica) (count_over_time(ingress_canary_route_reachable[5m])))


openshift-merge-bot · 2026-10-20T21:55:15Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

Arris Li (ArrisLee) · 2026-10-21T05:01:00Z

/test e2e-parallel

Simon Wiencki (swiencki) · 2026-10-21T17:10:01Z

+    # ----------------------------------------
+    # Medium Burn Alert (IcM Sev 3)
+    # ----------------------------------------
+    - alert: userJourneyIngressAvailabilityErrors6h30m


Any luck running these through alert-tester, and if prod access is the blocker can we find someone to run it so we can see how often each tier would have fired?

Arris Li (ArrisLee) · 2026-10-22T04:24:51Z

/test e2e-parallel

Copilot

🔵 Needs a closer look

A critical recording-rule issue still allows stale HA replica data to suppress outage errors and burn alerts, and the required fix, regeneration, and regression test are not yet complete.

3 open issues

_{Review effort: Lite}

…d alerts (ARO-25969) Adds SLI recording rules and burn-rate alerts for the Ingress customer user journey, covering both Availability and Latency dimensions. Recording rules (`ingress-availability-slo-recordingRule.yaml`): - Availability SLI: fraction of synthetic canary route observations that succeed, aggregated per HCP cluster via `sum/count by (_id, cluster, region)`. - Latency SLI: fraction of canary checks completing within 200ms, from the `ingress_canary_check_duration` histogram. - Derived `errors:*:error_rate` rules for burn-rate alert expressions. - SLO: 99.5% for both (error budget 0.5%). Alert rules (`ingress-availability-slo-prometheusRule.yaml`): - Fast burn (14.4x): >7.2% error rate for 5m → Sev 3 (would exhaust budget in ~50h) - Medium burn (6x): >3.0% error rate for 30m → Sev 3 (budget exhausted in ~5d) - Slow burn (1x): >0.5% error rate for 6h → Sev 4 (trend alert) All registered in `alerts-rp-hcps.yaml` → `generatedRPHCPPrometheusAlertingRules.bicep` (RP HCP on-call queue). Promtool unit tests verify all burn-rate tiers and cluster isolation. Note: The underlying SLI represents end-to-end synthetic route checks (DNS / load balancer / ingress routing / router pods), not pod availability directly. Annotations clarify this semantics per ARO-22170 review feedback. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Adds multi-window multi-burn-rate alerts for the ingress canary latency SLI, consuming errors:ingress_canary_latency:error_rate established in the SLI/SLO PR (Azure#6516). Mirrors the existing availability burn-rate tiers (Fast 14.4x / Medium 6x / Slow 1x) against the 99.5% SLO, completing the Availability + Latency alerting coverage required by ARO-25969. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@swiencki

… alerts Addresses review feedback from @swiencki on PR Azure#7097: the burn-rate tiers were gating the instantaneous errors:ingress_canary:error_rate with only `for:`, so the named 1h/6h/3d windows were never actually computed, and a single failed observation among few redundant canary routes (e.g. 1 of 2, a 50% instantaneous ratio) could already exceed every threshold outright. Adds windowed recording rules (errors:ingress_canary(_latency):{failed, total,error_rate}_{1h,6h,3d}) that accumulate real observation/check counts over each window via sum_over_time/count_over_time (availability, a gauge) and increase() (latency, a cumulative histogram), following the pattern already established in cluster-provision-slo-prometheusRule.yaml. Each alert tier now requires both a volume floor on the window's total observation/check count and the windowed error rate threshold, so a sparse or freshly-started window cannot page. Also rebases onto main now that PR Azure#6516 (SLI/SLO foundation) has merged, dropping the recording-rule files that were duplicated into this branch only so promtool tests could run standalone before that PR landed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@tkesharw-RH

…review Addresses review feedback from @tkesharw-RH and @swiencki on PR Azure#7097: - Each burn-rate tier now ANDs an independently-computed short-window condition with the long-window one (e.g. error_rate_5m > threshold AND error_rate_1h > threshold), each with its own volume floor, instead of gating only the long window with a `for:` duration. This matches UserJourneyKubeApiserverAvailabilityMonitor-prometheusRule-KSM.yaml: the short window confirms the burn is still active (so a recovered incident stops paging), the long window confirms it has consumed enough of the error budget to matter. Adds 5m/30m windowed recording rules alongside the existing 1h/6h/3d ones (6h is reused as both the medium tier's long window and the slow tier's short window). - Renames alerts to match the (unpublished) ingress-tsg.md runbook's expected routing: userJourneyIngressAvailabilityErrors{1h5m,6h30m,3d6h} and userJourneyIngressLatencyP99Errors{1h5m,6h30m,3d6h} (P99 since the 99.5% SLO target is closer to a P99 latency objective than P95). Tier suffixes now include the real short window (3d6h, not 3d). - correlationId now includes the alert family (userJourneyIngress{Availability,LatencyP99}Errors/{{ $labels._id }}) so the three burn tiers within a family still correlate into one IcM incident per HCP, while Availability and Latency (different SLIs, different problems) no longer collapse into the same incident. - Adds userJourneyIngressCanaryAbsent: none of the burn-rate tiers can see an HCP whose canary series disappears entirely (a missing series doesn't register as a high error rate, it registers as no data at all). Compares a 24h presence window against a 10m one so the alert stays firing for the whole outage instead of self-clearing once a narrower historical reference window ages out. All promtool tests pass; bicep regenerated and az bicep format-ed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Simon and Copilot both flagged two correctness issues in the ingress windowed recording rules and the liveness alert: - The availability and latency windowed recording rules summed sum_over_time/count_over_time/increase() directly over the raw canary metrics without collapsing the prometheus_replica dimension first. The managed Prometheus HA replicas both scrape the same canary targets, so every window's observation count (and therefore its volume floor) was silently doubled. Wrapped each window in max without (prometheus_replica), matching the existing frontend/nodepool/access-cluster SLO rules. - userJourneyIngressCanaryAbsent compared a 24h reference window against a 10m window to detect a silent canary. Once an outage ran longer than 24h, the reference window itself emptied out and the alert silently resolved even though the HCP never recovered. Widened the reference window to 7d to substantially shrink that blind spot; a metric-only detector can't eliminate it outright without an independent HCP-existence signal, which is out of scope here. Added regression tests for both (HA replica duplicates must not double windowed counts; the absence alert must still be firing past the old 24h mark) and verified each fails without its corresponding fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Simon pointed out there's no Grafana dashboard for this SLO, so on-call has nothing to open from these alerts. Dashboard work for all user journeys (not just ingress) is tracked separately in ARO-22171 and is intentionally out of scope here -- documenting that inline so it doesn't read as an oversight. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…I join Simon raised two gaps: the canary alerts carry no resource_id for on-call triage, and the absence alert keeps firing for up to 7 days after an HCP is actually deleted (not just silent). Both share a root cause: ingress_canary_route_reachable never carries resource_id (confirmed against real AMW data), so there's no way to attach it without joining against something that does. All 7 ingress alerts (3 availability + 3 latency burn-rate tiers, plus the absence alert) now multiply their condition by `hostedClusterAPI_kubeapiserver_available` joined on _id via group_left, which both injects resource_id and drops any _id with no corresponding HostedCluster record. This is the same metric UserJourneyKubeApiserverAvailabilityMonitor-prometheusRule-KSM.yaml already depends on for its own (higher-severity) alerting, so it isn't a new class of dependency for this alert group. For the absence alert specifically, this closes the common case of the 7-day window problem: a genuinely deleted HCP drops out of the join immediately rather than continuing to fire until its last known-good canary sample ages out of the 7d reference window. It does not close the residual case of an HCP that still exists but is silent for longer than 7d straight -- documented inline as a known limit of a time-window-based reference. Added a hostedClusterAPI_kubeapiserver_available series to every existing test case (so each keeps testing its own burn-rate/volume-floor behavior independently of the new join) and a new test proving the absence alert does not fire once the HostedCluster record is gone, even well past the `for: 10m` debounce. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Every burn-rate fixture in this file was either continuously failing or continuously healthy, so a regression that collapsed an alert's AND down to just its long-window leg would still pass every existing test (Copilot review feedback). Added a case where the short window (5m) recovers while the long window (1h) is still mostly failed, asserting the alert fires before recovery and stops firing once the short window clears, even though the long window alone would still justify it. Verified it fails if the short-window condition is removed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot kept re-flagging the replica-dedup thread as unresolved across three consecutive reviews even after the windowed recording rules were fixed. Traced it to the pre-existing instantaneous availability:ingress_canary:ratio and latency:ingress_canary:ratio rules (from Azure#6516, unmodified by this PR until now) -- same gap, just without a volume floor to visibly distort. With equal replica counts per instance this ratio happens to come out right regardless (both sum and count double by the same factor and it cancels), which is probably why it went unnoticed. It breaks once replica coverage is uneven across instances -- e.g. one replica misses a scrape for a single degraded canary route while continuing to double-count a healthy one -- skewing the ratio toward "healthy". Wrapped both rules in max without (prometheus_replica), matching the windowed group, and added a test for each reproducing that uneven-coverage scenario; confirmed both fail with the expected skewed value (0.667 instead of 0.5) without the fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ation Simon pointed out the canary's router pods and load balancer run on the customer's own workers and network, so a customer scaling their worker pool to zero or changing their own NSG rules fails the canary the same way a genuine platform-side ingress-operator or routing-stack problem would -- and there's no way today to tell them apart. Between gating on ready-worker-count (only covers the scale-to-zero case, not the NSG case, and needs an unvalidated new metric) and a uniform severity downgrade, went with the latter: all four fast/medium tiers (availability + latency) drop from Sev 3 to Sev 4 so customer-caused failures stop paging RP on-call at that severity. The slow tier was already Sev 4. This also slows response to genuinely platform-caused failures until the two can be told apart -- tracked in ARO-30462 to restore Sev 3 once that's possible. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Rebasing picked up upstream's new hypershiftUserJourney-prometheusRule.yaml registration in alerts-rp-hcps.yaml, which conflicted with this branch's own addition of ingress-availability-slo-prometheusRule.yaml to the same list; resolved by keeping both entries. The generated bicep and corpus fixture conflicts during the rebase were resolved with --ours as a placeholder (per the usual process for these generated artifacts), then fully regenerated here so the hypershiftCPODown alert this branch had temporarily dropped during conflict resolution is back, alongside this branch's own ingress alerts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot

🔵 Needs a closer look

Production paging behavior remains dependent on unverified live canary cardinality, scrape cadence, Azure Monitor PromQL semantics, and action-group routing, with key volume-floor and runtime validations deferred for human observability review.

3 open issues

_{Review effort: Lite}

openshift-merge-bot · 2026-10-29T00:08:55Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-parallel

openshift-ci · 2026-10-29T02:13:37Z

Arris Li (@ArrisLee): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	`adebafe`	link	true	`/test e2e-parallel`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copilot AI lite review requested due to automatic review settings September 23, 2026 00:52

openshift-ci Bot added do-not-merge/work-in-progress area/infrastructure Deployment/pipeline infrastructure config labels Sep 23, 2026

Copilot AI reviewed Sep 23, 2026

openshift-ci Bot added area/observability Automatically applied by owners-label for area/observability needs-ok-to-test labels Sep 23, 2026

Simon Wiencki (swiencki) reviewed Sep 23, 2026

View reviewed changes

Comment thread observability/alerts/ingress-availability-slo-prometheusRule.yaml Outdated

Arris Li (ArrisLee) force-pushed the huili/aro-25969-ingress-alerts branch from e9ab7f6 to 872455f Compare September 25, 2026 01:29

Copilot AI review requested due to automatic review settings September 25, 2026 01:29

Copilot AI reviewed Sep 25, 2026

Arris Li (ArrisLee) marked this pull request as ready for review September 25, 2026 01:31

openshift-ci Bot removed the do-not-merge/work-in-progress label Sep 25, 2026

openshift-ci Bot requested review from Brendan Bergen (SudoBrendan) and Steven Clarkson (sclarkso) September 25, 2026 01:31

Tejas Kesharwani (tkesharw-RH) reviewed Sep 28, 2026

View reviewed changes

Simon Wiencki (swiencki) reviewed Sep 28, 2026

View reviewed changes

openshift-ci Bot added the needs-rebase label Sep 28, 2026

Arris Li (ArrisLee) force-pushed the huili/aro-25969-ingress-alerts branch from 872455f to dda099f Compare September 30, 2026 03:02

Copilot AI lite review requested due to automatic review settings September 30, 2026 03:02

openshift-ci Bot removed the needs-rebase label Sep 30, 2026

Copilot AI reviewed Sep 30, 2026

openshift-ci Bot added the needs-rebase label Oct 2, 2026

Copilot AI lite review requested due to automatic review settings October 20, 2026 21:17

Arris Li (ArrisLee) force-pushed the huili/aro-25969-ingress-alerts branch from f813d7f to 29ffecb Compare October 20, 2026 21:17

Copilot started reviewing on behalf of Arris Li (ArrisLee) October 20, 2026 21:17

View session

Copilot AI reviewed Oct 20, 2026

View reviewed changes

Simon Wiencki (swiencki) reviewed Oct 21, 2026

View reviewed changes

Arris Li (ArrisLee) requested review from Simon Wiencki (swiencki) and a lite review from Copilot October 23, 2026 03:45

Copilot started reviewing on behalf of Arris Li (ArrisLee) October 28, 2026 05:05

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Arris Li (ArrisLee) and others added 11 commits October 29, 2026 12:32

Arris Li (ArrisLee) force-pushed the huili/aro-25969-ingress-alerts branch from a761495 to adebafe Compare October 28, 2026 23:36

Copilot AI lite review requested due to automatic review settings October 28, 2026 23:36

Copilot started reviewing on behalf of Arris Li (ArrisLee) October 28, 2026 23:37

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

Conversation

Arris Li (ArrisLee) commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing

Review history

Follow-ups (outside this PR)

Uh oh!

openshift-merge-bot Bot commented Sep 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Sep 23, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Tejas Kesharwani (tkesharw-RH) Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

Arris Li (ArrisLee) Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Simon Wiencki (swiencki) commented Sep 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Arris Li (ArrisLee) commented Sep 30, 2026

Uh oh!

Simon Wiencki (swiencki) commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

openshift-merge-bot Bot commented Oct 20, 2026

Uh oh!

Arris Li (ArrisLee) commented Oct 21, 2026

Uh oh!

Simon Wiencki (swiencki) Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Arris Li (ArrisLee) commented Oct 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

openshift-merge-bot Bot commented Oct 29, 2026

Uh oh!

openshift-ci Bot commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Arris Li (ArrisLee) commented Sep 23, 2026 •

edited

Loading

Arris Li (ArrisLee) Sep 30, 2026 •

edited

Loading