[chore] Add catch-up deploy profile overlay for credit-card analytics

finance-credit-card-chatbot/credit-card-analytics/package-catchup.json

@@ -0,0 +1,17 @@
+{
+  "engines": {
+    "flink": {
+      "config": {
+        "execution.checkpointing.interval": "1 h",
+        "execution.checkpointing.mode": "AT_LEAST_ONCE"
+      }
+    },
+    "postgres": {
+      "deployment": {
+        "replica-count": 0,
+        "create-indexes": false,
+        "data-checksums": false
+      }
+    }
+  }
+}

test-jobs/slow-query-test/README.md

@@ -0,0 +1,92 @@
+# Slow GraphQL Query Alert Test
+
+Deployment test job for the per-query slow-query alerts in cloud-backend
+(`metrics/metrics_alerting.sqrl` → `_SlowQueryAlerts`).
+
+## How the alert pipeline works
+
+1. The vertx server times every generated GraphQL query and publishes the
+   micrometer summary `sqrl_graphql_query_duration_seconds` tagged with
+   `name=<GraphQL field name>` and quantiles 0.5/0.99
+   (`GraphQLQueryMetricsInstrumentation`).
+2. Prometheus recording rules in `customer-dataplane-infrastructure`
+   (`sqrlpipeline-observability/.../vertx.yaml.tftpl`) derive
+   `x_sqrl_graphql_query_p50_seconds` / `x_sqrl_graphql_query_p99_seconds`
+   per `(deploymentId, service, name)` using `max_over_time(...[5m:])`.
+3. The cloud-backend metrics pipeline ingests these as
+   `BackendDeploymentMetrics` (the `name` label becomes `taskName`) and raises
+   an alert named `slow-query:<queryName>` when **p99 > 20s (level 1)** or
+   **p99 > 50s (level 2)** (thresholds in
+   `metrics/data-sources/static-data/metric_alerts_config.jsonl`).
+
+This job exposes two query endpoints:
+
+- `SlowQuery(n)` — cross-joins a 100k-row Postgres table with itself,
+  restricted to `id <= n` on both sides. Latency grows quadratically with `n`,
+  so you can dial p99 past either threshold at query time without redeploying.
+- `FastQuery(maxId)` — control; must never raise a slow-query alert.
+
+## Runbook
+
+### 1. Deploy
+
+Deploy this package (`slow-query-package.json`) to the target environment
+(e.g. staging). Wait for the Flink job to finish seeding (~10s of datagen at
+10k rows/sec; the 1 row/sec heartbeat source keeps the job running afterwards).
+
+### 2. Calibrate latency
+
+```bash
+ENDPOINT=https://<deployment-graphql-endpoint>/graphql
+
+time curl -s "$ENDPOINT" -H 'Content-Type: application/json' \
+  -d '{"query":"query { SlowQuery(n: 20000) { row_pairs max_id_sum } }"}'
+```
+
+Latency scales with `n²`: if `n=20000` takes `t` seconds, `n=20000*sqrt(k)`
+takes roughly `k*t`. Pick an `n` that lands in **25–40s** to trigger level 1,
+or **>50s** for level 2 (only reachable if no server/Postgres statement
+timeout cuts the request short — a timeout still records its duration in the
+timer, so a ~30s timeout caps observable p99 around 30s).
+
+### 3. Trigger the alert
+
+Fire a handful of slow queries so the p99 summary quantile reflects them
+(micrometer decays samples after a few minutes, so keep a slow trickle going):
+
+```bash
+for i in $(seq 1 10); do
+  curl -s "$ENDPOINT" -H 'Content-Type: application/json' \
+    -d '{"query":"query { SlowQuery(n: <calibrated>) { row_pairs max_id_sum } }"}' > /dev/null
+done
+```
+
+Also run the control a few times:
+
+```bash
+curl -s "$ENDPOINT" -H 'Content-Type: application/json' \
+  -d '{"query":"query { FastQuery(maxId: 100) { id payload } }"}'
+```
+
+### 4. Verify
+
+Allow ~5–10 min end to end (Prometheus scrape + `[5m:]` recording-rule window
++ metrics-pipeline ingestion). Then check the environment's alerts (UI alert
+panel or the metrics `getAlerts` GraphQL query) for the deployment:
+
+- `slow-query:SlowQuery` present — level 1 with trigger
+  `p99 latency <x>s exceeds threshold 20.0s`, or level 2 against 50s.
+- No `slow-query:FastQuery` alert.
+- Distinct slow queries alert independently (alert identity is
+  `slow-query:<name>`), so hitting both `SlowQuery` and another expensive query
+  yields separate alerts that snooze independently.
+
+Note: because the recording rule uses `max_over_time(...[5m:])`, the alert
+condition persists ~5 min after the last slow query.
+
+### 5. Clean up
+
+Delete the deployment. Logic-level coverage (no cluster needed) already exists
+in cloud-backend: `metrics` snapshot test `SlowQueryAlertTest` plus the
+`x_sqrl_graphql_query_*` rows in
+`metrics/cluster-events-local/BackendDeploymentMetrics.jsonl`.

test-jobs/slow-query-test/slow-query-package.json

@@ -0,0 +1,14 @@
+{
+  "version": "1",
+  "enabled-engines": ["vertx", "postgres", "flink"],
+  "script": {
+    "main": "slow-query.sqrl"
+  },
+  "engines": {
+    "flink": {
+      "config": {
+        "table.exec.source.idle-timeout": "1 s"
+      }
+    }
+  }
+}

test-jobs/slow-query-test/slow-query.sqrl

@@ -0,0 +1,64 @@
+/*
+ Test job for the per-query slow GraphQL query alerts in cloud-backend
+ (metrics_alerting.sqrl -> _SlowQueryAlerts).
+
+ The vertx server tags sqrl_graphql_query_duration_seconds with the GraphQL
+ field name, Prometheus recording rules derive x_sqrl_graphql_query_p50/p99_seconds
+ from it, and the metrics pipeline raises alerts when p99 exceeds 20s (level 1)
+ or 50s (level 2). SlowQuery below is deliberately expensive so those thresholds
+ can be crossed on demand; FastQuery is the control that must never alert.
+*/
+
+-- Bounded seed: 100k rows that land in Postgres via the SeedRows table.
+CREATE TABLE _SeedData (
+  id BIGINT NOT NULL,
+  payload STRING NOT NULL,
+  event_time AS NOW(),
+  WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
+) WITH (
+  'connector' = 'datagen',
+  'rows-per-second' = '10000',
+  'fields.id.kind' = 'sequence',
+  'fields.id.start' = '1',
+  'fields.id.end' = '100000',
+  'fields.payload.kind' = 'random',
+  'fields.payload.length' = '16'
+);
+
+SeedRows := DISTINCT _SeedData ON id ORDER BY event_time DESC;
+
+-- Unbounded trickle source so the Flink job keeps running after the bounded
+-- seed finishes and the deployment stays healthy while queries are tested.
+CREATE TABLE _Heartbeat (
+  beat DOUBLE NOT NULL,
+  hb_time AS NOW(),
+  WATERMARK FOR hb_time AS hb_time - INTERVAL '5' SECOND
+) WITH (
+  'connector' = 'datagen',
+  'rows-per-second' = '1',
+  'fields.beat.kind' = 'random',
+  'fields.beat.min' = '0.0',
+  'fields.beat.max' = '1.0'
+);
+
+_HeartbeatWindow := SELECT window_start, window_end, COUNT(*) AS beats
+FROM TABLE(TUMBLE(TABLE _Heartbeat, DESCRIPTOR(hb_time), INTERVAL '60' SECOND))
+GROUP BY window_start, window_end;
+
+EXPORT _HeartbeatWindow TO logger.heartbeat;
+
+/**
+  Deliberately slow query: Postgres scans :n x :n row pairs, so latency grows
+  quadratically with n. Tune n at query time to push p99 past the 20s (level 1)
+  or 50s (level 2) alert thresholds without redeploying.
+ */
+SlowQuery(n BIGINT NOT NULL) :=
+SELECT COUNT(*) AS row_pairs, MAX(a.id + b.id) AS max_id_sum
+FROM SeedRows a CROSS JOIN SeedRows b
+WHERE a.id <= :n AND b.id <= :n;
+
+/**
+  Fast control query: must never trigger a slow-query alert.
+ */
+FastQuery(maxId BIGINT NOT NULL) :=
+SELECT id, payload FROM SeedRows WHERE id <= :maxId ORDER BY id ASC LIMIT 10;