Indexing dashboard: generation column and spawn-aware prerender job links

[habdelra]

The Grafana indexing dashboard correlates indexing passes with the prerender_html jobs they spawn. The "View spawned prerender job" data link rode the job-id column unconditionally, so every row offered a link — including in-flight passes that haven't enqueued their prerender job yet and historical rows that predate the prerender split, where the link lands on an empty prerender-dashboard view. The Active Indexing table also had no way to see which realm generation an in-flight pass is producing.

Two changes to the Active Indexing and Completed Indexing Jobs panels:

Generation column on Active Indexing. An indexing pass computes its generation as realm_generations.current_generation + 1 at batch start and commits it in one write at the end, and per-realm indexing serializes — so the running pass owns the next generation for its whole lifetime. The panel derives it the same way: a LEFT JOIN on realm_generations (a primary-key lookup) plus one. Verified live: the value the panel showed for a running pass matched the generation its Completed row later reported from the job result.

Spawn-aware prerender links. Both panels resolve the spawned prerender_html job via the spawningJobId correlation the job carries in its args, exposed as a prerender_job_id column (newest wins when retries spawn more than one). A migration adds a partial expression index on jobs (((args->>'spawningJobId')::bigint), id) WHERE job_type = 'prerender_html' — built CONCURRENTLY so the production build doesn't lock queue writes — and each panel row resolves through a LATERAL probe of that index, so the lookup cost is independent of accumulated job history. The "View spawned prerender jobs" link now rides that column: a row links as soon as its prerender job exists, and a row with no spawned job shows an empty cell with nothing to click. The link keeps filtering the Prerender HTML dashboard by the spawning index job's id, which surfaces all of that pass's spawned jobs.

The existing activity-feed, job-logs, and delete-action links are untouched.

Screenshots from the local dev stack:

Active Indexing — in-flight pass with its generation and a link to the already-spawned prerender job:

Completed Indexing Jobs — generation and prerender_job_id columns; each prerender_job_id cell links to the prerender dashboard filtered to its spawning pass:

Verified against a live local Grafana reading the dev Postgres: the panel SQL runs on the dashboards' datasource, links carry the right var-spawning_job_id, a null prerender_job_id cell renders a zero-size anchor (invisible and unclickable), and the surrounding panels are unchanged.

🤖 Generated with Claude Code

[github-actions]

Observability diff (vs staging)

Show diff

diff --git a/tmp/remote-canon.5kcQGZ/dashboards/boxel-status/indexing.json b/tmp/committed-canon.FiiWOc/dashboards/boxel-status/indexing.json
index feff909..d7bf03e 100644
--- a/tmp/remote-canon.5kcQGZ/dashboards/boxel-status/indexing.json
+++ b/tmp/committed-canon.FiiWOc/dashboards/boxel-status/indexing.json
@@ -80,6 +80,10 @@
           "uid": "cef5v5sl9k7i8f"
         },
         "description": "System-wide operator action: queue a full reindex across every realm. The button disables itself while a `full-reindex` orchestration job is already pending or running. Per-realm reindex moved to the Realms dashboard. Click POSTs with `Authorization: Bearer ${grafana_secret}` (substituted from SSM at apply time, CS-10929).",
+        "fieldConfig": {
+          "defaults": {},
+          "overrides": []
+        },
         "gridPos": {
           "h": 8,
           "w": 24,
@@ -600,7 +604,7 @@
           "type": "grafana-postgresql-datasource",
           "uid": "cef5v5sl9k7i8f"
         },
-        "description": "One row per indexing job currently held by a worker (across all realm-server / worker tasks). `progress` and `current files` come from `job_progress` (CS-10930), populated by each realm-server's IndexingEventSink write-through. Click the realm cell to drill into the live activity feed; the job_id cell pivots to the Prerender HTML dashboard filtered to the prerender_html job this pass spawned.",
+        "description": "One row per indexing job currently held by a worker (across all realm-server / worker tasks). `progress` and `current files` come from `job_progress`, populated by each realm-server's IndexingEventSink write-through. `generation` is the realm generation this pass is producing (`realm_generations.current_generation + 1` — per-realm indexing serializes, so the in-flight pass owns the next generation until it commits it). `prerender_job_id` fills in once this pass's `prerender_html` job exists — typically at commit, when the enqueue lands atomically with the generation bump, or earlier when coalescing attached the work to an existing job — and pivots to the Prerender HTML dashboard filtered to this pass's spawned jobs; an empty cell means no prerender job has been spawned yet. Click the realm cell to drill into the live activity feed.",
         "fieldConfig": {
           "defaults": {
             "color": {
@@ -800,7 +804,7 @@
             {
               "matcher": {
                 "id": "byName",
-                "options": "job_id"
+                "options": "prerender_job_id"
               },
               "properties": [
                 {
@@ -808,7 +812,7 @@
                   "value": [
                     {
                       "targetBlank": true,
-                      "title": "View spawned prerender job",
+                      "title": "View spawned prerender jobs",
                       "url": "/d/b5eou6wleg7i6o?${__url_time_range}&var-spawning_job_id=${__data.fields.job_id}&orgId=1"
                     }
                   ]
@@ -881,7 +885,7 @@
             "editorMode": "code",
             "format": "table",
             "rawQuery": true,
-            "rawSql": "SELECT\n  job_id,\n  realm,\n  realm_url,\n  job_type,\n  files_completed,\n  total_files,\n  percent,\n  percent AS pct,\n  elapsed_seconds,\n  started_at,\n  worker_id,\n  reservation_id\nFROM (\n  SELECT\n    j.id AS job_id,\n    COALESCE(\n      NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n      REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n    ) AS realm,\n    COALESCE(j.args->>'realmURL','') AS realm_url,\n    j.job_type,\n    COALESCE(jp.files_completed, 0) AS files_completed,\n    COALESCE(jp.total_files, 0) AS total_files,\n    CASE WHEN COALESCE(jp.total_files, 0) > 0\n      THEN (jp.files_completed::float / jp.total_files) * 100\n      ELSE 0\n    END AS percent,\n    EXTRACT(EPOCH FROM (NOW() - jr.created_at)) AS elapsed_seconds,\n    jr.created_at AS started_at,\n    jr.worker_id,\n    jr.id AS reservation_id\n  FROM jobs j\n  JOIN job_reservations jr ON jr.job_id = j.id\n    AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n  LEFT JOIN job_progress jp ON jp.job_id = j.id\n  WHERE j.job_type IN ('from-scratch-index','incremental-index')\n    AND j.finished_at IS NULL\n    AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\n) active\nORDER BY started_at DESC;",
+            "rawSql": "SELECT\n  job_id,\n  realm,\n  realm_url,\n  job_type,\n  generation,\n  prerender_job_id,\n  files_completed,\n  total_files,\n  percent,\n  percent AS pct,\n  elapsed_seconds,\n  started_at,\n  worker_id,\n  reservation_id\nFROM (\n  SELECT\n    j.id AS job_id,\n    COALESCE(\n      NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n      REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n    ) AS realm,\n    COALESCE(j.args->>'realmURL','') AS realm_url,\n    j.job_type,\n    COALESCE(rg.current_generation, 0) + 1 AS generation,\n    sp.prerender_job_id,\n    COALESCE(jp.files_completed, 0) AS files_completed,\n    COALESCE(jp.total_files, 0) AS total_files,\n    CASE WHEN COALESCE(jp.total_files, 0) > 0\n      THEN (jp.files_completed::float / jp.total_files) * 100\n      ELSE 0\n    END AS percent,\n    EXTRACT(EPOCH FROM (NOW() - jr.created_at)) AS elapsed_seconds,\n    jr.created_at AS started_at,\n    jr.worker_id,\n    jr.id AS reservation_id\n  FROM jobs j\n  JOIN job_reservations jr ON jr.job_id = j.id\n    AND jr.completed_at IS NULL AND jr.locked_until > NOW()\n  LEFT JOIN job_progress jp ON jp.job_id = j.id\n  LEFT JOIN realm_generations rg ON rg.realm_url = j.args->>'realmURL'\n  LEFT JOIN LATERAL (\n    SELECT MAX(pj.id) AS prerender_job_id\n      FROM jobs pj\n     WHERE pj.job_type = 'prerender_html'\n       AND pj.args->>'spawningJobId' IS NOT NULL\n       AND (pj.args->>'spawningJobId')::bigint = j.id\n  ) sp ON TRUE\n  WHERE j.job_type IN ('from-scratch-index','incremental-index')\n    AND j.finished_at IS NULL\n    AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\n) active\nORDER BY started_at DESC;",
             "refId": "A"
           }
         ],
@@ -1178,7 +1182,7 @@
           "type": "grafana-postgresql-datasource",
           "uid": "cef5v5sl9k7i8f"
         },
-        "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). `generation` is the realm index generation the pass committed (NULL for full-reindex orchestration and results from builds that predate generation stamping). The id cell pivots to the Prerender HTML dashboard filtered to the prerender_html job this pass spawned. Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
+        "description": "Indexing jobs (from-scratch-index, incremental-index, full-reindex) that finished in the past 24 hours, newest first. `wait_seconds` is queue time (created_at → first reservation). `run_seconds` is the duration of the attempt that completed it (latest reservation → finished_at). `generation` is the realm index generation the pass committed (NULL for full-reindex orchestration and results from builds that predate generation stamping). `prerender_job_id` is the `prerender_html` job this pass spawned (the newest when retries spawned more than one); the cell pivots to the Prerender HTML dashboard filtered to this pass's spawned jobs, and an empty cell means none was spawned (e.g. rows that predate the prerender split). The lookup probes the partial expression index on the spawningJobId the prerender job's args carry. Force-cancelled jobs that never got a reservation appear with NULL started_at / wait_seconds / run_seconds / worker_id.",
         "fieldConfig": {
           "defaults": {
             "color": {
@@ -1211,7 +1215,7 @@
             {
               "matcher": {
                 "id": "byName",
-                "options": "id"
+                "options": "prerender_job_id"
               },
               "properties": [
                 {
@@ -1219,7 +1223,7 @@
                   "value": [
                     {
                       "targetBlank": true,
-                      "title": "View spawned prerender job",
+                      "title": "View spawned prerender jobs",
                       "url": "/d/b5eou6wleg7i6o?${__url_time_range}&var-spawning_job_id=${__data.fields.id}&orgId=1"
                     }
                   ]
@@ -1333,7 +1337,7 @@
             "editorMode": "code",
             "format": "table",
             "rawQuery": true,
-            "rawSql": "SELECT\n  j.id,\n  lr.reservation_id,\n  j.job_type,\n  CASE\n    WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n    ELSE COALESCE(\n      NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n      REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n    )\n  END AS realm,\n  j.status,\n  j.created_at,\n  lr.started_at,\n  j.finished_at,\n  EXTRACT(EPOCH FROM (fr.first_started_at - j.created_at)) AS wait_seconds,\n  EXTRACT(EPOCH FROM (j.finished_at - lr.started_at)) AS run_seconds,\n  (j.result->>'generation')::bigint AS generation,\n  lr.worker_id\nFROM jobs j\nLEFT JOIN LATERAL (\n  SELECT jr.id AS reservation_id, jr.created_at AS started_at, jr.worker_id\n  FROM job_reservations jr\n  WHERE jr.job_id = j.id\n  ORDER BY jr.created_at DESC\n  LIMIT 1\n) lr ON TRUE\nLEFT JOIN LATERAL (\n  SELECT MIN(jr.created_at) AS first_started_at\n  FROM job_reservations jr\n  WHERE jr.job_id = j.id\n) fr ON TRUE\nWHERE j.job_type IN ('from-scratch-index','incremental-index','full-reindex')\n  AND j.finished_at IS NOT NULL\n  AND j.finished_at > NOW() - INTERVAL '24 hours'\n  AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\nORDER BY j.finished_at DESC\nLIMIT 500;",
+            "rawSql": "SELECT\n  j.id,\n  lr.reservation_id,\n  j.job_type,\n  CASE\n    WHEN j.job_type = 'full-reindex' THEN '(all realms)'\n    ELSE COALESCE(\n      NULLIF(RTRIM(REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://[^/]+/', ''), '/'), ''),\n      REGEXP_REPLACE(COALESCE(j.args->>'realmURL',''), '^https?://([^./:]+).*$', '\\1')\n    )\n  END AS realm,\n  j.status,\n  j.created_at,\n  lr.started_at,\n  j.finished_at,\n  EXTRACT(EPOCH FROM (fr.first_started_at - j.created_at)) AS wait_seconds,\n  EXTRACT(EPOCH FROM (j.finished_at - lr.started_at)) AS run_seconds,\n  (j.result->>'generation')::bigint AS generation,\n  sp.prerender_job_id,\n  lr.worker_id\nFROM jobs j\nLEFT JOIN LATERAL (\n  SELECT jr.id AS reservation_id, jr.created_at AS started_at, jr.worker_id\n  FROM job_reservations jr\n  WHERE jr.job_id = j.id\n  ORDER BY jr.created_at DESC\n  LIMIT 1\n) lr ON TRUE\nLEFT JOIN LATERAL (\n  SELECT MIN(jr.created_at) AS first_started_at\n  FROM job_reservations jr\n  WHERE jr.job_id = j.id\n) fr ON TRUE\nLEFT JOIN LATERAL (\n  SELECT MAX(pj.id) AS prerender_job_id\n    FROM jobs pj\n   WHERE pj.job_type = 'prerender_html'\n     AND pj.args->>'spawningJobId' IS NOT NULL\n     AND (pj.args->>'spawningJobId')::bigint = j.id\n) sp ON TRUE\nWHERE j.job_type IN ('from-scratch-index','incremental-index','full-reindex')\n  AND j.finished_at IS NOT NULL\n  AND j.finished_at > NOW() - INTERVAL '24 hours'\n  AND (${job_id:sqlstring} = '' OR j.id::text = ${job_id:sqlstring})\nORDER BY j.finished_at DESC\nLIMIT 500;",
             "refId": "A"
           }
         ],

(Run: https://github.com/cardstack/boxel/actions/runs/29129447268)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 929ccf6596

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Updates the Boxel “Indexing” Grafana dashboard to better correlate indexing passes with the prerender jobs they spawn, and to expose the in-flight generation a running pass is producing.

Changes:

• Adds an inferred generation column to the Active Indexing table (realm_generations.current_generation + 1).
• Makes the “View spawned prerender job” link spawn-aware by joining prerender_html jobs via args.spawningJobId and linking only when a prerender_job_id exists.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files      1 suites   2h 54m 0s ⏱️
3 469 tests 3 454 ✅ 15 💤 0 ❌
3 488 runs  3 473 ✅ 15 💤 0 ❌

Results for commit c7ebeb4.

Realm Server Test Results

    1 files      1 suites   11m 42s ⏱️
1 815 tests 1 815 ✅ 0 💤 0 ❌
1 894 runs  1 894 ✅ 0 💤 0 ❌

Results for commit c7ebeb4.