Skip to content

feat: collect and export a relations array per query - #135

Open
AliakseiAndryiashau2026 wants to merge 5 commits into
ClickHouse:mainfrom
AliakseiAndryiashau2026:upstream-relations
Open

AliakseiAndryiashau2026 wants to merge 5 commits into
ClickHouse:mainfrom
AliakseiAndryiashau2026:upstream-relations

Conversation

@AliakseiAndryiashau2026

@AliakseiAndryiashau2026 AliakseiAndryiashau2026 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Adds a per-query "relations" field: the distinct tables/views/materialized views/foreign tables/partitioned tables a query touches, schema-qualified, with the actual pg_class.relkind carried per entry.

Collection

A new ExecutorCheckPerms_hook walks the resolved range table, dedupes same-relation-twice, and resolves names via get_namespace_name/get_rel_name. Capped by a new pg_stat_ch.relations_max_len GUC (default 1023 bytes); over-budget queries set relations_truncated rather than fail, same as query text already does.

A nested executor invocation -- a non-inlined SQL function, PL/pgSQL, dynamic EXECUTE, a trigger, SPI -- gets its own staging buffer (indexed by nesting depth) so it can't clobber an outer statement's already-collected relations.

The feature is gated by pg_stat_ch.relations_enabled (PGC_POSTMASTER, default off -- the ClickHouse exporter keeps its column set for the life of the bgworker, so toggling it requires a restart). The per-query budget, pg_stat_ch.relations_max_len (PGC_SUSET, default 1023 bytes), can still be changed live. A synthetic benchmark across join widths of 1/5/10/20 tables measured the added cost at +0.05us to +1.75us per query once enabled -- small enough to leave on in the config, not costly enough to ever need live disabling.

Export

The relations blob rides through the existing ring buffer via a new dsa_pointer field (relations_dsa), same pattern as query text and error messages. Exported as real array types on all three column-interface-based paths:

  • ClickHouse native: Array(LowCardinality(String)) / Array(FixedString(1)) via a new StatHCStringArray() column type
  • OTLP logs: an array-value log attribute
  • Unified Arrow exporter: list(utf8) columns

The legacy Arrow exporter doesn't carry relations yet as it would need a new schema column.

Schema: three arrays vs. a tuple column

Went with three parallel arrays (relation_namespaces, relation_names, relation_kinds) plus a relations_truncated flag, rather than a single Array(Tuple(namespace, name, relkind)) column. Benchmarked both on a 5M-row synthetic dataset (2000 distinct relations across 8 schemas, skewed access pattern, realistic per-query relation counts):

3-array tuple delta
storage, no codec (LZ4) 72.1 MiB 52.8 MiB tuple 27% smaller
storage, ZSTD(1) 37.4 MiB 31.9 MiB tuple 15% smaller
storage, ZSTD(3) 34.7 MiB 31.2 MiB tuple ~10% smaller
write (INSERT, 5M rows) 540ms 412ms tuple ~24% faster
read: lookup by relation name 11ms 21ms 3-array ~1.9x faster
read: top-20 relations by frequency 17ms 29ms 3-array ~1.7x faster

Tuple writes faster and takes less space, but neither cost shows up much in practice -- the background worker eats the write cost asynchronously, and the storage gap mostly closes once ZSTD(3) is on. Reads are where this field actually gets used (querying events_raw for "what tables did this touch" or "what's hot"), and that's exactly where three-array wins by a meaningful margin. Kept three-array.

Testing

  • Regression: 11/11, including the new GUC and no-crash coverage across plain tables, schema-qualified tables, self-joins, and views
  • Isolation: 3/3 (ring-buffer concurrency, exercises the new relations_dsa enqueue/dequeue path directly)
  • Unit tests: 19/19 (budget check, encoding/decoding, OTel attribute building)
  • TAP (t/039_relations_export.pl): 18/18 against a live ClickHouse instance -- actual captured values, the length-prefixed encoding's edge cases (special characters, every relkind, the truncation boundary, max-length identifiers), nested-invocation handling across four nesting mechanisms, relation drop/recreate/rename, and capture through an extension invoking SPI internally

Happy to share the benchmark script/methodology if anyone wants to reproduce the schema-shape numbers.


Note

Medium Risk
Touches the hot-path ExecutorEnd hook, catalog lookups, shared-memory enqueue/dequeue, and exporter column sets gated by postmaster GUCs—errors could affect query overhead or export compatibility if misconfigured.

Overview
Adds optional per-query relation tracking: when pg_stat_ch.relations_enabled is on (restart required; schema migration first), completed queries record distinct touched relations at ExecutorEnd into a compact blob bounded by pg_stat_ch.relations_max_len, with relations_truncated when the budget is exceeded.

Collection deduplicates OIDs, collapses partition/inheritance children to parents, and resolves schema/name/relkind via catalog lookups. Events and the shared ring buffer gain a relations field (relations + relations_dsa), including OOM handling via a sentinel length.

Export decodes the blob into three parallel arrays—relation_namespaces, relation_names, relation_kinds—plus relations_truncated on ClickHouse (new StatHCStringArray / Array(String)), OTLP (array log attributes), and the unified Arrow exporter (list(utf8)). Columns are omitted when the feature is disabled so older ClickHouse schemas still accept inserts.

Documentation, a ClickHouse migration, CI/regression (relations), expanded Arrow e2e checks, and comprehensive TAP coverage (t/039_relations_export.pl) land with the feature.

Reviewed by Cursor Bugbot for commit 6fa0cfa. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI balanced review requested due to automatic review settings October 12, 2026 13:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Executor nesting state can become permanently saturated, while Docker, unit-test, and migration wiring also contain blocking defects.

5 open findings
What changed in this PR

Adds structured per-query relation telemetry from PostgreSQL hooks through shared memory into ClickHouse, OTLP, and unified Arrow exports.

Changes:

  • Collects, deduplicates, encodes, and queues touched relations.
  • Adds relation arrays, truncation metadata, GUCs, schema migrations, and documentation.
  • Adds regression, TAP, and unit coverage with CI integration.
File Description
vcpkg.json Adds GoogleTest.
test/​unit/​relations_decode_test.cc Tests relation decoding.
test/​unit/​relations_budget_test.cc Tests byte-budget checks.
test/​unit/​otel_string_array_attr_test.cc Tests OTLP array attributes.
test/​regression/​sql/​relations.sql Adds relation regression scenarios.
test/​regression/​expected/​relations.out Defines expected regression output.
test/​regression/​expected/​guc.out Includes new GUCs.
t/​039_relations_export.pl Adds end-to-end export coverage.
src/​queue/​shmem.c Queues relation blobs through DSA.
src/​queue/​ring_entry.h Adds relation DSA metadata.
src/​queue/​event.h Adds relation storage to events.
src/​hooks/​hooks.c Collects and stages relations.
src/​export/​stats_exporter.cc Decodes and dispatches relation fields.
src/​export/​relations_decode.h Implements blob decoding.
src/​export/​otel_string_array_attr.h Builds OTLP array attributes.
src/​export/​otel_exporter.cc Exports OTLP relation arrays.
src/​export/​otel_arrow_exporter.cc Adds Arrow list columns.
src/​export/​exporter_interface.h Adds string-array column API.
src/​export/​clickhouse_exporter.cc Builds ClickHouse array columns.
src/​config/​guc.c Registers relation GUCs.
scripts/​run-tests.sh Runs relation regression tests.
schema/​migrations/​20261006000002_drop_relations_column.sql Replaces the legacy relation column.
schema/​migrations/​20261006000001_add_relation_namespace_name_kind_columns.sql Adds structured relation columns.
schema/​migrations/​20260928000001_add_relations_column.sql Adds the transitional relation column.
README.md Documents relation GUCs.
include/​hooks/​relations_budget.h Defines the budget helper.
include/​config/​guc.h Declares relation settings.
docs/​reference/​hooks.mdx Documents collection hooks.
docs/​reference/​events-schema.mdx Documents relation columns.
docs/​reference/​configuration.mdx Documents relation configuration.
docs/​guides/​opentelemetry.mdx Documents OTLP representation.
CMakeLists.txt Configures unit tests.
.github/​workflows/​ci.yml Adds relation regression coverage.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread CMakeLists.txt Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread CMakeLists.txt Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/hooks/hooks.c Resolved
Copilot AI balanced review requested due to automatic review settings October 12, 2026 14:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/hooks/hooks.c Resolved
Comment thread src/export/exporter_interface.h Resolved
Comment thread t/039_relations_export.pl Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/hooks/hooks.c Resolved
Copilot AI balanced review requested due to automatic review settings October 12, 2026 16:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/export/otel_string_array_attr.h Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread src/queue/event.h Resolved
Comment thread src/export/otel_arrow_exporter.cc Resolved
Comment thread test/unit/relations_nesting_test.cc Resolved
Copilot AI balanced review requested due to automatic review settings October 12, 2026 17:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/export/otel_arrow_exporter.cc Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/hooks/hooks.c Resolved
Copilot AI balanced review requested due to automatic review settings October 13, 2026 09:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Disabled collection is incorrectly marked truncated, and the legacy migration rollback can break its dependent materialized view.

2 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread src/hooks/hooks.c Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/hooks/hooks.c Resolved
Copilot AI balanced review requested due to automatic review settings October 13, 2026 10:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Committed subtransactions can leave nesting entries at the wrong level, allowing sibling savepoint aborts to corrupt live cursor relation state.

1 open finding
2 resolved since last review

🧠 Review effort: Balanced

Comment thread src/hooks/hooks.c Resolved
Copilot AI balanced review requested due to automatic review settings October 13, 2026 11:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The native-array wire contract depends on vendored clickhouse-c sources absent from this snapshot and requires live ClickHouse validation of serialization and casts into LowCardinality/String and FixedString array columns.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Comment thread src/hooks/hooks.c Resolved
Comment thread t/039_relations_export.pl Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread docs/reference/events-schema.mdx Resolved
Comment thread docs/reference/configuration.mdx Resolved
…st removal)

- Drop the stale per-query cost figure from configuration.mdx (measured
  before the ExecutorEnd refactor, now stale).
- Update hooks.c comments: partition/inheritance collapse no longer
  lists children, and document that error events carry no relations
  (collected only at ExecutorEnd).
- events-schema.mdx: clarify relation_namespaces wording for the same
  two points.
- Remove the GoogleTest unit tests and the vcpkg/CMake machinery behind
  them, following ClickHouse#80's precedent, including the now-dead "Run unit
  tests" CI step. Normal-path decoding is already proven end-to-end by
  t/039; add a TAP subtest there for the previously-untested
  failed-query path (relations_len=0).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The CI CTest step can pass without running unit tests because the referenced test sources and CMake test registrations are missing, and an empty test suite does not fail CI.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unavailable clickhouse-c sources and PostgreSQL/ClickHouse runtimes leave version-sensitive relation handling and native array export contracts unverified across supported versions.

No open issues 🎉

Review effort: Balanced

Comment thread docker/postgres-ext.Dockerfile Resolved
Comment thread src/export/otel_string_array_attr.h Resolved
Comment thread src/export/relations_decode.h Resolved
Comment thread t/039_relations_export.pl Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread src/hooks/hooks.c Resolved
Comment thread src/hooks/hooks.c Resolved
…ce TAP case

- Drop the now-unused -DPSCH_BUILD_UNIT_TESTS=OFF Dockerfile arg and restore
  CMakeLists.txt's trailing blank line, so neither appears in the diff.
- Drop stale header comments pointing at deleted test/unit/*_test.cc files.
- Size CollectRelations' seen-set with a small fixed constant instead of
  scaling it to the range table: the byte budget caps it at ~200 entries
  regardless, and planner-generated children never reach it.
- Add a TAP subtest for plain-inheritance collapse (t/039), mirroring the
  existing partition-collapse coverage.

Addresses jsc0218's 2026-11-17 review comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the optional per-query relation telemetry change.

No open issues 🎉

Review effort: Balanced

- Move PschRelidSeenEntry, the simplehash instantiation, and
  CollectRelations out of hooks.c into relations_collect.c/.h, dropping
  the forward declaration (item 7).
- Split CollectRelations into BuildChildMap, ShouldReportRte, and
  AppendRelationEntry (item 8).

Addresses jsc0218's 2026-11-17 review comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed because asynchronous export timing may let the test accept stale results without verifying relations after table recreation.

No open issues 🎉

Review effort: Balanced

Comment thread src/queue/event.h
// matrix across non-UTF-8/ICU locales; left as a documented gap for now.
char err_message[PSCH_MAX_ERR_MSG_LEN]; // Error message text (truncated if necessary)
char query[PSCH_MAX_QUERY_LEN]; // Query text, unterminated, see query_len
char relations[PSCH_MAX_RELATIONS_LEN]; // See CollectRelations in hooks.c for layout.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from the move: CollectRelations now lives in src/hooks/relations_collect.c, so this pointer is stale (the comment at lines 46-47 of this file was already updated). Suggested wording:

Suggested change
char relations[PSCH_MAX_RELATIONS_LEN]; // See CollectRelations in hooks.c for layout.
char relations[PSCH_MAX_RELATIONS_LEN]; // See CollectRelations in relations_collect.c for layout.

Comment thread t/039_relations_export.pl
Comment on lines +478 to +479
unlike($names, qr/'big_mid_599'/, "subpartitioned intermediate past the old cap not reported (got: $names)");
unlike($names, qr/'big_leaf_599'/, "leaf past the old cap not reported (got: $names)");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two messages still say "past the old cap", which refers to the earlier design (before collapsing). Now the reason these are not reported is that they are partitions collapsed into their parent. Suggested wording:

Suggested change
unlike($names, qr/'big_mid_599'/, "subpartitioned intermediate past the old cap not reported (got: $names)");
unlike($names, qr/'big_leaf_599'/, "leaf past the old cap not reported (got: $names)");
unlike($names, qr/'big_mid_599'/, "subpartitioned intermediate collapsed into its parent, not reported (got: $names)");
unlike($names, qr/'big_leaf_599'/, "leaf partition collapsed into its parent, not reported (got: $names)");

Comment thread src/config/guc.c
int psch_otel_max_block_bytes = 3 * 1024 * 1024; // 3 MiB (max: 16 MiB)
char* psch_extra_attributes = NULL;
char* psch_debug_arrow_dump_dir = NULL;
int psch_relations_max_len = 1023; // bytes, leaves room for the NUL in PSCH_MAX_RELATIONS_LEN=1024

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "room for the NUL" rationale doesn't apply to the relations blob

The relations blob is length-prefixed binary ([ns_len][ns][name_len][name][relkind]...), not a C string, so nothing in it is NUL-terminated, and CopyRelations copies exactly buf.len bytes with no terminator. The full 1024 bytes of event->relations are usable, so the cap of 1023 (here, and the same remark on the max at line 425) is one byte lower than needed for that reason.

Either raise the max to PSCH_MAX_RELATIONS_LEN (and update the default, docs and README to match), or keep 1023 and reword the comment so it doesn't give a reason that isn't true. This is harmless either way, just a misleading comment.

| `relation_namespaces` | `Array(LowCardinality(String))` | Schema name of each distinct table/view/materialized view/foreign table/partitioned table the query touched, positionally parallel to `relation_names` and `relation_kinds`. Partitions and inheritance children are reported through their parent, not individually (for example, `SELECT * FROM inh_p` lists only `inh_p`). Recorded only for queries that complete; error events have empty relation arrays. |
| `relation_names` | `Array(LowCardinality(String))` | Relation name of each distinct relation the query touched. |
| `relation_kinds` | `Array(FixedString(1))` | `pg_class.relkind` for each entry: `r` (table), `v` (view), `m` (materialized view), `f` (foreign table), `p` (partitioned table), etc. |
| `relations_truncated` | `Bool` | `true` if the query touched more relations than fit within `pg_stat_ch.relations_max_len`; the arrays above are incomplete for that event. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small doc gap: relations_truncated is also set when the whole blob was lost because the DSA string area was out of memory (the decoder reports truncated = true with empty arrays for the PSCH_RELATIONS_OOM_LEN sentinel). As written, a reader would assume the arrays only ever lose their tail after exceeding relations_max_len. Suggested wording:

Suggested change
| `relations_truncated` | `Bool` | `true` if the query touched more relations than fit within `pg_stat_ch.relations_max_len`; the arrays above are incomplete for that event. |
| `relations_truncated` | `Bool` | `true` if the arrays above are incomplete for that event: the query touched more relations than fit within `pg_stat_ch.relations_max_len`, or the relations data was dropped because the shared string area was out of memory (in which case the arrays are empty). |

@@ -0,0 +1,19 @@
-- Adds structured relation columns: namespace, name, and relkind kept as
-- separate parallel arrays. Collected in src/hooks/hooks.c at ExecutorEnd.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale pointer after the refactor: CollectRelations now lives in src/hooks/relations_collect.c (called from hooks.c at ExecutorEnd). Suggested wording:

Suggested change
-- separate parallel arrays. Collected in src/hooks/hooks.c at ExecutorEnd.
-- separate parallel arrays. Collected in src/hooks/relations_collect.c at ExecutorEnd.

Comment thread src/queue/event.h
char client_addr[46]; // Client IP address (INET6_ADDRSTRLEN=46)
uint8 client_addr_len; // Actual length
uint16 query_len; // Actual length of query text
uint16 relations_len; // Actual length of relations blob (0 for utility statements)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is only true for utility statements. relations_len is also 0 for error events (which never reach ExecutorEnd), and whenever pg_stat_ch.relations_enabled is off. Suggested wording:

Suggested change
uint16 relations_len; // Actual length of relations blob (0 for utility statements)
uint16 relations_len; // Actual length of relations blob (0 if not recorded: feature off, utility statement, or error event)

Comment thread src/config/guc.c
Comment on lines +419 to +420
"Mirrors the query-text truncation pattern: the relations blob is a fixed "
"backend-local buffer, truncated (not grown) when a query touches enough "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This long description is out of date after the move to ExecutorEnd: the relations blob is no longer a fixed backend-local buffer. CollectRelations builds it in a StringInfo allocated in es_query_cxt, and it is then copied into event->relations. Suggested wording:

Suggested change
"Mirrors the query-text truncation pattern: the relations blob is a fixed "
"backend-local buffer, truncated (not grown) when a query touches enough "
"Mirrors the query-text truncation pattern: the encoded list is truncated "
"(not grown) when a query touches enough "

Comment thread src/queue/shmem.c
Comment on lines 186 to +187
// Total shmem requested from the postmaster: the shared block plus the
// interner HTAB (which ShmemInitHash carves out of the same pool).
// query-text interner HTAB (carved from the same pool by ShmemInitHash).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, unrelated to relations: this hunk, the one at lines 193-194 and the one at lines 270-271 only reword existing comments about the query-text interner ("interner HTAB" to "query-text interner HTAB", re-wrapping "same handle"). They don't change behavior and aren't connected to this feature, so they add noise to the diff and to git blame. Suggest dropping these three hunks from the PR and keeping shmem.c to the relations changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants