Repository navigation
feat: collect and export a relations array per query - #135
AliakseiAndryiashau2026 wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Executor nesting state can become permanently saturated, while Docker, unit-test, and migration wiring also contain blocking defects.
5 open findings
What changed in this PR
Adds structured per-query relation telemetry from PostgreSQL hooks through shared memory into ClickHouse, OTLP, and unified Arrow exports.
Changes:
- Collects, deduplicates, encodes, and queues touched relations.
- Adds relation arrays, truncation metadata, GUCs, schema migrations, and documentation.
- Adds regression, TAP, and unit coverage with CI integration.
| File | Description |
|---|---|
vcpkg.json |
Adds GoogleTest. |
test/unit/relations_decode_test.cc |
Tests relation decoding. |
test/unit/relations_budget_test.cc |
Tests byte-budget checks. |
test/unit/otel_string_array_attr_test.cc |
Tests OTLP array attributes. |
test/regression/sql/relations.sql |
Adds relation regression scenarios. |
test/regression/expected/relations.out |
Defines expected regression output. |
test/regression/expected/guc.out |
Includes new GUCs. |
t/039_relations_export.pl |
Adds end-to-end export coverage. |
src/queue/shmem.c |
Queues relation blobs through DSA. |
src/queue/ring_entry.h |
Adds relation DSA metadata. |
src/queue/event.h |
Adds relation storage to events. |
src/hooks/hooks.c |
Collects and stages relations. |
src/export/stats_exporter.cc |
Decodes and dispatches relation fields. |
src/export/relations_decode.h |
Implements blob decoding. |
src/export/otel_string_array_attr.h |
Builds OTLP array attributes. |
src/export/otel_exporter.cc |
Exports OTLP relation arrays. |
src/export/otel_arrow_exporter.cc |
Adds Arrow list columns. |
src/export/exporter_interface.h |
Adds string-array column API. |
src/export/clickhouse_exporter.cc |
Builds ClickHouse array columns. |
src/config/guc.c |
Registers relation GUCs. |
scripts/run-tests.sh |
Runs relation regression tests. |
schema/migrations/20261006000002_drop_relations_column.sql |
Replaces the legacy relation column. |
schema/migrations/20261006000001_add_relation_namespace_name_kind_columns.sql |
Adds structured relation columns. |
schema/migrations/20260928000001_add_relations_column.sql |
Adds the transitional relation column. |
README.md |
Documents relation GUCs. |
include/hooks/relations_budget.h |
Defines the budget helper. |
include/config/guc.h |
Declares relation settings. |
docs/reference/hooks.mdx |
Documents collection hooks. |
docs/reference/events-schema.mdx |
Documents relation columns. |
docs/reference/configuration.mdx |
Documents relation configuration. |
docs/guides/opentelemetry.mdx |
Documents OTLP representation. |
CMakeLists.txt |
Configures unit tests. |
.github/workflows/ci.yml |
Adds relation regression coverage. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
7692911 to
9656a6b
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The depth-only relation stack can leak or associate relations with the wrong query when startup fails or portals end out of order.
6 open findings
Fix missing unit tests in Docker image configuration Track staged relations per query to prevent stack leaks and mix-ups · New Honor sub-header relation length budgets Update stale comment about array-column fallback behavior · New Test name and calculation use the wrong 1024-byte budget · New Run discovered unit tests in CI
2 resolved since last review
🧠 Review effort: Balanced
9656a6b to
c4b4560
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Relation loss, UTF-8 handling, OTLP sizing, and materialized-view data-loss risks remain unresolved.
8 open findings
Down migration loses recent history without backfill · New Materialized view recreation loses historical rows · New Payload estimate omits protobuf array element overhead · New Relation telemetry silently lost beyond 16 active QueryDesc entries · New Non-UTF-8 relation names can corrupt OTLP and Arrow exports · New Example shows incorrect schema-qualified relation_names format · New Make constant kKeys array immutable · New Run discovered unit tests in CI
5 resolved since last review
🧠 Review effort: Balanced
c4b4560 to
f70ec74
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The fixed nesting capacity can silently emit incorrect relation data, and Arrow list offsets are omitted from chunk sizing.
2 open findings
7 resolved since last review
Non-UTF-8 relation names can corrupt OTLP and Arrow exports Payload estimate omits protobuf array element overhead Materialized view recreation loses historical rows Down migration loses recent history without backfill Make constant kKeys array immutable Example shows incorrect schema-qualified relation_names format Run discovered unit tests in CI
🧠 Review effort: Balanced
f70ec74 to
4e3cb18
Compare
4e3cb18 to
f3e998a
Compare
f3e998a to
e5279e7
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The native-array wire contract depends on vendored clickhouse-c sources absent from this snapshot and requires live ClickHouse validation of serialization and casts into LowCardinality/String and FixedString array columns.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
…st removal) - Drop the stale per-query cost figure from configuration.mdx (measured before the ExecutorEnd refactor, now stale). - Update hooks.c comments: partition/inheritance collapse no longer lists children, and document that error events carry no relations (collected only at ExecutorEnd). - events-schema.mdx: clarify relation_namespaces wording for the same two points. - Remove the GoogleTest unit tests and the vcpkg/CMake machinery behind them, following ClickHouse#80's precedent, including the now-dead "Run unit tests" CI step. Normal-path decoding is already proven end-to-end by t/039; add a TAP subtest there for the previously-untested failed-query path (relations_len=0).
…ce TAP case - Drop the now-unused -DPSCH_BUILD_UNIT_TESTS=OFF Dockerfile arg and restore CMakeLists.txt's trailing blank line, so neither appears in the diff. - Drop stale header comments pointing at deleted test/unit/*_test.cc files. - Size CollectRelations' seen-set with a small fixed constant instead of scaling it to the range table: the byte budget caps it at ~200 entries regardless, and planner-generated children never reach it. - Add a TAP subtest for plain-inheritance collapse (t/039), mirroring the existing partition-collapse coverage. Addresses jsc0218's 2026-11-17 review comments.
- Move PschRelidSeenEntry, the simplehash instantiation, and CollectRelations out of hooks.c into relations_collect.c/.h, dropping the forward declaration (item 7). - Split CollectRelations into BuildChildMap, ShouldReportRte, and AppendRelationEntry (item 8). Addresses jsc0218's 2026-11-17 review comments.
| // matrix across non-UTF-8/ICU locales; left as a documented gap for now. | ||
| char err_message[PSCH_MAX_ERR_MSG_LEN]; // Error message text (truncated if necessary) | ||
| char query[PSCH_MAX_QUERY_LEN]; // Query text, unterminated, see query_len | ||
| char relations[PSCH_MAX_RELATIONS_LEN]; // See CollectRelations in hooks.c for layout. |
There was a problem hiding this comment.
Leftover from the move: CollectRelations now lives in src/hooks/relations_collect.c, so this pointer is stale (the comment at lines 46-47 of this file was already updated). Suggested wording:
| char relations[PSCH_MAX_RELATIONS_LEN]; // See CollectRelations in hooks.c for layout. | |
| char relations[PSCH_MAX_RELATIONS_LEN]; // See CollectRelations in relations_collect.c for layout. |
| unlike($names, qr/'big_mid_599'/, "subpartitioned intermediate past the old cap not reported (got: $names)"); | ||
| unlike($names, qr/'big_leaf_599'/, "leaf past the old cap not reported (got: $names)"); |
There was a problem hiding this comment.
These two messages still say "past the old cap", which refers to the earlier design (before collapsing). Now the reason these are not reported is that they are partitions collapsed into their parent. Suggested wording:
| unlike($names, qr/'big_mid_599'/, "subpartitioned intermediate past the old cap not reported (got: $names)"); | |
| unlike($names, qr/'big_leaf_599'/, "leaf past the old cap not reported (got: $names)"); | |
| unlike($names, qr/'big_mid_599'/, "subpartitioned intermediate collapsed into its parent, not reported (got: $names)"); | |
| unlike($names, qr/'big_leaf_599'/, "leaf partition collapsed into its parent, not reported (got: $names)"); |
| int psch_otel_max_block_bytes = 3 * 1024 * 1024; // 3 MiB (max: 16 MiB) | ||
| char* psch_extra_attributes = NULL; | ||
| char* psch_debug_arrow_dump_dir = NULL; | ||
| int psch_relations_max_len = 1023; // bytes, leaves room for the NUL in PSCH_MAX_RELATIONS_LEN=1024 |
There was a problem hiding this comment.
The "room for the NUL" rationale doesn't apply to the relations blob
The relations blob is length-prefixed binary ([ns_len][ns][name_len][name][relkind]...), not a C string, so nothing in it is NUL-terminated, and CopyRelations copies exactly buf.len bytes with no terminator. The full 1024 bytes of event->relations are usable, so the cap of 1023 (here, and the same remark on the max at line 425) is one byte lower than needed for that reason.
Either raise the max to PSCH_MAX_RELATIONS_LEN (and update the default, docs and README to match), or keep 1023 and reword the comment so it doesn't give a reason that isn't true. This is harmless either way, just a misleading comment.
| | `relation_namespaces` | `Array(LowCardinality(String))` | Schema name of each distinct table/view/materialized view/foreign table/partitioned table the query touched, positionally parallel to `relation_names` and `relation_kinds`. Partitions and inheritance children are reported through their parent, not individually (for example, `SELECT * FROM inh_p` lists only `inh_p`). Recorded only for queries that complete; error events have empty relation arrays. | | ||
| | `relation_names` | `Array(LowCardinality(String))` | Relation name of each distinct relation the query touched. | | ||
| | `relation_kinds` | `Array(FixedString(1))` | `pg_class.relkind` for each entry: `r` (table), `v` (view), `m` (materialized view), `f` (foreign table), `p` (partitioned table), etc. | | ||
| | `relations_truncated` | `Bool` | `true` if the query touched more relations than fit within `pg_stat_ch.relations_max_len`; the arrays above are incomplete for that event. | |
There was a problem hiding this comment.
Small doc gap: relations_truncated is also set when the whole blob was lost because the DSA string area was out of memory (the decoder reports truncated = true with empty arrays for the PSCH_RELATIONS_OOM_LEN sentinel). As written, a reader would assume the arrays only ever lose their tail after exceeding relations_max_len. Suggested wording:
| | `relations_truncated` | `Bool` | `true` if the query touched more relations than fit within `pg_stat_ch.relations_max_len`; the arrays above are incomplete for that event. | | |
| | `relations_truncated` | `Bool` | `true` if the arrays above are incomplete for that event: the query touched more relations than fit within `pg_stat_ch.relations_max_len`, or the relations data was dropped because the shared string area was out of memory (in which case the arrays are empty). | |
| @@ -0,0 +1,19 @@ | |||
| -- Adds structured relation columns: namespace, name, and relkind kept as | |||
| -- separate parallel arrays. Collected in src/hooks/hooks.c at ExecutorEnd. | |||
There was a problem hiding this comment.
Stale pointer after the refactor: CollectRelations now lives in src/hooks/relations_collect.c (called from hooks.c at ExecutorEnd). Suggested wording:
| -- separate parallel arrays. Collected in src/hooks/hooks.c at ExecutorEnd. | |
| -- separate parallel arrays. Collected in src/hooks/relations_collect.c at ExecutorEnd. |
| char client_addr[46]; // Client IP address (INET6_ADDRSTRLEN=46) | ||
| uint8 client_addr_len; // Actual length | ||
| uint16 query_len; // Actual length of query text | ||
| uint16 relations_len; // Actual length of relations blob (0 for utility statements) |
There was a problem hiding this comment.
This comment is only true for utility statements. relations_len is also 0 for error events (which never reach ExecutorEnd), and whenever pg_stat_ch.relations_enabled is off. Suggested wording:
| uint16 relations_len; // Actual length of relations blob (0 for utility statements) | |
| uint16 relations_len; // Actual length of relations blob (0 if not recorded: feature off, utility statement, or error event) |
| "Mirrors the query-text truncation pattern: the relations blob is a fixed " | ||
| "backend-local buffer, truncated (not grown) when a query touches enough " |
There was a problem hiding this comment.
This long description is out of date after the move to ExecutorEnd: the relations blob is no longer a fixed backend-local buffer. CollectRelations builds it in a StringInfo allocated in es_query_cxt, and it is then copied into event->relations. Suggested wording:
| "Mirrors the query-text truncation pattern: the relations blob is a fixed " | |
| "backend-local buffer, truncated (not grown) when a query touches enough " | |
| "Mirrors the query-text truncation pattern: the encoded list is truncated " | |
| "(not grown) when a query touches enough " |
| // Total shmem requested from the postmaster: the shared block plus the | ||
| // interner HTAB (which ShmemInitHash carves out of the same pool). | ||
| // query-text interner HTAB (carved from the same pool by ShmemInitHash). |
There was a problem hiding this comment.
Nit, unrelated to relations: this hunk, the one at lines 193-194 and the one at lines 270-271 only reword existing comments about the query-text interner ("interner HTAB" to "query-text interner HTAB", re-wrapping "same handle"). They don't change behavior and aren't connected to this feature, so they add noise to the diff and to git blame. Suggest dropping these three hunks from the PR and keeping shmem.c to the relations changes.



Adds a per-query "relations" field: the distinct tables/views/materialized views/foreign tables/partitioned tables a query touches, schema-qualified, with the actual
pg_class.relkindcarried per entry.Collection
A new
ExecutorCheckPerms_hookwalks the resolved range table, dedupes same-relation-twice, and resolves names viaget_namespace_name/get_rel_name. Capped by a newpg_stat_ch.relations_max_lenGUC (default 1023 bytes); over-budget queries setrelations_truncatedrather than fail, same as query text already does.A nested executor invocation -- a non-inlined SQL function, PL/pgSQL, dynamic
EXECUTE, a trigger, SPI -- gets its own staging buffer (indexed by nesting depth) so it can't clobber an outer statement's already-collected relations.The feature is gated by
pg_stat_ch.relations_enabled(PGC_POSTMASTER, default off -- the ClickHouse exporter keeps its column set for the life of the bgworker, so toggling it requires a restart). The per-query budget,pg_stat_ch.relations_max_len(PGC_SUSET, default 1023 bytes), can still be changed live. A synthetic benchmark across join widths of 1/5/10/20 tables measured the added cost at +0.05us to +1.75us per query once enabled -- small enough to leave on in the config, not costly enough to ever need live disabling.Export
The relations blob rides through the existing ring buffer via a new
dsa_pointerfield (relations_dsa), same pattern as query text and error messages. Exported as real array types on all three column-interface-based paths:Array(LowCardinality(String))/Array(FixedString(1))via a newStatHCStringArray()column typelist(utf8)columnsThe legacy Arrow exporter doesn't carry relations yet as it would need a new schema column.
Schema: three arrays vs. a tuple column
Went with three parallel arrays (
relation_namespaces,relation_names,relation_kinds) plus arelations_truncatedflag, rather than a singleArray(Tuple(namespace, name, relkind))column. Benchmarked both on a 5M-row synthetic dataset (2000 distinct relations across 8 schemas, skewed access pattern, realistic per-query relation counts):Tuple writes faster and takes less space, but neither cost shows up much in practice -- the background worker eats the write cost asynchronously, and the storage gap mostly closes once ZSTD(3) is on. Reads are where this field actually gets used (querying
events_rawfor "what tables did this touch" or "what's hot"), and that's exactly where three-array wins by a meaningful margin. Kept three-array.Testing
relations_dsaenqueue/dequeue path directly)t/039_relations_export.pl): 18/18 against a live ClickHouse instance -- actual captured values, the length-prefixed encoding's edge cases (special characters, every relkind, the truncation boundary, max-length identifiers), nested-invocation handling across four nesting mechanisms, relation drop/recreate/rename, and capture through an extension invoking SPI internallyHappy to share the benchmark script/methodology if anyone wants to reproduce the schema-shape numbers.
Note
Medium Risk
Touches the hot-path
ExecutorEndhook, catalog lookups, shared-memory enqueue/dequeue, and exporter column sets gated by postmaster GUCs—errors could affect query overhead or export compatibility if misconfigured.Overview
Adds optional per-query relation tracking: when
pg_stat_ch.relations_enabledis on (restart required; schema migration first), completed queries record distinct touched relations atExecutorEndinto a compact blob bounded bypg_stat_ch.relations_max_len, withrelations_truncatedwhen the budget is exceeded.Collection deduplicates OIDs, collapses partition/inheritance children to parents, and resolves schema/name/
relkindvia catalog lookups. Events and the shared ring buffer gain a relations field (relations+relations_dsa), including OOM handling via a sentinel length.Export decodes the blob into three parallel arrays—
relation_namespaces,relation_names,relation_kinds—plusrelations_truncatedon ClickHouse (newStatHCStringArray/Array(String)), OTLP (array log attributes), and the unified Arrow exporter (list(utf8)). Columns are omitted when the feature is disabled so older ClickHouse schemas still accept inserts.Documentation, a ClickHouse migration, CI/regression (
relations), expanded Arrow e2e checks, and comprehensive TAP coverage (t/039_relations_export.pl) land with the feature.Reviewed by Cursor Bugbot for commit 6fa0cfa. Bugbot is set up for automated code reviews on this repo. Configure here.