Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ jobs:
--outputdir=test/regression/results \
--temp-instance=test/regression/tmp_check \
--temp-config=test/regression/pg_stat_ch.conf \
basic version guc stats utility buffers cmd_type client_info error_capture upgrade
basic version guc stats utility buffers cmd_type client_info error_capture upgrade relations

- name: Run isolation tests
run: |
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,8 @@ SELECT * FROM pg_stat_ch_stats();
| `pg_stat_ch.clickhouse_use_tls` | bool | `off` | Restart | Enable TLS for ClickHouse connections |
| `pg_stat_ch.clickhouse_skip_tls_verify` | bool | `off` | Restart | Skip TLS certificate verification (insecure) |
| `pg_stat_ch.log_min_elevel` | enum | `warning` | Superuser | Minimum error level to capture (debug5..panic) |
| `pg_stat_ch.relations_enabled` | bool | `off` | Restart | Collect the relations array (tables/views a query touches); apply the relations schema migrations before enabling |
| `pg_stat_ch.relations_max_len` | int | `1023` | Superuser | Maximum encoded size in bytes of the per-query relations blob |

See [Error Level Values](docs/version-compatibility.md#error-fields) for the complete list of error levels and their numeric values in ClickHouse.

Expand Down
1 change: 1 addition & 0 deletions docs/guides/opentelemetry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ When `use_otel` is enabled, the ClickHouse connection parameters are ignored. Th
The OTel exporter maps pg_stat_ch events to OpenTelemetry semantic conventions:

- **Logs**: Each query execution becomes an OTel log record with attributes following the [database semantic conventions](https://opentelemetry.io/docs/specs/semconv/database/) (`db.name`, `db.user`, `db.operation.name`, `db.query.text`).
- **Relations**: tables/views touched by the query are exported as 3 separate native OTel array attributes, positionally parallel to each other: `relation_namespaces` (schema names), `relation_names` (relation names), `relation_kinds` (single-character `pg_class.relkind` per entry: `r`=table, `v`=view, `m`=matview, `f`=foreign table, `p`=partitioned table, ...). This mirrors the 3-array shape used for the ClickHouse-native export path; neither shape is final -- OTLP 1.9.0 added complex-type array attributes, and the ClickHouse schema may move to `Array(Tuple(...))`, so this representation may change.

The exporter builds OTLP log requests directly in the bgworker. pg_stat_ch's shared-memory queue already buffers events, and the exporter chunks those events into bounded gRPC requests.

Expand Down
41 changes: 41 additions & 0 deletions docs/reference/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,47 @@ Extract [sqlcommenter](https://google.github.io/sqlcommenter/) labels from query

When enabled, the background worker parses the `/* key='value' */` comment that ends each query and exports the pairs as a JSON object in the `labels` column (the `labels` attribute in OpenTelemetry). When disabled, no `labels` column is sent, so ClickHouse stores its `{}` default. Apply the `labels` schema migration before enabling it.

## Relations

### `pg_stat_ch.relations_enabled`

Collect the relations array (tables/views a query touches).

| | |
|---|---|
| **Type** | boolean |
| **Default** | `off` |
| **Context** | postmaster |

When enabled, pg_stat_ch resolves each query's touched relations (tables, views, materialized views, foreign tables, partitioned tables) and exports them as three parallel arrays -- `relation_namespaces`/`relation_names`/`relation_kinds` -- plus a `relations_truncated` flag. Toggling this off skips the catalog lookups entirely, not just the storage -- useful for isolating this feature's hot-path cost from pg_stat_ch's baseline overhead.

PGC_POSTMASTER: the ClickHouse exporter keeps its column set for the life of the bgworker, so changing this requires a restart. Apply the relations schema migrations before enabling it.

```sql
-- Enable relations collection after applying the schema migrations
ALTER SYSTEM SET pg_stat_ch.relations_enabled = on;
-- then restart PostgreSQL
```

### `pg_stat_ch.relations_max_len`

Maximum encoded size, in bytes, of the per-query relations blob.

| | |
|---|---|
| **Type** | integer |
| **Default** | `1023` |
| **Min** | `2` (just enough room for the 2-byte header) |
| **Max** | `1023` |
| **Context** | suset |

Queries touching enough distinct relations to exceed this budget are truncated: `relations_truncated` is set to `true` for that event, and the relation arrays contain only as many entries as fit. Lower this if you want to bound the per-event ClickHouse payload size further; it cannot be raised above the compiled-in maximum.

```sql
-- Lower the budget, e.g. to bound payload size more aggressively
SET pg_stat_ch.relations_max_len = 256;
```

## Queue

The shared-memory ring buffer sits between the foreground hooks and the background worker. These settings control its size and drain behavior.
Expand Down
13 changes: 13 additions & 0 deletions docs/reference/events-schema.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,19 @@ Query text is cut at 2,047 bytes, and text that reaches that length gets no labe
Label values are exported as written by the application. Don't put PII, tokens, or secrets in sqlcommenter comments.
</Note>

## Relations

| Column | Type | Description |
|---|---|---|
| `relation_namespaces` | `Array(LowCardinality(String))` | Schema name of each distinct table/view/materialized view/foreign table/partitioned table the query touched, positionally parallel to `relation_names` and `relation_kinds`. Partitions and inheritance children are reported through their parent, not individually (for example, `SELECT * FROM inh_p` lists only `inh_p`). Recorded only for queries that complete; error events have empty relation arrays. |
| `relation_names` | `Array(LowCardinality(String))` | Relation name of each distinct relation the query touched. |
| `relation_kinds` | `Array(FixedString(1))` | `pg_class.relkind` for each entry: `r` (table), `v` (view), `m` (materialized view), `f` (foreign table), `p` (partitioned table), etc. |
| `relations_truncated` | `Bool` | `true` if the query touched more relations than fit within `pg_stat_ch.relations_max_len`; the arrays above are incomplete for that event. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small doc gap: relations_truncated is also set when the whole blob was lost because the DSA string area was out of memory (the decoder reports truncated = true with empty arrays for the PSCH_RELATIONS_OOM_LEN sentinel). As written, a reader would assume the arrays only ever lose their tail after exceeding relations_max_len. Suggested wording:

Suggested change
| `relations_truncated` | `Bool` | `true` if the query touched more relations than fit within `pg_stat_ch.relations_max_len`; the arrays above are incomplete for that event. |
| `relations_truncated` | `Bool` | `true` if the arrays above are incomplete for that event: the query touched more relations than fit within `pg_stat_ch.relations_max_len`, or the relations data was dropped because the shared string area was out of memory (in which case the arrays are empty). |


<Note>
These columns require [`pg_stat_ch.relations_enabled`](/reference/configuration#pg_stat_chrelations_enabled) (off by default -- apply the relations schema migrations, then enable it). Once enabled, all three relation arrays are still empty for utility statements and for queries that touch no relations. See [`pg_stat_ch.relations_max_len`](/reference/configuration#pg_stat_chrelations_max_len) for the truncation budget. `events_recent_1h` keeps the columns it was created with, so it has no relation columns.
</Note>

## Shared buffer usage

These columns show how queries interact with PostgreSQL's `shared_buffers` cache. The cache hit ratio for a query is `shared_blks_hit / (shared_blks_hit + shared_blks_read)`. Target above 99% for OLTP workloads.
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/hooks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ This hook is used purely for nesting level tracking. The actual metric capture h
1. Finalizes instrumentation (`InstrEndLoop`)
2. Computes CPU time delta from `getrusage()`
3. Extracts all metrics from `QueryDesc`
4. Builds a `PschEvent` and enqueues it to shared memory
4. If `pg_stat_ch.relations_enabled` is on, walks `query_desc->plannedstmt->rtable`, resolving each distinct relation to its schema-qualified name and `pg_class.relkind` into a length-prefixed buffer bounded by `pg_stat_ch.relations_max_len` (queries that exceed the budget are truncated and flagged via `relations_truncated`)
5. Builds a `PschEvent` and enqueues it to shared memory

## Metrics Captured from QueryDesc

Expand Down Expand Up @@ -182,6 +183,7 @@ Parallel workers execute portions of a query plan. pg_stat_ch:
│ ├─ Finalize instrumentation │
│ ├─ Compute CPU delta │
│ ├─ Extract metrics from QueryDesc │
│ ├─ Collect touched relations (schema, name, relkind) │
│ └─ Enqueue PschEvent to shared memory │
│ │
│ ProcessUtility_hook │
Expand Down
2 changes: 2 additions & 0 deletions include/config/guc.h
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ extern bool psch_use_unified_arrow_exporter;
extern int psch_otel_max_block_bytes;
extern char* psch_extra_attributes;
extern char* psch_debug_arrow_dump_dir;
extern int psch_relations_max_len;
extern bool psch_relations_enabled;

// Initialize GUC variables
void PschInitGuc(void);
Expand Down
11 changes: 11 additions & 0 deletions include/hooks/relations_budget.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#ifndef PG_STAT_CH_INCLUDE_HOOKS_RELATIONS_BUDGET_H_
#define PG_STAT_CH_INCLUDE_HOOKS_RELATIONS_BUDGET_H_

// [truncated:u8][count:u8] -- the GUC's minimum must never go below this.
#define PSCH_RELATIONS_HEADER_LEN 2

// Sentinel relations_len for "blob lost to DSA OOM" -- distinct from 0 (no
// relations) and real blob lengths (>= PSCH_RELATIONS_HEADER_LEN).
#define PSCH_RELATIONS_OOM_LEN 1

#endif // PG_STAT_CH_INCLUDE_HOOKS_RELATIONS_BUDGET_H_
19 changes: 19 additions & 0 deletions schema/migrations/20261116000001_add_relation_columns.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- Adds structured relation columns: namespace, name, and relkind kept as
-- separate parallel arrays. Collected in src/hooks/hooks.c at ExecutorEnd.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale pointer after the refactor: CollectRelations now lives in src/hooks/relations_collect.c (called from hooks.c at ExecutorEnd). Suggested wording:

Suggested change
-- separate parallel arrays. Collected in src/hooks/hooks.c at ExecutorEnd.
-- separate parallel arrays. Collected in src/hooks/relations_collect.c at ExecutorEnd.


-- +goose Up

-- +goose StatementBegin
ALTER TABLE pg_stat_ch.events_raw
ADD COLUMN IF NOT EXISTS relation_namespaces Array(LowCardinality(String)) COMMENT 'Schema name per entry, positionally parallel to relation_names/relation_kinds.',
ADD COLUMN IF NOT EXISTS relation_names Array(LowCardinality(String)) COMMENT 'Relation name per entry, positionally parallel to relation_namespaces/relation_kinds.',
ADD COLUMN IF NOT EXISTS relation_kinds Array(FixedString(1)) COMMENT 'pg_class.relkind char per entry (r=table, v=view, m=matview, f=foreign table, p=partitioned table, ...).',
ADD COLUMN IF NOT EXISTS relations_truncated Bool DEFAULT false COMMENT 'True if pg_stat_ch.relations_max_len was exceeded and this list is incomplete.';
-- +goose StatementEnd

-- +goose Down

-- Keep the columns on rollback: ClickHouse refuses to drop them while
-- events_recent_1h's SELECT * depends on them, and producers that don't send
-- them still insert.
SELECT 1;
2 changes: 1 addition & 1 deletion scripts/run-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ run_regress() {
--outputdir=test/regression/results \
--temp-instance=test/regression/tmp_check \
--temp-config=test/regression/pg_stat_ch.conf \
basic version guc stats utility buffers cmd_type client_info error_capture drop_database_barrier upgrade
basic version guc stats utility buffers cmd_type client_info error_capture drop_database_barrier upgrade relations
}

# Run TAP tests
Expand Down
32 changes: 32 additions & 0 deletions src/config/guc.c
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#include <limits.h>

#include "config/guc.h"
#include "hooks/relations_budget.h"

// GUC variable storage
bool psch_enabled = true;
Expand Down Expand Up @@ -40,6 +41,8 @@ bool psch_use_unified_arrow_exporter = false;
int psch_otel_max_block_bytes = 3 * 1024 * 1024; // 3 MiB (max: 16 MiB)
char* psch_extra_attributes = NULL;
char* psch_debug_arrow_dump_dir = NULL;
int psch_relations_max_len = 1023; // bytes, leaves room for the NUL in PSCH_MAX_RELATIONS_LEN=1024

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "room for the NUL" rationale doesn't apply to the relations blob

The relations blob is length-prefixed binary ([ns_len][ns][name_len][name][relkind]...), not a C string, so nothing in it is NUL-terminated, and CopyRelations copies exactly buf.len bytes with no terminator. The full 1024 bytes of event->relations are usable, so the cap of 1023 (here, and the same remark on the max at line 425) is one byte lower than needed for that reason.

Either raise the max to PSCH_MAX_RELATIONS_LEN (and update the default, docs and README to match), or keep 1023 and reword the comment so it doesn't give a reason that isn't true. This is harmless either way, just a misleading comment.

bool psch_relations_enabled = false;

// Log level options (matches PostgreSQL's server_message_level_options pattern)
// clang-format off
Expand Down Expand Up @@ -410,6 +413,35 @@ void PschInitGuc(void) {
0,
NULL, NULL, NULL);

DefineCustomIntVariable(
"pg_stat_ch.relations_max_len",
"Maximum bytes for the encoded list of tables/views a query touches.",
"Mirrors the query-text truncation pattern: the relations blob is a fixed "
"backend-local buffer, truncated (not grown) when a query touches enough "
Comment on lines +419 to +420

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This long description is out of date after the move to ExecutorEnd: the relations blob is no longer a fixed backend-local buffer. CollectRelations builds it in a StringInfo allocated in es_query_cxt, and it is then copied into event->relations. Suggested wording:

Suggested change
"Mirrors the query-text truncation pattern: the relations blob is a fixed "
"backend-local buffer, truncated (not grown) when a query touches enough "
"Mirrors the query-text truncation pattern: the encoded list is truncated "
"(not grown) when a query touches enough "

"distinct relations to exceed this budget.",
&psch_relations_max_len,
1023, // bootValue
PSCH_RELATIONS_HEADER_LEN, // min -- the header alone must still fit
1023, // max (PSCH_MAX_RELATIONS_LEN - 1, room for the NUL)
PGC_SUSET,
0,
NULL, NULL, NULL);

DefineCustomBoolVariable(
"pg_stat_ch.relations_enabled",
"Collect the relations array (tables/views a query touches).",
"When enabled, each backend resolves a query's touched relations from "
"its plan's range table at ExecutorEnd and exports them as the "
"relation_namespaces/relation_names/relation_kinds/relations_truncated "
"columns. Apply the relations schema migrations before enabling. "
"PGC_POSTMASTER: the ClickHouse exporter keeps its column set for the "
"life of the bgworker, so a runtime change would leave a stale column.",
&psch_relations_enabled,
false,
PGC_POSTMASTER,
Comment thread
AliakseiAndryiashau2026 marked this conversation as resolved.
Resolved
0,
NULL, NULL, NULL);

DefineCustomBoolVariable(
"pg_stat_ch.track_labels",
"Extract sqlcommenter labels from query comments.",
Expand Down
38 changes: 38 additions & 0 deletions src/export/clickhouse_exporter.cc
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,9 @@ class ClickHouseExporter : public StatsExporter {
shared_ptr<Column<uint64_t>> StatHCUInt64(string_view name) final {
return MakeCol<FixedCol<uint64_t>>(name, "UInt64");
}
shared_ptr<Column<std::vector<string_view>>> StatHCStringArray(string_view name) final {
return MakeCol<ArrayStringCol>(name);
}

shared_ptr<Column<int64_t>> StatTimestamp(string_view name) final {
return MakeCol<FixedCol<int64_t>>(name, "DateTime64(6)");
Expand Down Expand Up @@ -263,6 +266,41 @@ class ClickHouseExporter : public StatsExporter {
chc_column col_{};
};

// Array(String) column: same flattened-bytes approach as StringCol, plus
// row_offsets_ tracking each row's cumulative element count.
class ArrayStringCol : public Column<std::vector<string_view>>, public ChColumn {
public:
ArrayStringCol(ClickHouseExporter* exp, string_view name) : exp_(exp), name_(name) {}
void Append(const std::vector<string_view>& arr) final {
for (const auto& s : arr) {
bytes_.insert(bytes_.end(), s.data(), s.data() + s.size());
value_offsets_.push_back(bytes_.size());
}
row_offsets_.push_back(value_offsets_.size());
}
bool Crunch() final {
values_col_ = chc_build_string(value_offsets_.data(),
reinterpret_cast<const uint8_t*>(bytes_.data()),
value_offsets_.size());
col_ = chc_build_array(row_offsets_.data(), row_offsets_.size(), &values_col_);
return exp_->AppendColumn(name_, "Array(String)", &col_);
}
void Clear() final {
bytes_.clear();
value_offsets_.clear();
row_offsets_.clear();
}

private:
ClickHouseExporter* const exp_;
const std::string name_;
std::vector<char> bytes_;
std::vector<uint64_t> value_offsets_;
std::vector<uint64_t> row_offsets_;
chc_column values_col_{};
chc_column col_{};
};

template <typename ColT, typename... Args>
shared_ptr<ColT> MakeCol(string_view name, Args&&... args) {
if (auto it = col_index_.find(name); it != col_index_.end()) {
Expand Down
18 changes: 18 additions & 0 deletions src/export/exporter_interface.h
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#include <memory>
#include <string>
#include <string_view>
#include <vector>

class StatsExporter {
protected:
Expand All @@ -21,6 +22,15 @@ class StatsExporter {
virtual ~Column() = default;
};

// No-op column: the default StatHCStringArray() implementation below
// returns this so callers never need a null check, matching the "not every
// backend implements every column" pattern SendArrowBatch already uses.
template <typename T>
class NoopColumn : public Column<T> {
public:
void Append(const T& /*t*/) final {}
};

public:
// ===========================================================================
// Cardinality-typed columns.
Expand Down Expand Up @@ -55,6 +65,14 @@ class StatsExporter {
virtual shared_ptr<Column<int64_t>> StatHCInt64(string_view name) = 0;
virtual shared_ptr<Column<uint64_t>> StatHCUInt64(string_view name) = 0;

// Array-of-strings column. Non-pure with a no-op default so a backend
// that doesn't implement it yet simply discards, rather than a pure
// virtual forcing every backend to handle it explicitly.
virtual shared_ptr<Column<std::vector<string_view>>> StatHCStringArray(string_view name) {
(void)name;
return std::make_shared<NoopColumn<std::vector<string_view>>>();
}

// Domain-specific. Caller appends a Unix-epoch microsecond timestamp.
// (PG-epoch values must be offset by kPostgresEpochOffsetUs before append;
// CH DateTime64(6) and OTel time_unix_nano both interpret the wire value
Expand Down
48 changes: 48 additions & 0 deletions src/export/otel_arrow_exporter.cc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ extern "C" {

#include <arrow/array/builder_binary.h>
#include <arrow/array/builder_dict.h>
#include <arrow/array/builder_nested.h>
#include <arrow/array/builder_primitive.h>
#include <arrow/io/memory.h>
#include <arrow/ipc/writer.h>
Expand Down Expand Up @@ -180,6 +181,9 @@ class OTelArrowExporter : public StatsExporter {
shared_ptr<Column<uint64_t>> StatHCUInt64(string_view n) final {
return MakeNum<uint64_t, arrow::UInt64Builder>(n, arrow::uint64());
}
shared_ptr<Column<std::vector<string_view>>> StatHCStringArray(string_view n) final {
return MakeListUtf8Sv(n);
}

shared_ptr<Column<int64_t>> StatTimestamp(string_view n) final { return MakeTimestamp(n); }

Expand Down Expand Up @@ -301,6 +305,39 @@ class OTelArrowExporter : public StatsExporter {
size_t* const bytes_;
};

// list(utf8) column for StatHCStringArray -- mirrors the OTelExporter's
// array_value log attribute (used for relation_namespaces/relation_names/
// relation_kinds, e.g. relation_names -> ["foo1","foo3"], namespace kept
// in the parallel relation_namespaces array, not inlined here), just
// built as an Arrow list instead of an OTLP AnyValue.array_value.
class ArrowListUtf8SvColumn : public Column<std::vector<std::string_view>> {
public:
ArrowListUtf8SvColumn(arrow::ListBuilder* builder, arrow::StringBuilder* value_builder,
size_t* bytes)
: builder_(builder), value_builder_(value_builder), bytes_(bytes) {}
void Append(const std::vector<std::string_view>& v) final {
arrow::Status s = builder_->Append();
if (!s.ok()) {
LogArrowFailure("Arrow list append", s);
return;
}
*bytes_ += kVarLenOffsetBytes;
for (const auto& elem : v) {
s = value_builder_->Append(elem.data(), static_cast<int32_t>(elem.size()));
if (!s.ok()) {
LogArrowFailure("Arrow list element append", s);
return;
}
*bytes_ += elem.size() + kVarLenOffsetBytes;
}
}

private:
arrow::ListBuilder* const builder_;
arrow::StringBuilder* const value_builder_;
size_t* const bytes_;
};

class ArrowTimestampColumn : public Column<int64_t> {
public:
ArrowTimestampColumn(arrow::TimestampBuilder* builder, size_t* bytes)
Expand Down Expand Up @@ -351,6 +388,17 @@ class OTelArrowExporter : public StatsExporter {
{std::string(name), arrow::field(std::string(name), arrow::utf8()), std::move(builder)});
return std::make_shared<ArrowUtf8SvColumn>(raw, &bytes_estimate_);
}
shared_ptr<Column<std::vector<string_view>>> MakeListUtf8Sv(string_view name) {
auto value_builder = std::make_shared<arrow::StringBuilder>();
auto* raw_value = value_builder.get();
auto builder =
std::make_shared<arrow::ListBuilder>(arrow::default_memory_pool(), value_builder);
Comment thread
AliakseiAndryiashau2026 marked this conversation as resolved.
Resolved
auto* raw = builder.get();
slots_.push_back({std::string(name),
arrow::field(std::string(name), arrow::list(arrow::utf8())),
std::move(builder)});
return std::make_shared<ArrowListUtf8SvColumn>(raw, raw_value, &bytes_estimate_);
}
shared_ptr<Column<int64_t>> MakeTimestamp(string_view name) {
auto builder =
std::make_shared<arrow::TimestampBuilder>(TimestampType(), arrow::default_memory_pool());
Expand Down
Loading