Skip to content

Document per-query and per-principal memory limits#462

Open
puzpuzpuz wants to merge 5 commits into
mainfrom
puzpuzpuz_per_query_mem_limit
Open

Document per-query and per-principal memory limits#462
puzpuzpuz wants to merge 5 commits into
mainfrom
puzpuzpuz_per_query_mem_limit

Conversation

@puzpuzpuz

@puzpuzpuz puzpuzpuz commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Documents the new memory-limit feature across configuration, monitoring, and access control.

OSS — per-workload memory limits

  • configuration/cairo-engine.md: new "Memory limits" section covering the three reloadable workload limits (cairo.query.memory.limit.bytes, cairo.mat.view.refresh.memory.limit.bytes, cairo.wal.apply.memory.limit.bytes), breach behavior, and the best-effort coverage caveat. Cross-links added from the WAL and materialized-views config pages.
  • query/functions/meta.md: query_activity gains the memory_used and memory_limit columns.

Enterprise — per-principal memory limits

  • query/sql/acl/alter-user.md, alter-service-account.md, and a new alter-group.md (+ sidebar): SET MEMORY LIMIT { <size> | UNLIMITED }.
  • security/rbac.md: new "Memory limits" section — how limits resolve, what a per-principal limit does and does not cover, how to inspect limits, the SET MEMORY LIMIT permission, and ALTER GROUP in the command list.
  • query/sql/show.md: the memory_limit column on SHOW USERS, SHOW GROUPS and SHOW GROUPS <user>, and SHOW SERVICE ACCOUNTS and SHOW SERVICE ACCOUNTS <user | group>.

Limit resolution — override, not min. Limits resolve by strict precedence: a principal's own limit → (users only) the most-restrictive limit among its groups → the global cairo.query.memory.limit.bytes workload limit. A 0/UNLIMITED value at a level means "not set" and falls through to the next. A more specific limit, when set, fully overrides the broader one and binds even when larger, so a per-user or per-group override can raise a principal's ceiling above the workload limit, not only lower it. Service accounts never inherit group limits. (Earlier drafts of these docs described a most-restrictive "min" behavior; revised per questdb/questdb-enterprise@fb4e050 "Make per-entity memory limits override, not min".)

Corrections made after checking against the implementation

Two statements in earlier revisions of this PR were wrong, and both were verified against the Enterprise code rather than re-reasoned:

  • UPDATE was described as never picking up a per-principal limit. False for a non-WAL table. UpdateOperation.apply() passes the caller's own SqlExecutionContext to UpdateOperatorImpl.executeUpdate, which calls queryRegistry.register(...), which acquires a QUERY-workload tracker from that context's security context — so a non-WAL UPDATE is capped by SET MEMORY LIMIT. An operator sizing one from the old text would have met an unexpected OOM abort. Only the WAL path (the default table type) runs under the WAL apply job's tracker. Replaced with a "What a per-principal limit covers" section that splits the two.
  • SHOW GROUPS <user> and SHOW SERVICE ACCOUNTS <user | group> were shown without a memory_limit column, with prose saying so explicitly. They carry it — AbstractShowLinkedEntitiesCursorFactory adds it to both metadata shapes, pinned by ShowAclTest. Since these are among the results that break a positional client, the omission hid half the breaking change.

Example tables that disagreed with each other on the same entity's limit were also made consistent.

Behaviour the docs previously did not mention at all

  • COPY ... TO exports are capped by the issuing principal's limit. The export runs under that principal's own context on the query workload, so a limit must be sized for the largest single thing the principal runs, exports included — not only for its interactive queries.
  • SET MEMORY LIMIT is self-escalating. It takes no entity name, so its holder can set any principal's limit including its own; combined with override-not-min, a non-admin holding it can raise its own ceiling above cairo.query.memory.limit.bytes. Documented as a warning, together with the GRANT ALL expansion — a principal granted ALL before the upgrade does not acquire the new permission.
  • The breaking change. Five results gain a column on upgrade (SHOW USERS, SHOW GROUPS and SHOW GROUPS <user>, SHOW SERVICE ACCOUNTS and SHOW SERVICE ACCOUNTS <user | group>), plus SELECT * on sys.acl_entities. Positional clients must be updated; name-based clients are unaffected.
  • sys.acl_entities stores the value, and is readable only by the built-in admin — a principal holding DATABASE ADMIN is still denied.
  • memory_limit means different things per statement: the effective limit in SHOW USERS, the entity's own limit in SHOW GROUPS / SHOW SERVICE ACCOUNTS, since neither inherits one.
  • An external (SSO/OIDC) user's inherited group limit refreshes at next login, not on its current session — the same refresh-on-login model that already governs group-granted permissions for external users.

Dependencies

Documents:

Web-console syntax highlighting for the new clause: questdb/sql-parser#27

Verification

yarn build, which this repo configures to throw on broken links, broken markdown links, and broken anchors.

@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown

🚀 Build success!

Latest successful preview: https://preview-462--questdb-documentation.netlify.app/docs/

Commit SHA: 1d8fbfc

📦 Build generates a preview & updates link on each commit.

puzpuzpuz and others added 4 commits June 5, 2026 16:39
- show.md: SHOW SERVICE ACCOUNTS <user|group> returns name, grant_option and
  SHOW GROUPS <user> returns name, external_alias; only the bare listings carry
  the new memory_limit column
- meta.md: query_activity example projects memory_used/memory_limit
- rbac.md: alphabetize ALTER entries in the SQL commands reference
- alter-user.md: add the --- separator before Syntax for parity with siblings
- capacity-planning.md: link the RAM section to the memory limits docs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror questdb/questdb-enterprise@fb4e050: the effective per-principal
memory limit resolves by strict precedence (user's own limit -> group
limit -> global workload limit) rather than the most-restrictive value.
A more specific limit, when set, fully overrides the broader one and
binds even when larger, so a per-entity override can raise a principal's
ceiling above the workload limit, not only lower it. Service accounts
never inherit group limits.

Updated rbac.md, alter-user/group/service-account.md, show.md, meta.md,
and cairo-engine.md accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mory limit

Reviewer note on questdb-enterprise#1039: the memory-limits section named
only mat-view refresh and WAL apply as running under internal (non-principal)
contexts, but UPDATE applied through the WAL runs under the allow-all root
context too (OperationExecutor.getRootContext, reports limit 0), so a large
UPDATE is not capped by a SET MEMORY LIMIT override. Name UPDATE explicitly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the gaps

Three corrections, each checked against the Enterprise implementation rather
than inferred:

- `UPDATE` was described as never picking up a per-principal limit. That is
  false for a non-WAL table: `UpdateOperation.apply()` passes the caller's own
  SqlExecutionContext to `UpdateOperatorImpl.executeUpdate`, which calls
  `queryRegistry.register(...)`, which acquires a QUERY tracker from that
  context's security context. So a non-WAL UPDATE IS capped, and an operator
  sizing one from the old text would meet an unexpected OOM abort. Only the WAL
  path -- the default table type -- runs under the WAL apply job's tracker.
  Replaced with a section that splits what a per-principal limit covers from
  what it does not.

- `SHOW GROUPS <user>` and `SHOW SERVICE ACCOUNTS <user | group>` were shown
  without a `memory_limit` column, and the prose said so explicitly. They carry
  it: AbstractShowLinkedEntitiesCursorFactory adds it to both GROUP_METADATA and
  METADATA, pinned by ShowAclTest. Since these are exactly the results that
  break a positional client, the omission hid half the breaking change.

- The example tables disagreed with each other on the same entity's limit; made
  them consistent.

Documents four things the page did not mention at all:

- `COPY ... TO` exports are capped by the issuing principal's limit, so a limit
  must be sized for exports and not only for interactive queries.
- `SET MEMORY LIMIT` is self-escalating. It takes no entity name, so its holder
  can set any principal's limit including its own, and a per-entity limit
  overrides the workload limit rather than tightening it -- a non-admin holding
  it can raise its own ceiling above `cairo.query.memory.limit.bytes`. Plus the
  `GRANT ALL` expansion, which means an upgrade does not confer it.
- The breaking change: five results gain a column on upgrade, and positional
  clients must be updated.
- `sys.acl_entities` stores the value, readable only by the built-in admin.

Also notes that `memory_limit` means the effective limit in `SHOW USERS` but the
entity's own limit in `SHOW GROUPS` / `SHOW SERVICE ACCOUNTS`, and that an
external user's inherited limit refreshes at next login rather than live.

Verified with `yarn build`, which is configured to throw on broken links and
anchors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant