Skip to content

A GROUP BY of a timestamptz's unit groups by a stored 15-minute bucket first (#1146) - #1149

Merged
tagspace merged 11 commits into
mainfrom
issue-1146-time-bucket
Nov 18, 2026
Merged

tagspace merged 11 commits into
mainfrom
issue-1146-time-bucket

Conversation

@tagspace

@tagspace tagspace commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

From Claude (Opus 5.5)

Closes #1146

What it does. Beside each timestamptz column, a synced table now holds an INTEGER: the value's 15 minutes of UTC. The first sync, the poll, the stream (whole rows and carried updates) and compaction each write it. A GROUP BY of the column's hour, day, week, month, quarter or year, planned in a zone other than UTC, groups by the bucket first. It works out the local unit once a bucket, instead of once a row (src/duck/bucket-group.ts). On a table every row of which its bucket decides, a cold query reads the bucket and not the time. Nothing is refused.

How the 1980 and infinity guard holds without going stale (round 2, the owner's decision)

  • A value no bucket decides (before 1980, from year 10000, ±infinity) gets the bucket BUCKET_NONE (-1). A NULL stays NULL. Every write path computes it from the row, so the write that adds such a row marks it in its own transaction.
  • The inner query is a UNION ALL. One branch takes the rows whose bucket is not -1 and reads only the bucket. The other takes the rows whose bucket is -1 and reads the time.
  • On a clean table, the engine's own statistics of the bucket rule out -1, and the engine plans that branch as an empty result. A join beside it reads one vector and stops. The plan's tree shows it (tests/fixtures/duck-plans.json).
  • The statistics are read as the statement is planned, in the transaction that runs it. I checked on the engine that a statement prepared before a -1 row committed, and run after it, still counts the row. Nothing is recorded beside the rows, and nothing a cached plan holds can go stale.
  • This differs from the suggested ledger flag read through getvariable. A session variable is set before the statement and folded in at bind, outside the statement's transaction. A write committed in between would meet a flag still saying "clean". The bucket's own statistics have no such gap.
  • A table that holds such a row reads the time only in the row groups whose bucket's minimum is -1. A deleted row leaves the statistics wide until a compaction rewrites the table. That costs speed, never an answer.
  • Both branches read the FROM and the WHERE. So a statement whose FROM or WHERE could answer twice differently keeps today's plan: a volatile call, a LIMIT or OFFSET, a sample, a LATERAL item, a function in the FROM, or a WITH query with any of these.

Other decisions

  • Six zones changed offset off a 15-minute boundary since 1980, contrary to the issue's premise: St. John's (with CNT and Canada/Newfoundland) and Goose Bay until 2011, Moncton until 2006, Gaza and Hebron in 2010 and 2011, and Casey from 2020 to 2022. tests/duck-bucket-zones.test.ts asserts this exact list over every zone the engine knows. The session's zone is folded in at bind, and in these zones every row is truncated from its time.
  • A fixed-offset session zone (the issue's 17 minutes) cannot be set. Mason refuses it with 22023, as main does.
  • Not planned in UTC, where today's arithmetic is as fast. Both plans are exact in any zone.
  • One column's bucket a statement. Every other key is one inner key computed per row. The two-column shape was 10% slower at 5 M rows (24 rows a bucket). At 50 M rows (240 a bucket) it is 44% faster, so it stays.
  • A float's sum or average keeps its plan, because last digits can move.
  • timestamp without time zone gets no bucket. Its truncation never reads the session zone.
  • A table synced before has no bucket, and keeps today's plan until it is synced again whole.

Measured on a laptop against main (5026d3d). The source has 50 M orders with two timestamptz columns, joined to 2.5 M customers, times in order with their own microseconds. Warm, over the Postgres API, min of 2 alternating runs. "Read cold" is the statement's SQL in a fresh engine instance over the file, with the engine's buffer cold: the profiler's total_bytes_read, then its time. The page cache was not cleared.

Query Zone 1 client, main → head 10 clients, main → head read cold, main → head
Revenue by month Asia/Kathmandu 456 → 60 ms 5,092 → 594 ms 182.7 → 44.6 MB, 533 → 62 ms
Orders by day Asia/Kathmandu 456 → 76 ms 4,424 → 607 ms 182.7 → 44.6 MB, 501 → 60 ms
Revenue by month joined to customers (the cold benchmark's) Asia/Kathmandu 1,185 → 775 ms 12,256 → 7,875 ms 336.9 → 212.1 MB, 1,275 → 832 ms
Month by status and ship week (two columns) Asia/Kathmandu 1,000 → 559 ms 10,014 → 5,969 ms 346.6 → 247.7 MB, 1,021 → 564 ms
Revenue by month UTC 126 → 134 ms 1,244 → 1,397 ms 182.7 → 185.6 MB, 145 → 142 ms
Revenue by month joined to customers UTC 565 → 497 ms 4,844 → 4,523 ms 336.9 → 353.1 MB, 499 → 553 ms
  • In UTC both send the engine the same SQL. Timed alone over the two files, alternating, revenue by month took 132 ms against 126 with one connection, and 1,435 against 1,435 with ten. The table's UTC differences are noise.
  • First sync: 12.04 s against 11.27 s (+7%). Synced data: 584.9 MB against 557.1 MB (+5%). With rows in random time order at 5 M rows, the worst case, it was +35%.
  • A query that also filters on the time (WHERE created_at >= …) reads the time for its own filter.

The benchmark's customer's year (not built). The joined query also groups by date_trunc('year', c.created_at), which is worked out per joined row after the join. By hand on the head's 50 M file, Kathmandu, warm, same answers:

Plan Time
main 1,036 ms
this PR 665 ms
the year worked out once per customer before the join 293 ms

So moving a second column's unit below the join would take the query to about a third of main's time. That is a change of its own.

Tested

  • tests/duck-time-bucket.test.ts holds every unit and every aggregate to Postgres (PGlite). It covers UTC, New York, Kolkata, Kathmandu, Lord Howe, Chatham, St. John's and Casey, with rows at and 1 µs either side of every change and month start, and rows before 1980, infinite and NULL.
  • The same file checks the engine's plan on a clean table: it reads the bucket and never the time, through a join and through a WHERE that is already an AND. In St. John's, and on a table of rows no bucket decides, the time is read.
  • A row no bucket decides, written by insert and by update after the same statement was planned and cached, is answered as Postgres answers it. A plan whose second branch is always empty fails this test.
  • tests/duck-bucket-zones.test.ts checks the zones, the engine's aggregates and the bucket's values. tests/integration/duck-time-bucket.itest.ts checks the bucket through the first sync, a poll, compaction and the stream, with Kathmandu answers equal to psql.
  • scripts/fails-without.ts PROVED each new test against e971531, and the first round's tests against 6889195.
  • The local oracle subset (issue 1146|date_trunc|histogram|time literals grouped) gave 2165 statements, 0 differ.
  • A zone sweep of 230 statements in 10 zones, including LEFT JOIN, WHERE subquery, LIMIT subquery, WITH and alias shapes, was 230 of 230 equal to Postgres at head and at main.
  • The sync, copy, wire, corpus, plans, gaps and conventions tests pass locally.

Left out: the integer join key (#1144), and the per-customer unit above.

🤖 Generated with Claude Code

tagspace and others added 5 commits November 18, 2026 12:53
…t first (WIP, #1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… boundary, and a float's sum keeps its plan (WIP, #1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mn's a statement, every other key whole (#1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se note (#1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tagspace
tagspace marked this pull request as ready for review November 18, 2026 14:07
@tagspace
tagspace marked this pull request as draft November 18, 2026 15:08
tagspace and others added 2 commits November 18, 2026 16:21
…ket decides is marked, and the engine's statistics leave its branch out (#1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… statistics as the flag, the two-column shape kept (#1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tagspace
tagspace marked this pull request as ready for review November 18, 2026 16:04
tagspace and others added 4 commits November 18, 2026 17:21
# Conflicts:
#	release-notes/next.md
#	src/duck/sync.ts
… and the inner keys are the GROUP BY's alone (#1146)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A key that reads no column (a literal, a parameter, a cast of either) was
matched by its form, so every equal literal in the list, the HAVING and the
ORDER BY took its inner column's place: a type modifier, EXTRACT's field, an
interval's quantity. Such a statement now keeps today's plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tagspace
tagspace merged commit 686f243 into main Nov 18, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A group by month, week, day or hour of a timestamp groups by a stored 15-minute bucket first, in every time zone

1 participant