Add QuestDB QWP support and benchmarking#28
Draft
RaphDal wants to merge 8 commits into
Draft
Conversation
QuestDB now speaks two ingestion protocols in the loader, selected with --protocol: qwp, the binary columnar QuestDB Wire Protocol over a WebSocket on port 9000, and ilp, the original hand-written line protocol text over TCP on 9009. QWP is the default; ILP stays available so runs remain comparable with the other TSBS targets, which all speak a text line protocol over a socket. The loader runs one QWP sender per worker with auto-flush off, publishes on every batch boundary and relies on a clean Close to drain outstanding server acknowledgements, so a successful run's row count is server-confirmed rather than bytes-on-a-socket. --qwp-await-ack makes every intermediate report acknowledged as well. A new generator format, questdb-qwp, writes the same points in a binary schema-and-dictionary encoding: names and symbol values are defined once and referenced by id, numbers keep their native width. The loader detects it from the file's magic, so nothing selects it by flag, and refuses to send it as ILP. Decoding a row costs 51ns against 592ns to parse the equivalent line of text, and the file is 88MB where the text is 297MB for 864k cpu-only rows. Measured on that data set with 8 workers against a local server, in rows/s the server confirmed: ILP 684k, QWP from text 1234k, QWP from binary 1867k. QWP support is not in a client release yet, so go.mod tracks the client's main branch as a pseudo-version. The compiled-binaries ignore rule matched the cmd/tsbs_* source directories too, so new files under them were invisible to git. Command directories are now re-included.
tsbs_run_queries_questdb can now send queries three ways, selected with --query-protocol: pg, the PostgreSQL wire default, http, the REST /exec endpoint, and qwp, the QuestDB Wire Protocol, which streams results back as columnar batches over the same WebSocket QWP ingestion uses. The --use-http boolean still selects http, so existing scripts keep working. The pgx path had its own copy of the parameter handling, so the two transports could have drifted apart. Both now share sqlFromQuery: arrays are inlined into the SQL, because QuestDB does not take an array bind for an IN clause, and the remaining placeholders are renumbered. Every transport therefore sends identical SQL with identical bind values, which is what makes comparing them meaningful. QWP bind indexes are zero-based, unlike the $1-based placeholders they fill. Verified against questdb/questdb:nightly over 864k rows, 200 queries per type: the rows returned by QWP match the HTTP response for all seven query types exercised, including the parameterized ones. In queries/sec QWP led five of the seven and was level with pg on the rest, with the largest margins on the heavy aggregations, where it roughly doubled HTTP. Loading that data also turned up a too-tight default: the client waits 5s for outstanding ACKs at Close, and Close is this loader's acknowledgement barrier, so the last batches of a large run can exceed it and the run fails rather than silently dropping rows. The wait is now configurable with --qwp-close-timeout-ms and defaults to 60s.
The loader only ever spoke line protocol over TCP, which is the legacy
transport: QuestDB serves it from a dedicated thread pool whose size
varies by build, so a TSBS ILP figure was measuring that pool as much as
the database. --protocol=ilp-http sends the same rows over HTTP using the
client library, so line protocol has a baseline that does not depend on
those settings. Both client-library transports now share one processor,
which also lets either of them read the binary data format.
Measured on an r8a.8xlarge, 32 vCPU, 69.1M cpu-only rows, 32 workers,
client and server on one box as in the published comparisons. Send rates:
ILP/TCP ILP/HTTP
9.4.3 release, defaults 9.3-12.5M 6.8-6.9M
9.4.4-SNAPSHOT nightly, defaults 1.7M 6.9-7.0M
nightly, 16 ILP io workers 8.0M -
ILP/TCP swings sevenfold across builds and settings while HTTP does not
move. The nightly gives the ILP/TCP pools 2 threads where every shared
pool gets 31; the release has no separate ILP pool at all. Neither the
client nor the connection count explains it: 32 connections were
established every time, and the loader parses that file at 17.3M rows/s
with --do-load=false.
Against a fairly configured server, committed throughput was QWP 6.7M,
ILP/HTTP 5.3M, ILP/TCP 5.0M rows/s. Every transport sends faster than the
write-ahead log applies, so committed rates converge and WAL apply, not
the wire, is the ceiling.
Queries went the other way from ingestion: QWP wins where result sets are
large, 2.1x HTTP and 1.4x PostgreSQL wire on a query returning 1.6M rows,
and is level with the others on aggregates returning tens of rows. Rows
returned over QWP matched the HTTP response for all 16 query types.
Two measurement traps are documented alongside the numbers. A QWP send
rate moves 25% with the ack policy alone while committed throughput moves
the other way, and ILP/TCP's fire-and-forget send rate ranks it first on
the wire and last on rows made visible. docs/questdb-qwp-benchmark.md has
the full write-up; docs/questdb.md carries the caveats next to the flags.
Close's drain default of 5s was also too tight to be the loader's
acknowledgement barrier and failed a large run outright, so it is now
configurable with --qwp-close-timeout-ms and defaults to 60s.
Two claims in the write-up were wrong and are replaced by measurements. QWP was reported as 1.3-1.7x ILP on throughput. That margin came from comparing against a build serving ILP/TCP from a 2-thread pool. Sized like the other pools, ILP/TCP reaches release-build speed, and on rows sent the two transports are level: at 4,000 hosts ILP 11.6M against QWP 11.1M, at 100,000 ILP 9.6M against QWP 11.6M. The best QWP configuration leads the best ILP one by 19% and 5% respectively, not by a multiple. Write-ahead log apply was described as the ceiling every transport converges on. It is not. Raising QDB_WAL_APPLY_WORKER_COUNT from its default of 3 to 8, 16 and 31 leaves committed throughput unchanged for every transport, and the transports do not converge: on the same table and the same apply path ILP commits at ~3.5M rows/s and QWP at ~6.5M. QWP's advantage is in rows made visible, it is 1.85x at identical configuration, and it survives the apply-thread sweep. Why the two differ upstream of apply is now the open question worth answering. The write-up also records that the nightly's ILP/TCP default costs 5.5x at 4,000 hosts and 8.7x at 100,000 against the release, that the published 8.39M and 11.36M figures both reproduce on the release build, that a QWP send rate moves 25% with the ack policy alone while committed moves the other way, and that ILP's committed rate falls as its pools grow while QWP shows no such tension. benchmark-artifacts/ holds every raw result and the harness that produced them, so each table can be traced to the run behind it.
Rows sent is the metric QuestDB publishes, so it now carries every table and the visibility metric moves to a closing note. The narrative follows the day instead of the conclusions. It opens with 4,000 hosts on stock defaults, where ILP/TCP reads 1.7M against QWP's 11.1M, then shows that figure to be the nightly's 2-thread ILP pool: sized like the other pools ILP/TCP reaches 12.9M, matching the release build. The fair comparison at 4,000 hosts follows, where plain QWP is marginally behind ILP/TCP and only per-batch acks put it ahead, by 19%. Then the other scales, where QWP leads by 10-20% at 1,000 and 100,000 hosts. Then queries, which is where QWP wins clearly and only on large result sets. The write-ahead log apply sweep is reduced to a paragraph recording that it changed nothing.
The same-host TSBS numbers hid QWP's advantage: client and server share the 32 cores, and QWP's binary encoding competes with the server for them, so the two look level on rows sent. Splitting the loader onto its own instance separates them. Over the private network both ILP transports pin at the 14.7 Gbit/s link ceiling at 5.3M rows/s, while QWP sustains 9-12M, because line protocol text is 3.4x larger on the wire. That is the result that matters and it now opens the write-up. Pre-encode replay then locates the bottleneck by taking the Go client's encoding out of the timed path. QWP tops out at three distinct ceilings, all with every row verified visible: 9.4M via the Go client over the network (client-bound), 18.5M pre-encoded over the network (network-bound, matching the wire), and 45-50M pre-encoded on localhost (the server's true ingest capacity, ~4-9x ILP). The Go client encoding is the single biggest lever for QWP ingestion. The write-up now also explains how the loader sends QWP: one sender per worker, rows built through the client's typed API and encoded per row, flushed per batch, published rather than committed by default. That per-row encoding is the client cost the replay path isolates, and ILP avoids it only because its text is already the wire format. benchmark-artifacts/ carries every raw result and the harness. The earlier split-replay files are kept for provenance but superseded by the corrected replay-localhost and replay-network files, whose committed counts were measured rather than inferred.
Folds three things that lived only in raw JSON or in conversation into the benchmark doc: - The direct-frame (pre-encode replay) send rate across cardinalities, including the 100,000-host network figure (~12M) that was only in the raw JSON. Records honestly that 4,000 hosts is the fully measured scale, 100,000 is send-only, and 1,000 was never run. Notes the finding that the server's pure-frame ingest falls with cardinality (18M at 4k to 12M at 100k) while the Go-client rate barely moves, because the client is its limit at both. - The CPU split sampled from /proc during a co-located load: the QWP loader burns ~10 of 32 cores encoding rows against ILP/TCP's ~1.7, and QWP is the most server-efficient transport per row. This was measured once and existed nowhere but the session transcript. Committed rows are deliberately out of scope for the send-rate benchmark, so the only remaining soft spot (100k direct-frame committed unverified) does not matter here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replay benchmark boundary
The replay rate is a server-capacity diagnostic, not an end-to-end client result. It includes WebSocket connection setup, network transfer, QWP server processing, and the final cumulative ACK. Binary input decoding, row building, and QWP encoding are reported separately and excluded. The loader has no local client replacement.
Validation
Repository-wide go vet ./... still reports three pre-existing warnings outside the modified loader package.