Only dedup query index matches when it can help

[brharrington]

DedupConsumer exists for :or clauses, where the same value is registered under more than one DNF term and a traversal can reach it twice. Every lookup paid for it: a DedupConsumer plus a HashSet, and on the first delivery a 16-slot table and a node.

For an index built only from :and queries none of that can ever suppress anything. Instrumenting the QueryIndexMatch workload — 5,000 subscriptions matched against 200 ids — counted 932 deliveries and 0 suppressions.

Two changes

Only wrap when it can help. add() records whether a value was registered under more than one term; a value reachable from a single term can be found once, so there is nothing to dedup.

The flag is set before the value is registered rather than after. Lookups run concurrently with updates — Evaluator.sync() adds on the config thread while update() runs on recording threads — so setting it afterwards leaves a window where a lookup finds the value under two terms and still reads the old flag, delivering it twice and double-counting that datapoint. It counts the terms in the dnf list rather than the ones that end up registered, which over-approximates and only ever costs a dedup that was not needed.

Sticky: a remove() can only make deduping unnecessary, never necessary, and recomputing would mean walking the index.

Defer the set. That still leaves the wrapper allocated for an index with any :or in it, so DedupConsumer now keeps the first value in a field and only builds the set when a second distinct value arrives. Most lookups deliver nothing or one value. A hasFirst boolean rather than a null sentinel, so a null value cannot be mistaken for an empty state.

Numbers

QueryIndexMatch with the gc profiler, against 08ac1af:

	time	alloc
no :or in the index	126.1 → 116.2 µs/op	49,953 → 3,201 B/op (−93.6%)
one :or among 5,000	126.1 → 121.5 µs/op	49,953 → 39,441 B/op (−21%)

The residual 3,201 B/op over 200 lookups is 16 B each — the benchmark's own bh::consume lambda — so the index itself allocates nothing per lookup in the first case.

The second row is the honest limit: the flag is index-wide, so a single :or subscription turns it off for everything and only the deferred set helps there. That is why both changes are here rather than just the flag.

Also

The private workers now take Consumer<T> rather than DedupConsumer<T>, since the decision is made once at the public entry, and DedupConsumer.from goes with it — the type it guarded against no longer reaches them. The forEachMatchImpl naming from #1295 is what keeps a recursive call from reaching the public method and re-wrapping.

add()'s javadoc now records that the guarantee is per call: adding the same value, or two equals values, under separate single-term calls can deliver it once for each. No in-repo caller does this, but nothing enforced it either.

Tests

Four cases, and each branch of the change has one that fails when it breaks — hard-wiring the flag off fails 4, always promoting to the set fails 3, and never suppressing fails 1.

dedupHandlesASecondValueAfterTheFirst is the one worth explaining. The carry of the first value into the set only matters when that value is delivered again after the set exists, which needs the delivery order X, Y, X. My first two attempts at it produced X, X, Y and passed with the carry deleted; the construction that works has the second query share a leaf with the :or's first branch.

Known limit, not addressed

terms > 1 counts registrations, not co-reachable leaves. Terms pinning different :eq values for the same key — name,a,:eq,name,b,:eq,:or, the shape of the existing queryNormalization test — can never co-match, yet turn the flag on for the whole index. Refining that means comparing Query.exactTags() across terms to rule out co-matching; it would widen where the win applies but is a separate change.