Repository navigation
Release v0.7 - #241
Merged
Merged
Release v0.7#241
Conversation
A column-to-column INTERSECTS join whose SELECT list held a scalar expression over qualified columns declined to the naive predicate, even though the join itself was an INNER pure-inequality join the dialect already routes through IE_JOIN. The projection shape alone cost the query its plan: a hash join on chrom with the position inequalities as a residual filter, quadratic when the chromosome key has low cardinality. That gate answers whether the projection rebuild can express a shape, and it was also deciding which join plan the query gets, though the projection takes no part in join planning. The rebuild needed nothing new. A plain aggregate already projects its argument columns inward as __giql_p<n> inner columns and re-emits the call over them on the outer wrapper SELECT; the same two steps applied to the leaves of an expression tree serve GREATEST, LEAST, arithmetic, CASE, casts and concatenation alike, because the wrapper relation carries exactly the leaf values the per-chromosome join produced. An unaliased expression is named __giql_expr_<n>, as an unaliased aggregate is named __giql_agg_<n>, since DuckDB's automatic name is derived from text the rewrite has changed. This costs two of the twelve bedtools intersect modes in the benchmark suite: the default clipped overlap span and -wo, both quadratic today. The decline from #205 is narrowed rather than reverted. A window aggregate, a FILTER clause, a scalar subquery, an aggregate nested in an expression, a bare literal and a star anywhere in the tree still fall back to the naive plan, so dialect="duckdb" stays consistent with every other backend. An out-of-scope leaf still raises: unqualified, unknown table, or a right-side reference under SEMI or ANTI, which now reaches the same diagnostic from anywhere in the tree rather than only at its root. The outer-join decomposition stays column-only, since its unmatched half would have to evaluate the expression over NULL-filled leaves from the other side.
The four tests that pinned an expression projection as a decline now assert it engages, and a.start + 1 leaves the unsupported-projection parametrizations. What replaces it there is the boundary the rebuild actually draws: each carve-out nested one level inside an otherwise rebuildable tree, since a window aggregate or a subquery reached through a GREATEST must decline as surely as one at the root; literal-only trees, which have no leaf to attribute to a side; and an expression sharing a SELECT list with a star or a window aggregate, where the older declines still win. The deep-leaf cases matter for the same reason on the error side: an unqualified or right-side column must raise from anywhere in the tree, not only where it used to sit. The execution tests cover what the property strategy cannot reach, because it generates integer arithmetic only: string concatenation, a boolean comparison, a CASE over literals and a cast. Alongside them the compositions the rebuild has to survive, each verified against the naive plan in rows and result schema: ANTI with a WHERE residual layered on the wrapper, an implicit comma join, DISTINCT over a rebuilt value, GROUP BY on an expression rather than on its leaf, LIMIT and OFFSET behind an ORDER BY alias, USING, custom column names, one-based closed coordinates where the rebuilt expression sees raw values while the join sees canonical ones, and case-folded user aliases. Four invariants are pinned that no single-shape test reaches. Both left-only shapes still plan through IE_JOIN rather than the nested loop the correlated EXISTS form exists to avoid. The unpartitioned branch above the contig ceiling carries the same projection list as the per-chromosome one. The empty-schema fallback types an expression column from the source tables under WHERE FALSE. And two queries whose inner projections coincide share one content-addressed session variable without contaminating each other's results, which is what makes that sharing safe now that the outer SELECT can differ while the inner one does not. Two property tests join the existing one. The first draws expression trees over left-only leaves under SEMI and ANTI with a WHERE residual and an ORDER BY, the path where a right-side leaf must raise instead. The second checks the clipped-span and overlap-width shapes against a Python reference rather than against the naive plan, so the bedtools semantics are pinned by something other than the implementation they are compared with.
These are the two modes the expression rebuild exists to serve, and nothing in the lane exercised them: the intersect oracle covered the join shapes whose projections are plain columns, which is every shape that already reached the fast path. Both new cases assert the dialect engaged as well as that the rows agree, because either would keep passing on the naive predicate and so would report nothing about the rewrite. The wrapper gains a clip mode for the first of them. Every existing caller selects a bedtools flag, and the default output with no flag at all -- each A feature clipped to its overlap -- had no way to be requested; the branch passes no flag and reads the result back through the same bed6 path the other flagless modes use.
The guide listed an expression among the shapes that fall back, which is now the opposite of what the dialect does, and a caller reading it would spell out a clipped overlap span expecting the naive predicate and get the fast path instead. Expression projections join the decorations the dialect absorbs, naming the bedtools modes they serve and the synthesized output name an unaliased one carries. The fallback entry keeps the shapes that still decline and now says so as a closed list rather than an open one, since the distinction between a rebuildable expression and a wrapper the rebuild cannot express is the whole of what a caller needs to predict the plan. Two boundaries are stated where they would otherwise surprise: an expression under a LEFT or RIGHT join still falls back, because the decomposition's unmatched half only NULL-fills columns; and an unqualified column inside an expression still raises, where the guide previously said every such wrapper falls back once qualified.
The LEFT-join scale test divides two wall-clock measurements and asserts the ratio stays under 2.5x. That bound has to sit between the 2.0x the decomposition should achieve and the 3.3x regression it exists to catch (#95), so the window it discriminates is narrow -- narrower than a shared runner's variance on a single sample. CI produced 2.64x for an implementation that measures 1.6x unloaded, failing the assertion without any change to the SQL either plan emits. The two plans also do not degrade alike under load. The dialect plan is a multi-statement per-chromosome UNION ALL with far more parallelism to lose than the naive plan's single hash join, so contention inflates the ratio by construction rather than symmetrically. Widening the bound is therefore not an option: it would have to cross into the band the regression occupies. Take the minimum of several runs per plan instead. Contention inflates a sample upward and never downward, so the minimum estimates the uncontended cost and the ratio stops depending on whether a stall happened to land in one measurement and not the other. The 2.5x bound is unchanged and still catches the regression. Costs about 14s on the two tests that measure ratios; the structural INTERSECT / NOT IN assertions that pin the mechanism are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-generated by the cut release workflow.