From 06277be5065e6e6a603790424615a8509de690e4 Mon Sep 17 00:00:00 2001 From: Conrad Date: Fri, 4 Sep 2026 09:25:00 -0400 Subject: [PATCH 1/3] perf: Rewrite a strand equality as an inequality pair to reach IE_JOIN DuckDB plans any join with an equality among its conditions as a HASH_JOIN on that key and demotes the position inequalities to a residual filter. Inside a per-chromosome subquery a strand key has two values, so each hash bucket is half the partition and the join is quadratic again: the bedtools intersect -s shape, whose residual is a.strand = b.strand, took 45.6s at 1e6 intervals per side single-threaded against 0.86s once the join reaches IE_JOIN. IE_JOIN needs no equality among the join conditions, sorts on the first two inequalities in ON order, and evaluates the rest as a per-candidate filter. The pair is the equality spelled without the equals sign: over a totally ordered domain x = y holds exactly when x >= y and x <= y both hold, strand values are strings DuckDB orders by collation, and NULL is NULL in both forms. A strand equality, spelled as =, NOT (<>), or a single-element IN, whose two operands are bare columns naming each side's strand column is emitted as that pair after the two positional overlap predicates. The order carries the result: the strand pair placed first makes the join sort on strand and enumerate nearly every pair (10.4s against 0.17s at 2e5 rows). The rewrite stands down when any other cross-side equality remains among the residuals, because the join would hash on that key regardless. The rewrite is gated on the strand column because it is the one cardinality signal the transpiler has without statistics. A key more selective than the positional overlap prunes more pairs as a hash key than the pair does as a filter: a 10,000-value key over a dense span runs in 0.32s as a hash join and 7.1s as a pair. Every other equality is emitted as written, which also keeps expression, quantified, volatile, and cross-type operands out of the duplicated pair; the per-key choice belongs to the cost-based optimizer (#67). The pair is emitted in every branch, including the unpartitioned query the chromosome-count ceiling selects, so a query binds the same way regardless of its chromosome count. There the chromosome hash join carries the pair as a residual filter at roughly 1.8x the cost of the composite key (INNER 1.48s against 2.71s at 300 chromosomes and 1e6 intervals per side). Strand columns of unlike types bind under the equality and raise DuckDB's binder error under the pair; an operand that is not a bare column is emitted as written, which is the escape hatch. Widening the partition key to (chrom, strand), the alternative the issue proposed, was measured and rejected: its 48 branches each full-scan both tables, and the widened plan ran 1.3 to 1.6x slower than the inequality reference single-threaded. The rewrite keeps the 24 chromosome branches and the shared per-chromosome scaffolding. --- src/giql/expanders/intersects_duckdb.py | 186 +++++++++++++++++++++--- src/giql/table.py | 4 +- 2 files changed, 171 insertions(+), 19 deletions(-) diff --git a/src/giql/expanders/intersects_duckdb.py b/src/giql/expanders/intersects_duckdb.py index 2ed56b4..aa04a27 100644 --- a/src/giql/expanders/intersects_duckdb.py +++ b/src/giql/expanders/intersects_duckdb.py @@ -41,6 +41,7 @@ from giql.constants import DEFAULT_CHROM_COL from giql.constants import DEFAULT_END_COL from giql.constants import DEFAULT_START_COL +from giql.constants import DEFAULT_STRAND_COL from giql.dialect import GIQLDialect from giql.expander import REGISTRY from giql.expander import ExpansionContext @@ -504,6 +505,130 @@ def _strip_intersects( return expr +def _rewrite_strand_equalities( + residuals: list[exp.Expression], strand_cols: dict[str, str] +) -> list[exp.Expression]: + """Return copies of ``residuals`` with each strand equality as an inequality pair. + + A conjunct is rewritten when it is a top-level ``=``, ``NOT (<>)``, or + single-element ``IN`` (under ``AND`` or parentheses) whose two operands + are bare columns naming the strand column of each side, one per side. + ``strand_cols`` maps each side's alias to its strand column name; a side + with no strand column is absent from the map. No residual is rewritten + when any other cross-side equality remains among them, because the join + would hash on that key regardless. Assumes the residuals have passed + `_validate_extra_qualifiers`, so every column carries a side alias. The + caller must emit the rewritten pair *after* the two positional overlap + predicates. + + The pair is the equality spelled without ``=``: over a totally ordered + domain ``x = y`` holds exactly when ``x >= y`` and ``x <= y`` both hold, + strand values are strings DuckDB orders by collation, and NULL is NULL, + hence false, in both forms. The pair carries no implicit cast, so strand + columns of unlike types bind under the equality and raise DuckDB's binder + error under the pair; an operand that is not a bare column is emitted as + written, which is the escape hatch. See :doc:`/transpilation/performance` + for the catalogue of shapes and the measurements. + + .. rubric:: Implementation notes + + DuckDB plans a join with any equality among its conditions as a + ``HASH_JOIN`` on that key, demoting the inequalities to a residual filter; + ``IE_JOIN`` requires no equality among the join conditions, sorts on the + first two inequalities in ``ON`` order, and evaluates the rest as a filter + over each candidate pair. Position therefore decides the sort keys: the + pair placed first makes the join sort on the strand and enumerate nearly + every pair in the partition (#98). ``NOT (a <> b)`` and a single-element + ``IN`` are the two spellings DuckDB's binder normalizes to the same hash + key, so they take the same rewrite. ``IS NOT DISTINCT FROM`` is NULL-safe, + and its pure-inequality form needs an ``OR`` arm for the both-NULL case, + so it is left as written. DuckDB derives collated ordering and equality + from the same collation key, so a collated pair agrees with the collated + equality. + """ + + def strand_side(node: exp.Expression) -> str | None: + if not isinstance(node, exp.Column): + return None + alias = _fold_identifier(node.table) + strand = strand_cols.get(alias) + if strand is None or _fold_identifier(strand) != _fold_identifier(node.name): + return None + return alias + + def as_equality( + node: exp.Expression, + ) -> tuple[exp.Expression, exp.Expression] | None: + if isinstance(node, exp.EQ): + return node.this, node.expression + if isinstance(node, exp.Not): + inner = node.this + while isinstance(inner, exp.Paren): + inner = inner.this + if isinstance(inner, exp.NEQ): + return inner.this, inner.expression + if ( + isinstance(node, exp.In) + and len(node.expressions) == 1 + and not any(node.args.get(key) for key in ("query", "unnest", "field")) + ): + return node.this, node.expressions[0] + return None + + def sides_of(node: exp.Expression) -> set[str]: + return {_fold_identifier(col.table) for col in node.find_all(exp.Column)} + + def strand_pair( + node: exp.Expression, + ) -> tuple[exp.Expression, exp.Expression] | None: + operands = as_equality(node) + if operands is None: + return None + left, right = (strand_side(operand) for operand in operands) + if left is None or right is None or left == right: + return None + return operands + + def is_hash_key(node: exp.Expression) -> bool: + if isinstance(node, exp.NullSafeEQ): + operands: tuple[exp.Expression, exp.Expression] | None + operands = node.this, node.expression + else: + operands = as_equality(node) + if operands is None: + return False + left, right = (sides_of(operand) for operand in operands) + return len(left) == 1 and len(right) == 1 and left != right + + def conjuncts(node: exp.Expression) -> list[exp.Expression]: + if isinstance(node, (exp.And, exp.Paren)): + return [c for child in node.args.values() for c in conjuncts(child)] + return [node] + + def rewrite(node: exp.Expression) -> exp.Expression: + if isinstance(node, exp.And): + return exp.And(this=rewrite(node.this), expression=rewrite(node.expression)) + if isinstance(node, exp.Paren): + return exp.Paren(this=rewrite(node.this)) + operands = strand_pair(node) + if operands is None: + return node + left, right = operands + return exp.And( + this=exp.GTE(this=left, expression=right), + expression=exp.LTE(this=left.copy(), expression=right.copy()), + ) + + copies = [residual.copy() for residual in residuals] + if any( + is_hash_key(conjunct) and strand_pair(conjunct) is None + for residual in copies + for conjunct in conjuncts(residual) + ): + return copies + return [rewrite(residual) for residual in copies] + + def _count_column_intersects(query: exp.Select) -> int: """Count column-to-column :class:`Intersects` predicates anywhere in *query*. @@ -1031,6 +1156,13 @@ class IntersectsDuckDBIEJoinTransformer: (wrapped to :class:`ValueError` at the public boundary). A right-side ``b.*`` under SEMI / ANTI declines to the naive plan with every other star (which then rejects the out-of-scope right table at bind time). + - A cross-side equality on the two tables' registered strand columns is + emitted as the equivalent ``>=`` / ``<=`` pair after the overlap + predicates, in every branch, so the per-chromosome join stays on + ``IE_JOIN`` (#98); `_rewrite_strand_equalities` has the mechanics. + Strand columns of unlike types bind under the equality but not under + the pair and raise DuckDB's binder error at execution; wrapping an + operand in an expression keeps the equality as written. - ``ANTI JOIN`` partitions on the chromosome INTERSECT like INNER and SEMI, then unions one non-partitioned branch carrying the left rows whose chromosome the right table lacks. Those rows cannot match, so a @@ -1733,14 +1865,14 @@ def _extract_extra_predicates( is only sound for INNER (where ``ON overlap WHERE r`` ≡ ``ON overlap AND r``) and SEMI (a left-only ``r`` is invariant over the right side); for ANTI it inverts the anti-join for rows failing ``r`` (#200). See - :meth:`_build_sql`, which routes WHERE residuals to an outer filter for - the left-only (SEMI / ANTI) shapes. + `_build_sql`, which routes WHERE residuals to an outer filter for the + left-only (SEMI / ANTI) shapes. - Note: :func:`_strip_intersects` only descends ``exp.And``. Predicates - whose AND tree wraps the INTERSECTS in ``exp.Or`` / ``exp.Not`` / + Note: `_strip_intersects` only descends ``exp.And``. Predicates whose + AND tree wraps the INTERSECTS in ``exp.Or`` / ``exp.Not`` / ``exp.Paren`` surface here with the INTERSECTS still embedded; - :meth:`_classify_extras` then routes them to the naive-predicate plan, while - :meth:`_validate_extra_qualifiers` enforces qualifier rules for the + `_classify_extras` then routes them to the naive-predicate plan, while + `_validate_extra_qualifiers` enforces qualifier rules for the remaining inlinable residuals. """ on_residuals: list[exp.Expression] = [] @@ -1811,17 +1943,20 @@ def _build_sql( rewrite composed on top never has to recover them by splitting rendered SQL, which would break on an identifier containing ``";\\n"``. + Inline residuals pass through `_rewrite_strand_equalities` before + rendering, in the per-chromosome and the unpartitioned form alike. + Returns ``None`` when a soft-fallback condition fires (the residual of the join carries a shape the naive-predicate plan can handle but the dialect cannot inline, or a single-column USING references a column that is not both tables' ``chrom_col``). May propagate - :class:`_DeclineIEJoin` from the :meth:`_resolve_projections` projection - pre-scan (a naive-valid projection the rebuild cannot express — a - window / FILTER / subquery / aggregate-nested wrapper or a bare - literal, #204, #205); :meth:`transform_to_parts` catches it and - declines to the naive plan. Raises :class:`_UnqualifiedProjectionError` - when a user-mistake condition fires; callers translating to the public - surface wrap the raise to :class:`ValueError`. + `_DeclineIEJoin` from the `_resolve_projections` projection pre-scan + (a naive-valid projection the rebuild cannot express — a window / + FILTER / subquery / aggregate-nested wrapper or a bare literal, #204, + #205); `transform_to_parts` catches it and declines to the naive plan. + Raises `_UnqualifiedProjectionError` when a user-mistake condition + fires; callers translating to the public surface wrap the raise to + `ValueError`. """ on_residuals, where_residuals = self._extract_extra_predicates(query, intersects) all_residuals = on_residuals + where_residuals @@ -1855,6 +1990,16 @@ def _build_sql( r_chrom = r_table.chrom_col if r_table else DEFAULT_CHROM_COL r_start = r_table.start_col if r_table else DEFAULT_START_COL r_end = r_table.end_col if r_table else DEFAULT_END_COL + l_strand = l_table.strand_col if l_table else DEFAULT_STRAND_COL + r_strand = r_table.strand_col if r_table else DEFAULT_STRAND_COL + strand_cols = { + alias: strand + for alias, strand in ( + (sides.left_user_alias, l_strand), + (sides.right_user_alias, r_strand), + ) + if strand is not None + } # USING() admission check (the gate already restricted to # single-column USING). The dialect's per-chromosome partition IS @@ -1898,16 +2043,21 @@ def _build_sql( r_table_ident = self._qualified_table_ident(sides.right_table) # The overlap predicate (canonical, strict) plus any inline residuals, - # shared by both the INNER join-ON form and the SEMI / ANTI EXISTS form. + # shared by the INNER join-ON form, the SEMI / ANTI EXISTS form, and + # the unpartitioned fallback. Residuals follow the overlap predicates + # -- the IE_JOIN ordering precondition in `_rewrite_strand_equalities` + # (#98). overlap_predicates = [ f"{l_start_expr} < {r_end_expr}", f"{l_end_expr} > {r_start_expr}", ] - for residual in inline_residuals: - overlap_predicates.append( + predicate_sql = " AND ".join( + overlap_predicates + + [ self._rewrite_refs_for_per_chrom_subquery(residual, sides) - ) - predicate_sql = " AND ".join(overlap_predicates) + for residual in _rewrite_strand_equalities(inline_residuals, strand_cols) + ] + ) # The dynamic SQL builder, expressed as a SQL string-concat that # ``string_agg`` aggregates per-chromosome. Single quotes are diff --git a/src/giql/table.py b/src/giql/table.py index a0a2068..516b750 100644 --- a/src/giql/table.py +++ b/src/giql/table.py @@ -46,7 +46,9 @@ class Table: (default: "end"). strand_col : str | None The physical column name storing strand information, or None if the - table has no strand column (default: "strand"). + table has no strand column (default: "strand"). The DuckDB IEJoin + dialect treats this column as low-cardinality and emits a cross-side + equality on it as a range pair; see :doc:`/transpilation/performance`. coordinate_system : Literal["0based", "1based"] The coordinate system used for positions (default: "0based"). interval_type : Literal["half_open", "closed"] From 3eeb310299a7d05e242296668b27749b6bfbbcb3 Mon Sep 17 00:00:00 2001 From: Conrad Date: Fri, 4 Sep 2026 09:25:00 -0400 Subject: [PATCH 2/3] test: Cover strand-equality residuals on the DuckDB IEJoin path Structure tests read the dynamic SQL back out of the session variable and parse it, asserting on the conjunct AST rather than on rendered text: the join predicate is the two positional overlap predicates followed by the strand pair, on both sides of the chromosome-count ceiling; every shape the rewrite claims (=, NOT (<>), single-element IN, either operand order, parenthesized, beside an inequality) yields the pair; every shape it leaves alone (another column, an expression, one-sided, under NOT or OR, mixed operands, collated, NULL-safe, quantified, volatile, or a strand pair beside another equality) is emitted as written; the rewrite follows each table's strand_col through custom, asymmetric, case-folded, defaulted, and disabled registrations; and a LEFT JOIN or count_overlaps shape carrying a strand equality declines the dialect. Plan tests read EXPLAIN (FORMAT JSON) over the dynamic SQL at a size where DuckDB picks a range join: the INNER form plans through IE_JOIN with no HASH_JOIN for all three spellings, the SEMI and ANTI EXISTS forms avoid BLOCKWISE_NL_JOIN, in each case with the positional conditions ahead of the strand pair; a non-strand equality still plans a HASH_JOIN, a strand equality beside another equality stays in the hash key, and the unpartitioned query hashes on the chromosome with the pair as a filter. Execution tests compare the dialect against the naive plan in rows and result schema over a stranded fixture in which one strand value exists on the left only, NULL strands exist on both sides, and one chromosome exists on the left only, across INNER, SEMI and ANTI joins and sixteen residual shapes drawn from one table; a property test draws random stranded tables, shapes and join kinds and asserts the same equivalence. A cross-type equality on a non-strand column agrees with the naive plan, and strand columns of unlike types raise DuckDB's binder error on either side of the ceiling. --- tests/test_duckdb_iejoin.py | 875 ++++++++++++++++++++++++++++++++++-- 1 file changed, 843 insertions(+), 32 deletions(-) diff --git a/tests/test_duckdb_iejoin.py b/tests/test_duckdb_iejoin.py index ecd1415..97834bf 100644 --- a/tests/test_duckdb_iejoin.py +++ b/tests/test_duckdb_iejoin.py @@ -1,9 +1,11 @@ """Tests for the DuckDB IEJoin dialect path on column-to-column INTERSECTS joins.""" +import json import re import time import pytest +from sqlglot import exp from sqlglot import parse_one from giql import Table @@ -29,6 +31,56 @@ #: whose output may not reference the right side. _LEFT_ONLY_LEAVES = ("a.start", 'a."end"') +#: Equality-residual shapes the dialect must answer like the naive plan, keyed +#: by a short id. +_EQUALITY_RESIDUAL_SHAPES = { + "strand_eq": "a.strand = b.strand", + "reversed": "b.strand = a.strand", + "paren_eq": "(a.strand = b.strand)", + "not_neq": "NOT (a.strand <> b.strand)", + "in_col": "a.strand IN (b.strand)", + "eq_plus_ineq": "a.strand = b.strand AND a.score > b.score", + "both_eq": "a.strand = b.strand AND a.score = b.score", + "score_eq": "a.score = b.score", + "name_eq": "a.name = b.name", + "expr_eq": "a.score + 1 = b.score", + "one_sided": "a.strand = '+'", + "under_not": "NOT (a.strand = b.strand)", + "under_or": "(a.strand = b.strand OR a.score > 5)", + "mixed_operand": "a.start + b.start = 100", + "collated": "a.strand COLLATE NOCASE = b.strand", + "is_not_distinct": "a.strand IS NOT DISTINCT FROM b.strand", +} + +#: The shapes above that the strand rewrite claims; every other shape is +#: emitted as written. +_REWRITTEN_SHAPES = ( + "strand_eq", + "reversed", + "paren_eq", + "not_neq", + "in_col", + "eq_plus_ineq", +) + +#: Shapes the naive plan cannot run under every join kind, or whose rows are +#: not deterministic, so they are pinned on the emitted SQL only. +_STRUCTURE_ONLY_SHAPES = { + "any_eq": "a.score = ANY(list_value(b.score, b.score + 4))", + "volatile_random": "a.score = b.score + CAST(random() * 3 AS INTEGER)", + "volatile_nextval": "a.score = b.score + nextval('s')", +} + +#: The shapes emitted as written, drawn from both tables. +_KEPT_SHAPES = tuple( + k for k in _EQUALITY_RESIDUAL_SHAPES if k not in _REWRITTEN_SHAPES +) + tuple(_STRUCTURE_ONLY_SHAPES) + + +def _residual(kind: str) -> str: + """Return the residual SQL for a shape id from either shape table.""" + return {**_EQUALITY_RESIDUAL_SHAPES, **_STRUCTURE_ONLY_SHAPES}[kind] + def _make_table(conn, name: str, rows: list[tuple]) -> None: conn.execute(f""" @@ -124,33 +176,37 @@ def _make_interval_tables(conn, peak_rows: list[tuple], gene_rows: list[tuple]) conn.executemany(f"INSERT INTO {table} VALUES (?, ?, ?)", rows) -def _make_planner_scale_tables(conn, *, genes_where: str | None = None) -> None: - """Recreate 20k-row peaks/genes tables over four chromosomes. +def _make_planner_scale_tables( + conn, *, genes_where: str | None = None, strand: bool = False, chroms: int = 4 +) -> None: + """Recreate 20k-row peaks/genes tables over *chroms* chromosomes. Sized so DuckDB's planner picks a range join rather than a nested loop, which is what the ``IE_JOIN`` plan assertions read. The two strides (37 and 41, the latter offset by 17) are coprime with the 100000 modulus, so starts spread evenly and the interval sets overlap without either side being a shifted copy of the other. *genes_where* restricts genes to a - subset of the chromosomes, leaving peaks with unmatched contigs. + subset of the chromosomes, leaving peaks with unmatched contigs. *strand* + adds a ``strand`` column alternating ``+`` / ``-`` by row, so a + strand-equality residual halves each per-chromosome candidate set. + *chroms* above ``MAX_PER_CHROM_PARTITIONS`` makes the dialect take its + unpartitioned branch. """ - conn.execute("DROP TABLE IF EXISTS peaks") - conn.execute(""" - CREATE TABLE peaks AS - SELECT 'chr' || ((i % 4) + 1) AS chrom, - ((i * 37) % 100000)::INTEGER AS "start", - (((i * 37) % 100000) + 200)::INTEGER AS "end" - FROM range(20000) t(i) - """) - conn.execute("DROP TABLE IF EXISTS genes") - conn.execute(f""" - CREATE TABLE genes AS - SELECT 'chr' || ((i % 4) + 1) AS chrom, - ((i * 41 + 17) % 100000)::INTEGER AS "start", - (((i * 41 + 17) % 100000) + 200)::INTEGER AS "end" - FROM range(20000) t(i) - {f"WHERE {genes_where}" if genes_where else ""} - """) + strand_column = ", CASE WHEN i % 2 = 0 THEN '+' ELSE '-' END AS strand" + for table, stride, offset, where in ( + ("peaks", 37, 0, None), + ("genes", 41, 17, genes_where), + ): + conn.execute(f"DROP TABLE IF EXISTS {table}") + conn.execute(f""" + CREATE TABLE {table} AS + SELECT 'chr' || ((i % {chroms}) + 1) AS chrom, + ((i * {stride} + {offset}) % 100000)::INTEGER AS "start", + (((i * {stride} + {offset}) % 100000) + 200)::INTEGER AS "end" + {strand_column if strand else ""} + FROM range(20000) t(i) + {f"WHERE {where}" if where else ""} + """) def _null_key(row: tuple) -> tuple: @@ -307,23 +363,67 @@ def _interval_lists( ) -def _explain_dynamic_sql(conn, sql: str, half: int = 0) -> str: - """Return the DuckDB EXPLAIN text of the per-chromosome dynamic SQL for *sql*. - - Runs the leading ``SET VARIABLE`` statements, reads the aggregated - per-chromosome ``UNION ALL`` string back out of the session variable named by - *half*, and returns the plain EXPLAIN for it. The outer - ``query(getvariable(...))`` wrapper is opaque to EXPLAIN, so the dynamic SQL - must be planned directly. *half* selects which variable to plan: ``0`` is the - only one for a single-relation shape and the matched half of a decomposed - outer join, ``1`` is that decomposition's unmatched half. +def _dynamic_sql(conn, sql: str, half: int = 0) -> str: + """Return the dynamic SQL that the ``SET VARIABLE`` statements of *sql* build. + + Runs the leading ``SET VARIABLE`` statements and reads the aggregated + per-chromosome ``UNION ALL`` string (or the unpartitioned query, above the + partition ceiling) back out of the session variable named by *half*: ``0`` + is the only one for a single-relation shape and the matched half of a + decomposed outer join, ``1`` is that decomposition's unmatched half. """ statements = [s for s in sql.split(";\n") if s.strip()] for statement in statements[:-1]: conn.execute(statement) var_names = _dynamic_sql_variables(sql) - dynamic_sql = conn.execute(f"SELECT getvariable('{var_names[half]}')").fetchone()[0] - return conn.execute("EXPLAIN " + dynamic_sql).fetchone()[1] + return conn.execute(f"SELECT getvariable('{var_names[half]}')").fetchone()[0] + + +def _explain_dynamic_sql(conn, sql: str, half: int = 0) -> str: + """Return the DuckDB EXPLAIN text of the dynamic SQL for *sql*. + + The outer ``query(getvariable(...))`` wrapper is opaque to EXPLAIN, so the + dynamic SQL must be planned directly. + """ + return conn.execute("EXPLAIN " + _dynamic_sql(conn, sql, half)).fetchone()[1] + + +def _plan_json(conn, sql: str, half: int = 0) -> dict: + """Return the DuckDB structured EXPLAIN of the dynamic SQL for *sql*. + + The JSON form carries each join's conditions as an ordered list under + ``extra_info``, which the text form only draws; the text form stays for + the operator-presence assertions that read best as substrings. + """ + plan = conn.execute( + "EXPLAIN (FORMAT JSON) " + _dynamic_sql(conn, sql, half) + ).fetchone()[1] + return json.loads(plan)[0] + + +def _find_operator(node: dict, name: str) -> dict | None: + """Return the first operator named *name* in a structured EXPLAIN tree.""" + if node.get("name") == name: + return node + for child in node.get("children", []): + found = _find_operator(child, name) + if found is not None: + return found + return None + + +def _join_predicate(conn, sql: str) -> exp.Expression: + """Return the join predicate of the dynamic SQL for *sql* as a parsed AST. + + Reads the ``JOIN ... ON`` predicate, or the ``EXISTS`` subquery's + ``WHERE`` for the SEMI / ANTI form, so a test can assert on the conjunct + order and shape without matching rendered SQL text. + """ + tree = parse_one(_dynamic_sql(conn, sql), read="duckdb") + join = tree.find(exp.Join) + if join is not None and join.args.get("on") is not None: + return join.args["on"] + return tree.find(exp.Exists).this.args["where"].this @pytest.fixture @@ -360,6 +460,43 @@ def peaks_genes(conn): return conn +@pytest.fixture +def stranded_peaks_genes(conn): + """Build peaks / genes whose strand column exercises every equality outcome. + + Strand ``.`` appears only on the left, NULL strands appear on both sides + (and can never match each other), and chr3 appears only on the left. + """ + _make_table( + conn, + "peaks", + [ + ("chr1", 100, 200, "p1", 10, "+"), + ("chr1", 150, 250, "p2", 20, "-"), + ("chr1", 300, 400, "p3", 30, "+"), + ("chr1", 500, 600, "p4", 40, "."), + ("chr2", 100, 200, "p5", 50, "-"), + ("chr2", 700, 800, "p6", 60, None), + ("chr3", 100, 200, "p7", 70, "+"), + ], + ) + _make_table( + conn, + "genes", + [ + ("chr1", 120, 180, "g1", 5, "+"), + ("chr1", 160, 260, "g2", 21, "-"), + ("chr1", 350, 450, "g3", 30, "-"), + ("chr1", 320, 380, "g8", 30, "+"), + ("chr1", 550, 650, "g4", 45, "+"), + ("chr2", 150, 250, "g5", 55, "-"), + ("chr2", 750, 850, "g6", 65, None), + ("chr2", 780, 800, "g7", 66, "+"), + ], + ) + return conn + + # Truth table for peaks_genes overlapping pairs (half-open): # - chr1: peaks[100,200) vs genes[150,250) -> overlap # - chr1: peaks[500,600) vs genes[500,600) -> overlap @@ -3799,6 +3936,258 @@ def test_transpile_should_reject_right_side_projection_when_join_is_anti(self): transpile(query, tables=["peaks", "genes"], dialect="duckdb") assert "left-side" in str(excinfo.value) + @pytest.mark.parametrize("join_kind", ["JOIN", "SEMI JOIN"]) + def test_transpile_should_rewrite_strand_equality_as_inequality_pair_after_overlap_predicates( # noqa: E501 + self, peaks_genes, join_kind + ): + """Test that a strand-equality residual is emitted as a ``>=`` / ``<=`` pair. + + Given: + Peaks / genes and a JOIN / SEMI JOIN INTERSECTS query whose ON + carries ``a.strand = b.strand``. + When: + The query is transpiled with ``dialect='duckdb'`` and its + per-chromosome dynamic SQL is read back and parsed. + Then: + The join predicate should be the two positional overlap predicates + followed by ``a.strand >= b.strand`` and ``a.strand <= b.strand``, + with no equality left. + """ + # Arrange + sql = transpile( + f"SELECT a.start FROM peaks a {join_kind} genes b " + "ON a.interval INTERSECTS b.interval AND a.strand = b.strand", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + conjuncts = list(_join_predicate(peaks_genes, sql).flatten()) + + # Assert + assert [type(c) for c in conjuncts] == [exp.LT, exp.GT, exp.GTE, exp.LTE] + assert conjuncts[2:] == [ + parse_one("a.strand >= b.strand", read="duckdb"), + parse_one("a.strand <= b.strand", read="duckdb"), + ] + + @pytest.mark.parametrize("join_kind", ["JOIN", "SEMI JOIN"]) + def test_transpile_should_rewrite_strand_equality_when_partition_exceeds_ceiling( + self, conn, join_kind + ): + """Test that the unpartitioned fallback carries the same strand pair. + + Given: + Stranded peaks / genes over more shared chromosomes than + ``MAX_PER_CHROM_PARTITIONS`` and a JOIN / SEMI JOIN INTERSECTS + query whose ON carries ``a.strand = b.strand``. + When: + The query is transpiled with ``dialect='duckdb'`` and the query the + session variable selects is read back and parsed. + Then: + The join predicate should carry the chromosome equality, the two + positional overlap predicates, and the ``>=`` / ``<=`` strand pair, + with no strand equality left. + """ + # Arrange + _make_planner_scale_tables( + conn, strand=True, chroms=MAX_PER_CHROM_PARTITIONS + 1 + ) + sql = transpile( + f"SELECT a.start FROM peaks a {join_kind} genes b " + "ON a.interval INTERSECTS b.interval AND a.strand = b.strand", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + conjuncts = list(_join_predicate(conn, sql).flatten()) + + # Assert + assert f"> {MAX_PER_CHROM_PARTITIONS}" in sql + assert [type(c) for c in conjuncts] == [exp.EQ, exp.LT, exp.GT, exp.GTE, exp.LTE] + assert conjuncts[3:] == [ + parse_one("a.strand >= b.strand", read="duckdb"), + parse_one("a.strand <= b.strand", read="duckdb"), + ] + + @pytest.mark.parametrize("residual_kind", _KEPT_SHAPES) + def test_transpile_should_keep_equality_residual_when_not_a_strand_pair( + self, peaks_genes, residual_kind + ): + """Test that an equality other than a bare strand pair is emitted as written. + + Given: + A JOIN INTERSECTS query whose ON residual is one of the shapes the + rewrite leaves alone: an equality on a non-strand column, over an + expression, one-sided, under ``NOT`` or ``OR``, mixing both sides + in one operand, collated, NULL-safe, quantified with ``ANY``, + carrying a volatile operand, or a strand pair beside another + cross-side equality. + When: + The query is transpiled with ``dialect='duckdb'`` and its + per-chromosome dynamic SQL is read back and parsed. + Then: + The residual should follow the two positional overlap predicates + unchanged, with no ``>=`` / ``<=`` pair anywhere in the predicate. + """ + # Arrange + residual = _residual(residual_kind) + sql = transpile( + "SELECT a.start FROM peaks a JOIN genes b " + f"ON a.interval INTERSECTS b.interval AND {residual}", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + predicate = _join_predicate(peaks_genes, sql) + + # Assert + expected = parse_one(residual, read="duckdb") + conjuncts = ( + list(expected.flatten()) if isinstance(expected, exp.And) else [expected] + ) + assert [c.unnest() for c in predicate.flatten()][2:] == [ + c.unnest() for c in conjuncts + ] + assert predicate.find(exp.GTE, exp.LTE) is None + + @pytest.mark.parametrize("residual_kind", _REWRITTEN_SHAPES) + def test_transpile_should_rewrite_only_the_strand_pair_when_residual_is_compound( + self, peaks_genes, residual_kind + ): + """Test that the strand pair is rewritten wherever it sits in an AND tree. + + Given: + A JOIN INTERSECTS query whose ON residual carries a bare strand + pair -- as ``=``, ``NOT (<>)``, or a single-element ``IN``, in + either operand order, parenthesized, or beside an inequality. + When: + The query is transpiled with ``dialect='duckdb'`` and its + per-chromosome dynamic SQL is read back and parsed. + Then: + The predicate should carry a ``>=`` / ``<=`` pair over the two + strand columns and no strand equality, while any other conjunct + is left as written. + """ + # Arrange + sql = transpile( + "SELECT a.start FROM peaks a JOIN genes b " + f"ON a.interval INTERSECTS b.interval AND {_residual(residual_kind)}", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + predicate = _join_predicate(peaks_genes, sql) + + # Assert + bounds = list(predicate.find_all(exp.GTE, exp.LTE)) + assert sorted(type(b).__name__ for b in bounds) == ["GTE", "LTE"] + for bound in bounds: + assert {c.name for c in bound.find_all(exp.Column)} == {"strand"} + assert predicate.find(exp.EQ, exp.NEQ, exp.In) is None + + @pytest.mark.parametrize( + ("tables", "residual", "rewritten"), + [ + ( + [Table("peaks", strand_col="str"), Table("genes", strand_col="str")], + "a.str = b.str", + True, + ), + ( + [Table("peaks", strand_col="str"), Table("genes", strand_col="strand")], + "a.str = b.strand", + True, + ), + ( + [Table("peaks", strand_col="Strand"), "genes"], + "a.STRAND = b.strand", + True, + ), + (["peaks"], "a.strand = b.strand", True), + ( + [Table("peaks", strand_col="str"), Table("genes", strand_col="str")], + "a.strand = b.strand", + False, + ), + ([Table("peaks", strand_col=None), "genes"], "a.strand = b.strand", False), + ], + ids=[ + "registered_name", + "asymmetric_names", + "case_folded", + "unregistered_table", + "unregistered_name", + "no_strand_column", + ], + ) + def test_transpile_should_gate_the_rewrite_on_the_registered_strand_column( + self, peaks_genes, tables, residual, rewritten + ): + """Test that the rewrite follows each table's registered ``strand_col``. + + Given: + A JOIN INTERSECTS query with a cross-side equality, and table + configs whose ``strand_col`` is a custom name, differs per side, + differs in case from the column as written, is left to the + default for an unregistered table, or is ``None``. + When: + The query is transpiled with ``dialect='duckdb'`` and its + per-chromosome dynamic SQL is read back and parsed. + Then: + The equality should be rewritten only when both operands name the + strand column registered, or defaulted, for their side. + """ + # Arrange + sql = transpile( + "SELECT a.start FROM peaks a JOIN genes b " + f"ON a.interval INTERSECTS b.interval AND {residual}", + tables=tables, + dialect="duckdb", + ) + + # Act + predicate = _join_predicate(peaks_genes, sql) + + # Assert + assert (predicate.find(exp.GTE) is not None) is rewritten + assert (predicate.find(exp.EQ) is None) is rewritten + + @pytest.mark.parametrize( + "query", + [ + "SELECT a.chrom, a.start, b.start AS b_start FROM peaks a " + "LEFT JOIN genes b ON a.interval INTERSECTS b.interval " + "AND a.strand = b.strand", + "SELECT a.chrom, a.start, COUNT(b.chrom) AS n FROM peaks a " + "LEFT JOIN genes b ON a.interval INTERSECTS b.interval " + "AND a.strand = b.strand GROUP BY a.chrom, a.start", + ], + ids=["left_join", "count_overlaps"], + ) + def test_transpile_should_decline_when_outer_join_carries_strand_equality( + self, query + ): + """Test that a strand equality on an outer join declines the dialect. + + Given: + A LEFT JOIN INTERSECTS query, plain or in the count_overlaps + shape, whose ON carries ``a.strand = b.strand``. + When: + ``transpile`` is called with ``dialect='duckdb'``. + Then: + It should emit the naive plan with no dynamic-SQL scaffolding, + because the outer-join decomposition admits no ON residual. + """ + # Act + sql = transpile(query, tables=["peaks", "genes"], dialect="duckdb") + + # Assert + assert "getvariable" not in sql + class TestTranspileDuckDBIEJoinCountOverlaps: """The count_overlaps fast path: LEFT JOIN + COUNT(b.col) + GROUP BY (#209).""" @@ -9511,6 +9900,428 @@ def test_query_should_order_anti_join_with_where_residual(self, conn): assert rows_duckdb == [(900,), (700,)] assert rows_duckdb == rows_naive + @pytest.mark.parametrize("residual_kind", ["strand_eq", "not_neq", "in_col"]) + def test_inner_join_plan_should_reach_iejoin_when_residual_is_strand_equality( + self, conn, residual_kind + ): + """Test that a strand-equality residual plans through IE_JOIN, not a hash join. + + Given: + 20k stranded rows per side over four chromosomes and an INNER JOIN + INTERSECTS query whose ON carries a strand equality spelled as + ``=``, ``NOT (<>)``, or a single-element ``IN``. + When: + The ``dialect='duckdb'`` per-chromosome dynamic SQL is planned with + ``EXPLAIN``. + Then: + It should plan through ``IE_JOIN`` with no ``HASH_JOIN``, sorting on + the two positional overlap predicates and filtering on the strand + pair. + """ + # Arrange + _make_planner_scale_tables(conn, strand=True) + sql = transpile( + "SELECT a.start, b.start AS b_start FROM peaks a " + "JOIN genes b ON a.interval INTERSECTS b.interval " + f"AND {_residual(residual_kind)}", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + plan = _plan_json(conn, sql) + + # Assert + assert _find_operator(plan, "HASH_JOIN") is None + conditions = _find_operator(plan, "IE_JOIN")["extra_info"]["Conditions"] + assert set(conditions[:2]) == {"start < end", "end > start"} + assert set(conditions[2:]) == {"strand >= strand", "strand <= strand"} + + @pytest.mark.parametrize("join_kind", ["SEMI JOIN", "ANTI JOIN"]) + def test_left_only_join_plan_should_avoid_nested_loop_when_residual_is_strand_equality( # noqa: E501 + self, conn, join_kind + ): + """Test that the SEMI / ANTI EXISTS form reaches IE_JOIN with a strand equality. + + Given: + 20k stranded rows per side over four chromosomes and a SEMI / ANTI + JOIN INTERSECTS query whose ON carries ``a.strand = b.strand``. + When: + The ``dialect='duckdb'`` per-chromosome dynamic SQL is planned with + ``EXPLAIN``. + Then: + It should plan through ``IE_JOIN`` and not the quadratic + ``BLOCKWISE_NL_JOIN``, sorting on the two positional predicates and + filtering on the strand pair. + """ + # Arrange + _make_planner_scale_tables(conn, strand=True) + sql = transpile( + f"SELECT a.start FROM peaks a {join_kind} genes b " + "ON a.interval INTERSECTS b.interval AND a.strand = b.strand", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + plan = _plan_json(conn, sql) + + # Assert + assert _find_operator(plan, "BLOCKWISE_NL_JOIN") is None + conditions = _find_operator(plan, "IE_JOIN")["extra_info"]["Conditions"] + assert set(conditions[:2]) == {"start < end", "end > start"} + assert set(conditions[2:]) == {"strand >= strand", "strand <= strand"} + + def test_inner_join_plan_should_keep_hash_join_when_residual_is_another_equality( + self, peaks_genes + ): + """Test that a cross-side equality on a non-strand column stays a hash key. + + Given: + Peaks / genes and an INNER JOIN INTERSECTS query whose ON carries + ``a.name = b.name``. + When: + The ``dialect='duckdb'`` per-chromosome dynamic SQL is planned with + ``EXPLAIN``. + Then: + It should plan a ``HASH_JOIN`` keyed on the name equality, since + the rewrite is gated on the registered strand column. + """ + # Arrange + sql = transpile( + "SELECT a.start FROM peaks a JOIN genes b " + "ON a.interval INTERSECTS b.interval AND a.name = b.name", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + plan = _plan_json(peaks_genes, sql) + + # Assert + hash_join = _find_operator(plan, "HASH_JOIN") + assert "name = name" in hash_join["extra_info"]["Conditions"] + + def test_inner_join_plan_should_keep_hash_join_when_another_equality_survives( + self, conn + ): + """Test that the strand equality stays a hash key beside another equality. + + Given: + 20k stranded rows per side over four chromosomes and an INNER JOIN + INTERSECTS query whose ON carries ``a.strand = b.strand AND + a.start = b.start``. + When: + The ``dialect='duckdb'`` per-chromosome dynamic SQL is planned with + ``EXPLAIN``. + Then: + It should plan a ``HASH_JOIN`` keyed on both equalities, since the + start equality keeps the join off ``IE_JOIN`` regardless and the + rewrite stands down. + """ + # Arrange + _make_planner_scale_tables(conn, strand=True) + sql = transpile( + "SELECT a.start FROM peaks a JOIN genes b " + "ON a.interval INTERSECTS b.interval " + "AND a.strand = b.strand AND a.start = b.start", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + plan = _plan_json(conn, sql) + + # Assert + conditions = _find_operator(plan, "HASH_JOIN")["extra_info"]["Conditions"] + assert "start = start" in conditions + assert "strand = strand" in conditions + assert "strand >= strand" not in conditions + + def test_unpartitioned_plan_should_filter_on_strand_when_partition_exceeds_ceiling( + self, conn + ): + """Test that the unpartitioned fallback carries the strand pair as a filter. + + Given: + 20k stranded rows per side over more shared chromosomes than + ``MAX_PER_CHROM_PARTITIONS`` and an INNER JOIN INTERSECTS query + whose ON carries ``a.strand = b.strand``. + When: + The query the ``dialect='duckdb'`` session variable selects is + planned with ``EXPLAIN``. + Then: + It should plan a ``HASH_JOIN`` keyed on the chromosome with the + ``>=`` / ``<=`` strand pair among its conditions and no strand + equality. + """ + # Arrange + _make_planner_scale_tables( + conn, strand=True, chroms=MAX_PER_CHROM_PARTITIONS + 1 + ) + sql = transpile( + "SELECT a.start, b.start AS b_start FROM peaks a " + "JOIN genes b ON a.interval INTERSECTS b.interval " + "AND a.strand = b.strand", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act + plan = _plan_json(conn, sql) + + # Assert + assert f"> {MAX_PER_CHROM_PARTITIONS}" in sql + conditions = _find_operator(plan, "HASH_JOIN")["extra_info"]["Conditions"] + assert "chrom = chrom" in conditions + assert {"strand >= strand", "strand <= strand"} <= set(conditions) + assert "strand = strand" not in conditions + + @pytest.mark.parametrize( + ("join_kind", "expected"), + [ + ( + "SEMI JOIN", + [("chr1", 100), ("chr1", 150), ("chr1", 300), ("chr2", 100)], + ), + ("ANTI JOIN", [("chr1", 500), ("chr2", 700), ("chr3", 100)]), + ], + ids=["semi", "anti"], + ) + def test_query_should_match_strand_equality_truth_table_when_dialect_is_duckdb( + self, stranded_peaks_genes, join_kind, expected + ): + """Test the strand-equality SEMI / ANTI truth table end to end. + + Given: + Stranded peaks / genes where one strand value (``.``) exists on the + left only, NULL strands exist on both sides, and one chromosome + exists on the left only. + When: + A SEMI / ANTI JOIN INTERSECTS query with ``a.strand = b.strand`` + is transpiled with ``dialect='duckdb'`` and executed. + Then: + SEMI should return exactly the left rows with a same-strand overlap + and ANTI the rest; a left-only strand value, a NULL strand (even + against another NULL) and a left-only chromosome should all count as + unmatched, and both should agree with the naive plan. + """ + # Arrange + query = ( + f"SELECT a.chrom, a.start FROM peaks a {join_kind} genes b " + "ON a.interval INTERSECTS b.interval AND a.strand = b.strand" + ) + + # Act + sql = _assert_agrees_with_naive(stranded_peaks_genes, query) + rows = sorted(stranded_peaks_genes.execute(sql).fetchall()) + + # Assert + assert rows == expected + + @pytest.mark.parametrize("join_kind", ["JOIN", "SEMI JOIN", "ANTI JOIN"]) + @pytest.mark.parametrize("residual_kind", list(_EQUALITY_RESIDUAL_SHAPES)) + def test_query_should_match_naive_plan_for_equality_residual_shapes( + self, stranded_peaks_genes, residual_kind, join_kind + ): + """Test dialect / naive equivalence across the equality-residual shapes. + + Given: + Stranded peaks / genes with left-only and NULL strand values, and a + JOIN / SEMI JOIN / ANTI JOIN INTERSECTS query whose ON residual is + one of ``_EQUALITY_RESIDUAL_SHAPES``. + When: + The query is transpiled with and without ``dialect='duckdb'`` and + both are executed. + Then: + Both plans should return the same rows and result schema. + """ + # Arrange + query = ( + f"SELECT a.chrom, a.start FROM peaks a {join_kind} genes b " + f"ON a.interval INTERSECTS b.interval AND {_residual(residual_kind)}" + ) + + # Act & assert + _assert_agrees_with_naive(stranded_peaks_genes, query) + + def test_query_should_match_naive_plan_for_where_strand_equality_when_join_is_inner( # noqa: E501 + self, stranded_peaks_genes + ): + """Test that an INNER-join WHERE strand equality is rewritten. + + Given: + Stranded peaks / genes and an INNER JOIN INTERSECTS query whose + ``a.strand = b.strand`` sits in the top-level WHERE rather than the + ON. + When: + The query is transpiled with and without ``dialect='duckdb'`` and + both are executed. + Then: + Both plans should return the same rows, and the per-chromosome join + predicate should carry the ``>=`` / ``<=`` pair. + """ + # Arrange + query = ( + "SELECT a.chrom, a.start, b.start AS b_start FROM peaks a " + "JOIN genes b ON a.interval INTERSECTS b.interval " + "WHERE a.strand = b.strand" + ) + + # Act + sql = _assert_agrees_with_naive(stranded_peaks_genes, query) + + # Assert + predicate = _join_predicate(stranded_peaks_genes, sql) + assert predicate.find(exp.GTE) is not None + assert predicate.find(exp.EQ) is None + + def test_query_should_match_naive_plan_when_equality_operands_differ_in_type( + self, conn + ): + """Test that a kept cross-type equality on a non-strand column binds. + + Given: + Peaks whose ``gene_id`` is VARCHAR and genes whose ``gene_id`` is + INTEGER, and an INNER JOIN INTERSECTS query whose ON carries + ``a.gene_id = b.gene_id``. + When: + The query is transpiled with and without ``dialect='duckdb'`` and + both are executed. + Then: + Both plans should return the same rows, because the equality is + emitted as written and DuckDB casts implicitly for ``=``. + """ + # Arrange + conn.execute( + 'CREATE TABLE peaks (chrom VARCHAR, "start" INTEGER, "end" INTEGER, ' + "gene_id VARCHAR)" + ) + conn.execute( + 'CREATE TABLE genes (chrom VARCHAR, "start" INTEGER, "end" INTEGER, ' + "gene_id INTEGER)" + ) + conn.execute( + "INSERT INTO peaks VALUES ('chr1', 100, 200, '7'), ('chr1', 300, 400, '8')" + ) + conn.execute( + "INSERT INTO genes VALUES ('chr1', 150, 250, 7), ('chr1', 350, 450, 9)" + ) + query = ( + "SELECT a.chrom, a.start FROM peaks a JOIN genes b " + "ON a.interval INTERSECTS b.interval AND a.gene_id = b.gene_id" + ) + + # Act & assert + _assert_agrees_with_naive(conn, query) + + @pytest.mark.parametrize( + "chroms", [4, MAX_PER_CHROM_PARTITIONS + 1], ids=["partitioned", "unpartitioned"] + ) + def test_query_should_raise_binder_error_when_strand_columns_differ_in_type( + self, conn, chroms + ): + """Test that unlike strand types raise the binder error on either branch. + + Given: + Peaks whose ``strand`` is VARCHAR and genes whose ``strand`` is + INTEGER, spread over fewer or more shared chromosomes than + ``MAX_PER_CHROM_PARTITIONS``, and an INNER JOIN INTERSECTS query + whose ON carries ``a.strand = b.strand``. + When: + The query is transpiled with ``dialect='duckdb'`` and executed. + Then: + It should raise DuckDB's binder error whichever branch the session + variable selects, since the rewritten ``>=`` / ``<=`` pair has no + implicit cast between the two types and is emitted in both. + """ + # Arrange + for table, strand in (("peaks", "'1'"), ("genes", "1")): + conn.execute(f""" + CREATE TABLE {table} AS + SELECT 'chr' || (i % {chroms}) AS chrom, + (i * 37)::INTEGER AS "start", + (i * 37 + 200)::INTEGER AS "end", + {strand} AS strand + FROM range({chroms * 4}) t(i) + """) + sql = transpile( + "SELECT a.chrom, a.start FROM peaks a JOIN genes b " + "ON a.interval INTERSECTS b.interval AND a.strand = b.strand", + tables=["peaks", "genes"], + dialect="duckdb", + ) + + # Act & assert + with pytest.raises(duckdb.BinderException): + conn.execute(sql).fetchall() + + @settings( + max_examples=40, + deadline=None, + suppress_health_check=[HealthCheck.function_scoped_fixture], + ) + @given( + peaks=st.lists( + st.tuples( + st.sampled_from(["chr1", "chr2"]), + st.integers(min_value=0, max_value=100), + st.integers(min_value=1, max_value=30), + st.sampled_from(["+", "-", None]), + st.one_of(st.none(), st.integers(min_value=0, max_value=3)), + ), + min_size=0, + max_size=6, + ), + genes=st.lists( + st.tuples( + st.sampled_from(["chr1", "chr2"]), + st.integers(min_value=0, max_value=100), + st.integers(min_value=1, max_value=30), + st.sampled_from(["+", "-", None]), + st.one_of(st.none(), st.integers(min_value=0, max_value=3)), + ), + min_size=0, + max_size=6, + ), + residual_kind=st.sampled_from(sorted(_EQUALITY_RESIDUAL_SHAPES)), + join_kind=st.sampled_from(["JOIN", "SEMI JOIN", "ANTI JOIN"]), + ) + def test_query_should_match_naive_plan_for_random_equality_residuals( + self, conn, peaks, genes, residual_kind, join_kind + ): + """Test cross-plan equivalence under random equality residuals. + + Given: + Hypothesis-generated peak / gene interval lists with (possibly + NULL) strand and score columns, a residual sampled from + ``_EQUALITY_RESIDUAL_SHAPES``, and a sampled INNER / SEMI / ANTI + join. + When: + The composed query is transpiled under both ``dialect=None`` and + ``dialect="duckdb"`` and executed. + Then: + Both plans should return the same rows and result schema. + """ + # Arrange + peak_rows = [ + (c, s, s + n, None, score, strand) for c, s, n, strand, score in peaks + ] + gene_rows = [ + (c, s, s + n, None, score, strand) for c, s, n, strand, score in genes + ] + conn.execute("DROP TABLE IF EXISTS peaks") + conn.execute("DROP TABLE IF EXISTS genes") + _make_table(conn, "peaks", peak_rows) + _make_table(conn, "genes", gene_rows) + query = ( + f'SELECT a.chrom, a.start, a."end" FROM peaks a {join_kind} genes b ' + f"ON a.interval INTERSECTS b.interval AND {_residual(residual_kind)}" + ) + + # Act & assert + _assert_agrees_with_naive(conn, query) + class TestTranspileDuckDBIEJoinKwargs: """Tests for the ``dialect`` kwarg surface. From e98bbd1b3f823c955d8807d63b02d3811f37159e Mon Sep 17 00:00:00 2001 From: Conrad Date: Fri, 4 Sep 2026 09:25:01 -0400 Subject: [PATCH 3/3] docs: Describe the strand-equality rewrite on the IEJoin path The performance guide gains a bullet explaining why a cross-side equality would force a hash join inside each per-chromosome subquery, what the >= / <= pair means and why it keeps the join on IE_JOIN, that the pair is emitted in every branch and what it costs above the partition ceiling, which shapes are rewritten and which are emitted as written, how the strand column is registered, the bind-time failure on strand columns of unlike types and its escape hatch, the INNER / SEMI / ANTI boundary, why the gate is the strand column, and the measurements on DuckDB 1.5.5. The dialect's two-bucket taxonomy names the bind-time case as the one supported shape outside both buckets. The bedtools strand flag links to the migration recipe instead of restating its predicates. --- docs/transpilation/performance.rst | 79 ++++++++++++++++++++++++++++-- 1 file changed, 76 insertions(+), 3 deletions(-) diff --git a/docs/transpilation/performance.rst b/docs/transpilation/performance.rst index 65e9911..3a5533e 100644 --- a/docs/transpilation/performance.rst +++ b/docs/transpilation/performance.rst @@ -556,7 +556,7 @@ them): ``RIGHT`` outer join an expression projection still declines, since the decomposition's unmatched half only NULL-fills columns. - **Extra JOIN / WHERE predicates.** Additional non-INTERSECTS - predicates ANDed onto the join ON or WHERE (e.g. ``a.score > + predicates ANDed onto the join ON or WHERE (e.g., ``a.score > b.score`` or ``WHERE a.score > 100``) are inlined into each per-chromosome subquery's ON, so DuckDB filters them inside each IEJoin candidate set. **Limitations:** the dialect peels extra @@ -572,6 +572,75 @@ them): ``dialect=None`` path, ``dialect="duckdb"`` is one workaround — the dialect inlines extras directly into each per-chromosome subquery and is unaffected by that bug. +- **Strand-equality residuals.** A cross-side equality inlined into a + per-chromosome subquery's ON turns the join into a hash join on that + key, with the two positional overlap predicates demoted to a residual + filter, and a two-valued key such as ``strand`` makes that join + quadratic inside every partition. The dialect therefore emits the + equality between the two tables' strand columns as the pair + ``a.strand >= b.strand AND a.strand <= b.strand``, placed after the + two positional overlap predicates. The pair is the equality spelled + without ``=``: over a totally ordered domain ``x = y`` holds exactly + when ``x >= y`` and ``x <= y`` both hold, strand values are strings + DuckDB orders by collation, and NULL is NULL, hence false, in both + forms, so the conjunction admits exactly the rows the equality does. + ``IE_JOIN`` requires no equality among the join conditions and sorts + on the first two inequalities it is given, so the join sorts on + position and filters on strand. The pair is emitted in every branch, + including the unpartitioned fallback above the partition ceiling, so a + query binds the same way regardless of its chromosome count; on that + branch the chromosome hash join carries the pair as a residual filter + at roughly 1.8x the cost of the composite key (INNER 1.48s against + 2.71s at 300 chromosomes and 1,000,000 intervals per side, eight + threads). + + The strand column is each table's registered ``strand_col``, which + defaults to ``strand`` and applies to an unregistered table too; + ``Table(strand_col=...)`` renames it and ``strand_col=None`` disables + the rewrite for that table. A conjunct is rewritten when it is a + top-level ``=``, ``NOT (<>)``, or single-element ``IN``, parenthesized + or not and in either operand order, whose two operands are bare + columns naming each side's strand column. The rewrite stands down + entirely when any other cross-side equality remains in the residuals, + because the join would hash on that key regardless. Every other + equality is emitted as written. Shapes DuckDB does not hash on need no + rewrite: a one-sided equality (``a.strand = '+'``, a scan filter + DuckDB pushes below the join), an equality under ``OR`` or as + ``NOT (a = b)``, or one mixing both sides in a single operand. Shapes + it does hash on are left alone deliberately: a cross-side equality on + any other column (``a.name = b.name``), whose selectivity the + transpiler cannot judge, and an expression, quantified (``= ANY``), or + collated operand. ``IS NOT DISTINCT FROM`` is NULL-safe, so its + pure-inequality form needs an ``OR`` arm and it is left as written + too. The pair carries no implicit cast: strand columns of incompatible + types on the two sides bind under the equality and fail at bind time + under the dialect, since DuckDB casts implicitly for ``=`` but not for + ``>=`` / ``<=``; wrapping either operand in an expression + (``CAST(a.strand AS INTEGER) = b.strand``) keeps the equality as + written. The rewrite reaches the INNER, SEMI, and ANTI shapes; a + ``LEFT`` / ``RIGHT`` join or the ``count_overlaps`` fast path carrying + a strand equality declines the dialect on any ON residual and keeps + the naive plan's hash join. + + The gate is the strand column rather than any cross-side key because + it is the one cardinality signal the transpiler has without + statistics. A key more selective than the positional overlap prunes + more pairs as a hash key than the pair does as a per-candidate filter, + and the loss grows with overlap density; choosing per key is the + cost-based optimizer's job + (`#67 `_). Measured on + DuckDB 1.5.5 with 1,000,000 intervals per side over 24 chromosomes, + ``SELECT a.chrom, a.start, a."end" FROM a SEMI JOIN b ON a.interval + INTERSECTS b.interval AND a.strand = b.strand`` (the ``bedtools + intersect -s -u`` shape with its columns spelled out; see + :doc:`../recipes/bedtools-migration`), the naive plan takes 45.6s at + one thread and 14.0s at eight; the dialect takes 0.86s and 0.37s. The + same query keyed on an arbitrary high-cardinality column (10,000 + distinct values over a 100,000-position span) runs in 0.32s at eight + threads as the hash join the dialect keeps, against 7.1s as a pair. + Below roughly a thousand rows per chromosome the two sorts the range + join needs cost more than the hash key saves, a bounded loss of tens + of milliseconds. The dialect splits unsupported shapes into two buckets. Soft-fallback shapes route to the naive-predicate plan automatically and return correct @@ -580,6 +649,10 @@ without risk of silent incorrectness. Hard-error shapes (enumerated further below) raise ``ValueError`` at transpile time — the dialect deliberately refuses them rather than silently producing the wrong SQL. +One supported shape is neither: a strand equality whose two columns have +incompatible types binds under the naive plan but fails at bind time +under the dialect, deterministically, with the escape hatch described +under **Strand-equality residuals** above. The soft-fallback shapes are: @@ -588,12 +661,12 @@ The soft-fallback shapes are: IEJoin path instead (see **Outer joins** above), except for the shapes listed there. A fallback also applies to any outer join that keeps its side modifier while the ``INTERSECTS`` lives in the top-level ``WHERE`` - (e.g. ``LEFT JOIN ... ON TRUE WHERE a.interval INTERSECTS b.interval``) — + (e.g., ``LEFT JOIN ... ON TRUE WHERE a.interval INTERSECTS b.interval``) — there the filter discards the very unmatched rows the outer join preserves, so it is a different query. - **SEMI / ANTI join with the INTERSECTS in the WHERE.** A ``SEMI`` / ``ANTI`` join whose column-to-column ``INTERSECTS`` sits in the - top-level ``WHERE`` rather than its own ``ON`` (e.g. ``ANTI JOIN ... ON + top-level ``WHERE`` rather than its own ``ON`` (e.g., ``ANTI JOIN ... ON TRUE WHERE a.interval INTERSECTS b.interval``) falls back. The right table is out of scope in the ``WHERE`` after a left-only join, so the reference plans reject it with a binder error; the dialect declines so