[conradbzura]

Summary

A residual is a non-INTERSECTS predicate the DuckDB IEJoin dialect inlines into each per-chromosome subquery's ON. A cross-side equality residual is an equi-join key, and DuckDB plans any join carrying one as a HASH_JOIN on that key with the position inequalities demoted to a residual filter. For a.strand = b.strand, the residual of bedtools intersect -s, the key has two values, so each hash bucket is half the partition and the join is quadratic again: 45.6s at 1e6 intervals per side single-threaded on DuckDB 1.5.5.

Emit the equality between the two tables' registered strand columns as the pair a.strand >= b.strand AND a.strand <= b.strand, placed after the two positional overlap predicates. The pair is the equality spelled without =: over any totally ordered domain, x = y holds exactly when x >= y and x <= y both hold, and strand values are strings that DuckDB orders by collation, so the conjunction admits a pair of rows only when the two strings are identical. NULL behaves the same in both forms, since NULL >= '+' AND NULL <= '+' is NULL, hence false, as NULL = '+' is. Nothing about the result changes; the spelling exists for the planner. DuckDB classifies any left = right condition as a hash key, while IE_JOIN requires no equality among the join conditions, sorts on the first two inequalities in ON order, and evaluates the rest as a filter over each candidate pair. Written as two inequalities after the positional predicates, the strand condition reaches the range join as a per-candidate filter rather than a hash key. The same query runs in 0.86s single-threaded and 0.37s on eight threads.

The rewrite is gated on the registered strand_col because it is the one cardinality signal the transpiler has without statistics. A key more selective than the positional overlap prunes more pairs as a hash key than the pair does as a per-candidate filter, and the loss grows with overlap density: a 10,000-value key over a dense 100,000-position span runs in 0.32s as the hash join the dialect keeps and 7.1s as a pair. Every other equality is emitted as written. That also keeps expression, quantified (= ANY), volatile, and cross-type operands out of the duplicated pair, where duplication would change results or fail at bind time. The per-key choice belongs to the cost-based optimizer (#67).

The per-chromosome branches carry the pair. The unpartitioned query the chromosome-count ceiling selects is a hash join on the chromosome, so its strand equality stays in the composite key and is emitted as written.

The issue proposed widening the partition key to (chrom, strand) behind a runtime cardinality gate, and asked that two unexplained probe signals be understood first. Both are explained in the investigation comment: the surviving HASH_JOIN is the decorrelation scaffolding of the correlated EXISTS and appears identically in the fast -S plan, and the negative thread scaling does not reproduce on DuckDB 1.4.3 or 1.5.5. Widening is the wrong trade: its 48 branches each full-scan both tables, and those scans alone cost more than the join work they save, so the widened plan runs 1.3 to 1.6x slower than the inequality reference single-threaded. The rewrite keeps the 24 chromosome branches, leaves the shared per-chromosome scaffolding untouched, and needs no cardinality gate, so the gate and its test are replaced by plan-structure and predicate-order tests.

Closes #98

Proposed changes

Rewrite a strand equality as an inequality pair

Add _strand_equality_as_range_pair to src/giql/expanders/intersects_duckdb.py. It walks a copy of a residual's top-level AND tree, descending parentheses, and replaces each EQ whose two operands are bare columns naming their side's registered strand column with GTE AND LTE. The alias-to-strand-column map comes from the Table registry, so a table with strand_col=None is never rewritten and an unqualified column never matches. The docstring states the contract and the ordering precondition, and carries the planner mechanics and the gate rationale under an Implementation notes rubric.

Build two predicate strings in _build_sql from the same residual ASTs. The rewritten one feeds the INNER ON and the SEMI and ANTI EXISTS forms; the as-written one feeds the unpartitioned query, which already hashes on the chromosome.

Cover the rewrite on the AST and the plan

Read the dynamic SQL back out of the session variable and parse it with sqlglot, so structure tests assert on the conjunct AST rather than on rendered text. Read EXPLAIN (FORMAT JSON) for plan tests, whose IE_JOIN node lists its conditions in order. Extend _make_planner_scale_tables with strand and chroms parameters, move the stranded rows into a fixture, and keep the residual shapes in one _EQUALITY_RESIDUAL_SHAPES table shared by the parametrized and property tests.

Document the rewrite

Add a strand-equality bullet to the DuckDB IEJoin section of docs/transpilation/performance.rst: why the equality forces a hash join, how the pair after the positional keys keeps the join on IE_JOIN, that the unpartitioned fallback keeps the equality, which shapes are rewritten and which are emitted as written, why the gate is the registered strand column, and the measurements on DuckDB 1.5.5. Link the bedtools strand flags to the migration recipe instead of restating its predicates.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestTranspileDuckDBIEJoinSQLStructure	A JOIN or SEMI JOIN INTERSECTS query whose ON carries a.strand = b.strand	The per-chromosome dynamic SQL is read back and parsed	The predicate is LT, GT, GTE, LTE with the strand pair last and no equality	Rewrite and predicate order
2	TestTranspileDuckDBIEJoinSQLStructure	Stranded tables over 300 chromosomes and the same query	The query the session variable selects is read back and parsed	The predicate carries the chromosome and strand equalities and no >= / <= pair	Unpartitioned fallback keeps the hash key
3	TestTranspileDuckDBIEJoinSQLStructure	An ON equality on a non-strand column, over an expression, one-sided, under NOT, collated, quantified with ANY, or carrying random() or nextval()	The per-chromosome dynamic SQL is read back and parsed	The residual follows the overlap predicates structurally unchanged with no >= / <= pair	Shapes emitted as written
4	TestTranspileDuckDBIEJoinSQLStructure	A strand pair in either operand order, parenthesised, or beside another equality or inequality	The per-chromosome dynamic SQL is read back and parsed	The strand pair is rewritten and every other conjunct is left as written	Compound residuals
5	TestTranspileDuckDBIEJoinSQLStructure	Table configs with a custom strand_col or strand_col=None	The per-chromosome dynamic SQL is read back and parsed	The equality is rewritten only when both operands name the registered strand column	Registration gate
6	TestTranspileDuckDBIEJoinExecution	20k stranded rows per side over four chromosomes and an INNER JOIN with a.strand = b.strand	The dynamic SQL is planned with EXPLAIN (FORMAT JSON)	The plan has IE_JOIN, no HASH_JOIN, and the positional conditions ahead of the strand pair	INNER plan structure
7	TestTranspileDuckDBIEJoinExecution	The same tables and a SEMI or ANTI JOIN with a.strand = b.strand	The dynamic SQL is planned with EXPLAIN (FORMAT JSON)	The plan has IE_JOIN, no BLOCKWISE_NL_JOIN, and the positional conditions ahead of the strand pair	SEMI / ANTI plan structure
8	TestTranspileDuckDBIEJoinExecution	An INNER JOIN with a.name = b.name	The dynamic SQL is planned with EXPLAIN (FORMAT JSON)	The plan has a HASH_JOIN keyed on the name equality	Non-strand keys stay hash keys
9	TestTranspileDuckDBIEJoinExecution	Stranded tables over 300 chromosomes and an INNER JOIN with a.strand = b.strand	The selected query is planned with EXPLAIN (FORMAT JSON)	The HASH_JOIN key carries both the chromosome and the strand equality	Unpartitioned plan
10	TestTranspileDuckDBIEJoinExecution	Stranded peaks and genes with a left-only strand value, NULL strands on both sides, and a left-only chromosome	A SEMI or ANTI strand-equality query is executed under both plans	SEMI returns exactly the same-strand overlaps, ANTI the rest, and both agree with the naive plan	Truth table with one-sided and NULL values
11	TestTranspileDuckDBIEJoinExecution	The stranded fixture and an INNER, SEMI, or ANTI JOIN with one of the eleven _EQUALITY_RESIDUAL_SHAPES	Executed under both plans	Rows and result schema match the naive plan	Residual shapes across join kinds
12	TestTranspileDuckDBIEJoinExecution	The stranded fixture and an INNER JOIN whose a.strand = b.strand sits in the WHERE	Executed under both plans	Rows match and the per-chromosome predicate carries the >= / <= pair	WHERE residual inlining
13	TestTranspileDuckDBIEJoinExecution	A VARCHAR gene_id on one side and an INTEGER gene_id on the other, joined on a.gene_id = b.gene_id	Executed under both plans	Rows match, since the equality is emitted as written	Cross-type non-strand key
14	TestTranspileDuckDBIEJoinExecution	A VARCHAR strand on one side and an INTEGER strand on the other, joined on a.strand = b.strand	Executed under dialect='duckdb'	DuckDB raises a binder error	The one shape the pair cannot bind
15	TestTranspileDuckDBIEJoinExecution	Hypothesis-generated stranded tables with NULL strands and scores, a sampled residual shape, and a sampled join kind	Executed under both plans	Rows and result schema match the naive plan	Property-based equivalence