Repository navigation
Conversation
The memtable's fuzzy path expanded every query token and unioned the expansions, so a fuzzy `operator=And` had no evaluation: `local_fts_query` and `validate_lsm_fts_query` both refused it. The base inverted index answers the same query, so an LSM read either errored or, in a consumer that declines what the memtable refuses, quietly dropped the fresh tier. `FtsQueryExpr::Fuzzy` carries an `operator`. `search_fuzzy_tokens` expands each token into its own group and, for `And`, keeps the rows matched by every group with the group scores summed -- the same conjunction `search_grouped_and` runs for exact terms, now shared as `search_groups_and`. `Or` is unchanged. Both rejections are lifted and the planner test that pinned one of them now asserts the row comes back. Also re-exports `MultivectorScoringExec` from `lance::io::exec` so a consumer can recognise the multivector plan shape by type. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The grouped intersection can support fuzzy AND, but an LSM query must have the same effective terms and expansion budget in every source arm. A base-plus-active reproduction returned different matches for identical rows under three supported query settings. Keep the grouping and align AUTO handling, explicit fuzzy tokenization, and the leaf-wide expansion cap with the committed index.
Please mark this PR with the breaking-change label.
| } | ||
| Ok(()) | ||
| } | ||
| IndexFtsQuery::Match(_) => Ok(()), |
There was a problem hiding this comment.
Accepting AUTO fuzzy AND here gives the two LSM arms different meanings. The dataset scanner currently rewrites fuzziness=None to exact, while the active memtable expands it. Identical rows therefore return different matches depending on their tier. Apply one effective AUTO mode to every arm before admitting this shape.
Reproducer run on this head
In the existing local_mode_unions_base_and_active_with_consistent_score_schema fixture, use one indexed base row (id=1, text="alpha beta") and one active row (id=2, text="alpha beta"), then run the query below through LsmFtsSearchPlanner.
MatchQuery::new("alphb beta".to_string())
.with_fuzziness(None)
.with_operator(Operator::And)A temporary cargo test -p lance --lib gate_repro_fuzzy_and_base_active_parity test asserted no hits under the current exact compatibility gate. It failed: the result was [2]; the indexed row was absent.
There was a problem hiding this comment.
Fixed in 551233de2: the LSM planner now rewrites AUTO to exact before dispatching every source arm. The base-plus-active alphb beta parity case now excludes both identical rows.
| m.prefix_length, | ||
| m.max_expansions, | ||
| ) | ||
| .with_operator(m.operator) |
There was a problem hiding this comment.
Explicit fuzzy AND now reaches FtsMemIndex::tokenize_for_search, which applies the index analyzer; the committed index deliberately uses SimpleTokenizer for explicit fuzziness. With the default stop-word filter, the active arm drops the from alpha the and matches alpha, while the base arm still requires the second fuzzy term. Use the same explicit-fuzzy query tokenization for both arms before grouping.
Reproducer run on this head
In the existing local_mode_unions_base_and_active_with_consistent_score_schema fixture, use one indexed base row (id=1, text="alpha beta") and one active row (id=2, text="alpha beta"), then run the query below through LsmFtsSearchPlanner.
MatchQuery::new("alpha the".to_string())
.with_fuzziness(Some(1))
.with_operator(Operator::And)The temporary base-plus-active test expected both rows to be excluded and failed with IDs [2] from the active arm only.
There was a problem hiding this comment.
Fixed in 551233de2: explicit fuzzy terms now use SimpleTokenizer in the memtable, matching the committed arm. The alpha the parity case now excludes both rows.
| let mut groups: Vec<Vec<String>> = Vec::with_capacity(tokens.len()); | ||
| for tok in tokens { | ||
| let max_dist = fuzziness.unwrap_or_else(|| auto_fuzziness(tok)); | ||
| let mut group: Vec<String> = Vec::new(); |
There was a problem hiding this comment.
This loop spends the full max_expansions budget separately for each query token. The committed Match path uses one deterministic vocabulary budget across all positions, established in #8758. An active row can therefore satisfy fuzzy AND when the identical indexed row cannot. Track the remaining leaf-wide budget while building the groups.
Reproducer run on this head
In the existing local_mode_unions_base_and_active_with_consistent_score_schema fixture, use one indexed base row (id=1, text="alpha beta") and one active row (id=2, text="alpha beta"), then run the query below through LsmFtsSearchPlanner.
MatchQuery::new("alpha beta".to_string())
.with_fuzziness(Some(1))
.with_operator(Operator::And)
.with_max_expansions(1)The temporary base-plus-active test expected no matches because one expansion cannot cover both AND positions. It failed with IDs [2] from the active arm only.
There was a problem hiding this comment.
Fixed in 551233de2: the memtable now spends one leaf-wide expansion budget in query order and selects terms lexicographically. The cap-one parity case now excludes both rows.
lance-gatekeeper
Bot
added
the
K-changes
…ve memtable An LSM fuzzy search could match an active row that the identical indexed row did not, because the memtable arm diverged from the committed index in three ways: - AUTO fuzziness: the dataset scanner runs it as exact, the memtable expanded it. The LSM planner now applies the same rewrite before planning any arm. - Explicit fuzziness tokenization: the committed index splits the query with a bare `SimpleTokenizer`, the memtable ran the index analyzer and dropped stop words. The memtable now tokenizes explicit fuzzy queries the same way. - Expansion budget: the committed index spends one `max_expansions` budget across the leaf on the lexicographically smallest candidates, the memtable gave each token its own budget and kept the closest matches. The memtable now does the former. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
…O terms Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…zy-and # Conflicts: # rust/lance/src/dataset/mem_wal/index/fts.rs # rust/lance/src/dataset/mem_wal/scanner/fts_search.rs
Refs lance-format#9883. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-approved
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lance-gatekeeper
Bot
removed
K-risk
There was a problem hiding this comment.
The exhausted-budget failure is fixed. The author accepts retaining fuzzy conjunctions with the documented per-source expansion behavior. When max_expansions binds, a source can return matches a merged index would exclude, so flushing or compacting between tiers can change results.
lance-gatekeeper
Bot
added
K-approved
This branch has not been deployed
1 participant
Adds support for FTS queries that include fuzzy matches in conjunction with the
ANDoperator.