fix(query): evaluate Near/Within for real and refine indexed Near to the circle (#165)

[CaffeinatedCoder]

Fixes #165.

Problem

GeoSpatialExtensions.Near and Within returned true when evaluated in memory, which was only correct while the planner turned them into an R-Tree search. Inside an ||, under a negation, or on any other path that falls back to the compiled predicate, every document matched. Indexed Near searched the circle's bounding box and never checked the distance, so it also returned points up to √2 × radius away. BLQL's $geoNear already computed the real distance, so LINQ and BLQL disagreed on the same data.

VectorSearch also returned true, so VectorSearch(...) || x matched every document.

Fix

Near / Within

• Near computes the great-circle distance with SpatialMath.DistanceKm; Within tests the box with its edges included. Both match GeoNearFilter / GeoWithinFilter in BLQL and the R-Tree's inclusive Intersects.
• New RTreeIndex.SearchNear searches the bounding box, then checks each hit against the exact distance. It uses the leaf entry's box, which for an indexed point is the point itself, so no document has to be loaded and the indexed Near plan stays Exact. CollectionSecondaryIndex.Near (typed NearAsync and LINQ) and DynamicCollection.NearAsync use it.

VectorSearch

• It now throws NotSupportedException when evaluated per document. Whether a document is among the k nearest depends on the whole collection, so there is no correct per-row answer.
• To keep VectorSearch(...) && x working, the planner lets a vector term drive any && plan. Before, x could win when it had its own index, and the post-filter then evaluated VectorSearch as true. The post-filter for a vector plan is compiled with the VectorSearch call replaced by true (IndexOptimizer.WithoutCall), since the index already satisfied it.
• VectorSearch(...) || x cannot be served and now throws instead of returning everything.

Planner

• Two spatial terms on the same index (Near(...) && Within(...)) were merged into a B-Tree range on the spatial index. One drives the plan now, and the other is applied as a post-filter.

Tests

NearWithinCorrectnessTests adds 11 tests. 10 fail on main:

• In memory: Near uses the great-circle distance, Within includes the edges, VectorSearch throws.
• The issue's repros: Near || x and Within || x on an indexed property, and indexed Near excluding a box corner via LINQ, NearAsync and Count.
• !Near(...), which runs as a full scan.
• Near && Within on the same index returns their intersection.
• VectorSearch || x throws.
• DynamicCollection.NearAsync excludes the box corner.

The 11th, Title == "B" && VectorSearch(...), guards the new post-filter. It passes on main because VectorSearch returned true there.

The existing spatial, vector, optimizer and BLQL tests (508) still pass. Full suite on macOS: 2390 passed, 3 failed, 4 skipped. The 3 failures are the known MultiProcessWalSharedMemoryTests .wal-shm lock timeouts.

Breaking change (release as 5.3.0)

Queries that evaluated VectorSearch outside an index plan used to return every document; they now throw NotSupportedException. This includes ||, a negation, a property without a vector index, and FindAll().Where(...) / AsEnumerable().Where(...) in memory. The old XML doc promised it "always returns true when evaluated in memory".

Suggested squash-message footer:

BREAKING CHANGE: VectorSearch throws NotSupportedException when it is evaluated per document (outside an index plan, inside ||, or on a property without a vector index) instead of returning true for every document.

Not in scope

• Near across the ±180° meridian and over the poles (Near misses points across the ±180° meridian and over the poles #166). The exact-distance check here only trims what the bounding box returns.
• BLQL $nearVector still matches every document and is not routed to the HNSW index.

Merges cleanly with #169, #170 and #172.

🤖 Generated with Claude Code

[mrdevrobot]

Thanks. The in-memory Near/Within match BLQL ($geoNear uses the same Haversine and <=, GeoWithinFilter and GeoBox.Intersects are inclusive), SearchNear on the leaf box is sound since every insert path stores GeoBox.FromPoint, WithoutCall finds its node (same lambda instance through TryOptimize and FetchAsync), and Count/Any/First go through the same plan. 518/518 targeted tests pass. One regression to fix and one pre-existing hole that this PR's description claims to cover.

1. Regression: Where(x => x.Emb.VectorSearch(q, k)).GroupBy(...) now throws NotSupportedException (BTreeQueryProvider.cs:302-322, :652-670).
FetchAsync serves the WHERE from the HNSW index, but when model.HasComplexOperators is set (GroupBy, Join, Sum/Min/Max with selector) or OrderBy follows a Select, ExecuteViaEnumerableRewriter rewrites the whole original expression, including the Where, to Enumerable.* and re-runs it over the fetched list. Before, VectorSearch returned true so the re-applied WHERE was a no-op; now it throws from VectorSearchExtensions.cs:20. Verified with a probe test: passes at 45dbc8c, fails at 224b332. Fix: when the WHERE was already applied, drop it or replace its VectorSearch nodes with true before the rewrite (the plan's VectorSearchCall is available from FetchAsync). Please add tests for Where(VectorSearch).GroupBy(...) and Where(VectorSearch).Sum(selector).

2. Near(a) || Near(b) on the same spatial index still returns nothing (IndexOptimizer.cs:201-236, :475-490).
A spatial/vector OptimizationResult has IsRange = false and null MinValue/MaxValue, so TryGetPointValues yields [null] and the OrElse branch merges the two terms into an InValues = [null] B-Tree probe on the R-Tree index name, which returns 0 rows. Pre-existing, but the PR fixes the && analogue (:180-181) and says Near inside || is now correct. Add !IsSpatialSearch && !IsVectorSearch to the OrElse merge guard (or make TryGetPointValues fail on a null MinValue) so it falls through to the in-memory path.

3. Breaking change. The old XML doc promised VectorSearch "always returns true when evaluated in memory"; FindAll().Where(...), AsEnumerable().Where(...) and FindAsync on a collection without a vector index now throw. I agree the throw is the right design, but it should ship in a minor (5.3.0) with a CHANGELOG "breaking" entry, not a patch.

Smaller: v1.VectorSearch(..) && v2.VectorSearch(..) throws with a message telling the user to combine with &&, which they did (IndexOptimizer.cs:171-178), detect two vector plans and give a clearer message. About 460 of the 508 diff lines in IndexOptimizer.cs are CRLF normalisation; please keep the file's line endings or do that in a separate commit so blame stays readable. The residual predicate is compiled per FetchAsync call (DocumentCollection.cs:3525-3528), unlike the cached GetCompiled().

Within with min.Longitude > max.Longitude: matching nothing on both paths is consistent; let's document that Within does not cross the antimeridian and leave it there.

Posted from the wrong account by mistake; the maintainer review from @mrdevrobot supersedes this.

[CaffeinatedCoder]

Thanks, good catch on the replay. Addressed in 97bfb7d; the branch also merges main (5.2.3).

1. Where(VectorSearch).GroupBy(...) regression. Before ExecuteViaEnumerableRewriter runs (both call sites), the provider replaces every VectorSearch call with true, but only when FetchAsync has applied the WHERE. A VectorSearch that reaches that point was necessarily served by the index: otherwise FetchAsync would already have thrown while evaluating it. I used a visitor rather than matching the plan's VectorSearchCall node, because chained Where calls are merged through ParameterReplacer, which creates new nodes. New test VectorSearch_FollowedByGroupByOrAggregate_UsesTheIndexResult covers Where(VectorSearch).GroupBy(...) and Where(VectorSearch).Sum(selector). It fails on the previous commit.

2. Near(a) || Near(b). The OrElse merge guard now excludes spatial and vector plans, so the expression falls through to the in-memory path. New test NearOrNear_OnTheSameIndex_ReturnsDocumentsNearEither.

3. Breaking change. Agreed. The commit carries a BREAKING CHANGE: footer describing the VectorSearch behaviour so versionize lists it, and I updated the PR description. If you squash, please keep that footer in the squash message, and release with --release-as 5.3.0 (the footer alone would bump the major).

Smaller:

• Two VectorSearch terms: the message now says "a query can contain only one VectorSearch". New test.
• Line endings: IndexOptimizer.cs is restored to its original mixed CRLF/LF endings line by line. Its diff against main is now 44 added / 2 removed lines. It came from my editing tooling normalising the file; sorry about that.
• The residual predicate is compiled lazily on the first candidate that needs it, like GetCompiled().
• Within docs now say the box doesn't wrap across ±180°: a min longitude greater than the max longitude matches nothing.

Full suite on macOS: 2406 passed, 0 failed, 4 skipped. #174 is updated on top of this.

🤖 Generated with Claude Code

[mrdevrobot]

Points 2, 3 and 4 are fixed, and the line-ending churn is gone. Point 1 needs another pass: the fix trades the exception for silently wrong results.

BTreeQueryProvider.cs:314-317 replaces every VectorSearch in the whole expression with true whenever model.WhereClause != null. That condition doesn't check whether the WHERE contained a VectorSearch, and the rewrite isn't limited to the WHERE that FetchAsync applied. Probes, with 3 documents A = [0,0,0], A = [1,1,1], B = [50,50,50]:

query	result now	correct
Where(VS(q,2)).Select(x => new { x.Title, ok = x.Emb.VectorSearch(q2,1) }).OrderBy(a => a.Title)	ok = true for both A	throw
Where(Title != null).Select(x => new { ok = VS(q,1) }).OrderBy(...)	ok = true for all 3	throw
Where(Title != null).GroupBy(Title).Select(g => g.Count(x => VS(q,1)))	A = 2, B = 1	throw
Where(Title != null).Sum(x => VS(q,1) ? 1 : 0)	3	throw
Where(Title != null).Count(x => VS(q,1))	3	throw

All five threw NotSupportedException before this commit, which is the behaviour we want. Please neutralise VectorSearch only inside the Queryable.Where predicates on the root chain that were folded into model.WhereClause. Simpler still, drop those Where nodes from the replay, since FetchAsync already applied them. Every other VectorSearch should keep throwing. Please add the probes above as regression tests.

Related but older, not for this PR: BTreeExpressionVisitor.VisitWhere (~L57-88) merges any later Where into WhereClause, including one on a Join inner sequence. Where(Title == "B").Join(VectorItems.Where(y => VS(q,1)), ...) returns [] instead of ["B/A"] both before and after. Worth an issue.

On release: agreed on the BREAKING CHANGE: footer. versionize would make the next auto release 6.0.0, so we'll release with --release-as 5.3.0 as you suggest.

[CaffeinatedCoder]

Point 1, second pass, is in db38b78.

• VectorSearchSatisfiedRewriter is gone. The replay now drops the Queryable.Where calls that FetchAsync already applied, and leaves every other VectorSearch alone.
• A Where is dropped only if it is on the root chain and only filtering, ordering or paging operators (Where, OrderBy*, ThenBy*, Skip, Take, SkipWhile, TakeWhile, Distinct, Reverse) lie between it and the root.
  • VisitWhere folds every such Where into WhereClause, and FetchAsync applies all of it. So every element that reaches that Where already satisfies it, and dropping it changes nothing except that VectorSearch is not evaluated again.
  • A Where after Select, GroupBy or Join, on a Join inner sequence, inside a lambda, or the indexed (x, i) overload is kept. A VectorSearch anywhere else still throws.
• All five probes from your table are regression tests in NearWithinCorrectnessTests (VectorSearch_In*_Throws). On 97bfb7d all five fail with "No exception was thrown". I also added a positive case, Where(VS(q,2)).Select(new { Title }).OrderBy(Title), which returns A, A.
• Full BLite.Tests: 2412 passed, 0 failed, 4 skipped.

Join inner Where: confirmed, and it isn't specific to VectorSearch. Where(Title == "B").Join(VectorItems.Where(y => y.Title == "A"), x => 1, y => 1, ...) also returns [] on main. Filed as #186.

Release: noted, --release-as 5.3.0.

[mrdevrobot]

Verified db38b78. All five probes from the last review now throw NotSupportedException, and Where(VectorSearch).GroupBy(...) / .Sum(selector) still use the index result. DropAppliedWheres only drops single-parameter Where calls on the root chain below subset-preserving operators, which are exactly the ones folded into the fetch filter, so it can't let unfiltered rows through. The new tests fail with the fix reverted. Full suite on main with #169/#170 merged: 2434 / 0 / 4 skipped.

Noted, not blocking: a VectorSearch Where after a non-subset-preserving operator, e.g. Select(x => x).Where(VS), now throws instead of using the index. That's the safe direction.

While probing I hit the older VisitWhere folding on main that #186 describes: Where(p1).Take(2).Where(p2) returns rows LINQ-to-Objects doesn't, because the second Where is applied before the paging. Not caused by this PR.

Approving and merging. The squash keeps your BREAKING CHANGE: footer; the release will go out as 5.3.0.