perf(query): LINQ Sum/Average/Min/Max from index keys or raw BSON

[CaffeinatedCoder]

LINQ aggregates with a selector (Sum, Average, Min, Max) materialised every matching entity and ran Enumerable over them. This PR computes them from index keys or raw BSON instead, with the result LINQ would compute over the entities. It also makes BSON scans read documents in place.

Independent of #180. Both touch BTreeQueryProvider, but in different places.

Why the existing push-down never ran

TryBsonAggregate invoked ScanAsync<TResult> through reflection and cast the result to IEnumerable. ScanAsync returns an IAsyncEnumerable, so the cast produced null and every aggregate fell through to full materialisation. Its semantics also differed from LINQ: it returned 0 instead of throwing on empty input, summed int unchecked, and ignored Take/Skip before the aggregate. This PR removes it. TryPushDownSelect has the same cast problem; it is not touched here (see follow-ups).

What it does

x => x.Property aggregates applied straight to Where clauses over the collection go through DocumentCollection.TryAggregateAsync:

1. Index keys. An int/long property (or nullable) with a B-Tree index, and either no filter or a filter that index covers exactly (same rule as the key-only Count). Values are decoded from the [value][id] keys with no document reads. Min reads the first entry and Max the last.
2. Raw BSON. Used when no index serves the filter. The field is read straight from each document's BSON, in primary-key order, which is the order the materialising path yields the entities in. The filter runs as the same BSON predicate FetchAsync already uses. The attempt gives up, and the query runs the usual way, if a document:
   • lacks the field (the mapper decides what a missing property becomes);
   • holds a BSON type other than the one the mapper writes for the property (it would be coerced on read);
   • holds a null in a non-nullable property (materialising it throws);
   • or if the property is renamed with [Column]/[BsonProperty]/[JsonPropertyName].

Anything else in front of the aggregate (Take, Skip, OrderBy, Select, Distinct, …) keeps the existing path, because it changes which values are aggregated or their order.

LinqAggregate<TValue> folds the values the way Enumerable does:

	Rule
Sum	int sums in a checked int, long in a checked long, float in a double cast back at the end
Average	int/long sum in a checked long; float returns (float)(sum / n)
Min/Max	IComparable<T>, replaces only on strict improvement, so the first of equal values wins (-0.0/0.0, 1.0m/1.00m) and NaN is the smallest value
empty / nulls	nullable selectors skip null; on no values Sum is 0 and the rest return null (nullable) or throw InvalidOperationException

The one difference, index sums only. Index keys come in value order, not document order, so an index Sum range-checks only the total. If the exact total fits, it returns it, even where LINQ would have overflowed on an intermediate running total in document order (e.g. int.MaxValue, 1, -2). It never returns a different number. If you'd rather keep LINQ's exception exactly, the index path could be limited to Min/Max/Average.

Why not other types on the index path:

• decimal: keys drop trailing zeros (1.50 becomes 1.5), and the first of equal Min/Max values can't be identified.
• double/float: sums depend on the order of the values, and so do the ties between -0.0 and 0.0.

BSON scans read documents in place

ScanAsync, CountScanAsync and FetchAsync's BSON-predicate path copied every document out of its page (ReadRawBytesAt) before evaluating it. VisitRawDocumentsAsync hands single-page documents to the caller straight from the page buffer and reassembles only overflow documents. Each page is still read once and cached for the scan, as before.

I first tried a bounded 64-page window instead, but it was ~60% slower. Primary-key order isn't page order (ObjectIds created in the same second aren't monotonic), so the window kept re-reading pages.

Numbers

10,000 documents, index on Count, macOS M-series, Release. The harness is the same code built against main and against this branch; each figure is the median of 100 runs after 60 warm-ups.

Query	main	this PR	allocated (main → PR)
Sum(x => x.Count) (indexed int)	4.58 ms	0.22 ms	6,464 KB → 2 KB
Where(Count in range).Sum(x => x.Count)	1.15 ms	0.10 ms	1,610 KB → 4 KB
Max(x => x.Count)	4.29 ms	0.02 ms	6,464 KB → 5 KB
Sum(x => x.Ratio) (double, no index)	4.10 ms	1.49 ms	6,464 KB → 1,041 KB
Average(x => x.Price) (decimal, no index)	4.39 ms	2.10 ms	6,464 KB → 1,431 KB
Where(x => x.Tag == "b").Sum(x => x.Ratio)	2.62 ms	2.17 ms	5,946 KB → 1,277 KB
Where(x => x.Tag == "b").ToList() (scan only)	2.68 ms	2.14 ms	5,994 KB → 2,791 KB

The new LinqAggregateBenchmarks (BenchmarkDotNet) compares each path against materialise + LINQ to Objects. With the index: Sum ~30× faster, a range Sum ~80×, Max ~1,000×.

What remains of a raw-BSON aggregate is the per-document scan machinery. That is the async primary-index iteration, plus an IndexKey allocation to check each document's _id against its entry. It is shared with every BSON scan, so I left it for a separate change.

Tests

LinqAggregateTests, 18 tests. Results are compared bit for bit (double/float bits, decimal.GetBits, exception type) with LINQ to Objects over the same entities in the same order:

• Every combination: 4 operators × 11 properties (int/long/double/float/decimal, plain and nullable) × 6 filters (none, non-indexed, covered range, range plus residue, no match, empty range). Each combination also checks whether it was pushed down or correctly left to the usual path.
• Floating point: NaN, ±0, ±∞, 1e308 overflow to infinity, and a float set where (float)(sum / n) ≠ (float)sum / n.
• Decimal scale; integer overflow; the documented index-sum difference.
• Gives up when it must: documents without the field, with another BSON type, or with null in a non-nullable property (materialising throws, so the aggregate throws alike); a renamed property whose conventional name is another property's field.
• Query shapes and maintenance: Take/Skip/OrderBy before the aggregate, chained Where, SumAsync/AverageAsync; index aggregates after updates and deletes.

Each rule was checked by reverting it: all 12 mutations fail at least one test. Full suite on macOS: 2410 passed, 0 failed, 4 skipped.

Found along the way (not changed here)

• Renamed properties fail on insert unless the new name happens to be another property's conventional name. The generated mapper honours [Column("x")]/[BsonProperty]/[JsonPropertyName], but BsonSchemaGenerator registers only lowercase property names, so WriteElementHeader throws "BSON Key 'x' not found in dictionary cache". Filed as Renamed properties ([Column]/[BsonProperty]/[JsonPropertyName]) fail on insert: field name not registered #184.
• TryPushDownSelect never runs (same IAsyncEnumerable cast). Making it run also means fixing the value-type projector, which returns default for filtered documents (Select followed by Any / Count(predicate) / Max / Sum returns false or 0: value-type default is taken as the push-down result #181).
• Where served by an index fetches in index order. So Where(indexed range).Sum(double) rounds differently from the same sum over ToList() in primary-key order. This is unchanged by this PR; such queries still take that path.

🤖 Generated with Claude Code

[mrdevrobot]

Thanks. The numeric semantics are right: I ran about 8,900 comparisons against LINQ-to-Objects (int/long/double/float/decimal, nullable, all-null, empty, overflow, NaN, ±0, decimal scale, indexed/non-indexed/partial-AND filters, OrderBy/Take/Skip before the aggregate) and they match bit for bit, apart from the cases below. The in-place visitor is sound: the span only reaches a synchronous delegate, rented buffers are returned on every path, overflow documents are reassembled, and transaction visibility is the same as the materialising path. Merged with main, full suite 2475 / 0 / 4 skipped; no conflicts with #180 or #193.

One blocker: the raw-BSON path assumes the field lowercase(Name) holds the property's value (DocumentCollection.cs:3447-3520, TryAggregateAsync, IsRenamedInBson, ToBsonName). That isn't always true, and these all returned the right value on main:

case	LINQ	this PR
[BsonIgnore] / [NotMapped] property over documents written before it was ignored, Sum	0	510
computed int Gone => Count * 10 over documents that stored gone earlier, Sum	100	5010
getter with logic, int G { get => _g * 2; set => _g = value; }, Sum (also Min/Max/Average)	7080	3540
non-key property named Id next to [Key] int Code, Sum(x => x.Id)	1000	10

Please take the field name and "stored verbatim" status from the mapper or generator metadata rather than the naming convention. At minimum, fall back when the property has an ignore attribute, isn't a compiler-generated auto-property with a setter, or is named Id without being the key. Attribute-name sniffing in IsRenamedInBson is fragile for the same reason.

Smaller:

• The index Sum (exactTotal: true, ~L3534) returns a value where LINQ throws OverflowException, e.g. {5, int.MinValue, -1} gives -2147483644. You documented it in the PR, but it should be in the public docs, or limit the index path to Min/Max/Average.
• s_scanProjectorMethod / s_compiledScanMethods are now used only by TryPushDownSelect, which never runs.
• AggregateRawFieldAsync: when TrySeekToField fails, SeekField restarts from the advanced position, so the name-based fallback can't find an earlier field.
• The index path decodes int/long keys without checking the key length; an assertion would guard against a type mismatch.

[CaffeinatedCoder]

Thanks for the thorough check. I merged main (4897eb1, 1d3e095; both clean) and addressed everything in 19c25ad, plus a test fix in a45fd45.

Field name and "stored verbatim" status now come from metadata.

• Generated mappers override a new DocumentMapperBase.GetVerbatimFieldName(propertyName) (default null). It returns the BSON field only when Deserialize restores the property from it unchanged, which requires all of:
  • both accessors compiler-implemented (a syntax check in source, [CompilerGenerated] for compiled properties);
  • a setter;
  • no converter;
  • a scalar type;
  • not passed to the constructor.
• Ignored and computed properties aren't mapped at all, so they return null. The key maps to _id, and renamed properties to their attribute name.
• TryAggregateAsync uses that field for the raw path, and requires it for the index path too: index keys are computed from the getter at write time, so for a property with accessor logic they aren't the materialised value either.
• IsRenamedInBson and the use of ToBsonName are gone. Renamed properties are now pushed down correctly. Custom mappers fall back to materialising.

One test per row of your table, each compared with LINQ-to-Objects and each failing without the fix with your numbers (510, 5010, 10):

• IgnoredProperty_… ([BsonIgnore] and [NotMapped]);
• ComputedProperty_…;
• AccessorsWithLogic_… (getter logic on an indexed int, setter logic on an indexed long, and the raw path);
• PropertyNamedId_ThatIsNotTheKey_….

I also added PropertySetThroughTheConstructor_…: a constructor that transforms its argument has the same problem. GeneratedMappers_NameTheFieldOnlyForPropertiesStoredVerbatim checks the generated metadata directly.

Index Sum overflow. I went for exact parity rather than documenting the difference. On the index path, int/long Sum/Average keep the positive and negative partial sums apart. Any running total LINQ computes, in any order, lies between those two sums. So if both fit the accumulator (int for Sum<int>, long otherwise), LINQ can't overflow, and the exact total is its result. If not, the outcome depends on order, so the aggregate runs in the order LINQ sees the entities: the raw scan in primary-key order, or materialising through the index for an index-served filter.

IndexKeySums_OverflowWhereLinqDoes covers four sets, all with explicit ascending ObjectIds, with and without such a filter:

• {MaxValue, 1, −2} and {MinValue, −1, 5}, which throw;
• {5, MinValue, −1} and {−2, MaxValue, 1}, which don't throw in primary-key order but do in index order under an index-served filter.

IndexKeySums_ThatCannotOverflow_AreTakenFromTheKeys checks the edge of the gate: positive values summing to exactly int.MaxValue and negative ones to exactly int.MinValue are still taken from the keys.

s_scanProjectorMethod / s_compiledScanMethods. Removed, along with TryPushDownSelect, s_tryCompileMethod and s_compiledSelectMethods. It wasn't just unreachable code: it ran for every Select without OrderBy/Skip/Take and always failed. It invoked ScanAsync<T>(projector, ct) with one argument (TargetParameterCountException: Parameter count mismatch), and the result would have been an IAsyncEnumerable, not an IEnumerable. So it compiled a projector per query and then fell through.

AggregateRawFieldAsync seek fallback. As far as I can see, TrySeekToField doesn't move the reader when it returns false: every false path returns before _position is assigned, and the doc comment says "the reader position is unchanged" (BsonSpanReader.cs:85-105). So SeekField did start from the first element. I made it explicit anyway: the code saves the position after ReadDocumentSize and seeks back before the name lookup. FieldAfterOthers_InDocumentsWithoutOffsetTable_IsFound pins the behaviour, using schemaless documents with no offset table where units comes after _id and tag. If you had a case where it misbehaved, I'd like to see it.

Key length / type. The index path now requires the index selector to be exactly x => x.Property with the aggregated type; only the boxing Convert to object is looked through. Keys are encoded from the selector's boxed value, so this pins the key layout. TryDecode also rejects keys with no document id after the value. IndexOnAConvertedProperty_IsNotDecodedAsThePropertyType uses an index on r => (long)r.Units, whose property path is Units; before the fix, its long keys were decoded as ints (−10 instead of −365).

Full suite on current main: 2550 passed / 4 skipped. The only failures were lock-timeout tests that pass in isolation or fail just as often on main (CrossCollectionWriteRaceTests fails 4 of 10 on a clean main here).

[mrdevrobot]

Thanks. Every point from the last review is resolved, and I verified each: verbatim field metadata from the generator, exact-parity index sums via split partial sums, dead push-down code removed, explicit seek-back, and KeysAreValuesOf/TryDecode guarding the key layout.

One regression was left on the index-key path, and I pushed a fix to the branch so this can land:

• eb21083: TryAggregateAsync trusted plan.MinValue/MaxValue even when the optimizer kept the constant's type, so the bound was encoded as a different key type than the index. With Count = 0..19 indexed, Where(r => r.Count < 9.5).Max(r => r.Count) returned 19 (LINQ and main: 9), and Where(r => r.Count > 4L).Sum(r => r.Count) returned 190 (180). The keys are now used only when each bound is null or exactly TValue; other filters take the post-filtered path as on main. FilterBoundOfAnotherType_IsNotUsedAsAKeyRange fails without the guard.
• bbb72ec: a test for an aggregate whose Where queries the collection again (q.Where(r => q.Any(...)).Sum(...)). It already passes and stays as a regression test, since the aggregate fast path runs ahead of the new HasSecondaryQuery check.
• Merges of main: fix(query): return the real result of terminals after Select and after an indexed top-N window (#181) #193 landed meanwhile and conflicted in BTreeQueryProvider. I kept this PR's removal of TryBsonAggregate/TryPushDownSelect and kept fix(query): return the real result of terminals after Select and after an indexed top-N window (#181) #193's HasSecondaryQuery replay right after the aggregate fast path.

The underlying optimizer issue (bounds not converted to the property type) also affects Count/ToList on main; I filed it as #222.

Full suite on the result: 2673 passed, 0 failed. I'll approve and merge once CI is green.

[mrdevrobot]

Approving with the fix in eb21083 (filter bounds of another type than the indexed property no longer used as a key range) and the merges of main described above. CI green, full suite 2673 / 0 / 4 skipped. The optimizer-level issue is tracked in #222.