Repository navigation
NCBC-4326: Add custom score and filter search queries - #239
Merged
Merged
Conversation
FIT performer imagePublished: ghcr.io/couchbase/dotnet-fit-performer:NCBC-4326Run FIT locally against this PR: fit run preset <preset-name> --performer dotnet-fit-performer:NCBC-4326Or run it from the workflow here. Note Each push to this PR replaces the image and edits this comment. The image is deleted 7 days after the last push. |
There was a problem hiding this comment.
🟡 Changes recommended
Missing global configuration bypasses custom-script capability validation, and custom-score/filter wrappers around Boolean queries dereference the outer message, causing conversion to throw before reaching the SDK.
2 open issues
What changed in this PR
Adds JavaScript-based custom scoring and filtering to the .NET SDK’s search APIs for Couchbase Server 8.5, with capability validation and explicit rejection over couchbase2.
Changes:
- Introduces uncommitted query types with optional fields, parameters, JSON export, and API documentation.
- Adds detection of custom queries in compound queries and vector prefilters, plus unsupported-query checks.
- Updates FIT protocol mappings and capabilities, with unit coverage for serialization and compatibility checks.
| File | Description |
|---|---|
| tests/Couchbase.UnitTests/Stellar/Search/StellarSearchClientTests.cs | Tests rejection of unsupported couchbase2 queries. |
| tests/Couchbase.UnitTests/Search/CustomScriptQueryTests.cs | Tests query export, validation, and configuration. |
| tests/Couchbase.UnitTests/Search/CustomScriptQueryCapabilityTests.cs | Tests nested detection and capability enforcement. |
| tests/Couchbase.UnitTests/Core/Configuration/Server/BucketConfigTests.cs | Covers the new search capability. |
| src/Couchbase/Stellar/Search/StellarSearchClient.cs | Rejects custom and unmapped query types. |
| src/Couchbase/Search/Queries/Script/CustomScriptQueryDetection.cs | Detects custom queries throughout search requests. |
| src/Couchbase/Search/Queries/Script/CustomScriptQuery.cs | Shares query state and JSON serialization. |
| src/Couchbase/Search/Queries/Script/CustomScoreQuery.cs | Adds the custom scoring API and documentation. |
| src/Couchbase/Search/Queries/Script/CustomFilterQuery.cs | Adds the custom filtering API and documentation. |
| src/Couchbase/Core/Configuration/Server/BucketConfig.cs | Defines search.udfQuery. |
| src/Couchbase/Cluster.cs | Adds capability validation and a testing seam. |
| fit/Couchbase.FitPerformer/Workload/SearchHelper.cs | Maps custom queries and structured parameters. |
| fit/Couchbase.FitPerformer/PerformerService.cs | Advertises custom-script query support. |
| fit/Couchbase.FitPerformer/gRPC/sdk.search.proto | Defines custom query protocol messages. |
| fit/Couchbase.FitPerformer/gRPC/sdk.caps.proto | Syncs capability definitions, including custom queries. |
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Custom-score and custom-filter queries with missing inner queries can trigger infinite recursion and a fatal StackOverflowException instead of returning a validation error.
1 open issue
2 resolved since last review
Review effort: Balanced
emilienbev
reviewed
Oct 30, 2026
Motivation ========== SDK-RFC 52 revision 15 adds two search query types that pass the results of an inner query through a user-defined JavaScript scoring or filtering function (Couchbase Server 8.5, Enterprise Edition). The SDK must raise FeatureNotAvailableException, before sending anything, when the cluster lacks the search.udfQuery capability, and always over couchbase2. Modification ============ Add CustomScoreQuery and CustomFilterQuery, marked Uncommitted. Both omit fields and params when empty and copy the caller's collections. Boost() throws InvalidArgumentException: FTS silently ignores a boost on these types, so it belongs on the inner query. The API docs say what the function receives, including that doc.fields holds analysed index terms, not stored text. Cluster.SearchAsync asserts search.udfQuery when the request holds one of these anywhere: in compound queries or in a vector query prefilter. This covers SearchQueryAsync and Scope.SearchAsync too. StellarSearchClient rejects them, and its query converter now throws for any type it cannot map; it used to send those as an empty query. The FIT performer maps both query types and declares SDK_SEARCH_CUSTOM_SCRIPT_QUERIES. Results ======= New unit tests cover the JSON, detection, the capability check and the couchbase2 paths; each check's tests fail when it is removed. Locally, FIT SearchCustomScriptQueryTest passes on 8.5.0-1308 and SearchCustomScriptQueryUnsupportedTest on 8.0.0.
Motivation ========== SearchHelper.ProtoSearchQueryToCore read boost and shouldMin from the root query's Boolean message, not the nested one. When the root is not a Boolean, the accessor returns null and the conversion throws, so a custom score or filter query that wraps a Boolean never reaches the SDK. When the root is a Boolean, a nested Boolean took its settings. Modification ============ Read both options from innerProtoQuery.Boolean. Results ======= Nested Boolean queries convert with their own boost and shouldMin.
Motivation ========== ProtoSearchQueryToCore falls back to the root query when the inner query is null. A custom score or filter request that leaves its query field unset therefore recursed until a StackOverflowException killed the performer. Modification ============ Throw UnsupportedException in both branches when the inner query is missing. Results ======= A malformed request fails with an error and the performer keeps running.
Motivation ========== Review asked for Boost() to match Java, shorter docs that describe less server behaviour, a better exception for an empty source, and a note that parameters use Newtonsoft. Modification ============ Boost() now does nothing and is marked [Obsolete], as in Java, so code that boosts every clause does not fail only in .NET. A null source throws ArgumentNullException and an empty or whitespace source throws ArgumentException. Shorten the Fields remarks, document that Parameters are serialized with Newtonsoft.Json, and tidy two Stellar comments. Results ======= Unit tests updated for Boost and the source checks.
emilienbev
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The issue in brief
Couchbase Server 8.5 lets a search query pass its results through a small JavaScript function written by the user. One kind of function changes each result's score. The other kind decides which results to keep. The SDK had no way to send these queries.
On a server that does not support the feature, the SDK must refuse the query before it sends anything. Couchbase2 (CNG) does not support the feature at all.
Change
CustomScoreQueryandCustomFilterQuery(SDK-RFC 52 rev 15), markedUncommitted. Each takes an inner query and the JavaScript source, plus optionalFields(...)andParameters(...).Boost()does nothing and is marked[Obsolete], as in Java. Against 8.5.0-1308, FTS accepts abooston these types but ignores it, so the boost belongs on the inner query.Cluster.SearchAsyncasserts thesearch.udfQuerycluster capability when the request contains a custom query anywhere, including inside compound queries and vector query prefilters.SearchQueryAsyncandScope.SearchAsyncgo through the same path.StellarSearchClientthrowsFeatureNotAvailableExceptionfor custom queries. Its query converter now also throws for any query type it cannot map. Before, such a type was sent as an empty query. This also affects a user's ownISearchQueryimplementation over couchbase2.doc._id,doc.score,doc.fields,params) and that the feature is off on the server by default. TheFields(...)docs explain thatdoc.fieldsholds the analyzer's terms, not the stored text: lowercased, de-duplicated and sorted, a string for one term and an array for several. They also say to use thekeywordanalyzer for exact values.SDK_SEARCH_CUSTOM_SCRIPT_QUERIESdeclared.Tests
SearchCustomScriptQueryTeston 8.5.0-1308: 51/51 passed.SearchCustomScriptQueryUnsupportedTeston 8.0.0: 38 passed, 1 expected skip (vector prefilter is not available throughSearchQueryAsync).op-multi-func-release(run 40755764646), from the JUnit reports:SearchCustomScriptQueryTest51/51 passed.SearchCustomScriptQueryUnsupportedTest13 skipped, as expected on a server with the capability.SearchCustomScriptQueryUnsupportedTest38 passed, 1 expected skip.SearchCustomScriptQueryTest17 skipped, as expected. No failures.ThreadSafetyMixedOpsGoldenPathTest.inserts: an ATR entry was left behind. This test is a known flake in the nightlies (6 failures in 68 runs).TtlPreservationTest.queryModeKvReplaceDoesNotPreserveTtl: the document kept its expiry after a query-mode replace. The test names the server issue MB-70540 as a possible cause.🤖 Generated with Claude Code