From 0b5a8d110f19bd5fd38b04458379a351b321146f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20Gro=C3=9F?= Date: Tue, 22 Sep 2026 20:54:10 +0200 Subject: [PATCH 1/3] fix: render JSON-member index parts as Npgsql's queries do; open 5.4.0 PostgreSQL uses an expression index only for a query whose expression matches it. Every JSON member was rendered as "doc" -> 'A' ->> 'B' text with no cast, which matched Npgsql's query translation only for a top-level string: indexes on nested members (#>> '{A,B}' in the query) or typed members (CAST(... AS integer)) applied, enforced uniqueness and served no query. Members now render exactly as NpgsqlQuerySqlGenerator renders them (identical in Npgsql 10 and 11), in index parts, typed expression indexes and index filters. Parts are resolved at diff time on both sides, so a rendering change alone never produces a migration. Every declaration now writes CustomIndex:RenderingVersion onto the model; a snapshot without it is rendered by the old rules, which turns the fix into one drop-and-create per affected index. Default names come from the path (NameToken), and ResolvedIndexPart equality ignores the derived metadata, so nothing is renamed or rebuilt without a reason. Exclusion constraint filters keep the old rendering. Date and time members stay text (their casts are not IMMUTABLE); a non-unique index leading with one is rejected at migrations add. Guards: rendering asserted against EF's own ToQueryString(); rollout on a real compiled snapshot; EXPLAIN on PostgreSQL 18 with seqscan off. Reverting each of the six parts of the fix turns its tests red. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 + CLAUDE.md | 29 +- Directory.Build.props | 4 +- README.md | 2 +- SECURITY.md | 4 +- docs/postgresql-indexes.md | 30 +- .../CHANGELOG.md | 14 + ...NpgsqlComplexIndexMigrationsModelDiffer.cs | 123 ++++- .../README.md | 9 +- src/EFCore.ComplexIndexes/CHANGELOG.md | 12 + .../ComplexIndexAnnotations.cs | 13 + .../ComplexIndexExtensions.cs | 2 + .../ComplexIndexStorage.cs | 22 +- .../CustomMigrationsModelDiffer.cs | 73 ++- .../IndexPartDefinition.cs | 7 + .../ResolvedIndexPart.cs | 38 +- .../DocumentationApiTests.cs | 4 +- .../NpgsqlJsonMemberIndexTests.cs | 7 +- .../NpgsqlJsonMemberRenderingTests.cs | 441 ++++++++++++++++++ .../PostgresIntegrationTests.cs | 102 ++++ .../SnapshotRoundTripTests.cs | 57 ++- 21 files changed, 955 insertions(+), 46 deletions(-) create mode 100644 test/EFCore.ComplexIndexes.Tests/NpgsqlJsonMemberRenderingTests.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index e4eefd5..dab872f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,14 @@ covering only what changed for that package: [PostgreSQL](src/EFCore.ComplexIndexes.PostgreSQL/CHANGELOG.md), [SQL Server](src/EFCore.ComplexIndexes.SqlServer/CHANGELOG.md). +## 5.4.0 + +Fixes to what 5.x already ships, found while testing the packages against EF Core 11: indexes on +JSON members that no query could use. + +- **Fixed:** indexes on members of a `ToJson()` complex property are written the way Npgsql's queries read the member, so PostgreSQL can use them. It uses an expression index only for a query whose expression matches, and until now every member was extracted as `"doc" -> 'A' ->> 'B'` text with no cast, which matched only a top-level string. A nested member (`#>> '{Address,City}'` in the query) or a typed one (`CAST("doc" ->> 'Rank' AS integer)`) got an index that applied cleanly, enforced uniqueness, and was never used by a single query. Members now render as Npgsql renders them — `->>` or `#>>`, cast to the member's store type unless it is a string, `decode(…, 'base64')` for `byte[]`, `jsonb` for a primitive collection — in index parts, typed expression indexes and index filters. That also makes a filter such as `x => x.Profile.Rank > 5` valid SQL: it compared text with an integer and failed at apply time. Default index names do not change. **Upgrading:** the first `migrations add` drops and re-creates each affected index under its existing name; indexes on top-level string members are untouched. Until that migration exists, `has-pending-model-changes` reports changes and `Migrate()` raises EF Core's pending-model-changes error. The re-create blocks writes while it builds, so on a large table declare the index with `IsCreatedConcurrently()` first. Exclusion constraint filters keep their rendering. The snapshot gains one `CustomIndex:RenderingVersion` annotation, which records the rules a model was declared under and is what lets the change reach databases built by earlier versions. +- **Changed:** a non-unique index that starts with a date or time JSON member (`DateTime`, `DateTimeOffset`, `DateOnly`, `TimeOnly`) is rejected at `migrations add`. EF Core's queries cast such a member to `timestamptz`, `date` or `time`, and PostgreSQL cannot index those casts (the conversion from text is not IMMUTABLE), so no query could ever use the index. A unique one is still allowed and enforces uniqueness on the stored text; a member in a later position leaves the index usable through the parts before it. + ## 5.3.0 One fix, found the first time a 5.2.0 converter-member path met the model snapshot diff --git a/CLAUDE.md b/CLAUDE.md index dc38b65..3fa105b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -360,13 +360,32 @@ differ let satellites resolve what the core cannot: - `IsForwardedIndexAnnotation` — the annotation whitelist (see below). - `ResolveUnmappedPart` — a path with no table column; the Npgsql differ builds a JSON extraction - (`"col" -> 'A' ->> 'B'`) when the path traverses a `ToJson()` complex property, honoring - `HasJsonPropertyName`. Members extract as text — no automatic casts (text→timestamptz casts are - not IMMUTABLE and would blow up `CREATE INDEX`). A path that *ends* at the JSON-mapped complex + when the path traverses a `ToJson()` complex property, honoring `HasJsonPropertyName`, rendered + **exactly as Npgsql's query translation renders the member** (`NpgsqlQuerySqlGenerator.VisitJsonScalar` + and `GenerateJsonPath`, identical in Npgsql 10 and 11): `->>` for one step, `#>> '{A,B}'` for + more (`ARRAY[…]::text[]` when a segment is not ASCII-alphanumeric), and a `CAST` to the member's + store type unless it is a string mapping, `decode(…, 'base64')` for `bytea`, `jsonb` for a + primitive collection. PostgreSQL uses an expression index only for a query whose expression + matches it, so this is not cosmetic: until 5.4.0 every member was `"col" -> 'A' ->> 'B'` text, + and indexes on nested or typed members applied, enforced, and served no query + (`NpgsqlJsonMemberRenderingTests` asserts each rendering against EF's own `ToQueryString()`). + Date and time members stay text — their casts are not IMMUTABLE and cannot appear in an index — + and the part records a `TextFallback`; `ValidateQueryUsableLeadingParts` rejects a non-unique + index that starts with one (target model only). A path that *ends* at the JSON-mapped complex property — or at a complex collection, which is always JSON — resolves to the container column as a plain **column** part (so a whole-document GIN needs no runtime wiring); a complex property - nested inside the document resolves to a `->` extraction yielding `jsonb`. A table-split complex - property stays unresolved: there is no single column to stand for it. + nested inside the document resolves to a `jsonb` extraction. A table-split complex property + stays unresolved: there is no single column to stand for it. + + **Rendering changes need `ComplexIndexAnnotations.RenderingVersion`.** Parts are resolved at diff + time on *both* sides, so changing how a member renders produces no migration at all — both sides + resolve alike and existing databases keep the old index forever. Every declaration writes the + version onto the model; a snapshot without it (scaffolded before 5.4.0) is resolved by the old + rules, which is what turns the change into one drop-and-create per affected index. Default index + names come from `ResolvedIndexPart.NameToken` (container plus path), never from the rendered SQL, + so a rendering change never renames an index; and `ResolvedIndexPart` equality ignores both + internal members, or identical SQL would still rebuild. Exclusion constraint filters pass + `renderingVersion: 1` and keep the old rendering: rebuilding a constraint buys no enforcement. - `ResolveTemplatePart` — substitutes template placeholders with quoted columns or parenthesized JSON extractions. Since 5.2.0 the core implements it, quoting through the `QuoteIdentifier` virtual (ANSI by default; the SQL Server satellite brackets), so satellites override neither. diff --git a/Directory.Build.props b/Directory.Build.props index d224267..96ef5bb 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -1,6 +1,6 @@ - 5.3.0 + 5.4.0 CaffeinatedCoder MIT true @@ -63,7 +63,7 @@ --> true - 5.2.0 + 5.3.0