Skip to content

fix(sqlserver): the idempotency key column exceeds the 900-byte clustered index key limit in NetEvolve.Pulse.SqlServer #850

Description

@samtrion

User Story

As a developer using the SQL Server or Entity Framework Core idempotency store, I want every idempotency key up to the advertised maximum length to be stored reliably, and longer keys to be rejected, so that commands with long keys still run, and two different keys are never treated as the same one.


Problem

In src/NetEvolve.Pulse.SqlServer/Scripts/IdempotencyKey.sql, line 33 declares [IdempotencyKey] NVARCHAR(500) NOT NULL and line 35 makes it PRIMARY KEY CLUSTERED ([IdempotencyKey]). NVARCHAR needs 2 bytes per character, so the key can take up to 1,000 bytes, which is more than SQL Server's 900-byte limit for a clustered index key. SQL Server creates the table anyway and only raises warning 1945.

Failure scenario:

  1. A command uses an idempotency key of 451 to 500 characters. That is within the advertised IdempotencyKeySchema.MaxLengths.IdempotencyKey = 500 (src/NetEvolve.Pulse.Extensibility/Idempotency/IdempotencyKeySchema.cs:55).
  2. IdempotencyCommandInterceptor calls TryReserveAsync, which calls ExistsAsync and then StoreAsync.
  3. The MERGE ... INSERT in usp_InsertIdempotencyKey (IdempotencyKey.sql:92 onwards) fails with Msg 1946 ("index entry of length N bytes ... exceeds the maximum length of 900 bytes").
  4. IsDuplicateKeyException (src/NetEvolve.Pulse.SqlServer/Idempotency/SqlServerIdempotencyKeyRepository.cs:158) only matches errors 2627 and 2601, so the catch at line 126 does not handle the error. The SqlException reaches the caller, and the command handler never runs for that key.

Keys longer than 500 characters are silently truncated:

  • SqlServerIdempotencyKeyRepository.cs:84 (ExistsAsync) and :116 (StoreAsync) bind new SqlParameter("@idempotencyKey", SqlDbType.NVarChar, 500). SqlClient truncates any value longer than Size.
  • The stored procedure parameters at IdempotencyKey.sql:56 and :92 are also NVARCHAR(500).
  • The repository, IdempotencyStore, and the interceptor only check for null or whitespace. None of them checks the length.

As a result, two different keys that share their first 500 characters are treated as the same key. The second command is reported as a duplicate and is silently skipped.

The EF Core provider has the same defect when it runs on SQL Server. src/NetEvolve.Pulse.EntityFramework/Configurations/IdempotencyKeyConfigurationBase.cs:70 applies .HasMaxLength(IdempotencyKeySchema.MaxLengths.IdempotencyKey) to the primary key column, which maps to nvarchar(500) as a clustered primary key.

No existing test uses a key longer than 450 characters.


Specification

CREATE INDEX (Transact-SQL), section "Index key size":

The maximum size for an index key is 900 bytes for a clustered index and 1,700 bytes for a nonclustered index. [...] Indexes on varchar columns that exceed the byte limit can be created if the existing data in the columns don't exceed the limit at the time the index is created; however, subsequent insert or update operations on the columns that cause the total size to be greater than the limit fail.

Maximum capacity specifications for SQL Server, rows "Bytes per index key" and "Bytes per primary key" (900 bytes):

The maximum number of bytes in a clustered index key can't exceed 900 [...] You can define a key using variable-length columns whose maximum sizes add up to more than the limit. However, the combined sizes of the data in those columns can never exceed the limit.


Requirements

  • Set IdempotencyKeySchema.MaxLengths.IdempotencyKey to 450, and use NVARCHAR(450) for the column (IdempotencyKey.sql:33) and the procedure parameters (:56, :92). The other option, PRIMARY KEY NONCLUSTERED with its 1,700-byte limit, only works on SQL Server 2016 and later. The 450 limit is simpler and the same for every provider.
  • Bind the @idempotencyKey parameters (SqlServerIdempotencyKeyRepository.cs:84, :116) with the size from IdempotencyKeySchema.MaxLengths.IdempotencyKey instead of the literal 500.
  • Reject keys longer than MaxLengths.IdempotencyKey with an ArgumentException before any database call, both in the store and in the repositories. Never truncate a key.
  • The EF Core configuration (IdempotencyKeyConfigurationBase.cs:70) follows the new central maximum. Check whether existing EF migrations or snapshots need a note.
  • Document how existing SQL Server deployments migrate, because the script only creates the table if it does not exist (ALTER TABLE ... ALTER COLUMN, or a note in the README or changelog).

Acceptance Criteria

  • A failing integration test comes first: storing and checking a key of 451 to 500 characters against SQL Server currently throws a SqlException (Msg 1946).
  • After the fix, keys of up to MaxLengths.IdempotencyKey characters are stored and found by the SQL Server provider and by the EF Core provider on SQL Server.
  • Keys longer than MaxLengths.IdempotencyKey throw an ArgumentException from ExistsAsync, StoreAsync and TryReserveAsync (unit tests).
  • Two keys that share a 450-character prefix but differ afterwards are never treated as the same key.
  • Creating the SQL Server table raises no warning 1945.
  • The migration steps for existing SQL Server tables are documented.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugIndicates an issue or flaw that needs to be fixed.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions