fix(idempotency): store and compare idempotency keys exactly on SQL Server and MySQL

decisions/2026-09-29-exact-idempotency-key-storage.md

@@ -0,0 +1,60 @@
+---
+authors:
+  - Martin Stühmer
+
+applyTo:
+  - "src/NetEvolve.Pulse.Extensibility/Idempotency/*.cs"
+  - "src/NetEvolve.Pulse/Idempotency/*.cs"
+  - "src/**/Idempotency/*.cs"
+  - "src/**/Scripts/IdempotencyKey.sql"
+  - "src/NetEvolve.Pulse.EntityFramework/Configurations/*IdempotencyKeyConfiguration*.cs"
+
+created: 2026-09-29
+
+lastModified: 2026-09-29
+
+state: proposed
+
+instructions: |
+  IdempotencyKeySchema.MaxLengths.IdempotencyKey is 450 characters, so NVARCHAR(450) (900 bytes) fits the SQL Server clustered index key limit.
+  IdempotencyStore and the SQL Server, MySQL and Entity Framework repositories MUST reject a key longer than MaxLengths.IdempotencyKey with an ArgumentException before any database call. A key MUST never be truncated.
+  Key columns MUST use a binary collation: Latin1_General_100_BIN2 on SQL Server and utf8mb4_bin on MySQL, in the provider scripts and in the EF Core configurations. Keys are case-sensitive and accent-sensitive on every provider.
+  Provider code MUST NOT use statements that turn data errors into warnings (MySQL INSERT IGNORE). Duplicate keys are detected by catching the provider's duplicate-key error.
+---
+
+# Decision: Store and Compare Idempotency Keys Exactly
+
+Idempotency keys are stored and compared exactly as the client sent them. The maximum length is 450 characters, keys that are too long are rejected instead of truncated, and the key columns use binary collations.
+
+## Context
+
+* `IdempotencyKeySchema.MaxLengths.IdempotencyKey` was 500. On SQL Server the key column was `NVARCHAR(500)`, and it is also the clustered primary key. [CREATE INDEX, Index key size](https://learn.microsoft.com/sql/t-sql/statements/create-index-transact-sql#index-key-size) states: "The maximum size for an index key is 900 bytes for a clustered index and 1,700 bytes for a nonclustered index." `NVARCHAR` needs 2 bytes per character, so keys of 451 to 500 characters failed with Msg 1946 (#850).
+* SqlClient silently cuts a parameter value to its `Size`. MySQL `INSERT IGNORE` turns `ER_DATA_TOO_LONG` into a warning and stores a prefix ([MySQL 8.0, INSERT](https://dev.mysql.com/doc/refman/8.0/en/insert.html): "With `IGNORE`, invalid values are adjusted to the closest values and inserted"). Two keys that share a long prefix collided, or a stored key was never found again (#858).
+* The MySQL column used `utf8mb4_unicode_ci`, and the SQL Server column used the database default collation. Both are usually case-insensitive, so `aBc123` and `ABC123` counted as the same key. Base62 and base64 client keys differ only by case all the time. PostgreSQL, SQLite, Redis and EF InMemory already compare keys exactly.
+
+## Decision
+
+* `MaxLengths.IdempotencyKey` is 450 for every provider. The SQL Server column and stored procedure parameters are `NVARCHAR(450)`.
+* `IdempotencyStore` (`ExistsAsync`, `StoreAsync`, `TryReserveAsync`) and the SQL Server, MySQL and Entity Framework repositories reject a longer key with `ArgumentOutOfRangeException` (an `ArgumentException`) before any database call.
+* SQL Server uses `Latin1_General_100_BIN2`. [Collation and Unicode support](https://learn.microsoft.com/sql/relational-databases/collations/collation-and-unicode-support) describes BIN2 as "a pure code-point comparison". MySQL uses `utf8mb4_bin`. The [MySQL 8.0 manual](https://dev.mysql.com/doc/refman/8.0/en/charset-binary-collations.html) says that for `_bin` collations "ordering is based on numeric character code values". The EF Core configurations set the same collations with `UseCollation`.
+* MySQL stores keys with a plain `INSERT` and treats error 1062 (`ER_DUP_ENTRY`) as a duplicate. `INSERT ... ON DUPLICATE KEY UPDATE` is not used, because MySql.Data reports found rows by default, so a duplicate would also report one affected row and `TryReserveAsync` would always return `true`.
+* The provider scripts upgrade existing tables on re-run. SQL Server drops the primary key, alters the column and re-creates the primary key in one transaction. MySQL changes the column collation through the `information_schema` guard pattern of the re-runnable MySQL scripts decision. The MySQL column stays `VARCHAR(500)`, because existing rows may already be longer than 450 characters. The core guard still enforces the 450 limit.
+
+## Consequences
+
+* Every key up to 450 characters works on every provider, and a longer key fails fast with a clear exception instead of a `SqlException` or a silent truncation.
+* Keys that differ only by case are distinct on every provider.
+* Trailing spaces are still ignored on SQL Server and MySQL. SQL Server pads strings before every `=` comparison, whatever the collation ([= (String comparison)](https://learn.microsoft.com/sql/t-sql/language-elements/string-comparison-assignment)), and `utf8mb4_bin` is a `PAD SPACE` collation. `utf8mb4_0900_bin` (`NO PAD`) was not chosen, because it would still differ from SQL Server and MariaDB does not provide it.
+* The public constant drops from 500 to 450. Code that compiled against the old value keeps 500 until it is recompiled.
+* EF Core users need a new migration for the column type and collation. ADO.NET users re-run the provider script.
+
+## Alternatives Considered
+
+* **`PRIMARY KEY NONCLUSTERED` with the 1,700-byte limit**: keeps 500 characters on SQL Server, but needs a separate clustered index or a heap, and the limit would differ between providers.
+* **Hash the key into a fixed-length column**: allows any length and exact comparison including trailing spaces, but changes the schema contract of every provider and makes stored keys unreadable for operators.
+* **Case-sensitive, accent-sensitive linguistic collations** (for example `Latin1_General_100_CS_AS`): still apply linguistic equivalence rules, so they are not an exact comparison.
+
+## Related Decisions (Optional)
+
+* [Refresh Expired Idempotency Keys on Reserve](2026-09-28-idempotency-refresh-expired-keys-on-reserve.md)
+* [Re-runnable MySQL Schema Scripts](2026-09-28-rerunnable-mysql-schema-scripts.md)

src/NetEvolve.Pulse.EntityFramework/Configurations/MySqlIdempotencyKeyConfiguration.cs

@@ -12,7 +12,7 @@ namespace NetEvolve.Pulse.Configurations;
 /// <remarks>
 /// <para><strong>Column Types:</strong></para>
 /// <list type="bullet">
-/// <item><description><c>varchar(500)</c> for the idempotency key</description></item>
+/// <item><description><c>varchar(500)</c> with the binary collation <c>utf8mb4_bin</c> for the idempotency key</description></item>
 /// <item><description><c>bigint</c> for <see cref="DateTimeOffset"/> — stored as UTC ticks via a <see langword="long"/> value converter</description></item>
 /// </list>
 /// <para><strong>Why bigint for DateTimeOffset:</strong></para>
@@ -23,6 +23,11 @@ namespace NetEvolve.Pulse.Configurations;
 /// </remarks>
 internal sealed class MySqlIdempotencyKeyConfiguration : IdempotencyKeyConfigurationBase
 {
+    /// <summary>
+    /// The column collation annotation of the Oracle provider (<c>MySql.EntityFrameworkCore</c>).
+    /// </summary>
+    private const string MySqlCollationAnnotation = "MySQL:Collation";
+
     /// <summary>
     /// Initializes a new instance of the <see cref="MySqlIdempotencyKeyConfiguration"/> class with default options.
     /// </summary>
@@ -39,7 +44,17 @@ public MySqlIdempotencyKeyConfiguration(IOptions<IdempotencyKeyOptions> options)
     /// <inheritdoc />
     protected override void ApplyColumnTypes(EntityTypeBuilder<IdempotencyKey> builder)
     {
-        _ = builder.Property(k => k.Key).HasColumnType("varchar(500)");
+        // The binary collation compares keys by code point, so keys differing only by case stay distinct.
+        // The Oracle provider ignores the relational collation set by UseCollation and reads its own
+        // "MySQL:Collation" annotation instead, so both are set.
+        // The model keeps MaxLength 500, so migrations created by earlier releases stay in sync;
+        // the store and the repository enforce IdempotencyKeySchema.MaxLengths.IdempotencyKey.
+        _ = builder
+            .Property(k => k.Key)
+            .HasColumnType("varchar(500)")
+            .HasMaxLength(500)
+            .UseCollation("utf8mb4_bin")
+            .HasAnnotation(MySqlCollationAnnotation, "utf8mb4_bin");
 
         // DateTimeOffset is stored as BIGINT (UTC ticks).
         // The Oracle MySQL provider lacks a proper DateTimeOffset type mapping for

src/NetEvolve.Pulse.EntityFramework/Configurations/PostgreSqlIdempotencyKeyConfiguration.cs

@@ -27,7 +27,9 @@ public PostgreSqlIdempotencyKeyConfiguration(IOptions<IdempotencyKeyOptions> opt
     /// <inheritdoc />
     protected override void ApplyColumnTypes(EntityTypeBuilder<IdempotencyKey> builder)
     {
-        _ = builder.Property(k => k.Key).HasColumnType("character varying(500)");
+        // The model keeps MaxLength 500, so migrations created by earlier releases stay in sync;
+        // the store and the repository enforce IdempotencyKeySchema.MaxLengths.IdempotencyKey.
+        _ = builder.Property(k => k.Key).HasColumnType("character varying(500)").HasMaxLength(500);
         // "timestamp with time zone" (timestamptz) preserves UTC correctly.
         _ = builder.Property(k => k.CreatedAt).HasColumnType("timestamp with time zone");
     }

src/NetEvolve.Pulse.EntityFramework/Configurations/SqlServerIdempotencyKeyConfiguration.cs

@@ -3,6 +3,7 @@ namespace NetEvolve.Pulse.Configurations;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.EntityFrameworkCore.Metadata.Builders;
 using Microsoft.Extensions.Options;
+using NetEvolve.Pulse.Extensibility.Idempotency;
 using NetEvolve.Pulse.Idempotency;
 
 /// <summary>
@@ -27,7 +28,12 @@ public SqlServerIdempotencyKeyConfiguration(IOptions<IdempotencyKeyOptions> opti
     /// <inheritdoc />
     protected override void ApplyColumnTypes(EntityTypeBuilder<IdempotencyKey> builder)
     {
-        _ = builder.Property(k => k.Key).HasColumnType("nvarchar(500)");
+        // 450 NVARCHAR characters take 900 bytes, the SQL Server limit for a clustered index key.
+        // The binary collation compares keys by code point, so keys differing only by case stay distinct.
+        _ = builder
+            .Property(k => k.Key)
+            .HasColumnType($"nvarchar({IdempotencyKeySchema.MaxLengths.IdempotencyKey})")
+            .UseCollation("Latin1_General_100_BIN2");
         _ = builder.Property(k => k.CreatedAt).HasColumnType("datetimeoffset");
     }
 }

src/NetEvolve.Pulse.EntityFramework/Configurations/SqliteIdempotencyKeyConfiguration.cs

@@ -32,7 +32,9 @@ public SqliteIdempotencyKeyConfiguration(IOptions<IdempotencyKeyOptions> options
     /// <inheritdoc />
     protected override void ApplyColumnTypes(EntityTypeBuilder<IdempotencyKey> builder)
     {
-        _ = builder.Property(k => k.Key).HasColumnType("TEXT");
+        // The model keeps MaxLength 500, so migrations created by earlier releases stay in sync;
+        // the store and the repository enforce IdempotencyKeySchema.MaxLengths.IdempotencyKey.
+        _ = builder.Property(k => k.Key).HasColumnType("TEXT").HasMaxLength(500);
         // DateTimeOffset stored as INTEGER (UTC ticks) for correct ordering in SQLite.
         _ = builder
             .Property(k => k.CreatedAt)

src/NetEvolve.Pulse.EntityFramework/Idempotency/EntityFrameworkIdempotencyKeyRepository{TContext}.cs

@@ -44,6 +44,10 @@ public Task<bool> ExistsAsync(
         cancellationToken.ThrowIfCancellationRequested();
 
         ArgumentException.ThrowIfNullOrWhiteSpace(idempotencyKey);
+        ArgumentOutOfRangeException.ThrowIfGreaterThan(
+            idempotencyKey.Length,
+            IdempotencyKeySchema.MaxLengths.IdempotencyKey
+        );
 
         if (validFrom.HasValue)
         {
@@ -66,6 +70,10 @@ public async Task StoreAsync(
         cancellationToken.ThrowIfCancellationRequested();
 
         ArgumentException.ThrowIfNullOrWhiteSpace(idempotencyKey);
+        ArgumentOutOfRangeException.ThrowIfGreaterThan(
+            idempotencyKey.Length,
+            IdempotencyKeySchema.MaxLengths.IdempotencyKey
+        );
 
         _ = await TryInsertAsync(idempotencyKey, createdAt, cancellationToken).ConfigureAwait(false);
     }
@@ -87,6 +95,10 @@ public async Task<bool> TryReserveAsync(
         cancellationToken.ThrowIfCancellationRequested();
 
         ArgumentException.ThrowIfNullOrWhiteSpace(idempotencyKey);
+        ArgumentOutOfRangeException.ThrowIfGreaterThan(
+            idempotencyKey.Length,
+            IdempotencyKeySchema.MaxLengths.IdempotencyKey
+        );
 
         if (validFrom.HasValue)
         {

src/NetEvolve.Pulse.EntityFramework/Idempotency/IdempotencyKey.cs

@@ -12,7 +12,7 @@ namespace NetEvolve.Pulse.Idempotency;
 /// to ensure interchangeability.
 /// <para><strong>Column Specifications:</strong></para>
 /// <list type="bullet">
-/// <item><description><see cref="Key"/>: VARCHAR(500), Primary Key — the client-supplied idempotency key.</description></item>
+/// <item><description><see cref="Key"/>: up to <see cref="Extensibility.Idempotency.IdempotencyKeySchema.MaxLengths.IdempotencyKey"/> characters, compared case-sensitively (binary collation on SQL Server and MySQL), Primary Key — the client-supplied idempotency key.</description></item>
 /// <item><description><see cref="CreatedAt"/>: DATETIMEOFFSET, NOT NULL — timestamp when the key was stored.</description></item>
 /// </list>
 /// </remarks>

src/NetEvolve.Pulse.EntityFramework/README.md

@@ -313,6 +313,13 @@ services.AddPulse(config => config
 );
 ```
 
+## Idempotency Key Length and Case Sensitivity
+
+- Keys can be up to 450 characters long (`IdempotencyKeySchema.MaxLengths.IdempotencyKey`). The store and the repository reject a longer key with an `ArgumentOutOfRangeException` before it touches the database.
+- On SQL Server the key column is `nvarchar(450)` with the collation `Latin1_General_100_BIN2`, because a clustered index key is limited to 900 bytes. On MySQL it is `varchar(500)` with the collation `utf8mb4_bin`. Keys are case-sensitive on every provider.
+
+Existing SQL Server and MySQL databases created with an earlier release need a new migration for these column changes (`dotnet ef migrations add IdempotencyKeyExactComparison`). Keys longer than 450 characters could never be stored in the old SQL Server column, so no key is truncated. The PostgreSQL (`character varying(500)`) and SQLite (`TEXT`) models are unchanged and need no migration.
+
 ## Processing Lease Reclaim
 
 A message claimed by `GetPendingAsync` stays in `Processing` until it is completed or failed. If a worker crashes or shuts down in between, the next pending poll reclaims the message once its `UpdatedAt` is older than `OutboxOptions.ProcessingLeaseTimeout` (default: 5 minutes, must be greater than zero).

src/NetEvolve.Pulse.Extensibility/Idempotency/IIdempotencyStore.cs

@@ -47,7 +47,10 @@ public interface IIdempotencyStore
     /// Determines whether the specified idempotency key has already been stored,
     /// indicating that the corresponding command was previously processed.
     /// </summary>
-    /// <param name="idempotencyKey">The idempotency key to look up. Must not be <see langword="null"/> or empty.</param>
+    /// <param name="idempotencyKey">The idempotency key to look up. Must not be <see langword="null"/> or empty.
+    /// Keys longer than <see cref="IdempotencyKeySchema.MaxLengths.IdempotencyKey"/> characters are not supported;
+    /// the built-in store rejects them with an <see cref="ArgumentOutOfRangeException"/>. Keys are compared case-sensitively.
+    /// </param>
     /// <param name="cancellationToken">A token to monitor for cancellation requests.</param>
     /// <returns>
     /// <see langword="true"/> if the key is already present in the store; otherwise <see langword="false"/>.
@@ -58,7 +61,10 @@ public interface IIdempotencyStore
     /// Persists the specified idempotency key so that future calls to <see cref="ExistsAsync"/>
     /// with the same key return <see langword="true"/>.
     /// </summary>
-    /// <param name="idempotencyKey">The idempotency key to store. Must not be <see langword="null"/> or empty.</param>
+    /// <param name="idempotencyKey">The idempotency key to store. Must not be <see langword="null"/> or empty.
+    /// Keys longer than <see cref="IdempotencyKeySchema.MaxLengths.IdempotencyKey"/> characters are not supported;
+    /// the built-in store rejects them with an <see cref="ArgumentOutOfRangeException"/>. Keys are compared case-sensitively.
+    /// </param>
     /// <param name="cancellationToken">A token to monitor for cancellation requests.</param>
     /// <returns>A task representing the asynchronous store operation.</returns>
     Task StoreAsync(string idempotencyKey, CancellationToken cancellationToken = default);
@@ -67,7 +73,10 @@ public interface IIdempotencyStore
     /// Attempts to reserve the specified idempotency key before the corresponding command executes,
     /// so that concurrent or subsequent submissions of the same key are rejected.
     /// </summary>
-    /// <param name="idempotencyKey">The idempotency key to reserve. Must not be <see langword="null"/> or empty.</param>
+    /// <param name="idempotencyKey">The idempotency key to reserve. Must not be <see langword="null"/> or empty.
+    /// Keys longer than <see cref="IdempotencyKeySchema.MaxLengths.IdempotencyKey"/> characters are not supported;
+    /// the built-in store rejects them with an <see cref="ArgumentOutOfRangeException"/>. Keys are compared case-sensitively.
+    /// </param>
     /// <param name="cancellationToken">A token to monitor for cancellation requests.</param>
     /// <returns>
     /// <see langword="true"/> if the key was newly reserved and the command may execute;

src/NetEvolve.Pulse.Extensibility/Idempotency/IIdempotentCommand{TResponse}.cs

@@ -32,7 +32,9 @@ public interface IIdempotentCommand<TResponse> : ICommand<TResponse>
     /// Gets the client-supplied idempotency key that uniquely identifies this logical operation.
     /// </summary>
     /// <remarks>
-    /// The key MUST be non-<see langword="null"/> and non-empty.
+    /// The key MUST be non-<see langword="null"/> and non-empty, and MUST NOT be longer than
+    /// <see cref="IdempotencyKeySchema.MaxLengths.IdempotencyKey"/> characters.
+    /// Keys are compared case-sensitively, so keys that differ only by case are distinct.
     /// </remarks>
     string IdempotencyKey { get; }
 }

src/NetEvolve.Pulse.Extensibility/Idempotency/IdempotencyKeySchema.cs

@@ -50,8 +50,13 @@ public static class Columns
     public static class MaxLengths
     {
         /// <summary>
-        /// Maximum length for the IdempotencyKey column (500 characters).
+        /// Maximum length for the IdempotencyKey column (450 characters).
         /// </summary>
-        public const int IdempotencyKey = 500;
+        /// <remarks>
+        /// 450 characters of <c>NVARCHAR</c> take 900 bytes, the SQL Server limit for a clustered index key.
+        /// Longer keys are rejected with an <see cref="System.ArgumentException"/>; keys are never truncated.
+        /// Keys are compared exactly, including their case.
+        /// </remarks>
+        public const int IdempotencyKey = 450;
     }
 }