Skip to content

fix: Store and compare idempotency keys exactly (length guard, binary collation) in NetEvolve.Pulse.MySql and NetEvolve.Pulse.SqlServer #858

Description

@samtrion

User Story

As a developer using the Pulse idempotency interceptor with NetEvolve.Pulse.MySql, I want every idempotency key to be stored and compared exactly as I sent it, so that duplicates are always detected and distinct commands are never rejected as conflicts.


Problem

The MySQL key column and insert statement lose key exactness in two ways (references on main):

  • src/NetEvolve.Pulse.MySql/Scripts/IdempotencyKey.sql:30-33 declares `IdempotencyKey` VARCHAR(500) NOT NULL as the primary key with COLLATE=utf8mb4_unicode_ci.
  • src/NetEvolve.Pulse.MySql/Idempotency/MySqlIdempotencyKeyRepository.cs:84 stores keys with INSERT IGNORE INTO .... The key is bound with AddWithValue and no size (:134), and the only guard is ArgumentException.ThrowIfNullOrWhiteSpace (:97, :127). src/NetEvolve.Pulse/Idempotency/IdempotencyStore.cs:49,61 checks nothing else either. IdempotencyKeySchema.MaxLengths.IdempotencyKey = 500 (src/NetEvolve.Pulse.Extensibility/Idempotency/IdempotencyKeySchema.cs:55) is only used by the EF Core configuration.
  • src/NetEvolve.Pulse/Interceptors/IdempotencyCommandInterceptor.cs:77-79 calls TryReserveAsync. The default implementation runs ExistsAsync (= @key, repository :70) and then StoreAsync, and the interceptor throws IdempotencyConflictException when the key exists.

(a) Keys longer than 500 characters: idempotency is silently off. IGNORE turns ER_DATA_TOO_LONG into a warning, so MySQL stores only the first 500 characters. The next ExistsAsync compares the full key against that stored prefix and never finds a match, so the handler runs again for every duplicate. Two different long keys that share the same first 500 characters also collide.

(b) Keys that differ only by case: false conflicts. utf8mb4_unicode_ci ignores case and accents, and with PAD SPACE it also ignores trailing spaces. Base62 and base64 client keys differ by case all the time. Once aBc123 is stored, ExistsAsync("ABC123") returns true and a different command is rejected with IdempotencyConflictException.

PostgreSQL and SQLite compare keys exactly. Other providers are affected too:

  • NetEvolve.Pulse.SqlServer: SqlServerIdempotencyKeyRepository.cs:84 and :116 bind the key as new SqlParameter("@idempotencyKey", SqlDbType.NVarChar, 500), and SqlClient silently cuts longer values to 500 characters on both lookup and store. Keys that share the same first 500 characters then collide and cause false conflicts. The table in src/NetEvolve.Pulse.SqlServer/Scripts/IdempotencyKey.sql:33 sets no collation, so it uses the database default. That default is usually case-insensitive (for example SQL_Latin1_General_CP1_CI_AS).
  • EF Core on MySQL: src/NetEvolve.Pulse.EntityFramework/Configurations/MySqlIdempotencyKeyConfiguration.cs:42 maps varchar(500) without a collation, so it uses the database collation, which is usually case-insensitive.

Specification

MySQL 8.0 Reference Manual, 15.2.7 INSERT Statement:

Data conversions that would trigger errors abort the statement if IGNORE is not specified. With IGNORE, invalid values are adjusted to the closest values and inserted; warnings are produced but the statement does not abort.

MySQL 8.0 Reference Manual, Server SQL Modes – The Effect of IGNORE on Statement Execution lists ER_DATA_TOO_LONG among the errors that IGNORE turns into warnings, and states that "when the IGNORE keyword and strict SQL mode are both in effect, IGNORE takes precedence."

MySQL 8.0 Reference Manual, B.3.4.1 Case Sensitivity in String Searches:

nonbinary string comparisons are case-insensitive by default ... If you want a column always to be treated in case-sensitive fashion, declare it with a case-sensitive or binary collation.

Pulse contract: IdempotencyKeySchema.MaxLengths.IdempotencyKey (500) is the documented maximum key length. No external standard defines whether keys are case-sensitive. The defect is that providers behave differently from each other and that case-insensitive matching is unsafe for base62 and base64 keys.


Requirements

  • Reject keys longer than IdempotencyKeySchema.MaxLengths.IdempotencyKey once, in the core IdempotencyStore (ExistsAsync, StoreAsync and TryReserveAsync), with an ArgumentException, so every provider behaves the same. Relying on ON DUPLICATE KEY UPDATE alone is not enough, because MySQL with a non-strict sql_mode still truncates silently.
  • MySQL script: declare the key column with a binary collation (utf8mb4_bin, or utf8mb4_0900_bin, which is also NO PAD).
  • MySQL repository: replace INSERT IGNORE with INSERT ... ON DUPLICATE KEY UPDATE IdempotencyKey = IdempotencyKey, so that data errors are no longer hidden.
  • EF Core MySQL configuration: set the same binary collation on the key column.
  • SQL Server: set a case-sensitive, accent-sensitive or binary collation (for example Latin1_General_100_BIN2) on the key column in the script and in the EF Core configuration.
  • Document the key length limit and that keys are case-sensitive in the package READMEs. Document the migration step for existing tables (ALTER TABLE ... MODIFY ... COLLATE ...).

Acceptance Criteria

  • A failing test is added first: storing aBc123 and then calling ExistsAsync("ABC123") on MySQL returns false.
  • A failing test is added first: a key with 501 characters is rejected with an ArgumentException and is not stored truncated.
  • Keys that differ only by case are treated as distinct on MySQL (ADO.NET and EF Core) and on SQL Server (ADO.NET and EF Core) (integration tests).
  • MySqlIdempotencyKeyRepository no longer uses INSERT IGNORE.
  • The MySQL and SQL Server scripts and EF Core configurations declare the key collation explicitly, and the READMEs document the length limit, case sensitivity and the migration for existing tables.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugIndicates an issue or flaw that needs to be fixed.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions