User Story
As a developer using the Pulse idempotency interceptor with NetEvolve.Pulse.MySql, I want every idempotency key to be stored and compared exactly as I sent it, so that duplicates are always detected and distinct commands are never rejected as conflicts.
Problem
The MySQL key column and insert statement lose key exactness in two ways (references on main):
src/NetEvolve.Pulse.MySql/Scripts/IdempotencyKey.sql:30-33 declares `IdempotencyKey` VARCHAR(500) NOT NULL as the primary key with COLLATE=utf8mb4_unicode_ci.
src/NetEvolve.Pulse.MySql/Idempotency/MySqlIdempotencyKeyRepository.cs:84 stores keys with INSERT IGNORE INTO .... The key is bound with AddWithValue and no size (:134), and the only guard is ArgumentException.ThrowIfNullOrWhiteSpace (:97, :127). src/NetEvolve.Pulse/Idempotency/IdempotencyStore.cs:49,61 checks nothing else either. IdempotencyKeySchema.MaxLengths.IdempotencyKey = 500 (src/NetEvolve.Pulse.Extensibility/Idempotency/IdempotencyKeySchema.cs:55) is only used by the EF Core configuration.
src/NetEvolve.Pulse/Interceptors/IdempotencyCommandInterceptor.cs:77-79 calls TryReserveAsync. The default implementation runs ExistsAsync (= @key, repository :70) and then StoreAsync, and the interceptor throws IdempotencyConflictException when the key exists.
(a) Keys longer than 500 characters: idempotency is silently off. IGNORE turns ER_DATA_TOO_LONG into a warning, so MySQL stores only the first 500 characters. The next ExistsAsync compares the full key against that stored prefix and never finds a match, so the handler runs again for every duplicate. Two different long keys that share the same first 500 characters also collide.
(b) Keys that differ only by case: false conflicts. utf8mb4_unicode_ci ignores case and accents, and with PAD SPACE it also ignores trailing spaces. Base62 and base64 client keys differ by case all the time. Once aBc123 is stored, ExistsAsync("ABC123") returns true and a different command is rejected with IdempotencyConflictException.
PostgreSQL and SQLite compare keys exactly. Other providers are affected too:
NetEvolve.Pulse.SqlServer: SqlServerIdempotencyKeyRepository.cs:84 and :116 bind the key as new SqlParameter("@idempotencyKey", SqlDbType.NVarChar, 500), and SqlClient silently cuts longer values to 500 characters on both lookup and store. Keys that share the same first 500 characters then collide and cause false conflicts. The table in src/NetEvolve.Pulse.SqlServer/Scripts/IdempotencyKey.sql:33 sets no collation, so it uses the database default. That default is usually case-insensitive (for example SQL_Latin1_General_CP1_CI_AS).
- EF Core on MySQL:
src/NetEvolve.Pulse.EntityFramework/Configurations/MySqlIdempotencyKeyConfiguration.cs:42 maps varchar(500) without a collation, so it uses the database collation, which is usually case-insensitive.
Specification
MySQL 8.0 Reference Manual, 15.2.7 INSERT Statement:
Data conversions that would trigger errors abort the statement if IGNORE is not specified. With IGNORE, invalid values are adjusted to the closest values and inserted; warnings are produced but the statement does not abort.
MySQL 8.0 Reference Manual, Server SQL Modes – The Effect of IGNORE on Statement Execution lists ER_DATA_TOO_LONG among the errors that IGNORE turns into warnings, and states that "when the IGNORE keyword and strict SQL mode are both in effect, IGNORE takes precedence."
MySQL 8.0 Reference Manual, B.3.4.1 Case Sensitivity in String Searches:
nonbinary string comparisons are case-insensitive by default ... If you want a column always to be treated in case-sensitive fashion, declare it with a case-sensitive or binary collation.
Pulse contract: IdempotencyKeySchema.MaxLengths.IdempotencyKey (500) is the documented maximum key length. No external standard defines whether keys are case-sensitive. The defect is that providers behave differently from each other and that case-insensitive matching is unsafe for base62 and base64 keys.
Requirements
- Reject keys longer than
IdempotencyKeySchema.MaxLengths.IdempotencyKey once, in the core IdempotencyStore (ExistsAsync, StoreAsync and TryReserveAsync), with an ArgumentException, so every provider behaves the same. Relying on ON DUPLICATE KEY UPDATE alone is not enough, because MySQL with a non-strict sql_mode still truncates silently.
- MySQL script: declare the key column with a binary collation (
utf8mb4_bin, or utf8mb4_0900_bin, which is also NO PAD).
- MySQL repository: replace
INSERT IGNORE with INSERT ... ON DUPLICATE KEY UPDATE IdempotencyKey = IdempotencyKey, so that data errors are no longer hidden.
- EF Core MySQL configuration: set the same binary collation on the key column.
- SQL Server: set a case-sensitive, accent-sensitive or binary collation (for example
Latin1_General_100_BIN2) on the key column in the script and in the EF Core configuration.
- Document the key length limit and that keys are case-sensitive in the package READMEs. Document the migration step for existing tables (
ALTER TABLE ... MODIFY ... COLLATE ...).
Acceptance Criteria
User Story
As a developer using the Pulse idempotency interceptor with
NetEvolve.Pulse.MySql, I want every idempotency key to be stored and compared exactly as I sent it, so that duplicates are always detected and distinct commands are never rejected as conflicts.Problem
The MySQL key column and insert statement lose key exactness in two ways (references on
main):src/NetEvolve.Pulse.MySql/Scripts/IdempotencyKey.sql:30-33declares`IdempotencyKey` VARCHAR(500) NOT NULLas the primary key withCOLLATE=utf8mb4_unicode_ci.src/NetEvolve.Pulse.MySql/Idempotency/MySqlIdempotencyKeyRepository.cs:84stores keys withINSERT IGNORE INTO .... The key is bound withAddWithValueand no size (:134), and the only guard isArgumentException.ThrowIfNullOrWhiteSpace(:97,:127).src/NetEvolve.Pulse/Idempotency/IdempotencyStore.cs:49,61checks nothing else either.IdempotencyKeySchema.MaxLengths.IdempotencyKey = 500(src/NetEvolve.Pulse.Extensibility/Idempotency/IdempotencyKeySchema.cs:55) is only used by the EF Core configuration.src/NetEvolve.Pulse/Interceptors/IdempotencyCommandInterceptor.cs:77-79callsTryReserveAsync. The default implementation runsExistsAsync(= @key, repository:70) and thenStoreAsync, and the interceptor throwsIdempotencyConflictExceptionwhen the key exists.(a) Keys longer than 500 characters: idempotency is silently off.
IGNOREturnsER_DATA_TOO_LONGinto a warning, so MySQL stores only the first 500 characters. The nextExistsAsynccompares the full key against that stored prefix and never finds a match, so the handler runs again for every duplicate. Two different long keys that share the same first 500 characters also collide.(b) Keys that differ only by case: false conflicts.
utf8mb4_unicode_ciignores case and accents, and withPAD SPACEit also ignores trailing spaces. Base62 and base64 client keys differ by case all the time. OnceaBc123is stored,ExistsAsync("ABC123")returnstrueand a different command is rejected withIdempotencyConflictException.PostgreSQL and SQLite compare keys exactly. Other providers are affected too:
NetEvolve.Pulse.SqlServer:SqlServerIdempotencyKeyRepository.cs:84and:116bind the key asnew SqlParameter("@idempotencyKey", SqlDbType.NVarChar, 500), and SqlClient silently cuts longer values to 500 characters on both lookup and store. Keys that share the same first 500 characters then collide and cause false conflicts. The table insrc/NetEvolve.Pulse.SqlServer/Scripts/IdempotencyKey.sql:33sets no collation, so it uses the database default. That default is usually case-insensitive (for exampleSQL_Latin1_General_CP1_CI_AS).src/NetEvolve.Pulse.EntityFramework/Configurations/MySqlIdempotencyKeyConfiguration.cs:42mapsvarchar(500)without a collation, so it uses the database collation, which is usually case-insensitive.Specification
MySQL 8.0 Reference Manual, 15.2.7 INSERT Statement:
MySQL 8.0 Reference Manual, Server SQL Modes – The Effect of IGNORE on Statement Execution lists
ER_DATA_TOO_LONGamong the errors thatIGNOREturns into warnings, and states that "when theIGNOREkeyword and strict SQL mode are both in effect,IGNOREtakes precedence."MySQL 8.0 Reference Manual, B.3.4.1 Case Sensitivity in String Searches:
Pulse contract:
IdempotencyKeySchema.MaxLengths.IdempotencyKey(500) is the documented maximum key length. No external standard defines whether keys are case-sensitive. The defect is that providers behave differently from each other and that case-insensitive matching is unsafe for base62 and base64 keys.Requirements
IdempotencyKeySchema.MaxLengths.IdempotencyKeyonce, in the coreIdempotencyStore(ExistsAsync,StoreAsyncandTryReserveAsync), with anArgumentException, so every provider behaves the same. Relying onON DUPLICATE KEY UPDATEalone is not enough, because MySQL with a non-strictsql_modestill truncates silently.utf8mb4_bin, orutf8mb4_0900_bin, which is alsoNO PAD).INSERT IGNOREwithINSERT ... ON DUPLICATE KEY UPDATE IdempotencyKey = IdempotencyKey, so that data errors are no longer hidden.Latin1_General_100_BIN2) on the key column in the script and in the EF Core configuration.ALTER TABLE ... MODIFY ... COLLATE ...).Acceptance Criteria
aBc123and then callingExistsAsync("ABC123")on MySQL returnsfalse.ArgumentExceptionand is not stored truncated.MySqlIdempotencyKeyRepositoryno longer usesINSERT IGNORE.