Skip to content

NFKC isn't idempotent on some Korean inputs #106

Description

@crtschin

For context, the failure was discovered through some property checks that tested some roundtrips. I had Claude convert the failing testcase into a minimal reproducer, also checking what text-icu returned. I've attached the reproducer, the following is the output returned when ran via cabal run.

L + compatibility V
  input                        U+1100 U+314F
  unicode-transforms NFKC      U+1100 U+1161
  unicode-transforms NFKC^2    U+AC00
  text-icu NFKC                U+AC00
  expected                     U+AC00
  unicode-transforms idempotent  False
  unicode-transforms correct     False
  text-icu correct               True
LV + compatibility T
  input                        U+AC00 U+313A
  unicode-transforms NFKC      U+AC00 U+11B0
  unicode-transforms NFKC^2    U+AC09
  text-icu NFKC                U+AC09
  expected                     U+AC09
  unicode-transforms idempotent  False
  unicode-transforms correct     False
  text-icu correct               True

Peculiarly, it does reach the right answer when ran twice.

The code includes references to Korean script, which I have not verified as I do not speak it. I've kept them in the code as it may still be useful for debugging.

unicode-transforms-idempotent.tar.gz

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions