Repository navigation
Conversation
NFC, NFD, lowercasing, the character categories and White_Space came from ICU through FFI. jolt carries the Unicode tables itself (Character/getType, java.text.Normalizer), so lev.tokenizer reads those, and the icu native is gone from deps.edn. That also fixes two portability problems. deficu picked the versioned symbol (u_charType_74) at expansion time, so a built binary ran only against the ICU major it was built with. And the candidate list missed ICU 70 (Ubuntu 22.04), where lev refused to start at all. char-type keeps ICU's numbering: the JVM's constants are the same but for the gap it leaves at 17. Checked over every code point against ICU 70: White_Space and NFC agree everywhere; categories, NFD and lowercase differ only on code points assigned or recategorized after Unicode 14, which ICU 70 predates. The suite gives the same results as with ICU.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tokenizers got NFC, NFD, lowercasing, the character categories and White_Space from ICU through FFI. jolt has its own Unicode tables (
Character/getType,java.text.Normalizer), solev.tokenizernow uses those, and theicunative is removed fromdeps.edn.This also fixes two portability problems:
deficupicked the versioned symbol (u_charType_74) when the macro expanded. A built binary therefore only ran against the ICU major version it was built with.char-typekeeps ICU's numbering. The JVM's constants are the same except for a gap at 17, so codes from 18 up shift down by one.Verification
englishcheckpoint and TinyDecide, before and after, with ICU removed from the library path for the after run. Every namespace gives the same result, including tokenizer (83 assertions), lang (225) and tinydecide (211). The only failures, in prepare, router and server, are the same tests before and after; they need the typed-decisions/multilingual checkpoints. tensors and checkpoints weren't run: they don't fit in this machine's 7 GB of RAM.