fix(arabic): add missing letters, harakat, tatweel and Arabic-Indic digits - #350
Open
mohaelmrabet wants to merge 1 commit into
Open
fix(arabic): add missing letters, harakat, tatweel and Arabic-Indic digits#350mohaelmrabet wants to merge 1 commit into
mohaelmrabet wants to merge 1 commit into
Conversation
…igits The Arabic ruleset was missing nine letters, including the plain alif — the most frequent letter in the language — and the ta marbuta. An unmapped character is not transliterated: it falls through to the separator regexp and becomes a dash, so words were cut at arbitrary points. The two existing Arabic test cases documented that behaviour as expected: هذه هي اللغة العربية -> hthh-hy-llgh-laarby مرحبا العالم -> mrhb-laa-lm Both are updated to the corrected output. Added: - Alif and its variants: ا إ آ ٱ - Ta marbuta ة -> ah, alef maksura ى -> a - Carried hamza ؤ -> o and ئ -> a; standalone hamza ء is dropped, it carries no sound of its own - Harakat/tashkeel (U+064B..U+0652, U+0670) are removed rather than left to become dashes, so a vocalised word and its bare spelling slug identically - Tatweel/kashida U+0640 removed: decorative elongation, never lexical - Arabic-Indic digits ٠-٩, which previously produced an empty slug Transliteration choices were reviewed by a native Arabic speaker. Existing rules are left untouched, so no current slug changes except for the characters that produced no output at all.
mohaelmrabet
force-pushed
the
fix/arabic-ruleset
branch
from
August 14, 2026 16:54
4d1f07b to
0875668
Compare
mohaelmrabet
marked this pull request as ready for review
August 14, 2026 17:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Arabic ruleset is missing nine letters, including the plain alif
ا— the most frequent letter in the language — and the ta marbutaة, the feminine ending found on a large share of Arabic nouns.A character with no rule is not transliterated. It falls through to the separator regexp and becomes a dash, so words are cut at arbitrary points and letters silently disappear.
The two existing Arabic test cases document the broken output as if it were expected:
هذه هي اللغة العربيةhthh-hy-llgh-laarbyhthh-hy-allghah-alaarbyahمرحبا العالمmrhb-laa-lmmrhba-alaaalmTwo more cases produce results that are arguably worse:
٢٠٥(Arabic-Indic digits)مَدْرَسَة(vocalised)m-d-r-sarabicis in the default ruleset, so this affects every user ofSlugifyout of the box.Changes
Only
Resources/rules/arabic.json(plus the regeneratedDefaultRuleProvider.phpand tests). No existing rule is modified — every addition covers a character that previously produced no output, so no current slug changes except the ones that were broken.اإآٱaةahىaؤoو → oئaءً ٌ ٍ َ ُ ِ ّ ْ ٰـ٠–٩0–9Removing the harakat rather than transliterating them is deliberate: it keeps a vocalised word and its bare spelling slugging identically, which matters for URLs.
Cross-checked against ICU
Every rule was compared with ICU's own transliterators (
Arabic-LatinandArabic-Latin/BGN, ICU 74.2) used as an independent oracle. The added letters agree with ICU, with two deliberate exceptions noted below.Arabic-Latin/BGNاإآaaٱa'ais the useful valueةahthh;ahrestores the vowel the stripped fatha carriedىayyعلى= 'ala)ئayءʾـ٠–٩0–90–90–9ICU is not usable as the implementation here: it requires
ext-intl, it emits non-ASCII output (ʿ,ā,ẗ) that still needs aLatin-ASCIIpass, andArabic-Latin/BGNleaves characters untransliterated on common words —رئيس→rئيs,فاطمة→fاtmh. It is used here only as a reference to check the table against.The same cross-check flags six pre-existing rules as diverging from ICU. They are left untouched in this PR, since changing them would alter slugs for current users:
جgjقkqذthdhظthzعaaʿوowHappy to open a separate PR for those if you consider a breaking change acceptable for a future major.
Notes
php bin/generate-default.phpwas run, per the contributing guide.defaultRuleProvider()per the guide, covering the missing letters, harakat, tatweel and digits.