Skip to content

fix(arabic): add missing letters, harakat, tatweel and Arabic-Indic digits - #350

Open
mohaelmrabet wants to merge 1 commit into
cocur:masterfrom
mohaelmrabet:fix/arabic-ruleset
Open

fix(arabic): add missing letters, harakat, tatweel and Arabic-Indic digits#350
mohaelmrabet wants to merge 1 commit into
cocur:masterfrom
mohaelmrabet:fix/arabic-ruleset

Conversation

@mohaelmrabet

@mohaelmrabet mohaelmrabet commented Aug 14, 2026

Copy link
Copy Markdown

Problem

The Arabic ruleset is missing nine letters, including the plain alif ا — the most frequent letter in the language — and the ta marbuta ة, the feminine ending found on a large share of Arabic nouns.

A character with no rule is not transliterated. It falls through to the separator regexp and becomes a dash, so words are cut at arbitrary points and letters silently disappear.

The two existing Arabic test cases document the broken output as if it were expected:

Input Current output Expected
هذه هي اللغة العربية hthh-hy-llgh-laarby hthh-hy-allghah-alaarbyah
مرحبا العالم mrhb-laa-lm mrhba-alaaalm

Two more cases produce results that are arguably worse:

Input Current output Note
٢٠٥ (Arabic-Indic digits) (empty slug) a page numbered in Arabic digits gets no slug at all
مَدْرَسَة (vocalised) m-d-r-s harakat become dashes inside a single word

arabic is in the default ruleset, so this affects every user of Slugify out of the box.

Changes

Only Resources/rules/arabic.json (plus the regenerated DefaultRuleProvider.php and tests). No existing rule is modified — every addition covers a character that previously produced no output, so no current slug changes except the ones that were broken.

Added Value Rationale
ا إ آ ٱ a alif and its variants; plain alif was absent entirely
ة ah ta marbuta, word-final feminine ending
ى a alef maksura, pronounced as a final alif
ؤ o carried hamza follows its carrier, matching the existing و → o
ئ a carried hamza, taking the vowel it actually carries
ء (removed) standalone hamza carries no sound of its own
ً ٌ ٍ َ ُ ِ ّ ْ ٰ (removed) harakat/tashkeel are vowel marks, not letters
ـ (removed) tatweel/kashida is decorative elongation, never lexical
٠٩ 09 Arabic-Indic digits

Removing the harakat rather than transliterating them is deliberate: it keeps a vocalised word and its bare spelling slugging identically, which matters for URLs.

مَدْرَسَة  ->  mdrsah
مدرسة      ->  mdrsah
مدرســـة  ->  mdrsah

Cross-checked against ICU

Every rule was compared with ICU's own transliterators (Arabic-Latin and Arabic-Latin/BGN, ICU 74.2) used as an independent oracle. The added letters agree with ICU, with two deliberate exceptions noted below.

Added This PR ICU Arabic-Latin ICU /BGN Note
ا إ آ a a agrees
ٱ a (unmapped) ' ICU leaves it as-is; a is the useful value
ة ah t h BGN agrees on h; ah restores the vowel the stripped fatha carried
ى a y y ICU follows the glyph, we follow the pronunciation (على = 'ala)
ئ a y (unmapped) deliberate, see below
ء (removed) ʾ (removed) BGN agrees
ـ (removed) (removed) agrees
٠٩ 09 09 09 agrees

ICU is not usable as the implementation here: it requires ext-intl, it emits non-ASCII output (ʿ, ā, ) that still needs a Latin-ASCII pass, and Arabic-Latin/BGN leaves characters untransliterated on common words — رئيسrئيs, فاطمةfاtmh. It is used here only as a reference to check the table against.

The same cross-check flags six pre-existing rules as diverging from ICU. They are left untouched in this PR, since changing them would alter slugs for current users:

Existing rule Current ICU
ج g j
ق k q
ذ th dh
ظ th z
ع aa ʿ
و o w

Happy to open a separate PR for those if you consider a breaking change acceptable for a future major.

Notes

  • Transliteration choices were reviewed by a native Arabic speaker.
  • php bin/generate-default.php was run, per the contributing guide.
  • Test cases added to defaultRuleProvider() per the guide, covering the missing letters, harakat, tatweel and digits.
  • Full suite green.

…igits

The Arabic ruleset was missing nine letters, including the plain alif — the
most frequent letter in the language — and the ta marbuta. An unmapped
character is not transliterated: it falls through to the separator regexp and
becomes a dash, so words were cut at arbitrary points.

The two existing Arabic test cases documented that behaviour as expected:

  هذه هي اللغة العربية  ->  hthh-hy-llgh-laarby
  مرحبا العالم          ->  mrhb-laa-lm

Both are updated to the corrected output.

Added:
- Alif and its variants: ا إ آ ٱ
- Ta marbuta ة -> ah, alef maksura ى -> a
- Carried hamza ؤ -> o and ئ -> a;
  standalone hamza ء is dropped, it carries no sound of its own
- Harakat/tashkeel (U+064B..U+0652, U+0670) are removed rather than left to
  become dashes, so a vocalised word and its bare spelling slug identically
- Tatweel/kashida U+0640 removed: decorative elongation, never lexical
- Arabic-Indic digits ٠-٩, which previously produced an empty slug

Transliteration choices were reviewed by a native Arabic speaker.

Existing rules are left untouched, so no current slug changes except for the
characters that produced no output at all.
@mohaelmrabet
mohaelmrabet marked this pull request as ready for review August 14, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant