Link FreeType against the bundled HarfBuzz in wheels

[akx]

Follows up on #8497.

This came up as a side effect of #9909, where the last commit needed to increase the allowed comparison epsilon for the Arabic-language test, since different platforms in the CI stack would render it slightly differently... and it's a doozy! 😄

FreeType's autofitter can use HarfBuzz to work out which glyphs a script covers, so it can derive blue zones for scripts such as Arabic.

Only the Windows wheels were built that way. macOS passed --with-harfbuzz=no (so no HarfBuzz interop, no way, no how), and Linux left it unconfigured, which meant FreeType defaulted to dlopening HarfBuzz at runtime with the name "libharfbuzz.so.0".

However, auditwheel renames shared libraries to avoid conflicts, so the Linux wheels ended up with a FreeType that would only find HarfBuzz if the system had its own copy with the name "libharfbuzz.so.0".

---

In the current 12.3 wheel off PyPI, the interop symbols do not exist:

$ wget https://files.pythonhosted.org/packages/c7/da/32c752228ae345f489e3a42499d817b6c3996da7e8a3bc7a04fc806b243b/pillow-12.3.0-cp314-cp314-macosx_11_0_arm64.whl
$ unzip pillow-12.3.0-cp314-cp314-macosx_11_0_arm64.whl PIL/.dylibs/libfreetype.6.dylib
$ nm -u PIL/.dylibs/libfreetype.6.dylib | grep _hb_ot_
$

but in a wheel built by CI off this branch (see akx#18):

$ unzip 'dist-macOS arm64.zip'
Archive:  dist-macOS arm64.zip
  inflating: pillow-13.0.0.dev0-cp311-cp311-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-cp312-cp312-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-cp313-cp313-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-cp314-cp314-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-cp314-cp314t-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-cp315-cp315-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-cp315-cp315t-macosx_11_0_arm64.whl
  inflating: pillow-13.0.0.dev0-pp311-pypy311_pp73-macosx_11_0_arm64.whl
$ unzip pillow-13.0.0.dev0-cp314-cp314-macosx_11_0_arm64.whl PIL/.dylibs/libfreetype.6.dylib
$ nm -u PIL/.dylibs/libfreetype.6.dylib | grep _hb_ot_
_hb_ot_layout_collect_lookups
_hb_ot_layout_lookup_collect_glyphs
_hb_ot_layout_lookup_would_substitute
_hb_ot_tags_from_script_and_language
$

[radarhere]

It is unusual that this should be required, right? Have FreeType or Harfbuzz discussed this anywhere?

[akx]

Google search autocomplete (on a different device, on mobile right now) suggests https://www.google.com/search?q=freetype+harfbuzz+circular+dependency if you start typing "freetype harf", so that suggests it's a known thing.

Here's a recent-ish Debian bug about it https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1124239

Freetype notes the circular dependency in the release announcement for 2.14.0 on https://freetype.org/

[radarhere]

The conclusion of the Debian bug and the FreeType release notes that the solution is to load HarfBuzz dynamically. Are you going to tell me that option isn't available to us because of auditwheel?

[akx]

As noted in the PR description, FT does attempt to load HarfBuzz dynamically on main already, but yes - auditwheel renames the bundled dynamic library and fixes up import tables but can't fix the dynamic dlopen call within FT.

The dlopen name is specified like https://github.com/freetype/freetype/blob/master/src/autofit/ft-hb.c#L34 so we'd need to know the name auditwheel will rename harfbuzz to before we compile FT to pass that preprocessor constant.

[radarhere]

@khaledhosny because you're obviously thinking about this sort of thing at the moment, did you have any thoughts on this approach?

[khaledhosny]

@khaledhosny because you're obviously thinking about this sort of thing at the moment, did you have any thoughts on this approach?

This is the usual solution to this issue; build freetype without harfbuzz support, then build harfbuzz with freetype support, then build freetype again with harfbuzz support.

The new way is to let freetype load harfbuzz at runtime which is discussed above.

You can break this circular dependency by either not using FreeType’s font functions in HarfBuzz and use HarfBuzz’s internal font functions, but if you are using horizontal hinting in FreeType you might get different glyph advance widths. Or by not using HarfBuzz in FreeType but the auto hinter will give different results which I think this is what this issue is trying to fix.

[radarhere]

Could you provide an example that renders differently with this change?

[akx]

Could you provide an example that renders differently with this change?

On my machine, comparing PyPI's Pillow and a CI build from here...

These comparison GIFs borrow complex script examples from test_imagefontctl. I don't know if either rendering is better or worse since I'm not exactly fluent in these scripts or languages though! 😅

From test_complex_unicode_text2

From test_x_max_and_y_offset

From test_y_offset