Pillow does not respect compiler library path

[CendioHalim]

I have a setup with Python 3.9 installed in /usr/lib for historical reasons, and zlib and libjpeg installed /usr/lib64.

When I attempt to build Pillow, it will fail since it cannot find zlib/libjpeg, even if the compiler will happlly link to both libs without issue.

In my case, I could work around this by explicitly setting LIBRARY_PATH=/usr/lib64.

[CendioHalim]

This issue for me seems to be due to two reasons:

1. pkg-config will strip -L linker flags that are part of the system library directory (/usr/lib64 in my case).
2. distutils sysconfig.get_config_var("LIBDIR") will return /usr/lib/, which will be set to self.compiler.library_dirs in setup.py.

This is almost related to #7634, but not really the same issue, since LIBDIR in my case is /usr/lib, which doesn't fix the issue.

I tried commenting out the dependency checks and hardcoding the library dirs manually in setup.py:

# for f in feature:
#   if not getattr(feature, f) and feature.require(f):
#      if f in ("jpeg", "zlib"):
            #raise RequiredDependencyException(f)
        #raise DependencyException(f)

# core library

libs = self.add_imaging_libs.split()
defs = []
feature.zlib="/usr/lib64"
libs.append(feature.zlib)
defs.append(("HAVE_LIBZ", None))

feature.jpeg="/usr/lib64"
libs.append(feature.jpeg)
defs.append(("HAVE_LIBJPEG", None))

With this, I could successfully build Pillow with jpeg and zlib support.

I'm not sure what a robust solution for this would be. Other build tools like autoconf works by attempting to compile a program to, for example, see if a library exists. This is maybe not suitable for this project, but maybe adding /usr/lib64 to the default library_dirs would make sense?

[radarhere]

I'm slightly confused. In your first comment, you said

In my case, I could work around this by explicitly setting LIBRARY_PATH=/usr/lib64.

but then you put that aside and modified Pillow instead? Could you explain what was wrong with your initial workaround?

[CendioHalim]

The modifications in the second comment were just to show that the dependency checks are insufficient.

It was enough to do this to get the build to work with zlib/jpeg:

libs = self.add_imaging_libs.split()
defs = []
feature.zlib="foo"
defs.append(("HAVE_LIBZ", None))

feature.jpeg="bar"
defs.append(("HAVE_LIBJPEG", None))

# if-statements below for jpeg and zlib are commented out

Could you explain what was wrong with your initial workaround?

There's not really anything wrong with it, I just think it's odd that the build works by commenting out some lines and setting some variables to junk. It would be cool if it just worked™ 😄, my compiler already knows to look in /usr/lib64.

[radarhere]

Pillow used to explicitly search /usr/lib64 in some Linux environments, until #3245 switched to using ldconfig instead.

https://man7.org/linux/man-pages/man8/ldconfig.8.html

On some 64-bit architectures such as x86-64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries.

So it would seem unexpected that ldconfig isn't solving this automatically for you.

What operating system are you using?
How did you install the libraries?
What does ldconfig -p give you? Or, if you're on FreeBSD, ldconfig -r?

[CendioHalim]

I'm using a custom Linux-based build system which is RedHat/Fedora based. My libraries are built/installed as RPM's in /usr/lib64.

Looks like we didn't have the file /etc/ld.so.cache, which seems to require ldconfig to be run once as root to generate. After running sudo ldconfig once, the build worked without issue.

[CendioHalim]

Doesn't this mean that you need to run ldconfig as root once before every build?

[radarhere]

It would seem the normal behaviour is to run ldconfig after a new library is installed.

https://unix.stackexchange.com/questions/256893/relationship-between-ldconfig-and-ld-so-cache

If I am installing a program, how does my computer know to use the new libraries that had been added? After apt-get install is ldconfig run?
...
Yes
...
apt-get and dpkg both invoke ldconfig to rebuild the cache.

https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

When you install a new version of a library, you install it in one of a few special directories and then run the program ldconfig

[CendioHalim]

Ah I see, looks like Fedora/RHEL has some RPM macro magic to run ldconfig when shared libraries are installed.

[radarhere]

So, would you consider this to be resolved then?

[CendioHalim]

I think the issue still persists in some cross-compilation setups, but the workaround works good enough for me.