Fix cross-compiling by searching the right lib and include directories - #7634
Conversation
8b7490d to
c8d257e
Compare
|
I slept on it and changed my mind about virtualenvs. I think dependencies may be installed there, so I have continued to support that case, even though it is incompatible with cross-compiling. |
We were previously searching the `{sys.prefix}/lib` and
`{sys.prefix}/include` directories unconditionally. This is problematic
when cross-compiling, as it does not take account of any sysroot where
alternative libraries and headers are located. Adding `-I/usr/include`
causes the build to explode, at least when cross-compiling from 64-bit
to 32-bit.
Python does not officially support cross-compiling, but Gentoo achieves
this by modifying the sysconfig variables like `LIBDIR` and `INCLUDEDIR`
with great results.
Assuming "lib" is bad. 64-bit Linux systems often use lib64, putting
32-bit libraries under lib. You cannot assume that either though, as
pure 64-bit Linux systems may just use lib instead. Things get even
stranger on RISC-V.
The value of `sys.prefix` changes when using a virtualenv. Dependencies
may be installed here, so it does make sense to continue supporting this
case, even if it is incompatible with cross-compiling. Unlike regular
environments, "lib" is generally used for libraries, although a lib64
symlink may also be present.
|
I don't suppose it would be possible for you to create a Dockerfile that demonstrates the need for this? |
|
I did try very hard, but it's difficult to demonstrate on other distros I'm familiar with. I could use Gentoo, but then it might just look like Gentoo is doing something weird. Debian and friends avoid the issue with their multilib approach to cross-compiling. Fedora doesn't have a formal cross-compile mechanism, but you can usually make it work by setting a few flags. Unfortunately, it has some quirks that make it fall over in this case, and while I could work around them, I think that would just distract from the issue at hand. It's better to focus on the command that leads to the errors. The above is truncated. It actually spews out pages and pages of this stuff. 64-bit x86_64 headers from /usr/include, particularly those under /usr/include/bits, massively break 32-bit targets like armv7a. Simply taking out |
hugovk
left a comment
hugovk
left a comment
There was a problem hiding this comment.
Looks okay, in our own native builds and wheel builds nothing should change.
Currently this isn't mergeable because the CI hasn't run in a while, so I'll close/re-open to re-trigger.
3 participants
Fix cross-compiling by searching the right lib and include directories
We were previously searching the
{sys.prefix}/liband{sys.prefix}/includedirectories unconditionally. This is problematic when cross-compiling, as it does not take account of any sysroot where alternative libraries and headers are located. Adding-I/usr/includecauses the build to explode, at least when cross-compiling from 64-bit to 32-bit.Python does not officially support cross-compiling, but Gentoo achieves this by modifying the sysconfig variables like
LIBDIRandINCLUDEDIRwith great results.Assuming "lib" is bad. 64-bit Linux systems often use lib64, putting 32-bit libraries under lib. You cannot assume that either though, as pure 64-bit Linux systems may just use lib instead. Things get even stranger on RISC-V.
The value of
sys.prefixchanges when using a virtualenv. Dependencies may be installed here, so it does make sense to continue supporting this case, even if it is incompatible with cross-compiling. Unlike regular environments, "lib" is generally used for libraries, although a lib64 symlink may also be present.