aarch64: can't find crate for `proc_macro`` when compiling proc-macro2

[5o50]

Version(s) of meta-rust

Master (d0dc19a)

Version(s) of poky and/or oe-core

poky: (Sep 8 2019) 6d2e12e79211b31cdf5ea824fb9a8be54ba9a9eb

Expected result

Using cargo only im able to successfully cross-compile and run my code on the target aarch64-unknown-linux-gnu.

Using Yocto and meta-rust the build fails with error[E0463]: can't find crate for proc_macro`` when compiling proc-macro2.

any idea ?

thank you

Actual result

Build Configuration:
BB_VERSION           = "1.42.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-fslc-linux"
MACHINE              = "imx8mmevk"
DISTRO               = "fslc-wayland"
DISTRO_VERSION       = "2.7"
TUNE_FEATURES        = "aarch64 cortexa53 crc crypto"
TARGET_FPU           = ""
meta                 
meta-poky            = "HEAD:6d2e12e79211b31cdf5ea824fb9a8be54ba9a9eb"
meta-oe              
meta-multimedia      
meta-python          = "HEAD:3bdbf72e3a4bf18a4a2c7afbde4f7ab773aeded9"
meta-freescale       = "HEAD:2142f7ded1b3115ccc21f7575fd83e2376247193"
meta-freescale-3rdparty = "HEAD:da422478d38e744283bcf61123c4a526396c7030"
meta-freescale-distro = "HEAD:d4e77ea682fa10d0d54a723b3d3099c44fc5e95c"
meta-networking      
meta-filesystems     = "HEAD:3bdbf72e3a4bf18a4a2c7afbde4f7ab773aeded9"
meta-rust            = "master:d0dc19aa78883c6927b8287267ba283760417fe7"
meta-browser         = "HEAD:5f365ef0f842ba4651efe88787cf9c63bc8b6cb3"
meta-noos            = "<unknown>:<unknown>"
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| error[E0463]: can't find crate for `proc_macro`
|   --> /usr/src/debug/pocket/0.1.0.AUTOINC+e82af10ec8-r0/cargo_home/bitbake/proc-macro2-1.0.6/src/lib.rs:86:1
|    |
| 86 | extern crate proc_macro;
|    | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
| 
| '+v8' is not a recognized feature for this target (ignoring feature)
| error: aborting due to previous error
| 
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| For more information about this error, try `rustc --explain E0463`.
| '+v8' is not a recognized feature for this target (ignoring feature)
| '+v8' is not a recognized feature for this target (ignoring feature)
| error: could not compile `proc-macro2`.
| 
| Caused by:
|   process didn't exit successfully: `rustc --edition=2018 --crate-name proc_macro2 /home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/cargo_home/bitbake/proc-macro2-1.0.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=871b2ad645127a9c -C extra-filename=-871b2ad645127a9c --out-dir /home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/git/target/aarch64-fslc-linux/release/deps --target aarch64-fslc-linux -C linker=/home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/wrapper/target-rust-ccld -L dependency=/home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/git/target/aarch64-fslc-linux/release/deps -L dependency=/home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/git/target/release/deps --extern unicode_xid=/home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/git/target/aarch64-fslc-linux/release/deps/libunicode_xid-5c908667fe25b206.rmeta --cap-lints allow -L /home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0/recipe-sysroot/usr/lib/rust --remap-path-prefix=/home/jimmy/Desktop/neo-noos-warrior/build-imx8mmevk-wayland/tmp/work/aarch64-fslc-linux/pocket/0.1.0.AUTOINC+e82af10ec8-r0=/usr/src/debug/pocket/0.1.0.AUTOINC+e82af10ec8-r0 --cfg use_proc_macro --cfg wrap_proc_macro` (exit code: 1)
| warning: build failed, waiting for other jobs to finish...

Steps to reproduce

bitbake core-image-full

[5o50]

after some testing it's clear that
the generated path [...]/recipe-sysroot/usr/lib/rust is missing a file likelibproc_macro[...].rlib

I'm able to find it on my host standalone cargo toolchain install for the same target.
and my build for the same target is successful.

I dont know why meta-rust when setting up the toolchain is not generating this file.
any idea ?

[remleduff]

I've been looking at this layer lately, and ran across a similar problem. The quickest, but maybe not most correct fix is to change meta-rust/recipes-devtools/rust/libstd-rs.inc:

from:
S = "${RUSTSRC}/src/libstd"

to:
S = "${RUSTSRC}/src/libtest"

Building libtest will actually build both libtest and libstd, and also includes libproc_macro.

[jwinarske]

I'm hitting this with a few commits back on master 1.51.0. Changing libstd-rs to libtest has no affect; even after

bitbake libstd-rs -c do_cleansstate -f
bitbake libstd-rs

libstd-rs is only building native version of proc_macro

dev@1ab55b927c25:~/tmp-glibc/work/aarch64-oe-linux/libstd-rs$ find -iname libproc*.rlib
./1.51.0-r0/recipe-sysroot-native/usr/lib/rustlib/x86_64-linux/lib/libproc_macro-d72bc99014fc97e5.rlib

Can anyone suggest another workaround?

[cmcqueen]

I'm encountering this issue while building a Rust project for aarch64. The project is based on the Rocket web server, v0.5-rc. I had no problems until I added the crate rocket_auth (v0.4.0), and now I'm getting this error trying to build the project in Yocto.

I'm currently using meta-rust 448047c.

[jwinarske]

Now that rust support is in OE core I need to test with updated branch. I suspect the issue is still there.

[cmcqueen]

I'm comparing my Yocto poky/build/tmp/work/aarch64-poky-linux/libstd-rs/1.54.0-r0/packages-split/libstd-rs-dev/usr/lib/rust to my ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib, seeing what *.rlib are there.

The local rustup dir has these that are not in the Yocto poky dir:

• libgetopts
• libproc_macro
• libprofiler_builtins
• librustc_std_workspace_std
• libterm
• libtest
• libunwind

Why are these ones in the rustup dir but not in the Yocto poky dir?

[jwinarske]

I suspect the build breaks at the missing libproc_macro, and never gets to the others. If you look at the source proc_macro has it's own directory, and is the exception. In my case what's odd is that it's attempting to link a cross compiled libproc_macro. Libproc_macro should only ever be host.

[jwinarske]

Did you try setting LD_LIBRARY RUNTIME to path in sysroot-native that has proc_macro? Given proc_macros are only built/run on host, LD_LIBRARY_RUNTIME can be used to augment paths from sysroot-native.

[cmcqueen]

@jwinarske wrote:

In my case what's odd is that it's attempting to link a cross compiled libproc_macro. Libproc_macro should only ever be host.

and also

Given proc_macros are only built/run on host, LD_LIBRARY_RUNTIME can be used to augment paths from sysroot-native.

I don't think this is correct. For non-Yocto builds, I installed the aarch64 cross-compile toolchain using rustup. In the provided lib directory ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib, I see libproc_macro-3f1222cb41e97635.rlib. That implies that the library is also needed for the cross-compile target.

I tested this:

1. Have a project that has a dependency on proc_macro. (in my case, I'm working on web frameworks, using either rocket 0.5-rc1 with rocket_auth, or alternatively using warp with askama)
2. cargo clean --target=aarch64-unknown-linux-gnu
3. In ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib, delete or rename libproc_macro-3f1222cb41e97635.rlib.
4. cargo build --target=aarch64-unknown-linux-gnu
5. The result is, I get a build error error[E0463]: can't find crate for `proc_macro` , when it tries to build proc-macro2-1.0.32.

So, I think your hypothesis is incorrect, and the Yocto libstd-rs recipe needs to build libproc_macro.rlib for the target as well.

[jwinarske]

@cmcqueen What's the machine architecture of libproc_macro-3f1222cb41e97635.rlib?
readelf -h libproc_macro-3f1222cb41e97635.rlib

[cmcqueen]

$ readelf -h libproc_macro-3f1222cb41e97635.rlib

File: libproc_macro-3f1222cb41e97635.rlib(proc_macro-3f1222cb41e97635.proc_macro.3f08mrfg-cgu.0.rcgu.o)
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          4736032 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         975
  Section header string table index: 1

File: libproc_macro-3f1222cb41e97635.rlib(lib.rmeta)
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          1848184 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         5
  Section header string table index: 4

[jwinarske]

@cmcqueen Is this a public project you can share?

[cmcqueen]

Is this a public project you can share?

No, it's private for my employer. But, I'll see if I can make a "Minimal Reproducible Example" (MRE).