static array of zeroes can take minutes to lint check

[kazcw]

This program takes an inordinate amount of time and memory to compile (it's worse if the array is actually used, but this is a minimal test case):

const SIZE: usize = 1 << 30;
static SLICE: [u8; SIZE] = [0u8; SIZE];
fn main() {}

I was hoping this would be a viable way to get a big chunk of .bss so I don't have to depend on an mmap crate to get a lot of static zeroes, but the effect on compile time makes that impractical.

If I run rustc -Z time-passes, the big offender is:
time: 47.666; rss: 2486MB lint checking

Although the time (and memory) is reportedly spent checking lints, setting --cap-lints allow doesn't make any difference. I'm guessing the "lint checking" pass includes some things that need to be checked even if lints are suppressed? If not, it seems like a separate issue is that a lot of work could be saved with cap-lints set (e.g. when compiling dependencies).

Here are the top results from perf report, for rustc 1.32.0-nightly (25a42b2ce 2018-11-07):

  31.62%  rustc     librustc_mir-714845413a99e6ff.so              [.] <rustc_mir::interpret::memory::Memory<'a, 'mir, 'tcx, M>>::copy_repeatedly
  22.94%  rustc     librustc_mir-714845413a99e6ff.so              [.] <core::iter::Map<I, F> as core::iter::iterator::Iterator>::fold
   7.32%  rustc     librustc-0eb8c117db37850c.so                  [.] rustc::mir::interpret::UndefMask::grow
   7.31%  rustc     librustc-0eb8c117db37850c.so                  [.] rustc::mir::interpret::UndefMask::set_range
   6.94%  rustc     libc-2.27.so                                  [.] __memmove_sse2_unaligned_erms
   5.91%  rustc     librustc_mir-714845413a99e6ff.so              [.] <rustc_mir::interpret::memory::Memory<'a, 'mir, 'tcx, M>>::check_bytes

So it looks like miri is actually creating the array and folding over it. I know it's not going to find any problems, because I have ECC memory 😆.

There are already some bugs relating to slow compilation of large arrays, with the most relevant I could find being #37155, #49330. I think this is separate from those cases because:

• the input in this case is a [0; _] array, whereas the others initialize arrays from sequences of elements
• those cases seem to refer to superlinear runtime; this issue appears roughly linear in time and space and only becomes noticeable for much larger arrays
• the bottleneck in this case occurs during lint checking, which I didn't see in any other array performance bugs

[oli-obk]

Have you seen any improvement after #58556 ?

[jonas-schievink]

rustc 1.34.0-nightly (146aa60f3 2019-02-18)   0:21.37elapsed
rustc 1.36.0-nightly (08bfe1612 2019-05-02)   0:07.22elapsed

[oli-obk]

I've wondered before whether we should special-case constants whose bits are all defined and zero. I'll open a discussion in the const-eval zulip channel.

[dbdr]

I also ran into this problem. @oli-obk, has that discussion happened?

[oli-obk]

Kind of. We will have special treatment for constants that consist solely of undef bytes (see #62655). This scheme can likely be extended to support all bytes being zero withour requiring time or space overhead.

[jyn514]

If not, it seems like a separate issue is that a lot of work could be saved with cap-lints set (e.g. when compiling dependencies).

I opened #74704 for this.

[jyn514]

Kind of. We will have special treatment for constants that consist solely of undef bytes (see #62655). This scheme can likely be extended to support all bytes being zero withour requiring time or space overhead.

#62655 was closed, are there still plans to implement this?

[oli-obk]

are there still plans to implement this?

Not right now... there are other more important construction sites in const eval.