thread 'rustc' has overflowed its stack

[avl]

I just tried the latest nightly (rustc 1.45.0-nightly (ad4bc33 2020-06-01)) and noticed it can't compile one of the crates I work on.

The crate does have an enum with a a huge number of variants (almost 6000), for which I use serde to derive Serialize and Deserialize.

When building, even with just 'cargo check', the build fails with

thread 'rustc' has overflowed its stack
fatal runtime error: stack overflow
error: could not compile `<crate_name>`.

It works on stable 1.43.1.

I tried running rustc under gdb. It didn't crash exactly the same way, I got SIGSEGV instead.

Looking at the callstack in gdb, it is suspiciously deep (15000 functions deep). The top most functions are:

#0  0x00007ff0bae9925b in rustc_mir_build::build::matches::simplify::<impl rustc_mir_build::build::Builder>::simplify_candidate ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#1  0x00007ff0bae9e7ad in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::match_candidates ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#2  0x00007ff0baef7fe4 in <core::iter::adapters::Map<I,F> as core::iter::traits::iterator::Iterator>::fold ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#3  0x00007ff0baea2f93 in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::test_candidates::{{closure}} ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#4  0x00007ff0bae9b2f1 in rustc_mir_build::build::matches::test::<impl rustc_mir_build::build::Builder>::non_scalar_compare ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#5  0x00007ff0baea1362 in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::match_simplified_candidates ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#6  0x00007ff0baea2fdc in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::test_candidates::{{closure}} ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#7  0x00007ff0bae9b2f1 in rustc_mir_build::build::matches::test::<impl rustc_mir_build::build::Builder>::non_scalar_compare ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#8  0x00007ff0baea1362 in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::match_simplified_candidates ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#9  0x00007ff0baea2fdc in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::test_candidates::{{closure}} ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#10 0x00007ff0bae9b2f1 in rustc_mir_build::build::matches::test::<impl rustc_mir_build::build::Builder>::non_scalar_compare ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so
#11 0x00007ff0baea1362 in rustc_mir_build::build::matches::<impl rustc_mir_build::build::Builder>::match_simplified_candidates ()
   from /home/anders/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-b6bcd19e02bce685.so

If I reduce the number of enum variants to 5000, or stop using serde to derive Serialize/Deserialize, it starts working again.

Since it works using stable, but not using nightly, I thought it might be some kind of regression. I totally realize this may not be a bug and that what I'm doing (Serde + giant enum) is just not a good idea. Accept my apologies if this kind of thing shouldn't be reported here!

[Mark-Simulacrum]

cc @oli-obk -- I believe #55617 should have fixed this but I guess we're missing at least this case?

[avl]

As an experiment I tried doubling the number of enum variants and compiling using stable. That still worked. So it seems the code wasn't just on the brink to stop working using the stable compiler.

[avl]

Here is a github repo with a minimized test case for this issue:
https://github.com/avl/repro_enum_serde_stackoverflow72933

[nagisa]

@Mark-Simulacrum It introduced a mechanism for dealing with this kind of issue, but I don’t doubt that we might have missed instances where the compiler is doing something recursive. MIR construction for matches appears to be one of these cases.

[Mark-Simulacrum]

Yeah, that's what I expected, I just recalled seeing some commentary suggesting that we'd perhaps covered all cases. Definitely not unexpected that we didn't, though, nor did I intend to imply we should have :)

[avl]

The beta seems to not have the problem. The problem does not occur with "rustc 1.44.0-beta.4 (02c25b3 2020-05-23)".

[avl]

I saw the 'E-needs-mcve'-tag. @nagisa Did you see my repro-case?

https://github.com/avl/repro_enum_serde_stackoverflow72933

(but maybe you're talking about something more self-contained, not referencing 'serde'?)