Earlier, clearer messages when an execution config makes a solve stall or fail late #643

We found these while running the EKL 2019 model from lcm-zoo at production scale: source5 grid, float64, one A100. The pin was pylcm 7d0ec63 (the head of #639).

Twice, a configuration that pylcm could have flagged instead burned hours of GPU time before anything told us what was wrong. Neither case needs a change to the solver, only better messages.

1. A query_cell width that makes a device loop enormous gets no message at all

ExecutionConfig(axis_widths={"query_cell": 1}) was a sensible way to bound memory when this width controlled a host loop. On the current engine it sets the chunk of a sequential device while loop over all query cells (_lcm/exit_queries.py, chunk_size=...get("query_cell", 64)). For EKL's married_school _chunks kernel at age 18, the trip count is 131,220,000 / W:

What the user sees. The log line compiling married_school _chunks (age 18, widths={}) ..., and then nothing:

• A full solve sat there for more than an hour before we cancelled it. On the earlier pin d7e90f0, a run sat there until its 6.5 h timeout.

• Compile-only probes of that one kernel through Model._compile_period_cores:

  W	XLA flags	result
  1	default	did not finish within 3300 s
  1	--xla_gpu_enable_dynamic_slice_fusion=false	did not finish within 10800 s
  64	default	lowered in 8.9 s, compiled in 28.8 s

• gdb samples taken during the stalled compile (at 6ad7442) show XLA's DynamicSliceAnnotator → AnalyzeDynamicSlice → HloEvaluator.

Requested:

• Plan-time warning: warn, or refuse above a threshold, when a width implies a very long sequential loop. Name the kernel, the trip count and the axis to raise. For example: "query_cell width 1 makes married_school _chunks (age 18) a sequential device loop of 131,220,000 iterations; XLA compile time grows with this. Raise axis_widths['query_cell'] (default 64)."
• Docstring: in ExecutionConfig.axis_widths, say that query_cell is a device loop chunk that trades memory for sequential iterations. It is not a host batch size.
• Compile heartbeat: log a line during a long compile, such as "still compiling X after N min", so that a stalled compile can be told apart from a dead process.

2. Narrowing logs repeat the full refusal paragraph

At widths 64 and 32, with value_storage="device", the solve compiled for about 4.5 h and was then refused for budget at the youngest period. The early-refusal part of that finding belongs to #633 and is posted there: #633 (comment)

Two general points remain:

• Repeated paragraph: every workspace narrowing logs the full explanation paragraph, 87 times in one run. One line per narrowing, with the full text only in the final error, would keep the log readable.
• Compile time not reported: a budget refusal does not say how much compile time was spent before it. Here it was 17,801 s across four compilation_waves.

3. Recorded-layout replay refuses PolicyQueryKernel without saying what to do

replay_period_on_recorded_layout supports only _GridSearchPeriodKernel, so for EKL it fails before any numerical work. That rules out the capture-and-replay route for performance work on any model that uses policy-query kernels.

Requested: either support it, or name the kernel class in the error and point to Model._compile_period_cores as the available route for reproducing a compile on its own.