Wait for wfi in one place; record traps only when taken

[davidharrishmc]

What

A waiting wfi now stalls the whole pipeline through StallW, as an LSU stall does, instead of stalling M and flushing W. Trap CSRs and trap HPM events are recorded only when the trap is taken.

This PR contains #1925's commits (merged, not rebased) and should merge after or together with #1925. It replaces #1930 (closed) and covers #1926, which was reverted by #1939. It supersedes #1923, #1933 and #1935.

Wait design

A waiting wfi stays in M. WaitM stalls the whole pipeline, so the instruction in W keeps forwarding.

• privdec:
  • WaitM = wfiM & ~IntPendingM & ~TWTimeoutM
  • One flop, WaitedM <= StallM & (WaitM | WaitedM), records that the wfi in M has waited.
  • One counter, WaitCount <= StallM ? WaitCount + (WaitM & ~msb) : 0, sets the mstatus.TW limit. WFI_TIMEOUT_BIT is unchanged. The counter holds while the wfi stays in M and clears when M advances.
• hazard: StallWCause |= WaitM & ~FlushWCause, FlushWCause = TrapM, StallM = StallW
• trap: ValidIntsM = (Committed | WaitedM) ? 0 : EnabledIntsM. InterruptM no longer gates on wfi.
• Removed: WFIStallM, WFIInterruptedM, StallMCause, LatestUnstalledW, wfiW and the ~wfiM | wfiW gate.

Behavior:

• Interrupt already enabled and pending when the wfi reaches M: the trap is taken on the wfi, and the wfi does not retire.
• wfi woken by a locally enabled interrupt: the wfi retires. If the interrupt is enabled, it is taken with mepc = pc + 4 (wfi_mepc_val).
• TW limit reached (below M with TW = 1, or U mode with S): illegal-instruction trap with mepc = the wfi. The wfi does not retire.

The logic is written so that Zawrs only adds wrs terms to WaitM, the counter and the timeouts.

Bugs fixed

• Stale operand after a woken wfi (hazard: stall Writeback while a wfi waits #1933). W was flushed every waiting cycle, so forwarding stopped. li a0,42; wfi; addi a1,a0,0 gave the old a0, and a following store wrote stale data.
• Interrupt taken one instruction late (trap: take an interrupt enabled before a wfi on the wfi itself #1935). An interrupt already pending, and enabled by a CSR write or xRET just before the wfi, was taken after the wfi retired. It is now taken on the wfi, as intr_mip_mie_xret_csrwr requires.
• wfi both retires and traps (Flush a trapping wfi in the Writeback stage #1926, reverted). A trapping wfi both retired (counted in minstret) and became mepc. Now FlushWCause = TrapM, with the corrected ecall/ebreak comments.
• Spurious trap on the next instruction (privdec: gate the WFI timeout with wfiM #1923). A wfi woken in the cycle its count reached the limit made the next instruction take an illegal-instruction trap.
• Wrong mcause. A TW timeout coinciding with an enabled interrupt wrote mcause = 5 without the interrupt bit. Gating ValidIntsM with WaitedM keeps CauseM consistent.
• Trap CSRs written for a trap not taken (Record trap CSRs and trap HPM events only when the trap is taken (~StallW) #1930). xEPC, xCAUSE and xTVAL were written every cycle TrapM was high, while mstatus, the privilege mode and the PC wait for ~StallW. Under ExternalStall, a trap that changed during the stall overwrote scause/sepc/stval for a trap that was never taken.
  • MTrapM/STrapM and HPM events 22 and 23 now use TrapM & ~StallW at each use. There is no named TrapTakenM signal.
  • Record trap CSRs and trap HPM events only when the trap is taken (~StallW) #1930's ~StallW gate on the WFI counter reset is not needed. The counter no longer uses TrapM; it clears when M advances, which for a trap is exactly when the trap is taken.

Testing

Tests in tests/coverage, listed in coverage64gc:

• From privdec: gate the WFI timeout with wfiM #1923/Flush a trapping wfi in the Writeback stage #1926/hazard: stall Writeback while a wfi waits #1933/trap: take an interrupt enabled before a wfi on the wfi itself #1935: wfitimeoutnext, wfiBackToBack, wfiForward, csrwfiInt.
• New:
  • wfitimeoutint: a TW timeout racing an enabled interrupt.
  • wfiTW: a TW wait woken before the limit; a timed-out wfi that does not retire; U mode.
• From Record trap CSRs and trap HPM events only when the trap is taken (~StallW) #1930: trapcsrstall, hpmtrapcount.

check (rv64gc)	result
Verilator: the 8 tests above, plus divflushstall	all pass
wfitimeout	runs to completion (it has no self-check)
coverage64gc	SUCCESS
Lockstep wfiForward, csrwfiInt	0 mismatches
ACT InterruptsSm/S, rv64gc and rv32gc	32/32 each
lint-wally	clean

On main, wfitimeoutnext, wfiBackToBack, wfiForward, csrwfiInt and wfitimeoutint fail.

wfiTW, wfitimeoutint and wfitimeoutnext are waived in lockstep. ImperasDV traps a wfi below M with TW = 1 at once, while Wally waits a bounded time. Both are legal: mstatus_tw_always_illegal allows the first and mstatus_tw_op the second.

Synthesis was measured on the combined wfi + Zawrs version (sky130, rv64gc core, 200 MHz). Compared with main + Zawrs:

• Hazard + privdec + trap area is 8.6% smaller.
• No path through StallW, StallM, FlushW, StallD or InterruptM gets slower.

It has not been re-measured for this wfi-only version.

🤖 Generated with Claude Code