Summary
The lru_lock_nmi test (merged Jun 7, commit 6e1e4a9d60ed) fails deterministically on aarch64 and in most runs on x86_64, blocking CI signal for all PRs. The test is not denylisted, so every failed run shows this test as the failure reason — obscuring real regressions. A kernel-side fix exists (#488) but has not been merged yet. Adding a DENYLIST entry restores CI usefulness immediately.
Failure Details
- Test / Component:
lru_lock_nmi (subtests: common_lru, no_common_lru, percpu_lru)
- Frequency: Every aarch64 run; most x86_64 runs (common_lru + percpu_lru). Observed in 8+ independent PRs over the past 48 hours alone.
- Failure mode: -EIO from
drain_then_verify_capacity — pending_free nodes not reclaimed before verification
- Affected architectures: aarch64 (deterministic), x86_64 (most runs). s390x skips the test (no perf HW events).
- CI runs observed (June 15 alone):
Root Cause Analysis
Same as documented in vmtest#488. The rqspinlock recovery path (commit 89edbdfc5d03) marks LRU nodes as pending_free=1 when NMI lock acquisition fails. Lazy reclamation happens during subsequent list operations, but the test verification runs immediately after stress — before reclamation completes.
New finding: Issue #488 documented x86_64 failures as "intermittent, percpu_lru subtest only." Current data shows x86_64 now also fails the common_lru subtest in most runs. The issue has worsened or the original characterization was incomplete.
Proposed Fix
Add lru_lock_nmi to the CI DENYLIST for aarch64 and x86_64. See attached patch for kernel-patches/vmtest repository.
The kernel-side fix (sleep + retry in the test) is tracked in vmtest#488 and should be submitted upstream. Once merged, the DENYLIST entry can be removed.
Impact
Without this DENYLIST entry, every aarch64 CI run and most x86_64 CI runs fail with lru_lock_nmi as the only failure — hiding real regressions from PR authors and maintainers. This is the single largest source of CI noise currently.
References
- vmtest#488: [bpf-ci-bot] lru_lock_nmi test fails deterministically on aarch64 due to pending_free node reclamation race
- Commit 6e1e4a9d60ed ("selftests/bpf: Stress LRU rqspinlock recovery paths")
- Commit 89edbdfc5d03 ("bpf: Fix NMI/tracepoint re-entry deadlock on lru locks")
Summary
The
lru_lock_nmitest (merged Jun 7, commit 6e1e4a9d60ed) fails deterministically on aarch64 and in most runs on x86_64, blocking CI signal for all PRs. The test is not denylisted, so every failed run shows this test as the failure reason — obscuring real regressions. A kernel-side fix exists (#488) but has not been merged yet. Adding a DENYLIST entry restores CI usefulness immediately.Failure Details
lru_lock_nmi(subtests:common_lru,no_common_lru,percpu_lru)drain_then_verify_capacity— pending_free nodes not reclaimed before verificationRoot Cause Analysis
Same as documented in vmtest#488. The rqspinlock recovery path (commit 89edbdfc5d03) marks LRU nodes as
pending_free=1when NMI lock acquisition fails. Lazy reclamation happens during subsequent list operations, but the test verification runs immediately after stress — before reclamation completes.New finding: Issue #488 documented x86_64 failures as "intermittent, percpu_lru subtest only." Current data shows x86_64 now also fails the
common_lrusubtest in most runs. The issue has worsened or the original characterization was incomplete.Proposed Fix
Add
lru_lock_nmito the CI DENYLIST for aarch64 and x86_64. See attached patch forkernel-patches/vmtestrepository.The kernel-side fix (sleep + retry in the test) is tracked in vmtest#488 and should be submitted upstream. Once merged, the DENYLIST entry can be removed.
Impact
Without this DENYLIST entry, every aarch64 CI run and most x86_64 CI runs fail with
lru_lock_nmias the only failure — hiding real regressions from PR authors and maintainers. This is the single largest source of CI noise currently.References