[bpf-ci-bot] lru_lock_nmi test fails deterministically on aarch64 due to pending_free node reclamation race

[kernel-patches-review-bot]

Summary

The newly-merged lru_lock_nmi selftest (commit 6e1e4a9d60ed, Jun 7 2026) fails deterministically on aarch64 and intermittently on x86_64 with drain_then_verify_capacity returning -EIO. The test's post-stress verification runs before the rqspinlock recovery path has time to reclaim nodes left in pending_free state.

Failure Details

• Test / Component: lru_lock_nmi (common_lru, no_common_lru, percpu_lru subtests)
• Frequency: Every aarch64 CI run since Jun 7; occasional x86_64 (percpu_lru subtest). Observed in 5+ independent PRs.
• Failure mode: -EIO from drain_then_verify_capacity — after NMI stress + drain + refill, some inserted keys are missing due to unexpected LRU eviction
• Affected architectures: aarch64 (deterministic), x86_64 (intermittent, percpu_lru subtest)
• CI runs observed:
  • https://github.com/kernel-patches/bpf/actions/runs/27132919096 (Non const src shift — aarch64 cpuv4)
  • https://github.com/kernel-patches/bpf/actions/runs/27130169123 (Uprobe path — aarch64 no_alu32)
  • https://github.com/kernel-patches/bpf/actions/runs/27138225449 (netdev CI — aarch64 no_alu32 + x86_64 llvm-21)
  • https://github.com/kernel-patches/bpf/actions/runs/27136325253 (arena split/fork — aarch64 cpuv4)

Root Cause Analysis

Commit 89edbdfc5d03 ("bpf: Fix NMI/tracepoint re-entry deadlock on lru locks") converts LRU locks to raw_res_spin_lock (rqspinlock) and adds recovery paths for lock acquisition failure. When a lock cannot be acquired (NMI re-entry), nodes are marked pending_free=1 and either:

• Published to a lockless free_llist for later pickup
• Left on the pending/inactive list with pending_free=1 for lazy reclamation

The lazy reclamation happens during:

• __local_list_flush() (called from bpf_lru_list_pop_free_to_local())
• __bpf_lru_list_shrink_inactive() (called during LRU shrink)

The test (tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c:228) calls drain_then_verify_capacity() immediately after destroying perf event links and joining hammer threads. This function:

1. Deletes all keys (drains the map)
2. Refills with exactly MAP_ENTRIES (64) unique keys spread across CPUs
3. Verifies all keys are retrievable

If pending_free nodes haven't been reclaimed yet, they still consume map capacity. The refill phase inserts 64 keys into a map that internally has fewer than 64 free slots (some occupied by pending_free nodes), triggering LRU eviction. Evicted keys then fail the lookup verification.

On aarch64, perf events don't use true NMI but IRQ-level exceptions. The slightly different timing characteristics compared to x86 NMI cause more frequent lock contention windows, making the failure deterministic rather than intermittent.

Proposed Fix

See 0001-selftests-bpf-Fix-lru_lock_nmi-post-stress-verificat.patch:

1. Add 10ms sleep after destroying perf links and joining threads — allows in-flight interrupt handlers to complete.
2. Retry verification up to 3 times — each drain+refill cycle triggers internal LRU list maintenance that reclaims pending_free nodes. After 2-3 cycles all stranded nodes are recovered.

This approach is robust because it doesn't rely on specific timing; it uses the kernel's own reclamation machinery to recover the nodes before asserting.

Impact

Every BPF CI run on aarch64 (test_progs, test_progs_no_alu32, test_progs_cpuv4) fails due to this test. This blocks meaningful CI signal for all submitted PRs on aarch64. On x86_64, the percpu_lru subtest fails intermittently, causing additional noise.

References

• Commit 89edbdfc5d03 ("bpf: Fix NMI/tracepoint re-entry deadlock on lru locks")
• Commit 6e1e4a9d60ed ("selftests/bpf: Stress LRU rqspinlock recovery paths")
• Patch series: https://lore.kernel.org/bpf/20260607-lru_map_spin-v3-0-bcd9332e911b@meta.com/
• Author: Mykyta Yatsenko yatsenko@meta.com

[kernel-patches-review-bot]

0001-selftests-bpf-Fix-lru_lock_nmi-post-stress-verificat.patch

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: BPF CI Bot <bot+bpf-ci@kernel.org>
Date: Sun, 8 Jun 2026 00:00:00 +0000
Subject: [PATCH] selftests/bpf: Fix lru_lock_nmi post-stress verification
 failure

The lru_lock_nmi test's drain_then_verify_capacity() fails
deterministically on aarch64 and intermittently on x86_64 (percpu_lru
subtest) with -EIO because nodes marked pending_free during the NMI
stress phase haven't been reclaimed yet when the verification runs.

The rqspinlock recovery path (commit 89edbdfc5d03 "bpf: Fix
NMI/tracepoint re-entry deadlock on lru locks") marks nodes
pending_free=1 when it cannot acquire the LRU list lock during NMI
re-entry. These nodes are reclaimed lazily during subsequent
__local_list_flush() or __bpf_lru_list_shrink_inactive() operations.
If the verification phase runs before this lazy reclamation completes,
the map appears to have less capacity than expected.

Fix this by:

1. Adding a 10ms sleep after destroying perf event links and joining
   hammer threads, giving any in-flight interrupt handlers time to
   complete.

2. Retrying drain_then_verify_capacity() up to 3 times. Each drain
   cycle triggers bpf_lru_push_free() -> __local_list_flush() which
   reclaims pending_free nodes. By the second or third attempt, all
   stranded nodes have been recovered.

Fixes: 6e1e4a9d60ed ("selftests/bpf: Stress LRU rqspinlock recovery paths")
Generated-by: BPF CI Bot (claude-opus-4-6-20250616) <bot+bpf-ci@kernel.org>
Signed-off-by: BPF CI Bot <bot+bpf-ci@kernel.org>
---
 .../selftests/bpf/prog_tests/lru_lock_nmi.c  | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c b/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c
index 60666a9ba41f..1ed94ce84516 100644
--- a/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c
+++ b/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c
@@ -18,6 +18,8 @@
 #define KEY_RANGE	(MAP_ENTRIES * 2)
 #define STRESS_NS	(500 * 1000 * 1000ULL)

+#define VERIFY_RETRIES	3
+
 struct hammer_arg {
 	int map_fd;
 	int cpu;
@@ -156,6 +158,7 @@ static void run_variant(enum bpf_map_type type, __u32 map_flags, const char *nam
 	struct lru_lock_nmi *skel = NULL;
 	int map_fd, i, err, nr_threads = 0, pmu_fd = -1;
 	__u64 deadline;
+	int retries;

 	nr_cpus = libbpf_num_possible_cpus();
 	if (!ASSERT_GT(nr_cpus, 0, "num_cpus"))
@@ -224,8 +227,21 @@ static void run_variant(enum bpf_map_type type, __u32 map_flags, const char *nam
 		}
 	}

+	/*
+	 * Allow in-flight perf/NMI handlers to complete before verifying.
+	 */
+	usleep(10000);
+
 	ASSERT_GT(skel->bss->hits, 0, "nmi_bpf_ran");
-	ASSERT_OK(drain_then_verify_capacity(map_fd, nr_cpus), "drain_then_verify_capacity");
+
+	/*
+	 * Retry: each drain+refill cycle triggers LRU list maintenance
+	 * that reclaims nodes left in pending_free by rqspinlock recovery.
+	 */
+	for (retries = 0; retries < VERIFY_RETRIES; retries++)
+		if (drain_then_verify_capacity(map_fd, nr_cpus) == 0)
+			break;
+	ASSERT_LT(retries, VERIFY_RETRIES, "drain_then_verify_capacity");

 cleanup:
 	for (i = 0; i < nr_cpus; i++) {
--
2.43.0