Repository navigation
feat(bench): record relay worker lock waits with perf #5277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
634197c
chore(quest): claim relay profiling harness
kixelated 7e8a20a
feat(bench): probe profiling support and classify worker lock waits
kixelated 8d47350
Merge origin/main into relay profiling
kixelated 5aade77
Validate profiling capabilities and retain refusal evidence
kixelated 6f92a5b
Merge origin/main into relay profiling
kixelated 695fd7c
feat(bench): capture relay lock waits without root via eu-stack sampling
kixelated 1b9ad55
fix(bench): require resolved MoQ frames for every profiled worker
kixelated ca0d8fc
Merge origin/main into relay profiling
kixelated f70254f
feat(bench): record relay lock waits with perf sched_switch
kixelated c9b7089
feat(bench): sample switch-outs without tracefs, drop frame pointers
kixelated e03c793
Merge origin/main into relay profiling
kixelated 3f02d0a
fix(bench): refuse dropped perf events and keep 16-worker stacks reso…
kixelated 1426999
Merge remote-tracking branch 'origin/main' into merge-5277
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| #!/usr/bin/env python3 | ||
| """Record relay worker off-CPU stacks with perf and attribute std mutex waits.""" | ||
| import argparse | ||
| import json | ||
| import os | ||
| import re | ||
| import subprocess | ||
| import time | ||
| from collections import Counter | ||
| from pathlib import Path | ||
|
|
||
| EVENT = 'context-switches' | ||
| WORKER = re.compile(r"moq-(?:uring|quic)-[0-9]+") | ||
| # The blocking call itself: std's futex wrapper or glibc's (allocator, pthread). | ||
| BLOCKED = re.compile(r"futex_wait|futex_abstimed_wait|lll_lock_wait") | ||
| # kio::Lock is a std Mutex; condvar, allocator and parking waits are separate. | ||
| # An inlined frame carries only its short name. | ||
| MUTEX = re.compile(r"(?:\bMutex>?::|^)lock_contended$") | ||
| HEADER = re.compile(r"\s*(\d+)\s+(\d+)\.(\d+):\s+(.*)") | ||
| FRAME = re.compile(r"\s+[0-9a-f]+\s+(.*?)\s+\((.*)\)$") | ||
| # A switch-out is sampled in kernel context (paranoid <= 1), and the ring buffer that keeps | ||
| # 16 workers from dropping samples exceeds the locked-memory limit unless paranoid is -1. | ||
| SETUP = 'sudo sysctl -w kernel.perf_event_paranoid=-1' | ||
|
|
||
|
|
||
| def require(): | ||
| """Refuse to record unless perf may sample the relay in kernel context with a large buffer.""" | ||
| paranoid = int(Path('/proc/sys/kernel/perf_event_paranoid').read_text()) | ||
| if paranoid != -1: | ||
| raise ValueError(f'perf needs kernel samples and unlimited buffers (perf_event_paranoid={paranoid}); run:\n{SETUP}') | ||
|
|
||
|
|
||
| def parse(text): | ||
| """Split `perf script` output into (tid, nanoseconds, event, innermost-first frames) records.""" | ||
| records, frames = [], None | ||
| for line in text.splitlines(): | ||
| if match := HEADER.match(line): | ||
| tid, seconds, fraction, event = match.groups() | ||
| frames = [] | ||
| nanoseconds = int(seconds) * 1_000_000_000 + int(fraction.ljust(9, '0')) | ||
| event = event.strip().rstrip(':') | ||
| # A dropped switch would silently shorten or lose a wait. | ||
| if event.startswith('PERF_RECORD_LOST'): | ||
| raise ValueError(f'perf dropped events ({event}); the recording is incomplete') | ||
| records.append((tid, nanoseconds, event, frames)) | ||
| elif frames is not None and (match := FRAME.match(line)): | ||
| symbol, dso = match.groups() | ||
| # Kernel frames all show the scheduler; the user stack names the wait. | ||
| if not dso.startswith('['): | ||
| frames.append(re.sub(r'\+0x[0-9a-f]+$', '', symbol)) | ||
| return records | ||
|
|
||
|
|
||
| def waits(records, end): | ||
| """Pair each switch-out stack with the thread's next switch-in for an exact off-CPU duration.""" | ||
| pending, result = {}, [] | ||
| for tid, at, event, frames in records: | ||
| if event == EVENT: | ||
| pending[tid] = (at, frames) | ||
| elif event.startswith('PERF_RECORD_SWITCH IN') and tid in pending: | ||
| start, stack = pending.pop(tid) | ||
| result.append((tid, at - start, stack)) | ||
| # A wait still open when recording stops counts up to the end of the window. | ||
| result.extend((tid, end - start, stack) for tid, (start, stack) in pending.items()) | ||
| return result | ||
|
|
||
|
|
||
| def classify(stack): | ||
| """An off-CPU interval is a mutex wait only when blocked in the futex under lock_contended.""" | ||
| if not any(BLOCKED.search(frame) for frame in stack): | ||
| return 'other' | ||
| return 'mutex' if any(MUTEX.search(frame) for frame in stack) else 'futex' | ||
|
|
||
|
|
||
| def reduce(names, intervals, before, after, window_ns): | ||
| """Sum each worker's off-CPU time by class, as a share of its CPU time.""" | ||
| workers, idle, folded = {}, [], Counter() | ||
| for tid, name in names.items(): | ||
| mine = [(wait, stack) for owner, wait, stack in intervals if owner == tid] | ||
| cpu = after[tid] - before[tid] | ||
| # Asleep for the whole window, such as a worker with no connections. | ||
| if not mine and cpu == 0: | ||
| idle.append(name) | ||
| continue | ||
| if not mine: | ||
| raise ValueError(f'worker {name} ({tid}) ran but never switched out') | ||
| if not any(re.search(r'\bmoq_|\bkio::', frame) for _, stack in mine for frame in stack): | ||
| raise ValueError(f'worker {name} ({tid}) has no resolved MoQ frames; build with the profiling profile and frame pointers') | ||
| if cpu <= 0: | ||
| raise ValueError(f'worker {name} ({tid}) has no advancing CPU counter') | ||
| total = Counter() | ||
| for wait, stack in mine: | ||
| kind = classify(stack) | ||
| total[kind] += wait | ||
| if kind == 'mutex': | ||
| folded[';'.join(reversed(stack))] += wait // 1000 | ||
| workers[tid] = {'comm': name, 'switches': len(mine), 'cpu_ns': cpu, 'mutex_wait_ns': total['mutex'], | ||
| 'other_futex_wait_ns': total['futex'], 'other_off_cpu_ns': total['other'], | ||
| 'mutex_wait_per_cpu': total['mutex'] / cpu} | ||
| if not workers: | ||
| raise ValueError('no worker ran during the window') | ||
| return {'version': 2, 'window_ns': window_ns, 'workers': workers, 'idle': sorted(idle), | ||
| 'scope': 'exact off-CPU time blocked in a futex under std Mutex::lock_contended; spinning is excluded'}, folded | ||
|
|
||
|
|
||
| def schedstat(pid, tids): | ||
| return {tid: int((Path('/proc') / str(pid) / 'task' / tid / 'schedstat').read_text().split()[0]) for tid in tids} | ||
|
|
||
|
|
||
| def record(pid, delay, duration, directory): | ||
| """Record one steady window of worker switch-outs, then symbolize after the relay is released.""" | ||
| require() | ||
| directory.mkdir(parents=True, exist_ok=True) | ||
| tasks = Path('/proc') / str(pid) / 'task' | ||
| names = {task.name: name for task in tasks.iterdir() if WORKER.fullmatch(name := (task / 'comm').read_text().strip())} | ||
| if not names: | ||
| raise ValueError(f'PID {pid} has no moq-uring-N or moq-quic-N workers') | ||
| time.sleep(delay) | ||
| data = directory / 'perf.data' | ||
| before, start = schedstat(pid, names), time.monotonic_ns() | ||
| # A period of 1 samples every switch-out. The sample fires in the outgoing thread, so -p | ||
| # never sees a worker's switch-in; --switch-events records it. glibc's syscall wrapper | ||
| # keeps no frame pointer, so frame-pointer unwinding drops its caller, lock_contended; | ||
| # DWARF unwinding keeps it. A monotonic clock lets open waits close at the window's end. | ||
| # The default 512 KiB buffer drops samples at 16 workers; 2 MiB does not. | ||
| subprocess.run(['perf', 'record', '--quiet', '--no-buildid-cache', '-e', EVENT, '-c', '1', '--call-graph', 'dwarf', '--switch-events', | ||
| '-k', 'monotonic', '-m', '2M', '-p', str(pid), '-o', str(data), '--', 'sleep', str(duration)], check=True) | ||
| after, end = schedstat(pid, names), time.monotonic_ns() | ||
| # Symbols come from the local binaries; never wait on a debuginfod server. | ||
| script = subprocess.run(['perf', 'script', '-i', str(data), '--show-switch-events', '--show-lost-events', '--inline', '--ns', | ||
| '-F', 'tid,time,event,ip,sym,dso'], capture_output=True, text=True, check=True, | ||
| env=dict(os.environ, DEBUGINFOD_URLS='')) | ||
| result, folded = reduce(names, waits(parse(script.stdout), end), before, after, end - start) | ||
| (directory / 'lock.json').write_text(json.dumps(result, indent=2) + '\n') | ||
| (directory / 'mutex.folded').write_text(''.join(f'{stack} {count}\n' for stack, count in folded.most_common())) | ||
|
|
||
|
|
||
| def main(): | ||
| parser = argparse.ArgumentParser(description=__doc__) | ||
| commands = parser.add_subparsers(dest='command', required=True) | ||
| commands.add_parser('check', help='refuse unless perf can sample the relay in kernel context') | ||
| capture = commands.add_parser('record', help='record worker off-CPU stacks of PID into DIRECTORY') | ||
| capture.add_argument('pid', type=int) | ||
| capture.add_argument('directory', type=Path) | ||
| capture.add_argument('--delay', type=float, default=0) | ||
| capture.add_argument('--duration', type=float, required=True) | ||
| args = parser.parse_args() | ||
| try: | ||
| if args.command == 'check': | ||
| require() | ||
| else: | ||
| record(args.pid, args.delay, args.duration, args.directory) | ||
| except (ValueError, OSError, subprocess.CalledProcessError) as error: | ||
| parser.exit(1, f'profile: {error}\n') | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| main() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| import importlib.util | ||
| import unittest | ||
| from pathlib import Path | ||
| from unittest.mock import patch | ||
|
|
||
| spec = importlib.util.spec_from_file_location('profile', Path(__file__).with_name('profile.py')) | ||
| profile = importlib.util.module_from_spec(spec) | ||
| spec.loader.exec_module(profile) | ||
|
|
||
| FIXTURE = Path(__file__).with_name('profile.test.txt').read_text() | ||
| IDLE, MUTEX, ALLOCATOR = '2152705', '2152709', '2152707' | ||
| # Every fixture wait closes, so the window end only has to follow the last event. | ||
| END = 271707 * 1_000_000_000 | ||
|
|
||
|
|
||
| class Parse(unittest.TestCase): | ||
| def test_recorded_switches_pair_into_exact_waits(self): | ||
| records = profile.parse(FIXTURE) | ||
| self.assertEqual([event for _, _, event, _ in records], ['context-switches', 'PERF_RECORD_SWITCH IN'] * 3) | ||
| # Kernel frames are dropped; inlined user frames are kept innermost first. | ||
| self.assertFalse(any('__schedule' in frame for _, _, _, frames in records for frame in frames)) | ||
| stacks = {tid: (wait, stack) for tid, wait, stack in profile.waits(records, END)} | ||
| self.assertEqual(stacks[IDLE][0], 21661) | ||
| self.assertEqual(stacks[MUTEX][0], 6315) | ||
| self.assertEqual(stacks[ALLOCATOR][0], 13709) | ||
| # DWARF unwinding keeps the caller of glibc's frameless syscall wrapper. | ||
| self.assertEqual(stacks[MUTEX][1][:3], ['syscall', 'std::sys::pal::unix::futex::futex_wait', | ||
| '<std::sys::sync::mutex::futex::Mutex>::lock_contended']) | ||
|
|
||
| def test_open_wait_runs_to_the_window_end(self): | ||
| # Even when the switch-out is the last recorded event. | ||
| records = [('2', 5, 'PERF_RECORD_SWITCH IN', []), ('1', 10, 'context-switches', ['a'])] | ||
| self.assertEqual(profile.waits(records, 50), [('1', 40, ['a'])]) | ||
|
|
||
| def test_lost_events_refused(self): | ||
| with self.assertRaises(ValueError): | ||
| profile.parse(FIXTURE + '2152705 271706.914950000: PERF_RECORD_LOST lost 3\n') | ||
|
|
||
|
|
||
| class Classify(unittest.TestCase): | ||
| def test_recorded_stacks(self): | ||
| stacks = {tid: stack for tid, _, stack in profile.waits(profile.parse(FIXTURE), END)} | ||
| self.assertEqual(profile.classify(stacks[MUTEX]), 'mutex') | ||
| self.assertEqual(profile.classify(stacks[ALLOCATOR]), 'futex') | ||
| self.assertEqual(profile.classify(stacks[IDLE]), 'other') | ||
|
|
||
| def test_spinning_is_not_a_wait(self): | ||
| self.assertEqual(profile.classify(['<std::sys::sync::mutex::futex::Mutex>::lock_contended', 'moq_net::serve']), 'other') | ||
|
|
||
|
|
||
| class Reduce(unittest.TestCase): | ||
| def test_recorded_waits_by_class_over_cpu(self): | ||
| intervals = profile.waits(profile.parse(FIXTURE), END) | ||
| names = {IDLE: 'moq-quic-0', MUTEX: 'moq-quic-1', ALLOCATOR: 'moq-quic-2'} | ||
| before = dict.fromkeys(names, 0) | ||
| result, folded = profile.reduce(names, intervals, before, dict.fromkeys(names, 3000), 1) | ||
| rows = result['workers'] | ||
| self.assertEqual(rows[MUTEX]['mutex_wait_ns'], 6315) | ||
| self.assertAlmostEqual(rows[MUTEX]['mutex_wait_per_cpu'], 6315 / 3000) | ||
| self.assertEqual(rows[ALLOCATOR]['other_futex_wait_ns'], 13709) | ||
| self.assertEqual(rows[IDLE]['other_off_cpu_ns'], 21661) | ||
| [(stack, weight)] = folded.items() | ||
| self.assertTrue(stack.endswith('lock_contended;std::sys::pal::unix::futex::futex_wait;syscall')) | ||
| self.assertEqual(weight, 6) | ||
|
|
||
| def test_silent_worker_refused(self): | ||
| intervals = [('1', 5, ['moq_net::serve'])] | ||
| # Worker 2 ran but recorded no switch-out, so the capture missed it. | ||
| with self.assertRaises(ValueError): | ||
| profile.reduce({'1': 'moq-quic-0', '2': 'moq-quic-1'}, intervals, {'1': 0, '2': 0}, {'1': 1, '2': 1}, 1) | ||
| # Switched out without a CPU advance: the counter is broken. | ||
| with self.assertRaises(ValueError): | ||
| profile.reduce({'1': 'moq-quic-0'}, intervals, {'1': 5}, {'1': 5}, 1) | ||
|
|
||
| def test_sleeping_worker_reported_idle(self): | ||
| # Light load at 16 workers leaves some without connections, asleep for the whole window. | ||
| intervals = [('1', 5, ['moq_net::serve'])] | ||
| result, _ = profile.reduce({'1': 'moq-quic-0', '2': 'moq-quic-1'}, intervals, {'1': 0, '2': 7}, {'1': 1, '2': 7}, 1) | ||
| self.assertEqual(list(result['workers']), ['1']) | ||
| self.assertEqual(result['idle'], ['moq-quic-1']) | ||
| with self.assertRaises(ValueError): | ||
| profile.reduce({'2': 'moq-quic-1'}, [], {'2': 7}, {'2': 7}, 1) | ||
|
|
||
| def test_unresolved_stacks_refused_per_worker(self): | ||
| # One resolved worker must not vouch for another whose unwinding failed. | ||
| intervals = [('1', 5, ['moq_net::serve']), ('2', 5, ['[unknown]', 'syscall'])] | ||
| with self.assertRaises(ValueError): | ||
| profile.reduce({'1': 'moq-quic-0', '2': 'moq-quic-1'}, intervals, {'1': 0, '2': 0}, {'1': 1, '2': 1}, 1) | ||
|
|
||
|
|
||
| class Require(unittest.TestCase): | ||
| def check(self, paranoid): | ||
| with patch.object(profile.Path, 'read_text', return_value=paranoid + '\n'): | ||
| profile.require() | ||
|
|
||
| def test_refusal_names_the_command(self): | ||
| # Ubuntu's default is 4; 2 forbids kernel samples; 1 and 0 cap the locked buffer. | ||
| for paranoid in ['4', '2', '1', '0']: | ||
| with self.assertRaises(ValueError) as refused: | ||
| self.check(paranoid) | ||
| self.assertIn('sudo sysctl -w kernel.perf_event_paranoid=-1', str(refused.exception)) | ||
| self.check('-1') | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| unittest.main() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Align CPU counters with the perf capture window.
Line 120 reads
schedstatbeforeperf recordstarts; Line 128 reads it afterperf recordexits. If a worker runs during perf setup and then sleeps throughout capture,reduce()rejects a valid recording because its CPU counter advanced but it has no recorded switch-out. The extra CPU time also lowers the reported wait/CPU ratio. Synchronize counter reads with capture start and stop; perf provides a control interface for coordinated measurement. (man7.org)🤖 Prompt for AI Agents
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is real, and the PR already lists it as a follow-up: perf records no switch-out samples for its first ~115 ms, while schedstat brackets the whole perf run. That biases wait/CPU about 2% low on the 5 s window.
I'm leaving it out of this PR. The planned fix is to derive CPU from the recorded switch-in/out intervals and drop schedstat entirely, rather than add perf's control FIFO to gate both reads. That removes the second clock and also closes the edge case you describe.
That edge case makes the harness refuse ("ran but never switched out") rather than publish a wrong number. It also can't hit the nightly
fanout-heavy 4run, where every worker is busy throughout.(Written by Claude Opus 5.5)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kixelatedThanks for the clarification. Deriving CPU time from recorded switch-in/out intervals avoids the separateschedstatmeasurement window.The edge case causes a refusal, not an incorrect report. Your explanation also limits the risk for nightly’s busy
fanout-heavy 4run. Deferring the fix is reasonable.Would you like me to open a follow-up issue for the interval-based CPU calculation?
✏️ Learnings added
You are interacting with an AI system.