Skip to content

Commit 991bbc8

Browse files
committed
Merge main into CUDA 13 source-root rename
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
2 parents 4fb18b9 + b288fc8 commit 991bbc8

8 files changed

Lines changed: 244 additions & 60 deletions

File tree

‎.github/actions/lychee-links/action.yml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ runs:
8484
echo "key=${prefix}${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "${GITHUB_OUTPUT}"
8585
8686
- name: Restore successful link checks
87+
id: cache-restore
8788
# Refresh mode deliberately skips earlier URL results.
8889
if: ${{ inputs.refresh != 'true' }}
8990
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
@@ -93,6 +94,25 @@ runs:
9394
restore-keys: ${{ steps.cache.outputs.prefix }}
9495
enableCrossOsArchive: true
9596

97+
- name: Explain a missing shared Lychee cache
98+
# Prefix restores are normal; cache-hit only identifies exact key matches.
99+
if: ${{ inputs.refresh != 'true' && steps.cache-restore.outputs.cache-matched-key == '' }}
100+
shell: bash --noprofile --norc -euo pipefail {0}
101+
env:
102+
KIND: ${{ inputs.kind }}
103+
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
104+
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/workflows/lychee.yml
105+
run: |
106+
message="No shared Lychee ${KIND} cache was restored for this checker configuration. Checking links without this baseline may take longer and make more network requests."
107+
context="This is expected after checker changes in a PR; a refresh on ${DEFAULT_BRANCH} cannot seed settings that exist only in a PR."
108+
hint="For an unexpected miss on ${DEFAULT_BRANCH}, maintainers can run CI: Lychee checks and caches (lychee.yml) on ${DEFAULT_BRANCH} with refresh-cache enabled."
109+
echo "::notice title=Lychee ${KIND} cache::${message} ${context} ${hint} ${WORKFLOW_URL}"
110+
{
111+
printf '### Lychee %s cache\n\n' "$KIND"
112+
printf '%s\n\n%s\n\n%s\n\n' "$message" "$context" "$hint"
113+
printf '[Open the cache refresh workflow](%s).\n' "$WORKFLOW_URL"
114+
} >> "$GITHUB_STEP_SUMMARY"
115+
96116
- name: Check links through the configured pre-commit hook
97117
shell: bash --noprofile --norc -euo pipefail {0}
98118
env:
@@ -107,7 +127,7 @@ runs:
107127
--report "${RUNNER_TEMP}/lychee-reports/lychee-${KIND}.json"
108128
109129
- name: Publish fresh successful link checks
110-
# Only nightly refreshes publish, so every pull request restores the same
130+
# Only refresh runs publish, so every pull request restores the same
111131
# baseline. Saving keeps original timestamps and never caches failures.
112132
if: ${{ !cancelled() && inputs.refresh == 'true' }}
113133
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0

‎.github/workflows/lychee.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@
44

55
# Warm authored and rendered link caches through the configured pre-commit
66
# hooks. Ordinary PRs reuse authored results; full docs CI reuses rendered
7-
# results. Nightly starts afresh, reports unresolved links as warnings, and is
8-
# the only producer of both shared baselines.
7+
# results. Nightly and manual dispatches with refresh-cache=true start afresh,
8+
# report unresolved links as warnings, and can produce both shared baselines.
99

10-
name: "CI: Documentation links"
10+
name: "CI: Lychee checks and caches"
1111

1212
on:
1313
workflow_call:

‎ci/README.md‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,9 +88,9 @@ existing allowance for at most ten distinct HTTP 429 URLs.
8888

8989
### Shared URL Caches
9090

91-
Nightly is the only producer. It skips restoring, checks afresh, and publishes
92-
successful results; every other job restores the newest baseline and never
93-
publishes. Failed URLs and filesystem targets are never cached. Successes keep
91+
Nightly and manual refreshes skip restoring, check afresh, and publish
92+
successful results; ordinary checks restore the newest baseline and never
93+
publish. Failed URLs and filesystem targets are never cached. Successes keep
9494
their original timestamps and expire after three days, so a missed nightly
9595
refresh does not break reuse, but a link that breaks after a nightly success
9696
can stay unnoticed for up to 72 hours.
@@ -104,6 +104,21 @@ fragments and Lychee's cache does not record which validation ran. URL results
104104
are shared across operating systems; pre-commit's installed environments keep
105105
their OS- and Python-specific caches.
106106

107+
#### Refreshing Shared Caches
108+
109+
To rebuild both shared caches manually, open
110+
[CI: Lychee checks and caches](https://github.com/NVIDIA/cuda-python/actions/workflows/lychee.yml)
111+
(`.github/workflows/lychee.yml`), select the default branch, and enable
112+
`refresh-cache` before choosing **Run workflow**. This runs the cache refresh
113+
without the rest of nightly CI.
114+
115+
Checks that restore no matching baseline add a notice and a job-summary hint.
116+
They still check links, with potentially more runtime and network traffic.
117+
Checker changes in a PR intentionally start with a cold cache; refreshing the
118+
default branch cannot seed a cache for settings that exist only in that PR.
119+
Refreshing rebuilds cached successes; broken links and upstream rate limits
120+
still need their usual investigation.
121+
107122
### Repository Links
108123

109124
Documentation can link to a path added or moved in the same PR using its

‎cuda_core/tests/AGENTS.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,15 @@ Follow these rules when adding or moving shared test code:
9696
`tests/helpers/` instead.
9797
- Import helpers explicitly from the test root, for example:
9898
`from helpers.memory import create_managed_memory_resource_or_skip`.
99+
- Reuse `run_child` and `noop` from `helpers.child_processes` for synchronous
100+
child-process tests. The runner applies the sanitizer-aware timeout, kills
101+
surviving children before asserting success, and closes process handles.
102+
Before measuring file-descriptor counts around spawned processes, call
103+
`warmup_multiprocessing` from the same module to start the persistent
104+
resource-tracker pipe outside the baseline. Keep explicit process management
105+
when the test needs to communicate with a running child or inspect its lifetime;
106+
use `kill_subprocesses` before assertions and `track_child_processes` for
107+
fixture cleanup so a stuck child cannot block IPC resource teardown.
99108
- Search `tests/helpers/` and `cuda_python_test_helpers/` for prior art
100109
before adding a new helper; consolidate duplicates across
101110

‎cuda_core/tests/helpers/child_processes.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111
"""
1212

1313
import contextlib
14+
import functools
15+
import multiprocessing as mp
1416
import multiprocessing.process
1517
import weakref
1618

@@ -64,6 +66,36 @@ def kill_subprocesses(*processes):
6466
return killed
6567

6668

69+
def run_child(target, *args):
70+
"""Run ``target(*args)`` in a child, killing it on timeout or interrupted join.
71+
72+
Fail unless the child exits successfully. Close the Process object before
73+
returning so its handles cannot affect file-descriptor leak checks.
74+
"""
75+
with contextlib.closing(mp.Process(target=target, args=args)) as process:
76+
try:
77+
process.start()
78+
process.join(timeout=child_timeout_sec())
79+
finally:
80+
survivors = kill_subprocesses(process)
81+
assert not survivors, "child did not exit within timeout"
82+
assert process.exitcode == 0, f"child exited with {process.exitcode}"
83+
84+
85+
def noop(*args):
86+
"""A picklable child target that accepts and discards any arguments."""
87+
88+
89+
@functools.cache
90+
def warmup_multiprocessing():
91+
"""Start spawn's persistent resource-tracker pipe before counting open FDs.
92+
93+
Cache only a successful warmup; the pipe outlives individual Process objects.
94+
"""
95+
assert mp.get_start_method() == "spawn"
96+
run_child(noop)
97+
98+
6799
@contextlib.contextmanager
68100
def track_child_processes():
69101
"""Context manager that kills any ``multiprocessing.Process`` children still

‎cuda_core/tests/memory_ipc/test_leaks.py‎

Lines changed: 4 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
else:
1414
HAVE_PSUTIL = True
1515
import pytest
16-
from helpers.child_processes import child_timeout_sec, kill_subprocesses
16+
from helpers.child_processes import child_timeout_sec, kill_subprocesses, noop, run_child, warmup_multiprocessing
1717

1818
CHILD_TIMEOUT_SEC = child_timeout_sec()
1919
NBYTES = 64
@@ -40,16 +40,7 @@ def test_alloc_handle(ipc_memory_resource):
4040
def exec_success(obj, number=1):
4141
"""Successfully run a child process."""
4242
for _ in range(number):
43-
process = mp.Process(target=child_main, args=(obj,))
44-
process.start()
45-
process.join(timeout=CHILD_TIMEOUT_SEC)
46-
survivors = kill_subprocesses(process)
47-
assert not survivors, "child did not exit within timeout"
48-
assert process.exitcode == 0
49-
50-
51-
def child_main(obj, *args):
52-
pass
43+
run_child(noop, obj)
5344

5445

5546
def exec_launch_failure(obj, number=1):
@@ -78,7 +69,7 @@ def exec_reduce_failure(obj, number=1):
7869
for _ in range(number):
7970
fails_to_reduce = Irreducible()
8071
with contextlib.suppress(RuntimeError):
81-
mp.Process(target=child_main, args=(obj, fails_to_reduce)).start()
72+
mp.Process(target=noop, args=(obj, fails_to_reduce)).start()
8273

8374

8475
class Irreducible:
@@ -137,7 +128,7 @@ def __init__(self):
137128
self.process = psutil.Process()
138129

139130
def __enter__(self):
140-
prime()
131+
warmup_multiprocessing()
141132
gc.collect()
142133
self.initial_fds = self.process.num_fds()
143134
return self
@@ -150,20 +141,3 @@ def __exit__(self, exc_type, exc_val, exc_tb):
150141
final_fds = self.process.num_fds()
151142
assert final_fds == self.initial_fds, f"file descriptors leaked: {self.initial_fds} -> {final_fds}"
152143
return False
153-
154-
155-
prime_was_run = False
156-
157-
158-
def prime():
159-
"""Multiprocessing consumes a file descriptor on first launch."""
160-
assert mp.get_start_method() == "spawn"
161-
global prime_was_run
162-
if not prime_was_run:
163-
process = mp.Process()
164-
process.start()
165-
process.join(timeout=CHILD_TIMEOUT_SEC)
166-
survivors = kill_subprocesses(process)
167-
assert not survivors, "child did not exit within timeout"
168-
assert process.exitcode == 0
169-
prime_was_run = True

‎cuda_core/tests/memory_ipc/test_vmm_ipc.py‎

Lines changed: 7 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
import threading
2323

2424
import pytest
25-
from helpers.child_processes import child_timeout_sec, kill_subprocesses
25+
from helpers.child_processes import child_timeout_sec, kill_subprocesses, noop, run_child, warmup_multiprocessing
2626
from helpers.contexts import assert_no_cuda_warning
2727

2828
from cuda.bindings import driver
@@ -149,20 +149,6 @@ class _CustomResource(VirtualMemoryResource):
149149
"""A subclass, to check that pickling preserves the type."""
150150

151151

152-
def _run_child(target, *args):
153-
"""Run ``target(*args)`` in a spawned child and fail if it does not exit cleanly."""
154-
process = mp.Process(target=target, args=args)
155-
process.start()
156-
process.join(timeout=CHILD_TIMEOUT_SEC)
157-
survivors = kill_subprocesses(process)
158-
assert not survivors, "child did not exit within timeout"
159-
assert process.exitcode == 0, f"child exited with {process.exitcode}"
160-
161-
162-
def _noop(*args):
163-
pass
164-
165-
166152
def _close_argument(buffer):
167153
buffer.close()
168154

@@ -260,7 +246,7 @@ def test_main(self, vmm_ipc_device):
260246
assert _open_fds() == fds_before + 3
261247
fds = [int(handle) for handle in desc._handles]
262248

263-
_run_child(self.child_main, device.device_id, desc, gran, queue)
249+
run_child(self.child_main, device.device_id, desc, gran, queue)
264250
result = queue.get(timeout=CHILD_TIMEOUT_SEC)
265251
assert result["chunks"] == [bytes([value]) * 16 for value in values]
266252
assert result["released"] is True
@@ -311,7 +297,7 @@ def test_main(self, vmm_ipc_device_x2):
311297

312298
peers_possible = dev1.can_access_peer(dev0)
313299
queue = mp.Queue()
314-
_run_child(self.child_main, dev0.device_id, dev1.device_id, desc, peers_possible, queue)
300+
run_child(self.child_main, dev0.device_id, dev1.device_id, desc, peers_possible, queue)
315301
result = queue.get(timeout=CHILD_TIMEOUT_SEC)
316302
assert f"device {dev0.device_id}" in result["wrong_device"]
317303
assert result["device_id"] == dev1.device_id
@@ -495,7 +481,7 @@ def test_export_and_pickling_raise_before_any_driver_call(self, vmm_ipc_device):
495481
with parent_conn, child_conn, pytest.raises(RuntimeError, match=_REEXPORT_MESSAGE):
496482
parent_conn.send(imported)
497483
with pytest.raises(RuntimeError, match=_REEXPORT_MESSAGE):
498-
mp.Process(target=_noop, args=(imported,)).start()
484+
mp.Process(target=noop, args=(imported,)).start()
499485
# The original descriptor still serves new importers, and the data is intact.
500486
_fill(buf, 0x3C)
501487
alias = Buffer.from_ipc_descriptor(mr, desc, stream=stream)
@@ -589,9 +575,7 @@ def test_exporter_holds_none_without_a_descriptor(self, vmm_ipc_device):
589575
@pytest.mark.thread_unsafe(reason="counts process-global file descriptors")
590576
def test_process_argument_releases_with_the_process_object(self, vmm_ipc_device):
591577
"""A buffer passed to a spawned Process imports there; its descriptor goes when the Process object does."""
592-
# The first spawn opens a process-wide resource-tracker pipe, which
593-
# outlives Process objects. Start it before taking the descriptor baseline.
594-
_run_child(_noop)
578+
warmup_multiprocessing()
595579
device = vmm_ipc_device
596580
mr = _resource(device)
597581
with mr.allocate(1) as buf:
@@ -926,7 +910,7 @@ def test_round_trip(self, vmm_ipc_device):
926910
assert alias.is_mapped
927911
alias.close()
928912
queue = mp.Queue()
929-
_run_child(self.child_main, device.device_id, desc, queue)
913+
run_child(self.child_main, device.device_id, desc, queue)
930914
assert queue.get(timeout=CHILD_TIMEOUT_SEC) == {"size": 0, "is_mapped": True}
931915
empty.close()
932916

@@ -948,7 +932,7 @@ def test_imported_buffer_feeds_a_consumer(self, vmm_ipc_device):
948932
buf = mr.allocate(1)
949933
_fill(buf, 0x7E)
950934
queue = mp.Queue()
951-
_run_child(self.child_main, device.device_id, buf, queue)
935+
run_child(self.child_main, device.device_id, buf, queue)
952936
result = queue.get(timeout=CHILD_TIMEOUT_SEC)
953937
assert result["dl_device"] == (int(DLDeviceType.kDLCUDA), device.device_id)
954938
assert result["shape"] == (buf.size,)

0 commit comments

Comments
 (0)