diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7bd81c17c52..2cc4bb91d26 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -85,16 +85,6 @@ repos: language: python files: '^cuda_core/tests/.*\.py$' - - id: check-build-shared-sync - name: Check _build_shared.py copies are identical - language: system - pass_filenames: false - files: '^(cuda_bindings|cuda_core)/_build_shared\.py$' - entry: bash -c - args: - - 'git diff --no-index --exit-code -- cuda_bindings/_build_shared.py cuda_core/_build_shared.py || { echo "ERROR: cuda_bindings/_build_shared.py and cuda_core/_build_shared.py must be byte-identical. Copy one onto the other and commit both." >&2; exit 1; }' - - "_" # fake script name, because bash considers $0 (the first argument) to be the script name - - id: no-markdown-in-docs-source name: Prevent markdown files in docs/source directories entry: bash -c diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 75c3056a59b..68e25ff0951 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,8 +21,8 @@ Thank you for your interest in contributing to CUDA Python! Based on the type of - [Table of Contents](#table-of-contents) - [Cloning the repository](#cloning-the-repository) - [Recommended clone](#recommended-clone) - - [Fixing an existing clone](#fixing-an-existing-clone) - [Updating your clone](#updating-your-clone) + - [Why the clone matters](#why-the-clone-matters) - [Symptoms of a bad clone](#symptoms-of-a-bad-clone) - [Development on Windows](#development-on-windows) - [Enabling git symlinks](#enabling-git-symlinks) @@ -45,11 +45,56 @@ Thank you for your interest in contributing to CUDA Python! Based on the type of ## Cloning the repository -> **Windows contributors (not WSL):** source builds and tests work with -> `core.symlinks=false`. Git symlink support is optional and only affects the -> remaining documentation and metadata links (`CLAUDE.md`, -> `cuda_python/README.md`, and `.git_archival.txt`). See -> [Enabling git symlinks](#enabling-git-symlinks) if you want those links materialized. +### Recommended clone + +**On native Windows (not WSL), first [enable symlink creation](#enabling-git-symlinks), +including the Developer Mode registry check.** Complete that setup before +running the clone command. + +From the directory where you want to create the checkout, run: + +```console +$ git clone --config core.symlinks=true --config submodule.recurse=true --recurse-submodules https://github.com/NVIDIA/cuda-python.git +$ cd cuda-python +``` + +This recipe sets up the repository for source builds and development: + +- `core.symlinks=true` materializes symlinks during checkout, including the + build-critical `cuda_core/_build_shared.py`. +- `--recurse-submodules` initializes the CCCL headers needed to build `cuda.core`. +- `submodule.recurse=true` keeps initialized submodules in sync on `git pull`, + `git checkout`, and `git switch`. +- The clone retains history and release tags for correct package versions. + Do not add `--depth` or `--no-tags`. + +Both configuration settings are stored in the new clone. Next, follow the +build and test instructions for +[`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/install.html), +[`cuda.bindings`](https://nvidia.github.io/cuda-python/cuda-bindings/latest/install.html), or +[`cuda.pathfinder`](https://nvidia.github.io/cuda-python/cuda-pathfinder/latest/install.html). + +### Updating your clone + +From the repository root, run: + +```console +$ git pull --recurse-submodules +$ git submodule update --init +``` + +The recommended clone configures `submodule.recurse=true`, so you can omit +`--recurse-submodules` from `git pull`. The second command initializes any +newly added submodules; it is usually a no-op. + +If you work from a fork, periodically fetch release tags from NVIDIA upstream +so package versions stay current: + +```console +$ git fetch --tags https://github.com/NVIDIA/cuda-python.git +``` + +### Why the clone matters Every package in this repository derives its version from git tags using [`setuptools-scm`](https://setuptools-scm.readthedocs.io/), so **how you clone @@ -79,59 +124,6 @@ working build therefore needs all of the following: 4. **The CCCL submodule** when building `cuda.core`. That package compiles against the headers pinned in `cuda_core/third_party/cccl`. -### Recommended clone - -Clone the repository with its submodules: - -```console -$ git clone --recurse-submodules https://github.com/NVIDIA/cuda-python.git -``` - -To keep initialized submodules in sync on `git pull`, `git checkout`, and -`git switch`, set this default once per clone: - -```console -$ git config --local submodule.recurse true -``` - -### Fixing an existing clone - -If you already have a shallow clone: - -```console -$ git fetch --unshallow --tags -``` - -If you cloned without submodules: - -```console -$ git submodule update --init cuda_core/third_party/cccl -``` - -If you are working from a personal fork, your fork's tags stop tracking upstream -the moment new releases are cut, which silently yields a stale version. Fetch -tags from upstream directly: - -```console -$ git remote add upstream https://github.com/NVIDIA/cuda-python.git -$ git fetch --tags upstream -``` - -Keep doing this periodically — a fork that was correct when you created it will -drift. - -### Updating your clone - -To update your clone and initialize any newly added submodules, run: - -```console -$ git pull --recurse-submodules && git submodule update --init -``` - -You can omit `--recurse-submodules` if you have already run -`git config --local submodule.recurse true`. The second command is usually a -no-op, but ensures that newly added submodules are initialized. - ### Symptoms of a bad clone Only case 3 below reports an error. The first two fail *silently*, producing a @@ -171,38 +163,34 @@ document. ### Enabling git symlinks -The repository still contains documentation and metadata symlinks, including +The repository contains documentation and metadata symlinks, including `CLAUDE.md`, `cuda_python/README.md`, and the per-package `.git_archival.txt` -files. With `core.symlinks=false`, Git checks them out as plain text files -containing the target path. This does not affect source builds or tests. - -If you want these links materialized as symlinks: - -1. **[Activate Developer Mode](https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development#activate-developer-mode)** - so Git can create symlinks without Administrator privileges. +files. It also currently tracks the build-critical +`cuda_core/_build_shared.py` as a symlink to the canonical +`cuda_bindings/_build_shared.py`. The core path may instead be maintained as a +byte-for-byte identical copy, but the current tree uses a symlink. With +`core.symlinks=false`, Git checks it out as a plain text file containing the +target path, and `cuda-core` source builds and tests fail. -2. **Enable Git symlink support globally** so newly-cloned repositories inherit - the setting: +[Activate Developer Mode](https://learn.microsoft.com/en-us/windows/advanced-settings/developer-mode) +so Git can create symlinks without Administrator privileges. Then verify +that Windows recorded Developer Mode in the machine-wide registry: - ```console - $ git config --global core.symlinks true - ``` - -Then clone as usual (see [Cloning the repository](#cloning-the-repository)). +```console +$ reg.exe query "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock" /v AllowDevelopmentWithoutDevLicense +``` -If you already cloned with `core.symlinks=false`, the repository-local -setting overrides the global setting. Enable it locally, then rematerialize -all tracked symlinks. For example: +The output must show `AllowDevelopmentWithoutDevLicense` as a `REG_DWORD` +with value `0x1`. If the value is missing or differs, Git may report +`unable to create symlink ...: Permission denied` even when Windows Settings +shows Developer Mode as enabled. In that case, run the following command +once from an Administrator shell: ```console -$ git config core.symlinks true -$ git rm --cached cuda_python/README.md -$ git checkout HEAD -- cuda_python/README.md +$ reg.exe add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock" /v AllowDevelopmentWithoutDevLicense /t REG_DWORD /d 1 /f ``` -Repeat the last two commands for every other tracked symlink. Re-cloning after -enabling Developer Mode and setting the global option is simpler if you want -all links restored. +Then return to [Recommended clone](#recommended-clone). ### Pre-commit lychee workaround diff --git a/cuda_bindings/docs/source/install.rst b/cuda_bindings/docs/source/install.rst index 2087002ad45..d712bbdf75f 100644 --- a/cuda_bindings/docs/source/install.rst +++ b/cuda_bindings/docs/source/install.rst @@ -115,6 +115,12 @@ published from the corresponding backport branch. Installing from Source ~~~~~~~~~~~~~~~~~~~~~~ +First clone the repository using `Recommended clone +`_. +On native Windows outside WSL, complete its Windows setup before cloning. +The recipe preserves release tags and configures symlinks and submodule +updates for development. + Requirements ^^^^^^^^^^^^ @@ -126,7 +132,12 @@ Requirements [^2]: The CUDA Runtime static library (``libcudart_static.a`` on Linux, ``cudart_static.lib`` on Windows) is part of the CUDA Toolkit. If using conda packages, it is contained in the ``cuda-cudart-static`` package. -[^3]: The version is derived from git tags via ``setuptools-scm``, so the clone must include tags reaching back to at least the latest ``v*`` tag. Clone with ``git clone https://github.com/NVIDIA/cuda-python.git``; do not use ``--depth`` or ``--no-tags``, since a shallow clone builds without error but produces a bogus version such as ``0.1.dev1+g0d22cb444``. See `Cloning the repository `_ for details and recovery steps. +[^3]: The version is derived from git tags via ``setuptools-scm``, so the clone +must include tags reaching back to at least the latest ``v*`` tag. Do not use +``--depth`` or ``--no-tags``: a clone without reachable tags can build with a +bogus version such as ``0.1.dev1+g0d22cb444``. See `Cloning the repository +`_ +for details. Source builds require that the provided CUDA headers are of the same major.minor version as the ``cuda.bindings`` you're trying to build. Despite this requirement, note that the minor version compatibility is still maintained. The build checks the header before it compiles anything. A mismatch stops the build with a message that names the ``cuda.h`` it found and the version this source tree needs. Use the ``CUDA_PATH`` (or ``CUDA_HOME``) environment variable to specify the location of your headers. If both are set, ``CUDA_PATH`` takes precedence. For example, if your headers are located in ``/usr/local/cuda/include``, then you should set ``CUDA_PATH`` with: @@ -143,10 +154,11 @@ See :doc:`Environment Variables ` for a description of ot Editable Install ^^^^^^^^^^^^^^^^ -You can use: +From the repository root, run: .. code-block:: console + $ cd cuda_bindings $ pip install -v -e . -to install the module as editable in your current Python environment (e.g. for testing of porting other libraries to use the binding). +This installs the module as editable in your current Python environment (e.g. for testing or porting other libraries to use the bindings). diff --git a/cuda_core/_build_shared.py b/cuda_core/_build_shared.py deleted file mode 100644 index 5a1231d1a75..00000000000 --- a/cuda_core/_build_shared.py +++ /dev/null @@ -1,485 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Build helpers shared by the cuda-bindings and cuda-core PEP 517 backends. - -``cuda_bindings/_build_shared.py`` is canonical. ``cuda_core/_build_shared.py`` -is either a symlink to it or a byte-for-byte identical copy. Both backends load -the helper through their package-local path, keeping ``Path(__file__)``-relative -state package-local. - -PEP 517 build isolation gives each backend its own ``build_hooks.py`` but not a -shared import path. Both packages declare ``backend-path = ["."]``, which puts -the package directory (and therefore this module) on ``sys.path`` while the -backend runs. - -Only what genuinely differs per package is parameterized, via the arguments of -``resolve_toolchain``: the C++ standard, warnings-as-errors, and an optional -``tweak`` hook for flags a single package needs. - -Besides the toolchain, this module owns the CUDA path lookup and the machinery that both backends use -around cythonize and build_ext: the opt-in Cython generated-source cache and -the build stamps that force a rebuild when the build configuration changed. - -Note: There is no support guarantee for environment variables like -CUDA_PYTHON_TOOLCHAIN. They may be removed or changed in the future. -""" - -import contextlib -import functools -import hashlib -import os -import shlex -import shutil -import sys -import sysconfig -import uuid -from pathlib import Path -from warnings import warn - -# ----------------------------------------------------------------------- -# CUDA path - - -def _import_get_cuda_path_or_home(): - """Import get_cuda_path_or_home, working around PEP 517 namespace shadowing. - - See https://github.com/NVIDIA/cuda-python/issues/1824 for why this helper is needed. - """ - try: - import cuda.pathfinder - except ModuleNotFoundError as exc: - if exc.name not in ("cuda", "cuda.pathfinder"): - raise - try: - import cuda - except ModuleNotFoundError: - cuda = None - - for p in sys.path: - sp_cuda = Path(p) / "cuda" - if (sp_cuda / "pathfinder").is_dir(): - cuda.__path__ = list(cuda.__path__) + [str(sp_cuda)] - break - else: - raise ModuleNotFoundError( - "cuda-pathfinder is not installed in the build environment. " - "Ensure 'cuda-pathfinder>=1.5' is in build-system.requires." - ) - import cuda.pathfinder - - pathfinder_dir = Path(cuda.pathfinder.__file__).parent - print( - f"Using cuda-pathfinder {cuda.pathfinder.__version__} from {pathfinder_dir}", - file=sys.stderr, - ) - return cuda.pathfinder.get_cuda_path_or_home - - -@functools.cache -def _get_cuda_path() -> str: - get_cuda_path_or_home = _import_get_cuda_path_or_home() - cuda_path = get_cuda_path_or_home() - if not cuda_path: - raise RuntimeError("Environment variable CUDA_PATH or CUDA_HOME is not set") - print("CUDA path:", cuda_path) - return cuda_path - - -# ----------------------------------------------------------------------- -# Toolchain selection - -_TOOLCHAINS_LINUX = ("gnu", "llvm") -_TOOLCHAINS_WINDOWS = ("msvc",) -_TOOLCHAIN_COMPILERS = { - "gnu": ("gcc", "g++"), - "llvm": ("clang", "clang++"), - "msvc": (None, None), -} - - -def _resolve_toolchain_name(): - """Read CUDA_PYTHON_TOOLCHAIN, validate it, return (name, allowed, cc, cxx). - - The default toolchain (gnu on Linux, msvc on Windows) is the first entry - of the platform's allowed tuple. cc/cxx are the compiler binaries for the - toolchain (None for msvc, which distutils discovers via the MSVC env). - """ - if sys.platform == "win32": - platform_key, allowed = "win32", _TOOLCHAINS_WINDOWS - else: - platform_key, allowed = "linux", _TOOLCHAINS_LINUX - name = os.environ.get("CUDA_PYTHON_TOOLCHAIN", allowed[0]).strip().lower() - if name not in allowed: - raise RuntimeError( - f"CUDA_PYTHON_TOOLCHAIN={name!r} is not supported on {platform_key}. Valid values: {', '.join(allowed)}." - ) - cc, cxx = _TOOLCHAIN_COMPILERS[name] - explicit = bool(os.environ.get("CUDA_PYTHON_TOOLCHAIN", "").strip()) - return name, allowed, cc, cxx, explicit - - -def _with_compiler(command, compiler): - """Replace the leading compiler on a linker command; keep flags. - - Conda ``LDCXXSHARED`` looks like ``g++ -pthread -B .../python_compiler_compat - -shared ...``. Only the executable changes so those flags stay on the - link line. The command is tokenized with shlex so quoted arguments - survive, and a leading ``env VAR=value`` prefix is preserved. CC/CXX are - not rewritten this way: they may already be a launcher plus compiler - (``sccache cc``). - """ - if not command or not command.strip(): - return compiler - parts = shlex.split(command) - # Keep an ``env VAR=value ...`` prefix: setuptools' C++ link step splits it - # off before it substitutes the compiler, so it still reaches the link line. - prefix_end = 0 - if parts and os.path.basename(parts[0]) == "env": - prefix_end = 1 - # Match setuptools' _split_env: any token with ``=`` is an env operand - # (covers both ``VAR=value`` and ``--unset=VAR`` long options). - while prefix_end < len(parts) and "=" in parts[prefix_end]: - prefix_end += 1 - # Everything else before the first flag is the old compiler (or a launcher - # for it; setuptools takes the launcher from CXX instead). - i = prefix_end - while i < len(parts) and not parts[i].startswith("-"): - i += 1 - return shlex.join([*parts[:prefix_end], compiler, *parts[i:]]) - - -def _with_sccache(current, compiler): - """Keep a leading sccache token when the toolchain picks a compiler. - - CI sets ``CC="sccache cc"`` or ``CC="/host/.../sccache cc"``. An explicit - toolchain then becomes ``CC="sccache clang"`` rather than a bare compiler. - """ - if current: - launcher = current.split()[0] - if os.path.basename(launcher) == "sccache": - return f"{launcher} {compiler}" - return compiler - - -def _apply_toolchain_env(cc, cxx, explicit): - """Set CC/CXX/LDCXXSHARED for an explicitly-chosen toolchain. - - The default path (CUDA_PYTHON_TOOLCHAIN unset) intentionally - does not touch the env, so an externally-set compiler (e.g. - CC="sccache cc" in CI) keeps working. An explicit CUDA_PYTHON_TOOLCHAIN - override (incl. =gnu) sets CC/CXX to the toolchain compiler; an existing - sccache prefix is kept (CC="sccache cc" + llvm -> CC="sccache clang"). - Extras on LDCXXSHARED (rpath, -pthread, -B, ...) are kept, taken from the - environment if set there and from sysconfig otherwise; only the compiler - is swapped. LDSHARED is left unset so distutils rewrites it from CC. - """ - if explicit and cc is not None: - os.environ["CC"] = _with_sccache(os.environ.get("CC", ""), cc) - os.environ["CXX"] = _with_sccache(os.environ.get("CXX", ""), cxx) - # An LDCXXSHARED the user already exported takes precedence over - # sysconfig's, as CC/CXX do; either way only the compiler is swapped. - ldcxxshared = ( - os.environ.get("LDCXXSHARED") - or sysconfig.get_config_var("LDCXXSHARED") - or sysconfig.get_config_var("LDSHARED") - ) - os.environ["LDCXXSHARED"] = _with_compiler(ldcxxshared, cxx) if ldcxxshared else f"{cxx} -shared" - - -def _check_toolchain_available(name): - """Preflight: verify the selected toolchain's tools are on PATH. - - No-op for the platform default (distutils discovers those). For llvm, - probes clang, clang++, and ld.lld so a missing toolchain fails fast with a - helpful message instead of a cryptic compile error. - """ - if name != "llvm": - return - tools = ("clang", "clang++", "ld.lld") - missing = [t for t in tools if shutil.which(t) is None] - if missing: - raise RuntimeError( - f"CUDA_PYTHON_TOOLCHAIN=llvm but required tool(s) not found on PATH: " - f"{', '.join(missing)}. Install clang and lld " - f"(e.g. `apt install clang lld` or `dnf install clang lld`) " - f"or set CUDA_PYTHON_TOOLCHAIN=gnu." - ) - - -# ----------------------------------------------------------------------- -# Compiler and linker flags - - -def _build_flags(name, cxx_std, debug, coverage, warnings_as_errors): - """Return (extra_compile_args, extra_link_args) for toolchain ``name``. - - The one flag set used by both packages: - - - Linux compile: ``-std=c++{cxx_std}``, then ``-g0 -O2`` (opt) or - ``-g -O0 -D _GLIBCXX_ASSERTIONS`` (debug). - - Linux link: ``-fuse-ld=lld`` (llvm) and ``-Wl,--strip-all`` (opt). - - MSVC compile: ``/std:c++{cxx_std}`` and ``/O2``. Modern setuptools no - longer forces ``/Ox``, so the optimization level is set explicitly for - symmetry with Linux ``-O2``. Debug builds are not supported on Windows. - - Coverage: Cython tracing defines. - - Warnings-as-errors: ``-Werror`` (Linux) or ``/WX`` (MSVC). - """ - extra_compile_args = [] - extra_link_args = [] - - if name == "msvc": - if debug: - raise RuntimeError("Debuggable builds are not supported on Windows.") - extra_compile_args += [f"/std:c++{cxx_std}", "/O2"] - else: - # Common Linux compile flags. - extra_compile_args += [f"-std=c++{cxx_std}"] - # Compiler-specific flags. - if name == "llvm": - extra_link_args += ["-fuse-ld=lld"] - # Common Linux debug/opt flags. - if debug: - extra_compile_args += ["-g", "-O0", "-D _GLIBCXX_ASSERTIONS"] - else: - extra_compile_args += ["-g0", "-O2"] - extra_link_args += ["-Wl,--strip-all"] - - if coverage: - # CYTHON_TRACE_NOGIL indicates to trace nogil functions. It is not - # related to free-threading builds. - extra_compile_args += ["-DCYTHON_TRACE_NOGIL=1", "-DCYTHON_USE_SYS_MONITORING=0"] - - if warnings_as_errors: - # The MSVC exemptions cover warnings that Cython's utility code - # produces in every module and the .pyx sources cannot fix: - # - C4551 ("function call missing argument list"), hundreds per - # module. - # - C4244 (narrowing): the overflow-check helpers that - # @cython.overflowcheck(True) instantiates for _layout.pxd narrow - # int64 to int inside Cython's own code. - # gcc and clang need no exemption. The one generated warning they - # report in cuda.core, the unused @overload wrappers of - # Graph.__getitem__, is silenced by a pragma in - # cuda/core/graph/_graph_builder.pyx. - if name == "msvc": - extra_compile_args += ["/WX", "/wd4551", "/wd4244"] - else: - extra_compile_args += ["-Werror"] - - return extra_compile_args, extra_link_args - - -def resolve_toolchain(*, cxx_std, debug=False, compile_for_coverage=False, warnings_as_errors=False, tweak=None): - """Resolve the C/C++ toolchain from CUDA_PYTHON_TOOLCHAIN. - - Returns (name, cc, cxx, extra_compile_args, extra_link_args). The default - toolchain (gnu on Linux, msvc on Windows) reproduces the previous build - behavior and does not touch CC/CXX/LDCXXSHARED, so an externally-set compiler - (e.g. CC="sccache cc") keeps working. A non-default toolchain (llvm on - Linux) selects clang/clang++ and lld and sets CC/CXX/LDCXXSHARED so distutils' - customize_compiler picks them up. - - The per-package choices are arguments: - - - ``cxx_std`` (required): the C++ standard, e.g. ``14``. There is - deliberately no shared default; each package picks its own. - - ``warnings_as_errors``: opt in to ``-Werror`` / ``/WX``. - - ``tweak``: optional callable ``tweak(name, extra_compile_args, - extra_link_args)`` returning the adjusted ``(extra_compile_args, - extra_link_args)`` pair, for the few flags a single package needs that - do not belong in the shared set. - """ - name, _allowed, cc, cxx, explicit = _resolve_toolchain_name() - - extra_compile_args, extra_link_args = _build_flags(name, cxx_std, debug, compile_for_coverage, warnings_as_errors) - if tweak is not None: - extra_compile_args, extra_link_args = tweak(name, extra_compile_args, extra_link_args) - - _apply_toolchain_env(cc, cxx, explicit) - - return name, cc, cxx, extra_compile_args, extra_link_args - - -# ----------------------------------------------------------------------- -# Cython cache helpers - - -# === Cython generated-source cache (opt-in via CUDA_PYTHON_CYTHON_CACHE_DIR) === -# Workaround for Cython issue #7532: Cython's native cache fingerprint omits -# `compiler_directives`, so builds with different directives (e.g. linetrace -# for coverage) could reuse stale generated C/C++ output. This helper -# namespaces the Cython cache by package and a digest of output-affecting -# build configuration so distinct configurations get distinct caches. -# -# Removal: once cython/cython#7532 is resolved in a released Cython version -# and cuda-python's minimum Cython version includes the fix, this helper -# and its workaround-specific tests can be deleted; cythonize() can then be -# called with `cache=` (or `cache=True`) without per-config namespacing. -# See https://github.com/cython/cython/issues/7532 -def _cython_cache_path( - package, - *, - compiler_directives=None, - compile_time_env=None, - language_level=None, - cplus=None, - debug=False, - cuda_major=None, -): - """Return a per-configuration Cython cache directory, or None to disable caching. - - Returns None when CUDA_PYTHON_CYTHON_CACHE_DIR is unset, so cythonize() - is called without ``cache=`` and existing workflows are unchanged. - """ - cache_root = os.environ.get("CUDA_PYTHON_CYTHON_CACHE_DIR") - if not cache_root: - return None - if sys.platform == "win32": - warn( - "CUDA_PYTHON_CYTHON_CACHE_DIR is set but Cython caching via symlinks " - "is not supported on Windows; caching will be disabled.", - stacklevel=2, - ) - return None - - h = hashlib.sha256() - h.update(package.encode("utf-8")) - # The Python version running cythonize affects generated C code - # (e.g. CYTHON_COMPRESS_STRINGS: zstd on 3.14, zlib on 3.12/3.13). - h.update(f"python={sys.version_info.major}.{sys.version_info.minor}".encode()) - - def _update(name, value): - h.update(name.encode("utf-8")) - h.update(repr(value).encode("utf-8")) - - # compiler_directives are not in Cython's native fingerprint (#7532). - if compiler_directives: - for key in sorted(compiler_directives): - _update(f"directive:{key}", compiler_directives[key]) - # compile_time_env, language_level, and cplus are already in Cython's - # fingerprint, but we include them so the namespace stays correct even - # if Cython's fingerprint logic changes. - if compile_time_env: - for key in sorted(compile_time_env): - _update(f"compile_time_env:{key}", compile_time_env[key]) - if language_level is not None: - _update("language_level", language_level) - if cplus is not None: - _update("cplus", cplus) - # debug toggles gdb_debug in cythonize(), which affects generated code. - _update("debug", debug) - if cuda_major is not None: - _update("cuda_major", cuda_major) - - return os.path.join(cache_root, f"{package}-{h.hexdigest()[:16]}") - - -@contextlib.contextmanager -def _stable_cython_alias(target: Path, alias: Path): - """Atomically create a stable directory symlink alias for a Cython include tree. - - Cython's cache fingerprint includes the absolute path of each resolved - .pxd dependency (via ``file_hash()``). PEP 517 build environments install - dependencies under randomized temporary prefixes, making those paths - unstable across runs. This context manager creates a fixed, worktree- - relative symlink so Cython sees a stable lexical path. - - The symlink is created in the package directory containing this file, not - in the cwd, to keep aliases package-local and avoid cross-package races. - - alias must not already exist as a real file or directory; if it is a - symlink (including a dangling one) it is atomically replaced. - - On exit the alias is removed only if it still points at ``target`` (a - racing replacement will not be deleted). - - POSIX only: directory symlinks require no elevated privileges on Linux. - """ - # Resolve the *parent* directory (must exist), then append the name. - # We deliberately do not follow a symlink that may already sit at alias. - if not alias.is_absolute(): - alias = Path(__file__).parent / alias - alias = alias.parent.resolve() / alias.name - target = target.resolve() - - if alias.exists() and not alias.is_symlink(): - raise RuntimeError( - f"Cannot create Cython include alias at {alias}: a real file or directory already exists there." - ) - - tmp_alias = alias.with_name(f".{alias.name}.{uuid.uuid4().hex[:8]}.tmp") - try: - os.symlink(target, tmp_alias, target_is_directory=True) - try: - os.replace(tmp_alias, alias) - except BaseException: - tmp_alias.unlink(missing_ok=True) - raise - rel = os.path.relpath(alias, start=Path.cwd()) - yield rel - finally: - tmp_alias.unlink(missing_ok=True) - # Only remove the alias we created; leave it alone if something else - # has already replaced it (readlink will differ). - try: - if alias.is_symlink() and Path(os.readlink(alias)).resolve() == target: - alias.unlink() - except OSError: - pass - - -# ----------------------------------------------------------------------- -# Build stamps -# -# Setuptools' freshness check covers neither the extension flags nor the -# build configuration, so a stale .so built under another configuration looks -# perfectly fresh. Each backend stamps the key of its last completed build -# (the toolchain for cuda-bindings; CUDA major, toolchain and debug/coverage -# for cuda-core). check_build_key() sets ``force_build_ext`` when the key -# changed, and setup.py hands that to build_ext. - -# Where per-configuration build artifacts live. Anchored to this file rather -# than the cwd, since a project can be built from anywhere. -_BUILD_DIR = Path(__file__).parent / "build" - -# Set by check_build_key(). Read it as ``build_hooks.force_build_ext``: both -# build_hooks modules re-export it. -force_build_ext = False - - -def _abi_stamp_path(stem): - """Return a stamp path scoped to this interpreter's extension ABI.""" - extension_suffix = sysconfig.get_config_var("EXT_SUFFIX") - if not extension_suffix: - raise RuntimeError("Python's EXT_SUFFIX build configuration is unavailable") - return _BUILD_DIR / f"{stem}{extension_suffix}" - - -def check_build_key(stamp, key, description): - """Set force_build_ext when ``key`` differs from the one stamped at ``stamp``. - - ``description`` names the key in the message that explains the rebuild. - """ - global force_build_ext - - try: - previous = stamp.read_text(encoding="utf-8").strip() - except FileNotFoundError: - previous = None - - # A missing stamp means the last build's key is unknown, so force too. - # On a first build that costs nothing: there are no artifacts to reuse. - if previous != key: - print(f"{description} of last build: {previous} (building {key}); forcing a full rebuild") - force_build_ext = True - - -def record_build_key(stamp, key): - """Stamp ``key`` at ``stamp``, once the build it describes has completed. - - A build that failed partway through must not claim outputs it never - produced, so callers record only after the build succeeded. - """ - stamp.parent.mkdir(parents=True, exist_ok=True) - stamp.write_text(key + "\n", encoding="utf-8") diff --git a/cuda_core/_build_shared.py b/cuda_core/_build_shared.py new file mode 120000 index 00000000000..31696363b94 --- /dev/null +++ b/cuda_core/_build_shared.py @@ -0,0 +1 @@ +../cuda_bindings/_build_shared.py \ No newline at end of file diff --git a/cuda_core/docs/source/install.rst b/cuda_core/docs/source/install.rst index 525d910d5b0..0b4374392b9 100644 --- a/cuda_core/docs/source/install.rst +++ b/cuda_core/docs/source/install.rst @@ -78,8 +78,8 @@ The conda-forge package depends on ``cuda-bindings`` of the same CUDA major. The Development environment ----------------------- -The sections above cover end-user installation. The section below focuses on -a repeatable *development* workflow (editable installs and running tests). +The sections below cover nightly builds, source installation, and repeatable +development workflows with editable installs and tests. Installing the latest nightly (top-of-tree builds) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -106,15 +106,62 @@ Replace ``python312`` with your Python version (e.g. ``python310``, ``python311` with ``linux-aarch64``; for Windows, use ``win-64``. Replace ``cu13`` with ``cu12`` for CUDA 12.x environments. +.. _cuda-core-source-install: + +Installing from Source +~~~~~~~~~~~~~~~~~~~~~~ + +A source build has two requirements. See :ref:`cuda-core-bindings-floor`. + +- ``cuda-bindings`` 12.x or 13.x at or above the release's floor for that major. An isolated + build, the default for ``pip install``, installs a ``cuda-bindings`` no newer than the + toolkit's minor. Other builds must provide it. +- A CUDA Toolkit, located through ``CUDA_PATH`` or ``CUDA_HOME``, whose ``cuda.h`` has the same + major.minor as the header that ``cuda-bindings`` was generated from. If the versions differ, + the build fails early. To build against a specific ``cuda-bindings`` in an isolated build, + constrain it with ``PIP_CONSTRAINT``. A build with ``--no-build-isolation`` uses the installed + ``cuda-bindings``. + +Use the repository's `Recommended clone +`_ +recipe below. On native Windows outside WSL, complete its Windows setup +*before* cloning so Git can create the build-critical +``cuda_core/_build_shared.py`` symlink. + +.. code-block:: console + + $ git clone --config core.symlinks=true --config submodule.recurse=true --recurse-submodules https://github.com/NVIDIA/cuda-python.git + $ cd cuda-python/cuda_core + $ pip install . + +.. note:: + + ``--recurse-submodules`` is required: the build compiles against the CCCL + headers pinned in the ``cuda_core/third_party/cccl`` submodule, and fails + without them. For a clone that already exists, run the following from the + repository root: + ``git submodule update --init cuda_core/third_party/cccl``. + +.. note:: + + The version is derived from git tags via ``setuptools-scm``, so the clone + must include tags reaching back to at least the latest ``cuda-core-v*`` tag. + Do not use ``--depth`` or ``--no-tags``: a shallow clone builds without + error but produces a bogus version such as ``0.1.dev1+g0d22cb444``. See + `Cloning the repository + `_ + for details. + Development with uv ~~~~~~~~~~~~~~~~~~~ -`uv`_ is a fast Python package and project manager. For example, to work on -``cuda-core`` against CUDA 13: +`uv`_ is a fast Python package and project manager. Before cloning, check the +:ref:`source-build prerequisites `. +For example, to work on ``cuda-core`` against CUDA 13: .. code-block:: console - $ git clone --recurse-submodules https://github.com/NVIDIA/cuda-python.git + $ git clone --config core.symlinks=true --config submodule.recurse=true --recurse-submodules https://github.com/NVIDIA/cuda-python.git $ cd cuda-python/cuda_core $ uv venv $ source .venv/bin/activate # On Windows: .venv\Scripts\activate @@ -132,11 +179,12 @@ Development with pixi ~~~~~~~~~~~~~~~~~~~~~ `pixi`_ provides a reproducible development environment across the repository. -From the repository root: +Before cloning, check the :ref:`source-build prerequisites `. +For example, to work against CUDA 13: .. code-block:: console - $ git clone --recurse-submodules https://github.com/NVIDIA/cuda-python.git + $ git clone --config core.symlinks=true --config submodule.recurse=true --recurse-submodules https://github.com/NVIDIA/cuda-python.git $ cd cuda-python $ pixi run -e cu13 test-core @@ -149,40 +197,3 @@ To run all repository tests (pathfinder → bindings → core): Use ``-e cu12`` to test against CUDA 12 instead. .. _pixi: https://pixi.sh/ - -Installing from Source -~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - $ git clone --recurse-submodules https://github.com/NVIDIA/cuda-python.git - $ cd cuda-python/cuda_core - $ pip install . - -A source build has two requirements. See :ref:`cuda-core-bindings-floor`. - -- ``cuda-bindings`` 12.x or 13.x at or above the release's floor for that major. An isolated - build, the default for ``pip install``, installs a ``cuda-bindings`` no newer than the - toolkit's minor. Other builds must provide it. -- A CUDA Toolkit, located through ``CUDA_PATH`` or ``CUDA_HOME``, whose ``cuda.h`` has the same - major.minor as the header that ``cuda-bindings`` was generated from. If the versions differ, - the build fails early. To build against a specific ``cuda-bindings`` in an isolated build, - constrain it with ``PIP_CONSTRAINT``. A build with ``--no-build-isolation`` uses the installed - ``cuda-bindings``. - -.. note:: - - ``--recurse-submodules`` is required: the build compiles against the CCCL - headers pinned in the ``cuda_core/third_party/cccl`` submodule, and fails - without them. For a clone that already exists, run - ``git submodule update --init cuda_core/third_party/cccl``. - -.. note:: - - The version is derived from git tags via ``setuptools-scm``, so the clone - must include tags reaching back to at least the latest ``cuda-core-v*`` tag. - Do not use ``--depth`` or ``--no-tags``: a shallow clone builds without - error but produces a bogus version such as ``0.1.dev1+g0d22cb444``. See - `Cloning the repository - `_ - for details and recovery steps. diff --git a/cuda_pathfinder/docs/source/install.rst b/cuda_pathfinder/docs/source/install.rst index aa8dcd942d7..6c2acbdd69a 100644 --- a/cuda_pathfinder/docs/source/install.rst +++ b/cuda_pathfinder/docs/source/install.rst @@ -57,9 +57,14 @@ the following commands: Installing from Source ~~~~~~~~~~~~~~~~~~~~~~ +Use the repository's `Recommended clone +`_ +recipe below. On native Windows outside WSL, complete its Windows setup steps +before cloning. + .. code-block:: console - $ git clone https://github.com/NVIDIA/cuda-python.git + $ git clone --config core.symlinks=true --config submodule.recurse=true --recurse-submodules https://github.com/NVIDIA/cuda-python.git $ cd cuda-python/cuda_pathfinder $ pip install . @@ -77,4 +82,4 @@ For an editable install (e.g. when developing ``cuda.pathfinder`` itself): error but produces a bogus version such as ``0.1.dev1+g0d22cb444``. See `Cloning the repository `_ - for details and recovery steps. + for details.