Skip to content

BLD: Insert CCCL also via FLAGS environment to work around conda nvcc - #3104

Open
seberg wants to merge 1 commit into
NVIDIA:mainfrom
seberg:issue-3098
Open

seberg wants to merge 1 commit into
NVIDIA:mainfrom
seberg:issue-3098

Conversation

@seberg

@seberg seberg commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Conda's cuda-nvcc inserts it's own CCCL in a way that setuptools ends up placing them before our includes.
However, using the system CCCL means we find a version that isn't (currently) compatible.

That means that in such a setup, the old "TMA doesn't link" bug resurfaces.

This is a somewhat weird work-around, but it was the best idea, I could think of...

Closes gh-3098

Conda's `cuda-nvcc` inserts it's own CCCL in a way that setuptools ends
up placing them before our includes.
However, using the system CCCL means we find a version that isn't (currently)
compatible.

That means that in such a setup, the old "TMA doesn't link" bug resurfaces.

This is a somewhat weird work-around, but it was the best idea, I could think of...
@seberg seberg self-assigned this Oct 14, 2026
@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Oct 14, 2026
@seberg seberg added the bug Something isn't working label Oct 14, 2026
@seberg seberg added this to the cuda.core 1.3.0 milestone Oct 14, 2026
@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • Build Improvements
    • Updated CUDA extension builds to prioritize the pinned CCCL headers and preserve existing compiler flags where available, improving consistency across supported build environments.

The build setup now prepends CCCL include flags to compiler environment variables on non-Windows platforms. It places CCCL headers before CUDA Toolkit headers for extensions and appends -cccl-first to build configuration keys.

Changes: CCCL include flag ordering

Layer / File(s) Summary
Configure CCCL compiler flags
cuda_core/build_hooks.py, cuda_core/tests/test_build_hooks.py
The build setup prepends CCCL include flags to CFLAGS, CXXFLAGS, and CPPFLAGS on non-Windows platforms. Extension include directories place CCCL before CUDA Toolkit headers. Build configuration key and generated build-directory expectations now include -cccl-first; tests isolate the compiler flag environment variables.

Suggested reviewers: juenglin

Priority: ⬇️ Low

Change: Bug fix

Merge Risk: 🔵 Low · up to 3aefb

The CCCL flag-ordering workaround lacks a focused regression test despite the repository’s behavior-change testing requirement. This is a bounded coverage gap; add the test before relying on the workaround.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 078222e2-d909-4ee6-9f89-e4d2826b80ca
📥 Commits

Reviewing files that changed from the base of the PR and between 3f58a7b and 3aefb73.

📒 Files selected for processing: 2
  • cuda_core/build_hooks.py
  • cuda_core/tests/test_build_hooks.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment thread cuda_core/build_hooks.py
Comment on lines +457 to +462
cccl_flags = shlex.join([f"-I{path}" for path in include_dirs])
default_flags = sysconfig.get_config_var("CFLAGS") or ""
for name in ("CFLAGS", "CXXFLAGS", "CPPFLAGS"):
flags = os.environ.get(name, "" if name == "CPPFLAGS" else default_flags)
if flags != cccl_flags and not flags.startswith(cccl_flags + " "):
os.environ[name] = f"{cccl_flags} {flags}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

git diff --no-ext-diff --unified=80 3f58a7bee17cef175a614a591b97733c2c0f569d 3aefb73e4ef474692b984a9d8857438ed0b2fc08 -- cuda_core/build_hooks.py cuda_core/tests/test_build_hooks.py
printf '\n--- helper and callers ---\n'
rg -n -C 12 '_configure_cccl_include_flags|_build_cuda_core|CFLAGS|CXXFLAGS|CPPFLAGS' cuda_core/build_hooks.py cuda_core/tests/test_build_hooks.py
printf '\n--- repository guidance ---\n'
find .. -name AGENTS.md -o -name CONTRIBUTING.md -o -name '*guideline*' -o -name '*style*' | head -80

Repository: NVIDIA/cuda-python

Length of output: 41763


🏁 Script executed:

set -e
printf '%s\n' '--- applicable guidance ---'
for f in AGENTS.md cuda_core/AGENTS.md cuda_core/tests/AGENTS.md CONTRIBUTING.md; do
  if [ -f "$f" ]; then
    printf '\n### %s\n' "$f"
    nl -ba "$f"
  fi
done

printf '%s\n' '--- head helper and callers ---'
rg -n -F -C 20 -- '_configure_cccl_include_flags' cuda_core/build_hooks.py cuda_core/tests/test_build_hooks.py

printf '%s\n' '--- all test references to helper/caller and environment flags ---'
rg -n -F -C 8 -e '_configure_cccl_include_flags' -e '_build_cuda_core(' -e 'CFLAGS' -e 'CXXFLAGS' -e 'CPPFLAGS' cuda_core/tests cuda_core | sed -n '1,1400p'

printf '%s\n' '--- test declarations and relevant blocks ---'
python3 - <<'PY'
from pathlib import Path
p = Path('cuda_core/tests/test_build_hooks.py')
for i, line in enumerate(p.read_text().splitlines(), 1):
    if any(x in line for x in ('def test_', 'class Test', '_capture_cythonize_kwargs', '_build_cuda_core()', '_configure_cccl_include_flags')):
        print(f'{i}:{line}')
PY

printf '%s\n' '--- changed test diff ---'
git diff --no-ext-diff --unified=30 3f58a7bee17cef175a614a591b97733c2c0f569d 3aefb73e4ef474692b984a9d8857438ed0b2fc08 -- cuda_core/tests/test_build_hooks.py

printf '%s\n' '--- exact head source ranges around helper and every _build_cuda_core test call ---'
nl -ba cuda_core/build_hooks.py | sed -n '440,510p'
nl -ba cuda_core/tests/test_build_hooks.py | sed -n '380,430p;840,910p;980,1060p;1060,1190p'

Repository: NVIDIA/cuda-python

Length of output: 41996


🏁 Script executed:

set -e
printf '%s\n' '--- scoped guidance ---'
for f in cuda_core/AGENTS.md cuda_core/tests/AGENTS.md; do
  printf '\n### %s\n' "$f"
  nl -ba "$f"
done

printf '%s\n' '--- helper call and test assertions ---'
rg -n -F -C 6 -e '_configure_cccl_include_flags' -e 'os.environ' -e 'CFLAGS' -e 'CXXFLAGS' -e 'CPPFLAGS' cuda_core/tests/test_build_hooks.py
printf '%s\n' '--- indirect caller test blocks ---'
nl -ba cuda_core/tests/test_build_hooks.py | sed -n '390,425p;870,905p;995,1045p'
printf '%s\n' '--- caller source ---'
nl -ba cuda_core/build_hooks.py | sed -n '451,520p'

Repository: NVIDIA/cuda-python

Length of output: 41756


suggestion: Add direct tests for CCCL flag ordering and repeated invocation.

_build_cuda_core() indirectly calls _configure_cccl_include_flags(), but existing tests do not assert the resulting environment flags. Add a focused test that supplies a competing -I flag, checks that the pinned CCCL path comes first, and calls the helper twice to check idempotence.

@github-actions

Copy link
Copy Markdown
Contributor

@seberg

seberg commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Restarted unrelated test failure test_concurrent_eviction_does_not_delete_replaced_file seems racy (probably without free-threaded as well, but free-threaded might make it more likely).

Details
==================================== ERRORS ====================================
___ ERROR at call of test_concurrent_eviction_does_not_delete_replaced_file ____

tmp_path = PosixPath('/tmp/pytest-of-root/pytest-1/test_concurrent_eviction_does_0/thread_0_iter_0')

    def test_concurrent_eviction_does_not_delete_replaced_file(tmp_path):
        """Eviction is stat-guarded: while one process is evicting an entry to
        bring the cache under its size cap, another process may have already
        ``os.replace``-d a fresh value into the same path. The evictor must
        refuse to unlink in that case, otherwise the racing rewriter's
        just-committed entry vanishes.
    
        The rewriter does NO post-race uncontested write -- the entry's
        survival has to come from the in-race rewrites being preserved by
        the stat-guarded eviction path."""
        from cuda.core.utils import FileStreamProgramCache
    
        root = str(tmp_path / "fc")
        payload = b"X" * 2000
        cap = 5000  # fits 2 raw 2000B entries; the third write triggers eviction
    
        ctx = _mp.get_context("spawn")
        done_event = ctx.Event()
        rewriter = ctx.Process(
            target=_worker_size_cap_rewriter,
            args=(root, b"survivor", payload, cap, done_event),
        )
        # Churning writer creates new keys faster than eviction can drain them,
        # forcing _enforce_size_cap to consider 'survivor' for eviction many times.
        churner = ctx.Process(
            target=_worker_size_cap_writer,
            args=(root, b"churn-", payload, 80, cap),
        )
        rewriter.start()
        churner.start()
        churner.join(timeout=60)
        done_event.set()
        rewriter.join(timeout=60)
        assert rewriter.exitcode == 0
        assert churner.exitcode == 0
    
        # If eviction blindly unlinked replaced files, the rewriter's last
        # in-race write would be gone -- there is no post-race write to mask
        # the failure. The value must also start with ``payload`` (not a
        # ``churn-`` value) because each rewriter iteration writes
        # ``payload + str(i).encode()``.
        with FileStreamProgramCache(root, max_size_bytes=cap) as cache:
            got = cache.get(b"survivor")
>           assert got is not None, "rewriter's entry was evicted by racing churner"
E           AssertionError: rewriter's entry was evicted by racing churner
E           assert None is not None

FileStreamProgramCache = <class 'cuda.core.utils._program_cache._file_stream.FileStreamProgramCache'>
cache      = <cuda.core.utils._program_cache._file_stream.FileStreamProgramCache object at 0x5b55c2c0410>
cap        = 5000
churner    = <SpawnProcess name='SpawnProcess-174' pid=22526 parent=11379 stopped exitcode=0>
ctx        = <multiprocessing.context.SpawnContext object at 0x5b51eb92410>
done_event = <Event at 0x5b55c260150 set>
got        = None
payload    = b'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
rewriter   = <SpawnProcess name='SpawnProcess-173' pid=22522 parent=11379 stopped exitcode=0>
root       = '/tmp/pytest-of-root/pytest-1/test_concurrent_eviction_does_0/thread_0_iter_0/fc'
tmp_path   = PosixPath('/tmp/pytest-of-root/pytest-1/test_concurrent_eviction_does_0/thread_0_iter_0')

tests/test_program_cache_multiprocess.py:191: AssertionError

@seberg seberg added the P1 Medium priority - Should do label Oct 14, 2026
@leofang

leofang commented Oct 15, 2026

Copy link
Copy Markdown
Member

I am still confused. We don’t need nvcc at build time. We only need its headers, and it’s how the cuda-cccl_<arch> package is brought in as a transitive dependency:
https://github.com/conda-forge/cuda-core-feedstock/actions/runs/36618553740/job/109577751781#step:4:881
but that package lives in the targets directory
https://github.com/conda-forge/cuda-cccl-feedstock/blob/31aa3cc377c7befaf9e0b846c884a619a3565fda/recipe/meta.yaml#L30
which is not added to the C[XX,PP]FLAGS (note -isystem $PREFIX/include there):
https://github.com/conda-forge/cuda-core-feedstock/actions/runs/36618553740/job/109577751781#step:4:991
so regardless of gcc or nvcc, there should not have a chance that we bring in the system CCCL headers. How to reproduce this locally?

@seberg

seberg commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

The issue is cuda-nvcc, which does not use -Isystem but rather just -I: https://github.com/conda-forge/cuda-nvcc-feedstock/blob/main/recipe/activate.sh

But there is a point, maybe we should just close this. It seems rather a cuda-nvcc packaging bug to not use -Isystem there.

Note 100% sure, let's see where this goes: conda-forge/cuda-nvcc-feedstock#110

@leofang

leofang commented Oct 15, 2026

Copy link
Copy Markdown
Member

Hmmm OK you're referring to -I$PREFIX/targets/<arch>/include:
https://github.com/conda-forge/cuda-core-feedstock/actions/runs/36618553740/job/109577751781#step:4:1430

Does this affect CuPy too? 🤔
https://github.com/conda-forge/cupy-feedstock/actions/runs/36045861869/job/107789122796#step:4:4309
Do we lose the ability of using bundled CCCL?

@leofang

leofang commented Oct 16, 2026

Copy link
Copy Markdown
Member

Looks like CuPy is affected too... Asking in the internal conda channel.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P1 Medium priority - Should do

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cuda.core: import fails with undefined symbol cudaGetErrorString when _tensor_map is built with -z now (after #2940)

2 participants