CORE: Get local rank from topology if unset

[MaayanGadishNvidia]

What

Adding automation for calculate local rank from topo if not provided by user

Why ?

Continue of #1189

[greptile-apps]

Greptile Summary

The PR adds opt-in automatic discovery of the node-local rank from an OOB topology exchange when NODE_LOCAL_ID is automatic.

• Adds context configuration for automatic node-local ID discovery.
• Exchanges context and host prefixes to construct a temporary topology.
• Derives a zero-based node-local rank and forwards it during transport-context creation.
• Adds validation and cleanup for the new address-storage layout.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/core/ucc_context.c	Implements the prefix exchange, topology-based rank discovery, full address exchange, validation, and cleanup without a remaining blocking failure.
src/components/topo/ucc_topo.h	Adds a guarded helper that returns the enabled node subgroup’s zero-based local rank.
src/core/ucc_context.h	Defines the new configuration field, address-storage layout flag, and exchange helper declarations consistently with the implementation.

_{Reviews (67): Last reviewed commit: "CORE: Get local rank from topology if un..." | Re-trigger Greptile}

[greptile-apps]

_{5 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{7 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Trailing whitespace added.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[greptile-apps]

_{6 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

src/core/ucc_context.c
Added components/topo/ucc_topo.h include to support automatic local rank computation from topology

[greptile-apps]

_{4 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{4 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[janjust]

/build

[janjust]

/build

[janjust]

@MaayanGadishNvidia let's rework so that init tim doesn't increase, please run osu_mpi_init tests to make sure we don't impact init

[MaayanGadishNvidia]

@MaayanGadishNvidia let's rework so that init tim doesn't increase, please run osu_mpi_init tests to make sure we don't impact init

As discussed, I ran MPI init tests and got high variance. The attached figure summarizes 50 iterations of the MPI init test for four configurations: the master branch; my PR with the computation of the local rank from topo disabled/enabled (when not provided by the user); and my PR when the local rank is provided by the user.
As can be seen, the results are quite noisy, and all UCC versions show similar behavior within the noise margins.

[janjust]

/build

[janjust]

@MaayanGadishNvidia fix linter please

[svcnbu-swx-hpcx]

🤖 CI Triage Agent — Lint (codestyle) · commit 9c5ec84e

TL;DR: The codestyle lint failed because the commit title UCC/CORE: Added automation to loading local rank from topo is 58 characters, exceeding the 50-character limit enforced by the commit-title check. Shorten the commit subject to ≤50 characters.

Full analysis

Summary: The "Lint (codestyle)" GitHub Actions job (codestyle job in .github/workflows/codestyle.yaml) failed its commit-title validation step with exit code 1.

Root cause: The commit-title check script computes ${#msg} and rejects any non-merge title longer than 50 chars. Commit 9c5ec84e's title UCC/CORE: Added automation to loading local rank from topo is 58 characters, so the script printed Commit title is too long: 58 / Bad commit title: ... and exited 1. (The header prefix UCC/CORE is actually not in the allowed prefix set either — the allowed CORE prefix is CORE, not UCC/CORE — but the length check fired first.)

Implicated commit: [REDACTED:Hex High Entropy String] — "UCC/CORE: Added automation to loading local rank from topo"

File: .github/workflows/codestyle.yaml (the inline check_title() shell function; the offending input is the commit message of 9c5ec84e)

Suggested fix: Amend the commit message to a title of ≤50 characters that uses a valid header prefix. For example: CORE: Auto-load local rank from topo (36 chars). Rebase/force-push the branch NIC_bid_auto (PR #1245) with the corrected title. Note the prefix should be one of the allowed tokens (CORE, TOPO, etc.) — UCC/CORE is not in the allow-list, so use CORE: or TOPO:.

Related: none

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 316340a8-0699-4de8-8037-740627698d50 in the triage console for the audit trail.

[MaayanGadishNvidia]

@MaayanGadishNvidia fix linter please

fixed @janjust

[janjust]

/build

[MaayanGadishNvidia]

@MaayanGadishNvidia fix linter please

fixed @janjust

I fixed again @janjust

[janjust]

/build

[svcnbu-swx-hpcx]

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit 3ecfd2cf

TL;DR: The build failed because the Slurm salloc on the gb300nvl72_ci partition could not obtain 2 GB300 NVL72 nodes within its 1-hour --immediate=3600 window and timed out; this is a cluster-capacity/infrastructure issue, not a code or test defect. Retry the build (or increase capacity/queue priority) — no source change will fix it.

Full analysis

Summary: Stage "Allocate Slurm job" (node 174) failed with salloc: error: Unable to allocate resources: Connection timed out before any UCC NVLS test ran.

Root cause: salloc -N 2 -p gb300nvl72_ci --immediate=3600 was issued at 20:11:10Z and blocked, legitimately waiting for resources (job 2021114 queued and waiting for resources). Exactly ~60 minutes later (21:11:19Z) the immediate-allocation timeout expired and Slurm returned "Unable to allocate resources: Connection timed out." The ~1-hour gap is not a hang — it is the intended --immediate=3600 wait for a scheduler grant. The gb300nvl72_ci partition simply had no 2-node slot free within the hour. The downstream test stages (Run UCC NVLS perftest, MPI allreduce/reduce_scatter) never started because the allocation never succeeded. The commit under test (3ecfd2c / PR #1245) is unrelated to the failure.

Implicated commit: unknown — not caused by any commit; this is a Slurm resource-availability failure.

File: N/A — failure is in the Jenkins slurm.allocation step (salloc on partition gb300nvl72_ci), not in repository source.

Suggested fix: Re-run the build; the failure is transient cluster contention on the scarce GB300 NVL72 partition. If it recurs, either (a) raise the immediateTimeout beyond 3600s so it can queue longer for a free node, (b) request higher scheduling priority for the blackwell-ci account on gb300nvl72_ci, or (c) add automatic retry/backoff around the slurm.allocation call in the pipeline so transient "Unable to allocate resources" errors don't fail the whole build. Do not treat this as a PR #1245 code regression.

Related: none found.

[svcnbu-swx-hpcx]

🤖 CI Triage Agent — UCC/ucc-test-nvls · commit aba9803a

TL;DR: The build failed in the "Allocate Slurm job" stage because salloc on the gb300nvl72_ci partition waited its full 1-hour --immediate=3600 window and Slurm then returned "Unable to allocate resources: Connection timed out" — a cluster resource-availability/infrastructure issue, not a code defect in the PR.

Full analysis

Summary: Stage "Allocate Slurm job" (node 174) failed after ~60 min when salloc could not obtain 2 nodes on partition gb300nvl72_ci.

Root cause: Slurm job 2021114 was queued and waited the entire --immediate=3600 (3600s) window without resources becoming available, then exited with salloc: error: Unable to allocate resources: Connection timed out. Timeline confirms this is a resource wait, not a hang: the command was issued at 20:11:10 and Slurm returned the allocation error at 21:11:19 — exactly the configured immediate timeout. The full runtime was spent legitimately waiting in the Slurm queue for nodes that never freed up. This is a cluster/infrastructure capacity problem on the GB300 CI partition, unrelated to commit aba9803a or PR #1245.

Implicated commit: unknown (not caused by a source commit — this is an infrastructure/scheduler resource shortage)

File: N/A — failure originates in slurm.allocation (swx-jenkins-lib), invoked with partition:gb300nvl72_ci, nodes:2, immediateTimeout:3600

Suggested fix: This is a retriable infra failure, not a code bug — re-run the build once the gb300nvl72_ci partition has free nodes. To reduce recurrence: (1) verify partition/node health and queue backlog with cluster admins, and (2) consider having the pipeline detect the "Unable to allocate resources" / immediate-timeout condition and automatically retry or mark the build as UNSTABLE/aborted rather than a hard FAILURE, so it isn't attributed to the PR under test.

Related: none (PR #1149 surfaced by search is unrelated to Slurm allocation).