Skip to content

Force a working BLAS kernel on the runners that need one - #897

Merged
iory merged 2 commits into
mainfrom
sdf-scale-ci-diagnostics
Sep 11, 2026
Merged

Force a working BLAS kernel on the runners that need one#897
iory merged 2 commits into
mainfrom
sdf-scale-ci-diagnostics

Conversation

@iory

@iory iory commented Sep 11, 2026

Copy link
Copy Markdown
Owner

test_sdf_from_robot_with_scale_parameter has been going red on some runs of a commit and green on others, most recently on 15d396f, which produced one failing run and one clean one. The assertion said the signed distance field had been built on a grid 7.4 times too wide:

worst |sdf| is 0.5357201464651443 (4.19 grid cells of 0.127984),
tolerance 0.19197599999999998

It is not scipy 1.8.0, which is what the failing cells had in common by coincidence. In run 34554196186 the numpy-latest/scipy-1.8.0 cell passed while numpy-1.23.5/scipy-1.8.0 failed; in 34552744465 the numpy-latest/scipy-1.8.0 cell failed.

What it is

numpy's bundled OpenBLAS detects the Intel(R) Xeon(R) 6973P-C as Core: Cooperlake, and that kernel returns wrong results for (4, 4) by (4, n) double-precision products once n passes roughly 128. Most rows come back as [10*x + 10, 0, 0]: the homogeneous row folded into the first column and the other two zeroed. 2311 of 2503 rows on the sample.

That is the product trimesh.transform_points makes, so every mesh a URDF scales or places is built from it. The collision mesh comes out a shape the bunny never had, SDFGen sizes the grid from its bounding box, and the failure surfaces four layers from the cause. The Mitsuba viewer sizing a Panda scene wrong in the same job was this too.

Minimal reproducer:

import numpy as np
n = 2503
V = np.random.default_rng(0).random((n, 3))
M = np.eye(4); M[:3, :3] = np.diag([10.0, 10.0, 10.0])
stack = np.column_stack((V, np.ones(n)))
print(np.abs(np.dot(M, stack.T).T[:, :3] - V * 10.0).max())
# 0.0 anywhere healthy, 10.0 on a Xeon 6973P-C

How it was pinned down

One runner per job, 145 jobs over five batches, printing the diagnostics whether or not the assertion held, because a machine can carry the defect and still pass: an over-wide grid on its own only moves the worst vertex to about half a cell, measured at four grid sizes.

CPU runners result
Intel Xeon 6973P-C 10 all reproduced the failure, to the last digit of both the resolution and the worst vertex
AMD EPYC 7763 / 9V45 / 9V74 120 all clean
Intel Xeon Platinum 8573C 15 all clean

On an affected runner:

max abs diff
np.dot (2503, 3) x (3, 3) 0.0
(4, 4) x (4, n), n = 4, 16, 64 0.0
(4, 4) x (4, n), n = 256 10.0, 128 of 256 rows
(4, 4) x (4, n), n = 2503 10.0, 2311 of 2503 rows
einsum, which does not use BLAS 0.0
OPENBLAS_NUM_THREADS=1 10.0
OPENBLAS_CORETYPE=HASWELL 0.0
OPENBLAS_CORETYPE=SKYLAKEX 0.0
OPENBLAS_CORETYPE=NEHALEM 0.0
OPENBLAS_CORETYPE=SAPPHIRERAPIDS 10.0

Single threads make no difference, so it is not a race. einsum and the (3, 3) product are exact, so it is that kernel and that shape.

What this changes

The workflow checks that product after installing numpy and scipy, and sets OPENBLAS_CORETYPE=HASWELL only on a runner that fails it, with a warning annotation naming the CPU. Haswell rather than SkylakeX because the setting has to be harmless on the machines it is not aimed at, and the EPYC 7763s that are most of the pool have no AVX-512. Healthy runners keep their own kernels. It cannot be done from inside the session: OpenBLAS reads the variable when it loads.

Confirmed end to end on a batch of forty runners: five were 6973P-C, the guard tripped on exactly those five, and all forty then produced the correct grid.

tests/conftest.py runs the same check at session start and stops with the reason and the command to set, so the same machine under someone's desk costs a sentence rather than an afternoon.

What this does not change

  • The test. It was right. The field really was wrong, and widening the tolerance to 4.19 grid cells would have taught it to accept a misshapen robot.
  • skrobot or trimesh. Both are correct; the multiply is not. Routing Link.collision_mesh around the matrix product would only make a broken machine look like it was working.
  • The Windows job. There is no evidence the affected CPU is in that pool, and an untested cross-platform version of this guard is the riskier change. Worth revisiting if unexplained geometry failures turn up there.

Not reported upstream. The closest existing reports are OpenBLAS #2168, #3454, #5267 and #2769; none names this CPU.

test_sdf_from_robot_with_scale_parameter went red on some runs of a commit
and green on others, and the assertion said the signed distance field had
been built on a grid 7.4 times too wide. Sampling one runner per job, 145
of them over five batches, split the result perfectly by CPU: every one of
the ten Intel Xeon 6973P-C machines reproduced the failure to the last digit
of both the resolution and the worst vertex, and none of the AMD EPYC 7763,
9V45, 9V74 or Xeon Platinum 8573C ones ever did.

numpy's bundled OpenBLAS detects that CPU as Cooperlake, and the kernel it
picks returns wrong numbers for (4, 4) by (4, n) dgemm once n passes about
128: most rows come back as [10*x + 10, 0, 0], the homogeneous row folded
into the first column and the other two zeroed, 2311 of 2503 rows on the
sample. That is the product trimesh.transform_points makes, so every mesh a
URDF scales or places is built from it -- the collision mesh comes out a
shape the bunny never had, SDFGen sizes the grid from its bounding box, and
the failure surfaces four layers from the cause. The Mitsuba viewer sizing a
Panda scene wrong in the same job was this too.

Single threads make no difference, so it is not a race; einsum and the
(3, 3) by (3, n) product are exact, so it is that kernel and that shape.
HASWELL, SKYLAKEX and NEHALEM all give exact results and SAPPHIRERAPIDS does
not, so force HASWELL -- the setting has to be harmless on the machines it is
not aimed at, and SKYLAKEX would put AVX-512 kernels on the EPYC 7763s that
are most of the pool and do not have them. Only on a runner that fails the
multiply, so everything else keeps its own kernel. It cannot be done from
inside the session, because OpenBLAS reads the variable when it loads.
Confirmed on a batch of forty: five 6973P-C, five trips, forty correct grids.

conftest runs the same check and stops with the reason and the command to
set, so the same machine under someone's desk costs them a sentence rather
than an afternoon.

The test is left exactly as it was. It was right: the field really was
wrong, and widening the tolerance to 4.19 grid cells would have taught it to
accept a misshapen robot.
@iory
iory merged commit 470b3df into main Sep 11, 2026
23 checks passed
@iory
iory deleted the sdf-scale-ci-diagnostics branch September 11, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant