From 46db0fe11ac595101c1be252c5a9982cd0986b3c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 07:22:14 +0000 Subject: [PATCH 1/4] Add CI to run pytest under AddressSanitizer Build the native extension (and the bundled onnx/protobuf) with -fsanitize=address and run the existing pytest suite with the ASan runtime preloaded, so memory errors (heap-buffer-overflow, use-after-free, ...) in onnxoptimizer's C++ passes are caught in CI. CPython is not built with ASan, so libasan is LD_PRELOADed and detect_leaks=0 suppresses leaks from the uninstrumented interpreter and third-party libraries. onnxruntime is omitted from the test dependencies because its uninstrumented C++ throws exceptions that trip ASan's __cxa_throw interceptor; it is only used for optional numerical comparison, so every optimize() call is still exercised. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GewbTty3fkFT4G9u6J1JJn Signed-off-by: Takeshi Watanabe Signed-off-by: take-cheeze --- .github/workflows/pytest-asan.yml | 68 +++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 .github/workflows/pytest-asan.yml diff --git a/.github/workflows/pytest-asan.yml b/.github/workflows/pytest-asan.yml new file mode 100644 index 000000000..e75f7086e --- /dev/null +++ b/.github/workflows/pytest-asan.yml @@ -0,0 +1,68 @@ +# Copyright (c) ONNX Project Contributors +# +# SPDX-License-Identifier: Apache-2.0 + +name: Pytest with AddressSanitizer + +on: + push: + branches: + - main + pull_request: + workflow_dispatch: + +permissions: + contents: read + +jobs: + pytest-asan: + name: Pytest (ASan) + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + submodules: recursive + persist-credentials: false + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.12" + + - name: Install build dependencies + run: python -m pip install --upgrade pip protobuf setuptools wheel "cmake>=3.22" + + - name: Build and install onnxoptimizer with AddressSanitizer + env: + # Build the native extension (and the bundled onnx / protobuf) with + # ASan. The flags are single-quoted so setup.py's shlex.split keeps + # each multi-token value as one -D... argument. -g / frame pointers + # give readable stack traces; the linker flags make libasan a + # DT_NEEDED of the shared objects. + ASAN_FLAGS: "-fsanitize=address -fno-omit-frame-pointer -g" + run: | + export CMAKE_ARGS="-DCMAKE_CXX_FLAGS='${ASAN_FLAGS}' -DCMAKE_C_FLAGS='${ASAN_FLAGS}' -DCMAKE_EXE_LINKER_FLAGS=-fsanitize=address -DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=address -DCMAKE_MODULE_LINKER_FLAGS=-fsanitize=address" + python -m pip install --no-build-isolation -v . + + - name: Install test dependencies + # onnxruntime is intentionally omitted: it is only used by the tests for + # optional numerical comparison (guarded by `has_ort`), and its + # uninstrumented C++ throws exceptions that trip ASan's __cxa_throw + # interceptor (CHECK failed: real___cxa_throw != 0) when the runtime is + # preloaded, aborting the interpreter before any real issue is found. + # Skipping it keeps ASan focused on onnxoptimizer's own C++ while still + # running every optimize() call and model check. + run: python -m pip install pytest pytest-xdist numpy + + - name: Run pytest under AddressSanitizer + env: + # CPython itself is not built with ASan, so the runtime has to be + # preloaded; otherwise dlopen of the instrumented extension fails + # with "ASan runtime does not come first in initial library list". + # detect_leaks=0 suppresses leaks from the uninstrumented interpreter + # and third-party libraries so the job only fails on genuine memory + # errors (heap-buffer-overflow, use-after-free, ...) in onnxoptimizer. + ASAN_OPTIONS: "detect_leaks=0:halt_on_error=1:symbolize=1" + run: | + export LD_PRELOAD="$(gcc -print-file-name=libasan.so)" + # Limit worker count to keep ASan's larger memory footprint in check. + python -m pytest onnxoptimizer/test/ -n 2 -v From 7a45bffeac724b58ca0659a383e6f3bd594efa8a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 11:08:51 +0000 Subject: [PATCH 2/4] Fix ASan pytest job aborting on onnx C++ exceptions The job aborted with "AddressSanitizer: CHECK failed ... real___cxa_throw != 0" on the first checker.check_model call. The cause is not onnxruntime (already omitted) but the uninstrumented `onnx` Python wheel: its C++ extension throws C++ exceptions from checker.check_model / shape inference, which the tests exercise constantly. With only libasan preloaded, ASan's __cxa_throw interceptor cannot resolve the real symbol at init and aborts the process on the first raised exception. Preload libstdc++ alongside libasan (libasan still first, as ASan requires) so the interceptor finds the real __cxa_throw and exceptions unwind normally; only genuine memory errors now fail CI. Update the stale comment that attributed the abort solely to onnxruntime. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GewbTty3fkFT4G9u6J1JJn Signed-off-by: Takeshi Watanabe Signed-off-by: take-cheeze --- .github/workflows/pytest-asan.yml | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/.github/workflows/pytest-asan.yml b/.github/workflows/pytest-asan.yml index e75f7086e..624f0ca7a 100644 --- a/.github/workflows/pytest-asan.yml +++ b/.github/workflows/pytest-asan.yml @@ -45,12 +45,9 @@ jobs: - name: Install test dependencies # onnxruntime is intentionally omitted: it is only used by the tests for - # optional numerical comparison (guarded by `has_ort`), and its - # uninstrumented C++ throws exceptions that trip ASan's __cxa_throw - # interceptor (CHECK failed: real___cxa_throw != 0) when the runtime is - # preloaded, aborting the interpreter before any real issue is found. - # Skipping it keeps ASan focused on onnxoptimizer's own C++ while still - # running every optimize() call and model check. + # optional numerical comparison (guarded by `has_ort`), so skipping it + # keeps ASan focused on onnxoptimizer's own C++ while still running every + # optimize() call and model check. run: python -m pip install pytest pytest-xdist numpy - name: Run pytest under AddressSanitizer @@ -63,6 +60,15 @@ jobs: # errors (heap-buffer-overflow, use-after-free, ...) in onnxoptimizer. ASAN_OPTIONS: "detect_leaks=0:halt_on_error=1:symbolize=1" run: | - export LD_PRELOAD="$(gcc -print-file-name=libasan.so)" + # libasan must come first in the preload list (ASan requirement), but + # libstdc++ must be preloaded alongside it: the `onnx` Python wheel is + # an uninstrumented C++ extension that throws C++ exceptions (e.g. from + # checker.check_model / shape_inference, which the tests exercise + # constantly). Without libstdc++ loaded at ASan init, the __cxa_throw + # interceptor can't resolve the real symbol and aborts the process with + # "CHECK failed: ... real___cxa_throw != 0" on the first raised + # exception. Preloading libstdc++ lets the interceptor find __cxa_throw + # so exceptions unwind normally and only genuine memory errors fail CI. + export LD_PRELOAD="$(gcc -print-file-name=libasan.so):$(gcc -print-file-name=libstdc++.so.6)" # Limit worker count to keep ASan's larger memory footprint in check. python -m pytest onnxoptimizer/test/ -n 2 -v From a959cf781f2c56ce9877392b8e0de8bd7a0d29f1 Mon Sep 17 00:00:00 2001 From: Takeshi Watanabe Date: Wed, 29 Jul 2026 21:05:46 +0900 Subject: [PATCH 3/4] Apply suggestion from @take-cheeze Signed-off-by: Takeshi Watanabe Signed-off-by: take-cheeze --- .github/workflows/pytest-asan.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pytest-asan.yml b/.github/workflows/pytest-asan.yml index 624f0ca7a..adb7206ce 100644 --- a/.github/workflows/pytest-asan.yml +++ b/.github/workflows/pytest-asan.yml @@ -40,7 +40,7 @@ jobs: # DT_NEEDED of the shared objects. ASAN_FLAGS: "-fsanitize=address -fno-omit-frame-pointer -g" run: | - export CMAKE_ARGS="-DCMAKE_CXX_FLAGS='${ASAN_FLAGS}' -DCMAKE_C_FLAGS='${ASAN_FLAGS}' -DCMAKE_EXE_LINKER_FLAGS=-fsanitize=address -DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=address -DCMAKE_MODULE_LINKER_FLAGS=-fsanitize=address" + export CXXFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address python -m pip install --no-build-isolation -v . - name: Install test dependencies From 1a0b8817aec540cd4c6a8dd95f881ea32b6c59ed Mon Sep 17 00:00:00 2001 From: take-cheeze Date: Wed, 29 Jul 2026 12:42:42 +0000 Subject: [PATCH 4/4] Build the ASan extension in place so pytest can import it The Pytest (ASan) job failed at collection with: ModuleNotFoundError: No module named 'onnxoptimizer.onnx_opt_cpp2py_export' `pip install .` puts the compiled extension only in site-packages, but the test step runs `pytest onnxoptimizer/test/` from the repo root. Because the test modules live inside the onnxoptimizer package, importing them puts the repo root on sys.path, so `import onnxoptimizer` resolves to the in-tree source package -- which has the .py sources but no compiled onnx_opt_cpp2py_export extension -- and collection fails before any test runs. Build the extension in place as well (reusing the cmake build the pip install just produced) so the compiled module sits next to the source package pytest imports. This is orthogonal to the ASan flags; the extension itself builds and imports correctly under ASan once it is present in the source tree. Signed-off-by: take-cheeze Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01GewbTty3fkFT4G9u6J1JJn --- .github/workflows/pytest-asan.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.github/workflows/pytest-asan.yml b/.github/workflows/pytest-asan.yml index adb7206ce..38072d5b4 100644 --- a/.github/workflows/pytest-asan.yml +++ b/.github/workflows/pytest-asan.yml @@ -42,6 +42,14 @@ jobs: run: | export CXXFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address python -m pip install --no-build-isolation -v . + # The test step runs `pytest onnxoptimizer/test/` from the repo root, so + # `import onnxoptimizer` binds to the in-tree source package instead of the + # installed one -- and the source tree has no compiled extension. Build it + # in place as well (reusing the cmake build pip just produced) so the + # onnx_opt_cpp2py_export module sits next to that source; without this the + # tests fail at collection with "ModuleNotFoundError: No module named + # 'onnxoptimizer.onnx_opt_cpp2py_export'". + python setup.py build_ext --inplace - name: Install test dependencies # onnxruntime is intentionally omitted: it is only used by the tests for