Skip to content

fix: give the vendored linux ggml libraries private sonames - #39455

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/37142-private-ggml-sonames
Nov 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/37142-private-ggml-sonames

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A source-built (GPU) pywhispercpp and Kiro Crew's embedding runtime each load their own ggml in one process on Linux, whichever loads first.

On Linux, a user who builds pywhispercpp from source for GPU speech gets one of two silent failures, depending on which side loads first in the gateway:

  • whisper first: memory search drops to keyword search, with one WARNING that libllama.so has an undefined symbol: ggml_lightning_indexer;
  • embeddings first: speech runs on the CPU, because whisper is bound to our CPU-only ggml and its GPU backend never loads.

Why it matters

Someone who builds whisper.cpp for their GPU, which is the route the STT docs point to, ends up with either speech silently on the CPU or memory search silently on keywords. Which one they get depends on load order, and they can't fix either from their side short of patching their own build.

Not a goal

What changed (motivation → approach → change)

The vendored Linux ggml libraries now have private names, so the dynamic loader can no longer mix them up with a user's own ggml.

Root cause: our linux_x86_64, linux_aarch64 and linux_aarch64_dotprod closures shipped libggml.so.0, libggml-base.so.0 and libggml-cpu.so.0 under upstream's public sonames. glibc reuses an already-loaded object whose soname matches a DT_NEEDED entry, whatever the dlopen mode. A source-built whisper.cpp installs the same public names, so whichever side loaded first supplied ggml for both.

Approach (the maintainer decision on the issue): auditwheel-style private names. The vendored libgomp files already use this scheme.

before after
file name and DT_SONAME libggml-base.so.0 libggml-base-<first 8 hex of the unpatched sha256>.so.0 (likewise libggml, libggml-cpu)
DT_NEEDED of libllama.so, libggml*, libggml-cpu* public names the private names in the same dir
libllama.so file name and soname, libgomp, RPATH unchanged (RPATH stays DT_RPATH)
.text / .rodata of every patched file byte-identical (checked for all 12 files)

The patch tool is pinned (patchelf 0.17.2, pip install patchelf==0.17.2.1), and two runs from the unpatched files give byte-identical output. Each patched file grows by 32–290 KB (patchelf adds a page-aligned load segment for the longer dynamic strings), about 1.7 MB across the three dirs.

Around the binaries:

  • _REQUIRED_VENDORED_LIBS (in src/kiro_crew/_llama_lib_path.py) names the new files.
  • scripts/vendor_manifest.sha256 is regenerated.
  • _PROVEN_RUNTIME_DIGEST is re-proven: TestRealModel passed against the shipped GGUF on this tree.
  • _vendor/README.md gets a "Private ggml names on Linux" subsection with the exact recipe, the check, and a before/after sha256 table. It also adds the rename step to the upgrade and local-divergence notes.
  • One sentence in the memory-skills-hooks spec.

Nothing in packaging changes: setup.cfg and MANIFEST.in select these files by directory, and the new names still end .so.0, so global-exclude *.so does not strip them.

Backwards compatibility

Compatible: the loader opens libllama.so by path and every other file is reached through libllama.so's own DT_NEEDED entries, which name the new files; an operator's LLAMA_CPP_LIB_PATH directory is not touched.

The diff renames shipped files, so the readers of the old names:

Reader: src/kiro_crew/embeddings_native.py:verify_vendored_libs -- release -- reads _REQUIRED_VENDORED_LIBS, which names the new files; test_source_tree_carries_every_required_lib
Reader: src/kiro_crew/embeddings_native.py:_linux_aarch64_libs_dir -- chat -- dotprod completeness check iterates the same declaration; test_an_incomplete_dotprod_dir_loads_todays_dir
Reader: src/kiro_crew/embeddings_native.py:_load_llama_class -- chat -- missing-file check iterates the same declaration; probe below loads and embeds in a clean process
Reader: src/kiro_crew/doctor_checks/features.py:verify_vendored_libs -- release -- same declaration; test/test_cli_stt_memory_doctor.py
Reader: scripts/verify_vendored_payload.py:_REQUIRED_VENDORED_LIBS -- release -- reads the literal with ast.literal_eval; TestPayloadVerifierWithoutRuntimeDependencies
Reader: setup.cfg:package_data / MANIFEST.in -- release -- per-directory * globs, names still end .so.0; test_package_data_declares_the_libs_explicitly, test_manifest_rules_keep_every_required_lib
Reader: src/kiro_crew/_vendor/llama_cpp/llama_cpp.py:load_shared_library -- chat -- opens libllama.so by base name, which keeps its name; probe below

Tests

  • test/test_vendored_llama_payload.py::test_linux_ggml_libraries_carry_private_sonames (new): a stdlib ELF64 reader opens every committed file in the three Linux dirs. It fails if a public ggml file name, soname or NEEDED entry comes back, if a libggml* soname differs from its file name or is not hash-named, if a libggml/libgomp dependency names a file the dir lacks, or if RPATH is lost or RUNPATH appears. It needs no readelf, so it runs on the Windows and macOS lanes too. Red on the base commit (linux_aarch64 ships a public ggml file name), green here.
  • test/test_embeddings_bundled_libs.py::…::test_an_incomplete_dotprod_dir_loads_todays_dir reads the dotprod ggml-cpu name from the declaration instead of hard-coding the public name.
  • test/test_embed_bounded_decode.py: _PROVEN_RUNTIME_DIGEST updated after TestRealModel passed with the shipped model.

Proof: not provable -- other: the production change is binary (patchelf-rewritten .so files) and prove.py cannot reverse-apply binary hunks; the pin test was run on the base commit instead and fails there

Manual verification

How it was verified, on Linux x86_64 (Ubuntu 24.04, glibc 2.39), with pywhispercpp 1.5.1 (f7bf621, whisper.cpp f24588a) built from source twice: a CPU build, and an OpenBLAS build (-DGGML_BLAS=ON). The BLAS build stood in for a GPU build, since it is a non-CPU ggml backend that needs no GPU. No CUDA or Vulkan GPU build was run. One fresh process per row; /proc/self/maps is the oracle.

tree whisper build load order _load_llama_class() ggml mapped (/proc/self/maps) whisper's ggml registries embed
base cpu whisper-first None: undefined symbol: ggml_lightning_indexer whisper's ['CPU'] -
base cpu llama-first Llama ours ['CPU'] 1024 dims (= clean)
base blas whisper-first None: undefined symbol: ggml_lightning_indexer whisper's ['BLAS', 'CPU'] -
base blas llama-first Llama ours ['CPU'] 1024 dims (= clean)
base none llama-only Llama ours - 1024 dims (= clean)
fix cpu whisper-first Llama ours + whisper's ['CPU'] 1024 dims (= clean)
fix cpu llama-first Llama ours + whisper's ['CPU'] 1024 dims (= clean)
fix blas whisper-first Llama ours + whisper's ['BLAS', 'CPU'] 1024 dims (= clean)
fix blas llama-first Llama ours + whisper's ['BLAS', 'CPU'] 1024 dims (= clean)
fix none llama-only Llama ours - 1024 dims (= clean)

Every embed is 1024 dims and bit-identical (max abs diff 0) to a clean process on either tree, as expected: .text and .rodata are unchanged.

detect() reads whisper_print_system_info(), which in this whisper.cpp lists only registries that export ggml_backend_get_features. ggml-blas does not export it, so detect() reports cpu even for the BLAS build loaded alone. The BLAS oracle is therefore the registry list of the object that holds soname libggml.so.0, i.e. what whisper binds to. In this ggml, only the CPU, CUDA, Metal and zDNN registries export it. A Vulkan build therefore reads as cpu there too. That limitation already exists and is outside this goal.

aarch64: under qemu-aarch64-static 8.2.2 (CPU max) with an arm64 python:3.12-slim rootfs, libllama.so was loaded by full path from copies of linux_aarch64/ and linux_aarch64_dotprod/ on both trees, followed by one embed through the vendored bindings. The fix maps the renamed ggml files, DOTPROD = 1 is reported for the dotprod build, and the vectors are bit-identical to base's.

Real-machine verification

Machine: EC2 m7i.2xlarge, Ubuntu 24.04.5 LTS (glibc 2.39), x86_64, 8 vCPU / 32 GiB; aarch64 under qemu-aarch64-static 8.2.2 on the same machine
Before (main 433d99b; main has not changed the vendored libraries or the loader since): env -u LLAMA_CPP_LIB_PATH PYTHONPATH=<tree>/src python probe.py <whisper-first|llama-first> <pywhispercpp build> <model>, one fresh process per row -> whisper-first: _load_llama_class() returns None with libllama.so: undefined symbol: ggml_lightning_indexer (CPU and BLAS builds); llama-first: only our ggml is mapped and the BLAS build's registries drop to ['CPU']
After (head 67e02db; the machine ran 80064f7e622, whose binaries, code and tests are identical, only README wording and its manifest line differ): the same command -> both orders load the Llama class, map both ggml sets, embed 1024 dims bit-identical to a clean process, and the BLAS build keeps ['BLAS', 'CPU']; aarch64 and aarch64_dotprod load and embed with the renamed files mapped; the pin test is red on the base and green on the head
Cleanup: the checkouts and builds lived in the job's own scratch directory; the machine was a tagged instance from a verification pool (no inbound rules, terminate-on-shutdown) and is terminated: it no longer appears in the pool's tagged-instance listing

Screenshots / video

No user-visible UI change: the diff touches no frontend path.

Related Issues

Closes #37142

Sibling sweep

Swept: git ls-files 'src/kiro_crew/_vendor/llama_cpp_libs/*' and readelf -d of every Linux file for a public libggml* name
Sibling: src/kiro_crew/_vendor/llama_cpp_libs/win_amd64/ggml.dll -- tracked in #39002
Sibling: src/kiro_crew/_vendor/llama_cpp_libs/macos_arm64/libggml.0.dylib -- not affected -- macOS is outside this goal (Not a goal), and the report and decision are Linux-only
Sibling: src/kiro_crew/_vendor/llama_cpp_libs/linux_x86_64/libllama.so -- not affected -- keeps soname libllama.so.0; whisper.cpp ships no libllama, and ctypes opens it by path

Pattern harvest

Rule candidate: lint
Pattern: a vendored native library that keeps upstream's public soname collides with a user's own copy loaded in the same process. The new payload test pins this for the Linux ggml files; a later vendored ELF closure should get the same soname/NEEDED pin.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 67e02db4f51f0c889c4702e52a0c534be5e3e091 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 67e02db4f51f0c889c4702e52a0c534be5e3e091 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix is at cause level: the public ggml sonames are what let glibc reuse whichever ggml loaded first, and this removes them. The Windows sibling is declared and deferred to #39002.

Not checked: whether the patched aarch64 libraries load at runtime. Only x86_64 ran TestRealModel.

Notes

  • Item 6 — oversized: the README's before/after sha256 table copies its "After" column from scripts/vendor_manifest.sha256 ("the after column equals the manifest; re-record both columns on every upgrade"). That is a second hand-kept copy of the same 12 digests, and it must be edited on every upgrade. Its "Before" digests are already encoded in the 8-hex file names, and the dotprod build block lists them again. Subtraction: delete the table and keep the one sentence pointing to the manifest.

What this change ships

Inventory (7 items) — 6 justified

Intent: let a user's source-built (GPU) whisper.cpp and Kiro Crew's embedding runtime each load their own ggml in one Linux process. FIX.

  1. Linux libggml* files renamed to private hash names, with the soname and libllama.so/ggml dependency entries rewritten in all three Linux dirs — justified
  2. _REQUIRED_VENDORED_LIBS names the new files — justified
  3. vendor_manifest.sha256 regenerated and _PROVEN_RUNTIME_DIGEST re-pinned — justified
  4. New stdlib ELF test that fails if a public ggml name, a mismatched soname or a RUNPATH comes back — justified
  5. Dotprod test reads the ggml-cpu name from the declaration instead of hard-coding it — justified
  6. _vendor/README.md rename recipe, upgrade/divergence notes and a before/after sha256 table — oversized: the table hand-copies the manifest
  7. One sentence in the memory-skills-hooks spec — justified

[FIRST-PRINCIPLES-REVIEWED] 67e02db

Description read

[DESCRIPTION-READ] 7aa2de1f0c8f4debe3376f8d946ca1de97e24c4dcb53d955ae18099c2f2cbb10

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 67e02db4f51f0c889c4702e52a0c534be5e3e091 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: renaming to private sonames fixes the cause (the glibc soname reuse), and every reader of the old names in this tree goes through _REQUIRED_VENDORED_LIBS or a per-directory glob.

Notes

  • _vendor/llama_cpp/_ggml.py asks for libggml.so by base name. Nothing imports it, and the base tree's libggml.so.0 didn't match that name either, so nothing changes. If it is ever imported on Linux it will fail.
  • The 8-hex names now sit in _llama_lib_path.py, the manifest and the README. Every vendor upgrade has to regenerate all three together. The new pin test catches drift between the binaries and the declaration, but not stale README tables.

[DESIGN-REVIEWED] 67e02db

Description read

[DESCRIPTION-READ] 870352047cd5fbdc2ce9e0f5ddb3e43350fae2fed92d330a68db396fa6358ff5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 67e02db4f51f0c889c4702e52a0c534be5e3e091 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 67e02db

Verdict parsed from the review's SHA-scoped output markers for commit 67e02db4f51f0c889c4702e52a0c534be5e3e091.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 67e02db4f51f0c889c4702e52a0c534be5e3e091 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 67e02db4f51f0c889c4702e52a0c534be5e3e091 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 67e02db

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 67e02db4f51f0c889c4702e52a0c534be5e3e091 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] CI triage for head 016fc78:

  • Backend Tests (3.12, 7): test/test_source_link_github_case.py::TestOneChipPerObject::test_case_variants_collapse_to_one_chip (assert 0 == 1). That is a test-order flake that is also red on main and on unrelated PRs; the fix is test: give every test the cold source-link settings snapshot #39475, recorded in Flaky: test_case_variants_collapse_to_one_chip (source_links_total 0 under xdist) #39503. This PR does not touch the dashboard or that test.
  • Backend Tests (3.12, 1/2/3/5/6/8), Real Adapter Contract Tests, E2E (stub ACP backend, offline), E2E Gateway Boot (ubuntu-latest): the runner could not install lsof from apt (mirror fetch failures), before any test ran. Coverage Gate failed because those jobs did.
  • Analyze (python): the CodeQL results upload hit the API rate limit.

The failed jobs get one re-run on a later pass. If shard 7 is still red after it, this PR rebases once #39475 lands.

The Linux llama.cpp closure shipped ggml under upstream's public names
(libggml.so.0, libggml-base.so.0, libggml-cpu.so.0). glibc reuses an
already-loaded object whose soname matches a DT_NEEDED entry, so a
source-built whisper.cpp (pywhispercpp, the GPU route for speech) in
the same process and the embedding runtime shared whichever ggml loaded
first. Whisper first: libllama.so fails on ggml_lightning_indexer,
which whisper's ggml lacks, and memory search falls back to keywords.
Embeddings first: whisper binds to our CPU-only ggml and its GPU or
BLAS backend never loads.

Rename the three files in linux_x86_64, linux_aarch64 and
linux_aarch64_dotprod to auditwheel-style private names
(libggml-<sha256 prefix>.so.0), set each soname to match and rewrite
the dependency entries of libllama.so and the ggml libraries, with
pinned patchelf 0.17.2. patchelf moves the dynamic strings into an
added load segment; .text and .rodata are byte-identical and the RPATH
is unchanged. libllama.so and libgomp keep their names.

The required-libs declaration and the vendor manifest name the new
files, the bounded-decode proof digest is re-proven against the shipped
model, and the vendoring notes carry the rename recipe so an upgrade
re-applies it. A new payload test reads each Linux file's dynamic
section and fails if a public ggml name, a mismatched soname or a
RUNPATH appears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_session_card_source_links_knob.py (#39470 stops test_put_persists_off from leaving the chip switch off, which made test/test_source_link_github_case.py red on Backend Tests shard 7 on both runs of the old head) and docs/system-specs/modules/memory-skills-hooks.md since this head's CI ran (433d99b -> b9576aa); the patch is unchanged (git range-diff: =) and CI is re-running.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/37142-private-ggml-sonames branch from 016fc78 to 67e02db Compare November 19, 2026 23:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On head 67e02db I inspected the dynamic sections of every vendored Linux library. Each ggml file now carries a private, hash-named soname that matches its file name. libllama.so and the ggml files list only those private names as dependencies, all present in the same directory, and RPATH is preserved. A section comparison against main showed .text and .rodata unchanged. The payload, bundled-libs, manifest, lib-path, doctor and bounded-decode tests pass, and so does the real-model embedding test against the shipped GGUF, which confirms the re-pinned runtime digest. A clean-process load maps only the renamed files. Packaging selects these files by directory, so nothing blocks this change.

@KiroCrewCaptain
KiroCrewCaptain merged commit 3fcd2f3 into main Nov 20, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/37142-private-ggml-sonames branch November 20, 2026 00:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Source-built (GPU) pywhispercpp clashes with the vendored llama.cpp ggml sonames

2 participants