Repository navigation
chore(vendor): track zvec c133f20; feat(fts)!: adapt to the embedded jieba dictionary - #21
Merged
Merged
Conversation
Upstream zvec moved six commits ahead of 60956b1. 52b72a4 embeds cppjieba's default dictionary and HMM model into libzvec_c_api as LZ4-compressed arrays and deletes the four C-API functions that configured an external dict directory: zvec_config_data_set_jieba_dict_dir zvec_config_data_get_jieba_dict_dir zvec_set_default_jieba_dict_dir zvec_get_default_jieba_dict_dir GlobalConfig::ConfigData::jieba_dict_dir is gone too; the tokenizer now resolves per-field extra_params["jieba_dict_dir"] > ZVEC_JIEBA_DICT_DIR > the embedded default, and CMake no longer installs data/jieba_dict into the SDK. Drop the corresponding Rust surface: the -sys declarations and build_time_jieba_dict_dir(), the dict discovery in zvec-sys/build.rs, ConfigBuilder::jieba_dict_dir, set/get_default_jieba_dict_dir, the auto-registration path and its dladdr/GetModuleHandleEx probe (plus the libdl link it needed), the data/jieba_dict staging in zvec-rust-build, and the dict bundling in build-prebuilt.yml. jieba_fts_test no longer skips when no dict directory is found: Chinese FTS must now work with zero configuration, which is what proves the bump. BREAKING CHANGE: zvec-rust removes set_default_jieba_dict_dir, get_default_jieba_dict_dir and ConfigBuilder::jieba_dict_dir. Use the per-field extra_params jieba_dict_dir or the ZVEC_JIEBA_DICT_DIR environment variable instead. The next release needs a minor bump. The other five commits are RaBitQ 0.7 modular targets (the submodule path moves to RaBitQ-Library-0.7.0), flat segment-locking batching, refined internal IDs, IVF primary-key brute force for filtered queries, and upstream's own Windows ARM64 wheel/SDK publishing, which matches the prebuilt target added in #18.
This was referenced Oct 20, 2026
Merged
mrcs64
added a commit
to mrcs64/zvec-rust
that referenced
this pull request
Oct 26, 2026
Upstream removed build-script cppjieba dictionary discovery in zvec-ai#21 (the dictionary is now embedded in libzvec_c_api). Resolve by dropping the remaining dictionary handling from this branch: the strict-mode dictionary paragraphs in README, the ZVEC_RUST_JIEBA_DICT_DIR registration, and the test that covered resolve_jieba_dict_dir. Strict mode now constrains only the native library, as the upstream design implies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream moved six commits ahead of
60956b1, and one of them removes four C-API functions. This PR tracksc133f20and adapts the Rust side.1. Track upstream zvec
c133f20vendor/zvec60956b1→c133f20(upstreammainHEAD). Compare API:status: ahead, ahead_by: 6, behind_by: 0.c549686build(rabitq): upgrade to RaBitQ 0.7 modular targets — the submodule path moves tothirdparty/RaBitQ-Library/RaBitQ-Library-0.7.0(de793e8), so an existing checkout needsgit submodule update --init --recursive.ab58382perf(flat): batch segment locking to fix the HNSW RaBitQ build regression.89e83dbperf(search): output refined internal IDs directly.e30317ffix(ivf): support primary-key brute-force search for filtered queries.52b72a4feat(fts): embed default Jieba resources with LZ4 — the breaking one, see below.c133f20ci: publish Windows ARM64 wheels and SDK — upstream now builds WoA onwindows-11-armwithmsvc_arch: arm64and Python ≥ 3.11 (CPython's WoA builds start at 3.11), i.e. exactly the setup chore(vendor): track zvec 60956b1; ci(prebuilt): add Windows ARM64 #18 added to our prebuilt matrix. Nothing to change on our side.Build inputs we pass are unaffected:
BUILD_C_BINDINGS/BUILD_TOOLSoptions and thezvec_c_apitarget all still exist atc133f20, and thelz4submodule the embedded dict links was already in.gitmodules.2. C-API change: the jieba dict-dir functions are gone
52b72a4embeds cppjieba'sjieba.dict.utf8andhmm_model.utf8intolibzvec_c_apias LZ4-compressed arrays (jieba_dict_lz4.inc,jieba_hmm_model_lz4.inc), so an external dict directory is no longer needed. Upstream deleted:zvec_config_data_set_jieba_dict_dirConfigData::set_jieba_dict_dirzvec_config_data_get_jieba_dict_dirzvec_set_default_jieba_dict_dirset_default_jieba_dict_dirzvec_get_default_jieba_dict_dirget_default_jieba_dict_dirGlobalConfig::ConfigData::jieba_dict_diris gone as well, and the tokenizer now resolves the dictionary as: per-fieldextra_params["jieba_dict_dir"]>ZVEC_JIEBA_DICT_DIR> embedded default. CMake no longer installsdata/jieba_dictinto the SDK, and upstream's_build_prebuilt.ymlstopped copyingdata/in the same commit.Rust changes
zvec-sys— dropped the fourexterndeclarations andbuild_time_jieba_dict_dir();build.rsno longer hunts for a cppjieba dict directory or emitsZVEC_RUST_JIEBA_DICT_DIR.zvec— removedConfigBuilder::jieba_dict_dir,set_default_jieba_dict_dir,get_default_jieba_dict_dirand the whole auto-registration path (auto_register_default_jieba_dict_dir,discover_jieba_dict_dir,is_jieba_dict_dir, and the unix/windows/fallbackloaded_library_dir).initializeis now a straightzvec_initializecall. Thedladdrprobe was the only reasonzvec/build.rslinkedlibdlon Linux, so that link is gone too.zvec-build— stopped stagingdata/jieba_dictnext to the built artifacts; it stages the shared library only.zvec/tests/jieba_fts_test.rs— the dict-discovery test is gone, and the Chinese-search test no longer skips when no dict directory is found: with the embedded dictionary it must always pass. That test is what actually proves the bump works end to end.build-prebuilt.yml— dropped the twodata/jieba_dictbundling blocks so our tarballs match upstream's SDK layout.extra_params>ZVEC_JIEBA_DICT_DIR, plus a note thatuser_dict_pathaugments whichever dictionary was selected.Breaking change for
zvec-rust: two public functions and one builder method are removed, so the next release needs a minor bump (0.8.0), not a patch. Versions are deliberately left untouched here.Validation
cargo fmt --all -- --checkclean.ZVEC_AUTO_BUILD=0 cargo clippy --workspace --all-targets -- -D warningsclean.ZVEC_AUTO_BUILD=0 cargo test --workspace --lib: 119 + 8 + 0 passed (was 123 + 8 + 0; the four missing ones are the removed jieba tests).ZVEC_AUTO_BUILD=0 cargo test -p zvec-rust-sys --test build_script: 10 passed.vendor/zvec/buildlibrary predates this bump and still exports the removed symbols, so it cannot exercise the embedded dictionary.ci.ymlbuildszvec_c_apifrom the bumped submodule on all five platforms and runscargo test --workspace --tests, which includesjieba_fts_test— that is the real validation.build-prebuilt.ymlstill parses to 8 targets; it is not triggered by PRs, so it is dispatched manually on this branch (link below).