Skip to content

chore(vendor): track zvec c133f20; feat(fts)!: adapt to the embedded jieba dictionary - #21

Merged
chinaux merged 1 commit into
mainfrom
chore/vendor-sync-embedded-jieba
Oct 20, 2026
Merged

chinaux merged 1 commit into
mainfrom
chore/vendor-sync-embedded-jieba

Conversation

@chinaux

@chinaux chinaux commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Upstream moved six commits ahead of 60956b1, and one of them removes four C-API functions. This PR tracks c133f20 and adapts the Rust side.

1. Track upstream zvec c133f20

vendor/zvec 60956b1 → c133f20 (upstream main HEAD). Compare API: status: ahead, ahead_by: 6, behind_by: 0.

  • c549686 build(rabitq): upgrade to RaBitQ 0.7 modular targets — the submodule path moves to thirdparty/RaBitQ-Library/RaBitQ-Library-0.7.0 (de793e8), so an existing checkout needs git submodule update --init --recursive.
  • ab58382 perf(flat): batch segment locking to fix the HNSW RaBitQ build regression.
  • 89e83db perf(search): output refined internal IDs directly.
  • e30317f fix(ivf): support primary-key brute-force search for filtered queries.
  • 52b72a4 feat(fts): embed default Jieba resources with LZ4 — the breaking one, see below.
  • c133f20 ci: publish Windows ARM64 wheels and SDK — upstream now builds WoA on windows-11-arm with msvc_arch: arm64 and Python ≥ 3.11 (CPython's WoA builds start at 3.11), i.e. exactly the setup chore(vendor): track zvec 60956b1; ci(prebuilt): add Windows ARM64 #18 added to our prebuilt matrix. Nothing to change on our side.

Build inputs we pass are unaffected: BUILD_C_BINDINGS / BUILD_TOOLS options and the zvec_c_api target all still exist at c133f20, and the lz4 submodule the embedded dict links was already in .gitmodules.

2. C-API change: the jieba dict-dir functions are gone

52b72a4 embeds cppjieba's jieba.dict.utf8 and hmm_model.utf8 into libzvec_c_api as LZ4-compressed arrays (jieba_dict_lz4.inc, jieba_hmm_model_lz4.inc), so an external dict directory is no longer needed. Upstream deleted:

removed C symbol our wrapper
zvec_config_data_set_jieba_dict_dir ConfigData::set_jieba_dict_dir
zvec_config_data_get_jieba_dict_dir declaration only
zvec_set_default_jieba_dict_dir set_default_jieba_dict_dir
zvec_get_default_jieba_dict_dir get_default_jieba_dict_dir

GlobalConfig::ConfigData::jieba_dict_dir is gone as well, and the tokenizer now resolves the dictionary as: per-field extra_params["jieba_dict_dir"] > ZVEC_JIEBA_DICT_DIR > embedded default. CMake no longer installs data/jieba_dict into the SDK, and upstream's _build_prebuilt.yml stopped copying data/ in the same commit.

Rust changes

  • zvec-sys — dropped the four extern declarations and build_time_jieba_dict_dir(); build.rs no longer hunts for a cppjieba dict directory or emits ZVEC_RUST_JIEBA_DICT_DIR.
  • zvec — removed ConfigBuilder::jieba_dict_dir, set_default_jieba_dict_dir, get_default_jieba_dict_dir and the whole auto-registration path (auto_register_default_jieba_dict_dir, discover_jieba_dict_dir, is_jieba_dict_dir, and the unix/windows/fallback loaded_library_dir). initialize is now a straight zvec_initialize call. The dladdr probe was the only reason zvec/build.rs linked libdl on Linux, so that link is gone too.
  • zvec-build — stopped staging data/jieba_dict next to the built artifacts; it stages the shared library only.
  • zvec/tests/jieba_fts_test.rs — the dict-discovery test is gone, and the Chinese-search test no longer skips when no dict directory is found: with the embedded dictionary it must always pass. That test is what actually proves the bump works end to end.
  • build-prebuilt.yml — dropped the two data/jieba_dict bundling blocks so our tarballs match upstream's SDK layout.
  • Both READMEs — jieba is documented as embedded; the override list is now per-field extra_params > ZVEC_JIEBA_DICT_DIR, plus a note that user_dict_path augments whichever dictionary was selected.

Breaking change for zvec-rust: two public functions and one builder method are removed, so the next release needs a minor bump (0.8.0), not a patch. Versions are deliberately left untouched here.

Validation

  • cargo fmt --all -- --check clean.
  • ZVEC_AUTO_BUILD=0 cargo clippy --workspace --all-targets -- -D warnings clean.
  • ZVEC_AUTO_BUILD=0 cargo test --workspace --lib: 119 + 8 + 0 passed (was 123 + 8 + 0; the four missing ones are the removed jieba tests).
  • ZVEC_AUTO_BUILD=0 cargo test -p zvec-rust-sys --test build_script: 10 passed.
  • Integration tests were not run locally on purpose: the local vendor/zvec/build library predates this bump and still exports the removed symbols, so it cannot exercise the embedded dictionary. ci.yml builds zvec_c_api from the bumped submodule on all five platforms and runs cargo test --workspace --tests, which includes jieba_fts_test — that is the real validation.
  • build-prebuilt.yml still parses to 8 targets; it is not triggered by PRs, so it is dispatched manually on this branch (link below).

Upstream zvec moved six commits ahead of 60956b1. 52b72a4 embeds
cppjieba's default dictionary and HMM model into libzvec_c_api as
LZ4-compressed arrays and deletes the four C-API functions that
configured an external dict directory:

  zvec_config_data_set_jieba_dict_dir
  zvec_config_data_get_jieba_dict_dir
  zvec_set_default_jieba_dict_dir
  zvec_get_default_jieba_dict_dir

GlobalConfig::ConfigData::jieba_dict_dir is gone too; the tokenizer now
resolves per-field extra_params["jieba_dict_dir"] > ZVEC_JIEBA_DICT_DIR
> the embedded default, and CMake no longer installs data/jieba_dict
into the SDK.

Drop the corresponding Rust surface: the -sys declarations and
build_time_jieba_dict_dir(), the dict discovery in zvec-sys/build.rs,
ConfigBuilder::jieba_dict_dir, set/get_default_jieba_dict_dir, the
auto-registration path and its dladdr/GetModuleHandleEx probe (plus the
libdl link it needed), the data/jieba_dict staging in zvec-rust-build,
and the dict bundling in build-prebuilt.yml. jieba_fts_test no longer
skips when no dict directory is found: Chinese FTS must now work with
zero configuration, which is what proves the bump.

BREAKING CHANGE: zvec-rust removes set_default_jieba_dict_dir,
get_default_jieba_dict_dir and ConfigBuilder::jieba_dict_dir. Use the
per-field extra_params jieba_dict_dir or the ZVEC_JIEBA_DICT_DIR
environment variable instead. The next release needs a minor bump.

The other five commits are RaBitQ 0.7 modular targets (the submodule
path moves to RaBitQ-Library-0.7.0), flat segment-locking batching,
refined internal IDs, IVF primary-key brute force for filtered queries,
and upstream's own Windows ARM64 wheel/SDK publishing, which matches
the prebuilt target added in #18.
@chinaux
chinaux merged commit 6949471 into main Oct 20, 2026
17 checks passed
@chinaux
chinaux deleted the chore/vendor-sync-embedded-jieba branch October 20, 2026 01:54
mrcs64 added a commit to mrcs64/zvec-rust that referenced this pull request Oct 26, 2026
Upstream removed build-script cppjieba dictionary discovery in zvec-ai#21 (the
dictionary is now embedded in libzvec_c_api). Resolve by dropping the
remaining dictionary handling from this branch: the strict-mode dictionary
paragraphs in README, the ZVEC_RUST_JIEBA_DICT_DIR registration, and the
test that covered resolve_jieba_dict_dir. Strict mode now constrains only
the native library, as the upstream design implies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant