blog: announce Zvec v0.8.0 in Chinese and English

content/blog/en/2026-10-20-zvec-release.mdx

@@ -0,0 +1,139 @@
+---
+title: "Announcing Zvec v0.8.0"
+date: "2026-10-20"
+image: /img/blog/2026-10-20-zvec-release/header.png
+description: "🎉 Faster queries, lower index build costs, and broader platform support"
+---
+
+> **TL;DR:** Zvec v0.8.0 focuses on **query performance, index build costs, and cross-platform support**. This release extensively refactors quantizers and distance computation, with particular gains for INT4 kernels and Linux ARM64; reduces peak memory during IVF and DiskANN builds and merges; accelerates HNSW RaBitQ construction; adds Intel Mac and Windows ARM64 support; and makes Chinese full-text search ready to use with embedded Jieba resources.
+
+You can find the complete [release notes on GitHub](https://github.com/alibaba/zvec/releases/tag/v0.8.0).
+
+---
+
+## Platform Support
+
+### Intel Mac and Windows ARM64
+
+v0.8.0 adds **macOS x86_64** and **Windows ARM64** build support. Intel Mac users get matching builds, wheels, and SDKs. Windows ARM64 builds use MSVC, with release workflows for native Python 3.11–3.14 wheels and ARM64 SDK packages. See the [platform support documentation](https://zvec.org/en/docs/db/platforms/) for the full compatibility matrix.
+
+### DiskANN on Windows and mobile
+
+DiskANN now supports three additional platforms:
+
+| Platform | New capability |
+| --- | --- |
+| Windows | Asynchronous, unbuffered reads through I/O completion ports |
+| Android | Portable disk reads using `pread` |
+| iOS | Portable disk reads using `pread` |
+
+Applications that need to keep indexes on disk can now use DiskANN on Windows desktops and mobile devices.
+
+### RaBitQ on Windows x86_64
+
+RaBitQ indexes now support Windows x86_64. This release also upgrades RaBitQ-Library to v0.7.0 and switches to the upstream modular core/FastScan build targets.
+
+---
+
+## Chinese Full-Text Search and Usability
+
+### Embedded Jieba dictionaries
+
+Chinese full-text search no longer requires deploying the default dictionary separately or registering its path when importing the Python module. v0.8.0 embeds the **default Jieba dictionary and HMM model** as LZ4-compressed resources in the native library, making them available directly from Python wheels and native SDKs.
+
+Parsed defaults are shared, while user dictionaries remain isolated between tokenizer instances. To add product names, business terms, or other domain vocabulary, use `user_dict_path` to supplement the default dictionary without replacing it.
+
+### UTF-8 document IDs
+
+Document IDs and collection names now have more flexible naming rules:
+
+- **Document IDs:** Valid UTF-8, from 1 to 1,024 bytes.
+- **Collection names:** Valid UTF-8, from 1 to 256 bytes.
+- **Field names:** The existing ASCII character set, with the length limit extended to 64 bytes.
+
+Document IDs and collection names still exclude control characters and certain separators. All lengths above are measured in bytes.
+
+---
+
+## Index Construction and Memory
+
+### Lower build memory for IVF and DiskANN
+
+An index fitting in memory does not mean its construction will fit within the same budget. Temporary copies created during training, vector conversion, encoding, and segment merging can all push peak memory higher.
+
+v0.8.0 reduces redundant vector copies in IVF and DiskANN builds, streams merged inputs, bounds temporary encoding batches, and releases intermediate state earlier after construction completes.
+
+These changes reduce **peak memory during index construction and merging**, making builds and Optimize easier to complete in memory-constrained environments.
+
+### Faster HNSW RaBitQ construction
+
+HNSW RaBitQ repeatedly accesses original vectors during neighbor pruning. This release reuses vectors that have already been fetched and batches Flat provider lookups and segment locking, reducing repeated reads and lock contention during graph construction.
+
+Builders also reuse the Optimize thread pool. Nested RaBitQ and DiskANN training respects the configured concurrency, reducing cases where internal training tasks exceed the thread budget.
+
+### Shared page caching across index types
+
+This release introduces buffer-backed storage and integrates **HNSW, IVF, and DiskANN with a shared page cache**. Indexes can access data through batched page reads and pinned pages, while DiskANN can fall back to direct I/O under cache pressure.
+
+Underlying storage reads now use `MemoryBlock`, tying the lifetime of the data to the memory block that owns it. This provides a stronger foundation for cache reuse, batched reads, and concurrent access.
+
+---
+
+## Query Performance
+
+### Turbo: gains across quantization formats and distance metrics
+
+Vector search repeatedly computes distances between a query and candidate vectors, so the efficiency of the underlying kernels directly affects query performance. v0.8.0 extensively refactors quantizers and distance computation, migrating the relevant Flat, HNSW, IVF, and DiskANN paths to the Turbo framework.
+
+The update covers multiple quantization formats and distance metrics, and expands AVX2, AVX512, AVX512-FP16, SSE, and ARM NEON kernels and runtime dispatch. **INT4 kernels and Linux ARM64 workloads benefit substantially from this refactoring**, making distance computation over compressed vectors more efficient and improving search on ARM servers. Performance improves by up to **131.8%** in the tested configuration: GIST 1M, L2 distance, INT8 quantization, HNSW, and a 16-core Linux ARM64 system.
+
+### Uniform quantization: less per-point score reconstruction
+
+This release adds **Uniform UINT4** and integrates Uniform UINT4, UINT7, and UINT8 quantizers with the database layer and HNSW.
+
+Uniform quantization uses a shared min/max range across all data points. Queries no longer need to reconstruct scores separately for each point, reducing computation and accelerating search.
+
+There is a recall trade-off: sharing one quantization range can increase quantization error for some data points and lower recall. Evaluate the balance between query speed and recall against your own data distribution when choosing Uniform quantization.
+
+### PQ INT4, FastScan, and OPQ
+
+Turbo gains more quantization options:
+
+- **PQ INT4:** Represents quantized subspace values with 4-bit codes, providing a more compact PQ storage option.
+- **PQ FastScan:** Accelerates 4-bit PQ distance computation with packed block scanning, with both SIMD and scalar implementations.
+- **OPQ preprocessing:** Rotates vectors before PQ quantization to improve their distribution across subspaces, offering more room to tune compression and recall.
+
+These capabilities are part of the Turbo framework. This release also consolidates shared training and serialization logic across PQ quantizers.
+
+### Less refinement and result conversion overhead
+
+Refinement typically recomputes distances for candidates from the coarse search. Alongside faster computation kernels, this release reduces candidate handoff, export, and ID lookup overhead.
+
+The new `Collection.query_internal_ids()` API returns internal IDs directly for read-only collections, reducing ID lookup costs in the refinement path. **HNSW and Vamana can refine with FP16 or UINT8 vectors**, while `scale_factor` controls the candidate count to balance refinement cost and recall.
+
+### Vamana graph layout and prefetching
+
+Vamana updates its graph and vector memory layout, optimizes search state and prefetching, and corrects entry-point selection. Two-pass graph construction is now enabled by default, using a second construction pass to improve graph quality.
+
+---
+
+## Stability and Correctness
+
+This release fixes several issues involving concurrency, index maintenance, and filtering:
+
+- **Concurrent reads:** Fixed crashes and missing data when `fetch()` or `query()` races with a segment switch or flush, and corrected lifetime handling for the Flat cache and key-map iterators.
+- **Index rebuilding:** Preserved vector-to-row mappings when rebuilding indexes that contain deleted records.
+- **Training and quantization:** Propagated IVF `n_iters` correctly, fixed Turbo source decoding during merge training, and corrected INT4 scoring metadata.
+- **Filtered queries:** Fixed deep OR expressions, overlapping `LIKE` prefix/suffix matches, inverted-index suffix matching, and empty-array queries; added primary-key brute-force search for filtered IVF queries.
+- **Floating-point range bounds:** Preserved exact FLOAT and DOUBLE bounds when the optimizer converts an inverted-index condition into a forward filter, ensuring exclusive comparisons do not incorrectly include values equal to the bound.
+- **Full-text index persistence:** Persisted sealed full-text postings before dropping auxiliary column families, and hardened WAL recovery, schema handling, and decoding.
+
+---
+
+## Upgrade Notes
+
+**Global Jieba dictionary configuration APIs have been removed.** If you use `zvec.init(jieba_dict_dir=...)`, `set_default_jieba_dict_dir()`, or `get_default_jieba_dict_dir()`, update those calls when upgrading. The corresponding C configuration accessors, C default-directory APIs, and C++ global configuration field and registration APIs have also been removed.
+
+If you use the embedded defaults, simply remove those settings. To override the main dictionary, use the field-level `FtsIndexParam.extra_params.jieba_dict_dir` or the `ZVEC_JIEBA_DICT_DIR` environment variable; field-level configuration takes precedence. To supplement the dictionary with domain vocabulary, use `user_dict_path`.
+
+For the full changelog and related PRs, see the [v0.8.0 release notes](https://github.com/alibaba/zvec/releases/tag/v0.8.0).

content/blog/zh/2026-10-20-zvec-release.mdx

@@ -0,0 +1,139 @@
+---
+title: "Zvec v0.8.0 正式发布"
+date: "2026-10-20"
+image: /img/blog/2026-10-20-zvec-release/header.png
+description: "🎉 查询提速、降低建图成本、扩展平台支持"
+---
+
+> **版本亮点：** Zvec v0.8.0 聚焦**查询性能、索引构建成本和跨平台体验**。本次更新大范围重构了量化器与距离计算，重点优化 INT4 内核和 Linux ARM64 性能；降低 IVF、DiskANN 建图与合并的峰值内存，加速 HNSW RaBitQ 构建；新增 Intel Mac、Windows ARM64 支持，并让中文全文检索通过内置 Jieba 资源开箱即用。
+
+完整变更请查看 [GitHub Release Notes](https://github.com/alibaba/zvec/releases/tag/v0.8.0)。
+
+---
+
+## 平台支持
+
+### Intel Mac 和 Windows ARM64
+
+v0.8.0 新增 **macOS x86_64** 与 **Windows ARM64** 构建支持。Intel Mac 用户获得对应的构建、wheel 和 SDK；Windows ARM64 使用 MSVC 构建，并增加原生 Python 3.11–3.14 wheel 和 ARM64 SDK 发布流程。详细的平台支持说明和矩阵请参见 [链接](https://zvec.org/zh/docs/db/platforms/)。
+
+### DiskANN 扩展到 Windows 和移动端
+
+DiskANN 本次新增三个平台方向：
+
+| 平台    | 新增能力                            |
+| ------- | ----------------------------------- |
+| Windows | 基于 I/O 完成端口的异步、无缓冲读取 |
+| Android | 基于可移植 `pread` 的磁盘读取       |
+| iOS     | 基于可移植 `pread` 的磁盘读取       |
+
+这让需要将索引放在磁盘上的应用，在 Windows 桌面和移动端也有了 DiskANN 选择。
+
+### RaBitQ 新增 Windows x86_64 支持
+
+RaBitQ 索引现已适配 Windows x86_64 环境。同时，本次更新将 RaBitQ-Library 升至 v0.7.0 版本，构建目标也切换为上游模块化的 core/FastScan。
+
+---
+
+## 中文全文检索与易用性
+
+### Jieba 词典内置，中文分词开箱即用
+
+中文全文检索不再需要单独部署默认词典或在导入 Python 模块时注册词典路径。v0.8.0 将 **Jieba 默认词典和 HMM 模型**以 LZ4 压缩资源的形式内置到原生库中，Python wheel 与原生 SDK 都可以直接使用。
+
+默认词典解析后共享，用户词典则在分词器实例之间保持隔离。如果需要加入产品名、业务术语或专有名词，可以通过 `user_dict_path` 补充，不必替换默认词典。
+
+### 文档 ID 支持 UTF-8
+
+文档 ID 和集合名的命名规则更加灵活：
+
+- **文档 ID**：支持有效 UTF-8，长度为 1–1,024 字节。
+- **集合名**：支持有效 UTF-8，长度为 1–256 字节。
+- **字段名**：保留原有 ASCII 字符集，长度上限扩展到 64 字节。
+
+文档 ID 和集合名仍会排除控制字符及指定分隔符；以上长度均以字节计算。
+
+---
+
+## 索引构建与内存
+
+### IVF 和 DiskANN 建图更省内存
+
+索引最终能放进内存，并不意味着建图过程也能承受同样的内存预算。训练、向量转换、编码和多段合并中的临时副本，都可能推高峰值内存。
+
+v0.8.0 针对 IVF 和 DiskANN 的构建流程，减少了重复向量拷贝，使用流式方式读取合并输入，限制临时编码批次的大小，并在构建完成后更早释放中间状态。
+
+这些改动降低了**建图和合并时的峰值内存**，让内存受限环境更容易完成索引构建和 Optimize。
+
+### HNSW RaBitQ 建图更快
+
+HNSW RaBitQ 在邻居筛选过程中会多次访问原始向量。本版本复用已经读取的向量，并批量执行 Flat 提供方的查找与分段加锁，减少重复读取和锁竞争，加速图构建。
+
+同时，构建器复用 Optimize 线程池，RaBitQ、DiskANN 的嵌套训练也会遵守配置的并发数，减少内部训练任务超出线程预算的情况。
+
+### 多种索引接入共享页缓存
+
+本版本引入了缓冲存储，并将 **HNSW、IVF 和 DiskANN 接入共享页缓存**。索引可以通过批量页读取和页面固定来访问数据，DiskANN 在缓存压力下也可以回退到直接 I/O。
+
+底层存储读取统一迁移到 `MemoryBlock`，使读取数据的生命周期与持有它的内存块保持一致。这为缓存复用、批量读取和并发访问提供了更稳固的基础。
+
+---
+
+## 查询性能
+
+### Turbo：多种量化格式和距离度量共同受益
+
+向量检索需要反复计算查询向量与候选向量之间的距离，底层计算内核的效率会直接影响查询性能。v0.8.0 对量化器和距离计算做了大范围重构，将 Flat、HNSW、IVF 和 DiskANN 的相关计算路径迁移到 Turbo 框架。
+
+这次更新覆盖多种量化格式和距离度量，并扩展了 AVX2、AVX512、AVX512-FP16、SSE 和 ARM NEON 内核及运行时分派。**INT4 内核和 Linux ARM64 场景大幅度受益于本次重构**，让压缩向量的距离计算更高效，也让 ARM 服务器上的检索更快，部分场景性能最高提升 **131.8%**（此处以 GIST 1M、L2 距离、INT8 量化、HNSW 索引、16 核 Linux ARM64 为例）。
+
+### Uniform 量化：减少逐点分数反算
+
+本版本新增 **Uniform UINT4**，并将 Uniform UINT4、UINT7、UINT8 量化器接入数据库层和 HNSW。
+
+Uniform 量化对所有数据点使用统一的 Min/Max 范围，查询时不再需要逐点反算分数，从而减少计算开销、加速检索。
+
+这一选择也有召回率上的权衡：所有点共享一个量化范围，可能使部分数据的量化误差增大，进而降低 Recall。选择 Uniform 量化时，可以结合自己的数据分布，评估查询速度与召回率的取舍。
+
+### PQ INT4、FastScan 与 OPQ
+
+Turbo 的量化能力也在继续扩展：
+
+- **PQ INT4**：以 4 bit 编码表示子空间中的量化结果，提供更紧凑的 PQ 存储选择。
+- **PQ FastScan**：通过打包后的分块扫描加速 4 bit PQ 距离计算，并提供 SIMD 与标量实现。
+- **OPQ 预处理**：在 PQ 量化前对向量进行旋转，改善向量在不同子空间中的分布，为压缩与召回率提供更多调优空间。
+
+这些能力位于 Turbo 框架中；本次更新还统一了多个 PQ 量化器的公共训练与序列化逻辑。
+
+### 减少精排和结果转换开销
+
+精排通常需要对粗排候选再次计算距离。除了优化计算内核，本版本还减少了候选传递、导出以及 ID 反查的开销。
+
+新增的 `Collection.query_internal_ids()` 面向只读集合，直接返回内部 ID，减少精排链路中的 ID 反查成本。**HNSW 和 Vamana 可以使用 FP16 或 UINT8 向量进行精排**，并通过 `scale_factor` 控制候选数量，在精排成本与召回率之间调整。
+
+### Vamana 图布局与预取优化
+
+Vamana 更新了图和向量的内存布局，优化搜索状态与预取，并修正了入口点选择。两遍建图现在默认开启，通过第二遍构建改善图质量。
+
+---
+
+## 稳定性与正确性
+
+本版本修复了多项与并发、索引维护和过滤相关的问题：
+
+- **并发读取**：修复 `fetch()`、`query()` 与分段切换或 Flush 并发时的崩溃、遗漏数据问题，并修正 Flat 缓存与键映射迭代器的生命周期。
+- **索引重建**：包含已删除记录的索引，在重建时正确保留向量与行号之间的对应关系。
+- **训练与量化**：正确传递 IVF 的 `n_iters`，修复合并训练中 Turbo 数据源的解码，以及 INT4 评分元数据的取值。
+- **过滤查询**：修复深层 OR 表达式、`LIKE` 前后缀重叠、倒排索引后缀匹配和空数组查询；支持过滤后的 IVF 主键暴力检索。
+- **浮点范围边界**：在优化器将倒排条件转为正排过滤时保留 FLOAT、DOUBLE 的精确边界，确保严格大于、小于等比较不会错误纳入边界值。
+- **全文索引持久化**：在删除辅助列族前持久化已封存的全文倒排数据，并加强 WAL 恢复、Schema 处理与解码。
+
+---
+
+## 升级说明
+
+**Jieba 全局词典配置接口已移除。** 如果使用过 `zvec.init(jieba_dict_dir=...)`、`set_default_jieba_dict_dir()` 或 `get_default_jieba_dict_dir()`，升级时需要调整调用。对应的 C 配置访问器、C 默认词典目录接口，以及 C++ 全局配置字段和注册接口也已移除。
+
+使用内置默认词典时，直接删除这些配置即可。需要自定义主词典时，可以使用字段级的 `FtsIndexParam.extra_params.jieba_dict_dir` 或环境变量 `ZVEC_JIEBA_DICT_DIR`，字段级配置优先；需要补充业务词汇时，使用 `user_dict_path`。
+
+完整变更与关联 PR 可查看 [v0.8.0 Release Notes](https://github.com/alibaba/zvec/releases/tag/v0.8.0)。