Skip to content

Feat/import export - #8

Merged
egolearner merged 7 commits into
zvec-ai:mainfrom
YongqiYin:feat/import-export
Sep 3, 2026
Merged

egolearner merged 7 commits into
zvec-ai:mainfrom
YongqiYin:feat/import-export

Conversation

@YongqiYin

@YongqiYin YongqiYin commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Import & Export: JSONL data exchange + collection snapshots

Adds data import/export to Zvec Studio, no engine changes required:
export reads through the iterator, import writes through the existing
insert/upsert primitives.

Features

  • Data export — GET /collections/{name}/documents:export streams the
    whole collection as JSONL with constant memory (snapshot isolation; writes
    stay available during export). Options: includeVector, includeFields,
    outputFields, mode=data|snapshot. A non-finite value detected before
    the response starts answers 422 instead of silently becoming null;
    one further downstream aborts the transfer mid-download (an incomplete
    file is a failed export).
  • Data import — POST /collections/{name}/documents:import streams a
    local JSONL file in batches (512/batch, SDK's 1024 write limit absorbed
    internally) and returns a per-row report. mode=replace|insert,
    onError=abort|skip; row failures stay in the 200 body. Under abort,
    rows after the first failure are not kept in insert mode (compensating
    delete); malformed lines count toward totalLines, so
    imported + failed == totalLines holds in skip mode.
  • Snapshot mode — mode=snapshot bundles manifest.json (full schema +
    export options) + documents.jsonl into a .tar.gz. Snapshots are
    faithful copies: the export dialog pins snapshot exports to full data
    (trimming options are data-mode only), so every UI-built snapshot can
    rebuild a collection.
  • Collection import — POST /collections:import rebuilds a whole
    collection from a snapshot: schema pre-check before any write (409 with
    mismatch list), optional rename. All-or-nothing: a mid-stream error or
    any row failure rolls the freshly created collection back and answers
    422 with the first failing line. Sidebar "+" menu → "Import
    Collection"; target directory pre-filled next to the snapshot file.
  • UI — Import/Export live on a dedicated Import / Export tab
    (?tab=data, a sibling of Overview/Browse/Query/Write); file picker via
    new includeFiles/extensions options on /fs/list; zh/en i18n. The
    import mode is labelled upsert-style ("insert new ids, overwrite existing
    ones").

Design notes (per review feedback)

  • JSONL first, extensible by design: formats sit behind
    ImportFormat/ExportFormat protocols plus a name/extension registry;
    both endpoints carry a format parameter from day one (unknown values
    answer 400 listing supported formats). Adding CSV/Parquet later means
    implementing the protocol and registering — no pipeline changes.
  • Schemas may declare an id column: zvec keeps the primary key beside
    columns, so when a column occupies id the primary key travels under the
    reserved $id row key (the engine rejects $ in column names, so this
    cannot collide). Without a clash, rows stay byte-identical to the existing
    API representation. Enforced by a shared representation module
    (doc_repr.py) mirrored in the frontend; ambiguous bare-id rows are
    rejected with guidance instead of guessing.

Pre-existing defects fixed along the way

  • Insert/upsert/update of >1024 documents answered 500 (SDK write-batch
    limit hit) — now batched internally, up to the documented 10,000.
  • Collections created via the SDK with an id column could not be opened
    (500), and the flat representation silently dropped the real primary
    key — both fixed by the $id scheme.
  • VECTOR_FP64 was offered but rejected by the engine at create time —
    removed from the enum.
  • Open collections are now globally unique by name; a clash answers 409
    with the blocking path (previously two same-named collections could be
    open and every name-only API silently resolved to the first).

Docs & tests

  • docs/api.md (new endpoints, $id rules, error catalog) and
    CHANGELOG.md updated; all new error codes mapped in the frontend.
  • 13 new backend test files + frontend tests covering roundtrip fidelity
    (export → import → byte-identical), the 1024 boundary, iterator release
    on every exit path, corrupt archives (truncated, non-gzip, deflate) →
    400, snapshot pre-check/rollback, abort prefix semantics, and
    name-uniqueness invariants. make verify green.

Copilot AI lite review requested due to automatic review settings August 28, 2026 02:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@egolearner

Copy link
Copy Markdown
  1. 导入导出放browse页面有些奇怪,可以考虑单独tab或者放overview页面最下面。
  2. 导出时没勾选向量,再导入集合会报错。
image
  1. 无法只导出向量,不导出标量。
  2. "导出期间读写仍可用;建索引、优化等维护操作会在导出完成前被暂时拒绝。"位置不对,可以放最上面。
image

@egolearner

Copy link
Copy Markdown
image image

覆盖/replace无法体现不存在的id将被插入,建议直接用upsert。

Comment thread apps/backend/zvec_studio/routers/collections.py
Comment thread apps/backend/zvec_studio/storage/sdk.py
Comment thread apps/backend/zvec_studio/routers/documents.py
Comment thread apps/backend/zvec_studio/storage/sdk.py
@YongqiYin

Copy link
Copy Markdown
Contributor Author
  1. 导入导出放browse页面有些奇怪,可以考虑单独tab或者放overview页面最下面。
  2. 导出时没勾选向量,再导入集合会报错。
image 4. 无法只导出向量,不导出标量。 5. "导出期间读写仍可用;建索引、优化等维护操作会在导出完成前被暂时拒绝。"位置不对,可以放最上面。 image

1、设置为了单独tab
2、报错原因是导出的manifest包含原集合的所有列,现在设置为了“快照模式”导出集合副本,数据列裁剪等语义可通过导出文件实现
4&5、done

- Export via zvec 0.7 iter_docs: GET /collections/{name}/documents:export
  streams JSONL with constant memory (includeVector/outputFields/format;
  non-finite values fail fast with 422).
- Batched import: POST /collections/{name}/documents:import with a per-row
  report (mode replace|insert, onError abort|skip). SDK write-batch limits are
  absorbed internally, so >1024-doc requests no longer fail with a 500.
- Snapshot mode: mode=snapshot streams manifest.json + documents.jsonl as a
  .tar.gz; POST /collections:import rebuilds a whole collection from it —
  schema pre-checked before any write, rolled back if the load fails midway.
- Document wire representation moved to storage/doc_repr.py: the primary key
  rides the reserved chain id -> $id -> $$id... whenever a column occupies
  "id", fixing silent key loss on such schemas.
- Open collection names are globally unique: create/open/import answer 409
  COLLECTION_ALREADY_EXISTS with the blocking path; close/destroy release it.
- Error taxonomy: DOCUMENT_CONFLICT(409), INVALID_DOCUMENT(422),
  MAINTENANCE_BLOCKED(409) while an export is open,
  UNSUPPORTED_VECTOR_DATA_TYPE(422), precise 400s for corrupt sources.

Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
…e tables

- Browse tab gains Import/Export dialogs: local JSONL import (write mode and
  error policy with row-level report), native streamed export download in
  data/snapshot modes (never fetch().blob()).
- Sidebar "+" menu gains "Import Collection": rebuilds a whole collection from
  a snapshot package with an auto-prefilled target directory and optional
  rename; the in-collection import dialog stays data-file only (.jsonl/.ndjson).
- Document tables across Browse/Query/Write follow the backend primary-key
  rules ($id when a column occupies id), including detail fetch and copies.
- File picker lists files for imports; partial imports surface as warnings;
  full zh-CN coverage; api-client types regenerated.

Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- docs/api.md: primary-key representation rules, write semantics
  (non-transactional internal batching), export/import/collection-import
  endpoints, fs listing files, maintenance blocked during exports.
- CHANGELOG: Added / Changed / Fixed / Removed entries for the feature set.
- gitignore runtime demo collections (./data) and local tool workspace (.qoder).

Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- Catch zlib.error in all tar.gz guards: deflate corruption leaking as
  500 now answers 400 IMPORT_MANIFEST_INVALID (pinned by a regression
  test that poisons the decompressor at the unwrapped advance-read).
- Register the nine new error codes in the frontend error mapper and
  add zh/en copy, so import/export failures no longer render as
  'Internal server error'.
- Clean up the snapshot staging directory when packaging fails, not
  only when staging fails.
- Propagate request-level manifest errors through the import row loop
  instead of digesting them as line-0 row failures; release the
  freshly opened collection handle when a name clash rejects open();
  single-pass finite-value scan on the export hot path.
- Drop dead code: DocumentsPanel and InsertDocumentDialog (no mount
  point; Write tab covers insert), plus 73 orphaned i18n keys per
  locale and stale ':restore' naming leftovers.
- docs: correct the maintenance-blocked note (flush stays available
  during export, verified against zvec 0.7.0) and testing.md pointers.

Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- Move import/export out of the Browse toolbar into a dedicated
  Import/Export tab (sibling of Overview/Browse/Query/Write).
- Snapshot exports are pinned to full data: trimming options are
  disabled in snapshot mode, so every UI-built snapshot can be
  rebuilt via Import Collection; data-file exports keep free
  trimming for exchange into existing collections.
- Collection import is all-or-nothing: row failures roll back the
  freshly created collection (422 with the first failing line).
- abort has true prefix semantics: rows after the first failure
  are compensated away (insert-confirmed ids deleted).
- Malformed lines count toward totalLines, so
  imported + failed == totalLines always holds.
- includeFields=false exports vectors only; import mode labelled
  upsert (insert new ids / overwrite existing ones).

CI fixes:
- fs-list size assertion writes bytes (Windows CRLF); skip the
  chmod-based test on Windows (no POSIX permission bits).
- E2E smoke picks the create action from the sidebar \"+\" menu.

Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- Register IMPORT/EXPORT_UNSUPPORTED_FORMAT in the frontend error
  mapper with zh/en copy (input errors no longer render as internal
  errors); regenerate the OpenAPI client (adds includeFields).
- Precise abort/replace copy: same-batch rows after the first
  failure cannot be rolled back in replace mode.
- CHANGELOG: dedicated Data tab entry + full error-code list.
- Drop dead i18n keys; fix stale comment; keep the tour demo tabs
  in sync with the real 5-tab page; e2e expects 5 detail tabs.

Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
@egolearner
egolearner merged commit dbefa53 into zvec-ai:main Sep 3, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants