Repository navigation
Feat/import export - #8
Merged
Merged
Conversation
egolearner
reviewed
Aug 28, 2026
egolearner
reviewed
Aug 28, 2026
egolearner
reviewed
Aug 28, 2026
egolearner
reviewed
Aug 28, 2026
Contributor
Author
- Export via zvec 0.7 iter_docs: GET /collections/{name}/documents:export
streams JSONL with constant memory (includeVector/outputFields/format;
non-finite values fail fast with 422).
- Batched import: POST /collections/{name}/documents:import with a per-row
report (mode replace|insert, onError abort|skip). SDK write-batch limits are
absorbed internally, so >1024-doc requests no longer fail with a 500.
- Snapshot mode: mode=snapshot streams manifest.json + documents.jsonl as a
.tar.gz; POST /collections:import rebuilds a whole collection from it —
schema pre-checked before any write, rolled back if the load fails midway.
- Document wire representation moved to storage/doc_repr.py: the primary key
rides the reserved chain id -> $id -> $$id... whenever a column occupies
"id", fixing silent key loss on such schemas.
- Open collection names are globally unique: create/open/import answer 409
COLLECTION_ALREADY_EXISTS with the blocking path; close/destroy release it.
- Error taxonomy: DOCUMENT_CONFLICT(409), INVALID_DOCUMENT(422),
MAINTENANCE_BLOCKED(409) while an export is open,
UNSUPPORTED_VECTOR_DATA_TYPE(422), precise 400s for corrupt sources.
Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
…e tables - Browse tab gains Import/Export dialogs: local JSONL import (write mode and error policy with row-level report), native streamed export download in data/snapshot modes (never fetch().blob()). - Sidebar "+" menu gains "Import Collection": rebuilds a whole collection from a snapshot package with an auto-prefilled target directory and optional rename; the in-collection import dialog stays data-file only (.jsonl/.ndjson). - Document tables across Browse/Query/Write follow the backend primary-key rules ($id when a column occupies id), including detail fetch and copies. - File picker lists files for imports; partial imports surface as warnings; full zh-CN coverage; api-client types regenerated. Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- docs/api.md: primary-key representation rules, write semantics (non-transactional internal batching), export/import/collection-import endpoints, fs listing files, maintenance blocked during exports. - CHANGELOG: Added / Changed / Fixed / Removed entries for the feature set. - gitignore runtime demo collections (./data) and local tool workspace (.qoder). Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- Catch zlib.error in all tar.gz guards: deflate corruption leaking as 500 now answers 400 IMPORT_MANIFEST_INVALID (pinned by a regression test that poisons the decompressor at the unwrapped advance-read). - Register the nine new error codes in the frontend error mapper and add zh/en copy, so import/export failures no longer render as 'Internal server error'. - Clean up the snapshot staging directory when packaging fails, not only when staging fails. - Propagate request-level manifest errors through the import row loop instead of digesting them as line-0 row failures; release the freshly opened collection handle when a name clash rejects open(); single-pass finite-value scan on the export hot path. - Drop dead code: DocumentsPanel and InsertDocumentDialog (no mount point; Write tab covers insert), plus 73 orphaned i18n keys per locale and stale ':restore' naming leftovers. - docs: correct the maintenance-blocked note (flush stays available during export, verified against zvec 0.7.0) and testing.md pointers. Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- Move import/export out of the Browse toolbar into a dedicated Import/Export tab (sibling of Overview/Browse/Query/Write). - Snapshot exports are pinned to full data: trimming options are disabled in snapshot mode, so every UI-built snapshot can be rebuilt via Import Collection; data-file exports keep free trimming for exchange into existing collections. - Collection import is all-or-nothing: row failures roll back the freshly created collection (422 with the first failing line). - abort has true prefix semantics: rows after the first failure are compensated away (insert-confirmed ids deleted). - Malformed lines count toward totalLines, so imported + failed == totalLines always holds. - includeFields=false exports vectors only; import mode labelled upsert (insert new ids / overwrite existing ones). CI fixes: - fs-list size assertion writes bytes (Windows CRLF); skip the chmod-based test on Windows (no POSIX permission bits). - E2E smoke picks the create action from the sidebar \"+\" menu. Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
- Register IMPORT/EXPORT_UNSUPPORTED_FORMAT in the frontend error mapper with zh/en copy (input errors no longer render as internal errors); regenerate the OpenAPI client (adds includeFields). - Precise abort/replace copy: same-batch rows after the first failure cannot be rolled back in replace mode. - CHANGELOG: dedicated Data tab entry + full error-code list. - Drop dead i18n keys; fix stale comment; keep the tour demo tabs in sync with the real 5-tab page; e2e expects 5 detail tabs. Signed-off-by: 琪霄 <qixiao.yyq@alibaba-inc.com>
YongqiYin
force-pushed
the
feat/import-export
branch
from
August 31, 2026 03:41
507a512 to
c93c117
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.






Import & Export: JSONL data exchange + collection snapshots
Adds data import/export to Zvec Studio, no engine changes required:
export reads through the iterator, import writes through the existing
insert/upsertprimitives.Features
GET /collections/{name}/documents:exportstreams thewhole collection as JSONL with constant memory (snapshot isolation; writes
stay available during export). Options:
includeVector,includeFields,outputFields,mode=data|snapshot. A non-finite value detected beforethe response starts answers
422instead of silently becomingnull;one further downstream aborts the transfer mid-download (an incomplete
file is a failed export).
POST /collections/{name}/documents:importstreams alocal JSONL file in batches (512/batch, SDK's 1024 write limit absorbed
internally) and returns a per-row report.
mode=replace|insert,onError=abort|skip; row failures stay in the200body. Underabort,rows after the first failure are not kept in
insertmode (compensatingdelete); malformed lines count toward
totalLines, soimported + failed == totalLinesholds in skip mode.mode=snapshotbundlesmanifest.json(full schema +export options) +
documents.jsonlinto a.tar.gz. Snapshots arefaithful copies: the export dialog pins snapshot exports to full data
(trimming options are data-mode only), so every UI-built snapshot can
rebuild a collection.
POST /collections:importrebuilds a wholecollection from a snapshot: schema pre-check before any write (
409withmismatch list), optional rename. All-or-nothing: a mid-stream error or
any row failure rolls the freshly created collection back and answers
422with the first failing line. Sidebar "+" menu → "ImportCollection"; target directory pre-filled next to the snapshot file.
(
?tab=data, a sibling of Overview/Browse/Query/Write); file picker vianew
includeFiles/extensionsoptions on/fs/list; zh/en i18n. Theimport mode is labelled upsert-style ("insert new ids, overwrite existing
ones").
Design notes (per review feedback)
ImportFormat/ExportFormatprotocols plus a name/extension registry;both endpoints carry a
formatparameter from day one (unknown valuesanswer
400listing supported formats). Adding CSV/Parquet later meansimplementing the protocol and registering — no pipeline changes.
idcolumn: zvec keeps the primary key besidecolumns, so when a column occupies
idthe primary key travels under thereserved
$idrow key (the engine rejects$in column names, so thiscannot collide). Without a clash, rows stay byte-identical to the existing
API representation. Enforced by a shared representation module
(
doc_repr.py) mirrored in the frontend; ambiguous bare-idrows arerejected with guidance instead of guessing.
Pre-existing defects fixed along the way
500(SDK write-batchlimit hit) — now batched internally, up to the documented 10,000.
idcolumn could not be opened(
500), and the flat representation silently dropped the real primarykey — both fixed by the
$idscheme.VECTOR_FP64was offered but rejected by the engine at create time —removed from the enum.
409with the blocking path (previously two same-named collections could be
open and every name-only API silently resolved to the first).
Docs & tests
docs/api.md(new endpoints,$idrules, error catalog) andCHANGELOG.mdupdated; all new error codes mapped in the frontend.(export → import → byte-identical), the 1024 boundary, iterator release
on every exit path, corrupt archives (truncated, non-gzip, deflate) →
400, snapshot pre-check/rollback, abort prefix semantics, andname-uniqueness invariants.
make verifygreen.