Feat/import export

CHANGELOG.md

@@ -9,7 +9,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+**Backend**
+- Document **export**: `GET /collections/{name}/documents:export` streams the
+  whole collection as a JSONL download via the Zvec 0.7 snapshot iterator
+  (constant memory, snapshot isolation; `includeVector` / `outputFields` /
+  `format` parameters). Non-finite values fail fast with
+  `422 EXPORT_NON_FINITE_VALUE`.
+- Document **import**: `POST /collections/{name}/documents:import` streams a
+  local JSONL file in batches with a per-row report (`mode`: replace/insert,
+  `onError`: abort/skip). Row failures stay in the `200` body.
+- **Snapshot mode**: `mode=snapshot` export bundles `manifest.json` +
+  `documents.jsonl` in a `.tar.gz`.
+- **Snapshot import**: `POST /collections:import` imports a whole
+  collection from a snapshot package (schema from the manifest + data load
+  in one pass, optional rename). It is a collection-level lifecycle
+  operation — a sibling of create/open — and is reached from the sidebar
+  "+" menu ("Import Collection"), not from the in-collection import dialog
+  (which accepts data files only). The target directory is pre-filled as a
+  sibling of the snapshot file and must not exist yet; if the data load fails
+  unexpectedly mid-stream, the freshly created collection is rolled back.
+- `GET /fs/list` gained `includeFiles` / `extensions` parameters and
+  `kind`/`size` entry fields for the import file picker.
+- New error codes: `DOCUMENT_CONFLICT` (409), `INVALID_DOCUMENT` (422),
+  `EXPORT_BLOCKED` (409), `EXPORT_NON_FINITE_VALUE` (422),
+  `MAINTENANCE_BLOCKED` (409), `UNSUPPORTED_VECTOR_DATA_TYPE` (422),
+  `IMPORT_*` request-level codes.
+
 **Frontend**
+- Import / Export dialogs on a dedicated collection Data tab (sibling of
+  Overview/Browse/Query/Write; file picker, import mode, error policy,
+  native streamed download) with full zh-CN coverage.
+- "Import Collection…" entry in the sidebar "+" menu (alongside Create
+  Collection): rebuilds a whole collection from a snapshot package, with
+  auto-prefilled target directory (sibling of the snapshot file) and
+  optional rename.
+- Collections whose schema declares an `id` column now render and write the
+  primary key under `$id` across Browse / Query / Write tabs.
 - Zvec 0.6 Group-By search controls and grouped result display.
 - FTS ASCII folding and language stemmer controls.
 - Optional random rotation for INT8/INT4 quantization.
@@ -28,6 +63,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 **Backend**
+- Zvec SDK runtime requirement raised to **0.7** (`iter_docs` is the export
+  read path).
+- `POST /documents`, `:upsert`, and `PATCH /documents` now write in internal
+  batches (up to 10,000 docs work); batching is not transactional — a later
+  failure leaves earlier chunks written (documented in `docs/api.md`).
+- Collections may declare columns named `id` / `_id`; the primary key moves
+  to the reserved `$id` row key in all document representations.
+- **Open collection names are now globally unique.** Creating, opening, or
+  importing a collection whose name matches one already open answers
+  `409` with the blocking path in `detail`; previously two same-named
+  collections could be open at once and every name-only API silently
+  resolved to whichever registered first. Destroying or closing frees the
+  name.
 - Upgraded the Zvec SDK runtime to 0.6.x and added Group-By search, expanded
   FTS token filters, and `QuantizerParam` adaptation.
 
@@ -48,6 +96,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fieldset legend sizing and management bar spacing polished in
   OverviewTab and CreateCollectionDialog.
 
+### Fixed
+
+- Insert/upsert/update of more than 1024 documents no longer answers
+  `500 INTERNAL_ERROR` (the SDK write-batch limit is absorbed internally).
+- Opening an SDK-created collection with an `id` column no longer fails with
+  `500`; the primary key is no longer silently overwritten by that column.
+- Corrupted or non-gzip `.tar.gz` import sources answer `400` instead of
+  `500`; maintenance during an export answers `409 MAINTENANCE_BLOCKED`
+  instead of an opaque error; a collection with an unsupported vector dtype
+  reports `422 UNSUPPORTED_VECTOR_DATA_TYPE` instead of a misleading 404.
+- An explicit `"id": null` on insert/upsert auto-generates a ULID again
+  (matching the historical "omit id" contract).
+
+### Removed
+
+- `VECTOR_FP64` is no longer offered as a vector data type — the Zvec engine
+  rejects it at create time ("dense_vector's data type only support FP32").
+
 **Documentation**
 - All project documentation updated to reflect current feature set,
   accurate test counts, and correct tab/field names.

apps/backend/pyproject.toml

@@ -31,7 +31,10 @@ dependencies = [
     "orjson>=3.10,<4.0",
     "typing-extensions>=4.12",
     # Zvec embedded vector database SDK (core runtime dependency).
-    "zvec>=0.6,<0.7",
+    # 0.7 is the floor: ``Collection.iter_docs()`` (streaming full-collection
+    # traversal) only exists from 0.7 onwards and the export feature is built
+    # on it.
+    "zvec>=0.7,<0.8",
     # Required by zvec for numpy-backed vector payloads.
     "numpy>=1.23,<2.5",
 ]

apps/backend/tests/integration/test_collections_router.py

@@ -46,6 +46,67 @@ async def _create(client: AsyncClient, path: Path, name: str) -> dict:
     return resp.json()
 
 
+class TestNameUniqueness:
+    """Open collection names are globally unique.
+
+    Nearly every document API resolves a collection by name alone, so two
+    same-named collections can never be open at once — otherwise requests
+    would silently resolve to whichever registered first (observed in the
+    wild as "deleted in A, B's browse changed too").
+    """
+
+    async def test_create_same_name_different_path_is_409(
+        self, client: AsyncClient, tmp_path: Path
+    ) -> None:
+        await _create(client, tmp_path / "first", "demo")
+
+        resp = await client.post(
+            f"{API}/collections",
+            json={"path": str(tmp_path / "second"), "schema": _payload("demo")},
+        )
+
+        assert resp.status_code == 409, resp.text
+        body = resp.json()
+        assert body["code"] == "COLLECTION_ALREADY_EXISTS"
+        # The message points at the blocking collection and says what to do.
+        assert "demo" in body["detail"]
+        assert str(tmp_path / "first") in body["detail"]
+
+    async def test_open_same_name_different_path_is_409(
+        self, client: AsyncClient, tmp_path: Path
+    ) -> None:
+        await _create(client, tmp_path / "first", "demo")
+        # A second on-disk collection with the same recorded name.
+        import zvec
+
+        other = tmp_path / "second"
+        zvec.create_and_open(
+            path=str(other),
+            schema=zvec.CollectionSchema(
+                name="demo",
+                vectors=[zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4)],
+                fields=[zvec.FieldSchema("title", zvec.DataType.STRING)],
+            ),
+        ).flush()
+
+        resp = await client.post(f"{API}/collections/open", json={"path": str(other)})
+
+        assert resp.status_code == 409, resp.text
+        assert resp.json()["code"] == "COLLECTION_ALREADY_EXISTS"
+
+    async def test_destroy_frees_the_name(
+        self, client: AsyncClient, tmp_path: Path
+    ) -> None:
+        await _create(client, tmp_path / "first", "demo")
+        await client.delete(f"{API}/collections/demo?path={tmp_path / 'first'}")
+
+        second = await client.post(
+            f"{API}/collections",
+            json={"path": str(tmp_path / "second"), "schema": _payload("demo")},
+        )
+        assert second.status_code == 201, second.text
+
+
 class TestCreate:
     async def test_create_returns_201_with_summary(
         self, client: AsyncClient, tmp_path: Path
@@ -60,25 +121,6 @@ async def test_create_returns_201_with_summary(
         assert body["schema"]["vectors"][0]["indexParam"]["metric"] == "COSINE"
         assert body["stats"]["documentCount"] == 0
 
-    async def test_duplicate_name_different_path_both_open(
-        self, client: AsyncClient, tmp_path: Path
-    ) -> None:
-        await _create(client, tmp_path / "a", "same")
-        resp = await client.post(
-            f"{API}/collections",
-            json={"path": str(tmp_path / "b"), "schema": _payload("same")},
-        )
-        assert resp.status_code == 201
-        body = resp.json()
-        assert body["name"] == "same"
-        assert body["path"] == str(tmp_path / "b")
-        # Both should be listed as open
-        list_resp = await client.get(f"{API}/collections")
-        items = list_resp.json()["items"]
-        paths = {i["path"] for i in items}
-        assert str(tmp_path / "a") in paths
-        assert str(tmp_path / "b") in paths
-
     async def test_duplicate_path_returns_409(
         self, client: AsyncClient, tmp_path: Path
     ) -> None: