Repository navigation
Merged
Conversation
…r sent it A definition can carry the schema of a collection as JSON, the SchemaJson that ChromaCollection reads from a server, and the create or get-or-create request sends it as it is, with the fields the client does not know: a restore creates the collection with the indexes and settings of the one it saved. It is the whole schema, so Schema and Configuration stay null, and the embedding function that a generator declares is not added to it. On Chroma Cloud its keys go through the same quota of bytes as the keys of a ChromaCollectionSchema. Tested against Chroma 1.5.9 and VecturaDB 1.0.0-preview.9: a collection created with the schema of another reads back the same schema.
…tion The Retry-After of a failed answer was internal, for the write retries of the client. Public next to StatusCode, it lets a caller that retries on its own wait as long as the server asks, as VecturaDB does when it refuses a write to a SPANN collection with 429.
…s comparison with the server
ChromaBackup.BackupAsync saves collections of the database of a client through the API into a zip file:
- manifest.json: the format and its version, the server, the times, and for each collection its counts and the SHA-256
of its records;
- collections/0001/collection.json: name, metadata, configuration_json and schema as the server sent them, and the
dimension;
- collections/0001/records.jsonl: a record a line, with the names of the lines of Chroma Data Pipes (id, metadata,
embedding, text_chunk) and the uri. The embeddings are the numbers that read back as the same floats, and the metadata
keeps the JSON of Chroma: an integer stays 1, a float 1.0, a list a list, a sparse vector
{"#type":"sparse_vector",...}.
The records come with gets of limit and offset, which every Chroma has; the search of the copy command of the CLI of
VecturaDB is not in Chroma 1.5.9. As the copy command: up to 5 gets at a time, written in their order; a request goes
again while waiting can fix the error (429, 408, a 5xx but 501, no answer), from 0.5 s doubled to 60 s or as its
Retry-After says; the count again at the end. The pages go on until one is not full, so the records added meanwhile come
in too, and an id given again by a later page goes once. A count that changed, an id given again or another number of
records says that the server took writes during the backup: the result and the manifest tell it. The file is written
next to its path and renamed at the end, and an existing file stays.
ChromaBackup.VerifyAsync compares the collections of a file with the server, as the copy command compares a source with
its target: documents and URIs the same, embeddings within the tolerances of np.allclose, metadata exactly the same, the
schema without the keys that the records brought in, the dimension and the metadata of the collection. A file whose
records do not match the digest of its manifest throws ChromaBackupFormatException.
Tested against Chroma 1.5.9, also with the netstandard2.0 builds, and VecturaDB 1.0.0-preview.9, sparse vectors
included.
…ce into its target Every collection of the file that exists on the server is compared first, and nothing is written before. One that is the same stays as it is. One that is not follows ChromaRestoreExisting, for all of them or by name: - Fail, the default, throws before any write and says what differs; - Resume adds the missing records, and leaves the different ones, the ones only on the server, the schema and the metadata; - Force makes the collection the same: with another schema or dimension it is deleted and created again; otherwise the records only on the server go, the different ones are deleted and added again, the missing ones added, and the metadata set. The copy command upserts a different record; on Chroma 1.0.21 to 1.5.9 an upsert of a record that exists, with its embedding, can lose a record from the index (KD-49), so the restore deletes and adds. A new collection is created with the schema of the file as it is (SchemaJson) and its metadata; as in the copy command, the old settings hnsw:* that Chroma refuses next to a schema come at the end with the metadata, and a 409 to the creation reads the collection. A record without an embedding, which the server sent as null, cannot go back, and a different one stays as the server has it. A server that refuses sparse vectors, as Chroma 1.5.9 with "Sparse vector indexing is not enabled in local", gets an error that names the keys, and a new collection that got no record is deleted. At the end the count must be the one written. Tested against Chroma 1.5.9, also with the netstandard2.0 builds, and VecturaDB 1.0.0-preview.9, and from each one into the other with CHROMA_TEST_SECOND_IMAGE, sparse vectors from VecturaDB into Chroma 1.5.9 included.
…arison, and Skip ChromaRestoreOptions.ChooseExisting gets what the comparison found for each collection that exists and is not the same, after every collection is compared and before any write, as the copy command of the CLI asks its questions; it chooses for the collections that ExistingByCollection does not name. ChromaRestoreExisting.Skip leaves a collection as it is while the others are restored, as the copy command leaves a collection it was not told what to do with; the result says Skipped.
…e and verify A dotnet tool on .NET 10, rolling forward to later versions, with System.CommandLine 3.0.0: - chromadotnet backup --url <server> (--all | --collections <name> ...) --out <file>, with --batch and --concurrent; - chromadotnet restore --url <server> --in <file> [--collections <name> ...] [--resume | --force] [--yes]; - chromadotnet verify --url <server> --in <file> [--collections <name> ...]. --token, --tenant and --database, or CHROMA_API_KEY, CHROMA_TENANT and CHROMA_DATABASE as Chroma names them, so that the key stays out of the history of the shell; on Chroma Cloud without a tenant, the one of the API key; --distributed for distributed Chroma not on Chroma Cloud. As the copy command, a restore asks for each collection that exists and is not the same, after comparing all of them: resume, force or leave; with --resume or --force it asks again unless --yes; without a terminal and without --yes it leaves the collection as it is, says how to run it again, and exits with 1. A backup of a server that took writes meanwhile exits with 2, and so does a verification that finds a difference. With a server the tests do not try, other than Chroma 1.5.9, VecturaDB 1.0.0-preview.9 and Chroma Cloud, the command warns and goes on: distributed Chroma installed by yourself too. Tested against Chroma 1.5.9, and installed from its package: backup, restore and verify of a collection of VecturaDB 1.0.0-preview.9.
…ed across the two The tests of the backup, the restore and the command chromadotnet against a server run on Chroma 1.5.9 and later and on a server already running (VecturaDB, distributed Chroma, Chroma Cloud), the servers the backup is tested with; on the older ones of the matrix they are skipped with that reason. The job of VecturaDB starts Chroma 1.5.9 as a second server (CHROMA_TEST_SECOND_IMAGE), so that a backup of each one is restored into the other.
…rites holds docs/backup.md, linked from the README: a server that takes writes during the backup, which can leave records of different moments in the file, and how the result and the command tell it; the command chromadotnet and the library; what the file holds and what a backup cannot save through the API; a restore into a collection that exists. Tested with Chroma 1.5.9, VecturaDB and Chroma Cloud, which runs on distributed Chroma; distributed Chroma installed by yourself: not tested.
…do not cover On Chroma 0.4.10 to 1.0.0 of the matrix the tests of the backup, the restore and the command failed instead of being skipped: an Assume in a SetUp of NUnit fails the test, and the TearDown then deleted a path that the SetUp had not set. The SetUp now calls Assert.Ignore, and the TearDown deletes only a path that was set. With the condition forced to false, the 17 tests of the four fixtures are skipped and none fails.
ambergrove
approved these changes
Nov 15, 2026
Chroma 1.5.9 answers 1.0.0 to /version, as every Chroma 1.x does, so the command warned that the server it is tested with was not tested. It now warns only for Chroma 0.x, and says that the versions 1.x cannot be told apart.
…Net.Tool too The job compares the packages of the CI with the ones on nuget.org by their ids, which were only the client and the package of dependency injection: with the two new packages it would fail before the signatures and the assets.
… that runs without .NET -p:NativeCommand=true publishes the command with NativeAOT. The job native of the CI publishes it for linux-x64, linux-arm64, linux-musl-x64 (in the AOT image of the SDK on Alpine, as the actions of GitHub do not run there), osx-arm64, osx-x64 and win-x64, every warning an error; on Linux the file backs up, restores and verifies a collection of Chroma 1.5.9. release-assets.yml attaches the six files to the release, with SHA256SUMS and their Sigstore signatures. docs/backup.md says how to download and check them, and that the files for glibc need 2.34 or later. Tested on linux-x64: 10 MB, no warning, and backup, restore and verify in debian:stable-slim without .NET, from VecturaDB 1.0.0-preview.9 into Chroma 1.5.9 and back.
ambergrove
approved these changes
Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two new packages save collections of a database into a zip file through the API of Chroma, and write them back into the same server or another one.
ChromaDotNet.Client.Backup (netstandard2.0, net462, net10.0)
ChromaBackup.BackupAsyncwrites a zip file:manifest.json(format and version, server, times, counts and the SHA-256 of the records of each collection), and for each collectioncollection.json(name, metadata,configuration_jsonandschemaas the server sent them, dimension) andrecords.jsonl, a record a line withid,metadata,embedding,text_chunkanduri. The first four are the names of the lines of Chroma Data Pipes.1,1.0, lists, sparse vectors).limitandoffset, up to 5 at a time, written in their order, until a page is not full. A request is tried again on 429, 408, a 5xx other than 501, or no answer, from 0.5 s doubled to 60 s or as itsRetry-Aftersays.ChangedDuringBackupin the result and in the manifest.ChromaBackup.VerifyAsynccompares a file with the server: documents and URIs the same, embeddings within the tolerances ofnp.allclose, metadata exactly, the schema without the keys that the records brought in, the dimension and the metadata of the collection.ChromaBackup.RestoreAsynccompares every collection of the file that exists before it writes anything, as thecopycommand of the CLI of VecturaDB does. A collection that is not the same followsChromaRestoreExisting:Fail(the default),Resume,ForceorSkip, for all of them, by name, or fromChooseExisting.Forcedeletes and adds a different record instead of an upsert, because of KD-49. A new collection gets the schema of the file as the server sent it.ChromaDotNet.Tool, the command
chromadotnet(.NET 10, with System.CommandLine 3.0.0)backup,restoreandverify, with--url,--tokenorCHROMA_API_KEY,--tenant,--databaseand--distributed.--yesit leaves it as it is and exits with 1. A backup of a server that took writes, or a verification that finds a difference, exits with 2.SHA256SUMSand their Sigstore signatures.Additions to the public API of ChromaDotNet.Client
ChromaCollectionDefinition.SchemaJson(JsonElement?): the schema of a new collection as JSON, sent as it is, in place ofSchemaandConfiguration.ChromaException.RetryAfter(TimeSpan?): the delay of theRetry-Afterof the answer; null when the answer has none.CI: the job
nativepublishes the command for the six platforms, every warning an error, and on Linux backs up, restores and verifies a collection of Chroma 1.5.9 with the file.release-assets.ymlattaches the files, and compares the two new packages with nuget.org too. The tests of the backup run on Chroma 1.5.9 and later and on VecturaDB, and are skipped on the older versions of the matrix. The job of VecturaDB starts Chroma 1.5.9 as a second server, so that a backup of each one is restored into the other.Documentation: docs/backup.md, linked from the README.
Tested with the SDK 11.0.100, as the CI builds: every test passed against Chroma 1.5.9, also in a tenant and database of their own and with the netstandard2.0 builds, and against VecturaDB 1.0.0-preview.9. By hand against Chroma Cloud: the tests of the backup, the restore and the command, and a backup of Chroma Cloud restored into VecturaDB and Chroma 1.5.9, and back. Distributed Chroma installed by yourself: not tested.
The native command: linux-x64, linux-arm64 and linux-musl-x64 back up, restore and verify a collection of Chroma 1.5.9 in the job
native; win-x64, the file of that job, did the same by hand from VecturaDB 1.0.0-preview.9 into Chroma 1.5.9 and back; osx-arm64 and osx-x64 are only built and answer--version: not tested against a server.