Skip to content

feat: add the zarr-indexing package (TensorStore-style index transforms, ndsel wire format) - #249

Open
d-v-b wants to merge 13 commits into
mainfrom
feat/zarr-indexing-package
Open

feat: add the zarr-indexing package (TensorStore-style index transforms, ndsel wire format)#249
d-v-b wants to merge 13 commits into
mainfrom
feat/zarr-indexing-package

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 28, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Adds zarr-indexing as a standalone uv-workspace package (packages/zarr-indexing, import name zarr_indexing), without wiring it into zarr's runtime — per the land-the-package-first sequencing. The runtime integration (lazy .lazy indexing, transform-routed selection I/O) stays on refactor/simplify-indexing (zarr-developers#3906) and will gain the zarr-indexing>=0.1.0 dependency only after 0.1.0 is on PyPI.

What the package is

  • TensorStore-style index-transform algebra: IndexTransform / IndexDomain / three output-map kinds, full-input-rank index arrays with dependency-axis-derived correlation (oindex outer-product vs vindex broadcast), composition, literal-coordinate slice semantics verified against tensorstore 0.1.84.
  • Chunk resolution: maps a composed transform to per-chunk selections against a DimensionGridLike protocol (public-name-only contract; zarr's per-dimension grids satisfy it structurally, no zarr import). Includes the touched-chunks-only enumeration and the sorted-1-D fast path from perf(zarr-transforms): partition sorted 1-D array maps by chunk #234.
  • ndsel-conformant wire format: parse_ndsel / normalize_ndsel implement the ndsel draft spec (all five message kinds, canonical normalize, full error taxonomy), validated against the byte-vendored conformance corpus plus a tensorstore IndexTransform(json=...) round-trip cross-check.

Contents

  • packages/zarr-indexing/ — sources, tests (273, all green against main's zarr), vendored ndsel corpus with provenance, own towncrier changelog.
  • .github/workflows/zarr-indexing.yml — pytest (3.12–3.14) / ruff / pyright, paths-filtered; test job uses uv sync --all-packages since zarr does not depend on the package here.
  • .github/workflows/zarr-indexing-release.yml — tag-triggered (zarr_indexing-v*) TestPyPI/PyPI publish with attestation and an isolated-wheel smoke test; byte-identical to the zarr-metadata release flow modulo the package name. A local dress rehearsal of the tagged build produced exactly 0.1.0 and the wheel imports with no zarr installed.
  • check_changelogs.yml — covers the package's changes/ dir.
  • Root pyproject.toml — uv workspace membership only (no runtime dependency).

Sequencing

  1. Land this PR.
  2. Configure PyPI trusted publishing for zarr-indexing → push zarr_indexing-v0.1.0.
  3. feat: add IndexTransform library for composable, lazy coordinate mappings zarr-developers/zarr-python#3906 then adds the runtime dependency and the integration layer (its packages/ tree converges with this one on merge).

History note: files were developed on the zarr-developers#3906 branch (with per-commit review); git log --follow traces their evolution there.

@d-v-b d-v-b mentioned this pull request Jul 28, 2026
Open
7 tasks
d-v-b added 7 commits July 30, 2026 11:08
@d-v-b
fix: minor correctness and hygiene fixes from the sync-pipeline audit (…
6f9724c 
…zarr-developers#4205)

- Codec construction warnings (e.g. sharding's "disables partial reads")
  fired twice per array open, and on every decode/encode through the fused
  pipeline's async fallback. Re-constructions of an already-validated codec
  chain now go through codecs_from_list_unchecked, which validates
  structure without repeating first-construction advisory warnings; each
  warning fires exactly once per open under both pipelines.
- concurrent_iter returned a lazy generator while its docstring promised
  eagerly scheduled tasks; it now materializes the task list so awaiting
  one at a time cannot serialize the batch.
- A garbage codec_pipeline.max_workers value (e.g. from the environment)
  raised ValueError mid-read; it now warns and falls back to the default,
  consistent with tolerant handling of config input.
- The as-completed pipeline helpers abandoned in-flight tasks when one
  failed, leaving stray background writes and "Task exception was never
  retrieved" warnings; failures now cancel and drain outstanding tasks.
- Benchmarks: seed the data generator for reproducibility; fix a
  copy-pasted docstring.
- Remove dead commented-out test blocks referencing the removed set_range
  API.

Assisted-by: ClaudeCode:claude-sonnet-5
@d-v-b
fix: byte-order handling for structured dtypes in the bytes codec (#220)
ff17c7e 
* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
feat: add the zarr-indexing package (TensorStore-style index transforms)
eb9a6e7 
Standalone workspace package extracted from the lazy-indexing branch
(zarr-developers#3906): composable, lazy coordinate transforms
(IndexTransform / IndexDomain / output maps), dependency-aware chunk
resolution against a DimensionGridLike protocol, and an ndsel-conformant
JSON wire format validated against the vendored conformance corpus.

zarr itself does not depend on zarr-indexing yet — the runtime wiring
lands separately once 0.1.0 is published. The package is numpy-only;
its tests exercise chunk resolution against zarr's concrete ChunkGrid,
so they run from the workspace root (uv sync --all-packages).

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
style: conventional submodule import in the chunk-resolution tests
9a744c3 
Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
perf(zarr-indexing): joint chunk enumeration for correlated vindex maps
fb8a2a7 
Candidate-chunk enumeration took the cartesian product of each correlated
ArrayMap's per-dimension distinct chunk ids and relied on intersect() to
filter untouched combinations. For a diagonal selection of P scattered
points that is P**2 intersect calls — quadratic in the number of selected
points, the same workload shape as zarr-developers#4174 (400 points:
~2.6s; 10k points: ~30min).

Group correlated maps jointly instead: broadcast their per-point chunk
ids, take the distinct rows (np.unique(axis=0), O(P log P)), and
enumerate exactly the touched combinations. Candidate slots now carry
chunk-coordinate tuples covering one or more output dimensions;
orthogonal/constant/slice dimensions keep their existing per-dimension
candidates. 400-point diagonal resolution drops from 2628ms to 14ms and
scales linearly.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
docs(zarr-indexing): standalone documentation site; add package justfile
3fd36cd 
Mirror the treatment zarr-metadata received in zarr-developers#4208/zarr-developers#4210 onto
zarr-indexing: a self-contained mkdocs site under the package (own
mkdocs.yml, landing page, ndsel wire-format guide, mkdocstrings page per
module, and .readthedocs.yaml for a dedicated RTD project), so the
package presents as a separate project with docs versioned by its own
zarr_indexing-v* release tags rather than zarr-python's. The zarr-python
site's API Reference nav links out to it, and each RTD project now skips
PR builds that do not touch its half of the repo.

The package gains a pinned docs dependency group, a docs build job in its
CI workflow, and a justfile with package-scoped dev recipes. Two recipes
deviate from the zarr-metadata original by design:

- `test` runs against the workspace-root environment (`uv run --project
  ../.. --all-packages --group test`), because the chunk-resolution tests
  exercise this package against zarr's chunk grids and `zarr` is
  deliberately not a dependency of this package.
- `typecheck` uses plain `pyright`, unpinned and on the default
  interpreter, mirroring this package's own CI invocation. The
  zarr-metadata pin exists for a PEP 661 sentinel regression that
  zarr-indexing's sources do not hit.

composition.py gains the module docstring the other modules already have,
since mkdocstrings renders it as the page introduction.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
chore: drop the already-released 4141 changelog fragment
733b3c3 
The bytes-codec byte-order fix this fragment describes shipped upstream
and its entry is already in docs/release-notes.md; the fragment survived
on this branch only as a rebase remnant, and would emit a duplicate entry
in the next release.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
d-v-b force-pushed the feat/zarr-indexing-package branch from d1e931f to 733b3c3 Compare July 30, 2026 11:41
d-v-b and others added 3 commits July 30, 2026 13:46
@d-v-b
docs(zarr-indexing): canonicalize ndsel references to zarr-developers…
c3ad000 
…/ndsel

Also aligns the zarr-indexing workflow's setup-uv pin (v8.3.2) with the
rest of the repo. The vendored-corpus sha is present upstream; the
historical d-v-b/ndsel#1 PR reference stays as provenance.

Assisted-by: ClaudeCode:claude-fable-5
@dependabot @d-v-b
chore(deps): bump the python-dependencies group across 1 directory wi…
ec8e70a 
…th 11 updates (zarr-developers#4216)

* chore(deps): bump the python-dependencies group across 1 directory with 11 updates

Bumps the python-dependencies group with 11 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [numpy](https://github.com/numpy/numpy) | `2.5.0` | `2.5.1` |
| [typer](https://github.com/fastapi/typer) | `0.26.8` | `0.27.0` |
| [coverage](https://github.com/coveragepy/coveragepy) | `7.14.3` | `7.15.2` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.155.7` | `6.160.0` |
| [tomlkit](https://github.com/python-poetry/tomlkit) | `0.15.0` | `0.15.1` |
| [uv](https://github.com/astral-sh/uv) | `0.11.26` | `0.11.31` |
| [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.6` | `9.7.7` |
| [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) | `1.0.4` | `1.0.6` |
| [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.1` | `1.12.3` |
| [ruff](https://github.com/astral-sh/ruff) | `0.15.20` | `0.15.22` |
| [mypy](https://github.com/python/mypy) | `2.1.0` | `2.3.0` |



Updates `numpy` from 2.5.0 to 2.5.1
- [Release notes](https://github.com/numpy/numpy/releases)
- [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst)
- [Commits](numpy/numpy@v2.5.0...v2.5.1)

Updates `typer` from 0.26.8 to 0.27.0
- [Release notes](https://github.com/fastapi/typer/releases)
- [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md)
- [Commits](fastapi/typer@0.26.8...0.27.0)

Updates `coverage` from 7.14.3 to 7.15.2
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.14.3...7.15.2)

Updates `hypothesis` from 6.155.7 to 6.160.0
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.155.7...v6.160.0)

Updates `tomlkit` from 0.15.0 to 0.15.1
- [Release notes](https://github.com/python-poetry/tomlkit/releases)
- [Changelog](https://github.com/python-poetry/tomlkit/blob/master/CHANGELOG.md)
- [Commits](python-poetry/tomlkit@0.15.0...0.15.1)

Updates `uv` from 0.11.26 to 0.11.31
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/0.11.31/CHANGELOG.md)
- [Commits](astral-sh/uv@0.11.26...0.11.31)

Updates `mkdocs-material[imaging]` from 9.7.6 to 9.7.7
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@9.7.6...9.7.7)

Updates `mkdocstrings` from 1.0.4 to 1.0.6
- [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases)
- [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/mkdocstrings@1.0.4...1.0.6)

Updates `markdown-exec[ansi]` from 1.12.1 to 1.12.3
- [Release notes](https://github.com/pawamoy/markdown-exec/releases)
- [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md)
- [Commits](pawamoy/markdown-exec@1.12.1...1.12.3)

Updates `ruff` from 0.15.20 to 0.15.22
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/0.15.22/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.15.20...0.15.22)

Updates `mypy` from 2.1.0 to 2.3.0
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](python/mypy@v2.1.0...v2.3.0)

---
updated-dependencies:
- dependency-name: coverage
  dependency-version: 7.15.2
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.160.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: markdown-exec[ansi]
  dependency-version: 1.12.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocs-material[imaging]
  dependency-version: 9.7.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocstrings
  dependency-version: 1.0.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mypy
  dependency-version: 2.3.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: numpy
  dependency-version: 2.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.15.22
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: tomlkit
  dependency-version: 0.15.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: typer
  dependency-version: 0.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.11.31
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: satisfy numpy 2.5.1 type stubs in indexing selection normalization

numpy 2.5.1 stubs infer np.asarray(<list>) as a float64 array, so mypy
now rejects the untyped asarray calls in replace_lists and
CoordinateIndexer. Pass dtype=np.intp where the integer dtype is
guaranteed, and cast to ArrayOfIntOrBool where the list contents are
only known at runtime.

Assisted-by: ClaudeCode:claude-fable-5

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
@d-v-b
Merge branch 'main' into feat/zarr-indexing-package
b0d574b
d-v-b added 3 commits July 30, 2026 14:28
@d-v-b
fix: reject malformed chunk keys in DefaultChunkKeyEncoding (zarr-dev…
57e66d9 
…elopers#4219)

* fix: reject malformed chunk keys in DefaultChunkKeyEncoding

decode_chunk_key stripped a single leading character and split the rest,
so any key at all decoded to something. "0/1" silently became (1,) -- the
"0" was eaten as if it were the "c" prefix -- and a key written with one
separator decoded wrongly under an encoding configured with the other.

Validate the "c<separator>" prefix and raise ValueError when it is absent,
so a key that is not a chunk key for this encoding is reported rather than
silently misread.

Adds the tests this method never had, covering the round trip for both
separators and each way a key can fail to carry the prefix.

Assisted-by: ClaudeCode:claude-fable-5

* Rename 250.bugfix.md to 4219.bugfix.md
@d-v-b
Merge branch 'main' into feat/zarr-indexing-package
7390206
@d-v-b
chore: drop the root uv-workspace wiring for zarr-indexing
7241e9a 
Per review: the root pyproject.toml should not change in this PR. The
package now operates fully standalone (like zarr-metadata); the test
invocations layer the package into the repo-root environment as an
editable overlay instead (python -m pytest, since a base-env console
script would not see the overlay).

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b d-v-b mentioned this pull request Jul 30, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b