Skip to content

NIfTI-2, NRRD and NWB decoders (0.9.0) - #9

Merged
aaronkanzer merged 3 commits into
mainfrom
feat/nifti2-nrrd
Oct 25, 2026
Merged

aaronkanzer merged 3 commits into
mainfrom
feat/nifti2-nrrd

Conversation

@aaronkanzer

@aaronkanzer aaronkanzer commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Three data formats surfaced by a pass over the NeuroSchema catalog (neuroschema.org/catalog), which the library did not read.

What

  • NIfTI-2: the 540-byte header (n+2 / ni2 magic, int64 dim, float64 pixdim) decodes through the same nifti decoder as NIfTI-1, in both byte orders, so an axis past 32767 voxels reads correctly. Both versions now report niftiVersion (1 or 2); every other field is unchanged.
  • NRRD: a new decoder for .nrrd and detached .nhdr headers (teem, ITK, 3D Slicer, the Allen CCFv3 mouse atlases). Sizes, dtype, encoding, endian, space, space directions and the per-axis spacing derived from them (or spacings), origin, units, kinds, labels, measurement frame, data file, key:=value pairs, and whether the header fit the 64 KiB read. The data is never touched, so gzip and bzip2 encodings are reported, not inflated.
  • NWB (.nwb, the DANDI archive format): an HDF5 file whose root carries nwb_version decodes as its own format: version, identifier, session description, start time and id, institution, lab, experimenters, keywords, publications, subject, devices, electrode groups and count, imaging planes, acquisition series (type, shape, dtype, unit, rate), processing modules and interfaces, unit count and columns, interval tables, stimuli, embedded schema namespaces and any ndx-* extension. Targeted lookups rather than a walk; every listing is taken before the per-series detail, and the detail passes stop before the read cap, so a file with dozens of series names them all and describes as many as the budget allows.

Two HDF5 reader changes came out of the NWB work and apply to every HDF5 flavour: identical reads and group listings are memoized within one decode (no byte is fetched that was not fetched before, but a lookup from the root no longer re-reads the root B-tree and heap), and the global heap scan widens in steps past its fixed 8 KiB window, bounded by the collection's own size. A pynwb session file keeps thousands of strings in one collection; two of the four DANDI test files read their subject as None before this.

Verification

  • tests/fixtures/nifti2-cases.json: NIfTI-2 headers written by nibabel 5.4.2, both byte orders and a 40000-voxel axis, with nibabel's own read recorded beside each expected decode.
  • tests/fixtures/nrrd-cases.json: the real header of the CCFv3 25 um annotation_25.nrrd (fetched from download.alleninstitute.org by Range request) plus three pynrrd 1.1.3 files. Every value checked against pynrrd's read_header.
  • tests/fixtures/hdf5-nwb-*.sparse.json: three DANDI files from three labs and writers (000127 Miller lab 2.4.0; 000004 Rutishauser lab 2.1.0; 000020 Allen patch-seq written by MIES, 2.2.5, ndx-mies, 38 acquisition and 38 stimulus series) and a pynwb 4.2.0 file. Every structural value was checked against h5py on the full downloads.
  • NIfTI-2 in the magic-seeded fuzz set.
  • 536 passed, 2 skipped with the optional libraries installed, and the same on a bare interpreter.

The NIfTI-2 and NRRD fixtures are copied into scigantic-edge in Scigantic/scigantic-edge#15, so the Go decoders are held to the same bytes. NWB is not ported there: edge has no generic HDF5 walker yet.

NIfTI-2: the 540-byte header (n+2 / ni2 magic, int64 dim, float64 pixdim)
decodes through the same nifti decoder as NIfTI-1, both byte orders, so an
axis past 32767 voxels reads correctly. Both versions now report
niftiVersion. Golden fixture: headers nibabel 5.4.2 wrote, with nibabel's
own read of each recorded beside the expected decode.

NRRD: a new decoder for .nrrd and detached .nhdr headers (teem, ITK, 3D
Slicer, the Allen CCFv3 atlases). Sizes, dtype, encoding, endian, space,
space directions and the per-axis spacing derived from them (or spacings),
origin, units, kinds, labels, measurement frame, data file, key:=value
pairs, and whether the header fit the 64 KiB read. Verified on the real
CCFv3 annotation_25.nrrd header and cross-checked against pynrrd 1.1.3 on
every fixture case.
A .nwb file (the DANDI archive format) is an HDF5 file whose root carries
nwb_version; it now decodes as its own format: version, identifier,
session description, start time and id, institution, lab, experimenters,
keywords, publications, subject, devices, electrode groups and count,
imaging planes, acquisition series (type, shape, dtype, unit, rate),
processing modules and interfaces, unit count and columns, interval
tables, stimuli, embedded schema namespaces and any ndx-* extension.
Targeted lookups rather than a walk, with every listing taken before the
per-series detail, and the detail passes stopping before the read cap so
a file with dozens of series names them all and describes as many as the
budget allows.

Two HDF5 reader changes made that possible and apply to every flavour:
identical reads and group listings are memoized within one decode, so a
lookup from the root no longer re-reads the root B-tree and heap (no byte
is fetched that was not fetched before); and the global heap scan widens
in steps past its fixed 8 KiB window, bounded by the collection's own
size, because a session file keeps thousands of strings in one collection
and two of the four DANDI test files read their subject as None before.

Verified against h5py on four DANDI files from four labs and writers
(NWB 2.1.0 to 2.4.0, pynwb and MIES) and on pynwb 4.2.0 output; three
real files and the pynwb one are sparse golden fixtures.
@aaronkanzer aaronkanzer changed the title NIfTI-2 and NRRD decoders (0.9.0) NIfTI-2, NRRD and NWB decoders (0.9.0) Oct 25, 2026
A symbol table entry for a soft link has an undefined object address; the
reader treated that as a malformed node and stopped, dropping every entry
after it. An NWB TimeSeries keeps its `electrode` soft link in front of
`starting_time`, so the sampling rate of every MIES-written series was
unreadable. The entry is now skipped and the rest of the node read. Also
unwrap one-element scalar attributes (MIES writes `rate` and `unit` so).
Fixture: an h5py libver-earliest file with a soft link mid-group; the
four NWB sparse recordings re-recorded, rate checked against h5py.
@aaronkanzer
aaronkanzer merged commit 42a4eaf into main Oct 25, 2026
3 checks passed
@aaronkanzer
aaronkanzer deleted the feat/nifti2-nrrd branch October 25, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant