Repository navigation
NIfTI-2, NRRD and NWB decoders (0.9.0) - #9
Merged
Merged
Conversation
NIfTI-2: the 540-byte header (n+2 / ni2 magic, int64 dim, float64 pixdim) decodes through the same nifti decoder as NIfTI-1, both byte orders, so an axis past 32767 voxels reads correctly. Both versions now report niftiVersion. Golden fixture: headers nibabel 5.4.2 wrote, with nibabel's own read of each recorded beside the expected decode. NRRD: a new decoder for .nrrd and detached .nhdr headers (teem, ITK, 3D Slicer, the Allen CCFv3 atlases). Sizes, dtype, encoding, endian, space, space directions and the per-axis spacing derived from them (or spacings), origin, units, kinds, labels, measurement frame, data file, key:=value pairs, and whether the header fit the 64 KiB read. Verified on the real CCFv3 annotation_25.nrrd header and cross-checked against pynrrd 1.1.3 on every fixture case.
A .nwb file (the DANDI archive format) is an HDF5 file whose root carries nwb_version; it now decodes as its own format: version, identifier, session description, start time and id, institution, lab, experimenters, keywords, publications, subject, devices, electrode groups and count, imaging planes, acquisition series (type, shape, dtype, unit, rate), processing modules and interfaces, unit count and columns, interval tables, stimuli, embedded schema namespaces and any ndx-* extension. Targeted lookups rather than a walk, with every listing taken before the per-series detail, and the detail passes stopping before the read cap so a file with dozens of series names them all and describes as many as the budget allows. Two HDF5 reader changes made that possible and apply to every flavour: identical reads and group listings are memoized within one decode, so a lookup from the root no longer re-reads the root B-tree and heap (no byte is fetched that was not fetched before); and the global heap scan widens in steps past its fixed 8 KiB window, bounded by the collection's own size, because a session file keeps thousands of strings in one collection and two of the four DANDI test files read their subject as None before. Verified against h5py on four DANDI files from four labs and writers (NWB 2.1.0 to 2.4.0, pynwb and MIES) and on pynwb 4.2.0 output; three real files and the pynwb one are sparse golden fixtures.
A symbol table entry for a soft link has an undefined object address; the reader treated that as a malformed node and stopped, dropping every entry after it. An NWB TimeSeries keeps its `electrode` soft link in front of `starting_time`, so the sampling rate of every MIES-written series was unreadable. The entry is now skipped and the rest of the node read. Also unwrap one-element scalar attributes (MIES writes `rate` and `unit` so). Fixture: an h5py libver-earliest file with a soft link mid-group; the four NWB sparse recordings re-recorded, rate checked against h5py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three data formats surfaced by a pass over the NeuroSchema catalog (neuroschema.org/catalog), which the library did not read.
What
n+2/ni2magic, int64dim, float64pixdim) decodes through the sameniftidecoder as NIfTI-1, in both byte orders, so an axis past 32767 voxels reads correctly. Both versions now reportniftiVersion(1 or 2); every other field is unchanged..nrrdand detached.nhdrheaders (teem, ITK, 3D Slicer, the Allen CCFv3 mouse atlases). Sizes, dtype, encoding, endian, space, space directions and the per-axis spacing derived from them (orspacings), origin, units, kinds, labels, measurement frame, data file,key:=valuepairs, and whether the header fit the 64 KiB read. The data is never touched, so gzip and bzip2 encodings are reported, not inflated..nwb, the DANDI archive format): an HDF5 file whose root carriesnwb_versiondecodes as its own format: version, identifier, session description, start time and id, institution, lab, experimenters, keywords, publications, subject, devices, electrode groups and count, imaging planes, acquisition series (type, shape, dtype, unit, rate), processing modules and interfaces, unit count and columns, interval tables, stimuli, embedded schema namespaces and anyndx-*extension. Targeted lookups rather than a walk; every listing is taken before the per-series detail, and the detail passes stop before the read cap, so a file with dozens of series names them all and describes as many as the budget allows.Two HDF5 reader changes came out of the NWB work and apply to every HDF5 flavour: identical reads and group listings are memoized within one decode (no byte is fetched that was not fetched before, but a lookup from the root no longer re-reads the root B-tree and heap), and the global heap scan widens in steps past its fixed 8 KiB window, bounded by the collection's own size. A pynwb session file keeps thousands of strings in one collection; two of the four DANDI test files read their subject as
Nonebefore this.Verification
tests/fixtures/nifti2-cases.json: NIfTI-2 headers written by nibabel 5.4.2, both byte orders and a 40000-voxel axis, with nibabel's own read recorded beside each expected decode.tests/fixtures/nrrd-cases.json: the real header of the CCFv3 25 umannotation_25.nrrd(fetched from download.alleninstitute.org by Range request) plus three pynrrd 1.1.3 files. Every value checked against pynrrd'sread_header.tests/fixtures/hdf5-nwb-*.sparse.json: three DANDI files from three labs and writers (000127 Miller lab 2.4.0; 000004 Rutishauser lab 2.1.0; 000020 Allen patch-seq written by MIES, 2.2.5,ndx-mies, 38 acquisition and 38 stimulus series) and a pynwb 4.2.0 file. Every structural value was checked against h5py on the full downloads.The NIfTI-2 and NRRD fixtures are copied into scigantic-edge in Scigantic/scigantic-edge#15, so the Go decoders are held to the same bytes. NWB is not ported there: edge has no generic HDF5 walker yet.