Electrophysiology rig formats and BIDS sidecars (0.10.0)

CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## 0.10.0
+
+- EDF, EDF+, BDF and BDF+ (`.edf`, `.bdf`): the electrophysiology container
+  clinical and research rigs write. Flavour and continuity, bit depth, start
+  date resolved to a four-digit year from the EDF+ recording field, the EDF+
+  patient and recording subfields, record count and duration, and per signal
+  the label, units, physical and digital range, samples per record, derived
+  rate, transducer and prefiltering. The annotations channel is named and kept
+  out of the recording's sampling rate. Verified on a PhysioNet Sleep-EDF
+  recording and against pyedflib.
+- BrainVision (`.vhdr`, `.vmrk`): header and marker files, one of the three
+  container formats BIDS allows for EEG and iEEG. Version, data file and
+  orientation, channel count, sampling interval and rate, binary format and
+  byte order, data points and duration, per-channel name, reference,
+  resolution and unit, coordinates present; and from the marker file the
+  counts per type, the first markers and the segment start timestamp.
+  Cross-checked against mne on all 64 non-empty `.vhdr` files in
+  bids-standard/bids-examples; all 64 agreed.
+- SpikeGLX (`.meta`): the run metadata beside every Neuropixels binary.
+  Stream type and kind, sample rate, AP/LF/sync channel counts, run name with
+  gate and trigger index, size, duration and derived sample count, probe type,
+  model, part number, serial, slot, port and dock, and microvolts per bit for
+  the probe generations the metadata reference defines it for. Per-channel
+  tables are counted, not copied. Verified against neo on Neuropixels 1.0 and
+  2.0 files; a 2016 Phase 3A file that neo cannot open still decodes.
+- Neuralynx (`.ncs`, `.nev`, `.nse`, `.ntt`, `.nvt`): the 16 KiB ASCII header,
+  in both the `##` comment-line dialect of Cheetah before 5.6.4 and the
+  property dialect of 5.7 onwards and Pegasus. File type and record kind,
+  record size, application and version, channel name, sampling rate, AD range
+  and microvolts per bit, DSP filters, open and close times. Checked against
+  python-neo's header grammar on four generations of file.
+- Open Ephys (`structure.oebin`): the JSON manifest of a binary-format
+  recording. GUI version, continuous streams with folder, stream and processor
+  names, sample rate, channel count and uniform scaling, plus event and spike
+  streams. Verified on three real manifests from GUI 0.4.4.1, 0.4.5 and 0.6.0.
+- BIDS sidecars (`.json`, `.tsv`): `dataset_description.json`, JSON metadata
+  sidecars, column data dictionaries, and the tabular files (events,
+  participants, sessions, samples, scans, channels, electrodes, optodes).
+  Both extensions are generic, so a file is claimed only when it carries a key
+  the standard defines or one of its column signatures. Swept over the whole
+  bids-standard/bids-examples corpus: 5853 files decoded, no exception, and
+  what declined is headerless derivative tables and non-core derivative JSON.
+- `.json` and `.tsv` are now registered extensions. Dispatch reaches a decoder
+  for any file with those extensions; a name-registered decoder (`zarr.json`,
+  `.zattrs`, `attributes.json`) still wins, and non-BIDS content is declined.
+
 ## 0.9.0
 
 - NIfTI-2: the 540-byte header (`n+2` / `ni2` magic, 64-bit dims, double

README.md

@@ -15,7 +15,7 @@ Read the metadata of a scientific file and return its fields (dimensions, data
 type, pixel size, columns, row count) as a dict, without reading the rest of the
 file. Decodes formats across cryo-EM, light microscopy and light-sheet imaging,
 arrays, tabular data, flow cytometry, mass spec, sequencing, genome annotation,
-and structural biology (the full list is below). The decode functions have no
+structural biology, and electrophysiology (the full list is below). The decode functions have no
 dependencies; separate reader functions fetch the bytes from a file or a URL:
 the leading bytes for most formats, the trailing bytes for Parquet (whose schema
 is in a footer), and a bounded handful of reads wherever the pointers lead for
@@ -111,6 +111,54 @@ Decoders today:
   study/series description. Reads technical fields only, never patient
   identifiers.
 
+Electrophysiology and BIDS (added in 0.10.0). What an EEG, iEEG or Neuropixels
+rig writes beside its raw samples, and the sidecars that describe a dataset
+organised to the Brain Imaging Data Structure:
+
+- **EDF / EDF+ / BDF / BDF+** (`.edf`, `.bdf`): the clinical and research
+  electrophysiology container. Flavour and continuity (`EDF+C` / `EDF+D`), bit
+  depth, start date and time (resolved to a four-digit year from the EDF+
+  recording field), the EDF+ patient and recording subfields, data record
+  count and duration, recording length, and per signal the label, units,
+  physical and digital range, samples per record, derived sampling rate,
+  transducer and prefiltering. The annotations channel is named and kept out
+  of the recording's sampling rate. Cross-checked against pyedflib.
+- **BrainVision** (`.vhdr`, `.vmrk`): Brain Products' header and marker files,
+  one of the three container formats BIDS allows for EEG and iEEG. Format
+  version, data file and orientation, channel count, sampling interval and
+  rate, binary format and byte order, data points and duration, per-channel
+  name, reference, resolution and unit, and whether electrode coordinates are
+  present; from the marker file, the marker count, the count per type, the
+  first markers and the segment start timestamp. Cross-checked against mne on
+  64 public files.
+- **SpikeGLX** (`.meta`): the run metadata SpikeGLX writes beside every
+  Neuropixels binary, which is the only description that headerless int16
+  matrix has. Stream type and kind, sample rate, channel counts split into AP,
+  LF and sync, run name with gate and trigger index, file size and duration,
+  derived sample count, probe type, model, part number and serial, slot, port
+  and dock, and the microvolts per bit for the two probe generations the
+  metadata reference defines it for. The per-channel tables are counted, not
+  copied. Cross-checked against neo.
+- **Neuralynx** (`.ncs`, `.nev`, `.nse`, `.ntt`, `.nvt`): the 16 KiB ASCII
+  header every Cheetah and Pegasus file opens with, in both dialects (the
+  older `##` comment lines with `-CheetahRev`, and the newer properties with
+  `-ApplicationName` and `-TimeCreated`). File type and record kind, record
+  size, application and version, channel name, sampling rate, AD range and
+  microvolts per bit, DSP filter settings, and the open and close times.
+  Checked against python-neo's header grammar.
+- **Open Ephys** (`structure.oebin`): the JSON manifest of a binary-format
+  recording directory, the only description of its headerless `.dat`
+  matrices. GUI version, continuous streams with folder, stream and processor
+  names, sample rate, channel count and the scaling where it is uniform
+  across the stream, plus the event and spike streams.
+- **BIDS sidecars** (`.json`, `.tsv`): `dataset_description.json`, the JSON
+  metadata sidecars, the column data dictionaries, and the tabular files
+  (events, participants, sessions, samples, scans, channels, electrodes,
+  optodes). Both extensions are generic, so a file is claimed only when it
+  carries a key the standard defines or one of its column signatures; key
+  names are kept exactly as the standard spells them. Swept over the whole
+  bids-standard/bids-examples corpus.
+
 Light microscopy and light-sheet (added in 0.6.0; most of these follow internal
 pointers, see "Pointer-based formats" below):
 
@@ -414,6 +462,11 @@ re-decoding.
   layouts and tested on constructed files; their tests say so. Olympus
   OIB (an OLE compound document) and OIR, Hamamatsu DCIMG, and raw `.bin` /
   `.dat` / `.raw` exports (which have no header to read) are not decoded.
+- **Neuralynx has no public sample** that was reachable, so its fixtures are
+  headers built to the per-version grammar documented in python-neo's
+  `NlxHeader` and parsed by neo, not copies of a rig's output. The binary
+  records after the header are not read, so a file's record count is not
+  reported.
 - **Legacy ND2** (JPEG2000 based) is recognised but not described.
 - BAM's BGZF framing and per-record structure are not handled; plain gzip is.
 - **Output is not a standard.** The fields are an ad-hoc dict, not an

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scigantic-headers"
-version = "0.9.0"
+version = "0.10.0"
 description = "Decode scientific file headers (cryo-EM, light microscopy and light-sheet imaging incl. TIFF/OME-TIFF, CZI, HDF5/Imaris/BigDataViewer, LIF, ND2, Zarr/N5; tabular, flow cytometry, mass spec, sequencing, structural biology) into typed fields. Pure, zero-dependency, with bounded readers that follow internal pointers without downloading the file."
 readme = "README.md"
 requires-python = ">=3.9"

src/scigantic_headers/__init__.py

@@ -14,6 +14,8 @@
 from .a3m import decode_a3m
 from .acquisition import Finding, compare_acquisition
 from .bed import decode_bed
+from .bids import decode_bids_json, decode_bids_tsv
+from .brainvision import decode_brainvision_vhdr, decode_brainvision_vmrk
 from .czi import decode_czi
 from .hdf5 import decode_hdf5
 from .ics import decode_ics
@@ -29,6 +31,7 @@
 from .zarr import decode_n5_attributes, decode_zarr_json, decode_zarray, decode_zattrs, decode_zgroup
 from .cryosparc import parse_cryosparc_optics, read_cryosparc_optics
 from .dicom import decode_dicom
+from .edf import decode_edf
 from .epu import decode_epu_xml, epu_pixel_size_candidates
 from .fastq import decode_fastq
 from .fcs import decode_fcs
@@ -37,10 +40,13 @@
 from .illumina import decode_illumina_run
 from .mmcif import decode_mmcif
 from .mzml import decode_mzml
+from .neuralynx import decode_neuralynx
+from .openephys import decode_openephys_oebin
 from .optics import read_session_optics
 from .parquet import decode_parquet
 from .pdb import decode_pdb
 from .sam import decode_sam
+from .spikeglx import decode_spikeglx_meta
 from .vcf import decode_vcf
 from .decoders import (
     HEADER_BYTES,
@@ -89,6 +95,9 @@
     "decode_parquet", "decode_fcs", "decode_mzml",
     "decode_a3m", "decode_fastq", "decode_illumina_run", "decode_vcf", "decode_sam", "decode_pdb",
     "decode_mmcif", "decode_genbank", "decode_gff", "decode_bed", "decode_dicom",
+    "decode_edf", "decode_brainvision_vhdr", "decode_brainvision_vmrk",
+    "decode_spikeglx_meta", "decode_neuralynx", "decode_openephys_oebin",
+    "decode_bids_json", "decode_bids_tsv",
     "decode_czi", "decode_hdf5", "decode_ics", "decode_klb", "decode_lif", "decode_nd2", "decode_nrrd", "decode_obf",
     "decode_tiff", "parse_ome_xml",
     "decode_bdv_xml", "decode_ome_xml", "decode_xml_sidecar",