VFB domain meshes

Aligned anatomical-region meshes (and filled volume masks) for Virtual Fly Brain painted domains, organised so any template and any source dataset can be added the same way.

Layout

<template_dir>/                         one directory per VFB template (id in the name)
  README.md                             template grid + provenance
  <source>/                             one directory per source dataset
    source.json                         server / space / mesh->template affine
    domains.csv                         per-domain FBbt mapping (curator-checkable)
    obj/                                region meshes (see coordinate spaces below)
    nrrd/                               filled uint8 masks (255 in / 0 out) on the template grid
    regions_diagram.svg                 every region of that source, outlined and numbered
templates.csv                           index of templates + their grids
domains.csv                             aggregate of every source's domains.csv
scripts/                                fetch + voxelise helpers (reusable for any source)
pending/                                built and QC'd, blocked on curation -- not publishable
publish/                                generated mkdir + scp command sets (see below)

Every NRRD is on the template's exact voxel grid — same sizes, spacing and origin as the template's volume.nrrd, so masks overlay directly. All 970 are uint8, gzip-encoded, values strictly {0, 255}.

OBJ coordinate space differs by source, and you have to read source.json to know which.

Sources	obj_space
the seven adult sources	already in template world microns — transformed, not source-native
L1_larval_CNS/CATMAID_L1EM	native CATMAID project nanometres; apply that file's affine_to_template to reach microns

An earlier version of this file claimed a single convention, "OBJ = source-native coordinates". That was the scaffold's intention and it is not what was built: build_meshes.py transforms the vertices and writes the result, so the adult OBJs are template-space meshes. It was checked rather than assumed — every adult OBJ's bounding box matches its own NRRD's to within a voxel, and the L1 OBJs match theirs only after the stated affine. Six source.json files repeated the same wrong claim and were corrected with it on 2026-08-04.

Practical consequence: to overlay an adult OBJ on its template, load it as-is. To re-derive one from the source connectome, do not treat the committed OBJ as input to a bridging transform — start from the mesh bucket pinned in source.json under mesh_source_pin.

Each region also carries a proposed FBbt class in domains.csv for curator sign-off.

regions_diagram.svg

Every source directory holds one, generated by scripts/generate_region_diagrams.py from the committed OBJs — the same geometry the NRRDs were voxelised from, so what the diagram shows is what was built, not a separate rendering of the source. Three orthogonal projections (x–y, x–z, y–z) with each region's silhouette outlined and numbered, over a dashed box marking the template bounding box, and a numbered legend giving roi_id, the proposed FBbt label and its id. Rows with a qc_class other than ok are drawn with a heavier dashed outline and marked * in red in the legend, so the flagged domains in EXCEPTIONS.md can be found by eye — the VNC nerves visibly leaving the dashed template box is the clearest single picture of why most of those flags are expected.

A region is numbered once, on whichever of the three views its centroid sits furthest from its neighbours', which is not always the view where it is largest. SVG rather than PNG because the dense sources need zooming: maleCNS_Berg2025a on the brain template carries 337 regions.

These are a curator aid, not an input to anything — nothing in the pipeline reads them.

Adding a new template/source

1. Add a row to templates.csv (read the template's volume.nrrd header for the grid — do not guess).
2. scripts/fetch_catmaid_volumes.py (or your own exporter) -> obj/. Record where the meshes came from in source.json under mesh_source_pin, with something immutable — a GCS object generation or a DVID node hash, not a release tag, which can move.
3. scripts/voxelize_to_template.py with that template's grid + the source->template affine -> nrrd/.
4. Write domains.csv with the FBbt mapping; get it curated (curator_* columns).
5. scripts/qc_and_annotate.py --dir <template>/<source> --template <template.nrrd> to fill filled_voxels / alignment_status / curator_notes.
6. scripts/classify_qc_exceptions.py to fill qc_class. It exits non-zero if any row lands in unreviewed, which means a flagged domain has no anatomical explanation yet — extend the FBbt tables at the top of that script, or record it as a defect.
7. Add the directory to ORDER in scripts/build_aggregate_csv.py and run it to rebuild the root domains.csv. Do not hand-edit the root file — it is derived, and the script errors out if a source directory is missing from ORDER so a dataset cannot be silently dropped.
8. scripts/generate_region_diagrams.py --dir <template>/<source> to draw regions_diagram.svg. It reads qc_class, so it has to come after step 6 or every region is drawn as ok.
9. scripts/verify_repo.py before opening the PR. It asserts what the rest of the pipeline assumes — that the aggregate is not stale, that every referenced OBJ and NRRD exists and nothing under obj/ or nrrd/ is unreferenced, that every mask is uint8, gzip (never raw), axis-aligned, strictly {0, 255} and on its template's exact grid, that filled_voxels matches the mask it describes, that no row is left unreviewed, and that nothing exceeds GitHub's 50 MB warning threshold. It exits non-zero on any failure. --fast skips decompressing the masks (seconds rather than minutes) and is enough for a quick check; the full run is what to trust before a PR.

domains.csv columns

template_id, template_label, source, dataset, roi_id, vfb_short_form, label, side, sex, is_a_label, fbbt_id, part_of_labels, part_of_fbbt_id, obj, nrrd, filled_voxels, source_mesh_ref, term_verified, alignment_status, qc_class, curator_fbbt_ok, curator_notes

All eight sources share this one schema. The L1 set used to carry a schema of its own with a map_l1() shim in build_aggregate_csv.py to bridge them; the schema was widened to hold L1's curator fields rather than flattening L1, and the shim is gone.

Two columns are worth understanding before reading the QC numbers:

alignment_status is measured, by scripts/qc_and_annotate.py: generated-truncated means more than 2% of the mesh volume falls outside the template bounding box, generated-outside- template-tissue means fewer than half the filled voxels land inside the Otsu-thresholded template, generated-empty means zero voxels.

qc_class is the judgement on that measurement, from scripts/classify_qc_exceptions.py. It exists because a low tissue overlap can mean a domain is broken or can mean it is a nerve, and nerves leave the neuropil by definition. Values are ok, four expected-* classes, and two defect-* classes. Only the defect-* rows need anyone to do anything — see EXCEPTIONS.md.

roi_id is the authoritative key: it is the filename value in the VFB curation record, and it is what ties a mesh here to the domain that will be painted with it. Never rename it. curator_fbbt_ok = loaded-in-KB means the class assignment has already been made and loaded by the VFB curator, so it is not up for re-litigation here.

Files are named <vfb_short_form>_<roi_id> — obj/VFB_00107gal_AL(L).obj, nrrd/VFB_00107gal_AL(L).nrrd. Read the path from the obj / nrrd column rather than rebuilding it from roi_id; those columns are authoritative. The short_form prefix is what makes the name unique — roi_id alone is not, because AL(L) and aL(L) in OpticLobe_Nern2024 differ only in case and collide on macOS and Windows, where the second checkout silently overwrites the first and the clone still looks complete. All 972 short_forms are unique case-insensitively, so the prefix closes that without touching a curation record. It also keeps the roi_id visible, so the tree is still readable by eye, and it matches the fact that VFB publishes each image under the short_form (see vfb_short_form below).

Because filename is the only handle the loader has, drift between it and roi_id is silent: the domain loads, nothing raises, and the ROI renders blank. No check internal to this repo can see that, so curation_filenames.csv snapshots the filename column of every archived record on the Clare72_curation branch of VirtualFlyBrain/curation, and scripts/verify_repo.py compares both directions against it — a roi_id with no curation row is a mesh nothing will ever ask for; a curation row with no roi_id is a domain that will ask for a mesh nobody generated. Regenerate the snapshot whenever a record is re-loaded.

vfb_short_form is the ID of that domain's anatomical individual in the VFB KB. It is not in the curation record and cannot be — IDs are minted by the loader at load time, and the curation record is written before the load. Once a record is loaded the KB holds both halves, so the two can be joined; scripts/fetch_kb_short_forms.py is that join, and kb_short_forms.csv is its committed snapshot (972 rows, one per loaded ROI). The KB keeps filename as a property on the in_register_with edge from the ROI's channel to the template's channel, not on the individual:

(:DataSet)<-[:has_source]-(:Individual)<-[:depicts]-(channel)-[r:in_register_with]->(template_channel)

so the key is (dataset, template, filename). It has to be all three — labels are not unique across datasets (AL(L) on JRC2018Unisex resolves to two individuals, one Nern and one Berg; ADMN(L) on JRC2018UnisexVNC to three), so a label join silently picks the wrong one.

Query kb.virtualflybrain.org, not pdb.virtualflybrain.org. pdb is the derived public database and lags the KB; at the time of writing it held none of these ROIs, and neither did the public Solr index, which makes it look like the individuals do not exist.

The short_form is worth carrying because it, not roi_id, decides where VFB publishes the image: /data/VFB/i/<sf[4:8]>/<sf[8:12]>/<template_id>/. A wrong or stale one means a mesh that is checked, QC'd and committed and then served against the wrong domain, which no other check here can see — scripts/verify_repo.py compares every row against the snapshot for exactly that reason, and fetch_kb_short_forms.py --check confirms the snapshot is not itself stale.

All 972 short_forms are unique, and still unique when lowercased, which is why the mesh and mask filenames are prefixed with one.

Status

All 970 domains across all eight sources have an OBJ and a template-grid NRRD committed.

Template	Source	ROIs	ok	Flagged	State
L1 larval CNS (VFB_00050000)	CATMAID_L1EM	27	26	1 edge-clip	loaded in KB; OBJ + NRRD generated
JRC2018Unisex (VFB_00101567)	maleCNS_Berg2025a	337	330	6 edge-clip, 1 CNS junction	loaded in KB; OBJ + NRRD generated
JRC2018Unisex (VFB_00101567)	BANC_Bates2026	195	193	2 edge-clip	loaded in KB; OBJ + NRRD generated
JRC2018Unisex (VFB_00101567)	OpticLobe_Nern2024	105	102	3 edge-clip	loaded in KB; OBJ + NRRD generated
JRC2018Unisex (VFB_00101567)	FlyWire_Dorkenwald2023	78	76	1 edge-clip, 1 outside neuropil	loaded in KB; OBJ + NRRD generated
JRCVNC2018U (VFB_00200000)	BANC_Bates2026	110	98	11 nerve, 1 defect	loaded in KB; OBJ + NRRD generated
JRCVNC2018U (VFB_00200000)	MANC_Takemura2023	59	46	10 nerve, 1 junction, 2 defect	loaded in KB; OBJ + NRRD generated
JRCVNC2018U (VFB_00200000)	maleCNS_Berg2025a	59	53	6 nerve	loaded in KB; OBJ + NRRD generated

924 rows are ok. Of the 46 flagged, 43 are expected — 27 nerves and tracts, which the JRC2018 stain does not capture; 13 clipped at the edge of a template (12 optic-lobe regions on the brain template, and the right larval brain hemisphere, whose CATMAID volume reaches 3.2 µm past the L1 template's first z slice); 2 spanning the brain/VNC junction; 1 dorsal to the neuropil. Three are genuine defects and are the only rows anyone needs to act on. EXCEPTIONS.md lists all 46 with the specific question for each, and section 6 records a separate finding that is not a QC flag at all: FlyWire_Dorkenwald2023 labels its L/R sides opposite to the other three brain sources, on 30/30 of the regions the two carry in common. scripts/check_side_convention.py reproduces it. Nothing has been changed on the strength of it — it needs a curator ruling first.

The count is 970 rather than 972 because OpticLobe_Nern2024/vnc-shell and maleCNS_Berg2025a/VNC are loaded in the KB against the brain template while both are whole-nerve-cord meshes. Voxelised onto VFB_00101567 they give 73 and 0 voxels, neither on tissue — which is a template problem, not a mesh one. Both have since been built against JRCVNC2018U and QC'd (99.2 % and 99.8 % of voxels on template tissue) and sit in pending/ until Clare moves the two curation rows to JRC2018UnisexVNC_c; see pending/README.md for the build, the QC and the byte-identity finding that decides which bridging chain vnc-shell takes. They are the one standing verify_repo.py FAIL, and that FAIL is the accurate signal — it should keep failing until the curation changes. Both OBJs were decimated to 300,000 faces to stay under the repo's 50 MB threshold (vnc-shell transforms to 78.8 MB raw); the largest committed file elsewhere is 35.6 MB.

"loaded in KB" = the ROI individuals already exist (curator cp390, July 2026); only the image files are outstanding. See KB_LOADED_ROIS.md for which records are loaded, and ALIGNMENT.md for the verified bridging path from each source space to its VFB template.

Before transforming anything from JRCFIB2022M or FLYWIRE to JRC2018U, read the first section of ALIGNMENT.md. navis's default bridging route goes via JRC2018M and is wrong by 12–18 µm; it has to be forced with via='JRC2018F'. This applies to skeletons as much as to meshes, and the masks it produces look entirely plausible in isolation — the error is only visible when scored against an independent reference.

Publishing

Images go into VFB's image tree under a DATA directory, at a path keyed on the individual's vfb_short_form and never on the mesh filename:

<DATA>/VFB/i/<sf[4:8]>/<sf[8:12]>/<template_id>/volume.nrrd      the mask
                                              /volume_man.obj    our mesh

so VFB_00107gal on VFB_00101567 lands in VFB/i/0010/7gal/VFB_00101567/. Nothing about the filename appears anywhere in the destination, which is why a wrong vfb_short_form publishes silently into another individual's folder and no check downstream can see it. (Templates themselves have no template subdirectory; only domains do.)

<DATA> is a variable, because the same directory has three different paths depending on who is writing. Only the root differs — VFB/i/... below it is identical everywhere:

who	DATA	set by
the Jenkins job	/IMAGE_WRITE (mounted)	$IMAGE_WRITE
ssh to vfbds0	/docker/VFB/DATA	$DATA
a shell on the NAS itself	/volume1/docker/VFB/DATA	$DATA

The last two are the same directory: /docker on vfbds0 is a bind mount of /volume1/docker, so the mkdir has to use the /volume1 form and scp must not. Nothing has a path baked in — the defaults live in scripts/make_publish_commands.py and every generated script reads its root from the environment.

Nothing is ever overwritten. A destination file that already exists is reported on stdout and skipped, and the run carries on. That makes a re-run cheap and safe — it fills in what is missing and leaves everything else alone — and it means replacing a published image is a deliberate act: remove it by hand first. The Jenkins script stats each destination; the scp script gets the same answer from a single remote find up front rather than 1940 round trips.

The Jenkins route

scripts/publish_images.sh is the job. It assumes the Jenkins defaults and needs no arguments:

scripts/publish_images.sh

variable	default
WORKSPACE	the script's parent directory	the repo checkout
IMAGE_WRITE	/IMAGE_WRITE	the DATA directory, mounted into the job
COPY_OBJ	1	publish the mesh as volume_man.obj as well as the mask
DRY_RUN	0	1 reports what it would do and touches nothing
LIST	publish/publish_files.tsv	the file list to walk

It makes its own directories, so mkdir_dirs.sh is not needed on this route. Each file is written as .part.$$ beside its destination and renamed, so a killed job cannot leave a half-written volume.nrrd that the next run would then refuse to replace. It prints a per-100 heartbeat and a final N rows: X copied, Y already present and left alone, Z missing from repo, and exits non-zero only on that last count — which means the file list and the checkout disagree, not that publishing failed.

Run it against a scratch directory first if in doubt:

IMAGE_WRITE=/tmp/imagetest DRY_RUN=1 scripts/publish_images.sh

The scp route

For publishing from a machine that has the repo but not the mount:

ssh vfbds0.inf.ed.ac.uk 'sh -s' < publish/mkdir_dirs.sh
REPO=/path/to/VFB_domain_meshes ./publish/scp_upload.sh

scp_upload.sh checks one known file exists in $REPO before it opens a connection, so a wrong checkout fails immediately rather than after an ssh handshake. It opens one multiplexed ssh connection (ControlMaster + ControlPersist) and reuses it for every file — without that, 970 files means 970 handshakes — and tears it down on exit via a trap. Unlike the Jenkins route it defaults to COPY_OBJ=0, masks only.

Regenerating

python3 scripts/make_publish_commands.py

after any change to domains.csv. It writes four files into publish/:

file	what it is
publish_files.tsv	rel_dir, nrrd, obj, short_form, roi_id — the list publish_images.sh walks. TSV rather than CSV so a shell read cannot be confused by the parentheses, apostrophes (a'L(L)) and spaces (Brain Hemisphere left) that appear in roi_ids; the generator refuses to emit a row containing a tab
publish_manifest.csv	the same rows for a human, with the resolved remote directory spelled out
mkdir_dirs.sh	970 mkdir -p lines, for the scp route
scp_upload.sh	970 uploads over one connection

The generator refuses to emit anything if a row has no vfb_short_form, if two rows resolve to the same destination, or if any referenced file is missing from disk. It reads the obj / nrrd columns rather than rebuilding names from roi_id. Local paths are shlex.quoted; remote paths are short_form-derived and need no quoting.

Nothing here runs any of it — these write into VFB's live image tree, so that stays a deliberate act. The generated destinations were cross-checked against the folder column of kb_short_forms.csv by re-parsing the generated list rather than the generator's own manifest: 943/943 populated rows agree exactly. The other 27 are L1 rows whose KB folder is empty, so there is nothing to disagree with. publish_images.sh was exercised end to end against a scratch tree — all 1940 files, including every awkward filename, plus a pre-seeded destination to prove the skip, a re-run to prove idempotency, and a missing-file row to prove the non-zero exit.

See templates.csv for grids and existing painted-domain coverage.