From 8e8c22f8e4165b70abe0c322035e87332211dd74 Mon Sep 17 00:00:00 2001 From: Carter Francis Date: Thu, 15 Oct 2026 10:59:50 -0500 Subject: [PATCH 1/3] Add the DE SpotNet diffraction spot detector and center refinement networks Six kind: weights entries for Direct Electron's networks, served as ONNX graphs from Zenodo record 23383572 (CC BY-NC 4.0): four spot detectors (DE_SpotNet, DE_SpotNet_Dynamical, DE_SpotNet_Production, DE_SpotNet_Base16) and the two center refiners that follow them (DE_CenterNet, DE_FriedelNet). The matching torch checkpoints sit in the same record for training and fine-tuning. --- emdatabase/data/__init__.pyi | 140 +++++++++++++++++--- emdatabase/index/DE_CenterNet.yaml | 41 ++++++ emdatabase/index/DE_FriedelNet.yaml | 41 ++++++ emdatabase/index/DE_SpotNet.yaml | 43 ++++++ emdatabase/index/DE_SpotNet_Base16.yaml | 38 ++++++ emdatabase/index/DE_SpotNet_Dynamical.yaml | 43 ++++++ emdatabase/index/DE_SpotNet_Production.yaml | 40 ++++++ 7 files changed, 364 insertions(+), 22 deletions(-) create mode 100644 emdatabase/index/DE_CenterNet.yaml create mode 100644 emdatabase/index/DE_FriedelNet.yaml create mode 100644 emdatabase/index/DE_SpotNet.yaml create mode 100644 emdatabase/index/DE_SpotNet_Base16.yaml create mode 100644 emdatabase/index/DE_SpotNet_Dynamical.yaml create mode 100644 emdatabase/index/DE_SpotNet_Production.yaml diff --git a/emdatabase/data/__init__.pyi b/emdatabase/data/__init__.pyi index 2618678..3883884 100644 --- a/emdatabase/data/__init__.pyi +++ b/emdatabase/data/__init__.pyi @@ -92,6 +92,102 @@ class CuZnHAADF(DownloadableDataset): https://raw.githubusercontent.com/hyperspy/exspy-demos/927d1f21b3b8aba4e2e622c2e621d3d9d5542d1c/EELS/datasets + """ + ... + +class DE_CenterNet(DownloadableDataset): + """ + DE_CenterNet + + Center refinement for DE SpotNet detections: segments each detected disk in a window of the raw pattern at native resolution and takes the coverage centroid, with a sigma head giving a positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the Friedel error of the detector's centers from 1.13 to 0.84 px. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_centernet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_FriedelNet(DownloadableDataset): + """ + DE_FriedelNet + + Friedel-partner center refinement, applied after DE CenterNet: reads each disk's window together with the 180-degree-rotated window at its Friedel mirror point, so it sees both dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_friedelnet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet(DownloadableDataset): + """ + DE_SpotNet + + Direct Electron's neural-network diffraction spot (Bragg disk) detector for 4D-STEM and scanning precession electron diffraction, the default detector in SpyDE, pyxem and quantem. Each pattern is rescaled so its disks are about 9 px across (diameter from the autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel offset. Trained across disk sizes, so it tolerates residual error in the rescale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Base16(DownloadableDataset): + """ + DE_SpotNet_Base16 + + Earlier edge-trained DE SpotNet detector, kept as a fallback. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet_base16.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Dynamical(DownloadableDataset): + """ + DE_SpotNet_Dynamical + + DE SpotNet fine-tuned on multislice (dynamical) disks with counted and integrating detector noise, then self-trained on real 4D-STEM frames whose labels are persistence- and Friedel- verified. A disk-coverage head centers each disk on its predicted aperture. Halves the center error on dynamical disks and cuts false positives on sparse counted data; on real SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet_dynamical.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Production(DownloadableDataset): + """ + DE_SpotNet_Production + + DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained scale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet_production.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + """ ... @@ -141,6 +237,23 @@ class InSituElectrochemGrowth(DownloadableDataset): https://zenodo.org/records/21632101/files + """ + ... + +class LayeredCuNb4DSTEM(DownloadableDataset): + """ + LayeredCuNb4DSTEM + + A 4D-STEM dataset of a layered Cu/Nb nanolaminate, acquired with a nearly parallel 1.58 mrad probe on a Direct Electron CeleritasXS. 128 x 128 probe positions (the central quarter of a 512 x 512 raster scan) of 256 x 256 pixel diffraction patterns, each the sum of 32 camera frames at 25000 fps. Reciprocal space is calibrated at 0.0078768 1/Angstrom per pixel and centred on the direct beam; the detector half-width is 1.008 1/Angstrom (25.3 mrad at 200 kV). The real-space scan step was not recorded by the scan controller, so the navigation axes are in pixels. Gain- and dark-corrected intensities were divided by 2 and rounded to uint16; multiply by 2 to recover ADU. + + DOI: 10.5281/zenodo.21632101 + + License: CC-BY-4.0 + + You can download this dataset here: + https://zenodo.org/records/21632101/files + + """ ... @@ -204,18 +317,16 @@ class LSMOSTOLineScanLowLoss(DownloadableDataset): """ ... -class LayeredCuNb4DSTEM(DownloadableDataset): +class MgONanoCrystals(DownloadableDataset): """ - LayeredCuNb4DSTEM - - A 4D-STEM dataset of a layered Cu/Nb nanolaminate, acquired with a nearly parallel 1.58 mrad probe on a Direct Electron CeleritasXS. 128 x 128 probe positions (the central quarter of a 512 x 512 raster scan) of 256 x 256 pixel diffraction patterns, each the sum of 32 camera frames at 25000 fps. Reciprocal space is calibrated at 0.0078768 1/Angstrom per pixel and centred on the direct beam; the detector half-width is 1.008 1/Angstrom (25.3 mrad at 200 kV). The real-space scan step was not recorded by the scan controller, so the navigation axes are in pixels. Gain- and dark-corrected intensities were divided by 2 and rounded to uint16; multiply by 2 to recover ADU. + MgONanoCrystals - DOI: 10.5281/zenodo.21632101 + A 4D STEM dataset of various MgO nanocrystals License: CC-BY-4.0 You can download this dataset here: - https://zenodo.org/records/21632101/files + https://zenodo.org/records/15490547/files """ @@ -235,21 +346,6 @@ class MOSS6Fig3(DownloadableDataset): https://zenodo.org/records/13958144/files - """ - ... - -class MgONanoCrystals(DownloadableDataset): - """ - MgONanoCrystals - - A 4D STEM dataset of various MgO nanocrystals - - License: CC-BY-4.0 - - You can download this dataset here: - https://zenodo.org/records/15490547/files - - """ ... @@ -376,4 +472,4 @@ class ZrNbPrecipitate(DownloadableDataset): """ ... -__all__ = ['AlNanocrystals', 'AmorphousFilm4nm4DSTEM', 'ApoferritinApollo15eps', 'BilayerWS2', 'CuZnEELSMapping', 'CuZnHAADF', 'FeAlStripes', 'HREBSDStrainPatterns', 'InSituElectrochemGrowth', 'LSMOLineScan', 'LSMOLineScanLowLoss', 'LSMOSTOLineScan', 'LSMOSTOLineScanLowLoss', 'LayeredCuNb4DSTEM', 'MOSS6Fig3', 'MgONanoCrystals', 'NiEBSDLarge', 'PdCuSiCrystallization', 'PdNiPGlass', 'PeakDetectionPolymers', 'SPEDAg', 'TutorialUNet', 'TwistedBilayerWSe2Themis', 'ZrNbPrecipitate'] \ No newline at end of file +__all__ = ['AlNanocrystals', 'AmorphousFilm4nm4DSTEM', 'ApoferritinApollo15eps', 'BilayerWS2', 'CuZnEELSMapping', 'CuZnHAADF', 'DE_CenterNet', 'DE_FriedelNet', 'DE_SpotNet', 'DE_SpotNet_Base16', 'DE_SpotNet_Dynamical', 'DE_SpotNet_Production', 'FeAlStripes', 'HREBSDStrainPatterns', 'InSituElectrochemGrowth', 'LayeredCuNb4DSTEM', 'LSMOLineScan', 'LSMOLineScanLowLoss', 'LSMOSTOLineScan', 'LSMOSTOLineScanLowLoss', 'MgONanoCrystals', 'MOSS6Fig3', 'NiEBSDLarge', 'PdCuSiCrystallization', 'PdNiPGlass', 'PeakDetectionPolymers', 'SPEDAg', 'TutorialUNet', 'TwistedBilayerWSe2Themis', 'ZrNbPrecipitate'] \ No newline at end of file diff --git a/emdatabase/index/DE_CenterNet.yaml b/emdatabase/index/DE_CenterNet.yaml new file mode 100644 index 0000000..6fdba4c --- /dev/null +++ b/emdatabase/index/DE_CenterNet.yaml @@ -0,0 +1,41 @@ +# $schema: ./json-schema.json +DE_CenterNet: + description: >- + Center refinement for DE SpotNet detections: segments each detected disk in a window of the + raw pattern at native resolution and takes the coverage centroid, with a sigma head giving a + positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the + Friedel error of the detector's centers from 1.13 to 0.84 px. An ONNX graph (BatchNorm + folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for + inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is + de_centernet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, + analyzing your own data at a company) is permitted; the weights may not be redistributed in, + or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_centernet.onnx + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Subpixel refinement + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: onnxruntime.InferenceSession + framework: onnx + latest: + url: https://zenodo.org/records/23383572/files/de_centernet.onnx + checksum: md5:1815cfa141da1a5acb1c191e82b72cdc + size_bytes: 124518 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_centernet.onnx + checksum: md5:1815cfa141da1a5acb1c191e82b72cdc + size_bytes: 124518 diff --git a/emdatabase/index/DE_FriedelNet.yaml b/emdatabase/index/DE_FriedelNet.yaml new file mode 100644 index 0000000..ce741f7 --- /dev/null +++ b/emdatabase/index/DE_FriedelNet.yaml @@ -0,0 +1,41 @@ +# $schema: ./json-schema.json +DE_FriedelNet: + description: >- + Friedel-partner center refinement, applied after DE CenterNet: reads each disk's window + together with the 180-degree-rotated window at its Friedel mirror point, so it sees both + dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. + An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the + model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training + and fine-tuning is de_friedelnet.pt in the same record. Licensed CC BY-NC 4.0: commercial + use (for example, analyzing your own data at a company) is permitted; the weights may not be + redistributed in, or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_friedelnet.onnx + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Subpixel refinement + - Friedel pairs + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: onnxruntime.InferenceSession + framework: onnx + latest: + url: https://zenodo.org/records/23383572/files/de_friedelnet.onnx + checksum: md5:127d5c86840377da01bd09b0f0eb0992 + size_bytes: 124793 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_friedelnet.onnx + checksum: md5:127d5c86840377da01bd09b0f0eb0992 + size_bytes: 124793 diff --git a/emdatabase/index/DE_SpotNet.yaml b/emdatabase/index/DE_SpotNet.yaml new file mode 100644 index 0000000..6b75682 --- /dev/null +++ b/emdatabase/index/DE_SpotNet.yaml @@ -0,0 +1,43 @@ +# $schema: ./json-schema.json +DE_SpotNet: + description: >- + Direct Electron's neural-network diffraction spot (Bragg disk) detector for 4D-STEM and + scanning precession electron diffraction, the default detector in SpyDE, pyxem and quantem. + Each pattern is rescaled so its disks are about 9 px across (diameter from the + autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel + offset. Trained across disk sizes, so it tolerates residual error in the rescale. An ONNX + graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model + metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and + fine-tuning is de_spotnet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for + example, analyzing your own data at a company) is permitted; the weights may not be + redistributed in, or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet.onnx + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - U-Net + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: onnxruntime.InferenceSession + framework: onnx + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet.onnx + checksum: md5:9ee0700aefd1ed64b8d6fbe2a3bc0c84 + size_bytes: 490476 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet.onnx + checksum: md5:9ee0700aefd1ed64b8d6fbe2a3bc0c84 + size_bytes: 490476 diff --git a/emdatabase/index/DE_SpotNet_Base16.yaml b/emdatabase/index/DE_SpotNet_Base16.yaml new file mode 100644 index 0000000..7efc80c --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Base16.yaml @@ -0,0 +1,38 @@ +# $schema: ./json-schema.json +DE_SpotNet_Base16: + description: >- + Earlier edge-trained DE SpotNet detector, kept as a fallback. An ONNX graph (BatchNorm + folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for + inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is + de_spotnet_base16.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, + analyzing your own data at a company) is permitted; the weights may not be redistributed in, + or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet_base16.onnx + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: onnxruntime.InferenceSession + framework: onnx + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet_base16.onnx + checksum: md5:99046ae5c26dc3f96c216faf871596df + size_bytes: 472862 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet_base16.onnx + checksum: md5:99046ae5c26dc3f96c216faf871596df + size_bytes: 472862 diff --git a/emdatabase/index/DE_SpotNet_Dynamical.yaml b/emdatabase/index/DE_SpotNet_Dynamical.yaml new file mode 100644 index 0000000..50f544a --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Dynamical.yaml @@ -0,0 +1,43 @@ +# $schema: ./json-schema.json +DE_SpotNet_Dynamical: + description: >- + DE SpotNet fine-tuned on multislice (dynamical) disks with counted and integrating detector + noise, then self-trained on real 4D-STEM frames whose labels are persistence- and Friedel- + verified. A disk-coverage head centers each disk on its predicted aperture. Halves the + center error on dynamical disks and cuts false positives on sparse counted data; on real + SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. An ONNX graph (BatchNorm + folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for + inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is + de_spotnet_dynamical.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for + example, analyzing your own data at a company) is permitted; the weights may not be + redistributed in, or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet_dynamical.onnx + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - Dynamical diffraction + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: onnxruntime.InferenceSession + framework: onnx + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet_dynamical.onnx + checksum: md5:e9cad9bccc007bffe14f7409ed260153 + size_bytes: 490831 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet_dynamical.onnx + checksum: md5:e9cad9bccc007bffe14f7409ed260153 + size_bytes: 490831 diff --git a/emdatabase/index/DE_SpotNet_Production.yaml b/emdatabase/index/DE_SpotNet_Production.yaml new file mode 100644 index 0000000..7b72f42 --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Production.yaml @@ -0,0 +1,40 @@ +# $schema: ./json-schema.json +DE_SpotNet_Production: + description: >- + DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly + higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained + scale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration + in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for + training and fine-tuning is de_spotnet_production.pt in the same record. Licensed CC BY-NC + 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the + weights may not be redistributed in, or incorporated into, software or services that are + sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet_production.onnx + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: onnxruntime.InferenceSession + framework: onnx + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet_production.onnx + checksum: md5:d485acbc0bc29429607cfe8d8e7e8c1e + size_bytes: 472866 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet_production.onnx + checksum: md5:d485acbc0bc29429607cfe8d8e7e8c1e + size_bytes: 472866 From 9db14906cb9657d89e6548df7c5f4f4e89c49a2b Mon Sep 17 00:00:00 2001 From: Carter Francis Date: Thu, 15 Oct 2026 11:10:47 -0500 Subject: [PATCH 2/3] Sort the index the same way on every platform dataset_files sorted Path objects, and Windows paths compare case-insensitively, so a stub generated on Windows put LayeredCuNb4DSTEM before LSMOLineScan and failed the stub check on Linux CI. Sort on the POSIX path string instead, which is the order CI already produced. --- emdatabase/data/__init__.pyi | 44 ++++++++++++++++++------------------ emdatabase/metadata.py | 10 ++++++-- 2 files changed, 30 insertions(+), 24 deletions(-) diff --git a/emdatabase/data/__init__.pyi b/emdatabase/data/__init__.pyi index 3883884..6f6971d 100644 --- a/emdatabase/data/__init__.pyi +++ b/emdatabase/data/__init__.pyi @@ -237,23 +237,6 @@ class InSituElectrochemGrowth(DownloadableDataset): https://zenodo.org/records/21632101/files - """ - ... - -class LayeredCuNb4DSTEM(DownloadableDataset): - """ - LayeredCuNb4DSTEM - - A 4D-STEM dataset of a layered Cu/Nb nanolaminate, acquired with a nearly parallel 1.58 mrad probe on a Direct Electron CeleritasXS. 128 x 128 probe positions (the central quarter of a 512 x 512 raster scan) of 256 x 256 pixel diffraction patterns, each the sum of 32 camera frames at 25000 fps. Reciprocal space is calibrated at 0.0078768 1/Angstrom per pixel and centred on the direct beam; the detector half-width is 1.008 1/Angstrom (25.3 mrad at 200 kV). The real-space scan step was not recorded by the scan controller, so the navigation axes are in pixels. Gain- and dark-corrected intensities were divided by 2 and rounded to uint16; multiply by 2 to recover ADU. - - DOI: 10.5281/zenodo.21632101 - - License: CC-BY-4.0 - - You can download this dataset here: - https://zenodo.org/records/21632101/files - - """ ... @@ -317,16 +300,18 @@ class LSMOSTOLineScanLowLoss(DownloadableDataset): """ ... -class MgONanoCrystals(DownloadableDataset): +class LayeredCuNb4DSTEM(DownloadableDataset): """ - MgONanoCrystals + LayeredCuNb4DSTEM - A 4D STEM dataset of various MgO nanocrystals + A 4D-STEM dataset of a layered Cu/Nb nanolaminate, acquired with a nearly parallel 1.58 mrad probe on a Direct Electron CeleritasXS. 128 x 128 probe positions (the central quarter of a 512 x 512 raster scan) of 256 x 256 pixel diffraction patterns, each the sum of 32 camera frames at 25000 fps. Reciprocal space is calibrated at 0.0078768 1/Angstrom per pixel and centred on the direct beam; the detector half-width is 1.008 1/Angstrom (25.3 mrad at 200 kV). The real-space scan step was not recorded by the scan controller, so the navigation axes are in pixels. Gain- and dark-corrected intensities were divided by 2 and rounded to uint16; multiply by 2 to recover ADU. + + DOI: 10.5281/zenodo.21632101 License: CC-BY-4.0 You can download this dataset here: - https://zenodo.org/records/15490547/files + https://zenodo.org/records/21632101/files """ @@ -346,6 +331,21 @@ class MOSS6Fig3(DownloadableDataset): https://zenodo.org/records/13958144/files + """ + ... + +class MgONanoCrystals(DownloadableDataset): + """ + MgONanoCrystals + + A 4D STEM dataset of various MgO nanocrystals + + License: CC-BY-4.0 + + You can download this dataset here: + https://zenodo.org/records/15490547/files + + """ ... @@ -472,4 +472,4 @@ class ZrNbPrecipitate(DownloadableDataset): """ ... -__all__ = ['AlNanocrystals', 'AmorphousFilm4nm4DSTEM', 'ApoferritinApollo15eps', 'BilayerWS2', 'CuZnEELSMapping', 'CuZnHAADF', 'DE_CenterNet', 'DE_FriedelNet', 'DE_SpotNet', 'DE_SpotNet_Base16', 'DE_SpotNet_Dynamical', 'DE_SpotNet_Production', 'FeAlStripes', 'HREBSDStrainPatterns', 'InSituElectrochemGrowth', 'LayeredCuNb4DSTEM', 'LSMOLineScan', 'LSMOLineScanLowLoss', 'LSMOSTOLineScan', 'LSMOSTOLineScanLowLoss', 'MgONanoCrystals', 'MOSS6Fig3', 'NiEBSDLarge', 'PdCuSiCrystallization', 'PdNiPGlass', 'PeakDetectionPolymers', 'SPEDAg', 'TutorialUNet', 'TwistedBilayerWSe2Themis', 'ZrNbPrecipitate'] \ No newline at end of file +__all__ = ['AlNanocrystals', 'AmorphousFilm4nm4DSTEM', 'ApoferritinApollo15eps', 'BilayerWS2', 'CuZnEELSMapping', 'CuZnHAADF', 'DE_CenterNet', 'DE_FriedelNet', 'DE_SpotNet', 'DE_SpotNet_Base16', 'DE_SpotNet_Dynamical', 'DE_SpotNet_Production', 'FeAlStripes', 'HREBSDStrainPatterns', 'InSituElectrochemGrowth', 'LSMOLineScan', 'LSMOLineScanLowLoss', 'LSMOSTOLineScan', 'LSMOSTOLineScanLowLoss', 'LayeredCuNb4DSTEM', 'MOSS6Fig3', 'MgONanoCrystals', 'NiEBSDLarge', 'PdCuSiCrystallization', 'PdNiPGlass', 'PeakDetectionPolymers', 'SPEDAg', 'TutorialUNet', 'TwistedBilayerWSe2Themis', 'ZrNbPrecipitate'] \ No newline at end of file diff --git a/emdatabase/metadata.py b/emdatabase/metadata.py index edb8e43..d0c4ea2 100644 --- a/emdatabase/metadata.py +++ b/emdatabase/metadata.py @@ -53,8 +53,14 @@ def dataset_files(directory: Path = INDEX_DIR) -> list[Path]: - """Every dataset collection YAML in ``directory``, sorted by name.""" - return sorted(p for p in directory.rglob("*.y*ml") if p.name not in NON_DATASET_FILES) + """Every dataset collection YAML in ``directory``, sorted by name. + + Sorted on the path string, not the ``Path``: Windows paths compare case-insensitively, + so on Windows ``LayeredCuNb4DSTEM`` would sort before ``LSMOLineScan`` and the generated + stub would disagree with the one CI builds on Linux. + """ + files = (p for p in directory.rglob("*.y*ml") if p.name not in NON_DATASET_FILES) + return sorted(files, key=lambda path: path.as_posix()) class IndexEntry(NamedTuple): From 808d00d3875d388e55a6864635bf74c5632c88ff Mon Sep 17 00:00:00 2001 From: Carter Francis Date: Thu, 15 Oct 2026 11:20:54 -0500 Subject: [PATCH 3/3] Add the DE SpotNet torch checkpoints as their own entries One *_Torch entry per network for the .pt checkpoints in the same Zenodo record, which training and fine-tuning start from (quantem's teach-on-your-marks needs them). Each ONNX entry now names its _Torch partner and vice versa. --- emdatabase/data/__init__.pyi | 110 ++++++++++++++++-- emdatabase/index/DE_CenterNet.yaml | 8 +- emdatabase/index/DE_CenterNet_Torch.yaml | 41 +++++++ emdatabase/index/DE_FriedelNet.yaml | 8 +- emdatabase/index/DE_FriedelNet_Torch.yaml | 41 +++++++ emdatabase/index/DE_SpotNet.yaml | 8 +- emdatabase/index/DE_SpotNet_Base16.yaml | 8 +- emdatabase/index/DE_SpotNet_Base16_Torch.yaml | 38 ++++++ emdatabase/index/DE_SpotNet_Dynamical.yaml | 8 +- .../index/DE_SpotNet_Dynamical_Torch.yaml | 43 +++++++ emdatabase/index/DE_SpotNet_Production.yaml | 9 +- .../index/DE_SpotNet_Production_Torch.yaml | 39 +++++++ emdatabase/index/DE_SpotNet_Torch.yaml | 43 +++++++ 13 files changed, 372 insertions(+), 32 deletions(-) create mode 100644 emdatabase/index/DE_CenterNet_Torch.yaml create mode 100644 emdatabase/index/DE_FriedelNet_Torch.yaml create mode 100644 emdatabase/index/DE_SpotNet_Base16_Torch.yaml create mode 100644 emdatabase/index/DE_SpotNet_Dynamical_Torch.yaml create mode 100644 emdatabase/index/DE_SpotNet_Production_Torch.yaml create mode 100644 emdatabase/index/DE_SpotNet_Torch.yaml diff --git a/emdatabase/data/__init__.pyi b/emdatabase/data/__init__.pyi index 6f6971d..b820d27 100644 --- a/emdatabase/data/__init__.pyi +++ b/emdatabase/data/__init__.pyi @@ -99,7 +99,23 @@ class DE_CenterNet(DownloadableDataset): """ DE_CenterNet - Center refinement for DE SpotNet detections: segments each detected disk in a window of the raw pattern at native resolution and takes the coverage centroid, with a sigma head giving a positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the Friedel error of the detector's centers from 1.13 to 0.84 px. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_centernet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + Center refinement for DE SpotNet detections: segments each detected disk in a window of the raw pattern at native resolution and takes the coverage centroid, with a sigma head giving a positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the Friedel error of the detector's centers from 1.13 to 0.84 px. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the DE_CenterNet_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_CenterNet_Torch(DownloadableDataset): + """ + DE_CenterNet_Torch + + Center refinement for DE SpotNet detections: segments each detected disk in a window of the raw pattern at native resolution and takes the coverage centroid, with a sigma head giving a positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the Friedel error of the detector's centers from 1.13 to 0.84 px. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_CenterNet entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. DOI: 10.5281/zenodo.23383571 @@ -115,7 +131,23 @@ class DE_FriedelNet(DownloadableDataset): """ DE_FriedelNet - Friedel-partner center refinement, applied after DE CenterNet: reads each disk's window together with the 180-degree-rotated window at its Friedel mirror point, so it sees both dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_friedelnet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + Friedel-partner center refinement, applied after DE CenterNet: reads each disk's window together with the 180-degree-rotated window at its Friedel mirror point, so it sees both dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine- tuning is the DE_FriedelNet_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_FriedelNet_Torch(DownloadableDataset): + """ + DE_FriedelNet_Torch + + Friedel-partner center refinement, applied after DE CenterNet: reads each disk's window together with the 180-degree-rotated window at its Friedel mirror point, so it sees both dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_FriedelNet entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. DOI: 10.5281/zenodo.23383571 @@ -131,7 +163,7 @@ class DE_SpotNet(DownloadableDataset): """ DE_SpotNet - Direct Electron's neural-network diffraction spot (Bragg disk) detector for 4D-STEM and scanning precession electron diffraction, the default detector in SpyDE, pyxem and quantem. Each pattern is rescaled so its disks are about 9 px across (diameter from the autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel offset. Trained across disk sizes, so it tolerates residual error in the rescale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + Direct Electron's neural-network diffraction spot (Bragg disk) detector for 4D-STEM and scanning precession electron diffraction, the default detector in SpyDE, pyxem and quantem. Each pattern is rescaled so its disks are about 9 px across (diameter from the autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel offset. Trained across disk sizes, so it tolerates residual error in the rescale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the DE_SpotNet_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. DOI: 10.5281/zenodo.23383571 @@ -147,7 +179,23 @@ class DE_SpotNet_Base16(DownloadableDataset): """ DE_SpotNet_Base16 - Earlier edge-trained DE SpotNet detector, kept as a fallback. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet_base16.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + Earlier edge-trained DE SpotNet detector, kept as a fallback. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the DE_SpotNet_Base16_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Base16_Torch(DownloadableDataset): + """ + DE_SpotNet_Base16_Torch + + Earlier edge-trained DE SpotNet detector, kept as a fallback. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_SpotNet_Base16 entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. DOI: 10.5281/zenodo.23383571 @@ -163,7 +211,23 @@ class DE_SpotNet_Dynamical(DownloadableDataset): """ DE_SpotNet_Dynamical - DE SpotNet fine-tuned on multislice (dynamical) disks with counted and integrating detector noise, then self-trained on real 4D-STEM frames whose labels are persistence- and Friedel- verified. A disk-coverage head centers each disk on its predicted aperture. Halves the center error on dynamical disks and cuts false positives on sparse counted data; on real SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet_dynamical.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + DE SpotNet fine-tuned on multislice (dynamical) disks with counted and integrating detector noise, then self-trained on real 4D-STEM frames whose labels are persistence- and Friedel- verified. A disk-coverage head centers each disk on its predicted aperture. Halves the center error on dynamical disks and cuts false positives on sparse counted data; on real SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the DE_SpotNet_Dynamical_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Dynamical_Torch(DownloadableDataset): + """ + DE_SpotNet_Dynamical_Torch + + DE SpotNet fine-tuned on multislice (dynamical) disks with counted and integrating detector noise, then self-trained on real 4D-STEM frames whose labels are persistence- and Friedel- verified. A disk-coverage head centers each disk on its predicted aperture. Halves the center error on dynamical disks and cuts false positives on sparse counted data; on real SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_SpotNet_Dynamical entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. DOI: 10.5281/zenodo.23383571 @@ -179,7 +243,39 @@ class DE_SpotNet_Production(DownloadableDataset): """ DE_SpotNet_Production - DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained scale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is de_spotnet_production.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained scale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the DE_SpotNet_Production_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Production_Torch(DownloadableDataset): + """ + DE_SpotNet_Production_Torch + + DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained scale. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_SpotNet_Production entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. + + DOI: 10.5281/zenodo.23383571 + + License: CC-BY-NC-4.0 + + Model weights hosted at https://zenodo.org/records/23383572/files; see ``.versions`` for the dated snapshots. + + + """ + ... + +class DE_SpotNet_Torch(DownloadableDataset): + """ + DE_SpotNet_Torch + + Direct Electron's neural-network diffraction spot (Bragg disk) detector for 4D-STEM and scanning precession electron diffraction, the default detector in SpyDE, pyxem and quantem. Each pattern is rescaled so its disks are about 9 px across (diameter from the autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel offset. Trained across disk sizes, so it tolerates residual error in the rescale. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_SpotNet entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the weights may not be redistributed in, or incorporated into, software or services that are sold. DOI: 10.5281/zenodo.23383571 @@ -472,4 +568,4 @@ class ZrNbPrecipitate(DownloadableDataset): """ ... -__all__ = ['AlNanocrystals', 'AmorphousFilm4nm4DSTEM', 'ApoferritinApollo15eps', 'BilayerWS2', 'CuZnEELSMapping', 'CuZnHAADF', 'DE_CenterNet', 'DE_FriedelNet', 'DE_SpotNet', 'DE_SpotNet_Base16', 'DE_SpotNet_Dynamical', 'DE_SpotNet_Production', 'FeAlStripes', 'HREBSDStrainPatterns', 'InSituElectrochemGrowth', 'LSMOLineScan', 'LSMOLineScanLowLoss', 'LSMOSTOLineScan', 'LSMOSTOLineScanLowLoss', 'LayeredCuNb4DSTEM', 'MOSS6Fig3', 'MgONanoCrystals', 'NiEBSDLarge', 'PdCuSiCrystallization', 'PdNiPGlass', 'PeakDetectionPolymers', 'SPEDAg', 'TutorialUNet', 'TwistedBilayerWSe2Themis', 'ZrNbPrecipitate'] \ No newline at end of file +__all__ = ['AlNanocrystals', 'AmorphousFilm4nm4DSTEM', 'ApoferritinApollo15eps', 'BilayerWS2', 'CuZnEELSMapping', 'CuZnHAADF', 'DE_CenterNet', 'DE_CenterNet_Torch', 'DE_FriedelNet', 'DE_FriedelNet_Torch', 'DE_SpotNet', 'DE_SpotNet_Base16', 'DE_SpotNet_Base16_Torch', 'DE_SpotNet_Dynamical', 'DE_SpotNet_Dynamical_Torch', 'DE_SpotNet_Production', 'DE_SpotNet_Production_Torch', 'DE_SpotNet_Torch', 'FeAlStripes', 'HREBSDStrainPatterns', 'InSituElectrochemGrowth', 'LSMOLineScan', 'LSMOLineScanLowLoss', 'LSMOSTOLineScan', 'LSMOSTOLineScanLowLoss', 'LayeredCuNb4DSTEM', 'MOSS6Fig3', 'MgONanoCrystals', 'NiEBSDLarge', 'PdCuSiCrystallization', 'PdNiPGlass', 'PeakDetectionPolymers', 'SPEDAg', 'TutorialUNet', 'TwistedBilayerWSe2Themis', 'ZrNbPrecipitate'] \ No newline at end of file diff --git a/emdatabase/index/DE_CenterNet.yaml b/emdatabase/index/DE_CenterNet.yaml index 6fdba4c..795fafd 100644 --- a/emdatabase/index/DE_CenterNet.yaml +++ b/emdatabase/index/DE_CenterNet.yaml @@ -6,10 +6,10 @@ DE_CenterNet: positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the Friedel error of the detector's centers from 1.13 to 0.84 px. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for - inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is - de_centernet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, - analyzing your own data at a company) is permitted; the weights may not be redistributed in, - or incorporated into, software or services that are sold. + inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the + DE_CenterNet_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your + own data at a company) is permitted; the weights may not be redistributed in, or + incorporated into, software or services that are sold. source: https://zenodo.org/records/23383572/files file: de_centernet.onnx license: CC-BY-NC-4.0 diff --git a/emdatabase/index/DE_CenterNet_Torch.yaml b/emdatabase/index/DE_CenterNet_Torch.yaml new file mode 100644 index 0000000..19d00d3 --- /dev/null +++ b/emdatabase/index/DE_CenterNet_Torch.yaml @@ -0,0 +1,41 @@ +# $schema: ./json-schema.json +DE_CenterNet_Torch: + description: >- + Center refinement for DE SpotNet detections: segments each detected disk in a window of the + raw pattern at native resolution and takes the coverage centroid, with a sigma head giving a + positional uncertainty. Used for spot radii of 5 px and above; on real GaN it lowers the + Friedel error of the detector's centers from 1.13 to 0.84 px. The torch checkpoint, for + training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded + with torch.load(path, weights_only=True). The ONNX graph for inference is the DE_CenterNet + entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own data at a + company) is permitted; the weights may not be redistributed in, or incorporated into, + software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_centernet.pt + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Subpixel refinement + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: quantem.core.ml.CenterNet + framework: torch + latest: + url: https://zenodo.org/records/23383572/files/de_centernet.pt + checksum: md5:ee19c495f722192886f1a2b0852a5acb + size_bytes: 141658 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_centernet.pt + checksum: md5:ee19c495f722192886f1a2b0852a5acb + size_bytes: 141658 diff --git a/emdatabase/index/DE_FriedelNet.yaml b/emdatabase/index/DE_FriedelNet.yaml index ce741f7..eb4b96f 100644 --- a/emdatabase/index/DE_FriedelNet.yaml +++ b/emdatabase/index/DE_FriedelNet.yaml @@ -5,10 +5,10 @@ DE_FriedelNet: together with the 180-degree-rotated window at its Friedel mirror point, so it sees both dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the - model metadata) for inference with ONNX Runtime; the matching torch checkpoint for training - and fine-tuning is de_friedelnet.pt in the same record. Licensed CC BY-NC 4.0: commercial - use (for example, analyzing your own data at a company) is permitted; the weights may not be - redistributed in, or incorporated into, software or services that are sold. + model metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine- + tuning is the DE_FriedelNet_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, + analyzing your own data at a company) is permitted; the weights may not be redistributed in, + or incorporated into, software or services that are sold. source: https://zenodo.org/records/23383572/files file: de_friedelnet.onnx license: CC-BY-NC-4.0 diff --git a/emdatabase/index/DE_FriedelNet_Torch.yaml b/emdatabase/index/DE_FriedelNet_Torch.yaml new file mode 100644 index 0000000..d5e418f --- /dev/null +++ b/emdatabase/index/DE_FriedelNet_Torch.yaml @@ -0,0 +1,41 @@ +# $schema: ./json-schema.json +DE_FriedelNet_Torch: + description: >- + Friedel-partner center refinement, applied after DE CenterNet: reads each disk's window + together with the 180-degree-rotated window at its Friedel mirror point, so it sees both + dynamical fills of g and -g. Trained on disks of 6-16 px radius and leaves others unchanged. + The torch checkpoint, for training and fine-tuning: a dict with the state_dict and the + architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for + inference is the DE_FriedelNet entry. Licensed CC BY-NC 4.0: commercial use (for example, + analyzing your own data at a company) is permitted; the weights may not be redistributed in, + or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_friedelnet.pt + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Subpixel refinement + - Friedel pairs + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: quantem.core.ml.CenterNet + framework: torch + latest: + url: https://zenodo.org/records/23383572/files/de_friedelnet.pt + checksum: md5:46436243d6864386f19f6b1dbd2b74f4 + size_bytes: 141218 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_friedelnet.pt + checksum: md5:46436243d6864386f19f6b1dbd2b74f4 + size_bytes: 141218 diff --git a/emdatabase/index/DE_SpotNet.yaml b/emdatabase/index/DE_SpotNet.yaml index 6b75682..07b137b 100644 --- a/emdatabase/index/DE_SpotNet.yaml +++ b/emdatabase/index/DE_SpotNet.yaml @@ -7,10 +7,10 @@ DE_SpotNet: autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel offset. Trained across disk sizes, so it tolerates residual error in the rescale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model - metadata) for inference with ONNX Runtime; the matching torch checkpoint for training and - fine-tuning is de_spotnet.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for - example, analyzing your own data at a company) is permitted; the weights may not be - redistributed in, or incorporated into, software or services that are sold. + metadata) for inference with ONNX Runtime; the torch checkpoint for training and fine-tuning + is the DE_SpotNet_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing + your own data at a company) is permitted; the weights may not be redistributed in, or + incorporated into, software or services that are sold. source: https://zenodo.org/records/23383572/files file: de_spotnet.onnx license: CC-BY-NC-4.0 diff --git a/emdatabase/index/DE_SpotNet_Base16.yaml b/emdatabase/index/DE_SpotNet_Base16.yaml index 7efc80c..0524d18 100644 --- a/emdatabase/index/DE_SpotNet_Base16.yaml +++ b/emdatabase/index/DE_SpotNet_Base16.yaml @@ -3,10 +3,10 @@ DE_SpotNet_Base16: description: >- Earlier edge-trained DE SpotNet detector, kept as a fallback. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for - inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is - de_spotnet_base16.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for example, - analyzing your own data at a company) is permitted; the weights may not be redistributed in, - or incorporated into, software or services that are sold. + inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the + DE_SpotNet_Base16_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing + your own data at a company) is permitted; the weights may not be redistributed in, or + incorporated into, software or services that are sold. source: https://zenodo.org/records/23383572/files file: de_spotnet_base16.onnx license: CC-BY-NC-4.0 diff --git a/emdatabase/index/DE_SpotNet_Base16_Torch.yaml b/emdatabase/index/DE_SpotNet_Base16_Torch.yaml new file mode 100644 index 0000000..02af6d6 --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Base16_Torch.yaml @@ -0,0 +1,38 @@ +# $schema: ./json-schema.json +DE_SpotNet_Base16_Torch: + description: >- + Earlier edge-trained DE SpotNet detector, kept as a fallback. The torch checkpoint, for + training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded + with torch.load(path, weights_only=True). The ONNX graph for inference is the + DE_SpotNet_Base16 entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your + own data at a company) is permitted; the weights may not be redistributed in, or + incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet_base16.pt + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: quantem.core.ml.SpotNet + framework: torch + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet_base16.pt + checksum: md5:d6438632aec861815e66f1130b509815 + size_bytes: 493770 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet_base16.pt + checksum: md5:d6438632aec861815e66f1130b509815 + size_bytes: 493770 diff --git a/emdatabase/index/DE_SpotNet_Dynamical.yaml b/emdatabase/index/DE_SpotNet_Dynamical.yaml index 50f544a..8b2c6bf 100644 --- a/emdatabase/index/DE_SpotNet_Dynamical.yaml +++ b/emdatabase/index/DE_SpotNet_Dynamical.yaml @@ -7,10 +7,10 @@ DE_SpotNet_Dynamical: center error on dynamical disks and cuts false positives on sparse counted data; on real SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration in the model metadata) for - inference with ONNX Runtime; the matching torch checkpoint for training and fine-tuning is - de_spotnet_dynamical.pt in the same record. Licensed CC BY-NC 4.0: commercial use (for - example, analyzing your own data at a company) is permitted; the weights may not be - redistributed in, or incorporated into, software or services that are sold. + inference with ONNX Runtime; the torch checkpoint for training and fine-tuning is the + DE_SpotNet_Dynamical_Torch entry. Licensed CC BY-NC 4.0: commercial use (for example, + analyzing your own data at a company) is permitted; the weights may not be redistributed in, + or incorporated into, software or services that are sold. source: https://zenodo.org/records/23383572/files file: de_spotnet_dynamical.onnx license: CC-BY-NC-4.0 diff --git a/emdatabase/index/DE_SpotNet_Dynamical_Torch.yaml b/emdatabase/index/DE_SpotNet_Dynamical_Torch.yaml new file mode 100644 index 0000000..7a1cd5e --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Dynamical_Torch.yaml @@ -0,0 +1,43 @@ +# $schema: ./json-schema.json +DE_SpotNet_Dynamical_Torch: + description: >- + DE SpotNet fine-tuned on multislice (dynamical) disks with counted and integrating detector + noise, then self-trained on real 4D-STEM frames whose labels are persistence- and Friedel- + verified. A disk-coverage head centers each disk on its predicted aperture. Halves the + center error on dynamical disks and cuts false positives on sparse counted data; on real + SPED Ag the in-grain strain speckle drops from 0.063 to 0.036 %. The torch checkpoint, for + training and fine-tuning: a dict with the state_dict and the architecture scalars, loaded + with torch.load(path, weights_only=True). The ONNX graph for inference is the + DE_SpotNet_Dynamical entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing + your own data at a company) is permitted; the weights may not be redistributed in, or + incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet_dynamical.pt + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - Dynamical diffraction + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: quantem.core.ml.SpotNet + framework: torch + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet_dynamical.pt + checksum: md5:d441483802cea81fbb8db58c022149e1 + size_bytes: 516186 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet_dynamical.pt + checksum: md5:d441483802cea81fbb8db58c022149e1 + size_bytes: 516186 diff --git a/emdatabase/index/DE_SpotNet_Production.yaml b/emdatabase/index/DE_SpotNet_Production.yaml index 7b72f42..97314f8 100644 --- a/emdatabase/index/DE_SpotNet_Production.yaml +++ b/emdatabase/index/DE_SpotNet_Production.yaml @@ -4,11 +4,10 @@ DE_SpotNet_Production: DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained scale. An ONNX graph (BatchNorm folded, opset 17, dynamic batch/height/width, configuration - in the model metadata) for inference with ONNX Runtime; the matching torch checkpoint for - training and fine-tuning is de_spotnet_production.pt in the same record. Licensed CC BY-NC - 4.0: commercial use (for example, analyzing your own data at a company) is permitted; the - weights may not be redistributed in, or incorporated into, software or services that are - sold. + in the model metadata) for inference with ONNX Runtime; the torch checkpoint for training + and fine-tuning is the DE_SpotNet_Production_Torch entry. Licensed CC BY-NC 4.0: commercial + use (for example, analyzing your own data at a company) is permitted; the weights may not be + redistributed in, or incorporated into, software or services that are sold. source: https://zenodo.org/records/23383572/files file: de_spotnet_production.onnx license: CC-BY-NC-4.0 diff --git a/emdatabase/index/DE_SpotNet_Production_Torch.yaml b/emdatabase/index/DE_SpotNet_Production_Torch.yaml new file mode 100644 index 0000000..ad6f1de --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Production_Torch.yaml @@ -0,0 +1,39 @@ +# $schema: ./json-schema.json +DE_SpotNet_Production_Torch: + description: >- + DE SpotNet variant with short-range-order augmentation and an edge-decoupled head; slightly + higher recall on crystalline 4D-STEM at the canonical disk size, weaker off its trained + scale. The torch checkpoint, for training and fine-tuning: a dict with the state_dict and + the architecture scalars, loaded with torch.load(path, weights_only=True). The ONNX graph + for inference is the DE_SpotNet_Production entry. Licensed CC BY-NC 4.0: commercial use (for + example, analyzing your own data at a company) is permitted; the weights may not be + redistributed in, or incorporated into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet_production.pt + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: quantem.core.ml.SpotNet + framework: torch + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet_production.pt + checksum: md5:6768177f96f61358d913bde8dcf5346e + size_bytes: 493490 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet_production.pt + checksum: md5:6768177f96f61358d913bde8dcf5346e + size_bytes: 493490 diff --git a/emdatabase/index/DE_SpotNet_Torch.yaml b/emdatabase/index/DE_SpotNet_Torch.yaml new file mode 100644 index 0000000..93ed4bf --- /dev/null +++ b/emdatabase/index/DE_SpotNet_Torch.yaml @@ -0,0 +1,43 @@ +# $schema: ./json-schema.json +DE_SpotNet_Torch: + description: >- + Direct Electron's neural-network diffraction spot (Bragg disk) detector for 4D-STEM and + scanning precession electron diffraction, the default detector in SpyDE, pyxem and quantem. + Each pattern is rescaled so its disks are about 9 px across (diameter from the + autocorrelation) and normalized; the network predicts a peak heat map and a sub-pixel + offset. Trained across disk sizes, so it tolerates residual error in the rescale. The torch + checkpoint, for training and fine-tuning: a dict with the state_dict and the architecture + scalars, loaded with torch.load(path, weights_only=True). The ONNX graph for inference is + the DE_SpotNet entry. Licensed CC BY-NC 4.0: commercial use (for example, analyzing your own + data at a company) is permitted; the weights may not be redistributed in, or incorporated + into, software or services that are sold. + source: https://zenodo.org/records/23383572/files + file: de_spotnet.pt + license: CC-BY-NC-4.0 + technique: + - 4D-STEM + - Diffraction + - ML - peak finding + doi: 10.5281/zenodo.23383571 + tags: + - Machine Learning + - Vector finding + - Bragg disk detection + - U-Net + - Direct Electron + authors: + Carter Francis: + affiliation: Direct Electron + kind: weights + model: + class: quantem.core.ml.SpotNet + framework: torch + latest: + url: https://zenodo.org/records/23383572/files/de_spotnet.pt + checksum: md5:00eec19c5498447f21aa08985ff5badd + size_bytes: 519732 + versions: + "261015": + url: https://zenodo.org/records/23383572/files/de_spotnet.pt + checksum: md5:00eec19c5498447f21aa08985ff5badd + size_bytes: 519732