metadata model, query API, docs and housekeeping

.github/ISSUE_TEMPLATE/new_dataset.yaml

@@ -27,11 +27,19 @@ body:
       placeholder: ex. Jane Doe
     validations:
       required: true
+  - type: input
+    id: affiliation
+    attributes:
+      label: --Affiliation--
+      description: Which institution or company are you at?
+      placeholder: ex. University of Wisconsin-Madison
+    validations:
+      required: true
   - type: input
     id: url
     attributes:
       label: --URL--
-      description: Please provide a URL where the dataset can be downloaded. This should be a direct link to the file itself.
+      description: Please provide a URL where the dataset can be downloaded. This must be a direct link to the file itself - the file name is taken from the end of it.
       placeholder: ex. https://zenodo.org/records/15490547/files/smallPtychography.hspy
     validations:
       required: true
@@ -106,11 +114,12 @@ body:
       description: Select the technique used to acquire the dataset.
       options:
         - 4D-STEM
-        - EBSD
-        - IN SITU TEM
-        - SEM
         - EELS
-        - XEDS
+        - EDS
+        - EBSD
+        - STEM
+        - In-situ TEM
+        - Cryo-EM
         - Other
   - type: input
     id: tags

.github/scripts/issue_to_yaml.py

@@ -1,66 +1,130 @@
-import sys
+"""Turn a filled-in "New Dataset" issue into a dataset YAML file.
+
+Run by ``.github/workflows/on_new_issue.yml``. The output is validated against
+``em_database/datasets/json-schema.json`` before it is written, so a malformed
+issue fails here rather than in the pull request the workflow opens.
+"""
+
 import re
-import yaml
+import sys
+import urllib.request
 from pathlib import Path
 
+import jsonschema
+import yaml
+
+from em_database.metadata import check_vendor, load_schema, load_vendors
+
+FIELDS = {
+    "Dataset Name": r"--Dataset Name--\s*(.*)",
+    "Author": r"--Author--\s*(.*)",
+    "Affiliation": r"--Affiliation--\s*(.*)",
+    "URL": r"--URL--\s*(.*)",
+    "Checksum": r"--Checksum--\s*(.*)",
+    "Description": r"--Description--\s*([\s\S]*?)--Detector Manufacturer--",
+    "Detector Manufacturer": r"--Detector Manufacturer--\s*(.*)",
+    "Detector Model": r"Detector Model\s*(.*)",
+    "Microscope Vendor": r"Microscope Vendor\s*(.*)",
+    "Microscope Model": r"Microscope Model\s*(.*)",
+    "Accelerating Voltage": r"Accelerating Voltage\s*(.*)",
+    "Dataset License": r"Dataset License\s*(.*)",
+    "Technique": r"Technique\s*(.*)",
+    "Tags": r"Tags\s*(.*)",
+}
+
+
+def clean(value):
+    """Trim what GitHub adds around an issue-form answer.
+
+    Answers are separated by ``###`` headings, and the multi-line ones run up
+    to the next heading; an optional field left blank comes through as the
+    literal ``_No response_``.
+    """
+    value = value.strip().rstrip("#").strip()
+    return "" if value == "_No response_" else value
+
+
 def parse_issue_body(text):
-    # Simple regex-based parser for the fields
-    fields = {
-        "Dataset Name": r"--Dataset Name--\s*(.*)",
-        "Author": r"--Author--\s*(.*)",
-        "URL": r"--URL--\s*(.*)",
-        "Checksum": r"--Checksum--\s*(.*)",
-        "Description": r"--Description--\s*([\s\S]*?)--Detector Manufacturer--",
-        "Detector Manufacturer": r"--Detector Manufacturer--\s*(.*)",
-        "Detector Model": r"Detector Model\s*(.*)",
-        "Microscope Vendor": r"Microscope Vendor\s*(.*)",
-        "Microscope Model": r"Microscope Model\s*(.*)",
-        "Accelerating Voltage": r"Accelerating Voltage\s*(.*)",
-        "Dataset License": r"Dataset License\s*(.*)",
-        "Technique": r"Technique\s*(.*)",
-        "Tags": r"Tags\s*(.*)"
-    }
     data = {}
-    for key, pattern in fields.items():
-        m = re.search(pattern, text)
-        if m:
-            data[key] = m.group(1).strip()
-        else:
-            data[key] = ""
+    for key, pattern in FIELDS.items():
+        match = re.search(pattern, text)
+        data[key] = clean(match.group(1)) if match else ""
     return data
 
+
+def content_length(url):
+    """The file's size in bytes, or None if the server will not say."""
+    request = urllib.request.Request(
+        url,
+        method="HEAD",
+        headers={"User-Agent": "em_database (https://github.com/CSSFrancis/em_data)"},
+    )
+    try:
+        with urllib.request.urlopen(request, timeout=60) as response:
+            length = response.headers.get("Content-Length")
+        return int(length) if length else None
+    except Exception:
+        return None
+
+
 def build_yaml(data):
-    # Convert tags to list
-    tags = [t.strip() for t in data["Tags"].split(",") if t.strip()]
-    # Use author as dataset name (sanitize)
-    d_name = re.sub(r'\W+', '', data["Dataset Name"])
-    yaml_data = {
-        d_name: {
-            "description": data["Description"],
-            "source": data["URL"],
-            "detector_manufacturer": data["Detector Manufacturer"],
-            "detector": data["Detector Model"],
-            "microscope_vendor": data["Microscope Vendor"],
-            "microscope": data["Microscope Model"],
-            "voltage": data["Accelerating Voltage"],
-            "license": data["Dataset License"],
-            "technique": data["Technique"],
-            "tags": tags,
-            "authors": {
-                data["Author"]: {}
-            }
-        }
+    """``(document, dataset_name)`` for a parsed issue."""
+    name = re.sub(r"\W+", "", data["Dataset Name"])
+    if not name:
+        sys.exit("the issue has no dataset name")
+    # The form asks for a direct link to the file; the YAML wants the directory
+    # and the file name separately.
+    url = data["URL"].rstrip("/")
+    source, _, filename = url.rpartition("/")
+    if not source or not filename:
+        sys.exit(f"{data['URL']!r} is not a direct link to a file")
+
+    entry = {
+        "description": data["Description"],
+        "source": source,
+        "checksum": data["Checksum"],
+        "file": filename,
+        "size_bytes": content_length(url),
+        "detector_manufacturer": data["Detector Manufacturer"],
+        "detector": data["Detector Model"],
+        "microscope_vendor": data["Microscope Vendor"],
+        "microscope_model": data["Microscope Model"],
+        "voltage": data["Accelerating Voltage"],
+        "license": data["Dataset License"],
+        "technique": data["Technique"],
+        "tags": [t.strip() for t in data["Tags"].split(",") if t.strip()],
     }
-    return yaml_data, dataset_name
+    if data["Author"]:
+        entry["authors"] = {data["Author"]: {"affiliation": data["Affiliation"] or "Unspecified"}}
+    return {name: {k: v for k, v in entry.items() if v not in (None, "", [])}}, name
+
+
+def report_vendors(entry):
+    """Print vendor warnings; exit on a name that is a likely misspelling."""
+    vendors = load_vendors()
+    failed = False
+    for field, known in (
+        ("detector_manufacturer", vendors["detector_manufacturer"]),
+        ("microscope_vendor", vendors["microscope_vendor"]),
+    ):
+        result = check_vendor(entry.get(field, ""), known)
+        if result is None:
+            continue
+        level, message = result
+        print(f"{level}: {field}: {message}")
+        failed = failed or level == "error"
+    if failed:
+        sys.exit("fix the vendor spelling in the issue and reopen it")
+
 
 if __name__ == "__main__":
-    issue_file = sys.argv[1]
-    out_dir = Path(sys.argv[2])
-    with open(issue_file) as f:
-        text = f.read()
-    data = parse_issue_body(text)
-    yaml_data, dataset_name = build_yaml(data)
+    issue_file, out_dir = sys.argv[1], Path(sys.argv[2])
+    document, dataset_name = build_yaml(parse_issue_body(Path(issue_file).read_text()))
+    jsonschema.validate(document, load_schema())
+    report_vendors(document[dataset_name])
+
     out_path = out_dir / f"{dataset_name}.yaml"
     with open(out_path, "w") as f:
         f.write("# $schema: ../json-schema.json\n")
-        yaml.dump(yaml_data, f, sort_keys=False)
+        yaml.dump(document, f, sort_keys=False)
+    print(f"wrote {out_path}")

.github/workflows/build.yml

@@ -9,6 +9,33 @@ env:
   MPLBACKEND: agg
 
 jobs:
+  lint:
+    name: lint and type check
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies and package
+        run: pip install -U -e .'[dev]'
+
+      - name: ruff check
+        run: ruff check --output-format=github .
+
+      - name: ruff format
+        run: ruff format --check --diff .
+
+      - name: basedpyright
+        run: basedpyright
+
+      - name: dataset stub is current
+        run: python -m em_database._create_stubs --check
+
   build-with-pip:
     name: ${{ matrix.os }}-py${{ matrix.python-version }}${{ matrix.LABEL }}
     runs-on: ${{ matrix.os }}

.github/workflows/check_sources.yml

@@ -0,0 +1,29 @@
+name: check sources
+
+# Every dataset's source URL, checked against the server. This is what rots over
+# time - a Zenodo record superseded, a GitHub ref rewritten - and it is invisible
+# until someone tries to download. It is one HEAD request per dataset, so it runs
+# weekly rather than on the six-way matrix of every push.
+on:
+  schedule:
+    - cron: "0 6 * * 1"
+  workflow_dispatch:
+
+jobs:
+  sources:
+    name: source URLs still resolve
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies and package
+        run: pip install -U -e .'[dev]'
+
+      - name: Check every source URL
+        run: pytest -m network -v

.github/workflows/documentation.yaml

@@ -26,6 +26,13 @@ jobs:
         run: |
           make html -C doc
 
+      - name: Upload the built site
+        uses: actions/upload-artifact@v4
+        with:
+          name: html-${{ github.ref_name }}
+          path: doc/build/html
+          retention-days: 14
+
       - name: Deploy to GitHub Pages
         uses: peaceiris/actions-gh-pages@v3
         if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}

.github/workflows/on_new_issue.yml

@@ -17,6 +17,9 @@ jobs:
         with:
           python-version: '3.x'
 
+      - name: Install dependencies and package
+        run: pip install -U -e .'[dev]'
+
       - name: Extract issue body
         run: echo "${{ github.event.issue.body }}" > issue_body.txt
 

.pre-commit-config.yaml

@@ -0,0 +1,7 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.16.5
+    hooks:
+      - id: ruff-check
+        args: [--fix]
+      - id: ruff-format

README.md

@@ -29,11 +29,37 @@ s = hs.load(path, lazy=True)
 ```
 
 By default the download runs on a background thread so a notebook cell returns
-immediately.  The handle it returns *is* the file path, so you can hand it straight
-to a loader as above — it only blocks at the moment the file is actually opened.
-Call `path.done()` to check progress without blocking, or `path.result()` to wait
-explicitly.  Pass `download(background=False)` to block and get the path as a plain
-string instead.
+immediately.  The handle it returns *is* the file path — a `pathlib.Path` subclass
+pointing at the file's final location — so you can hand it straight to a loader as
+above; it only blocks at the moment the file is actually opened.  Read `path.done`
+to check progress without blocking, or call `path.result()` to wait explicitly.
+`download(background=False)` blocks instead, and returns the same type.
+
+Any path pointing at the same file waits, however it was built, so a derived path
+(`handle.parent / handle.name`, `handle.with_suffix(...)`) behaves too.  The
+exceptions are `str(handle)` and `Path(handle)`: both hand back an ordinary value
+with no download attached, so `hs.load(str(handle))` will *not* wait.  Keeping
+`str()` non-blocking is deliberate — `repr()` needs it — so pass the handle itself.
+
+## Finding a dataset
+
+`search()` is the browser widget's search box, callable from Python; `filter()`
+matches named fields.  Both return dataset objects, so a result can be downloaded
+directly.
+
+```python
+import em_database
+
+em_database.list_datasets()                                     # everything
+em_database.search("amorphous")                            # any field
+em_database.search("jeol eels")                            # all terms, any field
+em_database.filter(technique="4D-STEM", tags="Strain")     # exact, case-insensitive
+em_database.filter(microscope_vendor=["JEOL", "Hitachi"])  # a list means any of
+em_database.filter(downloaded=True)                        # what is already here
+```
+
+An unknown field raises rather than being ignored, so a typo cannot quietly return
+the whole index.
 
 ## Settings
 
@@ -88,11 +114,16 @@ from the top-level key:
 MyDataset:
   description: What the data is, how it was acquired and how it is calibrated.
   source: https://zenodo.org/records/<record>/files
-  checksum: md5:<hash>
   file: MyDataset.zspy
-  data_size: 1.2 GB
+  checksum: md5:<hash>
+  size_bytes: 1200000000
   technique: 4D-STEM
   license: CC-BY-4.0
 ```
 
+`size_bytes` is the file's `Content-Length` in bytes; the test suite checks it against
+the server on every run. `em_database/datasets/vendors.yaml` lists the microscope
+vendors and detector manufacturers already in use - a new one is fine, but a name close
+to one already on the list fails CI as a misspelling.
+
 Open an issue with the new dataset template, or add the YAML file directly.