hygiene: add an escapes mode — LaTeX eaten by Python string escapes

agents/conductors/hygiene/AGENTS.md

@@ -26,7 +26,7 @@ kinds, which is what makes its count comparable (or not):
 
 - **debris** — finds directly-removable items; a real, rankable count (`tidy`).
 - **finding** — confirms a source-quality defect; a real, rankable count
-  (`docstrings`, `refs`, `optdeps`, `extras`).
+  (`docstrings`, `escapes`, `refs`, `optdeps`, `extras`).
 - **timing** — measures import cost; a real, rankable count of *slow* imports (`perf`).
 - **surface** — only *sizes* the audit; the real problems emerge when the
   delegated skill runs, so the count is **not** a problem count (`deps`, `docs`).
@@ -63,7 +63,8 @@ than its category — because category alone gets it wrong. The config layer is
 | ships a distribution | has a `pyproject.toml` | `deps` |
 | ships api docs | has a `docs/api/` tree | `docs` |
 
-The helper-backed modes (`docstrings`, `refs`, `optdeps`, `extras`, `config`)
+The helper-backed modes (`docstrings`, `escapes`, `refs`, `optdeps`, `extras`,
+`config`)
 are **not** on this list: they discover their own targets by walking the scan
 root for workspace-shaped directories, so they can find material the map never
 names — and they keep reporting even when the repo-array modes are `unscanned`.
@@ -78,13 +79,14 @@ names — and they keep reporting even when the repo-array modes are `unscanned`
 | `docs` | `docs/api/*.rst` + `currentmodule` counts across every managed repo shipping a `docs/api/` tree (**surface**) | `/audit_docs` (Heart, imports) |
 | `crlf` | executable scripts (`.sh` + shebang-`755` `.py`) with CRLF — the shebang breaks on Linux/HPC (**debris**, the ranked count); plain `.py` CRLF is reported separately as *cosmetic* (Python reads it fine — don't mass-normalise) | `/refactor` + `.gitattributes eol=lf` |
 | `docstrings` | consecutive module-level triple-quoted expressions separated only by whitespace in user-facing `*_workspace` and `HowTo*` root `*.py` entry scripts and `scripts/**/*.py` files (**finding**) | `/refactor` (mechanically merge each confirmed boundary) |
+| `escapes` | LaTeX in non-raw docstrings eaten by Python's escape handling in user-facing `*_workspace` and `HowTo*` `scripts/**/*.py`, in **two** classes (**finding**). `\s`/`\l`/`\[` are escapes Python does not recognise: it keeps them literal but warns. `\t` in `\theta`, `\f` in `\frac`, `\r` in `\rm`, `\b` in `\beta` are escapes it **does** recognise: the value is silently corrupted (`\theta_E` becomes TAB + `heta_E`) with **no diagnostic of any kind**, so a warning-only sweep reports a clean repo over a corrupted one. Both classes are counted separately and files with **only** silent damage are marked, because those are exactly the ones a warning sweep misses. The scan collects `DeprecationWarning` as well as `SyntaxWarning`: invalid escapes are only the latter on Python 3.12+, so a `SyntaxWarning`-only sweep returns a vacuous zero on 3.11 that is indistinguishable from "already fixed" | `/refactor` (prefix `r` on the enclosing docstring — **not** doubling the backslashes, which would leak into the rendered notebook prose) |
 | `refs` | file/folder references in user-facing `*_workspace` and `HowTo*` prose (`scripts/**/*.py` docstrings + comments, every `scripts/**/README.md` and `config/**/README.md`, and the top-level README) whose target no longer exists — restructure debt no health sweep can see, since the scripts still run (**finding**). Covers the README idioms a `scripts/`-anchored matcher cannot see: structure-list bullets (``- `slam_pipeline`: ``), slash-less relative folder paths (`data_preparation/imaging`), and config YAML names. Scans the **inverse direction** too: a folder that exists but whose own parent README never names it — a package can ship fully working, with every reference resolving, and still be invisible to a reader browsing the folder list (`interferometer/features/datacube` sat unlisted for three months) | `/refactor` (re-point each dead reference; add an entry for each undocumented folder, sourced from that folder's own README or a script docstring) |
 | `optdeps` | smoke-listed workspace scripts that construct an optional-dependency-gated API (`TransformerNUFFT` → `nufftax`) without the house `find_spec` skip guard, so they hard-fail the CI matrices that omit the extras (**finding**). AST-confirmed — prose mentions don't count; scripts outside `smoke_tests.txt` are never flagged | `/refactor` (add the skip guard) |
 | `extras` | the complement of `optdeps`: an optional dependency a library **declares** (in the `[optional]` extra `mode=release` installs) that the `workspace-validation.yml` **`mode=smoke`** leg never installs (**finding**). The extras chain only reaches each library's own `[jax]`, never a sibling's `[optional]`, so those need hand-adding and silently drift — the symptom is a script red in smoke and **green in release** | `/bug` (add the install; fix the install set, **never** the script) |
 | `config` | library `config/*.yaml` keys missing from the matching workspace config — recursive diff (**surface**) | `/refactor` (mirror keys) |
 | `artifacts` | tracked files that look like leaked run outputs / stray data (under `output/`, or data-ext outside fixtures) (**debris**) | `/repo_cleanup` (gitignore + `git rm --cached`) |
 | `packaging` | ignored, fully-untracked top-level `*.egg-info/` and `build/` directories in the managed code repos (**debris**) | preview then run `PyAutoBrain/bin/clean_slate.sh --packaging`; repo-set, exact-name, root-depth and tracked-file guards apply |
-| *(default)* | all of the above (**perf timing deferred** — it spawns real imports) | a ranked `HygieneDecision` worklist — recommends the highest-count direct mode (`tidy`/`crlf`/`docstrings`/`refs`/`artifacts`/`packaging`), then `hygiene perf`, then the periodic surface audits |
+| *(default)* | all of the above (**perf timing deferred** — it spawns real imports) | a ranked `HygieneDecision` worklist — recommends the highest-count direct mode (`tidy`/`crlf`/`docstrings`/`escapes`/`refs`/`artifacts`/`packaging`), then `hygiene perf`, then the periodic surface audits |
 
 ```
 pyauto-brain hygiene              # pre-scan across modes → ranked worklist
@@ -97,6 +99,7 @@ pyauto-brain hygiene deps         # dependency-cap surface → /dep_audit
 pyauto-brain hygiene docs         # API-docs surface → /audit_docs
 pyauto-brain hygiene crlf         # CRLF .py files → /refactor
 pyauto-brain hygiene docstrings   # adjacent top-level documentation → /refactor
+pyauto-brain hygiene escapes      # LaTeX eaten by string escapes (warned + SILENT) → /refactor
 pyauto-brain hygiene refs         # folder-list drift in workspace prose → /refactor
 pyauto-brain hygiene optdeps      # smoke-listed scripts missing an optional-dep skip guard → /refactor
 pyauto-brain hygiene extras       # optional deps the smoke CI leg never installs → /bug

agents/conductors/hygiene/_hygiene_escapes.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""Read-only scanner for LaTeX corrupted by Python's string-escape handling.
+
+Workspace tutorial prose carries LaTeX in module-level docstrings. Unless the
+docstring is raw, Python's escape handling damages it in two INDEPENDENT ways,
+and a scan that looks for only the first reports a clean sweep over a corpus
+full of the second:
+
+warned  ``\\s``, ``\\l``, ``\\[`` -- escapes Python does NOT recognise. It keeps
+        them literal but warns on every compile, and they are slated to become a
+        SyntaxError.
+silent  ``\\t`` in ``\\theta``, ``\\f`` in ``\\frac``, ``\\r`` in ``\\rm``,
+        ``\\b`` in ``\\beta``. Escapes Python DOES recognise: the value is
+        corrupted and there is NO diagnostic of any kind. ``\\theta_E`` becomes
+        a TAB followed by ``heta_E``.
+
+TWO INTERPRETER TRAPS, both of which have already produced a false "clean":
+
+1. Invalid escapes are a ``SyntaxWarning`` only on Python 3.12+; on 3.11 and
+   earlier they are a ``DeprecationWarning``. A ``SyntaxWarning``-only sweep
+   returns zero on 3.11 and is indistinguishable from "already fixed". Both
+   categories are collected here, so the scan is interpreter-independent.
+2. ``compileall`` needs ``-f``, or ``__pycache__`` suppresses recompilation and
+   the counts silently drop. This module compiles from source text and never
+   consults a cache.
+
+Read-only, stdlib-only, and consistent with the rest of the Hygiene Agent: it
+reports and delegates to /refactor; it never edits a file.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import json
+import warnings
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+SKIP_PARTS = {".git", "__pycache__", "build", "dist", ".ipynb_checkpoints"}
+# A newline in a string value is ordinary text, not evidence of a mangled macro.
+CONTROL_EXCEPT_NEWLINE = "\n"
+
+
+@dataclass(frozen=True)
+class Finding:
+    repo: str
+    file: str
+    warned: int
+    silent: int
+
+
+@dataclass(frozen=True)
+class ParseError:
+    repo: str
+    file: str
+    message: str
+
+
+def repository_paths(root: Path) -> list[Path]:
+    """Return user-facing ``*_workspace`` and ``HowTo*`` repositories."""
+    candidates = [*root.glob("*_workspace"), *root.glob("HowTo*")]
+    return sorted(
+        {path.resolve() for path in candidates if (path / "scripts").is_dir()},
+        key=lambda path: path.name.lower(),
+    )
+
+
+def warned_count(source: str, path: Path) -> int:
+    """Escapes Python does not recognise, on ANY interpreter version."""
+    with warnings.catch_warnings(record=True) as caught:
+        warnings.simplefilter("always")
+        try:
+            compile(source, str(path), "exec")
+        except SyntaxError:
+            return 0
+    return sum(
+        1
+        for item in caught
+        # Both categories: SyntaxWarning on 3.12+, DeprecationWarning on <=3.11.
+        if issubclass(item.category, (SyntaxWarning, DeprecationWarning))
+        and "invalid escape sequence" in str(item.message)
+    )
+
+
+def silent_count(source: str) -> int:
+    """Escapes Python DOES recognise, which corrupt the value with no warning.
+
+    A non-raw literal whose SOURCE carries a backslash but whose VALUE carries a
+    control character has had a macro eaten -- ``\\theta`` became TAB + ``heta``.
+    """
+    try:
+        tree = ast.parse(source)
+    except SyntaxError:
+        return 0
+    total = 0
+    for node in ast.walk(tree):
+        if not (isinstance(node, ast.Constant) and isinstance(node.value, str)):
+            continue
+        segment = ast.get_source_segment(source, node) or ""
+        if "\\" not in segment:
+            continue
+        if segment[:1] in "rR" or segment[:2].lower() in ("br", "rb", "fr", "rf"):
+            continue  # already raw: the escape never happened
+        if any(
+            ord(char) < 32 and char != CONTROL_EXCEPT_NEWLINE for char in node.value
+        ):
+            total += 1
+    return total
+
+
+def scan(root: Path) -> tuple[list[Finding], list[ParseError], int]:
+    findings: list[Finding] = []
+    errors: list[ParseError] = []
+    repositories = repository_paths(root)
+    for repository in repositories:
+        for script in sorted((repository / "scripts").rglob("*.py")):
+            if SKIP_PARTS & set(script.parts):
+                continue
+            try:
+                source = script.read_text(encoding="utf-8")
+            except (OSError, UnicodeDecodeError) as error:
+                errors.append(
+                    ParseError(repository.name, str(script.relative_to(repository)), str(error))
+                )
+                continue
+            warned, silent = warned_count(source, script), silent_count(source)
+            if warned or silent:
+                findings.append(
+                    Finding(
+                        repository.name,
+                        str(script.relative_to(repository)),
+                        warned,
+                        silent,
+                    )
+                )
+    return findings, errors, len(repositories)
+
+
+def summary_for(findings: list[Finding], errors: list[ParseError], repos: int) -> str:
+    warned = sum(finding.warned for finding in findings)
+    silent = sum(finding.silent for finding in findings)
+    silent_only = sum(1 for finding in findings if finding.silent and not finding.warned)
+    return (
+        f"{len(findings)} script(s) across {repos} repo(s) with LaTeX damaged by "
+        f"string escapes: {warned} warned, {silent} silent "
+        f"({silent_only} file(s) have ONLY silent damage, which a warning-only "
+        f"sweep would miss); {len(errors)} read error(s)"
+    )
+
+
+def row_for(root: Path) -> dict:
+    findings, errors, repository_count = scan(root)
+    if errors:
+        status = "partial"
+    elif findings:
+        status = "finding"
+    else:
+        status = "clean"
+    return {
+        "mode": "escapes",
+        "kind": "finding",
+        "status": status,
+        "count": len(findings),
+        "summary": summary_for(findings, errors, repository_count),
+        "delegate": "/refactor",
+        "findings": [asdict(finding) for finding in findings],
+        "parse_errors": [asdict(error) for error in errors],
+    }
+
+
+def render_human(row: dict) -> None:
+    print(row["summary"])
+    for finding in row["findings"]:
+        marker = "  <- silent only" if finding["silent"] and not finding["warned"] else ""
+        print(
+            f"  {finding['repo']}/{finding['file']}: "
+            f"{finding['warned']} warned, {finding['silent']} silent{marker}"
+        )
+    if row["parse_errors"]:
+        print("Read errors (scan incomplete):")
+        for error in row["parse_errors"]:
+            print(f"  {error['repo']}/{error['file']}: {error['message']}")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--root", type=Path, required=True)
+    output = parser.add_mutually_exclusive_group()
+    output.add_argument("--json-row", action="store_true")
+    output.add_argument("--summary", action="store_true")
+    args = parser.parse_args()
+
+    row = row_for(args.root.resolve())
+    if args.json_row:
+        print(json.dumps(row, sort_keys=True))
+    elif args.summary:
+        print(f"{row['count']}|{row['summary']}")
+    else:
+        render_human(row)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())