Adopt inspect-evals-lint 0.4.2 (template v1.2.0) and retire the .noautolint suppressions

.claude/skills/create-eval/SKILL.md

@@ -461,7 +461,7 @@ Fix any failures. For tests requiring Docker or datasets, note which tests need
 ### 6.1 Run Autolint
 
 ```bash
-uv run python tools/run_autolint.py <eval_name>
+uv run inspect-evals-lint <eval_name>
 ```
 
 The autolint surfaces the registry's recommended structural standards. In

.claude/skills/ensure-test-coverage/SKILL.md

@@ -71,7 +71,7 @@ Check if the eval has multiple `@task` functions or a single task with parameter
 Run all autolint checks in one command to get a baseline of structural test coverage:
 
 ```bash
-uv run python tools/run_autolint.py <eval_name>
+uv run inspect-evals-lint <eval_name>
 ```
 
 This checks: `tests_exist`, `tests_init`, `e2e_test`, `record_to_sample_test`, `custom_solver_tests`, `custom_scorer_tests`, `custom_tool_tests` (plus non-test checks). You do not need to report to the user specific details of this beyond what the autolint tool itself tells them.
@@ -307,7 +307,7 @@ uv run ruff check tests/<eval_name>/
 uv run ruff format tests/<eval_name>/
 
 # Re-run autolint to verify all checks pass
-uv run python tools/run_autolint.py <eval_name>
+uv run inspect-evals-lint <eval_name>
 
 # Re-measure coverage
 uv run pytest tests/<eval_name>/ --cov=src/<eval_name> --cov-report=term-missing -q --runslow
@@ -320,7 +320,7 @@ Fix any failures before presenting results.
 Show the user a summary:
 
 1. **Component inventory**: What testable components were found
-2. **Autolint results**: Pass/fail for each check (from `run_autolint.py` output)
+2. **Autolint results**: Pass/fail for each check (from `inspect-evals-lint` output)
 3. **Coverage**: Overall and per-file percentages, with missed line analysis
 4. **Quality gaps**: Issues found in Phase 2.3
 5. **Actions taken**: If tests were created/modified, list what was done

.github/workflows/checks.yml

@@ -120,7 +120,7 @@ jobs:
       - uses: hynek/build-and-inspect-python-package@v2
 
   autolint:
-    name: Autolint (inspect_evals standards)
+    name: Autolint (inspect-evals-lint)
     runs-on: ubuntu-latest
     needs: config
     # Default: advisory. Set ENFORCE_AUTOLINT=true in tools/enforcement.config
@@ -134,8 +134,8 @@ jobs:
           python-version: "3.12"
       - name: Install dependencies
         run: uv sync --frozen
-      - name: Run autolint
-        run: uv run python tools/run_autolint.py --all-evals
+      - name: Run inspect-evals-lint
+        run: uv run inspect-evals-lint --all --output-format github
 
   generated-docs:
     name: Generated READMEs up to date

.github/workflows/sync-template.yml

@@ -13,7 +13,7 @@ permissions:
 jobs:
   check-activity:
     runs-on: ubuntu-latest
-    if: github.repository != 'ArcadiaImpact/inspect-evals-template'
+    if: github.repository != 'Generality-Labs/inspect-evals-template'
     outputs:
       is_active: ${{ steps.activity.outputs.is_active }}
     steps:
@@ -54,7 +54,7 @@ jobs:
 
       - name: Add template remote
         run: |
-          git remote add template https://github.com/ArcadiaImpact/inspect-evals-template.git 2>/dev/null || true
+          git remote add template https://github.com/Generality-Labs/inspect-evals-template.git 2>/dev/null || true
           git fetch template main
 
       - name: Extract managed files list
@@ -154,7 +154,7 @@ jobs:
 
           # Build PR body
           body="## Template Sync"
-          body="$body\n\nThis PR updates managed files from the [inspect-eval-template](https://github.com/ArcadiaImpact/inspect-eval-template)."
+          body="$body\n\nThis PR updates managed files from the [inspect-evals-template](https://github.com/Generality-Labs/inspect-evals-template)."
           body="$body\n\n**Template version:** $template_version"
 
           if [ -n "$conflicts" ]; then

.template-sync-sha

@@ -1 +1 @@
-aa74b30159bea953e238d4021315e24e4d60ad38
+d2caec89178e122a6f6aa0872a880c7c2920e419

.template-version

@@ -1 +1 @@
-v1.1.0
+v1.2.0

AGENTS.md

@@ -90,7 +90,7 @@ This workflow runs a series of workflows each in turn. Each workflow is to be ru
 
 ### Useful Commands
 
-1. You can see our linting in the `.github/workflows/checks.yml` file. Run `make check` (which delegates to `tools/run_checks.sh`) when checking linting locally — it runs ruff, mypy, autolint, and the rest in one go and reports advisory vs enforced failures.
+1. You can see our linting in the `.github/workflows/checks.yml` file. Run `make check` (which delegates to `tools/run_checks.sh`) when checking linting locally — it runs ruff, mypy, `inspect-evals-lint`, and the rest in one go and reports advisory vs enforced failures.
 2. To run tests, run `uv run pytest tests/<eval_name>`.
 3. To run evaluations, run `uv run inspect eval <eval_name>/<task_name>` (the eval name comes from the entry-point registered in `pyproject.toml`).
 4. To run a specific task (i.e, a function with the @task decorator), run `uv run inspect eval <eval_name>/<task_name>`.

AUTOMATED_CHECKS.md

@@ -7,24 +7,30 @@
 > are **recommended, not required** — autolint is advisory by default. See
 > [Checks and enforcement](README.md#checks-and-enforcement) to opt in.
 
-We have a number of automatic checks that can be run using [tools/run_autolint.py](tools/run_autolint.py):
+The automated checks are provided by the [inspect-evals-lint](https://github.com/Generality-Labs/inspect-evals-lint) package. It is installed through the `dev` dependency group and configured under `[tool.inspect-evals-lint]` in `pyproject.toml` (the `template` layout preset). Run it with:
 
 ```bash
-uv run python tools/run_autolint.py <eval_name>
+uv run inspect-evals-lint <eval_name>
+uv run inspect-evals-lint --all               # every evaluation and src/utils
+uv run inspect-evals-lint --all --select IEBP # one category, or one rule by code or name
+uv run inspect-evals-lint --explain IEBP002   # a rule's documentation
 ```
 
-To suppress a check, use:
+Every check is a *rule* with a code (`IEFS` file structure, `IECQ` code quality, `IETS` tests, `IEBP` best practices, then a number) and a name; either works in `--select`, `--ignore` and suppression comments. A rule reports one finding per site with a file and line, so CI annotates pull requests at the offending lines.
 
-- Line-level: `# noautolint: <check_name>`
-- File-level: `# noautolint-file: <check_name>` (at top of file)
-- Directory-level: Add check name to `src/<eval_name>/<subdir>/.noautolint`
-- Eval-level: Add check name to `src/<eval_name>/.noautolint`
+Evaluations live directly under `src/`, one package each, registered under `[project.entry-points.inspect_ai]`. `src/utils` is linted as a helper package: it runs the checks about code behaviour (private imports, score values, model roles, dependencies, tests for custom components) and not the ones about an evaluation's structure. `src/examples` is not linted. The package version is pinned in `pyproject.toml` and `uv.lock`; check implementations and their unit tests are maintained in the package repository.
 
-You don't need to read these checks - they are presented here as a reference in case of linting errors.
+To suppress a finding:
+
+- Line-level: `# inspect-evals-lint: ignore[<rule>]` on the offending line (or any line of a multi-line statement), with a rule name or code, several comma-separated. A bare `ignore` is a configuration error.
+- File-level: `# inspect-evals-lint: ignore-file[<rule>]` within the first ten lines of the file.
+- Directory or package: `per-file-ignores = { "src/<eval_name>/<subdir>/**" = ["<rule>"] }` under `[tool.inspect-evals-lint]` in `pyproject.toml`; `exclude` lists globs the code rules never read at all, for code shipped into a sandbox.
+
+You don't need to read these checks - they are presented here as a reference in case of linting errors. The canonical description of each rule, with the reasoning behind it, is its page at [inspect-evals-lint.generality.org](https://inspect-evals-lint.generality.org/CHECKS/); `--explain <rule>` prints the same text.
 
 ## File Structure (Automated)
 
-- The evaluation is located in a sub-directory of `src/` (`eval_location`)
+- The evaluation is located in a sub-directory of `src/` (`package_location`)
 - `__init__.py` exports task and related functions (`init_exports`)
 - @task functions are contained within `src/<eval_name>/<eval_name>.py` (`main_file`)
 - Task registered in `pyproject.toml` under `[project.entry-points.inspect_ai]` (`registry`)
@@ -35,7 +41,9 @@ You don't need to read these checks - they are presented here as a reference in
 
 - No imports from private inspect_ai modules (those starting with `_`) (`private_api_imports`)
 - Score() calls use CORRECT/INCORRECT constants instead of literal strings (`score_constants`)
-- External eval-specific dependencies declared in `pyproject.toml` (`external_dependencies`)
+- `Score.unscored()` calls pass a `reason=`, and the former `metadata["unscored_reason"]` key does not appear (`unscored_reason`)
+- Every suppression marker is one the linter reads (`suppression_syntax`, warns): a `# noautolint` comment or `.noautolint` file, a bare `ignore`, an `ignore-file` past the header or a selector naming no rule suppresses nothing, so it is reported at its line with the replacement in the hint.
+- External eval-specific dependencies declared in `pyproject.toml` (`external_dependencies`). For `src/utils`, imports at module level must be in `[project].dependencies`, because every evaluation that imports the helper loads them; imports inside a function only need declaring in some optional group.
 
 ## Tests (Automated)
 
@@ -50,6 +58,10 @@ You don't need to read these checks - they are presented here as a reference in
 ## Best Practices (Automated)
 
 - `get_model()` only called inside @solver/@scorer decorated functions (`get_model_location`)
+- Model roles supply a model, a `default=`, or `required=True` to prevent an unbound role from falling back to the model under evaluation (`model_role_resolution`)
 - Sample() calls include an `id=` parameter for stable IDs (`sample_ids`)
 - @task functions provide defaults for overridable parameters (solver, scorer, etc.) (`task_overridable_defaults`)
+- Sandbox images pulled from a registry in compose files use an immutable tag or `@sha256` digest (`sandbox_image_pinning`). Untagged and `:latest` references fail; services built locally via `build:` and references interpolated from environment variables are skipped. Exceptions go under `[tool.inspect-evals-lint.allowlists.sandbox_image_pinning]` in `pyproject.toml`.
+- An evaluation whose `eval.yaml` declares `metadata.requires.gpu` ships a `<eval>_sandbox_check` task with `kind: maintenance` that certifies its image on GPU hardware (`gpu_sandbox_check`).
+- Dockerfile builds consume locked inputs (`dockerfile_locking`, warns): dependency installs use a committed `uv.lock` (`uv sync --locked`) or a hashed snapshot (`pip install --require-hashes -r`), `FROM` and `COPY --from` images carry an `@sha256` digest, Git dependencies name a full commit, and nothing is piped from `curl` into a shell. OS package installs cannot be locked and are named in the pass message.
 - Dataset pinning is enforced at runtime: `hf_dataset()`, `load_dataset()`, `snapshot_download()`, and `hf_hub_download()` wrappers require a `revision=` keyword argument

CONTRIBUTING.md

@@ -282,7 +282,7 @@ RUN_DATASET_DOWNLOAD_TESTS=1
 
 #### CI workflows
 
-- The template ships `.github/workflows/checks.yml` which runs ruff, mypy, the POSIX-code check, the unlisted-evals check, the package build, autolint, and a few advisory checks. By default this does not run pytest — the template assumes you run tests locally during development. If you want CI to run your tests, add a job to `checks.yml` (or a separate workflow) that calls `make test`.
+- The template ships `.github/workflows/checks.yml` which runs ruff, mypy, the POSIX-code check, the unlisted-evals check, the package build, `inspect-evals-lint`, and a few advisory checks. By default this does not run pytest — the template assumes you run tests locally during development. If you want CI to run your tests, add a job to `checks.yml` (or a separate workflow) that calls `make test`.
 - The upstream `inspect_evals` registry has additional CI (a `build.yml` that runs the test suite with `RUN_SLOW_TESTS=no`, plus a nightly heavy-tests workflow that detects unmarked slow/docker tests). If your fork wants the same coverage, those workflows are good references but they aren't shipped here.
 
 ### Manual testing
@@ -441,20 +441,36 @@ Each evaluation has an `eval.yaml` file in its directory (e.g., `src/<eval_name>
 
 - `metadata`: (Optional) Object containing system/configuration information. All fields are optional. May include:
   - `sandbox`: List of components that use a sandboxed environment. Can include `"solver"` and/or `"scorer"`. Omit if no sandbox is used.
-  - `requires_internet`: Set to `true` only if the evaluation requires internet access. Omit if false.
+  - `requires`: Capabilities the evaluation needs from its environment. `internet: true` if the runner needs outbound network access beyond fetching packages. `gpu: true` if a GPU is needed where the sandbox runs, or a mapping with `count` and an informational `products` list (e.g. `[T4, H100]`) naming the hardware the eval is designed or verified for. Unknown capability keys are rejected. Omit if neither applies.
   - `environment`: Special environment requirements (e.g., `"Kali-Linux"`). Only include if a non-standard environment is needed.
 
   Example:
 
   ```yaml
   metadata:
     sandbox: ["solver", "scorer"]  # Both solver and scorer use sandbox
-    requires_internet: true
+    requires:
+      internet: true
+      gpu:
+        count: 1
+        products: [T4]
     environment: "Kali-Linux"  # Only include if special environment is needed
   ```
 
   Omit the entire metadata field if none of its subfields are needed.
 
+- `kind` (per task): Set `kind: maintenance` on a task whose scores describe the evaluation harness rather than the model, such as a judge meta-evaluation or a sandbox image check. The default, `benchmark`, is what listings and evaluation reports assume; maintenance tasks are still registered and runnable but are listed separately in generated documentation.
+
+  ```yaml
+  tasks:
+    - name: my_bench
+      dataset_samples: 500
+    - name: my_bench_judge_check
+      dataset_samples: 200
+      kind: maintenance
+      comment: Scores the judge against human labels; not a model result.
+  ```
+
 - `human_baseline`: Optional field for evaluations with known human performance metrics
 
   ```yaml
@@ -483,7 +499,9 @@ dependency: "your_evaluation"  # Optional
 tags: ["Agent"]  # Optional
 metadata:  # optional metadata documenting eval information. All fields optional
   sandbox: []  # list of eval aspects that use a sandbox, can include "solver" and/or "scorer"
-  requires_internet: true  # boolean indicating whether the eval requires internet access
+  requires:  # capabilities the eval needs; omit keys that do not apply
+    internet: true  # runner needs outbound network access
+    gpu: false  # or `true`, or {count: 1, products: [T4]}
   environment: "Kali-Linux"  # optional environment information
 external_assets:  # required; use [] if none
   - type: huggingface

MANAGED_FILES.md

@@ -65,18 +65,6 @@ This means that you can add or remove files to this section as needed.
 - `tools/generate_readmes.py`
 - `tools/inject_managed_prefix.sh`
 - `tools/list_large_files.sh`
-- `tools/run_autolint.py`
-- `tools/run_autolint/__init__.py`
-- `tools/run_autolint/checks/__init__.py`
-- `tools/run_autolint/checks/best_practices.py`
-- `tools/run_autolint/checks/code_quality.py`
-- `tools/run_autolint/checks/dependencies.py`
-- `tools/run_autolint/checks/file_structure.py`
-- `tools/run_autolint/checks/tests.py`
-- `tools/run_autolint/checks/utils.py`
-- `tools/run_autolint/models.py`
-- `tools/run_autolint/output.py`
-- `tools/run_autolint/suppressions.py`
 - `tools/run_checks.sh`
 - `tools/sync_merge_file.sh`
 
@@ -90,7 +78,7 @@ These are yours — the sync mechanism will never modify them:
 - `tests/<your_eval>/` — your tests
 - `tests/conftest.py` — pytest fixtures and markers; the upstream version depends on `inspect_evals` package internals and is not synced
 - `README.md` — your evaluation's documentation
-- `pyproject.toml` — your project configuration (tool configs are managed, but `[project]` and `[project.entry-points]` sections are yours)
+- `pyproject.toml` — your project configuration (tool configs are managed, but `[project]` and `[project.entry-points]` sections are yours). Keep the `inspect-evals-lint` entry in the `dev` dependency group and the `[tool.inspect-evals-lint]` table: `checks.yml` and `tools/run_checks.sh` run `uv run inspect-evals-lint`, which needs the package installed. Repositories created before template v1.2.0 add both by hand.
 - `agent_artefacts/` — runtime artefacts from agent workflows
 - `LICENSE` — your chosen license. By default, the MIT license will be copied over, but this can be altered by you at any time.
 

pyproject.toml

@@ -14,6 +14,13 @@ where = ["src"]
 [tool.setuptools.package-data]
 "*" = ["eval.yaml"]
 
+# Evaluation checks (see AUTOMATED_CHECKS.md). The template preset expects one
+# package per evaluation directly under src/, registered under
+# [project.entry-points.inspect_ai]; src/utils is linted as a helper package
+# and src/examples is not linted.
+[tool.inspect-evals-lint]
+preset = "template"
+
 [tool.ruff]
 extend-exclude = [
     "agent_artefacts",
@@ -160,11 +167,16 @@ lab_bench_2 = "lab_bench_2"
 # Reference implementation's deterministic scoring code (CC BY-SA 4.0), scoped to
 # this eval rather than core. Pulled as a dependency rather than vendored; source
 # + pin in [tool.uv.sources]. See src/lab_bench_2/README.md.
-lab_bench_2 = ["labbench2"]
+lab_bench_2 = [
+    "labbench2",
+    "pillow",     # attachment_builder downscales oversized image attachments
+    "pydantic",   # scorers validate the grader's structured output
+]
 test = []
 
 [dependency-groups]
 dev = [
+    "inspect-evals-lint==0.4.2",  # Evaluation checks; see AUTOMATED_CHECKS.md
     "anthropic>=0.105.0",
     "inspect-scout>=0.3.0",  # llm_scanner API used by agent_artefacts/trajectory_analysis/inspect_scout/
     "mypy",