[MattFisher]

This PR contains

Description

This repository was created from inspect-evals-template v1.1.0, whose structural checks ran from a vendored tools/run_autolint. Template v1.2.0 replaced that with the inspect-evals-lint package, which is also what the register's daily lint service runs against this repository at its registered commit. The package removed the noautolint suppression syntax, so the service currently reports this repository's old suppression as a dead marker (src/lab_bench_2/.noautolint). This PR brings the repository onto the package and migrates the suppression.

What changes

• pyproject.toml (yours, so the template sync does not touch it): inspect-evals-lint==0.4.2 in the dev group and a [tool.inspect-evals-lint] table with the template preset.
• The managed files that the template changed for this in v1.2.0, merged three-way against your current copies so nothing local is lost: .github/workflows/checks.yml and tools/run_checks.sh run uv run inspect-evals-lint --all (CI annotates pull requests at the offending lines), AUTOMATED_CHECKS.md documents the rules and the new # inspect-evals-lint: ignore[<rule>] comment syntax, plus AGENTS.md, CONTRIBUTING.md, MANAGED_FILES.md, the two Claude skills and .template-version (v1.2.0). tools/run_autolint/ is removed. The autolint check stays advisory (ENFORCE_AUTOLINT=false), as before.
• src/utils/huggingface.py: the template's own line-level suppression for the deferred datasets import, which the helper package needs until an evaluation here declares datasets.
• src/lab_bench_2/.noautolint is deleted, and neither of its two entries needs replacing:
  • external_dependencies was suppressed because the linter reported the repository's own src/utils helper as an undeclared third-party package. inspect-evals-lint 0.4.2 treats every package under src/ as first-party.
  • tests_init was suppressed because a tests/lab_bench_2/__init__.py made pytest see two modules named lab_bench_2. Adding tests/__init__.py as well fixes that (the test package becomes tests.lab_bench_2); tests/lab_bench_2/__init__.py and tests/lab_bench_2/solvers/__init__.py are added too. All 242 tests still collect and mypy is clean.
• [project.optional-dependencies].lab_bench_2 gains pillow and pydantic, which attachment_builder.py and scorers.py import directly but which were only available transitively. (from evals.utils import MEDIA_TYPES resolves to the declared labbench2 distribution once it is installed, so it needs nothing.)

Result (uv run inspect-evals-lint --all with 0.4.2): utils passes. lab_bench_2 has one remaining failure, left for a separate decision: Score.unscored() at scorers.py:102 passes no reason=, which needs inspect_ai>=0.3.261 (the repository pins >=0.3.234). Three warnings do not fail the run: get_model() outside a @scorer in call_grader_with_retry, and two dockerfile_locking findings in solvers/Dockerfile (undigested FROM aisiuk/inspect-tool-support, an unlocked pip install list).

Verification: uv run pytest --collect-only, uv run mypy src tests and uv run ruff check/ruff format --check pass locally on this branch; uv lock is refreshed.

Checklist

• Are you adding a new eval?

  • If yes, please review the Evaluation Checklist.

• Does this change affect existing eval(s)?

  • If yes, have the affected task version(s) been incremented? To check if the version should be incremented, see when to bump the task version.

• Does this change affect how future contributors write or submit evaluations (e.g. new required fields, changed tooling, updated conventions)?

  • If yes, has the relevant documentation been updated (e.g. CONTRIBUTING.md, EVALUATION_CHECKLIST.md, AGENTS.md)? AUTOMATED_CHECKS.md, CONTRIBUTING.md and AGENTS.md are updated from the template.

Posted by Claude Code on Matt Fisher's behalf. Matt maintains the inspect_evals register that lists this evaluation.

🤖 Generated with Claude Code