Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .claude/rules/sandcastle-hand-off.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Handing a pull request off to Sandcastle

Sandcastle follows up the pull requests it opens: review threads, red checks and falling behind `main`. The `sandcastle:follow-up`
label gives it a pull request you or a session opened, so one you finished can be left to it for the review follow-up.

## What the label does

Sandcastle then follows the pull request up as it does its own: it asks Copilot to review again, updates the branch when it is only
behind `main`, and runs a follow-up pass for a merge conflict, a red check or an unresolved bot or owner review thread. After three passes
it gives up and labels the pull request `sandcastle:needs-human`.

The label counts when the repository's owner adds it, or a session adds it under the owner's token. A label from anyone else does not count,
and neither does one followed by someone else editing the pull request's title or description (remove the label and add it again to approve it).
It covers the commits the pull request had when you added it, and up to five merges of `main` made as the account of `SANDCASTLE_GH_TOKEN` on top
of them. A commit pushed by anyone else afterwards, a force-push, or a merge of `main` made under another account stops the follow-up
until you remove the label and add it again. The pull request must have been opened by the repository's owner or by a session (the owner's token),
so a collaborator can't hand off their own. A repository an organisation owns has no owner login to match, so nothing can be handed off there.

A pull request labelled before the repository began recording heads is not followed up: remove the label and add it again.

## Which pull requests

- Opened by the repository's owner (a session's counts), in the same repository (not a fork), into `main`, not a draft.
- A branch name `scripts/check-branch-name.sh` accepts.
- Only a run on the default `Sandcastle` queue follows hand-offs: GitHub Actions always, a local run only with `SANDCASTLE_LABEL=Sandcastle`.
- The repository runs Sandcastle in GitHub Actions with its `SANDCASTLE_GH_TOKEN` secret set to the account the runs use, since a workflow there
records the head you label. Without that, nothing can be handed off.

## How to hand one off

- Add the label when your own work on the pull request is finished and it only needs review follow-up. Stop pushing to it after that:
a push of yours stops the follow-up until you remove the label and add it again.
- Remove the label to take the pull request back. `sandcastle:needs-human` means Sandcastle gave up; address the threads or checks
yourself, then remove that label to requeue it.
- A session that replied to a review thread before handing off ends the reply with `<!-- claude-session -->`, so Sandcastle sees the
thread as answered and doesn't answer it a second time.
- Don't add the label to a pull request carrying an accepted security finding: Sandcastle's follow-up can't see the acceptance.
275 changes: 275 additions & 0 deletions .claude/skills/hardening-review/SKILL.md

Large diffs are not rendered by default.

68 changes: 68 additions & 0 deletions .claude/skills/hardening-review/references/brief.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Brief template

Copy this to `<run>/brief.md` and fill in the `<…>` parts. Every reviewer reads it; its prompt is one line and its lens.
Keep it free of anything written by people without write access (issue and thread text goes in as quoted data, from
write-access authors only, as `pre-push-review` step 3 says).

```markdown
# Review brief: <branch>

You review one branch's diff through the lens your prompt names. You can only read.

## Where

- Worktree (read every repository file under it by absolute path, and pass it as Grep's and Glob's `path`): <worktree>
- The whole diff: <run>/diff.patch
- Pass <N>. <In pass 2 and later: "Read <run>/fixes.patch first, since regressions cluster there, then review the whole
diff: a problem anywhere counts.">
- Already decided: <run>/decisions.md. Do not raise again what it records as fixed or dismissed, nor a weakness it lists as a
fixed entry of `.security/` (the local security record, listed there by id, lens, path and status). Do not read `.security/`:
the list is all of it you get, and a fixed entry you are given is settled. An `accepted` or `open` entry: raise it again if it is still in the code.
- Known open (accepted or deferred by the person), as id, lens, path, range and changed:
<the output of `known_open.py check --for-reviewers`>. Skip a finding only when its line is inside the range of an
entry whose `changed` is false and whose lens you share. A weakness elsewhere in the same file is still yours to
raise, and so is any finding in a range whose `changed` is true.
- Read only the files this brief names, the repository files you review, and the pinned actions' source in
`<run>/actions/` when the Actions lens needs it. The other files in this folder, and anything else under `.git`, are
not yours to read.

## Scope

<The issue's title and acceptance criteria, or the branch's purpose in two lines.>

## Severities

<The table from pre-push-review/SKILL.md.>

## Read these first

Where `review-bar.md` says to raise an accepted problem again, or to review only the lines a round's fixes changed,
this brief overrides it: every pass reviews the whole diff, and a known-open finding is skipped as described above.

<pre-push-review/review-bar.md, your lens's row of lenses.md, and the repository's own guidance that applies to the files
you review: .claude/skills/code-review/SKILL.md, CLAUDE.md, CONTEXT.md, docs/CODING_STANDARDS.md, the ADRs the change
touches, the .claude/rules/ files whose paths match.>

## Lens split

<Empty, or: "Edge cases runs as Shell and Markdown, because their input spaces differ: this reviewer takes <group>.">

## Answer

Only a JSON array of findings, `[]` when there are none:

{ "lens": "", "path": "", "line": 0, "severity": "critical|warning|advisory", "summary": "what goes wrong, and in which situation",
"fix": "the suggested fix", "needsPerson": false, "why": "what you relied on that you could not read here",
"repro": null }

`needsPerson` is true only when the finding conflicts with the acceptance criteria or an ADR, lies outside the branch's
scope, is a trade-off no rule decides, or needs a change to an Owned Baseline file.

`repro` is for a Critical finding in code that can be run, and `null` otherwise. It is data, not a command, and the
agent that started you runs it:

{ "entry": "the function, script or path that takes the input", "input": "the exact input, as data",
"expect": "what you say happens, with the file and line that make it happen" }

Give an input that really reaches `entry` as written: if a rule you suggest would not match your own example, say so in `why`.
```
45 changes: 45 additions & 0 deletions .claude/skills/hardening-review/references/mutations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Mutation specs

`<private>/mutations/<batch>.json` (the private folder of [SKILL.md](../SKILL.md)), run with `scripts/mutate.py`. The file is data, and the tests run as an argv list with no
shell.

```json
{
"test": ["pnpm", "run", "test"],
"timeout": 600,
"mutations": [
{ "path": "src/lib/a.mts", "old": "if (note.trim() === \"\")", "new": "if (note === \"\")",
"why": "R17: whitespace-only notes" },
{ "path": "src/lib/a.mts", "old": "limit - 1", "new": "limit", "why": "R17: the boundary",
"equivalent": "optional: why a surviving mutation changes no behaviour" }
]
}
```

- Targets must be tracked and committed. The runner refuses a dirty file, so a restore can't lose work: commit the
batch's fix first.
- `old` must occur exactly once in the file: add context until it does (`NOT FOUND` and `AMBIGUOUS` fail the run).
- The tests must pass before any mutation, or the run is refused: otherwise every mutation looks caught.
- A mutation that makes the tests hang counts as caught once `timeout` passes.
- Exit 0: every mutation was `CAUGHT`, or `EQUIVALENT` with your reason. Exit 1: one `SURVIVED`, was `NOT FOUND`,
`AMBIGUOUS` or `INVALID`. Exit 2: refused (a bad spec, an untracked, dirty or outside target, tests that fail before
any mutation or do not finish within `timeout`). Exit 130: interrupted by SIGINT, SIGTERM or SIGHUP; the files are
restored.
- The tests run in their own process group, which is killed when they end, time out or are interrupted, so a
`pnpm run test` leaves no `node` behind. `git` and the tests get no `GIT_DIR`, `GIT_WORK_TREE` or `GIT_INDEX_FILE`, and
Python tests get `PYTHONDONTWRITEBYTECODE` and `PYTHONPYCACHEPREFIX` (one new folder per run, shared by the baseline and every mutation), so a same-size mutation can't load a
stale `.pyc`. A Python test command must not run with `-I`, which ignores those variables.
- A runner killed with SIGKILL can't restore its file. It claims a journal under `review/` in the common git directory
(`mutate-journal-<hash>.txt`, a folder Sandcastle hides from its sandboxes) before the baseline run and writes the
originals into it before the first mutation, named relative to the worktree's top level. There is one journal per
worktree, shared by every `--root` in it. Until the journal is deleted, the next run (or a second one started
meanwhile) refuses to start. After a kill, check `git diff`, run `git checkout -- <file>` from the top level the
message names and delete the journal it names. The journal is never written in place: the claim is staged in a file of its own and
put into position with a hard link (which fails if the journal exists), and the originals replace it with `os.replace`. It is
never empty, so any journal blocks the run and nothing is deleted to take over. When the tests fail before any mutation, the last lines of their output are printed.
- POSIX only: the tests run in a process group, so on Windows outside WSL the runner refuses with exit 2.
- **CAUGHT only means the tests failed.** A mutation that leaves the file unparsable or uncompilable fails every test, so
write mutations that keep the file valid, and run the first one by hand to see that the test failing is one that
covers the fix.
- Write about one mutation per branch and guard in the fix. Don't pad the list to a number: a mutation that no test
could distinguish from the original is `equivalent`, with the reason.
34 changes: 34 additions & 0 deletions .claude/skills/hardening-review/references/state.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# `state.md`

Rewritten, not appended, whenever a step changes it. After a compaction this file, with `decisions.md`, is the run.

```markdown
# Hardening review state: <branch>

- Step: <the step to do next, e.g. "5 Triage, batch 2 of 3", or `recorded` once step 8's record has been made>
- Pass: <N> of at most <5, or the raised limit>
- Next finding id: R<n> (from `scripts/review.sh next`, then counted on)
- Key: <the key prepare printed this pass>
- HEAD: <the HEAD prepare printed this pass>
- Worktree: <absolute path>
- Run folder: <absolute path>
- Private folder: <absolute path>
- Verify checkout: <absolute path of the throwaway checkout, or none>

## Reviewers (this pass)

| Lens | Agent id | Answered |
| --- | --- | --- |
| Bar | <id> | yes |
| Edge cases (Shell) | <id> | no |

## Batch in progress

<The finding ids, the commit, the mutation spec, and whether mutate and the gate have passed.>

## Open items

<Ids that wait for the person, each as id, lens, path. Their reasons are in the private folder's known-open.json. When a fix moves an
accepted item's lines, its entry is `changed`: a re-accepted finding is removed and added again (`known_open.py remove`, then `add`), so
the line numbers here and in known-open.json agree.>
```
28 changes: 28 additions & 0 deletions .claude/skills/hardening-review/references/triage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Triage format

One batch is at most 6 findings, grouped by file or cause. Each is a question to the person, who answers them all in one
reply. Recommend an outcome for each; the person's answer is the outcome.

```text
❓ **F1** - **R17 Critical · Shell · scripts/review.sh:88 - <short title>**
<What goes wrong and in which situation, in two or three lines.>
Verified: confirmed (ran `<entry>` with `<input>`: <what it printed>) | unverifiable (<why>)
Options: (a) fix as suggested: <fix> (b) fix another way: <how> (c) accept (d) defer (e) dismiss, because <why>

➡️ <your recommended option and why>

Edge cases:
| Case | Test, or n/a and why |
| --- | --- |
| empty | `Review_EmptyNote_ReturnsError` |
| whitespace-only | … |
| Unicode / malformed / oversized / boundary / missing / duplicated | … |
```

- Refuted findings are not asked: they are dismissed with their evidence and listed in one line after the batch.
- A finding suppressed by the known-open list is not asked as a question, but is listed in one line (`R3 again: unchanged,
settled`); a matching Warning is confirmed there by the person, not decided for them.
- A Warning (security) is never accepted for the person and is never quoted in a public place. Ask by id, path and a
neutral description, and say the details are in the private report. A `Verified:` line, its input and its output, is
never quoted outside the session.
- The standing rule: every fix's tests include edge cases. A row marked `n/a` needs a reason the person can check.
Loading
Loading