From e27e1ccb9b8e7274ef87357a20a4a068e1d8ebd3 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:02:20 -0600 Subject: [PATCH 01/10] Trial P-62: the contents API answers a file over 1 MiB whole with the raw media type, and the git blobs API beside it Co-Authored-By: Claude Opus 5.5 --- docs/architecture/software-factory-design.md | 1 + trials/P-62.json | 357 +++++++++++++++++++ 2 files changed, 358 insertions(+) create mode 100644 trials/P-62.json diff --git a/docs/architecture/software-factory-design.md b/docs/architecture/software-factory-design.md index f6d93573..1c9e5312 100644 --- a/docs/architecture/software-factory-design.md +++ b/docs/architecture/software-factory-design.md @@ -2256,6 +2256,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is | P-59 | On a GitHub-hosted Linux runner, with bubblewrap and the subprocess scrub as the run job has them, is the pinned model CLI, given the read tools alone, held to its working folder: is every read outside it refused, do the folder's own settings or CLAUDE.md take effect, and does any write land? | Yes, held, with the flags a review Run would be given, or the review Role is not given the head to read. **Passed** on 2026-11-04 (`trials/P-59.json`), at 2.1.306, in the canary, with a stand-in model on the runner and no secret: given `--tools Read,Grep,Glob --allowedTools Read,Grep,Glob --permission-mode dontAsk --restricted --setting-sources ""`, each of 18 calls outside the folder was refused, by an absolute path, a `..`, three links, `/proc/self/environ` and its parent's environ, `/proc/self/cwd` through a link, `/proc/self/root` and the home directory, by Read, Grep and Glob alike; no marker from outside reached the model by any route; Bash and Write were no tools; the folder's `.claude` settings, which allowed every path, bypassed every permission and held two hooks and a key helper, did nothing, and its CLAUDE.md and AGENTS.md were not sent. `--restricted` alone held the same. `--setting-sources ""` alone kept the settings and instructions out but held no read: with the scrub set the CLI forces the permission mode to its default, so the allowed tools, named without a path, read anywhere, and Grep read the CLI's own environment and its parent's, which in a Run hold its credential. With neither flag, the settings also ran both hooks and the key helper and the CLAUDE.md was sent. So `--restricted` is what confines and `--setting-sources ""` keeps the head's settings out beside it. Grep does not follow a link inside the folder. The design of the review Role reading the head rests on this; how the head is fetched is P-60 | | P-60 | Does GitHub serve a commit's tarball as one redirect from api.github.com to codeload.github.com, which answers it with no Authorization header, and does codeload redirect again? | One redirect, so the transport may follow it to codeload alone and refuse any second (the Maintainer's "Tarball, allow codeload", 2026-11-04). **Passed** on 2026-11-04 (`trials/P-60.json`), on the canary, asked with no credential: the API answered 302 to `https://codeload.github.com///legacy.tar.gz/`, with no token in it, and codeload answered that with no Authorization header 200, a gzip with no `content-length`, whose tar opens with a pax global header naming the whole commit and holds one folder of directories and regular files. No address on codeload over HTTPS redirected; over plain HTTP it answered 301 to the same address over HTTPS. For a commit the canary does not hold the API still answered 302, and codeload 404. What codeload's Location holds for a private repo, read with an App's token, is not asked: it is owed before a live Run reads a private head. The same commit's tree, `GET /repos///git/trees/?recursive=1` asked with no credential on 2026-11-05 (`trials/P-60.tree.json`), answered 200 and not truncated, listing every folder and file with its mode, and each file with its size and a blob hash that matched the tarball's bytes, all 22; a commit the canary does not hold answered 422. The review's head is held to that tree (decision 0142) | | P-61 | Does an issue's timeline give each change of its type, and each link of a pull request by hand in its Development box and its unlinking, as an event of its own with its time to the second and who made it; and does the merge of a pull request linked by hand close the issue, naming it the closer? | Each as its own timed event, so that row 1 can read an issue's type and a link by hand as each stood at the merge ("Trial P-61: an issue's type changes and a pull request linked by hand, each timed on the issue's timeline", issue 551; decision 0141). **Not run yet**: `tools/trial_p61_timeline.sh` reads the timeline through the Maintainer's own login at each of eight steps he makes by hand in the canary, and keeps each answer redacted outside the repo; it waits on the eval replay | +| P-62 | Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64? | Whole, in one answer, so that the eval job reads a case's head and tree from the held-out home with the home's token alone ("A case's head is read from the private home, pinned by its hash, and the run is sharded", issue 611; the Factory Builder's ruling of 2026-11-08, (d)); the git blobs API is the fallback. **Passed** on 2026-11-09 (`trials/P-62.json`), in the canary, on a file of 1,577,185 random bytes on a branch made for it and deleted after: asked with `application/vnd.github.raw+json` with no credential, or with `application/vnd.github.raw` with the Maintainer's login or none, the API answered 200 with a `content-length` of the file's and the file's bytes exactly, by their sha256, and no redirect. Asked as JSON it answered the file's size with `content` empty and `encoding` `none`. The git blobs API answered the blob as base64 that decoded to the same bytes, and with the raw media type as `application/octet-stream`. `gh api` writes no body for a type that names json and exits 1, so a reader that is not the transport asks for `application/vnd.github.raw`. What the API answers for a private repo's file is not asked; the home is one | ### 6.1 The order to run them in diff --git a/trials/P-62.json b/trials/P-62.json new file mode 100644 index 00000000..01ba4428 --- /dev/null +++ b/trials/P-62.json @@ -0,0 +1,357 @@ +{ + "schema": "trial/1", + "trial": "P-62", + "question": "Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect, and does the git blobs API answer the same blob as base64?", + "pass": "pass: a GET of /repos/{owner}/{repo}/contents/{path} for a file of more than 1 MiB, asked with Accept application/vnd.github.raw+json, is answered 200 with a body that is the file's bytes exactly, by its length and its sha256, and no redirect, whether asked with the Maintainer's login or with no credential; and the same GET with application/vnd.github+json is answered 200 with the file's size and no content inline. fail otherwise. Recorded, deciding nothing, as the fallback: what GET /repos/{owner}/{repo}/git/blobs/{sha} answers for the same blob, as JSON and with the raw media type.", + "repo": "combsmith/software-factory-canary", + "ran_at": "2026-11-09T00:59:21Z", + "exchanges": [ + { + "request": { + "as": { + "kind": "person", + "name": "don-petry" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file trial-p62/blob.bin, 1577185 random bytes made for the trial, committed as fd71fb0 on the canary's main, 7737f8f, on the branch trial-p62 alone, which was deleted after; its sha256 is 691bf10a, asked as JSON", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "private, max-age=60, s-maxage=60", + "content-length": "952", + "content-type": "application/json; charset=utf-8", + "date": "Mon, 09 Nov 2026 00:59:22 GMT", + "etag": "\"f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5\"", + "last-modified": "Mon, 09 Nov 2026 00:59:21 GMT", + "server": "github.com", + "vary": "Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding, Accept, X-Requested-With", + "x-accepted-oauth-scopes": "repo", + "x-github-media-type": "github.v3; format=json", + "x-github-request-id": "F94E:1F6B:1898F4E:51296A5:6AF11AEA", + "x-ratelimit-limit": "5000", + "x-ratelimit-remaining": "4620", + "x-ratelimit-reset": "1794187179", + "x-ratelimit-resource": "core", + "x-ratelimit-used": "380" + }, + "body": { + "name": "blob.bin", + "path": "trial-p62/blob.bin", + "sha": "f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5", + "size": 1577185, + "url": "https://api.github.com/repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62", + "html_url": "https://github.com/combsmith/software-factory-canary/blob/trial-p62/trial-p62/blob.bin", + "git_url": "https://api.github.com/repos/combsmith/software-factory-canary/git/blobs/f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5", + "download_url": "https://raw.githubusercontent.com/combsmith/software-factory-canary/trial-p62/trial-p62/blob.bin", + "type": "file", + "content": "", + "encoding": "none", + "_links": { + "self": "https://api.github.com/repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62", + "git": "https://api.github.com/repos/combsmith/software-factory-canary/git/blobs/f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5", + "html": "https://github.com/combsmith/software-factory-canary/blob/trial-p62/trial-p62/blob.bin" + } + } + } + }, + { + "request": { + "as": { + "kind": "person", + "name": "don-petry" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the same file asked with the raw media type. gh formats a body whose type names json, and wrote none; asked so again in exchange 7 it exited 1, saying only that a buffer was short. What GitHub sent is in exchanges 2 and 5", + "headers": { + "accept": "application/vnd.github.raw+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "private, max-age=60, s-maxage=60", + "content-length": "1577185", + "content-type": "application/vnd.github.raw+json; charset=utf-8", + "date": "Mon, 09 Nov 2026 00:59:22 GMT", + "etag": "\"f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5\"", + "last-modified": "Mon, 09 Nov 2026 00:59:21 GMT", + "server": "github.com", + "vary": "Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding, Accept, X-Requested-With", + "x-accepted-oauth-scopes": "repo", + "x-github-media-type": "github.v3; param=raw; format=json", + "x-github-request-id": "F951:1ED98F:24351B9:76DEE7B:6AF11AEA", + "x-ratelimit-limit": "5000", + "x-ratelimit-remaining": "4619", + "x-ratelimit-reset": "1794187179", + "x-ratelimit-resource": "core", + "x-ratelimit-used": "381" + }, + "body": { + "summarized": "no body was written" + } + } + }, + { + "request": { + "as": { + "kind": "anonymous", + "name": "no credential" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (curl --include --max-redirs 0, no credential): the same file asked with the raw media type, no redirect followed", + "headers": { + "accept": "application/vnd.github.raw+json", + "x-github-api-version": "2022-11-28" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "public, max-age=60, s-maxage=60", + "content-length": "1577185", + "content-type": "application/vnd.github.raw+json; charset=utf-8", + "date": "Mon, 09 Nov 2026 00:59:22 GMT", + "etag": "\"f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5\"", + "last-modified": "Mon, 09 Nov 2026 00:59:21 GMT", + "vary": "Accept, Accept-Encoding, Accept, X-Requested-With", + "x-github-media-type": "github.v3; param=raw; format=json", + "server": "github.com", + "x-ratelimit-limit": "60", + "x-ratelimit-remaining": "59", + "x-ratelimit-used": "1", + "x-ratelimit-resource": "core", + "x-ratelimit-reset": "1794189562", + "x-github-request-id": "F952:194583:213E54E:6D3D6E1:6AF11AEA" + }, + "body": { + "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "length": 1577185, + "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", + "the_files_bytes": true + } + } + }, + { + "request": { + "as": { + "kind": "person", + "name": "don-petry" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/git/blobs/f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5 (gh api --include): the file's blob, by the sha exchange 0 names, asked as JSON; its content, base64, is recorded by the length and sha256 of what it decodes to", + "headers": { + "accept": "application/vnd.github+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "private, max-age=86400, s-maxage=86400", + "content-type": "application/json; charset=utf-8", + "date": "Mon, 09 Nov 2026 00:59:23 GMT", + "etag": "W/\"3d1e04d647e13d45fd2c394df4d5bc0a29fb2a3278aad94b005d05c47aebd571\"", + "server": "github.com", + "vary": "Accept, Authorization, Cookie, X-GitHub-OTP,Accept-Encoding, Accept, X-Requested-With", + "x-accepted-oauth-scopes": "", + "x-github-media-type": "github.v3; format=json", + "x-github-request-id": "F954:1BFA81:2470517:77743DF:6AF11AEB", + "x-ratelimit-limit": "5000", + "x-ratelimit-remaining": "4618", + "x-ratelimit-reset": "1794187179", + "x-ratelimit-resource": "core", + "x-ratelimit-used": "382" + }, + "body": { + "sha": "f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5", + "node_id": "B_kwDOVFD679oAKGYxY2ZiNzViOTA2MjVkY2YzZjdiM2M4MTFjZDVmZWU5ZjE3NWY1YTU", + "size": 1577185, + "url": "https://api.github.com/repos/combsmith/software-factory-canary/git/blobs/f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5", + "content": { + "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "length": 1577185, + "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", + "the_files_bytes": true + }, + "encoding": "base64" + } + } + }, + { + "request": { + "as": { + "kind": "person", + "name": "don-petry" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/git/blobs/f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5 (gh api --include): the same blob asked with the raw media type", + "headers": { + "accept": "application/vnd.github.raw+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "private, max-age=86400, s-maxage=86400", + "content-length": "1577185", + "content-type": "application/octet-stream", + "date": "Mon, 09 Nov 2026 00:59:24 GMT", + "etag": "\"1ea8e32093a4644e2eeda239b4e6697ad81b2610b5cfe448f0ceddb474496ed7\"", + "server": "github.com", + "vary": "Accept, Authorization, Cookie, X-GitHub-OTP,Accept-Encoding, Accept, X-Requested-With", + "x-accepted-oauth-scopes": "", + "x-github-media-type": "github.v3; param=raw; format=json", + "x-github-request-id": "F955:2B31E6:2375BFF:74693F5:6AF11AEC", + "x-ratelimit-limit": "5000", + "x-ratelimit-remaining": "4617", + "x-ratelimit-reset": "1794187179", + "x-ratelimit-resource": "core", + "x-ratelimit-used": "383" + }, + "body": { + "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "length": 1577185, + "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", + "the_files_bytes": true + } + } + }, + { + "request": { + "as": { + "kind": "person", + "name": "don-petry" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file asked with application/vnd.github.raw, a type that does not name json, on the same commit, its branch made again for this second pass and deleted after", + "headers": { + "accept": "application/vnd.github.raw" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "private, max-age=60, s-maxage=60", + "content-length": "1577185", + "content-type": "application/vnd.github.raw; charset=utf-8", + "date": "Mon, 09 Nov 2026 01:01:00 GMT", + "etag": "\"f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5\"", + "last-modified": "Mon, 09 Nov 2026 00:59:21 GMT", + "server": "github.com", + "vary": "Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding, Accept, X-Requested-With", + "x-accepted-oauth-scopes": "repo", + "x-github-media-type": "github.v3; param=raw", + "x-github-request-id": "F9EE:231DA5:2392CE8:74CD050:6AF11B4C", + "x-ratelimit-limit": "5000", + "x-ratelimit-remaining": "4611", + "x-ratelimit-reset": "1794187179", + "x-ratelimit-resource": "core", + "x-ratelimit-used": "389" + }, + "body": { + "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "length": 1577185, + "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", + "the_files_bytes": true + } + } + }, + { + "request": { + "as": { + "kind": "anonymous", + "name": "no credential" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (curl --include --max-redirs 0, no credential): the same, with no credential and no redirect followed", + "headers": { + "accept": "application/vnd.github.raw", + "x-github-api-version": "2022-11-28" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "public, max-age=60, s-maxage=60", + "content-length": "1577185", + "content-type": "application/vnd.github.raw; charset=utf-8", + "date": "Mon, 09 Nov 2026 01:01:01 GMT", + "etag": "\"f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5\"", + "last-modified": "Mon, 09 Nov 2026 00:59:21 GMT", + "vary": "Accept, Accept-Encoding, Accept, X-Requested-With", + "x-github-media-type": "github.v3; param=raw", + "server": "github.com", + "x-ratelimit-limit": "60", + "x-ratelimit-remaining": "58", + "x-ratelimit-used": "2", + "x-ratelimit-resource": "core", + "x-ratelimit-reset": "1794189562", + "x-github-request-id": "F9EF:D0ED2:22B9C26:71FAE5C:6AF11B4D" + }, + "body": { + "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "length": 1577185, + "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", + "the_files_bytes": true + } + } + }, + { + "request": { + "as": { + "kind": "person", + "name": "don-petry" + }, + "via": "rest", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): exchange 1 asked again: gh wrote no body, and exited 1", + "headers": { + "accept": "application/vnd.github.raw+json" + }, + "body": null + }, + "response": { + "status": 200, + "headers": { + "cache-control": "private, max-age=60, s-maxage=60", + "content-length": "1577185", + "content-type": "application/vnd.github.raw+json; charset=utf-8", + "date": "Mon, 09 Nov 2026 01:01:01 GMT", + "etag": "\"f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5\"", + "last-modified": "Mon, 09 Nov 2026 00:59:21 GMT", + "server": "github.com", + "vary": "Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding, Accept, X-Requested-With", + "x-accepted-oauth-scopes": "repo", + "x-github-media-type": "github.v3; param=raw; format=json", + "x-github-request-id": "F9F0:10630A:EEFC8E:30FC488:6AF11B4D", + "x-ratelimit-limit": "5000", + "x-ratelimit-remaining": "4610", + "x-ratelimit-reset": "1794187179", + "x-ratelimit-resource": "core", + "x-ratelimit-used": "390" + }, + "body": { + "summarized": "no body was written" + } + } + } + ], + "removed": [ + "exchanges[0].request.headers.authorization", + "exchanges[1].request.headers.authorization", + "exchanges[3].request.headers.authorization", + "exchanges[4].request.headers.authorization", + "exchanges[5].request.headers.authorization", + "exchanges[7].request.headers.authorization" + ], + "verdict": "pass" +} From c9d7ab0e79337fa3ac35c0e70bbe6636f8f1db8c Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:18:47 -0600 Subject: [PATCH 02/10] P-62: what the reads found: the summaries say how they were made, the gh calls what they show, and every reader asks for vnd.github.raw Co-Authored-By: Claude Opus 5.5 --- docs/architecture/software-factory-design.md | 2 +- trials/P-62.json | 34 ++++++++++---------- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/architecture/software-factory-design.md b/docs/architecture/software-factory-design.md index 1c9e5312..032231f1 100644 --- a/docs/architecture/software-factory-design.md +++ b/docs/architecture/software-factory-design.md @@ -2256,7 +2256,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is | P-59 | On a GitHub-hosted Linux runner, with bubblewrap and the subprocess scrub as the run job has them, is the pinned model CLI, given the read tools alone, held to its working folder: is every read outside it refused, do the folder's own settings or CLAUDE.md take effect, and does any write land? | Yes, held, with the flags a review Run would be given, or the review Role is not given the head to read. **Passed** on 2026-11-04 (`trials/P-59.json`), at 2.1.306, in the canary, with a stand-in model on the runner and no secret: given `--tools Read,Grep,Glob --allowedTools Read,Grep,Glob --permission-mode dontAsk --restricted --setting-sources ""`, each of 18 calls outside the folder was refused, by an absolute path, a `..`, three links, `/proc/self/environ` and its parent's environ, `/proc/self/cwd` through a link, `/proc/self/root` and the home directory, by Read, Grep and Glob alike; no marker from outside reached the model by any route; Bash and Write were no tools; the folder's `.claude` settings, which allowed every path, bypassed every permission and held two hooks and a key helper, did nothing, and its CLAUDE.md and AGENTS.md were not sent. `--restricted` alone held the same. `--setting-sources ""` alone kept the settings and instructions out but held no read: with the scrub set the CLI forces the permission mode to its default, so the allowed tools, named without a path, read anywhere, and Grep read the CLI's own environment and its parent's, which in a Run hold its credential. With neither flag, the settings also ran both hooks and the key helper and the CLAUDE.md was sent. So `--restricted` is what confines and `--setting-sources ""` keeps the head's settings out beside it. Grep does not follow a link inside the folder. The design of the review Role reading the head rests on this; how the head is fetched is P-60 | | P-60 | Does GitHub serve a commit's tarball as one redirect from api.github.com to codeload.github.com, which answers it with no Authorization header, and does codeload redirect again? | One redirect, so the transport may follow it to codeload alone and refuse any second (the Maintainer's "Tarball, allow codeload", 2026-11-04). **Passed** on 2026-11-04 (`trials/P-60.json`), on the canary, asked with no credential: the API answered 302 to `https://codeload.github.com///legacy.tar.gz/`, with no token in it, and codeload answered that with no Authorization header 200, a gzip with no `content-length`, whose tar opens with a pax global header naming the whole commit and holds one folder of directories and regular files. No address on codeload over HTTPS redirected; over plain HTTP it answered 301 to the same address over HTTPS. For a commit the canary does not hold the API still answered 302, and codeload 404. What codeload's Location holds for a private repo, read with an App's token, is not asked: it is owed before a live Run reads a private head. The same commit's tree, `GET /repos///git/trees/?recursive=1` asked with no credential on 2026-11-05 (`trials/P-60.tree.json`), answered 200 and not truncated, listing every folder and file with its mode, and each file with its size and a blob hash that matched the tarball's bytes, all 22; a commit the canary does not hold answered 422. The review's head is held to that tree (decision 0142) | | P-61 | Does an issue's timeline give each change of its type, and each link of a pull request by hand in its Development box and its unlinking, as an event of its own with its time to the second and who made it; and does the merge of a pull request linked by hand close the issue, naming it the closer? | Each as its own timed event, so that row 1 can read an issue's type and a link by hand as each stood at the merge ("Trial P-61: an issue's type changes and a pull request linked by hand, each timed on the issue's timeline", issue 551; decision 0141). **Not run yet**: `tools/trial_p61_timeline.sh` reads the timeline through the Maintainer's own login at each of eight steps he makes by hand in the canary, and keeps each answer redacted outside the repo; it waits on the eval replay | -| P-62 | Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64? | Whole, in one answer, so that the eval job reads a case's head and tree from the held-out home with the home's token alone ("A case's head is read from the private home, pinned by its hash, and the run is sharded", issue 611; the Factory Builder's ruling of 2026-11-08, (d)); the git blobs API is the fallback. **Passed** on 2026-11-09 (`trials/P-62.json`), in the canary, on a file of 1,577,185 random bytes on a branch made for it and deleted after: asked with `application/vnd.github.raw+json` with no credential, or with `application/vnd.github.raw` with the Maintainer's login or none, the API answered 200 with a `content-length` of the file's and the file's bytes exactly, by their sha256, and no redirect. Asked as JSON it answered the file's size with `content` empty and `encoding` `none`. The git blobs API answered the blob as base64 that decoded to the same bytes, and with the raw media type as `application/octet-stream`. `gh api` writes no body for a type that names json and exits 1, so a reader that is not the transport asks for `application/vnd.github.raw`. What the API answers for a private repo's file is not asked; the home is one | +| P-62 | Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64? | Whole, in one answer, so that the eval job reads a case's head and tree from the held-out home with the home's token alone ("A case's head is read from the private home, pinned by its hash, and the run is sharded", issue 611; the Factory Builder's ruling of 2026-11-08, (d)); the git blobs API is the fallback. **Passed** on 2026-11-09 (`trials/P-62.json`), in the canary, on a file of 1,577,185 random bytes on a branch made for it and deleted after: asked with `application/vnd.github.raw+json` with no credential, or with `application/vnd.github.raw` with the Maintainer's login or none, the API answered 200 with a `content-length` of the file's and the file's bytes exactly, by their sha256, and no redirect. Asked as JSON it answered the file's size with `content` empty and `encoding` `none`. The git blobs API answered the blob as base64 that decoded to the same bytes, and with the raw media type as `application/octet-stream`. `gh api` wrote no body for `application/vnd.github.raw+json` and, asked again, exited 1; and the transport reads any answer whose type ends in json as JSON, so every reader, the transport among them, asks for `application/vnd.github.raw`, whose answer's type does not. What the API answers for a private repo's file is not asked here. Should it answer the held-out home's otherwise, a redirect among them, the read fails closed: the transport follows no redirect for a file, and a head not read whole is `head_unreadable` | ### 6.1 The order to run them in diff --git a/trials/P-62.json b/trials/P-62.json index 01ba4428..8deaf985 100644 --- a/trials/P-62.json +++ b/trials/P-62.json @@ -1,7 +1,7 @@ { "schema": "trial/1", "trial": "P-62", - "question": "Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect, and does the git blobs API answer the same blob as base64?", + "question": "Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64?", "pass": "pass: a GET of /repos/{owner}/{repo}/contents/{path} for a file of more than 1 MiB, asked with Accept application/vnd.github.raw+json, is answered 200 with a body that is the file's bytes exactly, by its length and its sha256, and no redirect, whether asked with the Maintainer's login or with no credential; and the same GET with application/vnd.github+json is answered 200 with the file's size and no content inline. fail otherwise. Recorded, deciding nothing, as the fallback: what GET /repos/{owner}/{repo}/git/blobs/{sha} answers for the same blob, as JSON and with the raw media type.", "repo": "combsmith/software-factory-canary", "ran_at": "2026-11-09T00:59:21Z", @@ -13,7 +13,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file trial-p62/blob.bin, 1577185 random bytes made for the trial, committed as fd71fb0 on the canary's main, 7737f8f, on the branch trial-p62 alone, which was deleted after; its sha256 is 691bf10a, asked as JSON", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file trial-p62/blob.bin, 1577185 random bytes made for the trial, committed as fd71fb0, a child of the canary's main 7737f8f, on the branch trial-p62 alone, which was deleted after; its sha256 is 691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f, asked as JSON", "headers": { "accept": "application/vnd.github+json" }, @@ -66,7 +66,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the same file asked with the raw media type. gh formats a body whose type names json, and wrote none; asked so again in exchange 7 it exited 1, saying only that a buffer was short. What GitHub sent is in exchanges 2 and 5", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the same file asked with the raw media type. gh formats a body whose type names json, and wrote none; asked so again in exchange 7 it exited 1, saying only that a buffer was short. The body GitHub sends for this type is seen with no credential in exchange 2, and for this login, under application/vnd.github.raw, in exchange 5", "headers": { "accept": "application/vnd.github.raw+json" }, @@ -93,7 +93,7 @@ "x-ratelimit-used": "381" }, "body": { - "summarized": "no body was written" + "not_recorded": "gh wrote no body; GitHub's content-length was the file's" } } }, @@ -131,10 +131,10 @@ "x-github-request-id": "F952:194583:213E54E:6D3D6E1:6AF11AEA" }, "body": { - "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "summarized": "the body, split off the capture unread and hashed by a program, its headers redacted by tools.redact_trial: the trial's random bytes, recorded by their length and sha256 alone", "length": 1577185, "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", - "the_files_bytes": true + "matches_the_file": true } } }, @@ -175,10 +175,10 @@ "size": 1577185, "url": "https://api.github.com/repos/combsmith/software-factory-canary/git/blobs/f1cfb75b90625dcf3f7b3c811cd5fee9f175f5a5", "content": { - "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "summarized": "the content field, 2137965 characters of base64 wrapped at 60 in 35049 lines, decoded by a program and recorded by the length and sha256 of what it decodes to", "length": 1577185, "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", - "the_files_bytes": true + "matches_the_file": true }, "encoding": "base64" } @@ -217,10 +217,10 @@ "x-ratelimit-used": "383" }, "body": { - "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "summarized": "the body, split off the capture unread and hashed by a program, its headers redacted by tools.redact_trial: the trial's random bytes, recorded by their length and sha256 alone", "length": 1577185, "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", - "the_files_bytes": true + "matches_the_file": true } } }, @@ -231,7 +231,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file asked with application/vnd.github.raw, a type that does not name json, on the same commit, its branch made again for this second pass and deleted after", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file asked with application/vnd.github.raw, a type that does not name json, on the same commit, its branch made again for this second pass and deleted after. gh follows a redirect without saying so; the answer's headers are the API's own, and exchanges 2 and 6, which follow none, show no redirect", "headers": { "accept": "application/vnd.github.raw" }, @@ -258,10 +258,10 @@ "x-ratelimit-used": "389" }, "body": { - "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "summarized": "the body, split off the capture unread and hashed by a program, its headers redacted by tools.redact_trial: the trial's random bytes, recorded by their length and sha256 alone", "length": 1577185, "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", - "the_files_bytes": true + "matches_the_file": true } } }, @@ -299,10 +299,10 @@ "x-github-request-id": "F9EF:D0ED2:22B9C26:71FAE5C:6AF11B4D" }, "body": { - "summarized": "the body, the trial's random bytes, recorded by its length and sha256 alone", + "summarized": "the body, split off the capture unread and hashed by a program, its headers redacted by tools.redact_trial: the trial's random bytes, recorded by their length and sha256 alone", "length": 1577185, "sha256": "691bf10a3701a407ad9e231fbfbb0ef84161590d4a142673e1ec695552b11f6f", - "the_files_bytes": true + "matches_the_file": true } } }, @@ -313,7 +313,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): exchange 1 asked again: gh wrote no body, and exited 1", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): exchange 1 asked again: gh wrote no body, and exited 1 saying \"transform: short source buffer\"; asked a third time without --include, the body alone to a file, it did the same", "headers": { "accept": "application/vnd.github.raw+json" }, @@ -340,7 +340,7 @@ "x-ratelimit-used": "390" }, "body": { - "summarized": "no body was written" + "not_recorded": "gh wrote no body; GitHub's content-length was the file's" } } } From 99b4a3e009945715c7e2fcf9a7160b619f5872d9 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:19:57 -0600 Subject: [PATCH 03/10] P-62: the Factory Builder's ruling on its verdict, quoted beside the pass condition Co-Authored-By: Claude Opus 5.5 --- docs/architecture/software-factory-design.md | 2 +- trials/P-62.json | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture/software-factory-design.md b/docs/architecture/software-factory-design.md index 032231f1..af68f374 100644 --- a/docs/architecture/software-factory-design.md +++ b/docs/architecture/software-factory-design.md @@ -2256,7 +2256,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is | P-59 | On a GitHub-hosted Linux runner, with bubblewrap and the subprocess scrub as the run job has them, is the pinned model CLI, given the read tools alone, held to its working folder: is every read outside it refused, do the folder's own settings or CLAUDE.md take effect, and does any write land? | Yes, held, with the flags a review Run would be given, or the review Role is not given the head to read. **Passed** on 2026-11-04 (`trials/P-59.json`), at 2.1.306, in the canary, with a stand-in model on the runner and no secret: given `--tools Read,Grep,Glob --allowedTools Read,Grep,Glob --permission-mode dontAsk --restricted --setting-sources ""`, each of 18 calls outside the folder was refused, by an absolute path, a `..`, three links, `/proc/self/environ` and its parent's environ, `/proc/self/cwd` through a link, `/proc/self/root` and the home directory, by Read, Grep and Glob alike; no marker from outside reached the model by any route; Bash and Write were no tools; the folder's `.claude` settings, which allowed every path, bypassed every permission and held two hooks and a key helper, did nothing, and its CLAUDE.md and AGENTS.md were not sent. `--restricted` alone held the same. `--setting-sources ""` alone kept the settings and instructions out but held no read: with the scrub set the CLI forces the permission mode to its default, so the allowed tools, named without a path, read anywhere, and Grep read the CLI's own environment and its parent's, which in a Run hold its credential. With neither flag, the settings also ran both hooks and the key helper and the CLAUDE.md was sent. So `--restricted` is what confines and `--setting-sources ""` keeps the head's settings out beside it. Grep does not follow a link inside the folder. The design of the review Role reading the head rests on this; how the head is fetched is P-60 | | P-60 | Does GitHub serve a commit's tarball as one redirect from api.github.com to codeload.github.com, which answers it with no Authorization header, and does codeload redirect again? | One redirect, so the transport may follow it to codeload alone and refuse any second (the Maintainer's "Tarball, allow codeload", 2026-11-04). **Passed** on 2026-11-04 (`trials/P-60.json`), on the canary, asked with no credential: the API answered 302 to `https://codeload.github.com///legacy.tar.gz/`, with no token in it, and codeload answered that with no Authorization header 200, a gzip with no `content-length`, whose tar opens with a pax global header naming the whole commit and holds one folder of directories and regular files. No address on codeload over HTTPS redirected; over plain HTTP it answered 301 to the same address over HTTPS. For a commit the canary does not hold the API still answered 302, and codeload 404. What codeload's Location holds for a private repo, read with an App's token, is not asked: it is owed before a live Run reads a private head. The same commit's tree, `GET /repos///git/trees/?recursive=1` asked with no credential on 2026-11-05 (`trials/P-60.tree.json`), answered 200 and not truncated, listing every folder and file with its mode, and each file with its size and a blob hash that matched the tarball's bytes, all 22; a commit the canary does not hold answered 422. The review's head is held to that tree (decision 0142) | | P-61 | Does an issue's timeline give each change of its type, and each link of a pull request by hand in its Development box and its unlinking, as an event of its own with its time to the second and who made it; and does the merge of a pull request linked by hand close the issue, naming it the closer? | Each as its own timed event, so that row 1 can read an issue's type and a link by hand as each stood at the merge ("Trial P-61: an issue's type changes and a pull request linked by hand, each timed on the issue's timeline", issue 551; decision 0141). **Not run yet**: `tools/trial_p61_timeline.sh` reads the timeline through the Maintainer's own login at each of eight steps he makes by hand in the canary, and keeps each answer redacted outside the repo; it waits on the eval replay | -| P-62 | Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64? | Whole, in one answer, so that the eval job reads a case's head and tree from the held-out home with the home's token alone ("A case's head is read from the private home, pinned by its hash, and the run is sharded", issue 611; the Factory Builder's ruling of 2026-11-08, (d)); the git blobs API is the fallback. **Passed** on 2026-11-09 (`trials/P-62.json`), in the canary, on a file of 1,577,185 random bytes on a branch made for it and deleted after: asked with `application/vnd.github.raw+json` with no credential, or with `application/vnd.github.raw` with the Maintainer's login or none, the API answered 200 with a `content-length` of the file's and the file's bytes exactly, by their sha256, and no redirect. Asked as JSON it answered the file's size with `content` empty and `encoding` `none`. The git blobs API answered the blob as base64 that decoded to the same bytes, and with the raw media type as `application/octet-stream`. `gh api` wrote no body for `application/vnd.github.raw+json` and, asked again, exited 1; and the transport reads any answer whose type ends in json as JSON, so every reader, the transport among them, asks for `application/vnd.github.raw`, whose answer's type does not. What the API answers for a private repo's file is not asked here. Should it answer the held-out home's otherwise, a redirect among them, the read fails closed: the transport follows no redirect for a file, and a head not read whole is `head_unreadable` | +| P-62 | Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64? | Whole, in one answer, so that the eval job reads a case's head and tree from the held-out home with the home's token alone ("A case's head is read from the private home, pinned by its hash, and the run is sharded", issue 611; the Factory Builder's ruling of 2026-11-08, (d)); the git blobs API is the fallback. **Passed** on 2026-11-09 (`trials/P-62.json`), by the Factory Builder's ruling of 2026-11-08 that "the raw media type" in its pass condition is answered by `application/vnd.github.raw`, the type the transport must ask for, and that `gh`'s failure on `application/vnd.github.raw+json` with the Maintainer's login is a fault of the client on a type the Factory will not use, not GitHub's answer; in the canary, on a file of 1,577,185 random bytes on a branch made for it and deleted after: asked with `application/vnd.github.raw+json` with no credential, or with `application/vnd.github.raw` with the Maintainer's login or none, the API answered 200 with a `content-length` of the file's and the file's bytes exactly, by their sha256, and no redirect. Asked as JSON it answered the file's size with `content` empty and `encoding` `none`. The git blobs API answered the blob as base64 that decoded to the same bytes, and with the raw media type as `application/octet-stream`. With the login, `application/vnd.github.raw+json` was answered 200 with a `content-length` of the file's, but `gh api` wrote no body, saying "transform: short source buffer", three times; and the transport reads any answer whose type ends in json as JSON, so every reader, the transport among them, asks for `application/vnd.github.raw`, whose answer's type does not. What the API answers for a private repo's file is not asked here. Should it answer the held-out home's otherwise, a redirect among them, the read fails closed: the transport follows no redirect for a file, and a head not read whole is `head_unreadable` | ### 6.1 The order to run them in diff --git a/trials/P-62.json b/trials/P-62.json index 8deaf985..0c30df2f 100644 --- a/trials/P-62.json +++ b/trials/P-62.json @@ -66,7 +66,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the same file asked with the raw media type. gh formats a body whose type names json, and wrote none; asked so again in exchange 7 it exited 1, saying only that a buffer was short. The body GitHub sends for this type is seen with no credential in exchange 2, and for this login, under application/vnd.github.raw, in exchange 5", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the same file asked with the raw media type. gh formats a body whose type names json, and wrote none; asked so again in exchange 7 it exited 1, saying only that a buffer was short. The body GitHub sends for this type is seen with no credential in exchange 2, and for this login, under application/vnd.github.raw, in exchange 5. By the Factory Builder's ruling of 2026-11-08, quoted in exchange 5, a fault of the client, not GitHub's answer", "headers": { "accept": "application/vnd.github.raw+json" }, @@ -231,7 +231,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file asked with application/vnd.github.raw, a type that does not name json, on the same commit, its branch made again for this second pass and deleted after. gh follows a redirect without saying so; the answer's headers are the API's own, and exchanges 2 and 6, which follow none, show no redirect", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): the file asked with application/vnd.github.raw, a type that does not name json, on the same commit, its branch made again for this second pass and deleted after. gh follows a redirect without saying so; the answer's headers are the API's own, and exchanges 2 and 6, which follow none, show no redirect. The Factory Builder's ruling of 2026-11-08 on this trial's verdict, pass: \"the raw media type\" in the pass condition is answered by application/vnd.github.raw, the type the transport must ask for, since adapters/github.py reads any type ending in json as JSON; it returned the file's bytes exactly, by their sha256, with no redirect, with the Maintainer's login (this exchange) and with no credential (exchange 6). Asked as application/vnd.github.raw+json, it returned the bytes exactly with no credential (exchange 2); with the login its content-length was 1,577,185, but gh wrote no body, saying \"transform: short source buffer\" three times (exchanges 1 and 7): a fault of the client, on a type the Factory will not use, not GitHub's answer", "headers": { "accept": "application/vnd.github.raw" }, @@ -313,7 +313,7 @@ "name": "don-petry" }, "via": "rest", - "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): exchange 1 asked again: gh wrote no body, and exited 1 saying \"transform: short source buffer\"; asked a third time without --include, the body alone to a file, it did the same", + "call": "GET /repos/combsmith/software-factory-canary/contents/trial-p62/blob.bin?ref=trial-p62 (gh api --include): exchange 1 asked again: gh wrote no body, and exited 1 saying \"transform: short source buffer\"; asked a third time without --include, the body alone to a file, it did the same. By the Factory Builder's ruling of 2026-11-08, quoted in exchange 5, a fault of the client, not GitHub's answer", "headers": { "accept": "application/vnd.github.raw+json" }, From d8ea13145de9858828f4346eacf976415f223154 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:18:51 -0600 Subject: [PATCH 04/10] A case's head is read from the held-out home, pinned by the case, and held to its tree Co-Authored-By: Claude Opus 5.5 --- adapters/AGENTS.md | 1 + adapters/heldout.py | 37 ++- ...ad-tools-alone-held-to-it-by-restricted.md | 2 +- ...pinned-by-the-case-and-held-to-its-tree.md | 29 ++ examples/eval-case.also-at.json | 2 + examples/eval-case.json | 2 + examples/eval-case.met.json | 2 + examples/eval-case.with-issue.json | 2 + .../eval-case.a-category-nobody-owns.json | 2 + .../eval-case.a-head-as-an-object.json | 2 + ...case.a-place-in-also-at-with-no-lines.json | 2 + ...-case.a-place-in-also-at-with-no-path.json | 2 + examples/invalid/eval-case.a-short-head.json | 2 + .../eval-case.a-tree-pin-cut-short.json | 256 ++++++++++++++++++ .../invalid/eval-case.a-verdict-in-words.json | 2 + .../eval-case.an-issue-with-no-body.json | 2 + .../eval-case.four-places-in-also-at.json | 2 + examples/invalid/eval-case.no-head-pin.json | 255 +++++++++++++++++ examples/invalid/eval-case.no-head.json | 4 +- .../eval-case.no-place-in-also-at.json | 2 + harness/eval/AGENTS.md | 6 +- harness/eval/__init__.py | 56 ++-- harness/eval/canary.py | 19 +- harness/eval/fetch.py | 8 +- harness/head.py | 12 +- schemas/eval-case.schema.json | 12 + tests/test_adapters.py | 64 +++++ tests/test_evals.py | 229 +++++++++++++--- tools/AGENTS.md | 3 +- tools/heads_into_home.py | 98 +++++++ tools/mutants.py | 97 +++++-- 31 files changed, 1113 insertions(+), 101 deletions(-) create mode 100644 docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md create mode 100644 examples/invalid/eval-case.a-tree-pin-cut-short.json create mode 100644 examples/invalid/eval-case.no-head-pin.json create mode 100644 tools/heads_into_home.py diff --git a/adapters/AGENTS.md b/adapters/AGENTS.md index 92699cea..1a0b2358 100644 --- a/adapters/AGENTS.md +++ b/adapters/AGENTS.md @@ -34,6 +34,7 @@ Not built yet, by the stories that hold their rules: contract.py, binding.py, is | Closing references read short, of another repo or named twice, are not whole | `tests/test_adapters.py::test_closing_references_are_whole_only_of_this_repo_and_every_one_counted` | | A pull request to another base than the default is not whole on its contract | `tests/test_adapters.py::test_a_pull_request_to_another_base_than_the_default_is_not_read_whole_on_its_contract` | | A held-out case is the bytes its manifest hashed, or none and a code; one that could name another file is not asked | `tests/test_adapters.py::test_a_held_out_case_is_its_hashed_bytes`, `tests/test_adapters.py::test_a_held_out_case_not_read_whole_says_why` | +| What the home keeps beside a case, its head's tarball and tree, is read raw and handed back only as the bytes the case pins | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins` | | The title, and the head the snapshot was held to, are handed beside the facts, never in them | `tests/test_adapters.py::test_the_pull_requests_own_facts_and_the_title_beside_them`, `tests/test_adapters.py::test_a_snapshot_hands_back_its_head` | | A plan of writes not its pass record's sends nothing; a dispatch is sent once, never retried | `tests/test_effects.py::test_a_write_list_that_is_not_the_records_is_refused_whole`, `tests/test_effects.py::test_a_dispatch_is_sent_once_and_never_retried` | | Each write goes through its one credential, which sends only its own requests by path and body | `tests/test_effects.py::test_each_write_goes_through_its_one_credential_and_its_allowlist` | diff --git a/adapters/heldout.py b/adapters/heldout.py index d11d6f46..3e817e62 100644 --- a/adapters/heldout.py +++ b/adapters/heldout.py @@ -1,6 +1,7 @@ """The held-out home's cases, read for the eval job alone (design §3.13 and §4.9, issue 309): each case a manifest names, `GET /repos/{owner}/{repo}/contents/{role}/{id}.json`, through the one transport, with the Maintainer's read-only token for the home and nothing else (decision 0053). A case is handed back as the bytes its manifest hashed, or not at all, with a code that says why. +So is what the home keeps beside it, its head's tarball and its commit's tree, each by the sha256 the case pins (issue 611). The home's owner and name are the job's environment's, never this repo's, and no refusal, code or message here names them, a case's path or a byte of a case: what the job prints is counts and codes. A case is read from GitHub's JSON @@ -11,9 +12,13 @@ import hashlib import re -from adapters.github import Unanswered, request +from adapters.github import TARRED, Unanswered, raw, request LARGEST = 1024 * 1024 # bytes a case may be: a facts snapshot and a diff; GitHub inlines a file's content to 1 MiB +RAW = "application/vnd.github.raw" # a file's bytes as they are, whatever its size (P-62) +# What the home keeps beside a case, by its extension, and the bytes each may be: the head's tarball as the transport +# bounds one, and its tree, which GitHub truncates past 7 MB, as `harness/head.tree` refuses one truncated. +BESIDE = {"tar.gz": TARRED, "tree.json": 8 * 1024 * 1024} OWNER = re.compile(r"[A-Za-z0-9](?:[A-Za-z0-9-]{0,37}[A-Za-z0-9])?") NAME = re.compile(r"[A-Za-z0-9_.-]{1,100}") PART = re.compile(r"[a-z0-9][a-z0-9-]*") # a Role or a case id, as eval-set/1 holds them: no dot, no slash @@ -64,3 +69,33 @@ def case(transport, owner: str, name: str, role: str, entry) -> tuple: return None, "hash_differs" return data, None + +def beside(transport, owner: str, name: str, role: str, ident: str, kind: str, sha256) -> tuple: + """`(bytes, None)` for the file the home keeps beside a case, `{role}/{ident}.{kind}`, its head's tarball (`tar.gz`) + or its commit's tree (`tree.json`), read whole with the raw media type and hashing to the sha256 the case pins; + `(None, code)` otherwise, as `case` says them, and `too_large` over the kind's bound (issue 611). The raw media type + answers a file of more than 1 MiB whole, in one answer and no redirect (trials/P-62.json), which the JSON answer + `case` reads does not: it holds a file's content to 1 MiB alone.""" + if not (isinstance(role, str) and PART.fullmatch(role) and isinstance(ident, str) and PART.fullmatch(ident) + and kind in BESIDE and isinstance(sha256, str) and SHA256.fullmatch(sha256)): + return None, "not_this_set" + if not home(owner, name): + return None, "unanswered" + try: + answer = transport.send(request("GET", "/repos/%s/%s/contents/%s/%s.%s" % (owner, name, role, ident, kind), + accept=RAW)) + except Unanswered: + return None, "unanswered" + if answer["status"] == 404: + return None, "not_found" + body = answer["body"] + if answer["status"] != 200 or not isinstance(body, str) \ + or not str(answer["headers"].get("content-type", "")).startswith(RAW + ";"): + return None, "unanswered" + data = raw(body) + if len(data) > BESIDE[kind]: + return None, "too_large" + if hashlib.sha256(data).hexdigest() != sha256: + return None, "hash_differs" + return data, None + diff --git a/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md b/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md index 5888da93..32511943 100644 --- a/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md +++ b/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md @@ -41,7 +41,7 @@ What is owed: what codeload's address holds for a private repo read with an App' | Each archive it refuses, it refuses whole | `tests/test_harness.py::test_each_archive_the_unpacker_refuses_it_refuses_whole` | | It holds its caps, writes into a new folder, and nothing outside it | `tests/test_harness.py::test_the_unpacker_holds_its_caps_and_writes_nothing_outside_its_folder` | | An eval case requires its head, as 40 lowercase hex characters | `tools/check_examples.py`, `examples/invalid/eval-case.no-head.json`, `examples/invalid/eval-case.a-head-as-an-object.json`, `examples/invalid/eval-case.a-short-head.json` | -| Each case's head is read with no credential and removed once asked; one not read closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate` | +| Each case's head is read with no credential and removed once asked; one not read closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate` | | A case is reviewed with the read tools in its head, as a live Run is | `tests/test_evals.py::test_a_case_is_reviewed_by_the_task_and_the_engine_a_live_run_is` | | The read tools are held to the folder at the pin, and by which flag | `trials/P-59.json`, `tests/test_tools.py::test_the_read_tools_on_a_linux_runner_are_held_to_their_folder_only_with_both_flags` | | The tarball is one redirect to codeload, which redirects no further over HTTPS | `trials/P-60.json`, `tests/test_tools.py::test_a_commits_tarball_is_one_redirect_to_codeload_which_never_redirects_over_https` | diff --git a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md new file mode 100644 index 00000000..7539cc5c --- /dev/null +++ b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md @@ -0,0 +1,29 @@ +# NNNN. A case's head is read from the held-out home, pinned by the case, and held to its tree + +Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded", points (a) to (d): this is the first of its two pull requests, the heads; the second shards the run. It amends decision 0138's rows "A transport with no token" and "The eval case" where they say the eval job reads a case's head with no credential, and decision 0142's row "An eval case's head", which this closes. Nothing else in either changes. + +## Decision + +| What | Decided | Because | +|---|---|---| +| Where a head is kept | Beside each case in the held-out home, `/.json`, the home keeps the case's head as two files: `.tar.gz`, the commit's tarball as GitHub archived it, and `.tree.json`, the commit's tree, every level of it, as the API answered it. `eval-case/1` requires `head_sha256` and `tree_sha256`, the sha256 of each, beside `head` | The issue's: a head read anonymously shares the anonymous cap, 60 an hour, so a pool of three runs, or a run again within the hour, can fault on `head_unreadable`; and a head that is not on GitHub cannot be read at all. Pinned by the case, which the manifest pins, the head is the bytes the Maintainer chose | +| How it is read | `fetch` reads both files with the home's token, through the transport the case is read with, by `adapters/heldout.beside`: each with the raw media type, `application/vnd.github.raw`, whose answer is a file's bytes whole however large, in one answer and no redirect (`trials/P-62.json`). An answer of another type, `+json` among them, which the transport would read as JSON, is no file. A file over its bound (the transport's tarball bound, `TARRED`, or 8 MiB for a tree) is `too_large`, and one that does not hash to its pin `hash_differs`. Nothing in the eval job reads with no credential, and the home's token is still sent nowhere else | The Factory Builder's ruling (c): no transition, and no case without its pins, since an anonymous read kept for one would keep the read the issue removes. Ruling (d): the raw media type, grounded by its trial, the git blobs API its fallback | +| How it is held | The tarball is unpacked as before (decision 0138), and then held to the tree file by file, by git's blob hash, by the check a live head is held by (decision 0142): `harness/head.tree_of` reads the tree file with the checks `tree` reads GitHub's answer with, and `held` holds the head to it. A head not the one pinned is `hash_differs`, a fault of the run; one not read, unpacked or held whole is `head_unreadable`, the Engine's or the Factory's. Either closes the gate | The Factory Builder's ruling (b): pin the tree as well as the tarball, and hold every eval head to it. 0142 left an eval head unheld for want of room under the anonymous cap; read with the home's token, there is room. An eval case is now graded on the files a live review would be given | +| Who puts them there | The Maintainer, at his terminal, in his checkout of the home: `GH_TOKEN="$(gh auth token)" python3 -m tools.heads_into_home `. For each case of the folder, it reads the head's tarball and tree from GitHub through the one transport, holds the one to the other, writes both beside the case and adds their sha256 to it; a case it cannot pin it leaves as it was. It prints counts, then each pinned case's id and the sha256 of its new bytes, never a byte of a case. It takes a role folder and touches nothing else, so a gate set made later is pinned as the review Intent's set is | The Factory Builder's ruling (c): the tool serves the coming gate set too, and prints counts and hashes alone. A session never sees a case | +| The set changes with it | Pinning changes each case's bytes, so the set's manifest under `evals/` takes the hashes the tool prints, in the pull request that brings this change. It merges only once the home holds the files, and once `trials/P-62.json` is on main | The Factory Builder's ruling (c) and (d) | +| The canary | `harness/eval/canary.py` still reads its case's head with no credential: its case is this repo's own example, of this public repo, never one of the home's, and pins no file the home keeps | The canary is no step of the eval job (decision 0106) | +| Residual: a head only in the home | The tool reads a head from GitHub. A head that is not there, one built for a case, is written beside its case by hand, its tarball as `git archive` writes one with the commit in its pax header and one folder, and its tree in the API's shape; nothing builds that yet | No case of the set needs one yet. The eval reads it as it reads any other | +| Residual: the home's answer | P-62 asked a public repo. Should the API answer a private repo's file otherwise, a redirect among them, the read fails closed: the transport follows no redirect for a file, and a head not read whole is `head_unreadable` | Nothing the eval reads can be given in its place; the first dispatch after the Maintainer's run shows it | + +## Why + +A case's head was read with no credential, from the public repo its change was made in, which ties every run of the set to an hourly cap it shares with nothing else of the Factory's, and to GitHub keeping the commit. Kept in the home and pinned by the case, the head is read with the token every case is already read with, and is the same bytes on every run. The tree beside it lets the eval hold the head to the commit as the live review does, so a case can no longer be graded on files the live review would refuse. + +## Boundaries + +| Boundary | Check | +|---|---| +| What the home keeps beside a case is asked with the raw media type and the home's token, and handed back only as the bytes the case pins: an answer of another type, `+json` among them, one not found, refused, over its bound or of another hash is none, and a name, kind, pin or home that could name another file is not asked | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins`, `trials/P-62.json`, `adapters/heldout.py`, mutant: "a head beside a case of another hash is read", mutant: "what the home keeps beside a case is asked as JSON", mutant: "a head of any size is read", mutant: "a head answered under another type is read", mutant: "a head answered as raw JSON is read", mutant: "a head GitHub did not find is unanswered", mutant: "a head GitHub refused is read", mutant: "a file of any kind is asked beside a case", mutant: "a pin of any shape is asked by", mutant: "a file beside a case of any name is asked", mutant: "a home of any name is asked beside a case" | +| The fetch reads each case's head from the home by its pins, with no other transport, unpacks it and holds it to its tree; a head not the one pinned is `hash_differs`, one not read, unpacked or held whole `head_unreadable`, and the gate closes | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, `harness/eval/fetch.py`, `harness/head.py`, mutant: "the eval fetch reads no case's head", mutant: "a head is unpacked and never held to its tree", mutant: "a head not the one pinned is read as unreadable", mutant: "the tree's pin is never read", mutant: "a head is read beside another Role", mutant: "a tree that cannot be read stops the fetch" | +| An eval case requires both pins, each a whole sha256 | `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a tree pin may be cut short" | +| The Maintainer's tool writes a head beside its case only once it is held to its tree, pins what it wrote, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token" | diff --git a/examples/eval-case.also-at.json b/examples/eval-case.also-at.json index 677b63da..41a60425 100644 --- a/examples/eval-case.also-at.json +++ b/examples/eval-case.also-at.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/eval-case.json b/examples/eval-case.json index 71f4c29d..c5b29a6f 100644 --- a/examples/eval-case.json +++ b/examples/eval-case.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/eval-case.met.json b/examples/eval-case.met.json index c1988926..3359b0de 100644 --- a/examples/eval-case.met.json +++ b/examples/eval-case.met.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "met", "criterion": "PR-1", diff --git a/examples/eval-case.with-issue.json b/examples/eval-case.with-issue.json index b90c8556..ee635006 100644 --- a/examples/eval-case.with-issue.json +++ b/examples/eval-case.with-issue.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,3 +86,5 @@\n def parse(items):\n+ if not items:\n+ return []\n first = items[0]\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "met", "criterion": "PR-1", diff --git a/examples/invalid/eval-case.a-category-nobody-owns.json b/examples/invalid/eval-case.a-category-nobody-owns.json index 4921afac..c49ee422 100644 --- a/examples/invalid/eval-case.a-category-nobody-owns.json +++ b/examples/invalid/eval-case.a-category-nobody-owns.json @@ -237,6 +237,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.a-head-as-an-object.json b/examples/invalid/eval-case.a-head-as-an-object.json index 51aa0301..a94a35dd 100644 --- a/examples/invalid/eval-case.a-head-as-an-object.json +++ b/examples/invalid/eval-case.a-head-as-an-object.json @@ -240,6 +240,8 @@ "head": { "sha": "c514ff021d93934192c9dbb14b6c31c2b421e6c7" }, + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.a-place-in-also-at-with-no-lines.json b/examples/invalid/eval-case.a-place-in-also-at-with-no-lines.json index 0882bbc6..4c755c45 100644 --- a/examples/invalid/eval-case.a-place-in-also-at-with-no-lines.json +++ b/examples/invalid/eval-case.a-place-in-also-at-with-no-lines.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.a-place-in-also-at-with-no-path.json b/examples/invalid/eval-case.a-place-in-also-at-with-no-path.json index 4d839688..d7d7c46b 100644 --- a/examples/invalid/eval-case.a-place-in-also-at-with-no-path.json +++ b/examples/invalid/eval-case.a-place-in-also-at-with-no-path.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.a-short-head.json b/examples/invalid/eval-case.a-short-head.json index 4f7ea3fa..8e4aa0ba 100644 --- a/examples/invalid/eval-case.a-short-head.json +++ b/examples/invalid/eval-case.a-short-head.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff0", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.a-tree-pin-cut-short.json b/examples/invalid/eval-case.a-tree-pin-cut-short.json new file mode 100644 index 00000000..2a74e6b8 --- /dev/null +++ b/examples/invalid/eval-case.a-tree-pin-cut-short.json @@ -0,0 +1,256 @@ +{ + "schema": "eval-case/1", + "id": "synthetic-early-return", + "role": "pr-review", + "intent": "review-change", + "facts": { + "schema": "facts/1", + "item": { + "repo": "combsmith/software-factory", + "type": "pull_request", + "number": 42 + }, + "state": "open", + "is_draft": false, + "class": "maintainer", + "author": "don-petry", + "last_pusher": "don-petry", + "targets_protected": true, + "mergeable": true, + "content_id": "sha256:5d0c", + "diff": { + "files": 3, + "lines": 120, + "truncated": false, + "characters": 4800 + }, + "changed_paths": [ + { + "path": "kernel/derive.py", + "previous": "kernel/derive_old.py" + }, + { + "path": "tests/test_decide.py", + "previous": null + }, + { + "path": "docs/glossary.md", + "previous": null + } + ], + "label_events": [ + { + "label": "factory:hold", + "action": "labeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:40:00Z" + }, + { + "label": "factory:hold", + "action": "unlabeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:45:00Z" + } + ], + "last_push_at": "2026-10-11T12:00:00Z", + "last_reopened_at": null, + "description_edited_at": null, + "expired_reviews": [], + "last_writer_action_at": "2026-10-11T12:00:00Z", + "reviews": [ + { + "by": { + "kind": "machine_user", + "name": "donpetry-bot", + "can_write": true + }, + "state": "dismissed", + "commit": "4b1a9c0", + "content_id": "sha256:4b1a", + "submitted_at": "2026-10-11T11:50:00Z", + "dismissed_by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "dismissed_from": "approved" + }, + { + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "state": "changes_requested", + "commit": "5d0c7e2", + "content_id": "sha256:5d0c", + "submitted_at": "2026-10-11T12:18:00Z", + "dismissed_by": null, + "dismissed_from": null + } + ], + "threads": [ + { + "id": "PRRT_kw7", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "pr-review", + "can_write": true + }, + "category": "functional", + "severity": "major", + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw1", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "coderabbit", + "can_write": false + }, + "category": null, + "severity": null, + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw3", + "resolved": true, + "opened_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "category": "human", + "severity": null, + "resolved_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "opened_at": "2026-10-11T12:16:00Z", + "ruling": null + } + ], + "checks": [ + { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "Secret scan (gitleaks)", + "app_id": 15368, + "workflow": ".github/workflows/ci.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "CodeQL", + "app_id": 57789, + "workflow": null, + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + } + ], + "queue": false, + "acts_since_writer": 2, + "acceptance": { + "approved": { + "digest": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "commit": "1a2b3c4", + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "criteria": [ + { + "id": "AC-1", + "kind": "test" + }, + { + "id": "AC-2", + "kind": "test" + } + ] + }, + "head": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "results": [ + { + "criterion": "AC-1", + "content_id": "sha256:4b1a", + "result": "failing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-1", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-2", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + } + ] + }, + "complete": { + "acceptance": true, + "reviews": true, + "threads": true, + "checks": true, + "diff": true, + "changed_paths": true, + "runs": true, + "timeline": true + }, + "read_at": "2026-10-11T12:20:00Z" + }, + "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", + "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d242901", + "expected": { + "verdict": "finding", + "criterion": "PR-3", + "finding": { + "category": "functional", + "severity": "major", + "location": { + "path": "kernel/decide.py", + "start_line": 86, + "end_line": 94 + } + } + } +} diff --git a/examples/invalid/eval-case.a-verdict-in-words.json b/examples/invalid/eval-case.a-verdict-in-words.json index 699ba814..4d7a417d 100644 --- a/examples/invalid/eval-case.a-verdict-in-words.json +++ b/examples/invalid/eval-case.a-verdict-in-words.json @@ -237,6 +237,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "the reviewer should object", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.an-issue-with-no-body.json b/examples/invalid/eval-case.an-issue-with-no-body.json index 448315d3..8fd4df4c 100644 --- a/examples/invalid/eval-case.an-issue-with-no-body.json +++ b/examples/invalid/eval-case.an-issue-with-no-body.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,3 +86,5 @@\n def parse(items):\n+ if not items:\n+ return []\n first = items[0]\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "met", "criterion": "PR-1", diff --git a/examples/invalid/eval-case.four-places-in-also-at.json b/examples/invalid/eval-case.four-places-in-also-at.json index 6043f5c5..eabffbb9 100644 --- a/examples/invalid/eval-case.four-places-in-also-at.json +++ b/examples/invalid/eval-case.four-places-in-also-at.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/examples/invalid/eval-case.no-head-pin.json b/examples/invalid/eval-case.no-head-pin.json new file mode 100644 index 00000000..f23aa9d8 --- /dev/null +++ b/examples/invalid/eval-case.no-head-pin.json @@ -0,0 +1,255 @@ +{ + "schema": "eval-case/1", + "id": "synthetic-early-return", + "role": "pr-review", + "intent": "review-change", + "facts": { + "schema": "facts/1", + "item": { + "repo": "combsmith/software-factory", + "type": "pull_request", + "number": 42 + }, + "state": "open", + "is_draft": false, + "class": "maintainer", + "author": "don-petry", + "last_pusher": "don-petry", + "targets_protected": true, + "mergeable": true, + "content_id": "sha256:5d0c", + "diff": { + "files": 3, + "lines": 120, + "truncated": false, + "characters": 4800 + }, + "changed_paths": [ + { + "path": "kernel/derive.py", + "previous": "kernel/derive_old.py" + }, + { + "path": "tests/test_decide.py", + "previous": null + }, + { + "path": "docs/glossary.md", + "previous": null + } + ], + "label_events": [ + { + "label": "factory:hold", + "action": "labeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:40:00Z" + }, + { + "label": "factory:hold", + "action": "unlabeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:45:00Z" + } + ], + "last_push_at": "2026-10-11T12:00:00Z", + "last_reopened_at": null, + "description_edited_at": null, + "expired_reviews": [], + "last_writer_action_at": "2026-10-11T12:00:00Z", + "reviews": [ + { + "by": { + "kind": "machine_user", + "name": "donpetry-bot", + "can_write": true + }, + "state": "dismissed", + "commit": "4b1a9c0", + "content_id": "sha256:4b1a", + "submitted_at": "2026-10-11T11:50:00Z", + "dismissed_by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "dismissed_from": "approved" + }, + { + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "state": "changes_requested", + "commit": "5d0c7e2", + "content_id": "sha256:5d0c", + "submitted_at": "2026-10-11T12:18:00Z", + "dismissed_by": null, + "dismissed_from": null + } + ], + "threads": [ + { + "id": "PRRT_kw7", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "pr-review", + "can_write": true + }, + "category": "functional", + "severity": "major", + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw1", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "coderabbit", + "can_write": false + }, + "category": null, + "severity": null, + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw3", + "resolved": true, + "opened_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "category": "human", + "severity": null, + "resolved_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "opened_at": "2026-10-11T12:16:00Z", + "ruling": null + } + ], + "checks": [ + { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "Secret scan (gitleaks)", + "app_id": 15368, + "workflow": ".github/workflows/ci.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "CodeQL", + "app_id": 57789, + "workflow": null, + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + } + ], + "queue": false, + "acts_since_writer": 2, + "acceptance": { + "approved": { + "digest": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "commit": "1a2b3c4", + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "criteria": [ + { + "id": "AC-1", + "kind": "test" + }, + { + "id": "AC-2", + "kind": "test" + } + ] + }, + "head": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "results": [ + { + "criterion": "AC-1", + "content_id": "sha256:4b1a", + "result": "failing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-1", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-2", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + } + ] + }, + "complete": { + "acceptance": true, + "reviews": true, + "threads": true, + "checks": true, + "diff": true, + "changed_paths": true, + "runs": true, + "timeline": true + }, + "read_at": "2026-10-11T12:20:00Z" + }, + "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", + "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", + "expected": { + "verdict": "finding", + "criterion": "PR-3", + "finding": { + "category": "functional", + "severity": "major", + "location": { + "path": "kernel/decide.py", + "start_line": 86, + "end_line": 94 + } + } + } +} diff --git a/examples/invalid/eval-case.no-head.json b/examples/invalid/eval-case.no-head.json index 1078d64e..e1d6afd8 100644 --- a/examples/invalid/eval-case.no-head.json +++ b/examples/invalid/eval-case.no-head.json @@ -249,5 +249,7 @@ "end_line": 94 } } - } + }, + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010" } diff --git a/examples/invalid/eval-case.no-place-in-also-at.json b/examples/invalid/eval-case.no-place-in-also-at.json index 2b505196..7dfe2724 100644 --- a/examples/invalid/eval-case.no-place-in-also-at.json +++ b/examples/invalid/eval-case.no-place-in-also-at.json @@ -238,6 +238,8 @@ }, "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", "expected": { "verdict": "finding", "criterion": "PR-3", diff --git a/harness/eval/AGENTS.md b/harness/eval/AGENTS.md index 44ec90a3..9ca16451 100644 --- a/harness/eval/AGENTS.md +++ b/harness/eval/AGENTS.md @@ -1,7 +1,7 @@ # harness/eval/ -The eval job's steps, a module each, in the order `__init__.py` gives. Their rules are `evals/AGENTS.md`'s, beside the sets they run, but for what two issues add, which are here: -issue 612, "The eval job reads a named set, and a gate set's report is aggregates only", and issue 613, "The gate pools k dispatches on one commit and reads all of them". +The eval job's steps, a module each, in the order `__init__.py` gives. Their rules are `evals/AGENTS.md`'s, beside the sets they run, but for what three issues add, which are here: +issue 612, "The eval job reads a named set, and a gate set's report is aggregates only", issue 613, "The gate pools k dispatches on one commit and reads all of them", and issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded". A dispatch names its set. A set whose manifest says it is a gate set is reported by counts alone, `eval-aggregate/1`, and each case's outcome with its Result goes to a ledger, `eval-ledger/1`, kept in the scratch. `seal.py` encrypts the ledger with `openssl cms` to each of the Maintainer's certificates that its pins, in a protected path under `.github/`, name by sha256 (`seal.RECIPIENTS`). The ciphertext alone is uploaded, as `eval-ledger`. No key that opens a ledger is in the job or the repo, and nothing automated reads one. Until the Maintainer commits his certificates, a gate set's run fails at the seal, `no_recipient`. @@ -9,6 +9,7 @@ The ciphertext lives only as a run artifact under its retention until a storage A gate set gives its own bars and least of a kind. The Release's gate pools every dispatch on a commit of the gate set `config/harness.json` names, `gate_set`. Until one is named, the gate reads the review Intent's set by its newest successful dispatch, which a re-dispatch can still make the best of several. A gate set's dispatch asks its preflight first, a few cases of the review Intent's set scored apart, so that a broken pipeline stops it before any held-out case is asked. +A case's head is read from the home beside the case, its tarball and its tree each by the sha256 the case pins, and held to that tree (issue 611); nothing in the job reads with no credential. | Rule | Held by | |---|---| @@ -26,5 +27,6 @@ A gate set's dispatch asks its preflight first, a few cases of the review Intent | The pool is three runs at least, every one, each kind summed under the set's bars; one faulted closes it | `tests/test_evals.py::test_the_gate_pools_every_dispatch_on_the_commit_and_one_faulted_closes_it` | | A gate set's preflight is three to five of the review Intent's cases, none a gate set's by its id or its hash | `tests/test_evals.py::test_a_gate_sets_preflight_is_of_the_development_set_and_no_gate_sets_case` | | The preflight is asked first; a fault in it, or no pass, stops the dispatch before a case of the set is asked, and the gate reads it as faulted | `tests/test_evals.py::test_a_gate_sets_dispatch_asks_its_preflight_first_and_a_broken_one_stops_it` | +| Each case's head is read from the home by its pins, unpacked and held to its tree; one not the one pinned, or not read whole, closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate` | | The Release's gate judges a gate set named in the configuration by every completed dispatch on the commit | `tests/test_release_step.py::test_a_gate_set_is_judged_by_every_dispatch_on_the_commit` | | Each of these rules has a mutant that a test kills | `tools/mutants.py` | diff --git a/harness/eval/__init__.py b/harness/eval/__init__.py index 442a80bf..140ecc48 100644 --- a/harness/eval/__init__.py +++ b/harness/eval/__init__.py @@ -6,7 +6,8 @@ names; a set of no case ends the run here, `eval_set_empty`, before any credential is used (ruling 8); - `fetch` reads each case from the private home through `adapters/heldout.py`, with the Maintainer's read-only token, into a scratch directory outside the workspace, on a runner thrown away when the job ends, and each case's head, the - commit its diff was taken at, as its tarball with no credential, unpacked by hand beside it (decision 0138); + commit its diff was taken at, from the home too: its tarball and its commit's tree, kept beside the case and each + pinned by its sha256 there, the tarball unpacked by hand and held to the tree (decision 0138, issue 611); - `run` gives each case to the review Intent, its Task built and its Engine called by the functions a live Run's are (`harness/task.from_admission`, `harness/run.answer`), and scores its Result with `tools/eval_result.py`; for a gate set, the development cases its manifest names as its preflight first, scored apart (issue 613); @@ -358,43 +359,58 @@ def seen_of(review) -> dict: return said -def head_of(data: bytes, folder: pathlib.Path, ident: str, transport): - """The head a case names, its tarball read through `transport` and unpacked into `folder` under its id, or the code - `head_unreadable` when it cannot be read or unpacked whole; None, and nothing read, for a case with no head to read, - which its schema refuses after (decision 0138). Nothing of the head is printed or kept but its files there.""" +def head_of(env: dict, data: bytes, folder: pathlib.Path, role: str, ident: str, transport): + """The head a case names, read from the home through `transport` beside the case, its tarball and its commit's tree + each by the sha256 the case pins, unpacked into `folder` under its id and held to that tree file by file (issue 611; + decision 0142); or the code that says why not: `hash_differs` for a file not the one pinned, and `head_unreadable` + for one that cannot be read, unpacked or held whole. None, and nothing read, for a case with no head to read, which + its schema refuses after (decision 0138). Nothing of the head is printed or kept but its files there.""" + from adapters import heldout from harness import head try: case = json.loads(data) - repo, sha = case["facts"]["item"]["repo"], case["head"] + sha, pins = case["head"], (case["head_sha256"], case["tree_sha256"]) except (ValueError, TypeError, KeyError, RecursionError): return None - if not (isinstance(sha, str) and COMMIT.fullmatch(sha) and isinstance(repo, str)): + if not (isinstance(sha, str) and COMMIT.fullmatch(sha)): return None + read = {} + for kind, pin in zip(("tar.gz", "tree.json"), pins): + read[kind], code = heldout.beside(transport, env.get("HELDOUT_OWNER"), env.get("HELDOUT_REPO"), role, ident, + kind, pin) + if code is not None: + return "hash_differs" if code == "hash_differs" else "head_unreadable" folder.mkdir(mode=0o700, exist_ok=True) + into = None try: - head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident)) - except (head.Refused, OSError, Stop): - shutil.rmtree(str(folder / ident), ignore_errors=True) + tree = head.tree_of(json.loads(read["tree.json"])) + into = within(folder, ident) + head.unpack(read["tar.gz"], sha, into) + head.held(into, tree) + except (head.Refused, OSError, Stop, ValueError, RecursionError): + if into is not None: + shutil.rmtree(into, ignore_errors=True) return "head_unreadable" return None -def fetch(env: dict, root: pathlib.Path, workspace: pathlib.Path, transport, heads=None) -> str: +def fetch(env: dict, root: pathlib.Path, workspace: pathlib.Path, transport) -> str: """Each case the set names, read from the home into the scratch directory as the bytes its manifest hashed, and why each that is not there is not: `fetched.json`, by id, a code or null, and `fetch-causes.json`, by id, what a read - that raised raised, by its class and place alone. Each case's head is read through `heads`, with no credential, and - a case whose head cannot be read whole is not there either, `head_unreadable`. Heads are unpacked in `heads_place`, - apart from the cases (issue 623). A gate set's preflight is read first, the same way, into the scratch's own folder - of it, PREFLIGHT; its heads beside the set's, by their ids, which `preflight_of` holds apart (issue 613).""" + that raised raised, by its class and place alone. Each case's head is read from the home too, beside the case, by + `head_of`, and a case whose head cannot be read whole is not there either. No head is read with no credential + (issue 611). Heads are unpacked in `heads_place`, apart from the cases (issue 623). A gate set's preflight is read + first, the same way, into the scratch's own folder of it, PREFLIGHT; its heads beside the set's, by their ids, which + `preflight_of` holds apart (issue 613).""" kept, found, place = _planned(env, root, workspace) said = "" if kept["preflight"]: - ahead = _fetched(env, workspace, transport, heads, dict(found, cases=kept["preflight"]), place / PREFLIGHT) + ahead = _fetched(env, workspace, transport, dict(found, cases=kept["preflight"]), place / PREFLIGHT) said = ", %d of %d to preflight" % (ahead, len(kept["preflight"])) - return "fetched %d of %d cases" % (_fetched(env, workspace, transport, heads, found, place), found["count"]) + said + return "fetched %d of %d cases" % (_fetched(env, workspace, transport, found, place), found["count"]) + said -def _fetched(env: dict, workspace: pathlib.Path, transport, heads, found: dict, place: pathlib.Path) -> int: +def _fetched(env: dict, workspace: pathlib.Path, transport, found: dict, place: pathlib.Path) -> int: """How many of `found`'s cases `fetch` read into `place`, writing why each other was not read there.""" from adapters import heldout why, causes = {}, {} @@ -405,8 +421,8 @@ def _fetched(env: dict, workspace: pathlib.Path, transport, heads, found: dict, except Exception as err: data, code = None, "unanswered" causes[entry["id"]] = cause_of(err) - if data is not None and heads is not None: - code = head_of(data, heads_place(env, workspace), entry["id"], heads) + if data is not None: + code = head_of(env, data, heads_place(env, workspace), found["role"], entry["id"], transport) why[entry["id"]] = code if data is not None and code is None: path = within(place / "cases", "%s.json" % entry["id"]) diff --git a/harness/eval/canary.py b/harness/eval/canary.py index 8126a17d..187dca81 100644 --- a/harness/eval/canary.py +++ b/harness/eval/canary.py @@ -15,6 +15,7 @@ import json import os import pathlib +import shutil import subprocess import sys import tempfile @@ -49,6 +50,22 @@ def asked(args, env, timeout, **given): return asked +def head_of(data: bytes, folder: pathlib.Path, ident: str, transport): + """The example's head, its commit's tarball read through `transport`, with no credential, from this repo, which is + public, and unpacked into `folder` under its id; or `head_unreadable`. The eval job reads a case's head from the + held-out home instead, pinned by the case (issue 611): the example is no case of the home, and pins none it keeps.""" + from harness import head + case = json.loads(data) + repo, sha = case["facts"]["item"]["repo"], case["head"] + folder.mkdir(mode=0o700, exist_ok=True) + try: + head.unpack(head.fetch(transport, repo, sha), sha, job.within(folder, ident)) + except (head.Refused, OSError, job.Stop): + shutil.rmtree(str(folder / ident), ignore_errors=True) + return "head_unreadable" + return None + + def review(env: dict, call=live.cli_call, clock=time.time, heads=None) -> int: """One review by the eval job's path: 0 if the CLI answered in a status the Engine may write, its answer read and scored, and 1 otherwise, a `failed` Result among them. A scorer that raises fails the step. The example's head is @@ -72,7 +89,7 @@ def answered(given: dict) -> dict: if heads is None: from adapters import github heads = github.Live.anonymous() - fetched = job.head_of(data, pathlib.Path(unpacked), case["id"], heads) + fetched = head_of(data, pathlib.Path(unpacked), case["id"], heads) verdict, reasons = job.assess(found, {case["id"]: fetched}, place, answered)[case["id"]] status = results[0].get("status") if results else None print("status %s, verdict %s, reasons %s" % (status, verdict, ",".join(reasons) or "none")) diff --git a/harness/eval/fetch.py b/harness/eval/fetch.py index db558772..36411aba 100644 --- a/harness/eval/fetch.py +++ b/harness/eval/fetch.py @@ -1,7 +1,8 @@ """The eval job's fetch: each case from the private home, with the Maintainer's own token for it, read-only on the home alone (decision 0053), and nothing else. The token is used for reads alone: a request that is not a GET is refused -before it is sent, so no change to this step's code can write with it. Each case's head is read with no credential at -all, its tarball alone (decision 0138): the cases' repo is public, and the home's token is never sent anywhere else.""" +before it is sent, so no change to this step's code can write with it. Each case's head is read from the home as well, +its tarball and its commit's tree beside the case, each pinned by its sha256 there (issue 611): the home's token is +never sent anywhere else, and nothing is read with no credential.""" from harness.eval import Stop, exit_with, fetch @@ -27,8 +28,7 @@ def transport(env: dict): def step(env, root, workspace): - from adapters import github - return fetch(env, root, workspace, transport(env), github.Live.anonymous()) + return fetch(env, root, workspace, transport(env)) if __name__ == "__main__": diff --git a/harness/head.py b/harness/head.py index 6b92ed17..d00ad362 100644 --- a/harness/head.py +++ b/harness/head.py @@ -234,9 +234,15 @@ def tree(transport, repo: str, sha: str) -> dict: answer = transport.send(github.request("GET", path)) except github.Unanswered: raise Refused("the commit's tree could not be read") from None - body = answer.get("body") - if answer.get("status") != 200 or not isinstance(body, dict) or body.get("truncated") is not False \ - or not isinstance(body.get("tree"), list): + if answer.get("status") != 200: + raise Refused("the commit's tree was not answered whole") + return tree_of(answer.get("body")) + + +def tree_of(body) -> dict: + """A commit's tree, every level of it, as `tree` reads GitHub's answer, from the answer's body: so an eval case's + tree, kept beside it in the held-out home as GitHub answered it, is read by the same checks (issue 611).""" + if not isinstance(body, dict) or body.get("truncated") is not False or not isinstance(body.get("tree"), list): raise Refused("the commit's tree was not answered whole") files, links, seen = {}, set(), set() for entry in body["tree"]: diff --git a/schemas/eval-case.schema.json b/schemas/eval-case.schema.json index 2492886c..4eff02ed 100644 --- a/schemas/eval-case.schema.json +++ b/schemas/eval-case.schema.json @@ -57,6 +57,8 @@ "facts", "diff", "head", + "head_sha256", + "tree_sha256", "expected" ], "properties": { @@ -86,6 +88,16 @@ "type": "string", "pattern": "^[0-9a-f]{40}$" }, + "head_sha256": { + "description": "The sha256 of the head's tarball as GitHub archived the commit, kept beside the case in the held-out home, `/.tar.gz`: the eval job reads the head there, with the home's token, and none with no credential (issue 611).", + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, + "tree_sha256": { + "description": "The sha256 of the commit's tree, every level of it, as GitHub answered it, kept beside the case in the held-out home, `/.tree.json`: the head is held to it file by file (issue 611; decision 0142).", + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, "issue": { "description": "The issue the change serves, as a live Run's Task holds it (decision 0110): optional, so a case without one is reviewed on its diff alone.", "type": "object", diff --git a/tests/test_adapters.py b/tests/test_adapters.py index 9349abf4..974bef69 100644 --- a/tests/test_adapters.py +++ b/tests/test_adapters.py @@ -3489,6 +3489,17 @@ def home_file(name: str, data: bytes, path: str = None, **change) -> dict: "response": {"status": 200, "headers": {"content-type": "application/json; charset=utf-8"}, "body": body}} +def home_beside(name: str, kind: str, data: bytes, status: int = 200, role: str = "pr-review", **change) -> dict: + """What a stand-in home, `o/home`, keeps beside a case, `/.`, its head's tarball or its tree, as the + contents API answers the raw media type: the bytes as they are, under its own type (trials/P-62.json).""" + from adapters import github, heldout + headers = dict({"content-type": heldout.RAW + "; charset=utf-8", "content-length": str(len(data))}, **change) + return {"request": {"via": "rest", "call": "GET /repos/o/home/contents/%s/%s.%s" % (role, name, kind), + "headers": {"accept": heldout.RAW}}, + "response": {"status": status, "headers": headers, + "body": data.decode("utf-8", errors="surrogateescape") if status == 200 else "404: Not Found"}} + + def home_set(*named) -> dict: return {"schema": "eval-set/1", "role": "pr-review", "intent": "review-change", "count": len(named), "cases": [{"id": n, "sha256": hashlib.sha256(d).hexdigest()} for n, d in named]} @@ -3553,6 +3564,59 @@ def send(self, req): self.assertEqual(asked, []) self.assertEqual(heldout.case(Anything(), "o", "home", "pr-review", entry), (self.ONE, None)) self.assertEqual(asked, ["/repos/o/home/contents/pr-review/c-1.json"]) + + def test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins(self): # issue 611 + # A head's tarball and its tree, beside the case, are asked with the raw media type, whose answer is the file's + # bytes whole however large (trials/P-62.json), and handed back only as the bytes the case pinned. + tar, tree = bytes(range(256)) * 9, b'{"tree": [], "truncated": false}\n' + pins = {"tar.gz": hashlib.sha256(tar).hexdigest(), "tree.json": hashlib.sha256(tree).hexdigest()} + # The answer's headers as GitHub gave them to the Maintainer's login for a file of 1.5 MiB (P-62's exchange 5). + p62 = json.loads((ROOT / "trials" / "P-62.json").read_text())["exchanges"][5] + self.assertEqual(p62["request"]["headers"]["accept"], heldout.RAW) + grounded = {k: v for k, v in p62["response"]["headers"].items() if k in ("content-type",)} + home = github.Replay([home_beside("c-1", "tar.gz", tar, **grounded), home_beside("c-1", "tree.json", tree)]) + self.assertEqual([heldout.beside(home, "o", "home", "pr-review", "c-1", k, pins[k]) for k in ("tar.gz", "tree.json")], + [(tar, None), (tree, None)]) + + def read(found, kind="tar.gz", pin=pins["tar.gz"], owner="o", name="home", role="pr-review", ident="c-1"): + return heldout.beside(github.Replay(found), owner, name, role, ident, kind, pin) + json_typed = home_beside("c-1", "tar.gz", tar, **{"content-type": "application/json; charset=utf-8"}) + raw_json = home_beside("c-1", "tar.gz", tar, **{"content-type": heldout.RAW + "+json; charset=utf-8"}) + untyped = home_beside("c-1", "tar.gz", tar) + del untyped["response"]["headers"]["content-type"] + failed = home_beside("c-1", "tar.gz", tar) + failed["response"]["status"] = 500 + large = home_beside("c-1", "tar.gz", b"\0" * (heldout.BESIDE["tar.gz"] + 1)) + for found, code in (([home_beside("c-1", "tar.gz", tar[1:])], "hash_differs"), + ([home_beside("c-1", "tar.gz", tar, status=404)], "not_found"), + ([failed], "unanswered"), ([], "unanswered"), ([json_typed], "unanswered"), + ([raw_json], "unanswered"), ([untyped], "unanswered"), + ([large], "too_large"), ([home_beside("c-1", "tree.json", tree)], "unanswered")): + self.assertEqual(read(found), (None, code), code) + self.assertEqual(read([home_beside("c-1", "tree.json", b"\0" * (heldout.BESIDE["tree.json"] + 1))], + kind="tree.json", pin=pins["tree.json"]), (None, "too_large")) + self.assertEqual(read([home_beside("c-1", "tree.json", tree)], kind="tree.json", pin=pins["tar.gz"]), + (None, "hash_differs")) + # A name, a kind or a pin that could name another file, or a home that could name another repo, is refused + # before anything is asked. + asked = [] + + class Anything: + def send(self, req): + asked.append((req["path"], req["accept"])) + return home_beside("c-1", "tar.gz", tar)["response"] + for kind, role, ident, pin in (("json", "pr-review", "c-1", pins["tar.gz"]), ("tar.gz", "pr-review/..", "c-1", pins["tar.gz"]), + ("tar.gz", "pr-review", "../c-1", pins["tar.gz"]), ("tar.gz", "pr-review", "c-1.json", pins["tar.gz"]), + ("tar.gz", "pr-review", "c-1", "x"), ("tar.gz", "pr-review", "c-1", None), + ("tar.gz", None, "c-1", pins["tar.gz"]), (None, "pr-review", "c-1", pins["tar.gz"])): + self.assertEqual(heldout.beside(Anything(), "o", "home", role, ident, kind, pin), (None, "not_this_set"), + (kind, role, ident, pin)) + for owner, name in (("o", ".."), ("o/x", "home"), (None, "home"), ("o", None)): + self.assertEqual(heldout.beside(Anything(), owner, name, "pr-review", "c-1", "tar.gz", pins["tar.gz"]), + (None, "unanswered"), (owner, name)) + self.assertEqual(asked, []) + self.assertEqual(heldout.beside(Anything(), "o", "home", "pr-review", "c-1", "tar.gz", pins["tar.gz"]), (tar, None)) + self.assertEqual(asked, [("/repos/o/home/contents/pr-review/c-1.tar.gz", heldout.RAW)]) def per_hundred(found: dict) -> dict: """A fixture's exchange asked as the port asks a list: a full page of a hundred, GitHub's answer kept.""" found = copy.deepcopy(found) diff --git a/tests/test_evals.py b/tests/test_evals.py index 0a0c69a4..a26525fd 100644 --- a/tests/test_evals.py +++ b/tests/test_evals.py @@ -75,8 +75,9 @@ def opened(sealed: pathlib.Path, cert: pathlib.Path, key: pathlib.Path) -> bytes class Heads: - """A transport standing in for the anonymous one `fetch` reads each case's head with: each commit's tarball as - codeload serves one (P-60), or, for a commit in `refused`, a 404. What it was asked is kept by path.""" + """A transport standing in for the anonymous one the canary reads its example's head with: each commit's tarball as + codeload serves one (P-60), or, for a commit in `refused`, a 404. What it was asked is kept by path. The eval job + reads a case's head from the home instead (issue 611).""" def __init__(self, refused=()): self.refused, self.asked = set(refused), [] @@ -91,6 +92,46 @@ def send(self, req: dict) -> dict: return {"status": 200, "headers": {}, "body": support.tarball(sha).decode("utf-8", errors="surrogateescape")} +def stand_in_head(sha: str) -> tuple: + """`(tarball, tree)`: what the home keeps beside a case whose head is `sha`, its tarball as codeload serves one (P-60) + and its commit's tree as the API answers it (trials/P-60.tree.json), each byte for byte as the eval job reads it + there (issue 611). The head holds one file, as `tests/support.tarball` writes it.""" + from tests import support + body = b"# a repo\n" + blob = hashlib.sha1(b"blob %d\0" % len(body) + body).hexdigest() + tree = {"sha": "e" * 40, "truncated": False, + "tree": [{"path": "README.md", "mode": "100644", "type": "blob", "sha": blob, "size": len(body)}]} + return support.tarball(sha), (json.dumps(tree, indent=2) + "\n").encode() + + +def pinned(case: dict) -> dict: + """`case`, its head pinned by the sha256 of its stand-in tarball and tree, as the Maintainer's tool pins a case's.""" + tar, tree = stand_in_head(case["head"]) + return dict(case, head_sha256=hashlib.sha256(tar).hexdigest(), tree_sha256=hashlib.sha256(tree).hexdigest()) + + +def home_of(cases: dict, kept=None) -> list: + """The exchanges of a stand-in home for each of `cases`, in order: the case, and, for one that pins a head, what the + home keeps beside it, `kept(id, sha)` where given, as far as the job asks: the tree is not asked once the tarball is + not the one pinned.""" + from tests.test_adapters import home_beside, home_file + exchanges = [] + for ident, data in cases.items(): + exchanges.append(home_file(ident, data)) + try: + case = json.loads(data) + sha, pins = case["head"], (case["head_sha256"], case["tree_sha256"]) + except (ValueError, TypeError, KeyError, RecursionError): + continue + if not (isinstance(sha, str) and re.fullmatch(r"[0-9a-f]{40}", sha)): + continue + tar, tree = (kept or (lambda i, h: stand_in_head(h)))(ident, sha) + exchanges.append(home_beside(ident, "tar.gz", tar)) + if hashlib.sha256(tar).hexdigest() == pins[0]: + exchanges.append(home_beside(ident, "tree.json", tree)) + return exchanges + + def github_tarred() -> int: from adapters import github return github.TARRED @@ -924,7 +965,7 @@ def setUp(self): self.root, self.work, self.temp = base / "root", base / "work", base / "temp" for place in (self.root, self.work, self.temp): place.mkdir() - one = load("examples/eval-case.json") + one = pinned(load("examples/eval-case.json")) # A case's facts name the content its diff is, as the set's must. one["facts"]["content_id"] = "sha256:" + hashlib.sha256(one["diff"].encode()).hexdigest() two = dict(copy.deepcopy(one), id="poc-two") @@ -978,19 +1019,16 @@ def path(self) -> pathlib.Path: def step(self, step, *given, env=None): from harness import eval as job - if getattr(step, "__module__", None) == job.__name__ and getattr(step, "__name__", None) == "fetch" \ - and len(given) == 1: - given += (Heads(),) # each case's head, read as codeload would answer it (decision 0138) try: return step(dict(self.env, **(env or {})), self.root, self.work, *given) except job.Stop as stop: return stop.code def home(self, cases: dict, gate: bool = False): - """A home that answers for each of `cases`, in order; for a gate set's run, for its preflight first.""" + """A home that answers for each of `cases`, in order, with what it keeps beside each; for a gate set's run, for + its preflight first.""" from adapters import github - from tests.test_adapters import home_file - return github.Replay([home_file(i, d) for i, d in (dict(self.pre, **cases) if gate else cases).items()]) + return github.Replay(home_of(dict(self.pre, **cases) if gate else cases)) def test_a_set_of_no_case_ends_the_run_before_anything_is_read(self): import os @@ -1597,7 +1635,7 @@ def test_the_job_writes_each_kinds_word_and_the_judge_holds_it(self): # decisio for raw in self.cases.values(): if json.loads(raw)["expected"]["verdict"] == "met": met = json.loads(raw) - met = met or load("examples/eval-case.met.json") + met = met or pinned(load("examples/eval-case.met.json")) met["facts"] = dict(met["facts"], content_id="sha256:" + hashlib.sha256(met["diff"].encode()).hexdigest()) def run(miss): @@ -1649,7 +1687,7 @@ def reviewed(self): """A held-out case of the review Intent, as #183 writes them: its facts name the content its diff is, so its Task is a live Run's Task of that content. Made from tests/test_harness.py's change; nothing of the home.""" from tests import test_harness as live - case = dict(load("examples/eval-case.json"), id="poc-reviewed", facts=live.facts(), diff=live.DIFF.decode()) + case = pinned(dict(load("examples/eval-case.json"), id="poc-reviewed", facts=live.facts(), diff=live.DIFF.decode())) return case, live def test_the_facts_the_set_was_made_with_read_as_todays_once_filled(self): # decision 0105 @@ -1717,13 +1755,17 @@ def answering(the_task, *given, **named): built.append(json.dumps(the_task, sort_keys=True)) return answered(the_task, *given, **named) - def scored(heads=None, case=case): + def scored(kept=None, case=case): + from adapters import github seen["files"], seen["folders"] = {}, [] self.clean() # each run's scratch and heads are its own + if kept is not None: # the head the home keeps, and the case pins it + tar, tree = kept(case["id"], case["head"]) + case = dict(case, head_sha256=hashlib.sha256(tar).hexdigest(), tree_sha256=hashlib.sha256(tree).hexdigest()) cases = {case["id"]: (json.dumps(case, indent=2) + "\n").encode()} self.set(cases) self.step(job.plan) - self.step(job.fetch, self.home(cases), *([heads] if heads else [])) + self.step(job.fetch, github.Replay(home_of(cases, kept))) kept_plan = json.loads((self.temp / "heldout" / "plan.json").read_text()) (self.temp / "heldout" / "plan.json").write_text(json.dumps(dict(kept_plan, planned_at=1000))) cli = Reading(said=live.answer(live.judged(findings=[], effects={}))) @@ -1768,14 +1810,11 @@ def words(value): self.assertEqual(given[2], given[0]) # A head whose archive holds git's folder is refused whole: the case is unreadable by its code, and asks nothing. - class Historied(Heads): - def send(self, req): - from adapters import github - from tests import support - sha = github.TARBALL.fullmatch(req["path"]).group(3) - return {"status": 200, "headers": {}, "body": support.tarball(sha, [("README.md", b"0", b"# a repo\n"), - (".git/HEAD", b"0", b"ref: refs/heads/main\n")]).decode("utf-8", errors="surrogateescape")} - said, cli = scored(Historied()) + def historied(ident, sha): + from tests import support + return (support.tarball(sha, [("README.md", b"0", b"# a repo\n"), (".git/HEAD", b"0", b"ref: refs/heads/main\n")]), + stand_in_head(sha)[1]) + said, cli = scored(historied) self.assertEqual((said, cli.asked()), ("scored 1 of 1 cases, 0 passed", [])) self.assertEqual([c["reasons"] for c in json.loads((self.work / job.REPORT).read_text())["cases"]], [["head_unreadable"]]) @@ -2547,7 +2586,7 @@ def test_each_head_is_unpacked_apart_from_the_cases(self): # issue 623 # The plan makes the heads folder, as closed as the scratch (issue 626). self.assertTrue((self.temp / "heads").is_dir(), "the plan makes the heads folder") self.assertEqual(os.stat(str(self.temp / "heads")).st_mode & 0o777, 0o700) - self.assertEqual(self.step(fetch, self.home(cases), Heads()), "fetched 1 of 1 cases") + self.assertEqual(self.step(fetch, self.home(cases)), "fetched 1 of 1 cases") place = self.temp / "heldout" kept_plan = json.loads((place / "plan.json").read_text()) (place / "plan.json").write_text(json.dumps(dict(kept_plan, planned_at=1000))) @@ -2587,48 +2626,150 @@ def __call__(self, args, env, timeout, **given): (self.temp / "heldout").mkdir() self.assertEqual(self.step(plan), "scratch_unreadable") - def test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate(self): # decision 0138 - # The fetch reads each case's head, the commit its diff was taken at, as its tarball through a transport of its - # own, never the home's, and unpacks it under heads/; the run step reads it as the case's folder and removes it - # once the case is asked. A head that cannot be read whole is the case's `head_unreadable`, an Engine's or the - # Factory's fault, never the reviewer's: the gate closes. + def test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate(self): # issue 611 + # The fetch reads each case's head from the home, with the home's token, beside the case: its tarball, then its + # commit's tree, each by the raw media type and the sha256 the case pins; unpacks it under heads/ and holds it to + # the tree file by file (decision 0142). Nothing is read with no credential. The run step reads the head as the + # case's folder and removes it once the case is asked. A head not the one pinned is `hash_differs`, and one + # that cannot be read, unpacked or held whole `head_unreadable`, a fault of the run or the Engine, never the + # reviewer's: the gate closes. + from adapters import github, heldout from harness import eval as job from harness.eval import run as step + from tests import support + from tests.test_adapters import home_beside, home_file case, live = self.reviewed() - cases = {i: (json.dumps(dict(case, id=i, head=h), indent=2) + "\n").encode() - for i, h in (("poc-one", "1" * 40), ("poc-two", "2" * 40))} + tar, tree = stand_in_head(case["head"]) + other = {"sha": "e" * 40, "truncated": False, "tree": [dict(json.loads(tree)["tree"][0], path="OTHER.md")]} + truncated = dict(json.loads(tree), truncated=True) + kept = {"poc-one": (tar, tree), "poc-two": (None, None), "poc-three": (support.tarball("3" * 40), None), + "poc-four": (tar, (json.dumps(other) + "\n").encode()), "poc-five": (tar, b"not json\n"), + "poc-six": (tar, (json.dumps(truncated) + "\n").encode())} + cases, exchanges = {}, [] + for ident, (has, holds) in kept.items(): + # Each pins what the home keeps for poc-one, but poc-four, -five and -six pin the tree the home keeps. + pin = dict(case, id=ident) + if holds is not None: + pin["tree_sha256"] = hashlib.sha256(holds).hexdigest() + cases[ident] = (json.dumps(pin, indent=2) + "\n").encode() + exchanges.append(home_file(ident, cases[ident])) + exchanges.append(home_beside(ident, "tar.gz", b"", status=404) if has is None else home_beside(ident, "tar.gz", has)) + if has is not None and has == tar: + exchanges.append(home_beside(ident, "tree.json", holds)) self.set(cases) self.step(job.plan) - heads = Heads(refused={"2" * 40}) - self.assertEqual(self.step(job.fetch, self.home(cases), heads), "fetched 1 of 2 cases") - repo = case["facts"]["item"]["repo"] - self.assertEqual(heads.asked, [("/repos/%s/tarball/%s" % (repo, "1" * 40), True, github_tarred()), - ("/repos/%s/tarball/%s" % (repo, "2" * 40), True, github_tarred())]) + asked = [] + + class Home(github.Replay): + def send(self, req): + asked.append((req["path"].rsplit("/", 1)[-1], req["accept"])) + return super().send(req) + self.assertEqual(self.step(job.fetch, Home(exchanges)), "fetched 1 of 6 cases") + self.assertEqual(asked[:3], [("poc-one.json", github.JSON), ("poc-one.tar.gz", heldout.RAW), + ("poc-one.tree.json", heldout.RAW)]) + self.assertEqual(len(asked), len(exchanges), "each answer the home keeps is asked, once, in its order") place = self.temp / "heldout" - self.assertEqual(json.loads((place / "fetched.json").read_text()), {"poc-one": None, "poc-two": "head_unreadable"}) + self.assertEqual(json.loads((place / "fetched.json").read_text()), + {"poc-one": None, "poc-two": "head_unreadable", "poc-three": "hash_differs", + "poc-four": "head_unreadable", "poc-five": "head_unreadable", "poc-six": "head_unreadable"}) self.assertEqual(sorted(p.name for p in (self.temp / "heads").iterdir()), ["poc-one"]) - self.assertTrue((self.temp / "heads" / "poc-one" / "README.md").is_file()) + self.assertEqual((self.temp / "heads" / "poc-one" / "README.md").read_bytes(), b"# a repo\n") cli = live.CLI(said=live.answer(live.judged(findings=[], effects={}))) kept_plan = json.loads((place / "plan.json").read_text()) (place / "plan.json").write_text(json.dumps(dict(kept_plan, planned_at=1000))) self.step(step.step, cli, lambda: 1000.0, env=live.JOB) - asked, = cli.asked() - self.assertEqual(pathlib.Path(asked["head"]).parts[-2:], ("heads", "poc-one")) + asked_of, = cli.asked() + self.assertEqual(pathlib.Path(asked_of["head"]).parts[-2:], ("heads", "poc-one")) self.assertFalse((self.temp / "heads" / "poc-one").exists(), "removed once its case is asked") report = json.loads((self.work / job.REPORT).read_text()) + self.assertEqual({c["id"]: c["reasons"] for c in report["cases"]}["poc-three"], ["hash_differs"]) self.assertEqual({c["id"]: c["reasons"] for c in report["cases"]}["poc-two"], ["head_unreadable"]) self.assertIn("head_unreadable", job.ENGINE_FAULTS) self.assertEqual(self.step(job.judge.step if hasattr(job.judge, "step") else job.judge), "engine_fault") - # The fetch step's own transport for the heads sends no token: the home's is never sent anywhere else. + # The fetch step's one transport is the home's, read-only, with its token: no transport with no credential. import importlib - from adapters import github fetching = importlib.import_module("harness.eval.fetch") made = [] - with mock.patch.object(fetching, "fetch", lambda env, root, work, home, heads: made.append((home, heads)) or "x"): + with mock.patch.object(fetching, "fetch", lambda env, root, work, *given: made.append(given) or "x"): fetching.step({"HELDOUT_TOKEN": "a-stand-in-held-out-token"}, self.root, self.work) - (home, anonymous), = made - self.assertIsNone(anonymous._token) - self.assertIsInstance(anonymous, github.Live) + (home,), = made + self.assertIsInstance(home, fetching.ReadOnly) + self.assertEqual(home._transport._token, "a-stand-in-held-out-token") + + def test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone(self): # issue 611 + # Run at his terminal in his checkout of the home, the tool reads each case's head, its tarball and its tree, + # holds the one to the other, writes both beside the case and pins them in it; a case it cannot pin is left as + # it was. It prints counts, then each pinned case's id and new sha256, never a byte of a case, and what it + # writes is what the eval job's fetch reads. + from adapters import github + from harness import eval as job + from harness import head + from tests import support + from tests.test_adapters import home_beside, home_file + from tools import heads_into_home as tool + case, live = self.reviewed() + plain = {k: v for k, v in case.items() if k not in ("head_sha256", "tree_sha256")} + tar, tree = stand_in_head(case["head"]) + home = self.temp / "home" + (home / "pr-review").mkdir(parents=True) + written = {"poc-one": (json.dumps(dict(plain, id="poc-one"), indent=2) + "\n").encode(), + "poc-two": (json.dumps(dict(plain, id="poc-two", head="2" * 40), indent=2) + "\n").encode(), + "poc-three": b"no case\n", "poc-four": (json.dumps(dict(plain, id="poc-other"), indent=2) + "\n").encode()} + for ident, data in written.items(): + (home / "pr-review" / ("%s.json" % ident)).write_bytes(data) + (home / "pr-review" / "poc-old.tree.json").write_bytes(b"{}\n") # a tree an earlier run wrote: no case + asked = [] + + class GitHub: + """The repo the cases are of: each commit's tarball as codeload serves one, and its tree as the API answers; + but commit 2..2's tree lists a file its tarball does not hold.""" + def send(self, req): + asked.append(req["path"]) + tarred, listed = github.TARBALL.fullmatch(req["path"]), github.TREE.fullmatch(req["path"]) + if tarred: + return {"status": 200, "headers": {}, "body": support.tarball(tarred.group(3)).decode("utf-8", errors="surrogateescape")} + body = json.loads(stand_in_head(listed.group(3))[1]) + if listed.group(3) == "2" * 40: + body["tree"].append(dict(body["tree"][0], path="MISSING.md")) + return {"status": 200, "headers": {}, "body": body} + out = io.StringIO() + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 1) + new = (home / "pr-review" / "poc-one.json").read_bytes() + self.assertEqual(out.getvalue(), "pinned 1 of 4 cases; head_unreadable 1; not_a_case 2\n%s\n" + % json.dumps({"id": "poc-one", "sha256": hashlib.sha256(new).hexdigest()})) + for word in (case["diff"][:40], case["facts"]["item"]["repo"], case["head"]): + self.assertNotIn(word, out.getvalue()) + # The case pins what was written beside it, after its head, and is otherwise as it was. + self.assertEqual((home / "pr-review" / "poc-one.tar.gz").read_bytes(), tar) + self.assertEqual(json.loads((home / "pr-review" / "poc-one.tree.json").read_bytes()), json.loads(tree)) + self.assertEqual(json.loads(new), dict(plain, id="poc-one", head_sha256=hashlib.sha256(tar).hexdigest(), + tree_sha256=hashlib.sha256((home / "pr-review" / "poc-one.tree.json").read_bytes()).hexdigest())) + keys = list(json.loads(new)) + self.assertEqual(keys[keys.index("head"):keys.index("head") + 3], ["head", "head_sha256", "tree_sha256"]) + for ident in ("poc-two", "poc-three", "poc-four"): + self.assertEqual((home / "pr-review" / ("%s.json" % ident)).read_bytes(), written[ident], ident) + self.assertFalse((home / "pr-review" / ("%s.tar.gz" % ident)).exists(), ident) + repo = case["facts"]["item"]["repo"] + self.assertEqual(asked, [head.path_of(repo, case["head"]), head.tree_path(repo, case["head"]), + head.path_of(repo, "2" * 40), head.tree_path(repo, "2" * 40)]) + # Run again, it pins the case anew, to the same bytes. + (home / "pr-review" / "poc-two.json").unlink() + (home / "pr-review" / "poc-three.json").unlink() + (home / "pr-review" / "poc-four.json").unlink() + out = io.StringIO() + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 0) + self.assertEqual((home / "pr-review" / "poc-one.json").read_bytes(), new) + # What it wrote is what the eval job's fetch reads, with the home's token, beside the case. + cases = {"poc-one": new} + self.set(cases) + self.step(job.plan) + files = [home_file("poc-one", new), home_beside("poc-one", "tar.gz", (home / "pr-review" / "poc-one.tar.gz").read_bytes()), + home_beside("poc-one", "tree.json", (home / "pr-review" / "poc-one.tree.json").read_bytes())] + self.assertEqual(self.step(job.fetch, github.Replay(files)), "fetched 1 of 1 cases") + # Asked wrongly, or with no token, it reads nothing. + for argv in ([str(home)], [str(home), "pr-review/.."], [str(home), "absent"]): + self.assertEqual(tool.main(argv, {}, GitHub(), io.StringIO()), 2, argv) + self.assertEqual(tool.main([str(home), "pr-review"], {}, None, io.StringIO()), 2) def test_the_step_gives_each_case_the_release_dispatched_and_a_fresh_run(self): # issue 309, ruling 5 # Through the step as the workflow runs it: each case's Task names the Release the dispatch named, which plan diff --git a/tools/AGENTS.md b/tools/AGENTS.md index 2a77e5f3..bdeb0d7b 100644 --- a/tools/AGENTS.md +++ b/tools/AGENTS.md @@ -4,7 +4,7 @@ The checks, `verify.py` which runs them all, `mutants.py` which breaks the Kerne unless each commit names the Maintainer and a listed account opened it. `walk.py` lists this working tree's files, and nothing in another nested inside it. It skips local output, Python's caches and the corpus, only while git's index does not hold it, and reads the index as a file, so no git is needed. CI checks out the commit afresh, so there all else is committed; an export, with no `.git`, cannot tell, and skips it all. `make_corpus.py` records every decision the Kernel makes, under a directory git ignores, to hold another implementation to this one; the required check does not depend on it. `hooks/` holds the session hooks `.claude/settings.json` runs; `hooks/shell.py` reads a shell command as words for any of them. `eval_result.py` scores one Result -against one held-out case, where the case is, and is no check; `context_eval.sh` runs the context's test at the Maintainer's terminal and `context_eval.py` records a run, and neither is a check. `untested.py` reports Python no test notices (decision 0166). +against one held-out case, where the case is, and is no check; `context_eval.sh` runs the context's test at the Maintainer's terminal and `context_eval.py` records a run, and neither is a check. `untested.py` reports Python no test notices (decision 0166). `heads_into_home.py` pins each case's head beside it in the held-out home, at the Maintainer's terminal; it is no check. `rulesets_drift.sh` reads the repo's live rulesets through whoever is logged in to gh and redacts each answer outside the repo, and `rulesets_drift.py` compares what is left with `config/rulesets.json`; it is run by hand and is no check. `machine_user_drift.sh` does the same for `config/machine-user.json`, and `environments_drift.sh` holds each environment to `main` alone. @@ -33,6 +33,7 @@ and `rulesets_drift.py` compares what is left with `config/rulesets.json`; it is | A test kills a mutant only failing on it, again alone, and passing alone in an unbroken copy; else `FLAKY`, or `ERROR` on a hang: both fail a run | `tests/test_tools.py::test_a_failure_that_is_not_the_mutants_twice_is_flaky_and_fails_the_check` | | A `mutants.py` entry names text found once in its file (else `STALE`; update it) and tests that kill it, each existing, none in `WHOLE_TOO_LONG` nor its module | `mutants.py`, `tests/test_tools.py::test_every_mutant_names_a_test_that_exists` | | After the named tests, all of `tests/` runs in dealt parts; `mutants PATH...` breaks at most 40, two at once | `tests/test_tools.py::test_the_whole_suite_runs_in_dealt_parts_each_held_to_the_limit`, `tests/test_tools.py::test_a_filtered_run_is_capped` | +| `heads_into_home.py` writes a head beside its case only once it is held to its tree, pins what it wrote, and prints counts and hashes alone | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone` | A check collects every problem and does not stop at the first. Its message says what is wrong and what to do, in one sentence, because an author will read it with nothing else to go on. diff --git a/tools/heads_into_home.py b/tools/heads_into_home.py new file mode 100644 index 00000000..80cc0077 --- /dev/null +++ b/tools/heads_into_home.py @@ -0,0 +1,98 @@ +"""Each case's head, put beside the case in the held-out home and pinned by it (issue 611). Run at the Maintainer's +terminal, in his checkout of the home, never in a session: a session never sees a case. + + GH_TOKEN="$(gh auth token)" python3 -m tools.heads_into_home + +For each case `/.json`, it reads the commit the case names as its head, `head`, of the repo its facts +name, through the one transport with the token given: the commit's tarball, through codeload as a live Run reads one +(decision 0138), and its tree, every level of it (decision 0142). It holds the one to the other, as the eval job will, +and only then writes `.tar.gz` and `.tree.json` beside the case, and adds to the case the sha256 of each, +`head_sha256` and `tree_sha256`, after its `head`. It takes a role folder and touches nothing else: so the review +Intent's set and a gate set made later are pinned alike. + +It prints counts and hashes alone, never a byte of a case: how many cases it pinned and, by a code, how many it did not, +then each pinned case's id and the sha256 of its new bytes, one JSON object to a line, which is what the set's manifest +under `evals/` takes. A case it did not pin is left as it was. It exits 0 only when every case was pinned.""" +from __future__ import annotations + +import collections +import hashlib +import json +import os +import pathlib +import re +import sys +import tempfile + +PART = re.compile(r"[a-z0-9][a-z0-9-]*") # a Role or a case id, as eval-set/1 holds them: no dot, no slash +COMMIT = re.compile(r"[0-9a-f]{40}") +REPO = re.compile(r"[A-Za-z0-9_.-]{1,100}/[A-Za-z0-9_.-]{1,100}") + + +def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): + """`(the case's new bytes, None)` once its head's tarball and tree are written beside it in `folder`, or `(None, code)`: + `not_a_case` for bytes that are no case with a head of a repo, and `head_unreadable` for a head not read, unpacked or + held to its tree whole. Nothing is written for a case not pinned.""" + from adapters import github + from harness import head + try: + case = json.loads(data) + repo, sha = case["facts"]["item"]["repo"], case["head"] + except (ValueError, TypeError, KeyError, RecursionError): + return None, "not_a_case" + if not (isinstance(case, dict) and case.get("id") == ident and isinstance(sha, str) and COMMIT.fullmatch(sha) + and isinstance(repo, str) and REPO.fullmatch(repo)): + return None, "not_a_case" + try: + tar = head.fetch(transport, repo, sha) + answer = transport.send(github.request("GET", head.tree_path(repo, sha))) + if answer.get("status") != 200: + raise head.Refused("the commit's tree was not answered") + tree = head.tree_of(answer.get("body")) + with tempfile.TemporaryDirectory() as scratch: + into = os.path.join(os.path.realpath(scratch), ident) + head.unpack(tar, sha, into) + head.held(into, tree) + except (head.Refused, github.Unanswered, OSError): + return None, "head_unreadable" + listed = (json.dumps(answer["body"], indent=2) + "\n").encode("utf-8") + (folder / ("%s.tar.gz" % ident)).write_bytes(tar) + (folder / ("%s.tree.json" % ident)).write_bytes(listed) + out = {} + for key, value in case.items(): + if key not in ("head_sha256", "tree_sha256"): + out[key] = value + if key == "head": + out["head_sha256"], out["tree_sha256"] = hashlib.sha256(tar).hexdigest(), hashlib.sha256(listed).hexdigest() + return (json.dumps(out, indent=2, ensure_ascii=False) + "\n").encode("utf-8"), None + + +def main(argv: list, env=None, transport=None, out=None) -> int: + from adapters import github + env, out = os.environ if env is None else env, sys.stdout if out is None else out + if len(argv) != 2 or not PART.fullmatch(argv[1]) or not (pathlib.Path(argv[0]) / argv[1]).is_dir(): + out.write("usage: GH_TOKEN=... python3 -m tools.heads_into_home \n") + return 2 + folder = pathlib.Path(argv[0]) / argv[1] + if transport is None: + if not env.get("GH_TOKEN"): + out.write("no token: give one as GH_TOKEN\n") + return 2 + transport = github.Live(env["GH_TOKEN"]) + said, entries = collections.Counter(), [] + for path in sorted(p for p in folder.iterdir() if p.suffix == ".json" and not p.name.endswith(".tree.json")): + ident = path.name[:-len(".json")] + new, code = pinned(path.read_bytes(), transport, folder, ident) if PART.fullmatch(ident) else (None, "not_a_case") + said[code or "pinned"] += 1 + if new is not None: + path.write_bytes(new) + entries.append({"id": ident, "sha256": hashlib.sha256(new).hexdigest()}) + out.write("pinned %d of %d cases%s\n" % (said["pinned"], sum(said.values()), "".join( + "; %s %d" % (code, n) for code, n in sorted(said.items()) if code != "pinned"))) + for entry in entries: + out.write(json.dumps(entry) + "\n") + return 0 if said["pinned"] == sum(said.values()) else 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tools/mutants.py b/tools/mutants.py index f8a71038..483f1e2d 100644 --- a/tools/mutants.py +++ b/tools/mutants.py @@ -7192,20 +7192,17 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' head_.unpack(tarred, sha, os.path.join(folder, "head"))', ' pass', ('tests.test_steps.Reader.test_the_reader_writes_the_tasks_inputs',)), ("the eval fetch reads no case's head", 'harness/eval/__init__.py', - ' if data is not None and heads is not None:\n code = head_of(data, heads_place(env, workspace), entry["id"], heads)\n', - '', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate',)), + ' if data is not None:\n code = head_of(env, data, heads_place(env, workspace), found["role"], entry["id"], transport)\n', + '', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), ('a case is asked without its head', 'harness/eval/run.py', ' head = os.path.join(str(heads), case["id"]) if heads is not None else None', - ' head = None', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate',)), + ' head = None', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), ("a case's head is kept after it is asked", 'harness/eval/run.py', ' shutil.rmtree(head, ignore_errors=True)', - ' pass', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate',)), + ' pass', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), ("a head not read is the reviewer's fault", 'harness/eval/__init__.py', '"rubric_unreadable", "no_answer", "head_unreadable")', - '"rubric_unreadable", "no_answer")', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate',)), - ("the eval reads the heads with the home's token", 'harness/eval/fetch.py', - ' return fetch(env, root, workspace, transport(env), github.Live.anonymous())', - ' return fetch(env, root, workspace, transport(env), transport(env))', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_with_no_credential_and_one_not_read_closes_the_gate',)), + '"rubric_unreadable", "no_answer")', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), ("a tarball's owner is not quoted whole", 'harness/head.py', 'urllib.parse.quote(owner, safe=""), urllib.parse.quote(name, safe="")', 'owner, urllib.parse.quote(name, safe="")', ('tests.test_harness.Head.test_the_head_of_p60_is_unpacked_as_codeload_served_it',)), @@ -7987,8 +7984,9 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' if hashlib.sha256(data).hexdigest() != entry["sha256"]:\n return None, "hash_differs"\n', '', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), ('a held-out case GitHub did not find is unanswered', 'adapters/heldout.py', - ' if answer["status"] == 404:\n return None, "not_found"\n', - '', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), + ' if answer["status"] == 404:\n return None, "not_found"\n body = answer["body"]\n if answer["status"] != 200 or not isinstance(body, dict):\n', + ' body = answer["body"]\n if answer["status"] != 200 or not isinstance(body, dict):\n', + ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), ('a held-out case of any size is read', 'adapters/heldout.py', ' if size > LARGEST:\n', ' if False:\n', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), @@ -8017,8 +8015,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' and isinstance(role, str) and PART.fullmatch(role)):', ' and isinstance(role, str)):', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), ('a home of any name is asked', 'adapters/heldout.py', - ' if not home(owner, name):\n', - ' if False:\n', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), + ' return None, "not_this_set"\n if not home(owner, name):\n return None, "unanswered"\n path = ', + ' return None, "not_this_set"\n if False:\n return None, "unanswered"\n path = ', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), ('a home that climbs out is asked', 'adapters/heldout.py', ' and name not in (".", "..")', '', ('tests.test_adapters.Heldout.test_a_held_out_case_not_read_whole_says_why',)), @@ -8589,7 +8587,7 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' if os.path.commonpath([heads, other]) in (heads, other):\n', ' if heads == other or heads.startswith(other + os.sep) or other.startswith(heads + os.sep):\n', APART), ("the fetch unpacks each head in the scratch", 'harness/eval/__init__.py', - 'code = head_of(data, heads_place(env, workspace), entry["id"], heads)', 'code = head_of(data, place / "heads", entry["id"], heads)', APART), + 'code = head_of(env, data, heads_place(env, workspace), found["role"], entry["id"], transport)', 'code = head_of(env, data, place / "heads", found["role"], entry["id"], transport)', APART), ("the run step looks for each head in the scratch", 'harness/eval/run.py', 'reviewer(env, kept["release"], began, call, clock, heads_place(env, workspace))', 'reviewer(env, kept["release"], began, call, clock, place / "heads")', APART), # Issue 626: the scratch is read part by part, the plan makes the heads folder new, and the canary's head is apart. @@ -8622,15 +8620,15 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' return live.answer(the_task, env, config, watched, clock, head=head)\n', ' return live.answer(the_task, env, config, lambda a, **k: watched(a if a[1:] == ["--version"] else a + [case["expected"]["criterion"]], **k), clock, head=head)\n', NO_ANSWER), ("the head holds the case's answer", 'harness/eval/__init__.py', - ' head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident))\n', - ' head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident))\n (folder / ident / "answer.json").write_text(json.dumps(case["expected"]))\n', NO_ANSWER), + ' head.held(into, tree)\n', + ' head.held(into, tree)\n (folder / ident / "answer.json").write_text(json.dumps(case["expected"]))\n', NO_ANSWER), ("a folder of the head is named for the case's answer", 'harness/eval/__init__.py', - ' head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident))\n', - ' head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident))\n (folder / ident / case["expected"]["criterion"]).mkdir()\n', NO_ANSWER), + ' head.held(into, tree)\n', + ' head.held(into, tree)\n (folder / ident / case["expected"]["criterion"]).mkdir()\n', NO_ANSWER), # Issue 622: the head is compared byte for byte, severity varies, and the whole call is compared. ("a byte of the head is keyed on the case's verdict", 'harness/eval/__init__.py', - ' head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident))\n', - ' head.unpack(head.fetch(transport, repo, sha), sha, within(folder, ident))\n (folder / ident / "mark").write_bytes(b"\\x80" if case["expected"]["verdict"] == "finding" else b"\\x81")\n', NO_ANSWER), + ' head.held(into, tree)\n', + ' head.held(into, tree)\n (folder / ident / "mark").write_bytes(b"\\x80" if case["expected"]["verdict"] == "finding" else b"\\x81")\n', NO_ANSWER), ("the Task's texts say whether the expected Finding is blocking", 'harness/eval/run.py', ' if "issue" in case else []\n', ' if "issue" in case else [{"id": "issue:1", "by": {"kind": "human", "name": "a", "can_write": True}, "body": str((case["expected"]["finding"] or {}).get("severity") == "blocking")}]\n', NO_ANSWER), @@ -9247,6 +9245,59 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ('the fewest passes are read at the ends of the split alone', 'harness/eval/__init__.py', 'for cases in range(least, count - least + 1))', 'for cases in (least, count - least))', ('tests.test_evals.EvalJob.test_a_gate_set_gives_its_own_bars_and_least_and_its_record_states_them',)), + # Issue 611: a case's head is read from the held-out home, its tarball and its tree each by the sha256 the case pins. + ('a head beside a case of another hash is read', 'adapters/heldout.py', + ' if hashlib.sha256(data).hexdigest() != sha256:\n return None, "hash_differs"\n', '', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('what the home keeps beside a case is asked as JSON', 'adapters/heldout.py', + ' accept=RAW))', ' ))', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a head of any size is read', 'adapters/heldout.py', + ' if len(data) > BESIDE[kind]:\n', ' if False:\n', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a head answered under another type is read', 'adapters/heldout.py', + ' \\\n or not str(answer["headers"].get("content-type", "")).startswith(RAW + ";"):', ':', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ("a head answered as raw JSON is read", 'adapters/heldout.py', + '.startswith(RAW + ";")', '.startswith(RAW)', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a head GitHub did not find is unanswered', 'adapters/heldout.py', + ' if answer["status"] == 404:\n return None, "not_found"\n body = answer["body"]\n if answer["status"] != 200 or not isinstance(body, str)', + ' body = answer["body"]\n if answer["status"] != 200 or not isinstance(body, str)', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a head GitHub refused is read', 'adapters/heldout.py', + ' if answer["status"] != 200 or not isinstance(body, str) \\', ' if not isinstance(body, str) \\', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a file of any kind is asked beside a case', 'adapters/heldout.py', + ' and kind in BESIDE and isinstance(sha256, str)', ' and isinstance(sha256, str)', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a pin of any shape is asked by', 'adapters/heldout.py', + ' and isinstance(sha256, str) and SHA256.fullmatch(sha256)):', '):', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a file beside a case of any name is asked', 'adapters/heldout.py', + ' and isinstance(ident, str) and PART.fullmatch(ident)\n', '\n', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a home of any name is asked beside a case', 'adapters/heldout.py', + ' return None, "not_this_set"\n if not home(owner, name):\n return None, "unanswered"\n try:', + ' return None, "not_this_set"\n if False:\n return None, "unanswered"\n try:', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('a head is unpacked and never held to its tree', 'harness/eval/__init__.py', + ' head.held(into, tree)\n except', ' except', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ('a head not the one pinned is read as unreadable', 'harness/eval/__init__.py', + 'return "hash_differs" if code == "hash_differs" else "head_unreadable"', 'return "head_unreadable"', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ("the tree's pin is never read", 'harness/eval/__init__.py', + '(case["head_sha256"], case["tree_sha256"])', '(case["head_sha256"], case["head_sha256"])', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ('a head is read beside another Role', 'harness/eval/__init__.py', + 'heads_place(env, workspace), found["role"], entry["id"], transport)', 'heads_place(env, workspace), "pr-review-gate", entry["id"], transport)', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ('a tree that cannot be read stops the fetch', 'harness/eval/__init__.py', + ' except (head.Refused, OSError, Stop, ValueError, RecursionError):', ' except (head.Refused, OSError, Stop):', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ('a case need pin no head', 'schemas/eval-case.schema.json', ' "head_sha256",\n "tree_sha256",\n', ' "tree_sha256",\n', ('tests.test_evals.TheSet.test_a_case_dropped_under_evals_in_any_file_is_caught',)), + ('a tree pin may be cut short', 'schemas/eval-case.schema.json', + '"tree_sha256": {\n "description": "The sha256 of the commit\'s tree, every level of it, as GitHub answered it, kept beside the case in the held-out home, `/.tree.json`: the head is held to it file by file (issue 611; decision 0142).",\n "type": "string",\n "pattern": "^[0-9a-f]{64}$"', + '"tree_sha256": {\n "description": "The sha256 of the commit\'s tree, every level of it, as GitHub answered it, kept beside the case in the held-out home, `/.tree.json`: the head is held to it file by file (issue 611; decision 0142).",\n "type": "string",\n "pattern": "^[0-9a-f]{63,64}$"', ('tests.test_evals.TheSet.test_a_case_dropped_under_evals_in_any_file_is_caught',)), + ("the Maintainer's tool writes a head not held to its tree", 'tools/heads_into_home.py', + ' head.held(into, tree)\n', '', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool pins a case of another id", 'tools/heads_into_home.py', + 'case.get("id") == ident and ', '', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool reads a tree answered otherwise", 'tools/heads_into_home.py', + ' if answer.get("status") != 200:\n', ' if False:\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool says every case was pinned", 'tools/heads_into_home.py', + ' return 0 if said["pinned"] == sum(said.values()) else 1\n', ' return 0\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool reads a tree it wrote as a case", 'tools/heads_into_home.py', + ' and not p.name.endswith(".tree.json")', '', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool pins a tree it did not write", 'tools/heads_into_home.py', + 'hashlib.sha256(listed).hexdigest()', 'hashlib.sha256(json.dumps(answer["body"]).encode()).hexdigest()', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool reads with no token", 'tools/heads_into_home.py', + ' if not env.get("GH_TOKEN"):\n', ' if False:\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), # Issue 613, the preflight: each dispatch of a gate set asks a few development cases first, and a broken pipeline stops it. ('a gate set need name no preflight', 'harness/eval/__init__.py', 'OWN = ("bars", "kind_least", "record", "preflight")', @@ -14920,8 +14971,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: 'body.get("truncated") is not False', 'body.get("truncated") is True', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), ('a tree answered otherwise than 200 is read', 'harness/head.py', - 'if answer.get("status") != 200 or not isinstance(body, dict) or body.get("truncated")', - 'if not isinstance(body, dict) or body.get("truncated")', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), + ' if answer.get("status") != 200:\n raise Refused("the commit\'s tree was not answered whole")\n', + '', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), ('a path the tree names twice is read', 'harness/head.py', 'if not isinstance(path, str) or path in seen:', 'if not isinstance(path, str):', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), @@ -14974,8 +15025,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' if os.path.islink(full) or not os.path.isfile(full):\n raise Refused("the head holds what is not a regular file")', ' if os.path.islink(full):\n raise Refused("the head holds what is not a regular file")', ('tests.test_harness.Head.test_the_head_of_p60_is_held_to_its_commits_tree',)), ('a tree read short is read', 'harness/head.py', - 'or not isinstance(body, dict) or body.get("truncated") is not False', - 'or not isinstance(body, dict)', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), + 'if not isinstance(body, dict) or body.get("truncated") is not False or', + 'if not isinstance(body, dict) or', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), ('a blob by a sha that only starts as one is read', 'harness/head.py', 'isinstance(blob, str) and BLOB.fullmatch(blob)', 'isinstance(blob, str) and BLOB.match(blob)', ('tests.test_harness.Head.test_a_tree_not_answered_whole_or_holding_what_an_archive_cannot_is_refused',)), From ca15b2608993ee37648edbf9f62ff195f24eddc7 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:20:07 -0600 Subject: [PATCH 05/10] The eval blob is asked for as vnd.github.raw alone, by the ruling on P-62 Co-Authored-By: Claude Opus 5.5 --- ...e-held-out-home-pinned-by-the-case-and-held-to-its-tree.md | 4 ++-- tools/mutants.py | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md index 7539cc5c..459e6dde 100644 --- a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md +++ b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md @@ -7,7 +7,7 @@ Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A c | What | Decided | Because | |---|---|---| | Where a head is kept | Beside each case in the held-out home, `/.json`, the home keeps the case's head as two files: `.tar.gz`, the commit's tarball as GitHub archived it, and `.tree.json`, the commit's tree, every level of it, as the API answered it. `eval-case/1` requires `head_sha256` and `tree_sha256`, the sha256 of each, beside `head` | The issue's: a head read anonymously shares the anonymous cap, 60 an hour, so a pool of three runs, or a run again within the hour, can fault on `head_unreadable`; and a head that is not on GitHub cannot be read at all. Pinned by the case, which the manifest pins, the head is the bytes the Maintainer chose | -| How it is read | `fetch` reads both files with the home's token, through the transport the case is read with, by `adapters/heldout.beside`: each with the raw media type, `application/vnd.github.raw`, whose answer is a file's bytes whole however large, in one answer and no redirect (`trials/P-62.json`). An answer of another type, `+json` among them, which the transport would read as JSON, is no file. A file over its bound (the transport's tarball bound, `TARRED`, or 8 MiB for a tree) is `too_large`, and one that does not hash to its pin `hash_differs`. Nothing in the eval job reads with no credential, and the home's token is still sent nowhere else | The Factory Builder's ruling (c): no transition, and no case without its pins, since an anonymous read kept for one would keep the read the issue removes. Ruling (d): the raw media type, grounded by its trial, the git blobs API its fallback | +| How it is read | `fetch` reads both files with the home's token, through the transport the case is read with, by `adapters/heldout.beside`: each with the raw media type, `application/vnd.github.raw`, whose answer is a file's bytes whole however large, in one answer and no redirect (`trials/P-62.json`). An answer of another type, `+json` among them, which the transport would read as JSON, is no file. A file over its bound (the transport's tarball bound, `TARRED`, or 8 MiB for a tree) is `too_large`, and one that does not hash to its pin `hash_differs`. Nothing in the eval job reads with no credential, and the home's token is still sent nowhere else | The Factory Builder's ruling (c): no transition, and no case without its pins, since an anonymous read kept for one would keep the read the issue removes. Ruling (d): the raw media type, grounded by its trial, the git blobs API its fallback; and the ruling on P-62's verdict, that the raw media type is `application/vnd.github.raw`, which the transport must ask for | | How it is held | The tarball is unpacked as before (decision 0138), and then held to the tree file by file, by git's blob hash, by the check a live head is held by (decision 0142): `harness/head.tree_of` reads the tree file with the checks `tree` reads GitHub's answer with, and `held` holds the head to it. A head not the one pinned is `hash_differs`, a fault of the run; one not read, unpacked or held whole is `head_unreadable`, the Engine's or the Factory's. Either closes the gate | The Factory Builder's ruling (b): pin the tree as well as the tarball, and hold every eval head to it. 0142 left an eval head unheld for want of room under the anonymous cap; read with the home's token, there is room. An eval case is now graded on the files a live review would be given | | Who puts them there | The Maintainer, at his terminal, in his checkout of the home: `GH_TOKEN="$(gh auth token)" python3 -m tools.heads_into_home `. For each case of the folder, it reads the head's tarball and tree from GitHub through the one transport, holds the one to the other, writes both beside the case and adds their sha256 to it; a case it cannot pin it leaves as it was. It prints counts, then each pinned case's id and the sha256 of its new bytes, never a byte of a case. It takes a role folder and touches nothing else, so a gate set made later is pinned as the review Intent's set is | The Factory Builder's ruling (c): the tool serves the coming gate set too, and prints counts and hashes alone. A session never sees a case | | The set changes with it | Pinning changes each case's bytes, so the set's manifest under `evals/` takes the hashes the tool prints, in the pull request that brings this change. It merges only once the home holds the files, and once `trials/P-62.json` is on main | The Factory Builder's ruling (c) and (d) | @@ -23,7 +23,7 @@ A case's head was read with no credential, from the public repo its change was m | Boundary | Check | |---|---| -| What the home keeps beside a case is asked with the raw media type and the home's token, and handed back only as the bytes the case pins: an answer of another type, `+json` among them, one not found, refused, over its bound or of another hash is none, and a name, kind, pin or home that could name another file is not asked | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins`, `trials/P-62.json`, `adapters/heldout.py`, mutant: "a head beside a case of another hash is read", mutant: "what the home keeps beside a case is asked as JSON", mutant: "a head of any size is read", mutant: "a head answered under another type is read", mutant: "a head answered as raw JSON is read", mutant: "a head GitHub did not find is unanswered", mutant: "a head GitHub refused is read", mutant: "a file of any kind is asked beside a case", mutant: "a pin of any shape is asked by", mutant: "a file beside a case of any name is asked", mutant: "a home of any name is asked beside a case" | +| What the home keeps beside a case is asked with the raw media type and the home's token, and handed back only as the bytes the case pins: an answer of another type, `+json` among them, one not found, refused, over its bound or of another hash is none, and a name, kind, pin or home that could name another file is not asked | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins`, `trials/P-62.json`, `adapters/heldout.py`, mutant: "the eval blob is asked for as raw+json", mutant: "a head beside a case of another hash is read", mutant: "what the home keeps beside a case is asked as JSON", mutant: "a head of any size is read", mutant: "a head answered under another type is read", mutant: "a head answered as raw JSON is read", mutant: "a head GitHub did not find is unanswered", mutant: "a head GitHub refused is read", mutant: "a file of any kind is asked beside a case", mutant: "a pin of any shape is asked by", mutant: "a file beside a case of any name is asked", mutant: "a home of any name is asked beside a case" | | The fetch reads each case's head from the home by its pins, with no other transport, unpacks it and holds it to its tree; a head not the one pinned is `hash_differs`, one not read, unpacked or held whole `head_unreadable`, and the gate closes | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, `harness/eval/fetch.py`, `harness/head.py`, mutant: "the eval fetch reads no case's head", mutant: "a head is unpacked and never held to its tree", mutant: "a head not the one pinned is read as unreadable", mutant: "the tree's pin is never read", mutant: "a head is read beside another Role", mutant: "a tree that cannot be read stops the fetch" | | An eval case requires both pins, each a whole sha256 | `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a tree pin may be cut short" | | The Maintainer's tool writes a head beside its case only once it is held to its tree, pins what it wrote, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token" | diff --git a/tools/mutants.py b/tools/mutants.py index 483f1e2d..c3c8c73f 100644 --- a/tools/mutants.py +++ b/tools/mutants.py @@ -9248,6 +9248,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: # Issue 611: a case's head is read from the held-out home, its tarball and its tree each by the sha256 the case pins. ('a head beside a case of another hash is read', 'adapters/heldout.py', ' if hashlib.sha256(data).hexdigest() != sha256:\n return None, "hash_differs"\n', '', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), + ('the eval blob is asked for as raw+json', 'adapters/heldout.py', + 'RAW = "application/vnd.github.raw"', 'RAW = "application/vnd.github.raw+json"', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), ('what the home keeps beside a case is asked as JSON', 'adapters/heldout.py', ' accept=RAW))', ' ))', ('tests.test_adapters.Heldout.test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins',)), ('a head of any size is read', 'adapters/heldout.py', From 1447e232f1ca3aa797be10beae1026d383773c22 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:32:04 -0600 Subject: [PATCH 06/10] What the reads of the heads found: the tool writes whole and through no link, a pin that is no sha256 reads nothing, a tree answered otherwise is tested Co-Authored-By: Claude Opus 5.5 --- ...pinned-by-the-case-and-held-to-its-tree.md | 8 +-- harness/eval/__init__.py | 3 +- harness/eval/canary.py | 3 +- tests/test_adapters.py | 15 +++-- tests/test_evals.py | 56 ++++++++++++---- tools/heads_into_home.py | 66 +++++++++++++++---- tools/mutants.py | 17 +++++ 7 files changed, 130 insertions(+), 38 deletions(-) diff --git a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md index 459e6dde..a08adae0 100644 --- a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md +++ b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md @@ -1,6 +1,6 @@ # NNNN. A case's head is read from the held-out home, pinned by the case, and held to its tree -Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded", points (a) to (d): this is the first of its two pull requests, the heads; the second shards the run. It amends decision 0138's rows "A transport with no token" and "The eval case" where they say the eval job reads a case's head with no credential, and decision 0142's row "An eval case's head", which this closes. Nothing else in either changes. +Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded", points (a) to (d): this is the first of its two pull requests, the heads; the second shards the run. It amends decision 0138's rows "A transport with no token" and "The eval case" where they say the eval job reads a case's head with no credential, and its Boundaries row "Each case's head is read with no credential and removed once asked", whose check now holds the read from the home that replaces it; and decision 0142's row "An eval case's head", which this closes. Nothing else in either changes. ## Decision @@ -9,7 +9,7 @@ Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A c | Where a head is kept | Beside each case in the held-out home, `/.json`, the home keeps the case's head as two files: `.tar.gz`, the commit's tarball as GitHub archived it, and `.tree.json`, the commit's tree, every level of it, as the API answered it. `eval-case/1` requires `head_sha256` and `tree_sha256`, the sha256 of each, beside `head` | The issue's: a head read anonymously shares the anonymous cap, 60 an hour, so a pool of three runs, or a run again within the hour, can fault on `head_unreadable`; and a head that is not on GitHub cannot be read at all. Pinned by the case, which the manifest pins, the head is the bytes the Maintainer chose | | How it is read | `fetch` reads both files with the home's token, through the transport the case is read with, by `adapters/heldout.beside`: each with the raw media type, `application/vnd.github.raw`, whose answer is a file's bytes whole however large, in one answer and no redirect (`trials/P-62.json`). An answer of another type, `+json` among them, which the transport would read as JSON, is no file. A file over its bound (the transport's tarball bound, `TARRED`, or 8 MiB for a tree) is `too_large`, and one that does not hash to its pin `hash_differs`. Nothing in the eval job reads with no credential, and the home's token is still sent nowhere else | The Factory Builder's ruling (c): no transition, and no case without its pins, since an anonymous read kept for one would keep the read the issue removes. Ruling (d): the raw media type, grounded by its trial, the git blobs API its fallback; and the ruling on P-62's verdict, that the raw media type is `application/vnd.github.raw`, which the transport must ask for | | How it is held | The tarball is unpacked as before (decision 0138), and then held to the tree file by file, by git's blob hash, by the check a live head is held by (decision 0142): `harness/head.tree_of` reads the tree file with the checks `tree` reads GitHub's answer with, and `held` holds the head to it. A head not the one pinned is `hash_differs`, a fault of the run; one not read, unpacked or held whole is `head_unreadable`, the Engine's or the Factory's. Either closes the gate | The Factory Builder's ruling (b): pin the tree as well as the tarball, and hold every eval head to it. 0142 left an eval head unheld for want of room under the anonymous cap; read with the home's token, there is room. An eval case is now graded on the files a live review would be given | -| Who puts them there | The Maintainer, at his terminal, in his checkout of the home: `GH_TOKEN="$(gh auth token)" python3 -m tools.heads_into_home `. For each case of the folder, it reads the head's tarball and tree from GitHub through the one transport, holds the one to the other, writes both beside the case and adds their sha256 to it; a case it cannot pin it leaves as it was. It prints counts, then each pinned case's id and the sha256 of its new bytes, never a byte of a case. It takes a role folder and touches nothing else, so a gate set made later is pinned as the review Intent's set is | The Factory Builder's ruling (c): the tool serves the coming gate set too, and prints counts and hashes alone. A session never sees a case | +| Who puts them there | The Maintainer, at his terminal, in his checkout of the home: `GH_TOKEN="$(gh auth token)" python3 -m tools.heads_into_home `. For each case of the folder, it reads the head's tarball and tree from GitHub through the one transport, read-only, holds the one to the other, writes both beside the case, the tree compact as GitHub's answer held it, and writes the case anew with their sha256 after its `head`; a case it cannot pin, or whose head is over what the job reads, it leaves as it was. Each file is written whole and moved over its name, so a link there is replaced, never written through, and a case or a folder that is a link is not read. It prints counts, then each pinned case's id and the sha256 of its new bytes, never a byte of a case. It takes a role folder and touches nothing else, so a gate set made later is pinned as the review Intent's set is | The Factory Builder's ruling (c): the tool serves the coming gate set too, and prints counts and hashes alone. A session never sees a case | | The set changes with it | Pinning changes each case's bytes, so the set's manifest under `evals/` takes the hashes the tool prints, in the pull request that brings this change. It merges only once the home holds the files, and once `trials/P-62.json` is on main | The Factory Builder's ruling (c) and (d) | | The canary | `harness/eval/canary.py` still reads its case's head with no credential: its case is this repo's own example, of this public repo, never one of the home's, and pins no file the home keeps | The canary is no step of the eval job (decision 0106) | | Residual: a head only in the home | The tool reads a head from GitHub. A head that is not there, one built for a case, is written beside its case by hand, its tarball as `git archive` writes one with the commit in its pax header and one folder, and its tree in the API's shape; nothing builds that yet | No case of the set needs one yet. The eval reads it as it reads any other | @@ -24,6 +24,6 @@ A case's head was read with no credential, from the public repo its change was m | Boundary | Check | |---|---| | What the home keeps beside a case is asked with the raw media type and the home's token, and handed back only as the bytes the case pins: an answer of another type, `+json` among them, one not found, refused, over its bound or of another hash is none, and a name, kind, pin or home that could name another file is not asked | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins`, `trials/P-62.json`, `adapters/heldout.py`, mutant: "the eval blob is asked for as raw+json", mutant: "a head beside a case of another hash is read", mutant: "what the home keeps beside a case is asked as JSON", mutant: "a head of any size is read", mutant: "a head answered under another type is read", mutant: "a head answered as raw JSON is read", mutant: "a head GitHub did not find is unanswered", mutant: "a head GitHub refused is read", mutant: "a file of any kind is asked beside a case", mutant: "a pin of any shape is asked by", mutant: "a file beside a case of any name is asked", mutant: "a home of any name is asked beside a case" | -| The fetch reads each case's head from the home by its pins, with no other transport, unpacks it and holds it to its tree; a head not the one pinned is `hash_differs`, one not read, unpacked or held whole `head_unreadable`, and the gate closes | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, `harness/eval/fetch.py`, `harness/head.py`, mutant: "the eval fetch reads no case's head", mutant: "a head is unpacked and never held to its tree", mutant: "a head not the one pinned is read as unreadable", mutant: "the tree's pin is never read", mutant: "a head is read beside another Role", mutant: "a tree that cannot be read stops the fetch" | +| The fetch reads each case's head from the home by its pins, with no other transport, unpacks it and holds it to its tree; a head not the one pinned is `hash_differs`, one not read, unpacked or held whole `head_unreadable` and removed, a pin that is no sha256 reads nothing and the case is `not_this_set`, and the gate closes | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, `harness/eval/fetch.py`, `harness/head.py`, mutant: "the eval fetch reads no case's head", mutant: "a head is unpacked and never held to its tree", mutant: "a head not the one pinned is read as unreadable", mutant: "the tree's pin is never read", mutant: "a head is read beside another Role", mutant: "a tree that cannot be read stops the fetch", mutant: "a head not held is left in the heads folder", mutant: "a case whose pin is no sha256 has its head read" | | An eval case requires both pins, each a whole sha256 | `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a tree pin may be cut short" | -| The Maintainer's tool writes a head beside its case only once it is held to its tree, pins what it wrote, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token" | +| The Maintainer's tool writes a head beside its case only once it is held to its tree and within what the job reads, pins what it wrote, writes each file whole and through no link, reads no case or folder that is a link, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token", mutant: "the Maintainer's tool writes through a link", mutant: "the Maintainer's tool reads a case that is a link", mutant: "the Maintainer's tool pins a head the job will refuse", mutant: "the Maintainer's tool reads a folder that is a link", mutant: "a write that fails stops the Maintainer's tool", mutant: "the Maintainer's tool leaves a file half written" | diff --git a/harness/eval/__init__.py b/harness/eval/__init__.py index 140ecc48..f2bc9c12 100644 --- a/harness/eval/__init__.py +++ b/harness/eval/__init__.py @@ -372,7 +372,8 @@ def head_of(env: dict, data: bytes, folder: pathlib.Path, role: str, ident: str, sha, pins = case["head"], (case["head_sha256"], case["tree_sha256"]) except (ValueError, TypeError, KeyError, RecursionError): return None - if not (isinstance(sha, str) and COMMIT.fullmatch(sha)): + # A case whose head or pins are not whole is no case of this set, which its schema says after: nothing is read. + if not (isinstance(sha, str) and COMMIT.fullmatch(sha) and all(isinstance(p, str) and SHA256.fullmatch(p) for p in pins)): return None read = {} for kind, pin in zip(("tar.gz", "tree.json"), pins): diff --git a/harness/eval/canary.py b/harness/eval/canary.py index 187dca81..378acecd 100644 --- a/harness/eval/canary.py +++ b/harness/eval/canary.py @@ -69,7 +69,8 @@ def head_of(data: bytes, folder: pathlib.Path, ident: str, transport): def review(env: dict, call=live.cli_call, clock=time.time, heads=None) -> int: """One review by the eval job's path: 0 if the CLI answered in a status the Engine may write, its answer read and scored, and 1 otherwise, a `failed` Result among them. A scorer that raises fails the step. The example's head is - read through `heads`, by default with no credential, as the eval job's fetch reads a case's (decision 0138).""" + read through `heads`, by default with no credential, from this public repo (decision 0138); the eval job reads a case's + from the held-out home instead (issue 611).""" case = json.loads((job.ROOT / "examples" / "eval-case.json").read_text()) content = "sha256:" + hashlib.sha256(case["diff"].encode("utf-8")).hexdigest() case["facts"]["content_id"] = content diff --git a/tests/test_adapters.py b/tests/test_adapters.py index 974bef69..7a2012ce 100644 --- a/tests/test_adapters.py +++ b/tests/test_adapters.py @@ -283,8 +283,8 @@ def open(self, req, timeout=None): self.assertNotIn("secret", held(refused.exception)) def test_a_transport_with_no_token_sends_a_commits_tarball_alone(self): - # Decision 0138: the eval job and the canary read the public cases' repo's heads with no credential at all, and - # a transport made for that sends nothing else, so no other read is ever made with no token by mistake. + # Decision 0138: the canary reads its example's head from this public repo with no credential at all (the eval + # job reads a case's head from the home, issue 611), and a transport made for that sends nothing else, so no other read is ever made with no token by mistake. anonymous = github.Live.anonymous() sent = [] @@ -3586,15 +3586,18 @@ def read(found, kind="tar.gz", pin=pins["tar.gz"], owner="o", name="home", role= del untyped["response"]["headers"]["content-type"] failed = home_beside("c-1", "tar.gz", tar) failed["response"]["status"] = 500 - large = home_beside("c-1", "tar.gz", b"\0" * (heldout.BESIDE["tar.gz"] + 1)) + large = home_beside("c-1", "tar.gz", tar) + bounded = mock.patch.dict(heldout.BESIDE, {"tar.gz": len(tar) - 1, "tree.json": len(tree) - 1}) for found, code in (([home_beside("c-1", "tar.gz", tar[1:])], "hash_differs"), ([home_beside("c-1", "tar.gz", tar, status=404)], "not_found"), ([failed], "unanswered"), ([], "unanswered"), ([json_typed], "unanswered"), ([raw_json], "unanswered"), ([untyped], "unanswered"), - ([large], "too_large"), ([home_beside("c-1", "tree.json", tree)], "unanswered")): + ([home_beside("c-1", "tree.json", tree)], "unanswered")): self.assertEqual(read(found), (None, code), code) - self.assertEqual(read([home_beside("c-1", "tree.json", b"\0" * (heldout.BESIDE["tree.json"] + 1))], - kind="tree.json", pin=pins["tree.json"]), (None, "too_large")) + with bounded: + self.assertEqual(read([large]), (None, "too_large")) + self.assertEqual(read([home_beside("c-1", "tree.json", tree)], kind="tree.json", pin=pins["tree.json"]), + (None, "too_large")) self.assertEqual(read([home_beside("c-1", "tree.json", tree)], kind="tree.json", pin=pins["tar.gz"]), (None, "hash_differs")) # A name, a kind or a pin that could name another file, or a home that could name another repo, is refused diff --git a/tests/test_evals.py b/tests/test_evals.py index a26525fd..22cb2265 100644 --- a/tests/test_evals.py +++ b/tests/test_evals.py @@ -2656,6 +2656,9 @@ def test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_clos exchanges.append(home_beside(ident, "tar.gz", b"", status=404) if has is None else home_beside(ident, "tar.gz", has)) if has is not None and has == tar: exchanges.append(home_beside(ident, "tree.json", holds)) + # A pin that is no sha256 is no case of the set, as a pin left out is: nothing is read beside it. + cases["poc-seven"] = (json.dumps(dict(case, id="poc-seven", head_sha256="x"), indent=2) + "\n").encode() + exchanges.append(home_file("poc-seven", cases["poc-seven"])) self.set(cases) self.step(job.plan) asked = [] @@ -2664,14 +2667,15 @@ class Home(github.Replay): def send(self, req): asked.append((req["path"].rsplit("/", 1)[-1], req["accept"])) return super().send(req) - self.assertEqual(self.step(job.fetch, Home(exchanges)), "fetched 1 of 6 cases") + self.assertEqual(self.step(job.fetch, Home(exchanges)), "fetched 2 of 7 cases") self.assertEqual(asked[:3], [("poc-one.json", github.JSON), ("poc-one.tar.gz", heldout.RAW), ("poc-one.tree.json", heldout.RAW)]) self.assertEqual(len(asked), len(exchanges), "each answer the home keeps is asked, once, in its order") place = self.temp / "heldout" self.assertEqual(json.loads((place / "fetched.json").read_text()), {"poc-one": None, "poc-two": "head_unreadable", "poc-three": "hash_differs", - "poc-four": "head_unreadable", "poc-five": "head_unreadable", "poc-six": "head_unreadable"}) + "poc-four": "head_unreadable", "poc-five": "head_unreadable", "poc-six": "head_unreadable", + "poc-seven": None}) self.assertEqual(sorted(p.name for p in (self.temp / "heads").iterdir()), ["poc-one"]) self.assertEqual((self.temp / "heads" / "poc-one" / "README.md").read_bytes(), b"# a repo\n") cli = live.CLI(said=live.answer(live.judged(findings=[], effects={}))) @@ -2684,6 +2688,7 @@ def send(self, req): report = json.loads((self.work / job.REPORT).read_text()) self.assertEqual({c["id"]: c["reasons"] for c in report["cases"]}["poc-three"], ["hash_differs"]) self.assertEqual({c["id"]: c["reasons"] for c in report["cases"]}["poc-two"], ["head_unreadable"]) + self.assertEqual({c["id"]: c["reasons"] for c in report["cases"]}["poc-seven"], ["not_this_set"]) self.assertIn("head_unreadable", job.ENGINE_FAULTS) self.assertEqual(self.step(job.judge.step if hasattr(job.judge, "step") else job.judge), "engine_fault") # The fetch step's one transport is the home's, read-only, with its token: no transport with no credential. @@ -2714,15 +2719,22 @@ def test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_a (home / "pr-review").mkdir(parents=True) written = {"poc-one": (json.dumps(dict(plain, id="poc-one"), indent=2) + "\n").encode(), "poc-two": (json.dumps(dict(plain, id="poc-two", head="2" * 40), indent=2) + "\n").encode(), - "poc-three": b"no case\n", "poc-four": (json.dumps(dict(plain, id="poc-other"), indent=2) + "\n").encode()} + "poc-three": b"no case\n", "poc-four": (json.dumps(dict(plain, id="poc-other"), indent=2) + "\n").encode(), + "poc-five": (json.dumps(dict(plain, id="poc-five", head="5" * 40), indent=2) + "\n").encode()} for ident, data in written.items(): (home / "pr-review" / ("%s.json" % ident)).write_bytes(data) (home / "pr-review" / "poc-old.tree.json").write_bytes(b"{}\n") # a tree an earlier run wrote: no case + # A case that is a link is read as none, and a link where a head is written is replaced, never written through. + outside = self.temp / "outside" + outside.write_bytes(b"kept\n") + (self.temp / "linked.json").write_bytes((json.dumps(dict(plain, id="poc-six"), indent=2) + "\n").encode()) + os.symlink(str(self.temp / "linked.json"), str(home / "pr-review" / "poc-six.json")) + os.symlink(str(outside), str(home / "pr-review" / "poc-one.tar.gz")) asked = [] class GitHub: """The repo the cases are of: each commit's tarball as codeload serves one, and its tree as the API answers; - but commit 2..2's tree lists a file its tarball does not hold.""" + but commit 2..2's tree lists a file its tarball does not hold, and 5..5's is answered 404.""" def send(self, req): asked.append(req["path"]) tarred, listed = github.TARBALL.fullmatch(req["path"]), github.TREE.fullmatch(req["path"]) @@ -2731,31 +2743,34 @@ def send(self, req): body = json.loads(stand_in_head(listed.group(3))[1]) if listed.group(3) == "2" * 40: body["tree"].append(dict(body["tree"][0], path="MISSING.md")) - return {"status": 200, "headers": {}, "body": body} + return {"status": 404 if listed.group(3) == "5" * 40 else 200, "headers": {}, "body": body} out = io.StringIO() self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 1) new = (home / "pr-review" / "poc-one.json").read_bytes() - self.assertEqual(out.getvalue(), "pinned 1 of 4 cases; head_unreadable 1; not_a_case 2\n%s\n" + self.assertEqual(out.getvalue(), "pinned 1 of 6 cases; head_unreadable 2; not_a_case 3\n%s\n" % json.dumps({"id": "poc-one", "sha256": hashlib.sha256(new).hexdigest()})) for word in (case["diff"][:40], case["facts"]["item"]["repo"], case["head"]): self.assertNotIn(word, out.getvalue()) # The case pins what was written beside it, after its head, and is otherwise as it was. self.assertEqual((home / "pr-review" / "poc-one.tar.gz").read_bytes(), tar) + self.assertFalse((home / "pr-review" / "poc-one.tar.gz").is_symlink()) + self.assertEqual(outside.read_bytes(), b"kept\n") + self.assertEqual([p.name for p in (home / "pr-review").iterdir() if p.name.startswith(".")], [], "nothing half written") self.assertEqual(json.loads((home / "pr-review" / "poc-one.tree.json").read_bytes()), json.loads(tree)) self.assertEqual(json.loads(new), dict(plain, id="poc-one", head_sha256=hashlib.sha256(tar).hexdigest(), tree_sha256=hashlib.sha256((home / "pr-review" / "poc-one.tree.json").read_bytes()).hexdigest())) keys = list(json.loads(new)) self.assertEqual(keys[keys.index("head"):keys.index("head") + 3], ["head", "head_sha256", "tree_sha256"]) - for ident in ("poc-two", "poc-three", "poc-four"): + for ident in ("poc-two", "poc-three", "poc-four", "poc-five"): self.assertEqual((home / "pr-review" / ("%s.json" % ident)).read_bytes(), written[ident], ident) self.assertFalse((home / "pr-review" / ("%s.tar.gz" % ident)).exists(), ident) repo = case["facts"]["item"]["repo"] - self.assertEqual(asked, [head.path_of(repo, case["head"]), head.tree_path(repo, case["head"]), + self.assertEqual(asked, [head.path_of(repo, "5" * 40), head.tree_path(repo, "5" * 40), + head.path_of(repo, case["head"]), head.tree_path(repo, case["head"]), head.path_of(repo, "2" * 40), head.tree_path(repo, "2" * 40)]) # Run again, it pins the case anew, to the same bytes. - (home / "pr-review" / "poc-two.json").unlink() - (home / "pr-review" / "poc-three.json").unlink() - (home / "pr-review" / "poc-four.json").unlink() + for ident in ("poc-two", "poc-three", "poc-four", "poc-five", "poc-six"): + (home / "pr-review" / ("%s.json" % ident)).unlink() out = io.StringIO() self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 0) self.assertEqual((home / "pr-review" / "poc-one.json").read_bytes(), new) @@ -2766,8 +2781,23 @@ def send(self, req): files = [home_file("poc-one", new), home_beside("poc-one", "tar.gz", (home / "pr-review" / "poc-one.tar.gz").read_bytes()), home_beside("poc-one", "tree.json", (home / "pr-review" / "poc-one.tree.json").read_bytes())] self.assertEqual(self.step(job.fetch, github.Replay(files)), "fetched 1 of 1 cases") - # Asked wrongly, or with no token, it reads nothing. - for argv in ([str(home)], [str(home), "pr-review/.."], [str(home), "absent"]): + # A head over what the job reads is not pinned; nor is a case whose write fails, by its code alone. + from adapters import heldout + with mock.patch.dict(heldout.BESIDE, {"tree.json": 10}): + out = io.StringIO() + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 1) + self.assertEqual(out.getvalue(), "pinned 0 of 1 cases; head_unreadable 1\n") + with mock.patch.object(tool, "put", mock.Mock(side_effect=OSError("a path"))): + out = io.StringIO() + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 1) + self.assertEqual(out.getvalue(), "pinned 0 of 1 cases; not_written 1\n") + with mock.patch.object(tool.os, "replace", mock.Mock(side_effect=OSError("a path"))): + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), io.StringIO()), 1) + self.assertEqual([p.name for p in (home / "pr-review").iterdir() if p.name.startswith(".")], [], "nothing half written") + self.assertEqual((home / "pr-review" / "poc-one.json").read_bytes(), new) + # Asked wrongly, of a folder that is a link, or with no token, it reads nothing. + os.symlink(str(home / "pr-review"), str(home / "linked")) + for argv in ([str(home)], [str(home), "pr-review/.."], [str(home), "absent"], [str(home), "linked"]): self.assertEqual(tool.main(argv, {}, GitHub(), io.StringIO()), 2, argv) self.assertEqual(tool.main([str(home), "pr-review"], {}, None, io.StringIO()), 2) diff --git a/tools/heads_into_home.py b/tools/heads_into_home.py index 80cc0077..ed168943 100644 --- a/tools/heads_into_home.py +++ b/tools/heads_into_home.py @@ -7,8 +7,9 @@ name, through the one transport with the token given: the commit's tarball, through codeload as a live Run reads one (decision 0138), and its tree, every level of it (decision 0142). It holds the one to the other, as the eval job will, and only then writes `.tar.gz` and `.tree.json` beside the case, and adds to the case the sha256 of each, -`head_sha256` and `tree_sha256`, after its `head`. It takes a role folder and touches nothing else: so the review -Intent's set and a gate set made later are pinned alike. +`head_sha256` and `tree_sha256`, after its `head`, writing the case anew. Each file is written whole into the folder and +moved over its name, so a link there is replaced and never written through; a folder that is a link is refused. It +takes a role folder and touches nothing else: so the review Intent's set and a gate set made later are pinned alike. It prints counts and hashes alone, never a byte of a case: how many cases it pinned and, by a code, how many it did not, then each pinned case's id and the sha256 of its new bytes, one JSON object to a line, which is what the set's manifest @@ -16,11 +17,13 @@ from __future__ import annotations import collections +import contextlib import hashlib import json import os import pathlib import re +import stat import sys import tempfile @@ -29,11 +32,35 @@ REPO = re.compile(r"[A-Za-z0-9_.-]{1,100}/[A-Za-z0-9_.-]{1,100}") +def put(folder: pathlib.Path, name: str, data: bytes) -> None: + """`data` written as `folder/name`, whole or not at all: into a new file beside it, then moved over it, so that a + link already at that name is replaced, never written through, and nothing is left half written.""" + fd, temporary = tempfile.mkstemp(dir=str(folder), prefix=".heads-into-home-") + try: + with os.fdopen(fd, "wb") as out: + out.write(data) + os.replace(temporary, str(folder / name)) + except BaseException: + with contextlib.suppress(OSError): + os.unlink(temporary) + raise + + +def read(path: pathlib.Path): + """The bytes of the case at `path`, or None for one that is a link or not a regular file.""" + try: + fd = os.open(str(path), os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)) + except OSError: + return None + with os.fdopen(fd, "rb") as f: + return f.read() if stat.S_ISREG(os.fstat(f.fileno()).st_mode) else None + + def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): """`(the case's new bytes, None)` once its head's tarball and tree are written beside it in `folder`, or `(None, code)`: - `not_a_case` for bytes that are no case with a head of a repo, and `head_unreadable` for a head not read, unpacked or - held to its tree whole. Nothing is written for a case not pinned.""" - from adapters import github + `not_a_case` for bytes that are no case with a head of a repo, and `head_unreadable` for a head not read, unpacked, + held to its tree, or within what the eval job reads, whole. Nothing is written for a case not pinned.""" + from adapters import github, heldout from harness import head try: case = json.loads(data) @@ -43,6 +70,8 @@ def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): if not (isinstance(case, dict) and case.get("id") == ident and isinstance(sha, str) and COMMIT.fullmatch(sha) and isinstance(repo, str) and REPO.fullmatch(repo)): return None, "not_a_case" + if not github.TREE.fullmatch(head.tree_path(repo, sha)): + return None, "not_a_case" try: tar = head.fetch(transport, repo, sha) answer = transport.send(github.request("GET", head.tree_path(repo, sha))) @@ -55,22 +84,27 @@ def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): head.held(into, tree) except (head.Refused, github.Unanswered, OSError): return None, "head_unreadable" - listed = (json.dumps(answer["body"], indent=2) + "\n").encode("utf-8") - (folder / ("%s.tar.gz" % ident)).write_bytes(tar) - (folder / ("%s.tree.json" % ident)).write_bytes(listed) + # The tree as GitHub's answer held it, written compact, so that a tree GitHub gave whole is within what the job reads. + listed = (json.dumps(answer["body"], separators=(",", ":")) + "\n").encode("utf-8") + if len(tar) > heldout.BESIDE["tar.gz"] or len(listed) > heldout.BESIDE["tree.json"]: + return None, "head_unreadable" out = {} for key, value in case.items(): if key not in ("head_sha256", "tree_sha256"): out[key] = value if key == "head": out["head_sha256"], out["tree_sha256"] = hashlib.sha256(tar).hexdigest(), hashlib.sha256(listed).hexdigest() - return (json.dumps(out, indent=2, ensure_ascii=False) + "\n").encode("utf-8"), None + new = (json.dumps(out, indent=2, ensure_ascii=False) + "\n").encode("utf-8") + put(folder, "%s.tar.gz" % ident, tar) + put(folder, "%s.tree.json" % ident, listed) + return new, None def main(argv: list, env=None, transport=None, out=None) -> int: from adapters import github env, out = os.environ if env is None else env, sys.stdout if out is None else out - if len(argv) != 2 or not PART.fullmatch(argv[1]) or not (pathlib.Path(argv[0]) / argv[1]).is_dir(): + if len(argv) != 2 or not PART.fullmatch(argv[1]) or not (pathlib.Path(argv[0]) / argv[1]).is_dir() \ + or (pathlib.Path(argv[0]) / argv[1]).is_symlink(): out.write("usage: GH_TOKEN=... python3 -m tools.heads_into_home \n") return 2 folder = pathlib.Path(argv[0]) / argv[1] @@ -78,14 +112,20 @@ def main(argv: list, env=None, transport=None, out=None) -> int: if not env.get("GH_TOKEN"): out.write("no token: give one as GH_TOKEN\n") return 2 - transport = github.Live(env["GH_TOKEN"]) + from harness.eval.fetch import ReadOnly + transport = ReadOnly(github.Live(env["GH_TOKEN"])) said, entries = collections.Counter(), [] for path in sorted(p for p in folder.iterdir() if p.suffix == ".json" and not p.name.endswith(".tree.json")): ident = path.name[:-len(".json")] - new, code = pinned(path.read_bytes(), transport, folder, ident) if PART.fullmatch(ident) else (None, "not_a_case") + data = read(path) if PART.fullmatch(ident) else None + try: + new, code = pinned(data, transport, folder, ident) if data is not None else (None, "not_a_case") + if new is not None: + put(folder, path.name, new) + except Exception: # by its code alone: no message, which could quote a case + new, code = None, "not_written" said[code or "pinned"] += 1 if new is not None: - path.write_bytes(new) entries.append({"id": ident, "sha256": hashlib.sha256(new).hexdigest()}) out.write("pinned %d of %d cases%s\n" % (said["pinned"], sum(said.values()), "".join( "; %s %d" % (code, n) for code, n in sorted(said.items()) if code != "pinned"))) diff --git a/tools/mutants.py b/tools/mutants.py index c3c8c73f..d4664436 100644 --- a/tools/mutants.py +++ b/tools/mutants.py @@ -9300,6 +9300,23 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: 'hashlib.sha256(listed).hexdigest()', 'hashlib.sha256(json.dumps(answer["body"]).encode()).hexdigest()', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), ("the Maintainer's tool reads with no token", 'tools/heads_into_home.py', ' if not env.get("GH_TOKEN"):\n', ' if False:\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ('a head not held is left in the heads folder', 'harness/eval/__init__.py', + ' shutil.rmtree(into, ignore_errors=True)\n', ' pass\n', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ('a case whose pin is no sha256 has its head read', 'harness/eval/__init__.py', + ' and all(isinstance(p, str) and SHA256.fullmatch(p) for p in pins)', '', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ("the Maintainer's tool writes through a link", 'tools/heads_into_home.py', + ' os.replace(temporary, str(folder / name))\n', + ' (folder / name).write_bytes(pathlib.Path(temporary).read_bytes())\n os.unlink(temporary)\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool reads a case that is a link", 'tools/heads_into_home.py', + ' | getattr(os, "O_NOFOLLOW", 0)', '', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool pins a head the job will refuse", 'tools/heads_into_home.py', + ' if len(tar) > heldout.BESIDE["tar.gz"] or len(listed) > heldout.BESIDE["tree.json"]:\n', ' if False:\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool reads a folder that is a link", 'tools/heads_into_home.py', + ' \\\n or (pathlib.Path(argv[0]) / argv[1]).is_symlink():', ':', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("a write that fails stops the Maintainer's tool", 'tools/heads_into_home.py', + ' except Exception: # by its code alone', ' except ZeroDivisionError: # by its code alone', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ("the Maintainer's tool leaves a file half written", 'tools/heads_into_home.py', + ' with contextlib.suppress(OSError):\n os.unlink(temporary)\n', ' pass\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), # Issue 613, the preflight: each dispatch of a gate set asks a few development cases first, and a broken pipeline stops it. ('a gate set need name no preflight', 'harness/eval/__init__.py', 'OWN = ("bars", "kind_least", "record", "preflight")', From 2b6155b694246354f060f8e34d61537dd9cc9237 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:06:56 -0600 Subject: [PATCH 07/10] Verify's findings on the heads: the tool's imports, the record's names, each row's mutant and test, and the rules within their limits Co-Authored-By: Claude Opus 5.5 --- adapters/AGENTS.md | 3 +-- ...e-read-tools-alone-held-to-it-by-restricted.md | 2 +- ...ome-pinned-by-the-case-and-held-to-its-tree.md | 8 ++++---- harness/eval/AGENTS.md | 3 ++- tests/test_evals.py | 5 +++-- tools/AGENTS.md | 3 +-- tools/heads_into_home.py | 15 ++++++++------- 7 files changed, 20 insertions(+), 19 deletions(-) diff --git a/adapters/AGENTS.md b/adapters/AGENTS.md index 1a0b2358..77c4744b 100644 --- a/adapters/AGENTS.md +++ b/adapters/AGENTS.md @@ -33,8 +33,7 @@ Not built yet, by the stories that hold their rules: contract.py, binding.py, is | An issue asks for a contract only by an Intent that asks or a person's `factory:contract` label; if none asks, `acceptance` is null and whole | `tests/test_adapters.py::test_an_issue_asks_for_a_contract_only_by_a_persons_label` | | Closing references read short, of another repo or named twice, are not whole | `tests/test_adapters.py::test_closing_references_are_whole_only_of_this_repo_and_every_one_counted` | | A pull request to another base than the default is not whole on its contract | `tests/test_adapters.py::test_a_pull_request_to_another_base_than_the_default_is_not_read_whole_on_its_contract` | -| A held-out case is the bytes its manifest hashed, or none and a code; one that could name another file is not asked | `tests/test_adapters.py::test_a_held_out_case_is_its_hashed_bytes`, `tests/test_adapters.py::test_a_held_out_case_not_read_whole_says_why` | -| What the home keeps beside a case, its head's tarball and tree, is read raw and handed back only as the bytes the case pins | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins` | +| A held-out case, and its head beside it, are the bytes pinned, or none and a code; one that could name another file is not asked | `tests/test_adapters.py::test_a_held_out_case_not_read_whole_says_why`, `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins` | | The title, and the head the snapshot was held to, are handed beside the facts, never in them | `tests/test_adapters.py::test_the_pull_requests_own_facts_and_the_title_beside_them`, `tests/test_adapters.py::test_a_snapshot_hands_back_its_head` | | A plan of writes not its pass record's sends nothing; a dispatch is sent once, never retried | `tests/test_effects.py::test_a_write_list_that_is_not_the_records_is_refused_whole`, `tests/test_effects.py::test_a_dispatch_is_sent_once_and_never_retried` | | Each write goes through its one credential, which sends only its own requests by path and body | `tests/test_effects.py::test_each_write_goes_through_its_one_credential_and_its_allowlist` | diff --git a/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md b/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md index 32511943..d69c0b68 100644 --- a/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md +++ b/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md @@ -41,7 +41,7 @@ What is owed: what codeload's address holds for a private repo read with an App' | Each archive it refuses, it refuses whole | `tests/test_harness.py::test_each_archive_the_unpacker_refuses_it_refuses_whole` | | It holds its caps, writes into a new folder, and nothing outside it | `tests/test_harness.py::test_the_unpacker_holds_its_caps_and_writes_nothing_outside_its_folder` | | An eval case requires its head, as 40 lowercase hex characters | `tools/check_examples.py`, `examples/invalid/eval-case.no-head.json`, `examples/invalid/eval-case.a-head-as-an-object.json`, `examples/invalid/eval-case.a-short-head.json` | -| Each case's head is read with no credential and removed once asked; one not read closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate` | +| Each case's head is read with no credential and removed once asked; one not read closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, mutant: "the eval fetch reads no case's head" | | A case is reviewed with the read tools in its head, as a live Run is | `tests/test_evals.py::test_a_case_is_reviewed_by_the_task_and_the_engine_a_live_run_is` | | The read tools are held to the folder at the pin, and by which flag | `trials/P-59.json`, `tests/test_tools.py::test_the_read_tools_on_a_linux_runner_are_held_to_their_folder_only_with_both_flags` | | The tarball is one redirect to codeload, which redirects no further over HTTPS | `trials/P-60.json`, `tests/test_tools.py::test_a_commits_tarball_is_one_redirect_to_codeload_which_never_redirects_over_https` | diff --git a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md index a08adae0..0305556b 100644 --- a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md +++ b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md @@ -1,14 +1,14 @@ # NNNN. A case's head is read from the held-out home, pinned by the case, and held to its tree -Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded", points (a) to (d): this is the first of its two pull requests, the heads; the second shards the run. It amends decision 0138's rows "A transport with no token" and "The eval case" where they say the eval job reads a case's head with no credential, and its Boundaries row "Each case's head is read with no credential and removed once asked", whose check now holds the read from the home that replaces it; and decision 0142's row "An eval case's head", which this closes. Nothing else in either changes. +Status: accepted. The Factory Builder's rulings of 2026-11-08 on issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded", points (a) to (d): this is the first of its two pull requests, the heads; the second shards the run. It amends decision 0138's rows "A transport with no token" and "The eval case" where they say the eval job reads a case's head with no credential, and its Boundaries row "Each case's head is read with no credential and removed once asked", whose check is now the test of the read from the home; and decision 0142's row "An eval case's head", which this closes. Nothing else in either changes. ## Decision | What | Decided | Because | |---|---|---| | Where a head is kept | Beside each case in the held-out home, `/.json`, the home keeps the case's head as two files: `.tar.gz`, the commit's tarball as GitHub archived it, and `.tree.json`, the commit's tree, every level of it, as the API answered it. `eval-case/1` requires `head_sha256` and `tree_sha256`, the sha256 of each, beside `head` | The issue's: a head read anonymously shares the anonymous cap, 60 an hour, so a pool of three runs, or a run again within the hour, can fault on `head_unreadable`; and a head that is not on GitHub cannot be read at all. Pinned by the case, which the manifest pins, the head is the bytes the Maintainer chose | -| How it is read | `fetch` reads both files with the home's token, through the transport the case is read with, by `adapters/heldout.beside`: each with the raw media type, `application/vnd.github.raw`, whose answer is a file's bytes whole however large, in one answer and no redirect (`trials/P-62.json`). An answer of another type, `+json` among them, which the transport would read as JSON, is no file. A file over its bound (the transport's tarball bound, `TARRED`, or 8 MiB for a tree) is `too_large`, and one that does not hash to its pin `hash_differs`. Nothing in the eval job reads with no credential, and the home's token is still sent nowhere else | The Factory Builder's ruling (c): no transition, and no case without its pins, since an anonymous read kept for one would keep the read the issue removes. Ruling (d): the raw media type, grounded by its trial, the git blobs API its fallback; and the ruling on P-62's verdict, that the raw media type is `application/vnd.github.raw`, which the transport must ask for | -| How it is held | The tarball is unpacked as before (decision 0138), and then held to the tree file by file, by git's blob hash, by the check a live head is held by (decision 0142): `harness/head.tree_of` reads the tree file with the checks `tree` reads GitHub's answer with, and `held` holds the head to it. A head not the one pinned is `hash_differs`, a fault of the run; one not read, unpacked or held whole is `head_unreadable`, the Engine's or the Factory's. Either closes the gate | The Factory Builder's ruling (b): pin the tree as well as the tarball, and hold every eval head to it. 0142 left an eval head unheld for want of room under the anonymous cap; read with the home's token, there is room. An eval case is now graded on the files a live review would be given | +| How it is read | `fetch` reads both files with the home's token, through the transport the case is read with, by `beside` in `adapters/heldout.py`: each with the raw media type, `RAW` in `adapters/heldout.py`, whose answer is a file's bytes whole however large, in one answer and no redirect (`trials/P-62.json`). An answer of another type, `+json` among them, which the transport would read as JSON, is no file. A file over its bound (the transport's tarball bound, `TARRED`, or 8 MiB for a tree) is `too_large`, and one that does not hash to its pin `hash_differs`. Nothing in the eval job reads with no credential, and the home's token is still sent nowhere else | The Factory Builder's ruling (c): no transition, and no case without its pins, since an anonymous read kept for one would keep the read the issue removes. Ruling (d): the raw media type, grounded by its trial, the git blobs API its fallback; and the ruling on P-62's verdict, that the raw media type is GitHub's `vnd.github.raw`, which the transport must ask for | +| How it is held | The tarball is unpacked as before (decision 0138), and then held to the tree file by file, by git's blob hash, by the check a live head is held by (decision 0142): `tree_of` in `harness/head.py` reads the tree file with the checks `tree` reads GitHub's answer with, and `held` holds the head to it. A head not the one pinned is `hash_differs`, a fault of the run; one not read, unpacked or held whole is `head_unreadable`, the Engine's or the Factory's. Either closes the gate | The Factory Builder's ruling (b): pin the tree as well as the tarball, and hold every eval head to it. 0142 left an eval head unheld for want of room under the anonymous cap; read with the home's token, there is room. An eval case is now graded on the files a live review would be given | | Who puts them there | The Maintainer, at his terminal, in his checkout of the home: `GH_TOKEN="$(gh auth token)" python3 -m tools.heads_into_home `. For each case of the folder, it reads the head's tarball and tree from GitHub through the one transport, read-only, holds the one to the other, writes both beside the case, the tree compact as GitHub's answer held it, and writes the case anew with their sha256 after its `head`; a case it cannot pin, or whose head is over what the job reads, it leaves as it was. Each file is written whole and moved over its name, so a link there is replaced, never written through, and a case or a folder that is a link is not read. It prints counts, then each pinned case's id and the sha256 of its new bytes, never a byte of a case. It takes a role folder and touches nothing else, so a gate set made later is pinned as the review Intent's set is | The Factory Builder's ruling (c): the tool serves the coming gate set too, and prints counts and hashes alone. A session never sees a case | | The set changes with it | Pinning changes each case's bytes, so the set's manifest under `evals/` takes the hashes the tool prints, in the pull request that brings this change. It merges only once the home holds the files, and once `trials/P-62.json` is on main | The Factory Builder's ruling (c) and (d) | | The canary | `harness/eval/canary.py` still reads its case's head with no credential: its case is this repo's own example, of this public repo, never one of the home's, and pins no file the home keeps | The canary is no step of the eval job (decision 0106) | @@ -25,5 +25,5 @@ A case's head was read with no credential, from the public repo its change was m |---|---| | What the home keeps beside a case is asked with the raw media type and the home's token, and handed back only as the bytes the case pins: an answer of another type, `+json` among them, one not found, refused, over its bound or of another hash is none, and a name, kind, pin or home that could name another file is not asked | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins`, `trials/P-62.json`, `adapters/heldout.py`, mutant: "the eval blob is asked for as raw+json", mutant: "a head beside a case of another hash is read", mutant: "what the home keeps beside a case is asked as JSON", mutant: "a head of any size is read", mutant: "a head answered under another type is read", mutant: "a head answered as raw JSON is read", mutant: "a head GitHub did not find is unanswered", mutant: "a head GitHub refused is read", mutant: "a file of any kind is asked beside a case", mutant: "a pin of any shape is asked by", mutant: "a file beside a case of any name is asked", mutant: "a home of any name is asked beside a case" | | The fetch reads each case's head from the home by its pins, with no other transport, unpacks it and holds it to its tree; a head not the one pinned is `hash_differs`, one not read, unpacked or held whole `head_unreadable` and removed, a pin that is no sha256 reads nothing and the case is `not_this_set`, and the gate closes | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, `harness/eval/fetch.py`, `harness/head.py`, mutant: "the eval fetch reads no case's head", mutant: "a head is unpacked and never held to its tree", mutant: "a head not the one pinned is read as unreadable", mutant: "the tree's pin is never read", mutant: "a head is read beside another Role", mutant: "a tree that cannot be read stops the fetch", mutant: "a head not held is left in the heads folder", mutant: "a case whose pin is no sha256 has its head read" | -| An eval case requires both pins, each a whole sha256 | `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a tree pin may be cut short" | +| An eval case requires both pins, each a whole sha256 | `tests/test_evals.py::test_a_case_dropped_under_evals_in_any_file_is_caught`, `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a tree pin may be cut short" | | The Maintainer's tool writes a head beside its case only once it is held to its tree and within what the job reads, pins what it wrote, writes each file whole and through no link, reads no case or folder that is a link, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token", mutant: "the Maintainer's tool writes through a link", mutant: "the Maintainer's tool reads a case that is a link", mutant: "the Maintainer's tool pins a head the job will refuse", mutant: "the Maintainer's tool reads a folder that is a link", mutant: "a write that fails stops the Maintainer's tool", mutant: "the Maintainer's tool leaves a file half written" | diff --git a/harness/eval/AGENTS.md b/harness/eval/AGENTS.md index 9ca16451..5b834c5e 100644 --- a/harness/eval/AGENTS.md +++ b/harness/eval/AGENTS.md @@ -1,7 +1,8 @@ # harness/eval/ The eval job's steps, a module each, in the order `__init__.py` gives. Their rules are `evals/AGENTS.md`'s, beside the sets they run, but for what three issues add, which are here: -issue 612, "The eval job reads a named set, and a gate set's report is aggregates only", issue 613, "The gate pools k dispatches on one commit and reads all of them", and issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded". +issue 612, "The eval job reads a named set, and a gate set's report is aggregates only", issue 613, "The gate pools k dispatches on one commit and reads all of them", +and issue 611, "A case's head is read from the private home, pinned by its hash, and the run is sharded". A dispatch names its set. A set whose manifest says it is a gate set is reported by counts alone, `eval-aggregate/1`, and each case's outcome with its Result goes to a ledger, `eval-ledger/1`, kept in the scratch. `seal.py` encrypts the ledger with `openssl cms` to each of the Maintainer's certificates that its pins, in a protected path under `.github/`, name by sha256 (`seal.RECIPIENTS`). The ciphertext alone is uploaded, as `eval-ledger`. No key that opens a ledger is in the job or the repo, and nothing automated reads one. Until the Maintainer commits his certificates, a gate set's run fails at the seal, `no_recipient`. diff --git a/tests/test_evals.py b/tests/test_evals.py index 22cb2265..adfa81fc 100644 --- a/tests/test_evals.py +++ b/tests/test_evals.py @@ -2667,7 +2667,7 @@ class Home(github.Replay): def send(self, req): asked.append((req["path"].rsplit("/", 1)[-1], req["accept"])) return super().send(req) - self.assertEqual(self.step(job.fetch, Home(exchanges)), "fetched 2 of 7 cases") + self.assertEqual(self.step(getattr(job.fetch, "fetch", job.fetch), Home(exchanges)), "fetched 2 of 7 cases") self.assertEqual(asked[:3], [("poc-one.json", github.JSON), ("poc-one.tar.gz", heldout.RAW), ("poc-one.tree.json", heldout.RAW)]) self.assertEqual(len(asked), len(exchanges), "each answer the home keeps is asked, once, in its order") @@ -2780,7 +2780,8 @@ def send(self, req): self.step(job.plan) files = [home_file("poc-one", new), home_beside("poc-one", "tar.gz", (home / "pr-review" / "poc-one.tar.gz").read_bytes()), home_beside("poc-one", "tree.json", (home / "pr-review" / "poc-one.tree.json").read_bytes())] - self.assertEqual(self.step(job.fetch, github.Replay(files)), "fetched 1 of 1 cases") + fetch = getattr(job.fetch, "fetch", job.fetch) # the step, whether or not its module was imported + self.assertEqual(self.step(fetch, github.Replay(files)), "fetched 1 of 1 cases") # A head over what the job reads is not pinned; nor is a case whose write fails, by its code alone. from adapters import heldout with mock.patch.dict(heldout.BESIDE, {"tree.json": 10}): diff --git a/tools/AGENTS.md b/tools/AGENTS.md index bdeb0d7b..2a77e5f3 100644 --- a/tools/AGENTS.md +++ b/tools/AGENTS.md @@ -4,7 +4,7 @@ The checks, `verify.py` which runs them all, `mutants.py` which breaks the Kerne unless each commit names the Maintainer and a listed account opened it. `walk.py` lists this working tree's files, and nothing in another nested inside it. It skips local output, Python's caches and the corpus, only while git's index does not hold it, and reads the index as a file, so no git is needed. CI checks out the commit afresh, so there all else is committed; an export, with no `.git`, cannot tell, and skips it all. `make_corpus.py` records every decision the Kernel makes, under a directory git ignores, to hold another implementation to this one; the required check does not depend on it. `hooks/` holds the session hooks `.claude/settings.json` runs; `hooks/shell.py` reads a shell command as words for any of them. `eval_result.py` scores one Result -against one held-out case, where the case is, and is no check; `context_eval.sh` runs the context's test at the Maintainer's terminal and `context_eval.py` records a run, and neither is a check. `untested.py` reports Python no test notices (decision 0166). `heads_into_home.py` pins each case's head beside it in the held-out home, at the Maintainer's terminal; it is no check. +against one held-out case, where the case is, and is no check; `context_eval.sh` runs the context's test at the Maintainer's terminal and `context_eval.py` records a run, and neither is a check. `untested.py` reports Python no test notices (decision 0166). `rulesets_drift.sh` reads the repo's live rulesets through whoever is logged in to gh and redacts each answer outside the repo, and `rulesets_drift.py` compares what is left with `config/rulesets.json`; it is run by hand and is no check. `machine_user_drift.sh` does the same for `config/machine-user.json`, and `environments_drift.sh` holds each environment to `main` alone. @@ -33,7 +33,6 @@ and `rulesets_drift.py` compares what is left with `config/rulesets.json`; it is | A test kills a mutant only failing on it, again alone, and passing alone in an unbroken copy; else `FLAKY`, or `ERROR` on a hang: both fail a run | `tests/test_tools.py::test_a_failure_that_is_not_the_mutants_twice_is_flaky_and_fails_the_check` | | A `mutants.py` entry names text found once in its file (else `STALE`; update it) and tests that kill it, each existing, none in `WHOLE_TOO_LONG` nor its module | `mutants.py`, `tests/test_tools.py::test_every_mutant_names_a_test_that_exists` | | After the named tests, all of `tests/` runs in dealt parts; `mutants PATH...` breaks at most 40, two at once | `tests/test_tools.py::test_the_whole_suite_runs_in_dealt_parts_each_held_to_the_limit`, `tests/test_tools.py::test_a_filtered_run_is_capped` | -| `heads_into_home.py` writes a head beside its case only once it is held to its tree, pins what it wrote, and prints counts and hashes alone | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone` | A check collects every problem and does not stop at the first. Its message says what is wrong and what to do, in one sentence, because an author will read it with nothing else to go on. diff --git a/tools/heads_into_home.py b/tools/heads_into_home.py index ed168943..13d4dcf9 100644 --- a/tools/heads_into_home.py +++ b/tools/heads_into_home.py @@ -16,14 +16,12 @@ under `evals/` takes. A case it did not pin is left as it was. It exits 0 only when every case was pinned.""" from __future__ import annotations -import collections import contextlib import hashlib import json import os import pathlib import re -import stat import sys import tempfile @@ -47,13 +45,16 @@ def put(folder: pathlib.Path, name: str, data: bytes) -> None: def read(path: pathlib.Path): - """The bytes of the case at `path`, or None for one that is a link or not a regular file.""" + """The bytes of the case at `path`, or None for one that is a link or cannot be read as a file.""" try: fd = os.open(str(path), os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)) except OSError: return None - with os.fdopen(fd, "rb") as f: - return f.read() if stat.S_ISREG(os.fstat(f.fileno()).st_mode) else None + try: + with os.fdopen(fd, "rb") as f: + return f.read() + except OSError: + return None def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): @@ -114,7 +115,7 @@ def main(argv: list, env=None, transport=None, out=None) -> int: return 2 from harness.eval.fetch import ReadOnly transport = ReadOnly(github.Live(env["GH_TOKEN"])) - said, entries = collections.Counter(), [] + said, entries = {"pinned": 0}, [] for path in sorted(p for p in folder.iterdir() if p.suffix == ".json" and not p.name.endswith(".tree.json")): ident = path.name[:-len(".json")] data = read(path) if PART.fullmatch(ident) else None @@ -124,7 +125,7 @@ def main(argv: list, env=None, transport=None, out=None) -> int: put(folder, path.name, new) except Exception: # by its code alone: no message, which could quote a case new, code = None, "not_written" - said[code or "pinned"] += 1 + said[code or "pinned"] = said.get(code or "pinned", 0) + 1 if new is not None: entries.append({"id": ident, "sha256": hashlib.sha256(new).hexdigest()}) out.write("pinned %d of %d cases%s\n" % (said["pinned"], sum(said.values()), "".join( From d0a2adae958c2a9ddc6af0169f4565b4791ae7bd Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:28:02 -0600 Subject: [PATCH 08/10] The heads: a rule within its line, and 0138's row names the file its mutant changes Co-Authored-By: Claude Opus 5.5 --- adapters/AGENTS.md | 2 +- ...s-head-with-the-read-tools-alone-held-to-it-by-restricted.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/adapters/AGENTS.md b/adapters/AGENTS.md index 77c4744b..09f62930 100644 --- a/adapters/AGENTS.md +++ b/adapters/AGENTS.md @@ -33,7 +33,7 @@ Not built yet, by the stories that hold their rules: contract.py, binding.py, is | An issue asks for a contract only by an Intent that asks or a person's `factory:contract` label; if none asks, `acceptance` is null and whole | `tests/test_adapters.py::test_an_issue_asks_for_a_contract_only_by_a_persons_label` | | Closing references read short, of another repo or named twice, are not whole | `tests/test_adapters.py::test_closing_references_are_whole_only_of_this_repo_and_every_one_counted` | | A pull request to another base than the default is not whole on its contract | `tests/test_adapters.py::test_a_pull_request_to_another_base_than_the_default_is_not_read_whole_on_its_contract` | -| A held-out case, and its head beside it, are the bytes pinned, or none and a code; one that could name another file is not asked | `tests/test_adapters.py::test_a_held_out_case_not_read_whole_says_why`, `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins` | +| A held-out case and its head are the bytes pinned, or none and a code | `tests/test_adapters.py::test_a_held_out_case_not_read_whole_says_why`, `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins` | | The title, and the head the snapshot was held to, are handed beside the facts, never in them | `tests/test_adapters.py::test_the_pull_requests_own_facts_and_the_title_beside_them`, `tests/test_adapters.py::test_a_snapshot_hands_back_its_head` | | A plan of writes not its pass record's sends nothing; a dispatch is sent once, never retried | `tests/test_effects.py::test_a_write_list_that_is_not_the_records_is_refused_whole`, `tests/test_effects.py::test_a_dispatch_is_sent_once_and_never_retried` | | Each write goes through its one credential, which sends only its own requests by path and body | `tests/test_effects.py::test_each_write_goes_through_its_one_credential_and_its_allowlist` | diff --git a/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md b/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md index d69c0b68..8658f675 100644 --- a/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md +++ b/docs/decisions/0138-the-review-reads-the-pull-requests-head-with-the-read-tools-alone-held-to-it-by-restricted.md @@ -41,7 +41,7 @@ What is owed: what codeload's address holds for a private repo read with an App' | Each archive it refuses, it refuses whole | `tests/test_harness.py::test_each_archive_the_unpacker_refuses_it_refuses_whole` | | It holds its caps, writes into a new folder, and nothing outside it | `tests/test_harness.py::test_the_unpacker_holds_its_caps_and_writes_nothing_outside_its_folder` | | An eval case requires its head, as 40 lowercase hex characters | `tools/check_examples.py`, `examples/invalid/eval-case.no-head.json`, `examples/invalid/eval-case.a-head-as-an-object.json`, `examples/invalid/eval-case.a-short-head.json` | -| Each case's head is read with no credential and removed once asked; one not read closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, mutant: "the eval fetch reads no case's head" | +| Each case's head is read with no credential and removed once asked; one not read closes the gate | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, mutant: "the eval fetch reads no case's head" | | A case is reviewed with the read tools in its head, as a live Run is | `tests/test_evals.py::test_a_case_is_reviewed_by_the_task_and_the_engine_a_live_run_is` | | The read tools are held to the folder at the pin, and by which flag | `trials/P-59.json`, `tests/test_tools.py::test_the_read_tools_on_a_linux_runner_are_held_to_their_folder_only_with_both_flags` | | The tarball is one redirect to codeload, which redirects no further over HTTPS | `trials/P-60.json`, `tests/test_tools.py::test_a_commits_tarball_is_one_redirect_to_codeload_which_never_redirects_over_https` | From 231a36fcca1630b6417869810c767cb4e379494a Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 21:08:01 -0600 Subject: [PATCH 09/10] Main since P-62 was cut: #650, #652, #654 Co-Authored-By: Claude Opus 5.5 --- ...write-where-and-when-the-shell-makes-it.md | 2 + ...s-sys-code-from-text-and-shell-commands.md | 33 ++ tests/test_hooks.py | 64 ++++ tests/test_tools.py | 240 +++++++++++- tools/check_writes.py | 283 ++++++++++++-- tools/hooks/identity.py | 13 + tools/hooks/protected.py | 32 +- tools/hooks/shell.py | 28 ++ tools/mutants.py | 360 +++++++++++++++++- 9 files changed, 994 insertions(+), 61 deletions(-) create mode 100644 docs/decisions/0171-the-writes-check-reads-a-name-back-through-what-it-is-given-and-holds-an-objects-class-another-modules-names-sys-code-from-text-and-shell-commands.md diff --git a/docs/decisions/0032-the-identity-hook-reads-a-write-where-and-when-the-shell-makes-it.md b/docs/decisions/0032-the-identity-hook-reads-a-write-where-and-when-the-shell-makes-it.md index cf7200c5..d2f1676e 100644 --- a/docs/decisions/0032-the-identity-hook-reads-a-write-where-and-when-the-shell-makes-it.md +++ b/docs/decisions/0032-the-identity-hook-reads-a-write-where-and-when-the-shell-makes-it.md @@ -68,3 +68,5 @@ into a message's substitution. | An eval's script reads where its redirections' substitutions stand as a command's own do | `tests/test_hooks.py::test_a_substitution_in_a_redirection_or_an_expanded_here_document_asks` | | A substitution's own double quotes are read as the shell reads them: a `"` in a substitution nested in it, in a comment, or in single quotes outside its double quotes is none of them, so a `${...}` with a `'` after one is still double-quoted, and the hooks ask. A comment there begins where the reader begins one, outside a `${...}`, not after a substitution's `)` nor after a character a backslash holds, and at a backtick's start; in a backtick it ends at the backtick | `tests/test_hooks.py::test_a_substitutions_own_quotes_are_read_as_the_shell_reads_them`, `tools/hooks/shell.py`, mutant: "a substitution nested in a held one is read with its quotes", mutant: "a quote in a comment in a substitution is counted", mutant: "a double quote single quotes hold in a substitution is counted", mutant: "a # in a ${...} in a substitution begins a comment", mutant: "a # after a substitution's ) begins a comment", mutant: "a nested substitution's quote is kept past its end", mutant: "a nested substitution opens in its holder's quote", mutant: "a # after a character a backslash holds begins a comment", mutant: "a # that begins a backtick's script is a word", mutant: "a comment in a backtick in a substitution runs past it" | | Whether a command defines a function of a name that only prints is searched once a name, not once a command, so a command of many short ones is read in linear time | `tests/test_hooks.py::test_whether_a_command_defines_a_printers_name_is_searched_once_a_name`, `tools/hooks/identity.py`, mutant: "each printer's name is searched for again", mutant: "each command that prints searches the whole text again" | +| A double-quoted substitution or backtick nested in a double-quoted substitution has quotes of its own, to the protected hook as to the identity hook: its quotes do not end the outer one, so a command after it is read. The protected hook reads the command as it stands and again so spelt, each from where the command began, and asks what either reading finds; a spelt reading that fails, or one of a command longer than it reads again, leaves the first to stand. A backtick in double quotes is spelt as a `$(...)` is: it ends at its next backtick no backslash holds, and a backslash before a double quote, a backtick, a dollar sign or a backslash in it is taken out; one that never closes, or whose quote or comment would run past its end, is read as it stands | `tests/test_hooks.py::test_a_double_quoted_substitution_in_a_double_quoted_one_hides_no_command_after_it`, `tests/test_hooks.py::test_a_backtick_in_double_quotes_has_quotes_of_its_own`, `tools/hooks/protected.py`, `tools/hooks/identity.py`, mutant: "the protected hook reads a command once, as it stands", mutant: "the protected hook's second reading begins where the first ended", mutant: "a spelt reading that fails is the protected hook's failure", mutant: "a command of any length is read again, spelt", mutant: "a process substitution is not spelt again", mutant: "a backtick in double quotes is read flat", mutant: "a backtick's escaped double quote in double quotes keeps its backslash", mutant: "a backtick's escaped dollar in double quotes keeps its backslash", mutant: "a backtick in double quotes that never closes is spelt", mutant: "a backtick in double quotes whose quote or comment runs past it is spelt" | +| In a here-document's substitution, a backtick whose quote is still open at its end is a script the hook cannot read, not data: the hooks ask, as they do outside a here-document. A quote in a comment there is none, nor one in a `$(...)` in it, which has quotes of its own; a # after a character a backslash holds begins no comment | `tests/test_hooks.py::test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data`, `tests/test_hooks.py::test_a_double_quoted_substitution_in_a_double_quoted_one_hides_no_command_after_it`, `tools/hooks/shell.py`, mutant: "a quote past a backtick in a body's substitution is data", mutant: "a quote in a comment in a body's backtick is counted", mutant: "a # after a character a backslash holds in a body's backtick begins a comment", mutant: "a substitution in a body's backtick is scanned flat", mutant: "a backslash in a body's backtick holds no quote", mutant: "a backslash in single quotes in a body's backtick holds a quote" | diff --git a/docs/decisions/0171-the-writes-check-reads-a-name-back-through-what-it-is-given-and-holds-an-objects-class-another-modules-names-sys-code-from-text-and-shell-commands.md b/docs/decisions/0171-the-writes-check-reads-a-name-back-through-what-it-is-given-and-holds-an-objects-class-another-modules-names-sys-code-from-text-and-shell-commands.md new file mode 100644 index 00000000..e26d2fc5 --- /dev/null +++ b/docs/decisions/0171-the-writes-check-reads-a-name-back-through-what-it-is-given-and-holds-an-objects-class-another-modules-names-sys-code-from-text-and-shell-commands.md @@ -0,0 +1,33 @@ +# 0171. The writes check reads a name back through what it is given, and holds an object's class, another module's names set from outside, sys and loads reached other ways, code run from text and a command a shell runs; a runner's helper stays the review's + +Status: accepted. The Factory Builder's story of 2026-11-08, issue 640, "check_writes: routes still open after issue 596, a runner's helper, a class or instance reached through another name, sys.modules by getattr", raised by the reads of issue 596, "check_writes: further ways a write could be reached past the check, written to hide it". It amends decision 0167's row "Still not held". + +## Decision + +| What | Decided | Because | +|---|---|---| +| A name read back through what it is given | `tools/check_writes.py` reads what an attribute or a key is set on back through the names it is given: a module's constant that is another name, `P = Port` or `D = other.TABLE`, and a name a function or a function round it is given, by a default, read where the function is defined, or by an assignment, a walrus or a tuple's part at its place, `cls=Port`, `c = cls`, `a, b = Port, 1`, and an attribute of a module a local name holds, `m.P`, through that module's constants. A name a function declares `global` is read through what each such function assigns it, one a nested function assigns as `nonlocal` through that, a walrus in a comprehension as the function's, and a module's name stored under an `if`, a `try` or a walrus through what it is stored. `vars(m)` is m, and a walrus is what it holds. A name a nested scope binds is not the function's own | Each sets on a class, or on another module's name, as surely as the name itself does | +| An object's class | An attribute set on `type(x)`, `x.__class__`, `x.__mro__`, `x.__bases__` or `x.__base__`, or on what holds one, is refused, and so is `setattr` or `setitem` named as a value, and either's dunder | Which class an object has is a value the syntax does not give | +| A class's attribute, set other ways | A `for`, a `with` or a comprehension that sets an attribute sets it as an assignment does. A function under an `if` or a `try` is a function, and a nested `def` or `class`, or a caught exception's name, shadows a text constant | Decision 0167 refused an attribute set on a class from outside by an assignment alone | +| Another module's names | An attribute or a key set, or a container filled by `update`, `append` or their kind, or its names handed to a call as `vars(m)` or `m.__dict__`, from another module on what a module of the repo's defines or is, an instance, a dict, a function or the module itself, is refused, to anything but text. PATCHERS names who may, with each name: `tools/make_corpus.py`, which sets `sim.world.decide` to its recorder. A new one is the Maintainer's to allow, and an entry for a name its file does not set is refused | A caller reads what a module's own code gives its names; what another module sets on them is a value the syntax does not follow | +| sys and loads reached other ways | Refused as a load by name: `.modules` read of anything, or set on `sys`; `sys` named as a value, `s = sys` or `getattr(sys, 'modules')`; its namespace, `sys.__dict__`; `sys` reached through another module, `os.sys`, or imported from one, `from os import sys`; `getattr` given `"sys"` or `"modules"`, or a key `"sys"` or `"modules"` read, as a literal; `__builtins__` and `__import__` as an attribute; pkgutil and runpy, imported; and a loader imported from a module of the repo's, or named as a module's attribute | Each reaches a module by its name, which decision 0167 refuses by its plain names | +| Code run from text | `exec`, `eval` and `compile`, named where the module neither imports nor defines the name, and `exec` or `eval` as an attribute, are refused. `from re import compile` passes; a parameter or a name only assigned may still be the builtin, and is refused | What code a string holds is not syntax this check reads | +| A command a shell runs | gh or git is a command in any case, by a POSIX or a Windows path, with `.exe` or not: at the start of a string; after a new line, a shell's `;`, `&`, `|`, `(`, `{` or `!`; after `env`, `exec`, `command`, `nohup`, `xargs`, `sudo`, `time`, `nice`, `if`, `elif`, `while`, `until`, `then`, `do` or `else`, with its options and their values, or after `timeout` and its duration; after a variable's assignment; ended by a space, the end or a shell's `;`, `&`, `|`, `)` or `}`; or quoted, opening the quote. A bytes literal is read as a string. RUNNERS gains `tools/hooks/shell.py`, which reads a command it is given and holds a pattern for git and gh, and `tools/check_workflows.py`, which holds the workflows' text it checks, `gh api` calls among it | A shell runs each, `['sh', '-c', 'cd x && git push']` as surely as `'git push'` | +| A namespace package's `import *` | An import of all of a namespace package is refused, as one of a module's is | Its names are each a module's, and what they reach is not read | +| Still not held | A RUNNERS function that runs git with arguments it is given can be handed a write's by its caller. Eleven modules on main reach such a function, so listing its callers would list most of the harness, and holding each helper to read-only subcommands changes most of the twelve RUNNERS files, six of them hooks. A name given by a loop, a `with`, a container, a call's return or a starred part; `sys` or a module reached by `getattr` of a name not a literal, or by `unittest.mock.patch` given a module's name, or another Python run as a process; a command after a backtick, built from parts, or read from outside the code; a `match` capture's shadow, which Python 3.9 cannot parse and so refuses whole; a class attribute that holds another class, `H.P.w = ...`; an attribute set on a class from a class body in its own module. Nor are the routes the last reads of this change found: a stored name imported under another name, a second class or owner traced, a namespace handed on through a local name, a nonlocal value two steps away, `exec` let through by a binding that never runs, a loader reached by `getattr`, and `timeout`'s options or another word that runs the next, `eval`, `watch` or `stdbuf`. Each is issue 655's, "check_writes: routes still open after issue 640, an aliased stored name, a second class traced, exec by a binding that never runs". Text that runs nothing but reads as a command, `'~/.config/gh'` or a message that opens with Git, a local name `eval`, and an object's class counted, `type(self).n += 1`, are refused: each fails safe, and its message says what to do | The check is a net for code written plainly, as decision 0153 says. These need values the syntax does not give, or the helpers' own change, and are left to the review | + +## Why + +Those reads found these routes after that change's last fix round, and the reads of this change found more of the same kinds. Each but the runner's helper is a pattern the syntax names, so the check names it. On main only `tools/make_corpus.py` takes one of them, and it reads. + +## Boundaries + +| Boundary | Check | +|---|---| +| A class by a constant's or a function's name for it, by two steps, a tuple, a walrus or an outer function's default; an object's class, held by a name too; `setattr` as a value or by its dunder; a `for`, `with` or comprehension target; a setter under an `if`; and a text constant a nested `def`, `class` or catch shadows, are refused | `tests/test_tools.py::test_what_cannot_be_read_fails`, `tools/check_writes.py`, mutant: "a class by a constant's other name is not read", mutant: "a function's default is not given", mutant: "a keyword's default is not given", mutant: "an assignment is not given", mutant: "a walrus is not given", mutant: "a tuple's part is not given", mutant: "an outer function's name is not read", mutant: "type(x) is no object's class", mutant: "__class__ is no object's class", mutant: "__mro__ is no object's class", mutant: "__bases__ is no object's class", mutant: "__base__ is no object's class", mutant: "a default is read inside its function", mutant: "a walrus set on is not read", mutant: "a local module's constant is not followed", mutant: "a name declared global is not read", mutant: "a nested function's nonlocal is not given", mutant: "a module's name stored some other way is not read back", mutant: "a walrus in a comprehension is not given", mutant: "a walrus in a comprehension is not the function's", mutant: "setitem may set a key", mutant: "__setitem__ may set a key", mutant: "what an object's class holds is not read", mutant: "an object's class may be set", mutant: "setattr's dunder is let through", mutant: "a for target is not read", mutant: "a comprehension's target is not read", mutant: "a with target is not read", mutant: "a nested class does not shadow", mutant: "a nested def does not shadow", mutant: "a catch does not shadow", mutant: "a setter under an if is read as the top level" | +| An attribute or a key set, or a container filled, from another module on its instance, dict, function or the module itself, by any name read back, two keys deep too, or its names handed to a call, is refused; a name the function binds, one a nested scope binds, and a module's own names, pass | `tests/test_tools.py::test_what_cannot_be_read_fails`, `tools/check_writes.py`, mutant: "an instance, a dict or a module of another module may be set", mutant: "a name of another module's may be set by any module", mutant: "a container filled by a call is not read", mutant: "vars(m) is not m", mutant: "a key two deep is not followed to its owner", mutant: "a key set is named as an attribute", mutant: "an attribute of a local module is not read", mutant: "a local name is read as the module's", mutant: "a nested scope's assignment is given", mutant: "a module's own name set is refused", mutant: "another module's names handed to a call are let through" | +| `.modules`, `sys` as a value, its namespace, through or from another module, or by a literal `getattr` or key; `__builtins__`; `__import__` as an attribute; pkgutil and runpy; and `exec`, `eval` and `compile` unless the module imports or defines the name, are refused | `tests/test_tools.py::test_what_cannot_be_read_fails`, `tools/check_writes.py`, mutant: "sys named as a value is let through", mutant: "sys's namespace is read as an attribute of it", mutant: "sys through another module is let through", mutant: "sys imported from another module is not sys", mutant: "sys by a getattr of another module is let through", mutant: "sys by a literal key is let through", mutant: "sys's modules may be set", mutant: "__builtins__ is let through", mutant: "__import__ as an attribute is let through", mutant: "pkgutil may load a module by name", mutant: "runpy may run a module by name", mutant: "code run from text is let through", mutant: "exec is let through", mutant: "eval is let through", mutant: "compile is let through", mutant: "a name the module binds is read as exec", mutant: "exec as an attribute is let through", mutant: "a parameter named exec is the module's own", mutant: "a loader re-exported is let through", mutant: "a loader as an attribute is let through", mutant: "a store of .modules is refused" | +| gh or git in any case, by a Windows path, with `.exe`, after a new line, a shell's separator or brace, a word that runs it with options, or an assignment, ended by a shell's mark, quoted, or as bytes, is refused; text that only mentions git passes | `tests/test_tools.py::test_what_cannot_be_read_fails`, `tools/check_writes.py`, mutant: "a command is read in one case only", mutant: "a command named by a Windows path is no command", mutant: "a command's .exe hides it", mutant: "a shell's later command is no command", mutant: "a command after a new line is no command", mutant: "a command in braces is no command", mutant: "a command another word runs is no command", mutant: "a command after env's options is no command", mutant: "a command exec runs is no command", mutant: "a command after then is no command", mutant: "a command after if is no command", mutant: "an option's value hides a command", mutant: "a command after ! is no command", mutant: "a quoted command line ended by a shell's mark is no command", mutant: "timeout hides a command", mutant: "a command after a variable's assignment is no command", mutant: "a command ended by a shell's mark is no command", mutant: "a quoted command is no command", mutant: "a quoted command line is no command", mutant: "bytes are not read for a command", mutant: "a command is read only at the text's start" | +| An import of all of a namespace package is refused | `tests/test_tools.py::test_a_namespace_package_is_read_as_one`, `tools/check_writes.py`, mutant: "a namespace package's import * is read as nothing" | +| A PATCHERS entry for a name its file does not set is refused, and a listed file that sets another name is refused | `tests/test_tools.py::test_a_listed_caller_that_reaches_a_new_writer_or_none_fails`, `tools/check_writes.py`, mutant: "a PATCHERS entry that sets nothing is kept", mutant: "a patcher may set what PATCHERS does not name" | +| On main, `tools/make_corpus.py`'s recorder in `sim.world`'s place passes | `tests/test_tools.py::test_on_main_only_the_callers_listed_reach_a_write_and_each_what_it_lists`, `tools/check_writes.py`, mutant: "PATCHERS' sets are refused" | diff --git a/tests/test_hooks.py b/tests/test_hooks.py index 3f133b3b..30ca0c98 100644 --- a/tests/test_hooks.py +++ b/tests/test_hooks.py @@ -1601,6 +1601,46 @@ def test_a_substitutions_own_quotes_are_read_as_the_shell_reads_them(self): # i # So is one inside a nested one, which opens unquoted whatever quote holds it; main refused it, by its count. self.assertIn(["echo", "${x:-a}"], [w for w, _ in shell.commands('echo "$(echo "$(echo ${x:-\'a\'})")"')]) + def test_a_backtick_in_double_quotes_has_quotes_of_its_own(self): # issue 605 + # A backtick in double quotes ends at its next backtick, and a " in it is its own, as in a "$(...)": what follows + # its quotes is read as commands, \" \` \$ and \\ in it as the shell takes them out. + for command in ('echo "`echo "a b"; git push`"', 'echo "`echo \\"a b\\"; git push`"', + 'echo "`echo \\`date\\`; git push`"', 'echo "a `date` b"; git push'): + commands = shell.commands(identity._quoted(shell.without_heredocs(command, raw=True)), scopes=True) + self.assertIn(["git", "push"], [w for w, _ in commands if w], command) + self.assertIn(["echo", "a b"], [w for w, _ in shell.commands(identity._quoted('echo "`echo \\"a b\\"`"'))]) + self.assertEqual(identity._quoted('echo "`echo \\$HOME`"'), 'echo "$(echo $HOME)"') + # One escaped is text, and one that never closes is read as main reads it. + self.assertEqual(identity._quoted('echo "\\`git push\\`"'), 'echo "\\`git push\\`"') + self.assertEqual(identity._quoted('echo "`echo a'), 'echo "`echo a') + # So is one whose quote or comment would run past the ) it is given: main refuses the first and reads the + # second's push (Argon). One that never closes leaves what an expanded body runs read (the Claude review). + with self.assertRaises(ValueError): + shell.commands(identity._quoted('echo "`echo \'` $(git push) )\'"')) + for command in ('echo "`echo a #` $(git push)\n)"', 'cat < 1 or name in self.values} + # What the top level stores under an if, a try or a walrus, and what a function that declares a name global + # assigns it, each by name: a name stored so is read back through these (issue 640). + self.stored_as, self.declared = {}, {} + for node in self.nodes: + if id(node) not in inside and isinstance(node, (ast.Assign, ast.AnnAssign, ast.NamedExpr)) and node.value is not None: + for target in (node.targets if isinstance(node, ast.Assign) else [node.target]): + if isinstance(target, ast.Name): + self.stored_as.setdefault(target.id, []).append(node.value) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + for name in {g for n in ast.walk(node) if isinstance(n, ast.Global) for g in n.names}: + self.declared.setdefault(name, []).extend((node, value) for value, outer in _given(node, name) if not outer) def _filled_names(node) -> set: @@ -350,7 +383,7 @@ def _edges(self, mod: Module, qual: str, nodes: list) -> list: % (mod.rel, node.lineno, got[2], got[1])) if qual == "": self.problems += ["%s:%d imports * from %s, so what it reaches cannot be read; name what it imports" - % (mod.rel, line, source) for line, source in mod.stars if source in self.mods] + % (mod.rel, line, source) for line, source in mod.stars if _package(self.mods, source)] return out def sends(self, mod: Module, nodes: list) -> list: @@ -487,52 +520,211 @@ def _tests(mod: Module) -> list: def _sys(mods: dict, mod: Module, name: str, seen: frozenset = frozenset()) -> bool: """Whether `name` in `mod` is the module sys, imported here or re-exported from a module of the repo's.""" bound = mod.bindings.get(name) - if bound == ("module", "sys"): - return True + if bound == ("module", "sys") or bound and bound[0] == "attr" and bound[2] == "sys" and bound[1] not in mods: + return True # import sys, or from os import sys (issue 640) if bound and bound[0] == "attr" and bound[1] in mods and (bound[1], bound[2]) not in seen: return _sys(mods, mods[bound[1]], bound[2], seen | {(bound[1], bound[2])}) return False def _loaded(mods: dict, mod: Module) -> list: - """Each line that imports importlib or builtins, names __import__, or reads sys's modules, by any name or by an - import of all of sys: a module loaded or taken by its name is one the syntax cannot follow.""" + """Each line that imports importlib, builtins, pkgutil or runpy, names __import__ or __builtins__, reads `.modules` + of anything, or imports modules or all of sys; that names sys as a value or its namespace, `s = sys` or + `getattr(sys, 'modules')`; or that reaches sys through another module, `os.sys`, `getattr(os, 'sys')` or + `vars(os)['sys']`: a module loaded or taken by its name is one the syntax cannot follow.""" + held = {id(n.value) for n in mod.nodes if isinstance(n, ast.Attribute) and n.attr not in NAMESPACES} return sorted({n.lineno for n in mod.nodes if isinstance(n, ast.Import) and any(a.name.split(".")[0] in LOADER_MODULES for a in n.names) or isinstance(n, ast.ImportFrom) and (n.module or "").split(".")[0] in LOADER_MODULES or isinstance(n, ast.Name) and n.id == "__import__" or isinstance(n, ast.ImportFrom) and n.module == "sys" and any(a.name in ("modules", "*") for a in n.names) - or isinstance(n, ast.Attribute) and n.attr == "modules" and isinstance(n.value, ast.Name) - and _sys(mods, mod, n.value.id)}) + or isinstance(n, ast.Attribute) and n.attr in ("modules", "__import__") + and (isinstance(n.ctx, ast.Load) or isinstance(n.value, ast.Name) and _sys(mods, mod, n.value.id)) + or isinstance(n, ast.ImportFrom) and any(a.name in LOADER_MODULES for a in n.names) + or isinstance(n, ast.Attribute) and n.attr in LOADER_MODULES + or isinstance(n, ast.Name) and n.id == "__builtins__" + or isinstance(n, ast.Subscript) and isinstance(n.ctx, ast.Load) and isinstance(n.slice, ast.Constant) + and n.slice.value in ("sys", "modules") + or isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load) and id(n) not in held and _sys(mods, mod, n.id) + or isinstance(n, ast.Attribute) and n.attr == "sys" + or isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "getattr" and len(n.args) > 1 + and isinstance(n.args[1], ast.Constant) and n.args[1].value in ("sys", "modules")}) + + +def _runs_text(mod: Module) -> list: + """The lines of `mod` that name exec, eval or compile where the module neither imports nor defines the name, + `from re import compile`, or that call exec or eval as an attribute: what code a string holds is not syntax this + check reads. A name only assigned, or a parameter, may still be the builtin, and is refused.""" + own = {name for name, bound in mod.bindings.items() if bound[:2] != ("attr", "builtins")} | set(mod.classes) \ + | {name for name in mod.functions if "." not in name} + return sorted({n.lineno for n in mod.nodes if isinstance(n, ast.Name) and n.id in TEXT_RUNNERS and n.id not in own + or isinstance(n, ast.Attribute) and n.attr in TEXT_ATTRIBUTES}) + + +def _given(fn, name: str) -> list: + """What the function `fn` gives `name` in its own scope, each as (value, whether it is read outside `fn`): a default, + read where `fn` is defined, or an assignment, a walrus, a comprehension's walrus, the part of a tuple assigned to a + tuple at its place, or a nested function's assignment to it as nonlocal, read in `fn` (issue 640). Read once for + each function and name in a run.""" + if (id(fn), name) in _READ: + return _READ[(id(fn), name)] + inner = {id(n) for child in ast.walk(fn) if child is not fn and isinstance(child, OWN_SCOPES) for n in ast.walk(child)} + args = fn.args.posonlyargs + fn.args.args + out = [(d, True) for a, d in zip(args[len(args) - len(fn.args.defaults):], fn.args.defaults) if a.arg == name] + out += [(d, True) for a, d in zip(fn.args.kwonlyargs, fn.args.kw_defaults) if d is not None and a.arg == name] + for node in ast.walk(fn): + if id(node) in inner: + continue + pairs = [(t, node.value) for t in node.targets] if isinstance(node, ast.Assign) else \ + [(node.target, node.value)] if isinstance(node, (ast.AnnAssign, ast.NamedExpr)) and node.value is not None else [] + while pairs: + target, value = pairs.pop() + if isinstance(target, ast.Name) and target.id == name: + out.append((value, False)) + elif isinstance(target, (ast.Tuple, ast.List)) and isinstance(value, (ast.Tuple, ast.List)) \ + and len(target.elts) == len(value.elts): + pairs += list(zip(target.elts, value.elts)) + for child in ast.walk(fn): # def inner(): nonlocal c; c = Port + if child is not fn and isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)) \ + and any(isinstance(n, ast.Nonlocal) and name in n.names for n in ast.walk(child)): + out += [(value, False) for value, outer in _given(child, name) if not outer] + _READ[(id(fn), name)] = out + return out + + +def _scoped(fn) -> frozenset: + """`_scope`, once for each function in a run, with what a walrus in a comprehension of its own binds, which is the + function's.""" + if id(fn) in _READ: + return _READ[id(fn)] + inner = {id(n) for child in ast.walk(fn) if child is not fn and isinstance(child, OWN_SCOPES) for n in ast.walk(child)} + _READ[id(fn)] = frozenset(_scope(fn) | {n.target.id for n in ast.walk(fn) if isinstance(n, ast.NamedExpr) and id(n) not in inner}) + return _READ[id(fn)] + + +def _traced(mods: dict, mod: Module, node, chain: list = (), seen: frozenset = frozenset()) -> list: + """What `node` may name, each as `_lookup` gives it, read back through the names it is given: a module's constant + that is another name, `P = Port`, and a name a function in `chain`, innermost first, is given, `cls=Port`, `c = cls` + or `a, b = Port, 1`. `vars(m)` is m. `type(x)`, `x.__class__`, `x.__mro__` and `x.__bases__`, and what holds them, + are ("object's class", ...) (issue 640).""" + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in ("type", "vars") and node.args: + return [("object's class", mod.name, "")] if node.func.id == "type" else _traced(mods, mod, node.args[0], chain, seen) + if isinstance(node, ast.Attribute) and node.attr in CLASS_PARTS: + return [("object's class", mod.name, "")] + if isinstance(node, ast.NamedExpr): # (c := Port).w + return _traced(mods, mod, node.value, chain, seen) + if isinstance(node, ast.Subscript): # type(x).__mro__[0] + return [got for got in _traced(mods, mod, node.value, chain, seen) if got[0] == "object's class"] + root = node + while isinstance(root, ast.Attribute): + root = root.value + local = next((at for at, fn in enumerate(chain) if isinstance(root, ast.Name) and root.id in _scoped(fn)), None) + if local is not None and isinstance(node, ast.Name): + fn = chain[local] + if (id(fn), node.id) in seen: + return [] + return [got for value, outer in _given(fn, node.id) + for got in _traced(mods, mod, value, chain[local + outer:], seen | {(id(fn), node.id)})] + if local is not None: # m.PORT, where m = other_module: what other_module's PORT is + found = [got for base in _traced(mods, mod, node.value, chain, seen) if base[0] == "module" + for got in [_lookup(mods, base[1], node.attr)] if got] + else: + found = [got for got in [_resolve(mods, mod, node) if isinstance(node, (ast.Name, ast.Attribute)) else None] if got] + out = list(found) + for got in found: # a constant that is another name: what that name is + if got[0] == "value" and got[2] in mods[got[1]].values and (got[1], got[2]) not in seen: + out += [more for value in mods[got[1]].values[got[2]] + for more in _traced(mods, mods[got[1]], value, (), seen | {(got[1], got[2])})] + name = node.id if isinstance(node, ast.Name) else node.attr + if got[0] == "value" and got[2] == "" and (got[1], name) not in seen: # under an if, a try or a walrus + out += [more for value in mods[got[1]].stored_as.get(name, ()) + for more in _traced(mods, mods[got[1]], value, (), seen | {(got[1], name)})] + if local is None and isinstance(node, ast.Name) and ("global", node.id) not in seen: # global C; C = Port + out += [more for fn, value in mod.declared.get(node.id, ()) + for more in _traced(mods, mod, value, [fn], seen | {("global", node.id)})] + return out -def _class_set(mods: dict, mod: Module) -> list: +def _class_set(mods: dict, mod: Module) -> tuple: """An attribute set by `setattr`, or set on a class from a function or from another module than the class's to - anything but text: what a caller reaches through the class cannot be read (issue 596).""" - inside = {id(n) for fn in mod.functions.values() for n in ast.walk(fn)} - # A name any function binds, its parameters and what it or a function in it assigns, may shadow a text constant. - bound = {} - for fn in mod.functions.values(): - names = {m.id for m in ast.walk(fn) if isinstance(m, ast.Name) and isinstance(m.ctx, ast.Store)} \ - | {a.arg for a in ast.walk(fn) if isinstance(a, ast.arg)} - bound.update((id(n), names) for n in ast.walk(fn)) - out = [] + anything but text, by an assignment, a `for` or a `with`, and the class named by any name for it; one set on an + object's class; and an attribute or a key set, or a container filled, from another module on what a module of the + repo's defines or is, by any name for it: what a caller reaches through it cannot be read (issues 596, 640). The + last are given apart, each as (what, problem), as PATCHERS may set them.""" + parents, names = {id(child): node for node in mod.nodes for child in ast.iter_child_nodes(node)}, {} + + def around(node) -> list: + """The functions round `node`, innermost first.""" + out, at = [], parents.get(id(node)) + while at is not None: + out += [at] if isinstance(at, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)) else [] + at = parents.get(id(at)) + return out + + def shadows(node) -> set: + """What the functions round `node` bind: their parameters, and what they or a function in them assign, define + or catch, any of which may shadow a text constant. One a function imports is bound two ways, and refused.""" + out = set() + for fn in around(node): + if id(fn) not in names: + walked = list(ast.walk(fn)) + names[id(fn)] = {m.id for m in walked if isinstance(m, ast.Name) and isinstance(m.ctx, ast.Store)} \ + | {m.arg for m in walked if isinstance(m, ast.arg)} \ + | {m.name for m in walked if m is not fn and isinstance(m, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))} \ + | {m.name for m in walked if isinstance(m, ast.ExceptHandler) and m.name} + out |= names[id(fn)] + return out + + out, patched = [], [] for node in mod.nodes: - if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == "setattr": - out.append("%s:%d sets an attribute by setattr, so what it reaches cannot be read; assign it where it is " + if isinstance(node, ast.Name) and node.id in SETTERS or isinstance(node, ast.Attribute) and node.attr in SETTERS: + out.append("%s:%d sets an attribute by setattr, or may, so what it reaches cannot be read; assign it where it is " "defined" % (mod.rel, node.lineno)) - if not isinstance(node, (ast.Assign, ast.AnnAssign, ast.AugAssign)) or node.value is None or _text(node.value) is not None \ - or isinstance(node.value, ast.Name) and node.value.id in mod.texts and node.value.id not in bound.get(id(node), ()): + for arg in node.args + [k.value for k in node.keywords] if isinstance(node, ast.Call) else (): + if isinstance(arg, ast.Call) and isinstance(arg.func, ast.Name) and arg.func.id == "vars" \ + or isinstance(arg, ast.Attribute) and arg.attr == "__dict__": # dict.update(vars(other), w=...) + patched += [("a key of %s" % got[1], "%s:%d hands on %s's names, which a call may set, so what they reach " + "cannot be read; name what it sets" % (mod.rel, arg.lineno, got[1])) + for got in _traced(mods, mod, arg, around(arg)) if got[0] == "module" and got[1] != mod.name] + if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in FILLS: + targets = [node.func] # other.TABLE.update(w=...): a key of TABLE set + elif isinstance(node, (ast.Assign, ast.AnnAssign, ast.AugAssign)) and node.value is not None: + if _text(node.value) is not None or isinstance(node.value, ast.Name) and node.value.id in mod.texts \ + and node.value.id not in shadows(node): + continue + targets = list(node.targets if isinstance(node, ast.Assign) else [node.target]) + elif isinstance(node, (ast.For, ast.AsyncFor, ast.comprehension)): # for Port.w in [...] + targets = [node.target] + elif isinstance(node, (ast.With, ast.AsyncWith)): # with x as Port.w + targets = [item.optional_vars for item in node.items if item.optional_vars is not None] + else: continue - targets = list(node.targets if isinstance(node, ast.Assign) else [node.target]) while any(isinstance(target, (ast.Tuple, ast.List, ast.Starred)) for target in targets): # a, Port.w = ... targets = [part for target in targets for part in (target.elts if isinstance(target, (ast.Tuple, ast.List)) else [target.value] if isinstance(target, ast.Starred) else [target])] for target in targets: - got = _resolve(mods, mod, target.value) if isinstance(target, ast.Attribute) else None - if got and got[0] == "class" and (got[1] != mod.name or id(target) in inside): + if not isinstance(target, (ast.Attribute, ast.Subscript)): + continue + chain, filled = around(target), target is not node.func if isinstance(node, ast.Call) else True + found, part, direct = [], target.value, True + while part is not None and not found: # TABLE['a']['b'] = w sets a key of TABLE + found = _traced(mods, mod, part, chain) + if not found: + part, direct = part.value if isinstance(part, (ast.Attribute, ast.Subscript)) else None, False + classes = [got for got in found if got[0] == "class"] if direct and filled and isinstance(target, ast.Attribute) else [] + others = [got for got in found if got[0] in ("value", "class", "def", "module") and got[1] != mod.name] + if any(got[0] == "object's class" for got in found): + out.append("%s:%d sets %s on an object's class, so what it reaches cannot be read; assign it in the class" + % (mod.rel, target.lineno, target.attr if isinstance(target, ast.Attribute) else "a key")) + elif classes and (classes[0][1] != mod.name or chain): out.append("%s:%d sets %s on the class %s from outside its module's top level, so what it reaches " - "cannot be read; assign it in the class" % (mod.rel, node.lineno, target.attr, got[2])) - return out + "cannot be read; assign it in the class" % (mod.rel, target.lineno, target.attr, classes[0][2])) + elif others: + owner = ".".join(p for p in others[0][1:] if p) + what = "a key of %s" % owner if not filled or direct and isinstance(target, ast.Subscript) else \ + "%s.%s" % (owner, target.attr) if direct else "an attribute or a key of %s" % owner + patched.append((what, "%s:%d sets %s from another module, so what it reaches cannot be read; set it in its " + "own module" % (mod.rel, target.lineno, what))) + return out, patched def _commands(mod: Module) -> list: @@ -541,8 +733,9 @@ def _commands(mod: Module) -> list: docs = {id(n.body[0].value) for n in mod.nodes if isinstance(n, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)) and n.body and isinstance(n.body[0], ast.Expr) and isinstance(n.body[0].value, ast.Constant)} - return sorted({n.lineno for n in mod.nodes if isinstance(n, ast.Constant) and isinstance(n.value, str) - and COMMANDS.match(n.value) and id(n) not in docs}) + return sorted({n.lineno for n in mod.nodes if isinstance(n, ast.Constant) and isinstance(n.value, (str, bytes)) + and COMMANDS.search(n.value.decode("latin-1") if isinstance(n.value, bytes) else n.value) + and id(n) not in docs}) def _verbs(mod: Module) -> list: @@ -577,6 +770,7 @@ def _adapter(mod: Module) -> bool: def run(root: pathlib.Path = ROOT) -> list: + _READ.clear() problems, mods = [], {} sources = [p.relative_to(root).as_posix() for p in walk.files(root) if p.suffix == ".py"] sources = [rel for rel in sources if not rel.startswith("tests/")] @@ -590,7 +784,7 @@ def run(root: pathlib.Path = ROOT) -> list: mods[_name(rel)] = Module(rel, tree, known) reading = Reading(mods) problems += reading.problems - writers, commanding = set(), set() + writers, commanding, patching = set(), set(), set() for mod in mods.values(): if mod.name != GITHUB: problems += _by_hand(mod) + _edited(mod) @@ -601,7 +795,10 @@ def run(root: pathlib.Path = ROOT) -> list: if mod.rel not in NAME_LOADERS: problems += ["%s:%d loads a module by its name, or may, so what it reaches cannot be read; import it" % (mod.rel, line) for line in _loaded(mods, mod)] - problems += _class_set(mods, mod) + problems += ["%s:%d runs code from text, so what it reaches cannot be read" % (mod.rel, line) for line in _runs_text(mod)] + sets, patched = _class_set(mods, mod) + problems += sets + [problem for what, problem in patched if what not in PATCHERS.get(mod.rel, ())] + patching |= {(mod.rel, what) for what, _ in patched} if _commands(mod): commanding.add(mod.rel) if mod.rel not in RUNNERS: @@ -649,4 +846,6 @@ def run(root: pathlib.Path = ROOT) -> list: (node_mod.rel, ", ".join(sorted(found)) or "no write", ", ".join(sorted(listed)))) problems += ["CALLERS names %s, which is not here; take it off" % rel for rel in sorted(CALLERS) if _name(rel) not in mods] problems += ["RUNNERS names %s, which holds no command; take it off" % rel for rel in RUNNERS if rel not in commanding] + problems += ["PATCHERS names %s for %s, which it does not set; take it off" % (rel, what) for rel, names in + sorted(PATCHERS.items()) for what in sorted(names) if (rel, what) not in patching] return problems diff --git a/tools/hooks/identity.py b/tools/hooks/identity.py index c953cf09..0972300c 100644 --- a/tools/hooks/identity.py +++ b/tools/hooks/identity.py @@ -1433,6 +1433,19 @@ def _quoted(text: str) -> str: out.append("$(%s)" % _spelt(text[i + 2:end], room).replace("\\", "\\\\").replace('"', '\\"')) i = end + 1 continue + elif quote == '"' and c == "`": + # A backtick in double quotes ends at its next backtick no backslash holds, and its quotes are its own: it is + # spelt as a $(...) is, its \", \`, \$ and \\ the character, as the shell takes them out (#605 (g)). One that + # never closes, or whose quote or comment would run past the ) it is given, is read as main reads it (Argon). + end = shell._tick_end(text, i) + inner = re.sub(r'\\([\\`$"])', r"\1", text[i + 1:end]) + if end < len(text) and shell._left_open(inner) == (False, False): + out.append("$(%s)" % _spelt(inner, room).replace("\\", "\\\\").replace('"', '\\"')) + i = end + 1 + continue + out.append(c) + i += 1 + continue elif quote == '"': quote = None if c == '"' else quote elif text.startswith("$'", i): # $'...', in which a backslash escapes, \' among them diff --git a/tools/hooks/protected.py b/tools/hooks/protected.py index d62b8130..da172ed0 100644 --- a/tools/hooks/protected.py +++ b/tools/hooks/protected.py @@ -56,6 +56,8 @@ DEADLINE = 5.0 # seconds to read one tool call; the settings give the hook 30 LONGEST = 256 * 1024 # characters of a command the hook reads +RESPELT_MOST = 8 * 1024 # characters of a command read again, spelt (#605 (g)): past it, a nest of quoted + # substitutions is read in more than linear time (the security pass) LONGEST_PATH = 4096 # characters of a path NAME = 255 # characters of one name in a path MATCHES = 2000 # paths one wildcard may match @@ -515,6 +517,21 @@ def read(self) -> list: self.ask("command_unreadable", "the command cannot be split into words") return self.found self.sessions(commands) + try: + again = _respelt(self.text, self.spend) if len(self.text) <= RESPELT_MOST else [] + except (ValueError, RecursionError): + again = [] # _split's reading stands, and asks about what it cannot read + start = self.here + self.asks_of(commands) + if again and again != commands: + # Read again from where the command began, so a cd in the first reading moves none in the second; what + # either asks about is asked (check gives each ask once). + self.here = start + self.asks_of(again) + return self.found + + def asks_of(self, commands: list): + """Ask about what each of `commands` writes, from where the command stands.""" stood = [] # where the command stood when each subshell a redirection's operand runs began (issue 376) for at, (words, redirections) in enumerate(commands): self.spend() @@ -531,7 +548,6 @@ def read(self) -> list: if words: self.github(words, redirections) self.simple(words, redirections) - return self.found def redirections(self, redirections: list): for operator, target in redirections: @@ -1400,6 +1416,20 @@ def _split(text: str, spend) -> list: return commands +def _respelt(text: str, spend) -> list: + """The simple commands of a command read again with each substitution spelt as the identity hook spells one, its + quotes its own (identity._quoted), so a "$(...)" in a "$(...)" ends no quote there and what follows it is read as + commands (#605 (g)). A script it spells nothing new in, or gives up on, is left to _split's reading.""" + outer, inner = _process_substitutions(text) + commands = [] + for script in [outer] + inner: + raw = shell.without_heredocs(script, raw=True) + spelt = identity._quoted(raw) + if spelt != raw: + commands += shell.commands(spelt, spend, places=True) + return commands + + def _strings(value) -> list: """Every string a JSON value holds, its keys among them.""" if isinstance(value, dict): diff --git a/tools/hooks/shell.py b/tools/hooks/shell.py index d273ab45..f7748258 100644 --- a/tools/hooks/shell.py +++ b/tools/hooks/shell.py @@ -174,6 +174,10 @@ def _nested(script: str, room: list, comments: bool = True) -> str: inner.append(script[end + 1] if escaped else script[end]) end += 2 if escaped else 1 end, inner = end + 1, "".join(inner) + if _left_open(inner)[0]: + # The shell ends a backtick before it reads its quotes, fails this one and runs what follows: a + # script that would be data in a body is one the hook cannot read (#605 (d)). + raise Unread("a quote in a backtick in a substitution runs past where the backtick ends") else: end = _balanced(script, i + 2, ")") inner = script[i + 2:end - 1] if script[end - 1:end] == ")" else script[i + 2:] @@ -208,6 +212,30 @@ def _nested(script: str, room: list, comments: bool = True) -> str: return "".join(out) +def _left_open(script: str) -> tuple: + """(whether a quote that opens in `script`, a backtick's, is still open at its end; whether a comment begins in it). + A $(...) in it has quotes and comments of its own (Argon, Astra), and a # after a character a backslash holds + begins a word (Astra).""" + quote, i, escaped, commented = None, 0, -1, False + while i < len(script): + c = script[i] + if c == "\\" and quote != "'": + i = escaped = i + 2 + continue + if quote != "'" and script.startswith("$(", i): + i = _balanced(script, i + 2, ")") + continue + if quote is None and c == "#" and i != escaped and (i == 0 or script[i - 1] in COMMENTS_AFTER): + commented = True + i = len(script) if script.find("\n", i) < 0 else script.find("\n", i) + elif quote is None and c in "'\"": + quote = c + elif c == quote: + quote = None + i += 1 + return quote is not None, commented + + def _terminator(word: str, first: str) -> str: """The line that ends a here-document: its whole word after the <<, as the shell reads it once its quotes and backslashes are taken out, E\\OF as EOF (#410). Only a plain word, as the << pattern spells one, is taken: the line is diff --git a/tools/mutants.py b/tools/mutants.py index d4664436..c028c80a 100644 --- a/tools/mutants.py +++ b/tools/mutants.py @@ -144,6 +144,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: DATA_TESTS = (HOOK_TESTS + "test_a_word_given_to_any_other_program_is_data",) IDENTITY_TESTS = "tests.test_hooks.Identity." PROTECTED_NEST = "tests.test_hooks.MergesAndSettings.test_a_substitution_in_double_quotes_has_quotes_of_its_own_at_every_level" +PROTECTED_NESTED_IN_ONE = ("tests.test_hooks.MergesAndSettings." + "test_a_double_quoted_substitution_in_a_double_quoted_one_hides_no_command_after_it") NO_REFSPEC_TESTS = (IDENTITY_TESTS + "test_a_push_with_no_refspec_reads_what_git_would_push",) READINGS_TESTS = (IDENTITY_TESTS + "test_the_readings_sonarcloud_flagged_read_what_their_regexes_read",) PUSH_SETTING_TESTS = (IDENTITY_TESTS + "test_push_settings_given_in_the_command_are_refused",) @@ -2924,6 +2926,14 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ("a mint whose transfer failed is trusted", "tools/trial_reader_reads.sh", '{ [ -z "$token" ] || [ "$minted_exit" != 0 ]; }', '[ -z "$token" ]', ("tests.test_tools.ReaderTrialScript.test_a_mint_answered_whole_by_a_transfer_that_failed_is_revoked_and_told",)), + # Issue 424 (Argon): the reader's 201 path, as P-39's, P-8's and P-52's are held. + ("the reader's trial reads a 201 it cannot read, by a transfer that ended well, as a token minted", "tools/trial_reader_reads.sh", + '{ [ -z "$token" ] || [ "$minted_exit" != 0 ]; }', '[ "$minted_exit" != 0 ]', + ("tests.test_tools.ReachScript.test_every_script_that_mints_writes_nothing_of_a_mint_it_cannot_read_or_that_is_refused",)), + ("the reader's trial leaves a token found in a mint's answer it cannot read unrevoked", "tools/trial_reader_reads.sh", + ' token=$(token_in "$minted")\n minted=""\n if [ -n "$token" ]; then\n', + ' token=""\n minted=""\n if [ -n "$token" ]; then\n', + ("tests.test_tools.ReaderTrialScript.test_a_mint_answered_201_and_cut_short_is_revoked_and_told",)), ("the reader's raw answers may be kept inside the repo", "tools/trial_reader_reads.sh", 'case "$scratch/" in "$root"/*)', 'case "$scratch/" in "/nowhere"/*)', ("tests.test_tools.ReaderTrialScript.test_a_scratch_directory_inside_the_repo_is_refused_before_anything_is_minted",)), @@ -11670,28 +11680,27 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' if _package(mods, whole):\n return ("module", whole, "")\n', '', ('tests.test_tools.Writes.test_a_namespace_package_is_read_as_one',)), ("sys's modules are read", 'tools/check_writes.py', - ' or isinstance(n, ast.Attribute) and n.attr == "modules" and isinstance(n.value, ast.Name)\n', - ' or False and n.attr == "modules" and isinstance(n.value, ast.Name)\n', + 'n.attr in ("modules", "__import__")', 'n.attr in ("__import__",)', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ("sys's modules imported by name are read", 'tools/check_writes.py', 'any(a.name in ("modules", "*") for a in n.names)', 'any(a.name in ("path", "*") for a in n.names)', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), - ("setattr may set an attribute", 'tools/check_writes.py', - 'isinstance(node.func, ast.Name) and node.func.id == "setattr":', 'isinstance(node.func, ast.Name) and node.func.id == "set_attr":', + ('setattr may set an attribute', 'tools/check_writes.py', + 'SETTERS = ("setattr", "setitem", "__setattr__", "__setitem__")', 'SETTERS = ("setitem", "__setattr__", "__setitem__")', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ("a class's attribute may be set from another module", 'tools/check_writes.py', - 'if got and got[0] == "class" and (got[1] != mod.name or id(target) in inside):', 'if got and got[0] == "class" and id(target) in inside:', + 'elif classes and (classes[0][1] != mod.name or chain):', 'elif classes and chain:', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ("a class's attribute may be set from a function", 'tools/check_writes.py', - 'if got and got[0] == "class" and (got[1] != mod.name or id(target) in inside):', 'if got and got[0] == "class" and got[1] != mod.name:', + 'elif classes and (classes[0][1] != mod.name or chain):', 'elif classes and classes[0][1] != mod.name:', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), - ("a text constant set on a class from a function is refused", 'tools/check_writes.py', - ' or isinstance(node.value, ast.Name) and node.value.id in mod.texts and', ' or False and', + ('a text constant set on a class from a function is refused', 'tools/check_writes.py', + 'or isinstance(node.value, ast.Name) and node.value.id in mod.texts \\\n', 'or False and node.value.id in mod.texts \\\n', ('tests.test_tools.Writes.test_on_main_only_the_callers_listed_reach_a_write_and_each_what_it_lists',)), ("gh or git may be run from Python", 'tools/check_writes.py', ' if mod.rel not in RUNNERS:\n', ' if False:\n', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), - ("a command line is read only as its program alone", 'tools/check_writes.py', - 'COMMANDS = re.compile(r"\\s*(\\S*/)?(g" r"h|gi" r"t)(\\s|$)")', 'COMMANDS = re.compile(r"\\s*(\\S*/)?(g" r"h|gi" r"t)$")', + ('a command line is read only as its program alone', 'tools/check_writes.py', + '%s%s(\\s|$|[;&|)}])', '%s%s($|[;&|)}])', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ("a namespace package's other name is read as outside the repo", 'tools/check_writes.py', ' return ("unknown", module, name) if _package(mods, module) else None\n', ' return None\n', @@ -11702,18 +11711,290 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ("an import of all of sys is let through", 'tools/check_writes.py', 'any(a.name in ("modules", "*") for a in n.names)', 'any(a.name in ("modules",) for a in n.names)', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), - ("a shadowed text constant is taken as text", 'tools/check_writes.py', - ' and node.value.id not in bound.get(id(node), ()):\n', ':\n', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a shadowed text constant is taken as text', 'tools/check_writes.py', + ' and node.value.id not in shadows(node):\n', ':\n', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ("an unpacked target is not read", 'tools/check_writes.py', ' while any(isinstance(target, (ast.Tuple, ast.List, ast.Starred)) for target in targets):', ' while False:', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), - ("a command named by its path is no command", 'tools/check_writes.py', - 'COMMANDS = re.compile(r"\\s*(\\S*/)?(g"', 'COMMANDS = re.compile(r"\\s*(g"', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command named by its path is no command', 'tools/check_writes.py', + '_NAME = r"(\\S*[/\\\\])?(g"', '_NAME = r"(g"', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ("a RUNNERS entry that holds no command is kept", 'tools/check_writes.py', 'for rel in RUNNERS if rel not in commanding]', 'for rel in RUNNERS if False]', ('tests.test_tools.Writes.test_a_listed_caller_that_reaches_a_new_writer_or_none_fails',)), ("a runner's own gh or git is refused", 'tools/check_writes.py', ' if mod.rel not in RUNNERS:\n', ' if True:\n', ('tests.test_tools.Writes.test_on_main_only_the_callers_listed_reach_a_write_and_each_what_it_lists',)), + # Issue 640: a name read back through what it is given, an object's class, setattr by another name, a for or + # with target, a shadow by a nested def, class or catch, a setter under an if, another module's name set or + # filled, sys and loads reached other ways, code run from text, and a command a shell runs. + ("a namespace package's import * is read as nothing", 'tools/check_writes.py', + 'for line, source in mod.stars if _package(self.mods, source)]', 'for line, source in mod.stars if source in self.mods]', + ('tests.test_tools.Writes.test_a_namespace_package_is_read_as_one',)), + ('a command is read in one case only', 'tools/check_writes.py', + 'COMMANDS = re.compile(r"(?i)%s', 'COMMANDS = re.compile(r"%s', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command named by a Windows path is no command', 'tools/check_writes.py', + '_NAME = r"(\\S*[/\\\\])?(g"', '_NAME = r"(\\S*/)?(g"', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a command's .exe hides it", 'tools/check_writes.py', + 'r"t)(\\.exe)?"', 'r"t)"', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a shell's later command is no command", 'tools/check_writes.py', + '(r"(^\\s*|[;&|({\\n!]\\s*|\\b(env', '(r"(^\\s*|\\b(env', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command after a new line is no command', 'tools/check_writes.py', + '[;&|({\\n!]', '[;&|({!]', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command in braces is no command', 'tools/check_writes.py', + '[;&|({\\n!]', '[;&|(\\n!]', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command another word runs is no command', 'tools/check_writes.py', + '|\\b(env|exec|command|nohup|xargs|sudo|time|nice|if|elif|while|until|then|do|else)"\n r"(\\s+-\\S+(\\s+[^-\\s]\\S*)?)*\\s+|', '|"\n r"', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a command after env's options is no command", 'tools/check_writes.py', + 'r"(\\s+-\\S+(\\s+[^-\\s]\\S*)?)*\\s+|', 'r"\\s+|', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command exec runs is no command', 'tools/check_writes.py', + '(env|exec|', '(env|', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command after then is no command', 'tools/check_writes.py', + '|then|do|else)', '|do|else)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a command after a variable's assignment is no command", 'tools/check_writes.py', + '|\\b\\w+=\\S*\\s+)"', ')"', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a command ended by a shell's mark is no command", 'tools/check_writes.py', + '(\\s|$|[;&|)}])', '(\\s|$)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a quoted command is no command', 'tools/check_writes.py', + '% (_LEAD, _NAME, _NAME))', '% (_LEAD, _NAME, "(?!)"))', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a quoted command line is no command', 'tools/check_writes.py', + '[\\"\']\\s*%s(\\s|[\\"\';&|)}]|$)', '[\\"\']%s[\\"\']', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('bytes are not read for a command', 'tools/check_writes.py', + 'isinstance(n.value, (str, bytes))', 'isinstance(n.value, str)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a command is read only at the text's start", 'tools/check_writes.py', + 'COMMANDS.search(', 'COMMANDS.match(', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('sys named as a value is let through', 'tools/check_writes.py', + 'or isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load) and id(n) not in held and _sys(mods, mod, n.id)', 'or False and isinstance(n.ctx, ast.Load) and id(n) not in held and _sys(mods, mod, n.id)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("sys's namespace is read as an attribute of it", 'tools/check_writes.py', + 'isinstance(n, ast.Attribute) and n.attr not in NAMESPACES}', 'isinstance(n, ast.Attribute)}', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('sys through another module is let through', 'tools/check_writes.py', + ' or isinstance(n, ast.Attribute) and n.attr == "sys"\n', ' or False\n', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('sys imported from another module is not sys', 'tools/check_writes.py', + 'if bound == ("module", "sys") or bound and bound[0] == "attr" and bound[2] == "sys" and bound[1] not in mods:', 'if bound == ("module", "sys"):', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('sys by a getattr of another module is let through', 'tools/check_writes.py', + 'n.args[1].value in ("sys", "modules")', 'n.args[1].value in ("modules",)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('sys by a literal key is let through', 'tools/check_writes.py', + 'and n.slice.value in ("sys", "modules")', 'and False', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('__builtins__ is let through', 'tools/check_writes.py', + 'or isinstance(n, ast.Name) and n.id == "__builtins__"', 'or False', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('__import__ as an attribute is let through', 'tools/check_writes.py', + 'n.attr in ("modules", "__import__")', 'n.attr in ("modules",)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('pkgutil may load a module by name', 'tools/check_writes.py', + 'LOADER_MODULES = ("importlib", "builtins", "pkgutil", "runpy")', 'LOADER_MODULES = ("importlib", "builtins", "runpy")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('runpy may run a module by name', 'tools/check_writes.py', + 'LOADER_MODULES = ("importlib", "builtins", "pkgutil", "runpy")', 'LOADER_MODULES = ("importlib", "builtins", "pkgutil")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('code run from text is let through', 'tools/check_writes.py', + ' problems += ["%s:%d runs code from text, so what it reaches cannot be read" % (mod.rel, line) for line in _runs_text(mod)]\n', '', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('exec is let through', 'tools/check_writes.py', + 'TEXT_RUNNERS = ("exec", "eval", "compile")', 'TEXT_RUNNERS = ("eval", "compile")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('eval is let through', 'tools/check_writes.py', + 'TEXT_RUNNERS = ("exec", "eval", "compile")', 'TEXT_RUNNERS = ("exec", "compile")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('compile is let through', 'tools/check_writes.py', + 'TEXT_RUNNERS = ("exec", "eval", "compile")', 'TEXT_RUNNERS = ("exec", "eval")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a name the module binds is read as exec', 'tools/check_writes.py', + 'and n.id in TEXT_RUNNERS and n.id not in own', 'and n.id in TEXT_RUNNERS', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('exec as an attribute is let through', 'tools/check_writes.py', + '\n or isinstance(n, ast.Attribute) and n.attr in TEXT_ATTRIBUTES})', '})', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a class by a constant's other name is not read", 'tools/check_writes.py', + 'if got[0] == "value" and got[2] in mods[got[1]].values and (got[1], got[2]) not in seen:', 'if False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a function's default is not given", 'tools/check_writes.py', + 'out = [(d, True) for a, d in zip(args[len(args) - len(fn.args.defaults):], fn.args.defaults) if a.arg == name]', 'out = []', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a keyword's default is not given", 'tools/check_writes.py', + 'out += [(d, True) for a, d in zip(fn.args.kwonlyargs, fn.args.kw_defaults) if d is not None and a.arg == name]', 'out += []', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('an assignment is not given', 'tools/check_writes.py', + 'pairs = [(t, node.value) for t in node.targets] if isinstance(node, ast.Assign) else', 'pairs = [] if isinstance(node, ast.Assign) else', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a walrus is not given', 'tools/check_writes.py', + 'isinstance(node, (ast.AnnAssign, ast.NamedExpr))', 'isinstance(node, ast.AnnAssign)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a tuple's part is not given", 'tools/check_writes.py', + 'pairs += list(zip(target.elts, value.elts))', 'pass', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a nested scope's assignment is given", 'tools/check_writes.py', + ' if id(node) in inner:\n continue\n', '', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("an outer function's name is not read", 'tools/check_writes.py', + 'local = next((at for at, fn in enumerate(chain) if', 'local = next((at for at, fn in enumerate(chain[:1]) if', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a local name is read as the module's", 'tools/check_writes.py', + 'local = next((at for at, fn in enumerate(chain) if isinstance(root, ast.Name) and root.id in _scoped(fn)), None)', 'local = None', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('an attribute of a local module is not read', 'tools/check_writes.py', + 'if base[0] == "module"\n', 'if False\n', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("type(x) is no object's class", 'tools/check_writes.py', + 'node.func.id in ("type", "vars")', 'node.func.id in ("vars",)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('vars(m) is not m', 'tools/check_writes.py', + 'node.func.id in ("type", "vars")', 'node.func.id in ("type",)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("__class__ is no object's class", 'tools/check_writes.py', + 'CLASS_PARTS = ("__class__", "__mro__", "__bases__", "__base__")', 'CLASS_PARTS = ("__mro__", "__bases__", "__base__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("__mro__ is no object's class", 'tools/check_writes.py', + 'CLASS_PARTS = ("__class__", "__mro__", "__bases__", "__base__")', 'CLASS_PARTS = ("__class__", "__bases__", "__base__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("__bases__ is no object's class", 'tools/check_writes.py', + 'CLASS_PARTS = ("__class__", "__mro__", "__bases__", "__base__")', 'CLASS_PARTS = ("__class__", "__mro__", "__base__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("what an object's class holds is not read", 'tools/check_writes.py', + ' return [got for got in _traced(mods, mod, node.value, chain, seen) if got[0] == "object\'s class"]\n', ' return []\n', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("setattr's dunder is let through", 'tools/check_writes.py', + 'SETTERS = ("setattr", "setitem", "__setattr__", "__setitem__")', 'SETTERS = ("setattr", "setitem", "__setitem__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a container filled by a call is not read', 'tools/check_writes.py', + 'if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in FILLS:', 'if False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a for target is not read', 'tools/check_writes.py', + 'elif isinstance(node, (ast.For, ast.AsyncFor, ast.comprehension)):', 'elif isinstance(node, (ast.AsyncFor, ast.comprehension)):', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a comprehension's target is not read", 'tools/check_writes.py', + 'elif isinstance(node, (ast.For, ast.AsyncFor, ast.comprehension)):', 'elif isinstance(node, (ast.For, ast.AsyncFor)):', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a with target is not read', 'tools/check_writes.py', + 'elif isinstance(node, (ast.With, ast.AsyncWith)):', 'elif False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a nested class does not shadow', 'tools/check_writes.py', + 'isinstance(m, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))}', 'isinstance(m, (ast.FunctionDef, ast.AsyncFunctionDef))}', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a nested def does not shadow', 'tools/check_writes.py', + 'isinstance(m, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))}', 'isinstance(m, (ast.ClassDef,))}', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a catch does not shadow', 'tools/check_writes.py', + 'isinstance(m, ast.ExceptHandler) and m.name}', 'isinstance(m, ast.ExceptHandler) and False}', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a setter under an if is read as the top level', 'tools/check_writes.py', + 'out += [at] if isinstance(at, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)) else []', 'out += [at] if isinstance(at, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)) and at in mod.functions.values() else []', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("an object's class may be set", 'tools/check_writes.py', + 'if any(got[0] == "object\'s class" for got in found):', 'if False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('an instance, a dict or a module of another module may be set', 'tools/check_writes.py', + ' elif others:\n', ' elif False:\n', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a module's own name set is refused", 'tools/check_writes.py', + 'others = [got for got in found if got[0] in ("value", "class", "def", "module") and got[1] != mod.name]', 'others = [got for got in found if got[0] in ("value", "class", "def", "module")]', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a key two deep is not followed to its owner', 'tools/check_writes.py', + 'part, direct = part.value if isinstance(part, (ast.Attribute, ast.Subscript)) else None, False', 'part, direct = None, False', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a key set is named as an attribute', 'tools/check_writes.py', + 'if not filled or direct and isinstance(target, ast.Subscript) else', 'if not filled else', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a name of another module's may be set by any module", 'tools/check_writes.py', + 'problems += sets + [problem for what, problem in patched if what not in PATCHERS.get(mod.rel, ())]', 'problems += sets', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("PATCHERS' sets are refused", 'tools/check_writes.py', + 'problems += sets + [problem for what, problem in patched if what not in PATCHERS.get(mod.rel, ())]', 'problems += sets + [problem for what, problem in patched]', + ('tests.test_tools.Writes.test_on_main_only_the_callers_listed_reach_a_write_and_each_what_it_lists',)), + ('a patcher may set what PATCHERS does not name', 'tools/check_writes.py', + 'if what not in PATCHERS.get(mod.rel, ())]', 'if mod.rel not in PATCHERS]', + ('tests.test_tools.Writes.test_a_listed_caller_that_reaches_a_new_writer_or_none_fails',)), + ('a PATCHERS entry that sets nothing is kept', 'tools/check_writes.py', + 'if (rel, what) not in patching]', 'if False]', + ('tests.test_tools.Writes.test_a_listed_caller_that_reaches_a_new_writer_or_none_fails',)), + ("__base__ is no object's class", 'tools/check_writes.py', + 'CLASS_PARTS = ("__class__", "__mro__", "__bases__", "__base__")', 'CLASS_PARTS = ("__class__", "__mro__", "__bases__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command after if is no command', 'tools/check_writes.py', + '|if|elif|while|until|then', '|then', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("an option's value hides a command", 'tools/check_writes.py', + '(\\s+-\\S+(\\s+[^-\\s]\\S*)?)*', '(\\s+-\\S+)*', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a quoted command line ended by a shell's mark is no command", 'tools/check_writes.py', + '(\\s|[\\"\';&|)}]|$)', '(\\s|[\\"\']|$)', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a command after ! is no command', 'tools/check_writes.py', + '[;&|({\\n!]', '[;&|({\\n]', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a parameter named exec is the module's own", 'tools/check_writes.py', + ' own = {name for name, bound in mod.bindings.items()', ' own = {n.arg for n in mod.nodes if isinstance(n, ast.arg)} | {name for name, bound in mod.bindings.items()', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a default is read inside its function', 'tools/check_writes.py', + 'chain[local + outer:]', 'chain[local:]', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a walrus set on is not read', 'tools/check_writes.py', + ' if isinstance(node, ast.NamedExpr): # (c := Port).w\n return _traced(mods, mod, node.value, chain, seen)\n', '', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a local module's constant is not followed", 'tools/check_writes.py', + ' for got in found: # a constant that is another name', ' for got in (found if local is None else []): # a constant that is another name', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('setitem may set a key', 'tools/check_writes.py', + 'SETTERS = ("setattr", "setitem", "__setattr__", "__setitem__")', 'SETTERS = ("setattr", "__setattr__", "__setitem__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('__setitem__ may set a key', 'tools/check_writes.py', + 'SETTERS = ("setattr", "setitem", "__setattr__", "__setitem__")', 'SETTERS = ("setattr", "setitem", "__setattr__")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a name declared global is not read', 'tools/check_writes.py', + 'if local is None and isinstance(node, ast.Name) and ("global", node.id) not in seen:', 'if False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a nested function's nonlocal is not given", 'tools/check_writes.py', + 'and any(isinstance(n, ast.Nonlocal) and name in n.names for n in ast.walk(child)):', 'and False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a module's name stored some other way is not read back", 'tools/check_writes.py', + 'if got[0] == "value" and got[2] == "" and (got[1], name) not in seen:', 'if False:', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a walrus in a comprehension is not given', 'tools/check_writes.py', + 'isinstance(child, OWN_SCOPES) for n in ast.walk(child)}\n args =', 'isinstance(child, SCOPES) for n in ast.walk(child)}\n args =', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("a walrus in a comprehension is not the function's", 'tools/check_writes.py', + ' | {n.target.id for n in ast.walk(fn) if isinstance(n, ast.NamedExpr) and id(n) not in inner})', ')', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a loader re-exported is let through', 'tools/check_writes.py', + 'or isinstance(n, ast.ImportFrom) and any(a.name in LOADER_MODULES for a in n.names)', 'or False', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a loader as an attribute is let through', 'tools/check_writes.py', + 'or isinstance(n, ast.Attribute) and n.attr in LOADER_MODULES', 'or False', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('a store of .modules is refused', 'tools/check_writes.py', + 'and (isinstance(n.ctx, ast.Load) or', 'and (True or', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("another module's names handed to a call are let through", 'tools/check_writes.py', + 'for arg in node.args + [k.value for k in node.keywords] if isinstance(node, ast.Call) else ():', 'for arg in ():', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ('timeout hides a command', 'tools/check_writes.py', + '|\\btimeout(\\s+-\\S+)*\\s+\\S+\\s+', '', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + ("sys's modules may be set", 'tools/check_writes.py', + ' or isinstance(n.value, ast.Name) and _sys(mods, mod, n.value.id))', ')', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), # Issue 587: outside adapters/, only the callers tools/check_writes.py lists reach a GitHub write. ('a caller no list names may reach a write', 'tools/check_writes.py', 'if found and listed is None:', 'if False:', ('tests.test_tools.Writes.test_a_new_caller_of_a_writer_fails_until_it_is_listed',)), @@ -11736,7 +12017,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ('a name no module defines is read as nothing', 'tools/check_writes.py', ' else:\n self.problems.append("%s:%d names %s, which', ' elif False:\n self.problems.append("%s:%d names %s, which', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ('an import * is read as nothing', 'tools/check_writes.py', - 'for line, source in mod.stars if source in self.mods]', 'for line, source in mod.stars if False]', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + 'for line, source in mod.stars if _package(self.mods, source)]', 'for line, source in mod.stars if False]', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ('a relative import * is not read', 'tools/check_writes.py', ' if alias.name == "*":\n self.stars.append((node.lineno, source))\n', '', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ('a module handed on as a value is read as nothing', 'tools/check_writes.py', @@ -11850,7 +12132,8 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ("a request's alias is not followed", 'tools/check_writes.py', ' grown, names = not more <= names, names | more\n', ' grown = False\n', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ('builtins may load a module by name', 'tools/check_writes.py', - 'LOADER_MODULES = ("importlib", "builtins")', 'LOADER_MODULES = ("importlib",)', ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), + 'LOADER_MODULES = ("importlib", "builtins", "pkgutil", "runpy")', 'LOADER_MODULES = ("importlib", "pkgutil", "runpy")', + ('tests.test_tools.Writes.test_what_cannot_be_read_fails',)), ('super() skips a base of the same name in another module', 'tools/check_writes.py', 'and not (above and m[0] == mod.name and m[1].startswith(', 'and not (above and m[1].startswith(', ('tests.test_tools.Writes.test_a_writer_is_reached_by_any_name_it_goes_by',)), ('a name a function imports is read as its parameter', 'tools/check_writes.py', @@ -14595,6 +14878,51 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ' if name not in seen:\n', ' if True:\n', (IDENTITY_TESTS + 'test_whether_a_command_defines_a_printers_name_is_searched_once_a_name',)), ("each command that prints searches the whole text again", 'tools/hooks/identity.py', '_defined(program, text, defines)', '_defined(program, text)', (IDENTITY_TESTS + 'test_whether_a_command_defines_a_printers_name_is_searched_once_a_name',)), + # Issue 605, the rest: a double-quoted substitution in one, as the protected hook reads it; a backtick in double + # quotes; and a quote past a backtick in a here-document's substitution. + ("the protected hook reads a command once, as it stands", 'tools/hooks/protected.py', + ' if again and again != commands:', ' if False:', (PROTECTED_NESTED_IN_ONE,)), + ("the protected hook's second reading begins where the first ended", 'tools/hooks/protected.py', + ' self.here = start\n', '', (PROTECTED_NESTED_IN_ONE,)), + ("a spelt reading that fails is the protected hook's failure", 'tools/hooks/protected.py', + " again = [] # _split's reading stands", ' raise #', (PROTECTED_NESTED_IN_ONE,)), + ("a command of any length is read again, spelt", 'tools/hooks/protected.py', + ' if len(self.text) <= RESPELT_MOST else []', '', (PROTECTED_NESTED_IN_ONE,)), + ("a process substitution is not spelt again", 'tools/hooks/protected.py', + ' for script in [outer] + inner:\n raw =', ' for script in [outer]:\n raw =', (PROTECTED_NESTED_IN_ONE,)), + ("a backtick in double quotes is read flat", 'tools/hooks/identity.py', + ''' elif quote == '"' and c == "`":''', ' elif False:', + (IDENTITY_TESTS + 'test_a_backtick_in_double_quotes_has_quotes_of_its_own', PROTECTED_NESTED_IN_ONE)), + ("a backtick's escaped double quote in double quotes keeps its backslash", 'tools/hooks/identity.py', + '''inner = re.sub(r'\\\\([\\\\`$"])', r"\\1", text[i + 1:end])''', '''inner = re.sub(r'\\\\([\\\\`$])', r"\\1", text[i + 1:end])''', + (IDENTITY_TESTS + 'test_a_backtick_in_double_quotes_has_quotes_of_its_own',)), + ("a backtick's escaped dollar in double quotes keeps its backslash", 'tools/hooks/identity.py', + '''inner = re.sub(r'\\\\([\\\\`$"])', r"\\1", text[i + 1:end])''', '''inner = re.sub(r'\\\\([\\\\`"])', r"\\1", text[i + 1:end])''', + (IDENTITY_TESTS + 'test_a_backtick_in_double_quotes_has_quotes_of_its_own',)), + ("a backtick in double quotes that never closes is spelt", 'tools/hooks/identity.py', + ' if end < len(text) and shell._left_open(inner) == (False, False):', + ' if shell._left_open(inner) == (False, False):', (IDENTITY_TESTS + 'test_a_backtick_in_double_quotes_has_quotes_of_its_own',)), + ("a backtick in double quotes whose quote or comment runs past it is spelt", 'tools/hooks/identity.py', + ' if end < len(text) and shell._left_open(inner) == (False, False):', + ' if end < len(text):', (IDENTITY_TESTS + 'test_a_backtick_in_double_quotes_has_quotes_of_its_own',)), + ("a quote past a backtick in a body's substitution is data", 'tools/hooks/shell.py', + ' if _left_open(inner)[0]:', ' if False:', + (IDENTITY_TESTS + 'test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data', PROTECTED_NESTED_IN_ONE)), + ("a quote in a comment in a body's backtick is counted", 'tools/hooks/shell.py', + ' if quote is None and c == "#" and i != escaped and (i == 0 or script[i - 1] in COMMENTS_AFTER):', + ' if False:', (IDENTITY_TESTS + 'test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data',)), + ("a # after a character a backslash holds in a body's backtick begins a comment", 'tools/hooks/shell.py', + ' if quote is None and c == "#" and i != escaped and (', ' if quote is None and c == "#" and (', + (IDENTITY_TESTS + 'test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data',)), + ("a substitution in a body's backtick is scanned flat", 'tools/hooks/shell.py', + ''' if quote != "'" and script.startswith("$(", i):''', ' if False:', + (IDENTITY_TESTS + 'test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data',)), + ("a backslash in a body's backtick holds no quote", 'tools/hooks/shell.py', + ''' if c == "\\\\" and quote != "'":\n i = escaped = i + 2''', ''' if False:\n i = escaped = i + 2''', + (IDENTITY_TESTS + 'test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data',)), + ("a backslash in single quotes in a body's backtick holds a quote", 'tools/hooks/shell.py', + ''' if c == "\\\\" and quote != "'":\n i = escaped = i + 2''', ''' if c == "\\\\":\n i = escaped = i + 2''', + (IDENTITY_TESTS + 'test_a_quote_past_a_backtick_in_a_here_documents_substitution_is_not_read_as_data',)), # Issue 410: a backtick glued to a redirection, a nested ${...}, and what only prints beside a script. ("a backtick inside a word leaves the word read as what came before it", 'tools/hooks/shell.py', ' out.append("$")\n', ' pass\n', (IDENTITY_TESTS + 'test_a_backtick_glued_to_a_redirection_and_a_substitution_in_a_nested_expansion_are_read',)), From b94aeac706b5a2d77093f60d0f53922a598cf868 Mon Sep 17 00:00:00 2001 From: Don Petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 8 Nov 2026 21:50:32 -0600 Subject: [PATCH 10/10] What Muse's read of the heads found: both pins held by counter-examples, and the tool writes a case's three files all or none Co-Authored-By: Claude Opus 5.5 --- ...pinned-by-the-case-and-held-to-its-tree.md | 4 +- .../eval-case.a-head-pin-cut-short.json | 256 ++++++++++++++++++ examples/invalid/eval-case.no-tree-pin.json | 255 +++++++++++++++++ tests/test_evals.py | 11 + tools/heads_into_home.py | 44 +-- tools/mutants.py | 9 +- 6 files changed, 559 insertions(+), 20 deletions(-) create mode 100644 examples/invalid/eval-case.a-head-pin-cut-short.json create mode 100644 examples/invalid/eval-case.no-tree-pin.json diff --git a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md index 0305556b..d04f9afb 100644 --- a/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md +++ b/docs/decisions/NNNN-a-cases-head-is-read-from-the-held-out-home-pinned-by-the-case-and-held-to-its-tree.md @@ -25,5 +25,5 @@ A case's head was read with no credential, from the public repo its change was m |---|---| | What the home keeps beside a case is asked with the raw media type and the home's token, and handed back only as the bytes the case pins: an answer of another type, `+json` among them, one not found, refused, over its bound or of another hash is none, and a name, kind, pin or home that could name another file is not asked | `tests/test_adapters.py::test_what_the_home_keeps_beside_a_case_is_read_raw_by_the_hash_the_case_pins`, `trials/P-62.json`, `adapters/heldout.py`, mutant: "the eval blob is asked for as raw+json", mutant: "a head beside a case of another hash is read", mutant: "what the home keeps beside a case is asked as JSON", mutant: "a head of any size is read", mutant: "a head answered under another type is read", mutant: "a head answered as raw JSON is read", mutant: "a head GitHub did not find is unanswered", mutant: "a head GitHub refused is read", mutant: "a file of any kind is asked beside a case", mutant: "a pin of any shape is asked by", mutant: "a file beside a case of any name is asked", mutant: "a home of any name is asked beside a case" | | The fetch reads each case's head from the home by its pins, with no other transport, unpacks it and holds it to its tree; a head not the one pinned is `hash_differs`, one not read, unpacked or held whole `head_unreadable` and removed, a pin that is no sha256 reads nothing and the case is `not_this_set`, and the gate closes | `tests/test_evals.py::test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate`, `harness/eval/__init__.py`, `harness/eval/fetch.py`, `harness/head.py`, mutant: "the eval fetch reads no case's head", mutant: "a head is unpacked and never held to its tree", mutant: "a head not the one pinned is read as unreadable", mutant: "the tree's pin is never read", mutant: "a head is read beside another Role", mutant: "a tree that cannot be read stops the fetch", mutant: "a head not held is left in the heads folder", mutant: "a case whose pin is no sha256 has its head read" | -| An eval case requires both pins, each a whole sha256 | `tests/test_evals.py::test_a_case_dropped_under_evals_in_any_file_is_caught`, `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a tree pin may be cut short" | -| The Maintainer's tool writes a head beside its case only once it is held to its tree and within what the job reads, pins what it wrote, writes each file whole and through no link, reads no case or folder that is a link, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token", mutant: "the Maintainer's tool writes through a link", mutant: "the Maintainer's tool reads a case that is a link", mutant: "the Maintainer's tool pins a head the job will refuse", mutant: "the Maintainer's tool reads a folder that is a link", mutant: "a write that fails stops the Maintainer's tool", mutant: "the Maintainer's tool leaves a file half written" | +| An eval case requires both pins, each a whole sha256 | `tests/test_evals.py::test_a_case_dropped_under_evals_in_any_file_is_caught`, `tools/check_examples.py`, `schemas/eval-case.schema.json`, `examples/invalid/eval-case.no-head-pin.json`, `examples/invalid/eval-case.no-tree-pin.json`, `examples/invalid/eval-case.a-head-pin-cut-short.json`, `examples/invalid/eval-case.a-tree-pin-cut-short.json`, mutant: "a case need pin no head", mutant: "a case need pin no tree", mutant: "a head pin may be cut short", mutant: "a tree pin may be cut short" | +| The Maintainer's tool writes a head beside its case only once it is held to its tree and within what the job reads, pins what it wrote, writes each file whole and through no link, reads no case or folder that is a link, leaves a case it cannot pin as it was, reads no tree file as a case, prints counts and hashes alone, and says so in its exit; what it writes is what the fetch reads | `tests/test_evals.py::test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone`, `tools/heads_into_home.py`, mutant: "the Maintainer's tool writes a head not held to its tree", mutant: "the Maintainer's tool pins a case of another id", mutant: "the Maintainer's tool reads a tree answered otherwise", mutant: "the Maintainer's tool says every case was pinned", mutant: "the Maintainer's tool reads a tree it wrote as a case", mutant: "the Maintainer's tool pins a tree it did not write", mutant: "the Maintainer's tool reads with no token", mutant: "the Maintainer's tool writes through a link", mutant: "the Maintainer's tool reads a case that is a link", mutant: "the Maintainer's tool pins a head the job will refuse", mutant: "the Maintainer's tool reads a folder that is a link", mutant: "a write that fails stops the Maintainer's tool", mutant: "the Maintainer's tool leaves a file half written", mutant: "the Maintainer's tool writes a head beside a case it cannot pin" | diff --git a/examples/invalid/eval-case.a-head-pin-cut-short.json b/examples/invalid/eval-case.a-head-pin-cut-short.json new file mode 100644 index 00000000..afff1335 --- /dev/null +++ b/examples/invalid/eval-case.a-head-pin-cut-short.json @@ -0,0 +1,256 @@ +{ + "schema": "eval-case/1", + "id": "synthetic-early-return", + "role": "pr-review", + "intent": "review-change", + "facts": { + "schema": "facts/1", + "item": { + "repo": "combsmith/software-factory", + "type": "pull_request", + "number": 42 + }, + "state": "open", + "is_draft": false, + "class": "maintainer", + "author": "don-petry", + "last_pusher": "don-petry", + "targets_protected": true, + "mergeable": true, + "content_id": "sha256:5d0c", + "diff": { + "files": 3, + "lines": 120, + "truncated": false, + "characters": 4800 + }, + "changed_paths": [ + { + "path": "kernel/derive.py", + "previous": "kernel/derive_old.py" + }, + { + "path": "tests/test_decide.py", + "previous": null + }, + { + "path": "docs/glossary.md", + "previous": null + } + ], + "label_events": [ + { + "label": "factory:hold", + "action": "labeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:40:00Z" + }, + { + "label": "factory:hold", + "action": "unlabeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:45:00Z" + } + ], + "last_push_at": "2026-10-11T12:00:00Z", + "last_reopened_at": null, + "description_edited_at": null, + "expired_reviews": [], + "last_writer_action_at": "2026-10-11T12:00:00Z", + "reviews": [ + { + "by": { + "kind": "machine_user", + "name": "donpetry-bot", + "can_write": true + }, + "state": "dismissed", + "commit": "4b1a9c0", + "content_id": "sha256:4b1a", + "submitted_at": "2026-10-11T11:50:00Z", + "dismissed_by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "dismissed_from": "approved" + }, + { + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "state": "changes_requested", + "commit": "5d0c7e2", + "content_id": "sha256:5d0c", + "submitted_at": "2026-10-11T12:18:00Z", + "dismissed_by": null, + "dismissed_from": null + } + ], + "threads": [ + { + "id": "PRRT_kw7", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "pr-review", + "can_write": true + }, + "category": "functional", + "severity": "major", + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw1", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "coderabbit", + "can_write": false + }, + "category": null, + "severity": null, + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw3", + "resolved": true, + "opened_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "category": "human", + "severity": null, + "resolved_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "opened_at": "2026-10-11T12:16:00Z", + "ruling": null + } + ], + "checks": [ + { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "Secret scan (gitleaks)", + "app_id": 15368, + "workflow": ".github/workflows/ci.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "CodeQL", + "app_id": 57789, + "workflow": null, + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + } + ], + "queue": false, + "acts_since_writer": 2, + "acceptance": { + "approved": { + "digest": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "commit": "1a2b3c4", + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "criteria": [ + { + "id": "AC-1", + "kind": "test" + }, + { + "id": "AC-2", + "kind": "test" + } + ] + }, + "head": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "results": [ + { + "criterion": "AC-1", + "content_id": "sha256:4b1a", + "result": "failing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-1", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-2", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + } + ] + }, + "complete": { + "acceptance": true, + "reviews": true, + "threads": true, + "checks": true, + "diff": true, + "changed_paths": true, + "runs": true, + "timeline": true + }, + "read_at": "2026-10-11T12:20:00Z" + }, + "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", + "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec", + "tree_sha256": "c8037a8b9d2fca258427534558b219fa94635816faf136ab59684a97d2429010", + "expected": { + "verdict": "finding", + "criterion": "PR-3", + "finding": { + "category": "functional", + "severity": "major", + "location": { + "path": "kernel/decide.py", + "start_line": 86, + "end_line": 94 + } + } + } +} diff --git a/examples/invalid/eval-case.no-tree-pin.json b/examples/invalid/eval-case.no-tree-pin.json new file mode 100644 index 00000000..b06e7f3c --- /dev/null +++ b/examples/invalid/eval-case.no-tree-pin.json @@ -0,0 +1,255 @@ +{ + "schema": "eval-case/1", + "id": "synthetic-early-return", + "role": "pr-review", + "intent": "review-change", + "facts": { + "schema": "facts/1", + "item": { + "repo": "combsmith/software-factory", + "type": "pull_request", + "number": 42 + }, + "state": "open", + "is_draft": false, + "class": "maintainer", + "author": "don-petry", + "last_pusher": "don-petry", + "targets_protected": true, + "mergeable": true, + "content_id": "sha256:5d0c", + "diff": { + "files": 3, + "lines": 120, + "truncated": false, + "characters": 4800 + }, + "changed_paths": [ + { + "path": "kernel/derive.py", + "previous": "kernel/derive_old.py" + }, + { + "path": "tests/test_decide.py", + "previous": null + }, + { + "path": "docs/glossary.md", + "previous": null + } + ], + "label_events": [ + { + "label": "factory:hold", + "action": "labeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:40:00Z" + }, + { + "label": "factory:hold", + "action": "unlabeled", + "by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "at": "2026-10-11T11:45:00Z" + } + ], + "last_push_at": "2026-10-11T12:00:00Z", + "last_reopened_at": null, + "description_edited_at": null, + "expired_reviews": [], + "last_writer_action_at": "2026-10-11T12:00:00Z", + "reviews": [ + { + "by": { + "kind": "machine_user", + "name": "donpetry-bot", + "can_write": true + }, + "state": "dismissed", + "commit": "4b1a9c0", + "content_id": "sha256:4b1a", + "submitted_at": "2026-10-11T11:50:00Z", + "dismissed_by": { + "kind": "human", + "name": "don-petry", + "can_write": true + }, + "dismissed_from": "approved" + }, + { + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "state": "changes_requested", + "commit": "5d0c7e2", + "content_id": "sha256:5d0c", + "submitted_at": "2026-10-11T12:18:00Z", + "dismissed_by": null, + "dismissed_from": null + } + ], + "threads": [ + { + "id": "PRRT_kw7", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "pr-review", + "can_write": true + }, + "category": "functional", + "severity": "major", + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw1", + "resolved": false, + "opened_by": { + "kind": "lens", + "name": "coderabbit", + "can_write": false + }, + "category": null, + "severity": null, + "resolved_by": null, + "opened_at": "2026-10-11T12:15:00Z", + "ruling": null + }, + { + "id": "PRRT_kw3", + "resolved": true, + "opened_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "category": "human", + "severity": null, + "resolved_by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "opened_at": "2026-10-11T12:16:00Z", + "ruling": null + } + ], + "checks": [ + { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "Secret scan (gitleaks)", + "app_id": 15368, + "workflow": ".github/workflows/ci.yml", + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + }, + { + "name": "CodeQL", + "app_id": 57789, + "workflow": null, + "conclusion": "success", + "completed_at": "2026-10-11T12:06:00Z" + } + ], + "queue": false, + "acts_since_writer": 2, + "acceptance": { + "approved": { + "digest": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "commit": "1a2b3c4", + "by": { + "kind": "human", + "name": "a-colleague", + "can_write": true + }, + "criteria": [ + { + "id": "AC-1", + "kind": "test" + }, + { + "id": "AC-2", + "kind": "test" + } + ] + }, + "head": "sha256:c0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0dec0de", + "results": [ + { + "criterion": "AC-1", + "content_id": "sha256:4b1a", + "result": "failing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-1", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + }, + { + "criterion": "AC-2", + "content_id": "sha256:5d0c", + "result": "passing", + "check": { + "name": "verify", + "app_id": 15368, + "workflow": ".github/workflows/verify.yml" + } + } + ] + }, + "complete": { + "acceptance": true, + "reviews": true, + "threads": true, + "checks": true, + "diff": true, + "changed_paths": true, + "runs": true, + "timeline": true + }, + "read_at": "2026-10-11T12:20:00Z" + }, + "diff": "--- a/kernel/decide.py\n+++ b/kernel/decide.py\n@@ -86,9 +86,9 @@\n- if not items:\n- return []\n first = items[0]\n+ if not items:\n+ return []\n return parse(first, items[1:])\n", + "head": "c514ff021d93934192c9dbb14b6c31c2b421e6c7", + "head_sha256": "6bbe1f08b91c69085d33a368e265ba4f7ef7438228aa0047546ab018b61eeec2", + "expected": { + "verdict": "finding", + "criterion": "PR-3", + "finding": { + "category": "functional", + "severity": "major", + "location": { + "path": "kernel/decide.py", + "start_line": 86, + "end_line": 94 + } + } + } +} diff --git a/tests/test_evals.py b/tests/test_evals.py index adfa81fc..d23f21e9 100644 --- a/tests/test_evals.py +++ b/tests/test_evals.py @@ -2792,6 +2792,17 @@ def send(self, req): out = io.StringIO() self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 1) self.assertEqual(out.getvalue(), "pinned 0 of 1 cases; not_written 1\n") + # A name that is no file, here a folder where the tree goes, fails the case before its tarball is written. + (home / "pr-review" / "poc-one.tar.gz").unlink() + (home / "pr-review" / "poc-one.tree.json").unlink() + (home / "pr-review" / "poc-one.tree.json").mkdir() + (home / "pr-review" / "poc-one.tree.json" / "kept").write_bytes(b"") + out = io.StringIO() + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), out), 1) + self.assertEqual(out.getvalue(), "pinned 0 of 1 cases; not_written 1\n") + self.assertFalse((home / "pr-review" / "poc-one.tar.gz").exists(), "no head half written") + shutil.rmtree(str(home / "pr-review" / "poc-one.tree.json")) + self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), io.StringIO()), 0) with mock.patch.object(tool.os, "replace", mock.Mock(side_effect=OSError("a path"))): self.assertEqual(tool.main([str(home), "pr-review"], {}, GitHub(), io.StringIO()), 1) self.assertEqual([p.name for p in (home / "pr-review").iterdir() if p.name.startswith(".")], [], "nothing half written") diff --git a/tools/heads_into_home.py b/tools/heads_into_home.py index 13d4dcf9..3ecfe6b0 100644 --- a/tools/heads_into_home.py +++ b/tools/heads_into_home.py @@ -30,17 +30,26 @@ REPO = re.compile(r"[A-Za-z0-9_.-]{1,100}/[A-Za-z0-9_.-]{1,100}") -def put(folder: pathlib.Path, name: str, data: bytes) -> None: - """`data` written as `folder/name`, whole or not at all: into a new file beside it, then moved over it, so that a - link already at that name is replaced, never written through, and nothing is left half written.""" - fd, temporary = tempfile.mkstemp(dir=str(folder), prefix=".heads-into-home-") +def put(folder: pathlib.Path, files: dict) -> None: + """Each of `files`, `{name: bytes}`, written in `folder`, all or none: each name held to be nothing yet, a link or a + regular file, each written whole into a new file beside it, and only then each moved over its name, so that a link + there is replaced, never written through, and no case is left with its head half written.""" + for name in files: + target = str(folder / name) + if os.path.lexists(target) and not (os.path.islink(target) or os.path.isfile(target)): + raise OSError("not a file") + staged = {} try: - with os.fdopen(fd, "wb") as out: - out.write(data) - os.replace(temporary, str(folder / name)) + for name, data in files.items(): + fd, staged[name] = tempfile.mkstemp(dir=str(folder), prefix=".heads-into-home-") + with os.fdopen(fd, "wb") as out: + out.write(data) + for name, temporary in staged.items(): + os.replace(temporary, str(folder / name)) except BaseException: - with contextlib.suppress(OSError): - os.unlink(temporary) + for temporary in staged.values(): + with contextlib.suppress(OSError): + os.unlink(temporary) raise @@ -58,9 +67,10 @@ def read(path: pathlib.Path): def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): - """`(the case's new bytes, None)` once its head's tarball and tree are written beside it in `folder`, or `(None, code)`: + """`({name: bytes}, None)`: the head's tarball and tree to write beside the case in `folder`, and the case anew, + pinning them, which `put` writes all or none; or `(None, code)`: `not_a_case` for bytes that are no case with a head of a repo, and `head_unreadable` for a head not read, unpacked, - held to its tree, or within what the eval job reads, whole. Nothing is written for a case not pinned.""" + held to its tree, or within what the eval job reads, whole. Nothing is written here.""" from adapters import github, heldout from harness import head try: @@ -96,9 +106,7 @@ def pinned(data: bytes, transport, folder: pathlib.Path, ident: str): if key == "head": out["head_sha256"], out["tree_sha256"] = hashlib.sha256(tar).hexdigest(), hashlib.sha256(listed).hexdigest() new = (json.dumps(out, indent=2, ensure_ascii=False) + "\n").encode("utf-8") - put(folder, "%s.tar.gz" % ident, tar) - put(folder, "%s.tree.json" % ident, listed) - return new, None + return {"%s.tar.gz" % ident: tar, "%s.tree.json" % ident: listed, "%s.json" % ident: new}, None def main(argv: list, env=None, transport=None, out=None) -> int: @@ -120,9 +128,11 @@ def main(argv: list, env=None, transport=None, out=None) -> int: ident = path.name[:-len(".json")] data = read(path) if PART.fullmatch(ident) else None try: - new, code = pinned(data, transport, folder, ident) if data is not None else (None, "not_a_case") - if new is not None: - put(folder, path.name, new) + files, code = pinned(data, transport, folder, ident) if data is not None else (None, "not_a_case") + new = None + if files is not None: + put(folder, files) + new = files[path.name] except Exception: # by its code alone: no message, which could quote a case new, code = None, "not_written" said[code or "pinned"] = said.get(code or "pinned", 0) + 1 diff --git a/tools/mutants.py b/tools/mutants.py index c028c80a..5a194866 100644 --- a/tools/mutants.py +++ b/tools/mutants.py @@ -9292,6 +9292,13 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: 'heads_place(env, workspace), found["role"], entry["id"], transport)', 'heads_place(env, workspace), "pr-review-gate", entry["id"], transport)', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), ('a tree that cannot be read stops the fetch', 'harness/eval/__init__.py', ' except (head.Refused, OSError, Stop, ValueError, RecursionError):', ' except (head.Refused, OSError, Stop):', ('tests.test_evals.EvalJob.test_each_cases_head_is_read_from_the_home_by_its_pins_and_one_not_read_closes_the_gate',)), + ('a case need pin no tree', 'schemas/eval-case.schema.json', ' "head_sha256",\n "tree_sha256",\n', ' "head_sha256",\n', ('tests.test_evals.TheSet.test_a_case_dropped_under_evals_in_any_file_is_caught',)), + ('a head pin may be cut short', 'schemas/eval-case.schema.json', + '"head_sha256": {\n "description": "The sha256 of the head\'s tarball as GitHub archived the commit, kept beside the case in the held-out home, `/.tar.gz`: the eval job reads the head there, with the home\'s token, and none with no credential (issue 611).",\n "type": "string",\n "pattern": "^[0-9a-f]{64}$"', + '"head_sha256": {\n "description": "The sha256 of the head\'s tarball as GitHub archived the commit, kept beside the case in the held-out home, `/.tar.gz`: the eval job reads the head there, with the home\'s token, and none with no credential (issue 611).",\n "type": "string",\n "pattern": "^[0-9a-f]{63,64}$"', ('tests.test_evals.TheSet.test_a_case_dropped_under_evals_in_any_file_is_caught',)), + ("the Maintainer's tool writes a head beside a case it cannot pin", 'tools/heads_into_home.py', + ' if os.path.lexists(target) and not (os.path.islink(target) or os.path.isfile(target)):\n', + ' if False:\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), ('a case need pin no head', 'schemas/eval-case.schema.json', ' "head_sha256",\n "tree_sha256",\n', ' "tree_sha256",\n', ('tests.test_evals.TheSet.test_a_case_dropped_under_evals_in_any_file_is_caught',)), ('a tree pin may be cut short', 'schemas/eval-case.schema.json', '"tree_sha256": {\n "description": "The sha256 of the commit\'s tree, every level of it, as GitHub answered it, kept beside the case in the held-out home, `/.tree.json`: the head is held to it file by file (issue 611; decision 0142).",\n "type": "string",\n "pattern": "^[0-9a-f]{64}$"', @@ -9326,7 +9333,7 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple: ("a write that fails stops the Maintainer's tool", 'tools/heads_into_home.py', ' except Exception: # by its code alone', ' except ZeroDivisionError: # by its code alone', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), ("the Maintainer's tool leaves a file half written", 'tools/heads_into_home.py', - ' with contextlib.suppress(OSError):\n os.unlink(temporary)\n', ' pass\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), + ' with contextlib.suppress(OSError):\n os.unlink(temporary)\n', ' pass\n', ('tests.test_evals.EvalJob.test_the_maintainers_tool_pins_each_head_beside_its_case_and_prints_counts_and_hashes_alone',)), # Issue 613, the preflight: each dispatch of a gate set asks a few development cases first, and a broken pipeline stops it. ('a gate set need name no preflight', 'harness/eval/__init__.py', 'OWN = ("bars", "kind_least", "record", "preflight")',