Trials P-30 and P-63: the release identity and the eval job, recorded from their runs (#189, #309)

.claude/skills/run-a-platform-trial/SKILL.md

@@ -21,12 +21,12 @@ everything a trial leaves in it: its runs, logs and artifacts.
 - **Only the canary.** A trial runs in combsmith/software-factory-canary. Set
   `GH_REPO=combsmith/software-factory-canary` for every command of the trial, and name no other
   repo. P-8 needs twelve repos, P-11 the control room and the hub, P-19 a separate public repo,
-  P-30 the Factory's own repo, and P-31 every repo an App is installed on: for these, stop and
+  P-30 and P-63 the Factory's own repo, and P-31 every repo an App is installed on: for these, stop and
   raise it with the Maintainer before anything runs. The fixture's schema accepts the canary,
   or the org `combsmith` for a trial about the org and its Apps alone, such as P-39 (decision
   0035), or the Factory's own repo for a trial `tools/check_trials.py` names in
-  `TRIALS_OF_THE_FACTORY`, which is P-30 alone, or, for a vendor's CLI, no repo (decision 0033), so
-  any other trial outside the canary needs that decided first.
+  `TRIALS_OF_THE_FACTORY`, which is P-30 and P-63 alone, or, for a vendor's CLI, no repo
+  (decision 0033), so any other trial outside the canary needs that decided first.
 - **Nothing here reads the canary.** Only a document, a fixture, the fixture's example and its
   schema name it here, and so does `config/apps.json`, which declares where the Apps are
   installed and which nothing runs. `tools/check_trials.py` refuses any other file that names

docs/architecture/software-factory-design.md

@@ -2220,7 +2220,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is
 | P-31 | Does the Controller App's webhook reach a hosted function for every repo it is installed on, and can that function cause a nudge with no credential over the Factory? | Yes to both |
 | P-28 | Can the actor who last pushed a pull request's head be read as a fact? | Yes, or I-26 has to be met another way |
 | P-29 | Is the hash of a pull request's own diff stable when only the base moves, and how does a truncated diff present? | Stable; truncation is detectable |
-| P-30 | Which identity can cut a Release and move a channel tag under a tag ruleset, from a workflow in the Factory repo? | One identity, through one path |
+| P-30 | Which identity can cut a Release and move a channel tag under a tag ruleset, from a workflow in the Factory repo? | One identity, through one path. **Passed** on 2026-11-09 (`trials/P-30.json`), recorded from the release runs of v0.1.0: the release App, combsmith-sf-release, cut it and moved the channel tag from `release.yml` on `main`, and the re-cut wrote nothing. The tag ruleset's bypass list names that App and the admin role alone, and the admin role is the Maintainer's login alone, both read live. That read stands in for each other identity's refusal; none was exercised (decision 0173). Decision 0114 is why the App and not the poller's job token |
 | P-21 | How often may the control room's page be rebuilt, and can a page read the feed files directly from a browser? | The page reads the feed directly and rebuilds only when its own code changes |
 | P-23 | How many threads may one App resolve in an hour before GitHub's secondary limits apply? | Enough for about 22 threads a pull request at the Factory's merge rate. **Passed** on 2026-10-28 (`trials/P-23.json`), in the canary: an App with `contents: write` resolved 30 threads it had opened, one after another as fast as GitHub answered, each 200 with no error, in 40 seconds, the GraphQL count rising from 8 to 37 of 5,000, with no secondary limit met. An hour's rate was not measured: no limit was reached |
 | P-24 | With the Maintainer as the only code owner of a path, does the machine user's approval fail to satisfy code-owner review there? Not needed for the Factory's repo while its code owners follow the org's standard | It fails to satisfy it |
@@ -2257,6 +2257,7 @@ it observed as a recorded fixture, which is what the simulator's fake GitHub is
 | P-60 | Does GitHub serve a commit's tarball as one redirect from api.github.com to codeload.github.com, which answers it with no Authorization header, and does codeload redirect again? | One redirect, so the transport may follow it to codeload alone and refuse any second (the Maintainer's "Tarball, allow codeload", 2026-11-04). **Passed** on 2026-11-04 (`trials/P-60.json`), on the canary, asked with no credential: the API answered 302 to `https://codeload.github.com/<owner>/<repo>/legacy.tar.gz/<sha>`, with no token in it, and codeload answered that with no Authorization header 200, a gzip with no `content-length`, whose tar opens with a pax global header naming the whole commit and holds one folder of directories and regular files. No address on codeload over HTTPS redirected; over plain HTTP it answered 301 to the same address over HTTPS. For a commit the canary does not hold the API still answered 302, and codeload 404. What codeload's Location holds for a private repo, read with an App's token, is not asked: it is owed before a live Run reads a private head. The same commit's tree, `GET /repos/<owner>/<repo>/git/trees/<sha>?recursive=1` asked with no credential on 2026-11-05 (`trials/P-60.tree.json`), answered 200 and not truncated, listing every folder and file with its mode, and each file with its size and a blob hash that matched the tarball's bytes, all 22; a commit the canary does not hold answered 422. The review's head is held to that tree (decision 0142) |
 | P-61 | Does an issue's timeline give each change of its type, and each link of a pull request by hand in its Development box and its unlinking, as an event of its own with its time to the second and who made it; and does the merge of a pull request linked by hand close the issue, naming it the closer? | Each as its own timed event, so that row 1 can read an issue's type and a link by hand as each stood at the merge ("Trial P-61: an issue's type changes and a pull request linked by hand, each timed on the issue's timeline", issue 551; decision 0141). **Not run yet**: `tools/trial_p61_timeline.sh` reads the timeline through the Maintainer's own login at each of eight steps he makes by hand in the canary, and keeps each answer redacted outside the repo; it waits on the eval replay |
 | P-62 | Does the contents API, asked with the raw media type, answer a file of more than 1 MiB with its bytes whole, in one answer and no redirect; and does the git blobs API answer the same blob as base64? | Whole, in one answer, so that the eval job reads a case's head and tree from the held-out home with the home's token alone ("A case's head is read from the private home, pinned by its hash, and the run is sharded", issue 611; the Factory Builder's ruling of 2026-11-08, (d)); the git blobs API is the fallback. **Passed** on 2026-11-09 (`trials/P-62.json`), by the Factory Builder's ruling of 2026-11-08 that "the raw media type" in its pass condition is answered by `application/vnd.github.raw`, the type the transport must ask for, and that `gh`'s failure on `application/vnd.github.raw+json` with the Maintainer's login is a fault of the client on a type the Factory will not use, not GitHub's answer; in the canary, on a file of 1,577,185 random bytes on a branch made for it and deleted after: asked with `application/vnd.github.raw+json` with no credential, or with `application/vnd.github.raw` with the Maintainer's login or none, the API answered 200 with a `content-length` of the file's and the file's bytes exactly, by their sha256, and no redirect. Asked as JSON it answered the file's size with `content` empty and `encoding` `none`. The git blobs API answered the blob as base64 that decoded to the same bytes, and with the raw media type as `application/octet-stream`. With the login, `application/vnd.github.raw+json` was answered 200 with a `content-length` of the file's, but `gh api` wrote no body, saying "transform: short source buffer", three times; and the transport reads any answer whose type ends in json as JSON, so every reader, the transport among them, asks for `application/vnd.github.raw`, whose answer's type does not. What the API answers for a private repo's file is not asked here. Should it answer the held-out home's otherwise, a redirect among them, the read fails closed: the transport follows no redirect for a file, and a head not read whole is `head_unreadable` |
+| P-63 | Does the review Intent's eval job, run from `main`, read the private held-out home with its environment's secret and upload its `eval-report/1` by hashes alone, and is that environment refused to every other branch? | The job reads and uploads on `main`, and the environment admits `main` alone with no admin bypass. **Passed** on 2026-11-09 (`trials/P-63.json`), recorded from run 42347416962, the gate's run for `factory/v0.1.0`, by hashes, codes and run metadata alone: its fetch and upload succeeded, its report's set and cases are `evals/pr-review/cases.json`'s at the run's commit, and `factory-eval` deploys from `main` alone with no admin bypass, read live. No dispatch from another branch was exercised; the live read stands in for it (decision 0173). It runs in the Factory's own repo, as P-30 does (decision 0113) |
 
 ### 6.1 The order to run them in
 

docs/decisions/0173-the-eval-jobs-trial-records-the-factorys-own-repo-beside-p-30-and-a-live-read-stands-for-a-refusal.md

@@ -0,0 +1,25 @@
+# 0173. The eval job's trial records the Factory's own repo beside P-30, and a live read stands for a refusal
+
+Status: accepted. The Factory Builder's rulings of 2026-11-09 on the fixtures of "Epic: slice 1, first approval. A Release on next reviews a pull request to this repo, and the machine user's approval stands" (issue 182), for "Workflows: eval.yml runs the review Intent's held-out cases and writes eval-report/1" (issue 309) and "Trial P-30: one identity cuts a Release and moves a channel tag under the tag ruleset" (issue 189). Pre-approved by the Maintainer on 2026-11-08. It amends decision 0113's row "Which trials may", whose "P-30 alone" now reads "P-30 and P-63 alone", and changes nothing else of it.
+
+## Decision
+
+| What | Decided | Because |
+|---|---|---|
+| The eval job's trial | P-63, a new row of section 6: the job, run from `main`, reads the private held-out home with `factory-eval`'s secret and uploads its `eval-report/1` by hashes alone, and that environment is refused to every other branch. `trials/P-63.json` records run 42347416962, the gate's run for the Release v0.1.0 | The eval job's story asks a platform trial to record the job, and no row of section 6 asked it |
+| Where it records it ran | `combsmith/software-factory`: `TRIALS_OF_THE_FACTORY` in `tools/check_trials.py` names P-63 beside P-30, and still no other | The job and its environment are this repo's, and the canary has neither, as with P-30's tag ruleset under 0113 |
+| What it keeps of the report | The report's commit, Release, set, totals and one-word bars, and each case by its id and sha256 alone. Each case's verdict and reasons are left out, and the archive's own sha256 is the artifact's digest GitHub lists | A reason such as `finding_not_raised` tells a case's kind, and no count of a kind is written in this repo (decision 0120). The id and the hash are already `evals/pr-review/cases.json`'s |
+| What stands for a refusal | P-63: `factory-eval`'s one deployment branch policy, `main`, with no admin bypass, read live, stands for a dispatch from another branch refused. P-30: tag ruleset 25183656's bypass list, the admin role and the release App alone, read live, stands for each other identity's refusal. Each fixture states its residual in its pass condition: no dispatch from another branch, and no other identity's refusal, was exercised | A rule GitHub enforces, read as it stands, says whom it refuses. Exercising each refusal needs a dispatch the Maintainer alone makes, or a credential for each identity, which no session holds |
+| P-30 | `trials/P-30.json` records release runs 42348605675, which cut the Release v0.1.0 and moved the channel tag v0-next to it, its Deployment naming the release App as creator, and 42348650504, the same cut refused, with both tag rulesets and the repo's admins read live. Decision 0114 is why the release App and not the poller's job token | The first two boxes of P-30's story |
+| When each pass condition was written | After the runs, from the story's own condition as the ruling of 2026-11-09 stands it in. This sets aside, for these two fixtures alone, the trial skill's step 1, that the condition comes before anything runs: the runs were the Maintainer's dispatches for the Release, made before any trial was asked of them, and each condition names its stand-in and residual rather than fitting the answer | A fixture of a run already made can only be written after it; saying so is better than a condition that reads as written first |
+
+## Why
+
+Both stories needed a fixture to close, and both runs had already happened. A fixture made from those runs, with the rule that refuses everyone else read beside it, says what happened and what it stands on, and names what it does not show.
+
+## Boundaries
+
+| Boundary | Check |
+|---|---|
+| P-63 recorded on this repo holds, and recorded on the canary is refused | `tests/test_tools.py::test_the_eval_jobs_trial_records_the_factorys_own_repo`, `tools/check_trials.py`, mutant: "the eval job's trial is no trial of the Factory" |
+| No trial but P-30 and P-63 records this repo | `examples/invalid/trial.another-trial-in-the-factorys-repo.json`, `tests/test_tools.py::test_a_cost_measurements_exchanges_name_each_repo_they_span`, `tools/check_trials.py`, mutant: "any trial may record the Factory's own repo" |

schemas/trial.schema.json

@@ -35,7 +35,7 @@
       "minLength": 1
     },
     "repo": {
-      "description": "Where the trial ran: the canary repo; the org itself for a trial about the org and its Apps, such as P-39 (decision 0035); the Factory's own repo for a trial tools/check_trials.py names in TRIALS_OF_THE_FACTORY, such as P-30; or null for a trial that asks a vendor's CLI and not GitHub, such as P-13 (decision 0033). Nothing else.",
+      "description": "Where the trial ran: the canary repo; the org itself for a trial about the org and its Apps, such as P-39 (decision 0035); the Factory's own repo for a trial tools/check_trials.py names in TRIALS_OF_THE_FACTORY, P-30 and P-63; or null for a trial that asks a vendor's CLI and not GitHub, such as P-13 (decision 0033). Nothing else.",
       "enum": [
         "combsmith/software-factory-canary",
         "combsmith",

tests/test_tools.py

@@ -5633,6 +5633,14 @@ def test_a_write_is_recorded_only_on_a_node_a_read_of_the_org_answered(self):
         outside["exchanges"][0]["request"]["call"] = "GET /repos/outside/x/pulls/7"
         self.assertTrue(any("on a node no read" in p for p in check_trials.fixture_problems(outside)))
 
+    def test_the_eval_jobs_trial_records_the_factorys_own_repo(self):  # issue 309, decision 0173
+        # P-63, the eval job's trial, ran in this repo from main: recorded there it holds, and on the canary it is refused.
+        data = json.loads((ROOT / "trials" / "P-63.json").read_text())
+        self.assertEqual(check_trials.fixture_problems(data), [])
+        canary = check_trials.ORG + "/" + check_trials.CANARY
+        self.assertEqual(check_trials._of_the_factory(dict(data, repo=canary)),
+                         [("repo", "is %s, and P-63 runs in %s; record where it ran" % (canary, check_trials.FACTORY))])
+
     def test_a_write_in_a_fixture_of_the_factory_is_held_to_a_node_of_the_factory(self):
         # What holds a canary fixture's writes holds this one's, with the Factory's id in the canary's. Each node id is
         # a prefix and base64url of msgpack [0, the repository's id, ...], as GitHub writes one.

tools/check_trials.py

@@ -57,10 +57,11 @@
 TRIALS_OF_FORKS = {"P-27"}
 TRIALS_OF_REACH = {"P-4"}
 # A trial that can only run in the Factory's own repo, and records it: P-30 cuts a Release under the tag ruleset of the
-# repo that holds the Releases, which the canary is not. It is held to every rule of a fixture of the canary, with this
-# repo in the canary's place, and no other trial records this repo.
+# repo that holds the Releases, which the canary is not, and P-63 runs the eval job from its environment, which the
+# canary has not either. Each is held to every rule of a fixture of the canary, with this repo in the canary's place, and
+# no other trial records this repo.
 FACTORY = ORG + "/software-factory"
-TRIALS_OF_THE_FACTORY = {"P-30"}
+TRIALS_OF_THE_FACTORY = {"P-30", "P-63"}
 # A command-line read names a repo by --repo, -R or GH_REPO, with or without a space, an equals sign, quotes or the
 # site's address; by `gh repo <verb> owner/name`; or by a path or an address in its words.
 CLI_REPO = re.compile(r"(?:--repo|-R|GH_REPO)[ =]?[\"']?(?:https://github\.com/)?([A-Za-z0-9_.-]+)/([A-Za-z0-9_.-]+)"

tools/mutants.py

@@ -2656,11 +2656,15 @@ def moved_before(rel: str, block: str, anchor: str) -> tuple:
      "        if not actions and rel is not None:\n", "        if False:\n", ("tests.test_tools.Workflows.test_a_lens_run_by_actions_names_its_workflow_and_no_other_does",)),
     ("a Role may write through the release App", "tools/check_authority.py",
      'if key in ("controller", "reader", "release")}', 'if key in ("controller", "reader")}', ("tests.test_tools.Authority.test_a_role_writes_through_no_app_that_is_no_roles",)),
-    # P-30 runs in the Factory's own repo, the one trial that records it, and is held as a canary fixture is.
+    # P-30 and P-63 run in the Factory's own repo, the only trials that record it, each held as a canary fixture is.
     ("any trial may record the Factory's own repo", "tools/check_trials.py",
      'if repo == FACTORY and trial not in TRIALS_OF_THE_FACTORY:', 'if False:', ("tests.test_tools.Trials.test_a_cost_measurements_exchanges_name_each_repo_they_span",)),
     ("a trial of the Factory may record the canary", "tools/check_trials.py",
      'if trial in TRIALS_OF_THE_FACTORY and repo != FACTORY:', 'if False:', ("tests.test_tools.Trials.test_a_cost_measurements_exchanges_name_each_repo_they_span",)),
+    # Issue 309, decision 0173: the eval job's trial records this repo, beside P-30.
+    ("the eval job's trial is no trial of the Factory", "tools/check_trials.py",
+     'TRIALS_OF_THE_FACTORY = {"P-30", "P-63"}', 'TRIALS_OF_THE_FACTORY = {"P-30"}',
+     ("tests.test_tools.Trials.test_the_eval_jobs_trial_records_the_factorys_own_repo",)),
     ("a fixture of the Factory's repo is held to no repo it names", "tools/check_trials.py",
      'data.get("repo") in ("%s/%s" % (ORG, CANARY), FACTORY):', 'data.get("repo") == "%s/%s" % (ORG, CANARY):', ("tests.test_tools.Trials.test_a_cost_measurements_exchanges_name_each_repo_they_span",)),
     ("an App in a fixture of the Factory's repo is held to no reach", "tools/check_trials.py",