codex-local 8/9: automated upstream sync + sync-upstream skill

.claude/skills/sync-upstream/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: sync-upstream
+description: Merge a new openai/codex rust-v* release into codex-local, repair conflicts and breakage within the allowed edit set, and escalate otherwise. Use for the upstream-sync issue/PR or when asked to sync upstream.
+---
+
+# sync-upstream
+
+Driver: `python3 local/scripts/sync_upstream.py <subcommand>`. Each prints ONE JSON
+object `{ok, step, conflicts[], build_errors[], test_failures[], surface_violations[], next_action}`.
+Exit codes: **0** ok, **1** needs repair (you fix it), **2** escalate (you stop).
+Always do what `next_action` says. Never guess past it.
+
+## Checklist
+
+1. `status` -> if `new_release` is false, stop (nothing to do). Else note `latest`.
+2. `merge <latest>` -> creates `upstream-sync/<tag>`, merges the tag, takes upstream's
+   `codex-rs/Cargo.lock` and regenerates it, bumps `local/UPSTREAM_VERSION`.
+   Exit 0: go to step 4. Exit 1: step 3. Exit 2: ESCALATE.
+3. `conflicts` -> repair each listed path by `kind`, `git add` it, rerun `conflicts` until
+   exit 0, then `git commit --no-edit`.
+   - `hook`: take UPSTREAM's version (`git checkout --theirs <path>`), then re-apply the hook
+     by following that hook's recipe in `local/FORK_SURFACE.md`. Keep the edit as small as the
+     recorded line count. Do not hand-merge the old hunk.
+   - `local`: keep ours unless the file only exists upstream-side.
+   - `generated` (Cargo.lock): `git checkout --theirs`, then `cargo update --workspace` in `codex-rs/`.
+   - `other`: ESCALATE (see below).
+4. `check` -> runs build, local crate tests, hook tests, `check_fork_surface.py`, and
+   the workflow check. Loop while exit is 1 (see Repair loop). Exit 0: go to step 5.
+5. `check > /tmp/check.json; report --check-json /tmp/check.json --out /tmp/report.md`.
+   Use `markdown` from the report as the PR body (success) or issue comment (failure).
+   Push the branch and open the PR to `codex-local/main` only when `check` exited 0.
+
+## Repair loop (exit 1 from `check`)
+
+Fix in this order, rerun `check` after each batch:
+- `build_errors`: upstream API drift. Typical causes: a hook call site no longer matches a
+  renamed/re-signatured upstream function, or a `codex-rs/local/*` crate uses a changed
+  upstream API. Read the compiler error, fix the call site (hook file) or the local crate.
+- `test_failures`: fix the local crate or re-apply the hook; never delete or skip a test.
+- `surface_violations`: a file outside the allowlist or over budget is modified. Revert it
+  to upstream's version (`git checkout <tag> -- <path>`). If a hook legitimately needs a
+  bigger edit, that is an escalation, not a budget bump.
+- `workflows_to_disable`: run `gh workflow disable <name> -R <owner/repo>`; not a code edit.
+
+## Allowed edits (nothing else)
+
+1. Allowlisted hook files (the `allow` list in `local/fork-surface.toml`), re-applied from the
+   `FORK_SURFACE.md` recipe onto upstream's version.
+2. `codex-rs/local/**` and `local/**`.
+3. Generated files (`codex-rs/Cargo.lock`).
+
+Never edit other upstream files to make things compile. Never add `codex-local-*`
+dependencies to upstream crates. Never lower tests, loosen the budget, or edit the allowlist
+to hide a violation.
+
+## Escalate (exit 2: stop, do not keep trying)
+
+- Any conflict with `kind: other` (outside the allowlist).
+- A hook's target symbol (named in `FORK_SURFACE.md`) no longer exists or moved so the recipe
+  cannot be applied as written.
+- `check` still exits 1 after 3 repair rounds, or the work needs more than ~150 changed lines
+  outside `codex-rs/local/**`.
+- Any command exits 2.
+
+To escalate: leave the branch as is, run `report` (with `--check-json` if you have one), and
+comment on the open `upstream-sync` issue with: the tag, the failing step, the JSON
+`conflicts`/errors, and which rule above triggered. Then stop.
+
+## Notes
+
+- `check` skips cargo steps if `cargo` is missing and says so under `skipped`; a skipped step
+  is not a pass (`check` exits 1). `check_fork_surface.py` missing is also reported as skipped.
+- Hook unit tests run if `FORK_SURFACE.md` has a line like ``Tests: `cargo test -p codex-core router` ``.
+- Exit 2 from `check`: a fork-surface budget violation or a non-allowlisted path. Escalate.
+- `check` exits 1 if any new upstream workflow could not be disabled (listed in
+  `workflows_to_disable`).
+- `--tag`/`merge <tag>` must match `^rust-v\d+\.\d+\.\d+$`; anything else exits 2.
+- **Secret `UPSTREAM_SYNC_TOKEN`** (PAT or GitHub App token with `workflows` scope): the
+  upstream-sync workflow uses it to push and open the PR when set. When a merge touches
+  `.github/workflows/**` and it is NOT set, the workflow fails early with a clear message
+  (GITHUB_TOKEN cannot push workflow-file changes). Otherwise it falls back to GITHUB_TOKEN.
+- Options: `--upstream-remote NAME` (default `upstream`, else `origin` if it is openai/codex);
+  `check --repo OWNER/NAME` disables new upstream workflows via `gh`.

.codex/skills/sync-upstream/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: sync-upstream
+description: Merge a new openai/codex rust-v* release into codex-local, repair conflicts and breakage within the allowed edit set, and escalate otherwise. Use for the upstream-sync issue/PR or when asked to sync upstream.
+---
+
+# sync-upstream
+
+Driver: `python3 local/scripts/sync_upstream.py <subcommand>`. Each prints ONE JSON
+object `{ok, step, conflicts[], build_errors[], test_failures[], surface_violations[], next_action}`.
+Exit codes: **0** ok, **1** needs repair (you fix it), **2** escalate (you stop).
+Always do what `next_action` says. Never guess past it.
+
+## Checklist
+
+1. `status` -> if `new_release` is false, stop (nothing to do). Else note `latest`.
+2. `merge <latest>` -> creates `upstream-sync/<tag>`, merges the tag, takes upstream's
+   `codex-rs/Cargo.lock` and regenerates it, bumps `local/UPSTREAM_VERSION`.
+   Exit 0: go to step 4. Exit 1: step 3. Exit 2: ESCALATE.
+3. `conflicts` -> repair each listed path by `kind`, `git add` it, rerun `conflicts` until
+   exit 0, then `git commit --no-edit`.
+   - `hook`: take UPSTREAM's version (`git checkout --theirs <path>`), then re-apply the hook
+     by following that hook's recipe in `local/FORK_SURFACE.md`. Keep the edit as small as the
+     recorded line count. Do not hand-merge the old hunk.
+   - `local`: keep ours unless the file only exists upstream-side.
+   - `generated` (Cargo.lock): `git checkout --theirs`, then `cargo update --workspace` in `codex-rs/`.
+   - `other`: ESCALATE (see below).
+4. `check` -> runs build, local crate tests, hook tests, `check_fork_surface.py`, and
+   the workflow check. Loop while exit is 1 (see Repair loop). Exit 0: go to step 5.
+5. `check > /tmp/check.json; report --check-json /tmp/check.json --out /tmp/report.md`.
+   Use `markdown` from the report as the PR body (success) or issue comment (failure).
+   Push the branch and open the PR to `codex-local/main` only when `check` exited 0.
+
+## Repair loop (exit 1 from `check`)
+
+Fix in this order, rerun `check` after each batch:
+- `build_errors`: upstream API drift. Typical causes: a hook call site no longer matches a
+  renamed/re-signatured upstream function, or a `codex-rs/local/*` crate uses a changed
+  upstream API. Read the compiler error, fix the call site (hook file) or the local crate.
+- `test_failures`: fix the local crate or re-apply the hook; never delete or skip a test.
+- `surface_violations`: a file outside the allowlist or over budget is modified. Revert it
+  to upstream's version (`git checkout <tag> -- <path>`). If a hook legitimately needs a
+  bigger edit, that is an escalation, not a budget bump.
+- `workflows_to_disable`: run `gh workflow disable <name> -R <owner/repo>`; not a code edit.
+
+## Allowed edits (nothing else)
+
+1. Allowlisted hook files (the `allow` list in `local/fork-surface.toml`), re-applied from the
+   `FORK_SURFACE.md` recipe onto upstream's version.
+2. `codex-rs/local/**` and `local/**`.
+3. Generated files (`codex-rs/Cargo.lock`).
+
+Never edit other upstream files to make things compile. Never add `codex-local-*`
+dependencies to upstream crates. Never lower tests, loosen the budget, or edit the allowlist
+to hide a violation.
+
+## Escalate (exit 2: stop, do not keep trying)
+
+- Any conflict with `kind: other` (outside the allowlist).
+- A hook's target symbol (named in `FORK_SURFACE.md`) no longer exists or moved so the recipe
+  cannot be applied as written.
+- `check` still exits 1 after 3 repair rounds, or the work needs more than ~150 changed lines
+  outside `codex-rs/local/**`.
+- Any command exits 2.
+
+To escalate: leave the branch as is, run `report` (with `--check-json` if you have one), and
+comment on the open `upstream-sync` issue with: the tag, the failing step, the JSON
+`conflicts`/errors, and which rule above triggered. Then stop.
+
+## Notes
+
+- `check` skips cargo steps if `cargo` is missing and says so under `skipped`; a skipped step
+  is not a pass (`check` exits 1). `check_fork_surface.py` missing is also reported as skipped.
+- Hook unit tests run if `FORK_SURFACE.md` has a line like ``Tests: `cargo test -p codex-core router` ``.
+- Exit 2 from `check`: a fork-surface budget violation or a non-allowlisted path. Escalate.
+- `check` exits 1 if any new upstream workflow could not be disabled (listed in
+  `workflows_to_disable`).
+- `--tag`/`merge <tag>` must match `^rust-v\d+\.\d+\.\d+$`; anything else exits 2.
+- **Secret `UPSTREAM_SYNC_TOKEN`** (PAT or GitHub App token with `workflows` scope): the
+  upstream-sync workflow uses it to push and open the PR when set. When a merge touches
+  `.github/workflows/**` and it is NOT set, the workflow fails early with a clear message
+  (GITHUB_TOKEN cannot push workflow-file changes). Otherwise it falls back to GITHUB_TOKEN.
+- Options: `--upstream-remote NAME` (default `upstream`, else `origin` if it is openai/codex);
+  `check --repo OWNER/NAME` disables new upstream workflows via `gh`.

.github/workflows/local-ci.yml

@@ -77,10 +77,12 @@ jobs:
           workspaces: codex-rs
       - name: Local crate tests
         run: cargo test -p codex-local-cli -p codex-local-models -p codex-local-resume
-      - name: Hook test, tool-name unflatten
-        run: cargo test -p codex-core --lib unflatten_tool_name
-      - name: Hook test, model provider factory
-        run: cargo test -p codex-model-provider local_models_factory
+      - name: Hook tests (Tests lines in local/FORK_SURFACE.md)
+        run: |
+          set -euo pipefail
+          grep -E '^[^A-Za-z0-9]*Tests?:' ../local/FORK_SURFACE.md \
+            | sed -E 's/.*`(cargo test [^`]+)`.*/\1/' \
+            | while read -r cmd; do echo "+ ${cmd}"; ${cmd}; done
       - name: Build codex-local
         run: cargo build --bin codex-local -p codex-local-cli
       - name: verify_install --quick

.github/workflows/upstream-sync.yml

@@ -0,0 +1,195 @@
+# upstream-sync: merges the newest upstream rust-v* release into codex-local/main via PR.
+#
+# Secret UPSTREAM_SYNC_TOKEN (optional but needed for some merges): a PAT or GitHub App token
+# with `contents`, `pull requests` and `workflows` scope. Used to push the sync branch and open
+# the PR. GITHUB_TOKEN cannot push changes to .github/workflows/**, so when the upstream range
+# touches workflow files and this secret is unset, the run fails early with a clear message.
+# When the upstream range does not touch workflows, GITHUB_TOKEN is used as a fallback.
+name: upstream-sync
+
+on:
+  schedule:
+    - cron: "17 6 * * *"
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Upstream tag to sync (default: newest stable rust-v*)"
+        required: false
+        type: string
+      force:
+        description: "Sync even if the tag is not newer than local/UPSTREAM_VERSION"
+        required: false
+        type: boolean
+        default: false
+
+permissions:
+  contents: write
+  pull-requests: write
+  issues: write
+  actions: write
+
+concurrency:
+  group: upstream-sync
+  cancel-in-progress: false
+
+env:
+  BASE: codex-local/main
+
+jobs:
+  sync:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 120
+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: codex-local/main
+          fetch-depth: 0
+          # No persisted GITHUB_TOKEN auth header: it would override the
+          # UPSTREAM_SYNC_TOKEN credentials used by the push step below.
+          persist-credentials: false
+
+      - name: Configure git and upstream remote
+        run: |
+          git config user.name "codex-local upstream-sync"
+          git config user.email "upstream-sync@users.noreply.github.com"
+          git remote add upstream https://github.com/openai/codex.git
+          git fetch upstream --tags --quiet
+
+      - name: Status
+        id: status
+        env:
+          TAG_INPUT: ${{ inputs.tag }}
+          FORCE_INPUT: ${{ inputs.force }}
+        run: |
+          set -o pipefail
+          args=()
+          [ -n "$TAG_INPUT" ] && args+=(--tag "$TAG_INPUT")
+          [ "$FORCE_INPUT" = "true" ] && args+=(--force)
+          python3 local/scripts/sync_upstream.py status "${args[@]}" | tee status.json
+          echo "new=$(jq -r .new_release status.json)" >> "$GITHUB_OUTPUT"
+          echo "tag=$(jq -r .latest status.json)" >> "$GITHUB_OUTPUT"
+
+      - name: Require UPSTREAM_SYNC_TOKEN when workflows change
+        if: steps.status.outputs.new == 'true'
+        env:
+          SYNC_TOKEN: ${{ secrets.UPSTREAM_SYNC_TOKEN }}
+          TAG: ${{ steps.status.outputs.tag }}
+        run: |
+          changed="$(git diff --name-only "$(git merge-base HEAD "$TAG")" "$TAG" -- .github/workflows)"
+          if [ -n "$changed" ] && [ -z "$SYNC_TOKEN" ]; then
+            echo "::error title=UPSTREAM_SYNC_TOKEN required::Upstream $TAG changes .github/workflows/** but secret UPSTREAM_SYNC_TOKEN (PAT or App token with workflows scope) is not set; GITHUB_TOKEN cannot push workflow changes. Add the secret and re-run."
+            printf 'Workflow files changed upstream:\n%s\n' "$changed"
+            exit 1
+          fi
+
+      - name: Install Rust toolchain
+        if: steps.status.outputs.new == 'true'
+        run: |
+          channel=$(sed -n 's/^channel *= *"\(.*\)"/\1/p' codex-rs/rust-toolchain.toml)
+          rustup toolchain install "$channel" --profile minimal -c clippy -c rustfmt
+          rustup default "$channel"
+
+      - uses: Swatinem/rust-cache@v2
+        if: steps.status.outputs.new == 'true'
+        with:
+          workspaces: codex-rs
+
+      - name: Merge
+        id: merge
+        if: steps.status.outputs.new == 'true'
+        continue-on-error: true
+        env:
+          TAG: ${{ steps.status.outputs.tag }}
+        run: |
+          set +e
+          python3 local/scripts/sync_upstream.py merge "$TAG" | tee merge.json
+          echo "code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Check
+        id: check
+        if: steps.status.outputs.new == 'true' && steps.merge.outputs.code == '0'
+        run: |
+          set +e
+          python3 local/scripts/sync_upstream.py check --repo "$GITHUB_REPOSITORY" | tee check.json
+          echo "code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
+          exit 0
+
+      - name: Report
+        id: report
+        if: steps.status.outputs.new == 'true'
+        env:
+          TAG: ${{ steps.status.outputs.tag }}
+          MERGE_CODE: ${{ steps.merge.outputs.code }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          # merge.json is the report input when the merge itself failed.
+          src=check.json; [ "$MERGE_CODE" = "0" ] || src=merge.json
+          if [ "$MERGE_CODE" != "0" ]; then
+            # conflicts left in the index: record them, abort the half-done merge
+            git merge --abort 2>/dev/null || true
+            git checkout -q "$BASE" 2>/dev/null || true
+          fi
+          python3 - "$src" <<'PY'
+          import json, os, sys
+          d = json.load(open(sys.argv[1]))
+          lines = [f"## Upstream sync: {os.environ['TAG']}", "",
+                   f"Result: {'PASS' if d['ok'] else 'FAIL'} (step {d['step']})", ""]
+          for k in ("conflicts", "build_errors", "test_failures", "surface_violations"):
+              lines += [f"### {k}"] + ([f"- `{x}`" for x in d[k]] or ["- none"]) + [""]
+          lines += [f"Next action: {d['next_action']}", "",
+                    f"Run: {os.environ['RUN_URL']}"]
+          open("report.md", "w").write("\n".join(lines) + "\n")
+          PY
+
+      - name: Open PR (success)
+        if: steps.status.outputs.new == 'true' && steps.merge.outputs.code == '0' && steps.check.outputs.code == '0'
+        env:
+          TAG: ${{ steps.status.outputs.tag }}
+          SYNC_TOKEN: ${{ secrets.UPSTREAM_SYNC_TOKEN }}
+        run: |
+          tag="$TAG"
+          branch="upstream-sync/$tag"
+          # PAT/App token (workflows scope) when set, else GITHUB_TOKEN.
+          token="${SYNC_TOKEN:-$GH_TOKEN}"
+          export GH_TOKEN="$token"
+          git push "https://x-access-token:${token}@github.com/${GITHUB_REPOSITORY}.git" "$branch"
+          # Supersede older sync PRs.
+          gh pr list --state open --json number,headRefName \
+            --jq ".[] | select(.headRefName | startswith(\"upstream-sync/\")) | select(.headRefName != \"$branch\") | .number" |
+            while read -r n; do
+              gh pr close "$n" --delete-branch --comment "Superseded by $tag."
+            done
+          gh pr create --base "$BASE" --head "$branch" \
+            --title "Upstream sync: $tag" --body-file report.md
+
+      - name: Open or update issue (failure)
+        if: steps.status.outputs.new == 'true' && (steps.merge.outputs.code != '0' || steps.check.outputs.code != '0')
+        env:
+          TAG: ${{ steps.status.outputs.tag }}
+          MERGE_CODE: ${{ steps.merge.outputs.code }}
+          SYNC_TOKEN: ${{ secrets.UPSTREAM_SYNC_TOKEN }}
+        run: |
+          tag="$TAG"
+          branch="upstream-sync/$tag"
+          token="${SYNC_TOKEN:-$GH_TOKEN}"
+          export GH_TOKEN="$token"
+          # Keep the repair branch only when the merge itself completed.
+          if [ "$MERGE_CODE" = "0" ]; then
+            git push --force "https://x-access-token:${token}@github.com/${GITHUB_REPOSITORY}.git" "$branch"
+          fi
+          gh pr list --state open --json number,headRefName \
+            --jq ".[] | select(.headRefName | startswith(\"upstream-sync/\")) | select(.headRefName != \"$branch\") | .number" |
+            while read -r n; do
+              gh pr close "$n" --delete-branch --comment "Superseded by $tag."
+            done
+          gh label create upstream-sync --color 0e8a16 --description "Upstream sync tracking" 2>/dev/null || true
+          n=$(gh issue list --label upstream-sync --state open --json number --jq '.[0].number')
+          if [ -n "$n" ]; then
+            gh issue edit "$n" --title "Upstream sync needs repair: $tag"
+            gh issue comment "$n" --body-file report.md
+          else
+            gh issue create --label upstream-sync --title "Upstream sync needs repair: $tag" --body-file report.md
+          fi