Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .github/scripts/probe_issue.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ def runtime_context(repository: str, issue_number: int, branch: str) -> str:

`run_tox` runs `tox -e format,lint,unit` only — not integration tests. \
Integration tests require a Juju controller and a packed `.charm` file. They \
run in per-charm CI after the PR is marked ready for review. You can write \
run in per-charm CI automatically when the PR is created. You can write \
integration tests and validate that they import and type-check via `run_tox` \
(lint runs pyright), but you cannot run them yourself. Write the test, \
validate with `run_tox`, and let CI confirm or refute.
Expand Down Expand Up @@ -404,8 +404,7 @@ def test_deploy(charm, juju: jubilant.Juju):
dependency in pyproject.toml, run_tox will uv lock and install it — make \
sure the version spec is valid.
9. After you exit, the workflow enforces the path allowlist and creates the \
PR as a draft. CI checks don't run until the reviewer marks the PR ready for \
review.
PR. CI runs after the reviewer approves the workflow runs.

### Version-dependent claims

Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/kepler.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: kepler

on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
types: [opened, synchronize, reopened]
paths:
- .github/workflows/kepler.yaml
- kepler/**
Expand All @@ -13,11 +13,6 @@ permissions: {}
jobs:
pack-and-test:
runs-on: ubuntu-latest
# Skip CI while the PR is a draft. CI runs when the PR is marked
# ready for review.
if: |
github.event_name != 'pull_request' ||
!github.event.pull_request.draft
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/kosmos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: kosmos

on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
types: [opened, synchronize, reopened]
paths:
- .github/workflows/kosmos.yaml
- kosmos/**
Expand All @@ -13,11 +13,6 @@ permissions: {}
jobs:
pack-and-test:
runs-on: ubuntu-latest
# Skip CI while the PR is a draft. CI runs when the PR is marked
# ready for review.
if: |
github.event_name != 'pull_request' ||
!github.event.pull_request.draft
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/meteor.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: meteor

on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
types: [opened, synchronize, reopened]
paths:
- .github/workflows/meteor.yaml
- meteor/**
Expand All @@ -13,11 +13,6 @@ permissions: {}
jobs:
pack-and-test:
runs-on: ubuntu-latest
# Skip CI while the PR is a draft. CI runs when the PR is marked
# ready for review.
if: |
github.event_name != 'pull_request' ||
!github.event.pull_request.draft
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/micron.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: micron

on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
types: [opened, synchronize, reopened]
paths:
- .github/workflows/micron.yaml
- micron/**
Expand All @@ -13,11 +13,6 @@ permissions: {}
jobs:
pack-and-test:
runs-on: ubuntu-latest
# Skip CI while the PR is a draft. CI runs when the PR is marked
# ready for review.
if: |
github.event_name != 'pull_request' ||
!github.event.pull_request.draft
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down
8 changes: 3 additions & 5 deletions .github/workflows/probe-issue.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -155,13 +155,11 @@ jobs:
--head "$branch" \
--title "$title" \
--body-file "$RUNNER_TEMP/reasoning.md" \
--draft 2>/dev/null || \
2>/dev/null || \
gh pr edit --repo "$REPOSITORY" "$branch" \
--title "$title" \
--body-file "$RUNNER_TEMP/reasoning.md")
echo "pr_url=$pr_url" >> "$GITHUB_OUTPUT"
# Comment on the PR telling the reviewer how to run CI.
gh pr comment "$pr_url" --body "This PR is a draft. Inspect the changes, then mark it ready for review to run CI."

- name: Comment on issue
if: always()
Expand All @@ -181,8 +179,8 @@ jobs:

**Blocker:** $blocker"
else
body="The doc-validation agent opened a draft PR: $PR_URL
body="The doc-validation agent opened a PR: $PR_URL

Review the PR, then mark it ready for review to run CI. Inspect the CI runs to determine whether the doc was validated or refuted."
CI will run after you approve the workflow runs. Inspect the CI runs to determine whether the doc was validated or refuted."
fi
gh issue comment "$ISSUE_NUMBER" --repo "$REPOSITORY" --body "$body"
12 changes: 6 additions & 6 deletions AGENT_DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The `run_tox` tool is the exception: it lets the agent trigger tox inside an iso
- Verify `.git/` was not modified (checks `git diff` and `git ls-files` for `.git/` paths). Reject if any `.git/` files were changed — this prevents the agent from planting hooks that would fire during `git add` or `git push`.
10. Configure git credentials using `GITHUB_TOKEN` — only now, after enforcement passes and the agent has exited.
11. `git add --all`, commit, push branch `probe/issue-<n>`.
12. `gh pr create --draft` with title `verify: <first line of reasoning>`, the agent's reasoning file as the PR body. The body does not include `Closes #<n>`. The PR is created as a draft so CI doesn't run automatically. Comment on the PR telling the reviewer to mark it ready for review to run CI.
12. `gh pr create` with title `verify: <first line of reasoning>`, the agent's reasoning file as the PR body. The body does not include `Closes #<n>`. GitHub requires approval before running CI workflows on PRs created by `GITHUB_TOKEN`.
13. Comment on the issue with the result (PR link, blocker, or failure message). This step always runs.

## Prompt composition
Expand Down Expand Up @@ -182,19 +182,19 @@ Even if the agent injected malicious commands into `tox.ini`, `pyproject.toml`,

The agent calls the tool on demand to validate its work: write code, call `run_tox`, see the output, fix issues, call again. This happens within the single agent session — no separate fix sessions are needed. After the agent exits, the workflow enforces the path allowlist and creates the PR.

`tox -e integration` is never run by this workflow. Integration tests require a Juju controller and are slow; they run in the per-charm CI workflows (`kepler.yaml`, etc.) after the PR is marked ready for review.
`tox -e integration` is never run by this workflow. Integration tests require a Juju controller and are slow; they run in the per-charm CI workflows (`kepler.yaml`, etc.) automatically when the PR is created or updated.

## CI gating on the PR
## CI on the PR

The per-charm CI workflows (`kepler.yaml`, `kosmos.yaml`, `meteor.yaml`, `micron.yaml`) trigger on `pull_request` activity types `opened`, `synchronize`, `reopened`, and `ready_for_review`. The job has a condition that skips if the PR is a draft (`if: !github.event.pull_request.draft`). The probe-issue workflow creates the PR as a draft (`gh pr create --draft`), so CI doesn't run when the PR is first created. The reviewer inspects the changes, then marks the PR as ready for review — this triggers the `ready_for_review` event, which runs CI.
The per-charm CI workflows (`kepler.yaml`, `kosmos.yaml`, `meteor.yaml`, `micron.yaml`) trigger on `pull_request` activity types `opened`, `synchronize`, and `reopened`. The probe-issue workflow creates the PR as a non-draft PR (`gh pr create` without `--draft`). Since the PR is created by a GitHub Actions workflow using `GITHUB_TOKEN`, GitHub requires approval before running CI workflows triggered by the PR — this provides the human review gate without needing draft mode.

This is necessary because the per-charm CI runs `tox -e unit` (which executes the agent's test code) and `tox -e integration` (which deploys the charm). The `run_tox` tool runs inside a container, but the per-charm CI runs on the runner directly. The draft gate ensures a human reviews the code before it executes outside the container.
The per-charm CI runs `tox -e unit` (which executes the agent's test code) and `tox -e integration` (which deploys the charm). The `run_tox` tool runs inside a container, but the per-charm CI runs on the runner directly. Since the PR is created by a GitHub Actions workflow using `GITHUB_TOKEN`, GitHub requires approval before running workflows triggered by the PR — this provides the human review gate without needing draft mode.

## Remaining risks

Agent reads committed secrets (e.g., a `.env` file in the repo): low. Cannot exfiltrate without bash or network. Don't commit secrets.

Agent makes subtle malicious changes (e.g., typosquat a dependency in `pyproject.toml`): medium. Mitigated by human PR review and dependency scanning. The `uv lock` step in the container regenerates the lockfile, but a malicious package in `pyproject.toml` would still be installed inside the container (without secrets). On the runner (after the PR is marked ready for review), the malicious package would execute with the runner's environment — but the reviewer inspects the PR before marking it ready.
Agent makes subtle malicious changes (e.g., typosquat a dependency in `pyproject.toml`): medium. Mitigated by human PR review and dependency scanning. The `uv lock` step in the container regenerates the lockfile, but a malicious package in `pyproject.toml` would still be installed inside the container (without secrets). On the runner, the malicious package would execute with the runner's environment — but GitHub requires approval before running workflows on PRs created by `GITHUB_TOKEN`, providing a human review gate.

Prompt injection from issue or docs content: low-medium. Mitigated by `<untrusted-content>` delimiters and system constraints. The agent can only edit files, which are reviewed.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To perform an adversarial test:

3. Run the [Probe issue](https://github.com/dwilding/basic-charms/actions/workflows/probe-issue.yaml) workflow, entering the issue number in the **Run workflow** UI.

4. Wait for a draft PR to be created.
4. Wait for a PR to be created.

The PR will modify one or more of the basic charms (and possibly their unit tests or integration tests) to test the documentation claim you described in the issue.

Expand All @@ -22,7 +22,7 @@ To perform an adversarial test:

> I believe `<claim>` is true. I added a test asserting it, which is expected to pass. If CI passes, the doc is correct.

5. Review the PR to make sure the agent's changes are meaningful and trustworthy. Then mark the PR ready for review to run the CI checks.
5. Review the PR to make sure the agent's changes are meaningful and trustworthy. Once you're satisfied, approve the PR's workflow runs to trigger CI.

6. After the CI checks have completed, use the PR description to draw a conclusion about the documentation.

Expand Down