Skip to content

refactor(agent-harness): share ACP probe eligibility - #15481

Merged
bolichen97 merged 1 commit into
mainfrom
feat/share-acp-eligibility
Oct 5, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/share-acp-eligibility

Conversation

@kyleseaman

Copy link
Copy Markdown
Contributor

Stacked on #13888. Base is feat/onboarding-other-agents; this PR's diff is one commit on top of it. Retarget to main and rebase once #13888 merges.

Problem / Motivation

Goal: First-run setup and Settings > Agent Harness always agree on what an ACP probe row says about an agent, because both read one shared verdict.

#13888 reads each GET /api/acp-backends row's installed, restart_required and selectable fields separately in the first-run gate (KiroPrerequisiteGate.tsx) and in Settings > Agent Harness (AgentBackendTab.tsx). The two agree today only by convention, so a later edit to one could make one surface offer an agent the other disables (#14517).

Why it matters

A disagreement would show up as an agent that is usable in Settings but refused at setup, or the reverse, with no signal to the reader which screen is right. Setup completion is durable, so a drift in the gate's direction can record setup as finished for an agent that cannot start.

Not a goal

What changed (motivation → approach → change)

Two copies of the probe-row rule → one shared state with two readings, because the surfaces deliberately differ in how much proof they need:

  • acpProbeState(probe) in website/src/api/acpBackend.ts: unprobed / unselectable / missing / restart_required / unknown / installed, in that precedence. Mirrors the gateway's marker rule (installed != INSTALLED or restart_required).
  • acpProbeBlocksUse(probe): a KNOWN block (unselectable, missing, restart_required). Settings disables Use on these; a failed or absent check stays live, as before.
  • acpProbeConfirmsUse(probe): POSITIVE proof (installed). The first-run gate requires this before it lets setup finish, as before.

configuredBackendCanStart (gate) and cannotUse (Settings) now call these instead of reading the fields inline. Folding selectable: false into Settings' cannotUse is behaviour-neutral: an unselectable agent is either filtered out of offered or shown build-excluded with no Use button, so no rendered button changes state.

Backwards compatibility

Compatible: frontend-only refactor; no API, config or copy changes, and every existing test on both surfaces passes unchanged.

Tests

  • website/src/test/acpBackend.eligibility.test.ts (new): the state table (no row, installed, missing, check failed, restart required, unselectable) for all three helpers, that no row both blocks and confirms, and the precedence order.
  • AgentBackendTab.test.tsx and KiroPrerequisiteGate.test.tsx: the same states rendered on each surface. Settings: Use enabled for installed and check-failed, disabled for missing and restart-required, unselectable not listed. Setup: Use enabled only for installed, unselectable not listed.
  • Mutation check: swapping the gate to acpProbeBlocksUse fails at least 5 gate tests (including the new check-failed row); swapping Settings to acpProbeConfirmsUse fails at least 8 Settings tests (including the new check-failed row). Both restored.
  • Ran: the three files (318 tests), tsc, and eslint on the changed files. Not run: the full frontend suite.

Manual verification

N/A — no rendered change; the rendered-state tests on both surfaces cover each probe state.

Screenshots / video

Why no screenshot: pure refactor of the eligibility check behind both screens; every state renders exactly as on #13888, which the per-surface tests pin.

Related Issues

Closes #14517. Stacked on #13888.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

N/A — placeholder pending OSPO wording.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ed4533beaaf6e54f63a7c341f20729001c0af631 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ed4533b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ed4533beaaf6e54f63a7c341f20729001c0af631: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ✅ PASS

UX-level review of ed4533beaaf6e54f63a7c341f20729001c0af631 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Confirm the unselectable-row claim holds in a real build: a selectable: false harness never renders a Use button in Settings — the tests pin it, a quick manual look seals it.

[UX-REVIEWED] ed4533b

Description read

[DESCRIPTION-READ] 321a104959364ac7f5ed8acfe2fb9b566dbf8f901aebf58343a20400a3a0b022

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of ed4533beaaf6e54f63a7c341f20729001c0af631 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] ed4533b

Description read

[DESCRIPTION-READ] 6c47233b35cc2340249e1f24181963f8110c101d19b59e8efb7c561194f506a9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of ed4533beaaf6e54f63a7c341f20729001c0af631 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the selectable: false fold in Settings' cannotUse with one full-suite run — the author ran only three test files; neutrality holds by reading, not by a run.

Not justified as shipped

  • Item 4 — undeclared, rides along: AcpBackendProbe now imported from ./client/config instead of ./client; harm-free (both export it), the description never mentions it.

What this change ships

Inventory (5 items) — 4 justified

Intent: FIX — stop first-run setup and Settings reading the same ACP probe row by two private conventions that could drift (#14517, linked).

  1. One shared probe-row verdict (acpProbeState + two predicates) both screens now read — justified
  2. First-run gate's "can start" check reads the shared strict verdict, same outcome as before — justified
  3. Settings' Use-disable check reads the shared block verdict, now including selectable: false — justified (verified: a rendered Use button requires offered/non-current, which already excludes unselectable rows; AgentDetail.tsx:77 omits the button entirely)
  4. AcpBackendProbe imported from ./client/config instead of ./client — undeclared, rides along
  5. The row-state table pinned in one new test file plus the same states rendered per surface — justified

The rewritten cannotUse comment ("Selectability is deliberately NOT part of this") is not a reversed pin: the description declares the fold, and the pin's own rationale — only actionable reasons disable a rendered button — is preserved and still true in the new comment and code.

[FIRST-PRINCIPLES-REVIEWED] ed4533b

Description read

[DESCRIPTION-READ] 9950a3361c00fd60114777007e0617c6167da1f7dbf0278adaee84f6fa3ff1c2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed ed4533beaaf6e54f63a7c341f20729001c0af631 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ed4533b

Verdict parsed from the review's SHA-scoped output markers for commit ed4533beaaf6e54f63a7c341f20729001c0af631.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ed4533beaaf6e54f63a7c341f20729001c0af631: <one-sentence reason>

@kyleseaman
kyleseaman force-pushed the feat/share-acp-eligibility branch from fd50ada to fa76c83 Compare September 30, 2026 17:04
@kyleseaman
kyleseaman force-pushed the feat/share-acp-eligibility branch from fa76c83 to df10fde Compare September 30, 2026 20:37
@kyleseaman
kyleseaman force-pushed the feat/share-acp-eligibility branch from df10fde to 0803232 Compare October 1, 2026 13:49
@kyleseaman
kyleseaman force-pushed the feat/share-acp-eligibility branch from 0803232 to f3b0514 Compare October 1, 2026 19:36
@kyleseaman
kyleseaman force-pushed the feat/share-acp-eligibility branch from f3b0514 to de1b8aa Compare October 2, 2026 13:54
@kyleseaman
kyleseaman force-pushed the feat/share-acp-eligibility branch from de1b8aa to 0ef5bf8 Compare October 2, 2026 14:42
@kyleseaman
kyleseaman marked this pull request as ready for review October 2, 2026 16:01
@kyleseaman
kyleseaman requested a review from a team as a code owner October 2, 2026 16:01
@kyleseaman
kyleseaman requested review from bolichen97 and removed request for a team October 2, 2026 16:01
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@kyleseaman
kyleseaman added this pull request to stack #16184 October 2, 2026 16:03
iamwhatever
iamwhatever previously approved these changes Oct 2, 2026
Base automatically changed from feat/onboarding-other-agents to main October 2, 2026 16:38
@iamwhatever
iamwhatever requested a review from a team October 2, 2026 16:38
…ng and Settings

First-run setup and Settings > Agent Harness each read an ACP probe row's
install, restart and selectable fields on their own, and agreed only by
convention. Both now decide from one shared verdict in api/acpBackend.ts:

- acpProbeState: unprobed / unselectable / missing / restart_required /
  unknown / installed, in that precedence.
- acpProbeBlocksUse: a known block. Settings disables Use on these; a
  failed or absent check stays live.
- acpProbeConfirmsUse: positive proof. First-run setup requires it, since
  finishing setup is durable.

Onboarding keeps its host sandbox, independent_setup and KAS checks local;
Settings keeps the config schema's selectable set local. No behaviour change.

Closes #14517
@iamwhatever
iamwhatever force-pushed the feat/share-acp-eligibility branch from 0ef5bf8 to ed4533b Compare October 2, 2026 16:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Two non-blocking notes after reading this against the gateway side (dashboard/handlers/acp_backend_status.py and agent_sdk/backend_install.py). Approval stands.

1. acpProbeState is fail-open where the old gate was fail-closed.
The old gate refused anything where probe.installed !== 'installed'. The new acpProbeState only checks for 'missing' and 'unknown', then ends with return 'installed'. So a payload with an absent or unexpected installed value now reads as installed, and acpProbeConfirmsUse lets first-run setup finish and record the durable marker. The AcpBackendInstalled type has three values, so this is not reachable from a current gateway. It does differ from the stated rule, "mirrors the gateway's marker rule (installed != INSTALLED or restart_required)", which refuses anything that is not exactly installed. A one-line hardening:

if (probe.installed !== 'installed') return 'unknown'

in place of the fall-through, plus one test row with an unexpected installed value expecting unknown and confirms: false.

2. The new comments say more than the probe proves.
acpProbeState documents installed as "confirmed installed and spawnable", and acpProbeConfirmsUse as "POSITIVELY confirms the harness can start". The gateway probe only checks that the harness binaries resolve on disk. It does not run them and reads no credential (backend_install.py says so in its module docstring), so an installed but signed-out harness still passes and then fails at session/new. Behavior is unchanged by this PR, but I would reword those two comments to "binaries resolve on this machine", so the next reader does not treat the setup gate as a start guarantee.

Also worth knowing, no action needed: the gateway caches each verdict for 30 seconds, and restart_required is never set for the kiro and kas rows.

Not verified: I did not run the suites, and I did not read kiro_prerequisite.py, so I have not confirmed the gateway marker rule matches the quoted wording.

🤖 AI-assisted comment 🤖

@bolichen97
bolichen97 merged commit 53a3c29 into main Oct 5, 2026
97 checks passed
@bolichen97
bolichen97 deleted the feat/share-acp-eligibility branch October 5, 2026 06:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Share ACP probe eligibility between onboarding and Agent Settings

3 participants