Skip to content

refactor: split source-provider handler into responsibility owners - #14543

Merged
chenmingwei23 merged 1 commit into
mainfrom
refactor/b01-source-providers
Sep 28, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
refactor/b01-source-providers

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Maintainers can read and change one part of the pull-request / issue source panel's backend without working through a 7,925-line file, and every caller, patch and HTTP response behaves exactly as before.

src/kiro_crew/dashboard/handlers/source_providers.py held everything behind the Changes and Issues panels in one module:

  • URL validation and the GitLab/Jira allowlists
  • the sandboxed gh/glab runner
  • GitHub, GitLab and Jira reads
  • the Atlassian-document-to-markdown converter
  • the full-payload cache and the sidebar chip cache
  • the chip refresh loop
  • every owner-authenticated write

It is one of the most-patched files in the test suite. A change to any one of these parts meant reading all of them, and every open PR in this area collided on the same file.

Why it matters

Every source-provider change pays for the size today: review cost, merge conflicts across the many PRs that touch this file, and the risk of editing a security-relevant rule (redaction, host pinning, the owner gate) without seeing where else it applies. This split is also the first step of the dashboard spec chain B01 → B11 → B12. The Cron and AutoNudge refactors start from a base that contains it.

Not a goal

  • Changing any behaviour: HTTP status, body, error code, audit event, cache TTL, redaction or subprocess bound.
  • Changing the frontend, or any other module that imports from the handler.
  • Editing the unclaimed docs and comments that cite the old file path (listed under Related Issues as follow-ups for their owners).
  • Redesigning the caches, the chip/full-payload invalidation protocol, or the plugin seam.
  • Removing the compatibility surface. Every name the handler defined, and every import callers reach through it, stays importable and patchable there.

What changed (motivation → approach → change)

The source panel's backend is now sixteen modules, each named for the one thing it owns, so changing one part means reading only that part. Users and callers see no difference. Each responsibility has one owner module under src/kiro_crew/dashboard/source_providers/, and the code moved there verbatim: an AST comparison against the base shows every moved definition is identical once sibling qualification is ignored. The handler file keeps the /api/source/* routes, their audit and error mapping, and the dashboard owner gate.

Owner Holds
contract refs, payload shapes, SourceProviderPlugin, errors
sanitize the redaction chokepoint and the payload byte cap
hosts GitLab/Jira allowlists and the chip switch
plugins the registered-provider seam, chip labels, path markers, search seam
links URL validation
runner the bounded, audited, sandboxed gh/glab run
projection the payload, chip and merge-state vocabulary every read shares
github, gitlab, jira provider reads
adf Jira ADF to markdown
chip_status the chip cache, status deltas, the public-repo visibility gate
cache coalesced full, checks, issue and contributor reads, admission, revalidation, invalidation
chip_refresh background chip reads
mutations, review owner writes

Callers and tests patch these names through the handler, so the handler reuses the security package's re-export pattern (#13933). _OWNED_NAMES/_EXPORTS map each moved name to the module that holds it. A read resolves the owner from sys.modules on each access, and _ReExportModule forwards a write or delete to that owner. A patch through the handler therefore reaches every caller. For that to hold, owners reach a sibling as a module (runner._run_json(...)) and copy only types. Owners import only owners earlier in the table order, so the owners form no import cycle.

Two shared entries follow the move, as granted by the campaign coordinator.

  • NON_EGRESS_REDACTION_MODULES in src/kiro_crew/security_posture.py lists the five owners that call a redactor (adf, cache, chip_refresh, review, sanitize) in place of the handler, which no longer calls one.
  • test_spawn_audit's agent-influenced key now names dashboard/source_providers/runner.py::_run_provider, the function that spawns the child. The old …::_run_json key named a function with no spawn in it, so that check could not fail.

The learn-cron-dashboard.md bullet now carries the owner map.

Backwards compatibility

Compatible. Every one of the 417 names the handler exposed still resolves at kiro_crew.dashboard.handlers.source_providers, with the same object. Writes and deletes through it reach the owning module. Every HTTP route, status, body and error code is unchanged. The only intentional differences:

  • Log records come from owners that use the handler's logger name.
  • json and time are plain imports on the handler rather than forwarded, because several owners import their own.

Tests

  • test_source_providers_refactor_handlers.py, ..._visibility.py and ..._jira.py are new characterization tests. They pin every handler's status/body/code/audit mapping, the public-repo visibility gate, and the Jira failure messages and request shape. All of them passed against the unsplit base before the move.

  • test_source_providers_refactor_facade.py pins the surface and the protocol:

    • all 417 historical names resolve;
    • each name has exactly one home, and a read answers from its owner;
    • a write or delete through the handler reaches the owner;
    • a patch reaches a caller in another owner;
    • no handler reads a moved name bare;
    • owners copy only types and import only earlier owners;
    • owners log under the historical name.

    Each check was shown to fail when its property is broken: an export was dropped, forwarding was removed, github was made to copy _run_json, and a handler was made to read a name bare.

  • Two existing owned tests read the module that holds the code they check, since the handler no longer holds it:

    • test_source_providers.py: the ADF chokepoint scan reads the converter's module.
    • test_source_provider_plugin.py: the check that the transcript-search collector keeps no spelling cap of its own asks the collector's module as well as the handler. Adding that constant to plugins turns it red.
  • Existing: the 707 owned tests and 69 files of adjacent tests (source links, session cards, the owner gate, security posture, error codes, spawn audit, mirrored-owner storage) pass. The only edits to them are the two above and the one-key test_spawn_audit change.

  • On Python 3.12 (CI's version):

    • the 7 owned files at this head: 826 passed;
    • mypy src/kiro_crew/: clean, and the adjacent and guard files: 5,517 passed, 16 skipped, on production code identical to this head.
  • scripts/local-gate.py --base f57c1dee --dry-run selects 2,251 backend and 1,175 frontend related files, because more than 100 test files import or patch this handler. That is beyond this host's shared test budget, so, with the campaign coordinator's approval, the suites above ran locally and CI owns the full related set. The diff touches no website/** path.

Manual verification

Built the wheel and installed it into a separate target. The handler and all 16 owners imported from the installed package, and dashboard.routes.chat imported against it. Importing each owner module first in a fresh interpreter also works. There is no UI change.

Related Issues

no linked issue: campaign scope B01 (source-provider core refactor).

These unclaimed files cite code that moved. They still resolve through the handler, and the campaign coordinator logs them as follow-ups for their owners:

  • docs/architecture/resource-protection.md:88, :419 → source_providers/runner.py
  • docs/system-specs/modules/security.md:1960 (_run_json() audit) → source_providers/runner.py; the owner gate stays in the handler
  • src/kiro_crew/hooks.py:4411, src/kiro_crew/sandbox.py:13930 → source_providers/runner.py
  • src/kiro_crew/history_search.py:543 (provider registry) → source_providers/plugins.py
  • test/fixtures/adf_markdown_safety.json _comment and website/src/test/MarkdownRenderer.adfSafety.test.tsx:18 (escape set) → source_providers/adf.py
  • website/docs/extension-seams.md:537, :557 → source_providers/plugins.py, source_providers/contract.py

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The 7,925-line dashboard/handlers/source_providers.py held URL
validation, the provider-CLI runner, three providers' reads, the ADF
converter, two caches, the chip refresh loop and every write. Each now
has one owner module under dashboard/source_providers/, moved verbatim.

The handler keeps the /api/source/* routes, their audit and error
mapping, and the dashboard owner gate. Every other name it defined still
resolves there. Reads come from the owner and writes are forwarded to
it, reusing the security package's re-export pattern (#13933), so
existing imports and patches still reach the running code. Owners reach
each other as modules and import only earlier owners, so the owners
form no cycle.

The owners that redact provider data join security_posture's non-egress
allowlist in place of the handler. The spawn-audit tripwire now names
runner._run_provider, the function that actually spawns. The dashboard
spec records the owner map.
@bolichen97
bolichen97 requested a review from a team as a code owner September 27, 2026 22:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify CI green on the full 2,251-file related set the local gate could not run — local runs covered only the owned and adjacent suites.

[DESIGN-REVIEWED] 84ffe73

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify CI's full related-test set is green on this head: the author locally ran only the owned and adjacent suites, deferring the 2,251-file set to CI.

Not justified as shipped

  • Item 7 — rides along: the spawn-audit key change goes beyond the forced path edit — runner.py::_run_json would still pass trivially; pointing it at _run_provider (verified: the spawn is there, runner.py:476,512) makes a dead check live. Declared and coordinator-granted; no premise risk.

What this change ships

Inventory (10 items) — 9 justified

Intent: split the 7,925-line source-provider handler into per-responsibility modules with zero caller-visible change — a refactor (ADDITION of structure, not a FIX).

  1. The source panel's backend moved verbatim into 16 owner modules; the handler keeps routes, audit/error mapping, owner gate — justified
  2. Every historical name still imports, patches and writes through the handler, forwarded live to its owner — justified
  3. Sixteen new importable module paths plus a package __init__ (LOGGER_NAME) — justified
  4. Owners log under the handler's historical logger name — justified
  5. json/time are the handler's own attributes, no longer forwarded — justified
  6. Non-egress redaction list swaps the handler entry for the five redacting owners — justified
  7. Spawn-audit key now names the function that actually spawns — rides along: declared; makes a dead check live
  8. The dashboard spec bullet carries the owner map, same commit — justified
  9. Four new characterization/facade test files pin the surface and forwarding protocol — justified
  10. ADF chokepoint scan reads the converter's defining module instead of the handler source — justified

The forwarding class is the repo's established per-package pattern (8th instance; 7 exist in security/, config/, dashboard/, diag/, crew_log/, stt/, mcp_gateway/, precedent #13933) — not relitigated.

[FIRST-PRINCIPLES-REVIEWED] 84ffe73

@bolichen97 bolichen97 added the posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only label Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 84ffe73

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Cross-Platform Portability — disposition for the three flagged added lines (posix-only-approved, applied by the campaign coordinator)

The gate reads added lines. All three are verbatim moves out of src/kiro_crew/dashboard/handlers/source_providers.py at base f57c1dee. Neither is a filesystem path, so pathlib/os.path.join would be the wrong fix:

  • source_providers/runner.py _PROVIDER_SYSTEM_PATH = "/usr/bin:/bin:/usr/sbin:/sbin" (base L144). This is the fixed POSIX PATH handed to the sandboxed gh/glab child. On Windows no sandbox backend exists, so sandboxed_spawn_argv refuses the spawn before PATH matters.
  • source_providers/links.py best[0].split("/") (base L1181). It splits the GitLab project path taken from a URL to reject ./../empty segments. URL paths always use /.
  • source_providers/review.py _segments_match(pattern.split("/"), ref.split("/")) (base L6284). It splits git branch names and branch-protection patterns, which use / on every OS.

The gate documents the label as the escape hatch for exactly this case. Rewording the lines only to dodge the regex would be the spelling-chase AGENTS.md rejects, so the code stays as it was.

@bolichen97

bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=a14339ea5877 Item 7 — spawn-audit key names the function that spawns (rebutted: keep)

Moving _run_json out of the handler leaves the old key dashboard/handlers/source_providers.py::_run_json naming nothing. It was already vacuous at base, because the spawn sits in _run_provider.
Keeping it would leave test_agent_influenced_sites_are_routed unable to fail for the source-provider spawn.
runner.py::_run_provider is in _collect_spawn_functions(), so if routing is ever removed the check goes red.
It is a one-line, coordinator-granted entry edit that the move forces, with no behaviour change.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106 — this comment is updated in place on each push.

Review details

Non-blocking: the relocated GitLab merge-state re-read omits host=, so _run_provider refuses it on every GitLab instance.

FINDING — src/kiro_crew/dashboard/source_providers/gitlab.py:135 — data = await runner._run_json("glab", "api", mr_api) passes no host, so runner._run_provider hits if not host: (runner.py:404) and raises SourceProviderError("a GitLab host is required for glab calls"), which the loop's except SourceProviderError: break swallows — an MR whose first read returns an unsettled detailed_merge_status (e.g. "checking") is therefore reported to the panel as mergeable/mergeStateStatus = "unknown" after a wasted 0.8 s sleep, plus a misleading denied/host_not_specified SEL provider-CLI event; every sibling glab call in the file passes host=ref.host → Fix: data = await runner._run_json("glab", "api", mr_api, host=ref.host).

[OPUS-REVIEWED] 84ffe73

Verdict parsed from the review's SHA-scoped output markers for commit 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 84ffe73a84f75eb7262aac6eea2cb4f4bdb8e106: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Genuine, and pre-existing: the same call sits unchanged at base, src/kiro_crew/dashboard/handlers/source_providers.py:2320 @ f57c1dee.
This PR moves it verbatim into source_providers/gitlab.py; the AST comparison against base shows it unchanged.
Adding host=ref.host changes behaviour: the re-read would start reaching the provider. This behaviour-preserving refactor's frozen goal excludes that.
Tracked in #14556 (deferred-finding, assignee, Due 2026-10-11), with the one-line fix and a test that pins it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 28, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 28, 2026 00:27
@chenmingwei23
chenmingwei23 merged commit 2b7cb3e into main Sep 28, 2026
113 of 115 checks passed
@chenmingwei23
chenmingwei23 deleted the refactor/b01-source-providers branch September 28, 2026 00:33
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 28, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

Fixed in the follow-up #14603 (03acd53), because this PR merged at the reviewed head before the repair was pushed.
The write and delete tests in test/test_source_providers_refactor_facade.py no longer call monkeypatch.undo() mid-test. Each patches in its own pytest.MonkeyPatch.context() and asserts restoration after the block, so no fixture's patch is undone early.

This PR merged before its review repairs were pushed. They land in #14603, together with the coordinator's facade-hygiene requirements. Open items on this PR and where each went:

  1. GPT, span 116759990009 (fixture patches undone mid-test): fixed in fix(source-providers): close the review findings left on #14543 #14603 (disposition above).
  2. Opus, span 9e4b40fb36bb (GitLab merge-state re-read without host=): accepted-and-deferred to GitLab merge-state re-read is refused because _run_json is called without host= #14556, as already answered on this PR. It is pre-existing behaviour, and fix(source-providers): close the review findings left on #14543 #14603 does not change it.
  3. Coordinator review, test/test_source_provider_plugin.py: the spelling-cap check could no longer see the collector's module. Fixed in fix(source-providers): close the review findings left on #14543 #14603; the check now also asks the module that defines source_search_ref, and a mutation proves it turns red.
  4. Coordinator review, the learn-cron-dashboard.md bullet: verb agreement fixed, and the claim that every other name resolves to an owner now names what the handler binds itself. Fixed in fix(source-providers): close the review findings left on #14543 #14603.
  5. Facade hygiene, fixed in fix(source-providers): close the review findings left on #14543 #14603 with one test each:
    • a patch that may create a forwarded name (create=True), which would strip the owner's binding on exit, is refused by test_source_providers_refactor_create_guard.py;
    • __all__ is the historical public names;
    • __getattr__ is hidden from type checkers.

bolichen97 added a commit that referenced this pull request Sep 28, 2026
The source-provider split merged before its review repairs were
pushed. This lands them, together with the facade hygiene the
campaign asks every re-export facade to hold.

The handler facade keeps forwarding both writes and deletes to the
owner module, the canonical re-export pattern, so monkeypatch and
mock.patch restore exactly in any nesting order. mock.patch with
create=True skips the write-back that restore relies on and would
leave the owner without the name, so a guard test refuses any patch
in the test trees that may create a forwarded name. It resolves each
call from the syntax tree: the patch callable under any alias, and
targets through aliases, module-name constants, f-strings and
concatenation. A target it cannot resolve fails the guard rather
than passing it. The forwarding
__getattr__ is hidden from type checkers, and the facade binds every
public name it always bound, so a star import sees what it saw
before the split.

Two tests regain their teeth: the plugin spelling-cap check asks
the collector's own module, and the facade write test patches in a
private MonkeyPatch instead of undoing the shared fixture mid-test.
The spec bullet drops an overclaim and fixes its verb agreement.
bolichen97 added a commit that referenced this pull request Sep 28, 2026
The source-provider split merged before its review repairs were
pushed. This lands them, together with the facade hygiene the
campaign asks every re-export facade to hold.

The handler facade keeps forwarding both writes and deletes to the
owner module, the canonical re-export pattern, so monkeypatch and
mock.patch restore exactly in any nesting order. mock.patch with
create=True skips the write-back that restore relies on and would
leave the owner without the name, so a guard test refuses any patch
in the test trees that may create a forwarded name. It resolves each
call from the syntax tree: the patch callable under any alias, and
targets through aliases, module-name constants, f-strings and
concatenation, reading every test file that names the handler. A
target it cannot resolve, a helper call's result included, fails
the guard rather than passing it. The forwarding
__getattr__ is hidden from type checkers, and the facade binds every
public name it always bound, so a star import sees what it saw
before the split.

Two tests regain their teeth: the plugin spelling-cap check asks
the collector's own module, and the facade write test patches in a
private MonkeyPatch instead of undoing the shared fixture mid-test.
The spec bullet drops an overclaim and fixes its verb agreement.
chenmingwei23 pushed a commit that referenced this pull request Sep 28, 2026
The source-provider split merged before its review repairs were
pushed. This lands them, together with the facade hygiene the
campaign asks every re-export facade to hold.

The handler facade keeps forwarding both writes and deletes to the
owner module, the canonical re-export pattern, so monkeypatch and
mock.patch restore exactly in any nesting order. mock.patch with
create=True skips the write-back that restore relies on and would
leave the owner without the name, so a guard test refuses any patch
in the test trees that may create a forwarded name. It resolves each
call from the syntax tree: the patch callable under any alias, and
targets through aliases, module-name constants, f-strings and
concatenation, reading every test file that names the handler. A
target it cannot resolve, a helper call's result included, fails
the guard rather than passing it. The forwarding
__getattr__ is hidden from type checkers, and the facade binds every
public name it always bound, so a star import sees what it saw
before the split.

Two tests regain their teeth: the plugin spelling-cap check asks
the collector's own module, and the facade write test patches in a
private MonkeyPatch instead of undoing the shared fixture mid-test.
The spec bullet drops an overclaim and fixes its verb agreement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

posix-only-approved Cross-Platform Portability findings reviewed and accepted as intentionally POSIX-only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants