Skip to content

refactor(knowledge): split ingestion.py into budget and rebuild owners - #22271

Merged
bolichen97 merged 1 commit into
mainfrom
refactor/linecap-s80-ingestion
Oct 19, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
refactor/linecap-s80-ingestion

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Bring src/kiro_crew/knowledge/ingestion.py (2,012 lines) under the 2,000-line cap without changing what the ingestion pipeline does, so new work on the explicit-import budget and the corpus re-embed sweep lands in modules sized for it.

ingestion.py is past the repository's 2,000-line cap for hand-written files and is listed in .github/file-line-cap-baseline.txt (row 2,007; it has grown 5 lines past that row). One file holds the ingestion pipeline, the explicit-import chunk budget and the signature-gated re-embed sweep that the dashboard rebuild and the watcher's self-heal both run.

Why it matters

A file over the cap makes every feature PR that touches it grow a baselined file, which the line-cap guard reports and the campaign must later undo. The pipeline, the budget and the re-embed sweep each keep collecting rules, and today they all land in one file.

Not a goal

  • No behaviour change: every moved definition keeps its body (AST-identical); the re-embed owner is black-formatted, which changes layout only.
  • No change to which names kiro_crew.knowledge.ingestion offers, or to how a test patches them.
  • No decomposition of IngestionPipeline: it stays whole in ingestion.py, with the one baselined on-loop fs site (_ingest_file_impl) in place.

What changed (motivation → approach → change)

ingestion.py keeps the pipeline and becomes a kiro_crew._facade_mirror facade over two sibling owners. Every name it bound before still resolves on it to the same object, and a patch through it reaches every module holding the name: the bulk-pacing tests patch ingestion.bulk_pace_delay, which the re-embed sweep now reads in its owner, and the mirror writes that patch there.

File Lines Owns
knowledge/ingestion.py 2,012 → 1,569 IngestionPipeline, the ingestion gate and handoff_gate_context, EmbedRateLimiter, run_to_completion, FileTooLargeError, the redaction and chunking helpers the pipeline reads
knowledge/ingestion_budget.py 210 ImportChunkBudget, ImportChunkBudgetError, the window constant and the live knowledge.import_chunk_budget read
knowledge/ingestion_rebuild.py 358 rebuild_embeddings, start_rebuild_job, count_stale_items and the sweep's off-loop write and pace helpers

Neither owner imports the facade or the other owner, and neither logs. ImportChunkBudgetError keeps kiro_crew.knowledge.ingestion as its __module__, since tracebacks and the dashboard's error text name it. The owning spec, knowledge.md, gains the two owners in its Key Files table and cites ImportChunkBudget and the re-embed in their new homes.

Backwards compatibility

Compatible: every public name kiro_crew.knowledge.ingestion bound before the split still resolves on it and is the same object (test/campaign_facade_inventories/kiro_crew.knowledge.ingestion.json: 35 names, check_inventory.py missing 0, extra 0); every module-level name the file bound (less stdlib modules) still resolves (test_ingestion_facade.py::test_every_base_module_name_resolves_through_the_facade); and every name the test trees patch through the facade reaches every module holding it (test_ingestion_facade.py::test_a_facade_patch_of_every_scanned_name_reaches_every_module_holding_it).

The take-away check flags the three public defs that moved to ingestion_rebuild; each is still served by the facade:

Reader: src/kiro_crew/knowledge/watcher.py:count_stale_items -- crew page -- from .ingestion import count_stale_items resolves through the facade's mirror to the owner's function (test_knowledge_embedding_rebuild.py)
Reader: src/kiro_crew/knowledge/watcher.py:rebuild_embeddings -- crew page -- the watcher's self-heal imports it from the facade, the same object as the owner's (test_knowledge_rebuild_bulk_pacing.py::test_watcher_self_heal_sweep_is_paced_on_the_bulk_class)
Reader: src/kiro_crew/knowledge/watcher.py:start_rebuild_job -- crew page -- imported from the facade, same object (test_knowledge_embedding_rebuild.py)
Reader: src/kiro_crew/dashboard/handlers/knowledge_embeddings.py:rebuild_embeddings -- crew page -- the dashboard rebuild imports it from the facade (test_knowledge_rebuild_bulk_pacing.py::test_dashboard_trigger_runs_unpaced_at_the_interactive_class)
Reader: src/kiro_crew/dashboard/handlers/knowledge_embeddings.py:start_rebuild_job -- crew page -- imported from the facade, same object (test_knowledge_embedding_rebuild.py)
Reader: src/kiro_crew/dashboard/handlers/knowledge.py:rebuild_embeddings -- crew page -- its TYPE_CHECKING import from the facade type-checks through the facade's owner manifest (mypy src/kiro_crew)

Move audit

move_audit.py --base c3f2b27de7 --head 42cc988f75 (the merge-base and this head):

src/kiro_crew/knowledge/ingestion.py: 2012 -> 1569 lines; moved 100.0% (python-ast)

All 65 units are AST-identical to BASE (0 changed). Non-move edits, all glue:

Edit Where Why
+import importlib as _importlib, +import sys as _sys, TypeVar → TYPE_CHECKING, TypeVar, +from kiro_crew import _facade_mirror facade header the mirror install at the foot
drop timedelta, PRIORITY_BULK, bulk_pace_delay, MAX_CHUNKS_PER_FILE from the header imports facade header only moved code reads them; the facade serves them from the owners and names them for type checkers under TYPE_CHECKING
+from .ingestion_budget import ImportChunkBudget, _import_chunk_budget facade header the pipeline reads both as globals, and the budget tests patch ingestion._import_chunk_budget
owner map comment, _OWNERS, import loop, install(...), class swap, derived __all__, two TYPE_CHECKING groups facade foot the mirror facade shape of kiro_crew._facade_mirror
ImportChunkBudgetError.__module__ = "kiro_crew.knowledge.ingestion" ingestion_budget.py the exception keeps the module its callers and tracebacks name
each owner's docstring and import header owners glue

L36 parity (default diff vs --minimal): knowledge.md 18/3 both; file-line-cap-baseline.txt 0/1 both; ingestion.py default 62/505, --minimal 61/504. The one extra + line under the default diff is a blank line in the facade foot (the default diff matches one deleted blank line of the moved re-embed block differently); no kept non-blank line is re-shown and no gate token is on it. This is the L41 glue-only exception, accepted by the campaign Master: file src/kiro_crew/knowledge/ingestion.py, default 62 added / 505 removed, --minimal 61 / 504, the differing line is blank; three foot layouts were measured and all differ by the same one blank line. Diff size: 91,000 bytes.

Tests

  • test/test_ingestion_facade.py (new, 18 tests): the facade serves a mirror over exactly the ingestion_* siblings, with no exclusion and no divergence; every BASE module-level name resolves to each holder's object; every name the test trees patch through the facade (8, from facade_mirror_scan, this file excluded) reaches every holder and is restored, with a non-vacuity check of the known seams; a behavioural pin that a facade patch of bulk_pace_delay is what _embed_row_paced reads in ingestion_rebuild; the owners load in layer order and never import the facade (relative spellings resolved, with a self-test); the family logs as kiro_crew.knowledge.ingestion; ImportChunkBudgetError keeps the facade module and round-trips by qualname; every served name is named for type checkers (owner manifest and outside-family group, with a dropped-name self-test); no module outside the family imports an owner (with a planted-importer self-test); and leaf_test_scope.corpus_gates() selects the file.
  • Mutation proofs (each turns its test red with exit 1; the file is restored after): the owner reading the pace derivation under another binding; watcher.py importing an owner; the exception losing its module line; a served name dropped from either TYPE_CHECKING group; an owner importing the facade by a relative spelling; a BASE name renamed in its owner; an owner binding its own logger; the corpus losing its bulk_pace_delay patches; IngestionPipeline reporting another module.
  • Determinism: the new file 5 runs at -n0, once in reversed order and once at -n 3, 18 passed each time; no sleeps, clocks, ports or order-dependent assertions (sets are compared sorted).
  • Local gate floor at -n 3 (run on the original base 2cf3d0d69f, on a tree that differs from this head's patch only in two owner docstring sentences, one census assertion relaxed to a subset and the spec's facade paragraph; after the rebase onto main 3e39e7a963, the 25 files that drive the pipeline, the budget and the re-embed plus the mirror, inventory, line-cap and handler-facade guards ran again: 1,573 passed; the push-time rebase onto c3f2b27de7 was clean and left the -U0 patch and the path set identical): every test file that imports or patches kiro_crew.knowledge.ingestion, the watcher and the dashboard knowledge handlers, the related backend set from local-gate --dry-run (85 files), the campaign and tree-scan guards and every corpus_gates facade test: 176 files, 7,717 passed, 1 failed — test_name_grant_facade.py::test_the_patch_scan_reads_every_spelling_and_nothing_else[helper-binding], which fails the same way on an unmodified export of the base (main's red there; test: clear main's four reds left by #21753, #21598 and later backend edits #22178 fixed it, and this branch is rebased onto that fix). The frontend related set (385 files, selected because .github/ changed) was not run: no website/ file changes. mypy src/kiro_crew: no issues in 3,305 files. isort --check-only and flake8 over src/kiro_crew test conftest.py xdist_budget.py: clean. Comment-history, brand, black, subprocess-encoding, agent-sdk boundary, sync-io (both ledgers), memory-store seam, feature-map and line-cap gates and docs_lint.py: clean; harness parity was red at the base only on main's stale ACP_BACKENDS_MEMBER_CAPABILITIES [claude] row, which test: clear main's four reds left by #21753, #21598 and later backend edits #22178 removed. test_flake_pattern_ratchet.py: 60 passed. Semgrep 1.78 with the CI configs and --baseline-commit 2cf3d0d69f over the four changed Python files: 0 findings, 0 parse errors. Installed-mode smoke: a wheel built from a git archive export of this head ships all three knowledge/ingestion*.py modules, and each imports from the unpacked wheel under the isolated env -i wrapper.
  • Per-file line coverage (the 20 test files that drive the pipeline, the budget and the re-embed): ingestion.py 90.0%, ingestion_budget.py 96.7%, ingestion_rebuild.py 96.9% (floor 80%).

Manual verification

N/A — a pure move of definitions; the ingest, budget, watcher self-heal and dashboard rebuild tests that drive these code paths run unchanged against the facade and pass (see Tests).

Citation edits (L29)

  • docs/system-specs/modules/knowledge.md Key Files table: knowledge/ingestion_budget.py and knowledge/ingestion_rebuild.py rows added; the knowledge/ingestion.py row names it the owners' import path.
  • docs/system-specs/modules/knowledge.md explicit-import ceiling: `ImportChunkBudget` in `ingestion.py` → `ImportChunkBudget` in `ingestion_budget.py`
  • docs/system-specs/modules/knowledge.md section header ### Embedding vector space (...) gains ingestion_rebuild.py (the section describes rebuild_embeddings).
  • docs/system-specs/modules/knowledge.md gains the facade-contract paragraph after the dashboard handlers' one: the mirror over the two owners, what the facade keeps, patch reach, the owners' import and logger rules, ImportChunkBudgetError's __module__, where new work goes, and the pinning test.
  • docs/system-specs/modules/knowledge.md Writers list gains knowledge/ingestion_rebuild.py (the signature-gated re-embed's vector writes and its job row).

Import-path spellings (knowledge.ingestion.rebuild_embeddings, ingestion.count_stale_items) are unchanged: the facade still serves those names.

CI path references

None: no workflow path filter or explicit file list names knowledge/ingestion.py; no AUTOSDE.yaml rule or sensitive_change_review.py glob matches knowledge/ingestion*.

Notes for reviewers

  • Logger name (L25): neither owner logs; every log call stays in ingestion.py under kiro_crew.knowledge.ingestion.
  • ImportChunkBudget (a plain class) and the moved functions report their owner module; nothing in the tree reads their __module__ or __qualname__, and nothing pickles them.
  • .github/sync-io-in-async-fs-baseline.txt keeps 1 src/kiro_crew/knowledge/ingestion.py: the site (uri = str(p.resolve()) in IngestionPipeline._ingest_file_impl) stays in place, verbatim. The owners have no fs site.
  • security_posture.NON_EGRESS_REDACTION_MODULES keeps knowledge/ingestion.py: _redact and _redact_for_ingest and their calls stay there; the moved code calls no redactor.
  • Two comments outside this PR already cited stale line numbers at the base (dashboard/handlers/knowledge_ingest.py:290 "ingestion.py:1128 / 1144" and website/playwright/{artifacts,embed-popout}.spec.ts "ingestion.py:341"); the constructs they mean stay in ingestion.py, so they are left as they are.

Related Issues

Part of #20192

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

knowledge/ingestion.py was 2,012 lines, past the 2,000-line cap. It
keeps IngestionPipeline and the helpers the pipeline reads, and
becomes a kiro_crew._facade_mirror facade over two sibling owners:
ingestion_budget.py (ImportChunkBudget, ImportChunkBudgetError and the
live budget read) and ingestion_rebuild.py (rebuild_embeddings,
start_rebuild_job, count_stale_items and the sweep's helpers).

Every moved definition is AST-identical to before. Every name the
module bound still resolves on it to the same object, and a patch
through it reaches every module holding the name, so the bulk-pacing
tests' patch of bulk_pace_delay still reaches the re-embed sweep.
ImportChunkBudgetError keeps kiro_crew.knowledge.ingestion as its
module. test/test_ingestion_facade.py pins the facade contract; the
facade inventory, the owning spec and the line-cap baseline row
follow the new layout.

Part of #20192
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 19, 2026 19:25
@bolichen97
bolichen97 requested a review from buluoray October 19, 2026 19:25
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 19, 2026 19:35
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that ImportChunkBudgetError.__module__ being overridden to the facade name still matches what the dashboard's error text and pickling expect.

What this change ships

Inventory (6 items) — 6 justified

Intent: bring ingestion.py under the 2,000-line cap with no behaviour change, so new budget and re-embed work lands in modules sized for it. This is a FIX for a breach of the line-cap invariant in AGENTS.md.

  1. ingestion.py drops from 2,012 to 1,569 lines and its row leaves the line-cap baseline — justified
  2. The explicit-import chunk budget now lives in ingestion_budget.py — justified
  3. The corpus re-embed sweep (rebuild_embeddings, start_rebuild_job, count_stale_items) now lives in ingestion_rebuild.py — justified
  4. ingestion.py becomes a _facade_mirror facade, so every old name and test patch still resolves; this reuses the mechanism 196 existing modules already use — justified
  5. ImportChunkBudgetError keeps kiro_crew.knowledge.ingestion as its reported module — justified
  6. New facade inventory JSON and test_ingestion_facade.py, matching the 129 inventories and 84 *_facade.py tests already in the repo — justified

[FIRST-PRINCIPLES-REVIEWED] 42cc988

Description read

[DESCRIPTION-READ] 54aaa20c2bc45fe808530996e71839ea6df32e85965e435c7acae2eb8c55430b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the facade-patch test catches dashboard or watcher code that patches ingestion.rebuild_embeddings at runtime, not only in the test trees.

[DESIGN-REVIEWED] 42cc988

Description read

[DESCRIPTION-READ] 3bd544ac7d957b74486f8db87a5a9e397c403f20482321d5be3c77962282b709

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 42cc988

Verdict parsed from the review's SHA-scoped output markers for commit 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 42cc988

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 42cc988f757f74a17cbc1ce88dcaa6ec92cb319a: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Does ImportChunkBudgetError.__module__ set to the facade name still match the dashboard's error text and pickling?

rebutted: it matches both, because the override restores the class's BASE module rather than inventing one.
The dashboard error text is str(exc) (knowledge_ingest.py, knowledge_sync.py, publication.py answer 429 import_budget_exceeded with it); __init__ is AST-identical to BASE, so the message is byte-identical.
Pickling by reference resolves kiro_crew.knowledge.ingestion.ImportChunkBudgetError, which the mirror serves as the same class: pickle.loads(pickle.dumps(ImportChunkBudgetError)) is ImportChunkBudgetError holds at this head.
test_ingestion_facade.py::test_the_refusal_exception_keeps_the_facade_module pins __module__, __qualname__ and getattr(import_module(cls.__module__), cls.__qualname__) is cls; mutation M3 (drop the line) goes red with rc 1.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Does the facade-patch test catch dashboard or watcher code that patches ingestion.rebuild_embeddings at runtime, not only in the test trees?

rebutted: no production code patches the facade at runtime; git grep over src/ finds no setattr on, or assignment to, kiro_crew.knowledge.ingestion.<name>. The watcher and the dashboard import the names (from .ingestion import rebuild_embeddings) and hold their own bindings, exactly as at BASE.
Reach is a property of the module object, not of the tests: the facade's class is MirroredModule, whose __setattr__ writes every holder, so a runtime setattr(ingestion, "rebuild_embeddings", f) reaches ingestion_rebuild too (checked at this head: the owner sees the new object, and the restore reaches it).
The test-tree scan exists to prove the corpus's own patches reach their readers (8 names; non-vacuity mutation M9 goes red with rc 1).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: refactor (7 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: code-move only, splits knowledge/ingestion.py into ingestion_budget.py and ingestion_rebuild.py behind a mirror facade; the moved SQL and embed calls are re-wrapped, not changed. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/knowledge.md.

@bolichen97
bolichen97 merged commit 1f520f3 into main Oct 19, 2026
106 of 113 checks passed
@bolichen97
bolichen97 deleted the refactor/linecap-s80-ingestion branch October 19, 2026 21:32
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants