Repository navigation
refactor(knowledge): split ingestion.py into budget and rebuild owners - #22271
Conversation
knowledge/ingestion.py was 2,012 lines, past the 2,000-line cap. It keeps IngestionPipeline and the helpers the pipeline reads, and becomes a kiro_crew._facade_mirror facade over two sibling owners: ingestion_budget.py (ImportChunkBudget, ImportChunkBudgetError and the live budget read) and ingestion_rebuild.py (rebuild_embeddings, start_rebuild_job, count_stale_items and the sweep's helpers). Every moved definition is AST-identical to before. Every name the module bound still resolves on it to the same object, and a patch through it reaches every module holding the name, so the bulk-pacing tests' patch of bulk_pace_delay still reaches the re-embed sweep. ImportChunkBudgetError keeps kiro_crew.knowledge.ingestion as its module. test/test_ingestion_facade.py pins the facade contract; the facade inventory, the owning spec and the line-cap baseline row follow the new layout. Part of #20192
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that What this change shipsInventory (6 items) — 6 justifiedIntent: bring
[FIRST-PRINCIPLES-REVIEWED] 42cc988 Description read[DESCRIPTION-READ] 54aaa20c2bc45fe808530996e71839ea6df32e85965e435c7acae2eb8c55430b That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the facade-patch test catches dashboard or watcher code that patches [DESIGN-REVIEWED] 42cc988 Description read[DESCRIPTION-READ] 3bd544ac7d957b74486f8db87a5a9e397c403f20482321d5be3c77962282b709 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: refactor (7 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: code-move only, splits knowledge/ingestion.py into ingestion_budget.py and ingestion_rebuild.py behind a mirror facade; the moved SQL and embed calls are re-wrapped, not changed. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/knowledge.md.
Problem / Motivation
Goal: Bring
src/kiro_crew/knowledge/ingestion.py(2,012 lines) under the 2,000-line cap without changing what the ingestion pipeline does, so new work on the explicit-import budget and the corpus re-embed sweep lands in modules sized for it.ingestion.pyis past the repository's 2,000-line cap for hand-written files and is listed in.github/file-line-cap-baseline.txt(row 2,007; it has grown 5 lines past that row). One file holds the ingestion pipeline, the explicit-import chunk budget and the signature-gated re-embed sweep that the dashboard rebuild and the watcher's self-heal both run.Why it matters
A file over the cap makes every feature PR that touches it grow a baselined file, which the line-cap guard reports and the campaign must later undo. The pipeline, the budget and the re-embed sweep each keep collecting rules, and today they all land in one file.
Not a goal
kiro_crew.knowledge.ingestionoffers, or to how a test patches them.IngestionPipeline: it stays whole iningestion.py, with the one baselined on-loop fs site (_ingest_file_impl) in place.What changed (motivation → approach → change)
ingestion.pykeeps the pipeline and becomes akiro_crew._facade_mirrorfacade over two sibling owners. Every name it bound before still resolves on it to the same object, and a patch through it reaches every module holding the name: the bulk-pacing tests patchingestion.bulk_pace_delay, which the re-embed sweep now reads in its owner, and the mirror writes that patch there.knowledge/ingestion.pyIngestionPipeline, the ingestion gate andhandoff_gate_context,EmbedRateLimiter,run_to_completion,FileTooLargeError, the redaction and chunking helpers the pipeline readsknowledge/ingestion_budget.pyImportChunkBudget,ImportChunkBudgetError, the window constant and the liveknowledge.import_chunk_budgetreadknowledge/ingestion_rebuild.pyrebuild_embeddings,start_rebuild_job,count_stale_itemsand the sweep's off-loop write and pace helpersNeither owner imports the facade or the other owner, and neither logs.
ImportChunkBudgetErrorkeepskiro_crew.knowledge.ingestionas its__module__, since tracebacks and the dashboard's error text name it. The owning spec,knowledge.md, gains the two owners in its Key Files table and citesImportChunkBudgetand the re-embed in their new homes.Backwards compatibility
Compatible: every public name
kiro_crew.knowledge.ingestionbound before the split still resolves on it and is the same object (test/campaign_facade_inventories/kiro_crew.knowledge.ingestion.json: 35 names,check_inventory.pymissing 0, extra 0); every module-level name the file bound (less stdlib modules) still resolves (test_ingestion_facade.py::test_every_base_module_name_resolves_through_the_facade); and every name the test trees patch through the facade reaches every module holding it (test_ingestion_facade.py::test_a_facade_patch_of_every_scanned_name_reaches_every_module_holding_it).The take-away check flags the three public defs that moved to
ingestion_rebuild; each is still served by the facade:Reader: src/kiro_crew/knowledge/watcher.py:count_stale_items -- crew page --
from .ingestion import count_stale_itemsresolves through the facade's mirror to the owner's function (test_knowledge_embedding_rebuild.py)Reader: src/kiro_crew/knowledge/watcher.py:rebuild_embeddings -- crew page -- the watcher's self-heal imports it from the facade, the same object as the owner's (test_knowledge_rebuild_bulk_pacing.py::test_watcher_self_heal_sweep_is_paced_on_the_bulk_class)
Reader: src/kiro_crew/knowledge/watcher.py:start_rebuild_job -- crew page -- imported from the facade, same object (test_knowledge_embedding_rebuild.py)
Reader: src/kiro_crew/dashboard/handlers/knowledge_embeddings.py:rebuild_embeddings -- crew page -- the dashboard rebuild imports it from the facade (test_knowledge_rebuild_bulk_pacing.py::test_dashboard_trigger_runs_unpaced_at_the_interactive_class)
Reader: src/kiro_crew/dashboard/handlers/knowledge_embeddings.py:start_rebuild_job -- crew page -- imported from the facade, same object (test_knowledge_embedding_rebuild.py)
Reader: src/kiro_crew/dashboard/handlers/knowledge.py:rebuild_embeddings -- crew page -- its TYPE_CHECKING import from the facade type-checks through the facade's owner manifest (mypy src/kiro_crew)
Move audit
move_audit.py --base c3f2b27de7 --head 42cc988f75(the merge-base and this head):All 65 units are AST-identical to BASE (0 changed). Non-move edits, all glue:
+import importlib as _importlib,+import sys as _sys,TypeVar→TYPE_CHECKING, TypeVar,+from kiro_crew import _facade_mirrortimedelta,PRIORITY_BULK,bulk_pace_delay,MAX_CHUNKS_PER_FILEfrom the header importsTYPE_CHECKING+from .ingestion_budget import ImportChunkBudget, _import_chunk_budgetingestion._import_chunk_budget_OWNERS, import loop,install(...), class swap, derived__all__, twoTYPE_CHECKINGgroupskiro_crew._facade_mirrorImportChunkBudgetError.__module__ = "kiro_crew.knowledge.ingestion"ingestion_budget.pyL36 parity (default diff vs
--minimal):knowledge.md18/3 both;file-line-cap-baseline.txt0/1 both;ingestion.pydefault 62/505,--minimal61/504. The one extra+line under the default diff is a blank line in the facade foot (the default diff matches one deleted blank line of the moved re-embed block differently); no kept non-blank line is re-shown and no gate token is on it. This is the L41 glue-only exception, accepted by the campaign Master: filesrc/kiro_crew/knowledge/ingestion.py, default 62 added / 505 removed,--minimal61 / 504, the differing line is blank; three foot layouts were measured and all differ by the same one blank line. Diff size: 91,000 bytes.Tests
test/test_ingestion_facade.py(new, 18 tests): the facade serves a mirror over exactly theingestion_*siblings, with no exclusion and no divergence; every BASE module-level name resolves to each holder's object; every name the test trees patch through the facade (8, fromfacade_mirror_scan, this file excluded) reaches every holder and is restored, with a non-vacuity check of the known seams; a behavioural pin that a facade patch ofbulk_pace_delayis what_embed_row_pacedreads iningestion_rebuild; the owners load in layer order and never import the facade (relative spellings resolved, with a self-test); the family logs askiro_crew.knowledge.ingestion;ImportChunkBudgetErrorkeeps the facade module and round-trips by qualname; every served name is named for type checkers (owner manifest and outside-family group, with a dropped-name self-test); no module outside the family imports an owner (with a planted-importer self-test); andleaf_test_scope.corpus_gates()selects the file.watcher.pyimporting an owner; the exception losing its module line; a served name dropped from eitherTYPE_CHECKINGgroup; an owner importing the facade by a relative spelling; a BASE name renamed in its owner; an owner binding its own logger; the corpus losing itsbulk_pace_delaypatches;IngestionPipelinereporting another module.-n0, once in reversed order and once at-n 3, 18 passed each time; no sleeps, clocks, ports or order-dependent assertions (sets are compared sorted).-n 3(run on the original base2cf3d0d69f, on a tree that differs from this head's patch only in two owner docstring sentences, one census assertion relaxed to a subset and the spec's facade paragraph; after the rebase onto main3e39e7a963, the 25 files that drive the pipeline, the budget and the re-embed plus the mirror, inventory, line-cap and handler-facade guards ran again: 1,573 passed; the push-time rebase ontoc3f2b27de7was clean and left the-U0patch and the path set identical): every test file that imports or patcheskiro_crew.knowledge.ingestion, the watcher and the dashboard knowledge handlers, the related backend set fromlocal-gate --dry-run(85 files), the campaign and tree-scan guards and everycorpus_gatesfacade test: 176 files, 7,717 passed, 1 failed —test_name_grant_facade.py::test_the_patch_scan_reads_every_spelling_and_nothing_else[helper-binding], which fails the same way on an unmodified export of the base (main's red there; test: clear main's four reds left by #21753, #21598 and later backend edits #22178 fixed it, and this branch is rebased onto that fix). The frontend related set (385 files, selected because.github/changed) was not run: nowebsite/file changes. mypysrc/kiro_crew: no issues in 3,305 files.isort --check-onlyandflake8oversrc/kiro_crew test conftest.py xdist_budget.py: clean. Comment-history, brand, black, subprocess-encoding, agent-sdk boundary, sync-io (both ledgers), memory-store seam, feature-map and line-cap gates anddocs_lint.py: clean; harness parity was red at the base only on main's staleACP_BACKENDS_MEMBER_CAPABILITIES [claude]row, which test: clear main's four reds left by #21753, #21598 and later backend edits #22178 removed.test_flake_pattern_ratchet.py: 60 passed. Semgrep 1.78 with the CI configs and--baseline-commit 2cf3d0d69fover the four changed Python files: 0 findings, 0 parse errors. Installed-mode smoke: a wheel built from agit archiveexport of this head ships all threeknowledge/ingestion*.pymodules, and each imports from the unpacked wheel under the isolatedenv -iwrapper.ingestion.py90.0%,ingestion_budget.py96.7%,ingestion_rebuild.py96.9% (floor 80%).Manual verification
N/A — a pure move of definitions; the ingest, budget, watcher self-heal and dashboard rebuild tests that drive these code paths run unchanged against the facade and pass (see Tests).
Citation edits (L29)
docs/system-specs/modules/knowledge.mdKey Files table:knowledge/ingestion_budget.pyandknowledge/ingestion_rebuild.pyrows added; theknowledge/ingestion.pyrow names it the owners' import path.docs/system-specs/modules/knowledge.mdexplicit-import ceiling:`ImportChunkBudget` in `ingestion.py`→`ImportChunkBudget` in `ingestion_budget.py`docs/system-specs/modules/knowledge.mdsection header### Embedding vector space (...)gainsingestion_rebuild.py(the section describesrebuild_embeddings).docs/system-specs/modules/knowledge.mdgains the facade-contract paragraph after the dashboard handlers' one: the mirror over the two owners, what the facade keeps, patch reach, the owners' import and logger rules,ImportChunkBudgetError's__module__, where new work goes, and the pinning test.docs/system-specs/modules/knowledge.mdWriters list gainsknowledge/ingestion_rebuild.py(the signature-gated re-embed's vector writes and its job row).Import-path spellings (
knowledge.ingestion.rebuild_embeddings,ingestion.count_stale_items) are unchanged: the facade still serves those names.CI path references
None: no workflow path filter or explicit file list names
knowledge/ingestion.py; noAUTOSDE.yamlrule orsensitive_change_review.pyglob matchesknowledge/ingestion*.Notes for reviewers
ingestion.pyunderkiro_crew.knowledge.ingestion.ImportChunkBudget(a plain class) and the moved functions report their owner module; nothing in the tree reads their__module__or__qualname__, and nothing pickles them..github/sync-io-in-async-fs-baseline.txtkeeps1 src/kiro_crew/knowledge/ingestion.py: the site (uri = str(p.resolve())inIngestionPipeline._ingest_file_impl) stays in place, verbatim. The owners have no fs site.security_posture.NON_EGRESS_REDACTION_MODULESkeepsknowledge/ingestion.py:_redactand_redact_for_ingestand their calls stay there; the moved code calls no redactor.dashboard/handlers/knowledge_ingest.py:290"ingestion.py:1128 / 1144" andwebsite/playwright/{artifacts,embed-popout}.spec.ts"ingestion.py:341"); the constructs they mean stay iningestion.py, so they are left as they are.Related Issues
Part of #20192
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)