Repository navigation
feat(knowledge): expand local AsciiDoc include:: directives at ingest - #21778
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The include record and stale-includer sweep is a mechanism-level fix the description itself calls incomplete, and it is larger than the feature it protects. Not justified as shipped
What this change shipsInventory (6 items) — 3 justifiedIntent: index an AsciiDoc book with the text of the local files it includes, so that searches find its content in context. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] fd17c76 Description read[DESCRIPTION-READ] 93265560bdd36791fb937d841fc62c49ce7da25b086b6a1fcc698728569a53ad That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS An edited part never refreshes the copy inside its includer: change detection still keys on the includer's own mtime and hash. Text redacted from a part stays searchable indefinitely. Watch
Suggestions
[DESIGN-REVIEWED] fd17c76 Description read[DESCRIPTION-READ] 30c4f4999c04fc2a34a64bde34a24270aac44cb3a17e4474dfbc1dffaf970f19 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
3a2f897 to
70f6605
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
|
70f6605 to
fe6fd55
Compare
|
Rebased: main carried the fix for |
fe6fd55 to
792c5b1
Compare
|
self-added: yes
|
|
|
|
84c32fa to
fd17c76
Compare
|
mechanism: an includer is never paired by the fuzzy dedup tier, so its claim always lives under its own text hash
|
|
CI note: Backend Tests (Windows) (7) was red on 72e3d5e and 84c32fa in this PR's own tests. On 72e3d5e six legacy-plaintext tests failed: the fixture wrote its literal includer with |
|
|
|
|
|
|
|
|
|
|
|
|
Review-ready at fd17c76: PR Readiness passed (all 103 checks green, including the Windows shards), GPT and Opus have no blocking finding on this head, and every design and first-principles item has a disposition. For the maintainer:
Not merging, approving or enabling auto-merge from this side. |
chenmingwei23
left a comment
There was a problem hiding this comment.
Approving per maintainer bulk readiness pass (readiness: passed, no failing lanes).
Problem / Motivation
Goal: An AsciiDoc file added to Knowledge is indexed with the text of the local files it pulls in through
include::, so a search hit on that text carries the including document's context.AsciiDoc books and runbooks are often one short top file that stitches many small files together with
include::path[]. Since #14405, Knowledge reads.adoc/.asciidocas plain text, so the top file is indexed as a list of rawinclude::lines and its real content is missing from it.Why it matters
A search for something written in an included part finds only the bare fragment, with no heading or chapter around it. A search that should land on the guide itself finds a file that holds no content at all.
Not a goal
include::targets that are URLs, Antora resource ids (partial$x.adoc), absolute paths, or paths outside the top document's folder. Ingest stays local-only.tag=,lines=,leveloffset=): the whole target file is indexed.xref:or<<anchor>>links. They are links, not inclusions; their targets are indexed as their own documents.What changed (motivation → approach → change)
An AsciiDoc document in a folder source is indexed with its local includes pasted in, the way Asciidoctor itself would assemble it. An include can only pull in a file the folder scan already picked for that source, so
.kiroignore,ignore_patterns, skipped folders, ignored names and unsupported file types are honored without a second set of rules.knowledge/folder_watcher.pyknowledge/ingestion.pyingest_file(..., include_paths=None, included=None)hands both to the readerknowledge/readers.py_read_asciidocreplaces eachinclude::target[...]line with the target's text, recursively, only when the set is given, and reports the real path of each target it inlinedknowledge/store_parts/schema_migrations.py,item_rows.pyfolder_include_targetstable; the source cascade deletes its rowsknowledge/dedup.pyA target is read only when its
realpathis in the set and stays inside the top document's own folder, with no dot-named part on the way (.env). The read goes throughpinned_fs.open_fenced_for_read, and the kernel's own path for the opened file must pass the same checks, so a folder swapped for a symlink around the open is refused, and so is a hardlinked file. A NUL in the target name or the file's bytes, a cycle, and the depth, include-count, per-file and total size caps all leave the directive in the text exactly as written.Each successful read records the real paths the includer inlined in
folder_include_targets, in the same write as its items or its dedup claim. Every sweep compares each record with the new walk. When an includer's recorded target is gone (deleted, newly excluded, or rejected by a walk rule), its stale items are archived, or its old dedup claim released, in that same sweep. It is then re-read first, pastmax_files, whatever its own mtime, unless it waits for the user (failed/skipped). Staleness comes from the record and the walk, and a retired row keeps an empty file hash until it is re-read, so a pause, a budget stop, a crash or the part coming back leaves the same answer for the next sweep. Claim release on deletion falls back to the file hash for a legacy row with no text hash. The knowledge spec describes the rules and the remaining gaps.Known limit: only folder sources expand includes. A single local-file source, a dashboard upload, an artifact or an agent document keeps its
include::lines as text. This can be revisited later.Accepted risk: editing a part, rather than removing or excluding it, does not refresh the copy inside the documents that include it until each of them is next re-read. That includes text redacted by an edit. Removing or excluding the part is what takes it out of its includers. The include record is a mechanism-level fix for removal only; the cause, that an includer's change key is its own mtime and hash, stays open for edits. The record makes the follow-up small: compare each recorded target's walk mtime with the includer's last read.
Backwards compatibility
Compatible: every file main reads still reads the same way;
ingest_fileandFileReader.readgain an optional keyword that defaults to the old behavior, and an.adoc/.asciidocfile outside a folder source yields the same text as on main.Removes nothing: the reader only adds included text; a directive it cannot resolve stays in the indexed text unchanged.
Schema: one new table,
folder_include_targets, created byCREATE TABLE IF NOT EXISTS; no backfill, since main inlines nothing, so no record is the true state of every existing row. An older build ignores the table, and a record it leaves behind can only cause an extra re-read.Tests
test/test_knowledge_asciidoc_include.py: nested and relative includes expand (including a code snippet inside a listing block) when the target is in the scan set built by the real folder walk; a target excluded by.kiroignore,ignore_patternsorinclude_extensions, a.pem, anode_modules/file, a lock file,.git/configand.envstay as written; a read with no set never expands; a document with noinclude::never iterates the set;../targets and outward symlinks are not read; URL, Antora id,{attribute}, escaped, indented, missing and absolute targets stay as written; a parent folder swapped for a symlink, kept or restored around the open, and a witness re-pointed before the fence, are refused, and so is a hardlinked target; a NUL in a target name keeps the directive; trailing whitespace afterinclude::...[]still expands while leading whitespace keeps the line as text; a deduped includer records its targets with its claim, and drops that claim when a part is removed, also when the part shares its text or two includers share it, while a row outside the sweep keeps a shared claim; a sweep cancelled before it deletes the removed file still re-reads the includer next time; failed directives count toward the include cap; the depth, count and total-size caps hold; a real folder scan indexes a selected part and not a.kiroignore-excluded one; a part later excluded or deleted leaves its includer on the next sweep, also when the first sweep stops before the includer, and a selected name re-pointed after the walk at a rejected file is not inlined; an includer whose mtime is 0 drops a removed part (done or deduped, after a pause, and ahead of a newer file a chunk budget then defers), while an unchanged file with mtime 0 keeps the fast path; a part beyondmax_files, an includer beyondmax_files, a symlinked includer, a.mdname linked to an includer and every level of a nested include are re-read when the part goes; removing a file nothing inlined re-reads nothing; afailedorskippedincluder, and one whose re-read fails, lose their stale items, atomically; every stale includer loses its stale text in a sweep the chunk budget stops early, and one that sweep did not reach is still re-read after its part comes back; a retried (pending) deduped includer still releases its claim; a stalescanningincluder keeps its retry budget and is retired; a record that does not parse forces a re-read; the source cascade drops its records; an includer that would fuzzy-match another folder's file is never merged, so removing its part and then deleting the other source leaves no untracked item; a legacy row with a null or empty text hash releases its claim on re-read and on deletion, and a legacy copy still keeps a shared claim.test/test_knowledge_ingest_scan_off_loop.py: the pipeline stand-ins acceptincluded.test/test_knowledge.py: the AsciiDoc extension test pins the_read_asciidocdispatch.test/test_store_parts_bindings.py: the pinned signature ofdetach_source_location_by_hashnow includesexclude_folder_paths: AbstractSet[str]..kiroignore,ignore_patterns,include_extensionsand folder-scan tests fail against the previous reader (secret text inlined).2791b5ded81b, the folder-match version.test/test_knowledge*.py,test/test_folder_watcher.py,test/test_pinned_fs*.py,test/test_store_parts*.py,test/test_folder_watch_handlers.pyandtest/test_agent_folder_dedupe.py, 1999 passed.Manual verification
N/A — unit coverage sufficient: the tests drive the reader and real folder scans against real files on disk and a real store.
Related Issues
Part of #4699
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)