Skip to content

feat(knowledge): expand local AsciiDoc include:: directives at ingest - #21778

Merged
iamwhatever merged 1 commit into
mainfrom
fix/4699-asciidoc-include
Oct 19, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/4699-asciidoc-include

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An AsciiDoc file added to Knowledge is indexed with the text of the local files it pulls in through include::, so a search hit on that text carries the including document's context.

AsciiDoc books and runbooks are often one short top file that stitches many small files together with include::path[]. Since #14405, Knowledge reads .adoc / .asciidoc as plain text, so the top file is indexed as a list of raw include:: lines and its real content is missing from it.

Why it matters

A search for something written in an included part finds only the bare fragment, with no heading or chapter around it. A search that should land on the guide itself finds a file that holds no content at all.

Not a goal

  • Following include:: targets that are URLs, Antora resource ids (partial$x.adoc), absolute paths, or paths outside the top document's folder. Ingest stays local-only.
  • Applying include attributes (tag=, lines=, leveloffset=): the whole target file is indexed.
  • Rewriting xref: or <<anchor>> links. They are links, not inclusions; their targets are indexed as their own documents.
  • AsciiDoc-aware chunking or markup rendering.
  • Re-indexing an including document when only one of its included files changes.

What changed (motivation → approach → change)

An AsciiDoc document in a folder source is indexed with its local includes pasted in, the way Asciidoctor itself would assemble it. An include can only pull in a file the folder scan already picked for that source, so .kiroignore, ignore_patterns, skipped folders, ignored names and unsupported file types are honored without a second set of rules.

Part Change
knowledge/folder_watcher.py records the exact resolved path it screened for each selected file and uses that set, normcased once, for every file it ingests; keeps each includer's include record and re-reads the includers whose recorded targets left the walk
knowledge/ingestion.py ingest_file(..., include_paths=None, included=None) hands both to the reader
knowledge/readers.py _read_asciidoc replaces each include::target[...] line with the target's text, recursively, only when the set is given, and reports the real path of each target it inlined
knowledge/store_parts/schema_migrations.py, item_rows.py new folder_include_targets table; the source cascade deletes its rows
knowledge/dedup.py the fuzzy tier never pairs an includer, since a fuzzy collapse moves its claim under a hash nothing records (main's wider gap: #21954)

A target is read only when its realpath is in the set and stays inside the top document's own folder, with no dot-named part on the way (.env). The read goes through pinned_fs.open_fenced_for_read, and the kernel's own path for the opened file must pass the same checks, so a folder swapped for a symlink around the open is refused, and so is a hardlinked file. A NUL in the target name or the file's bytes, a cycle, and the depth, include-count, per-file and total size caps all leave the directive in the text exactly as written.

Each successful read records the real paths the includer inlined in folder_include_targets, in the same write as its items or its dedup claim. Every sweep compares each record with the new walk. When an includer's recorded target is gone (deleted, newly excluded, or rejected by a walk rule), its stale items are archived, or its old dedup claim released, in that same sweep. It is then re-read first, past max_files, whatever its own mtime, unless it waits for the user (failed/skipped). Staleness comes from the record and the walk, and a retired row keeps an empty file hash until it is re-read, so a pause, a budget stop, a crash or the part coming back leaves the same answer for the next sweep. Claim release on deletion falls back to the file hash for a legacy row with no text hash. The knowledge spec describes the rules and the remaining gaps.

Known limit: only folder sources expand includes. A single local-file source, a dashboard upload, an artifact or an agent document keeps its include:: lines as text. This can be revisited later.

Accepted risk: editing a part, rather than removing or excluding it, does not refresh the copy inside the documents that include it until each of them is next re-read. That includes text redacted by an edit. Removing or excluding the part is what takes it out of its includers. The include record is a mechanism-level fix for removal only; the cause, that an includer's change key is its own mtime and hash, stays open for edits. The record makes the follow-up small: compare each recorded target's walk mtime with the includer's last read.

Backwards compatibility

Compatible: every file main reads still reads the same way; ingest_file and FileReader.read gain an optional keyword that defaults to the old behavior, and an .adoc / .asciidoc file outside a folder source yields the same text as on main.

Removes nothing: the reader only adds included text; a directive it cannot resolve stays in the indexed text unchanged.

Schema: one new table, folder_include_targets, created by CREATE TABLE IF NOT EXISTS; no backfill, since main inlines nothing, so no record is the true state of every existing row. An older build ignores the table, and a record it leaves behind can only cause an extra re-read.

Tests

  • test/test_knowledge_asciidoc_include.py: nested and relative includes expand (including a code snippet inside a listing block) when the target is in the scan set built by the real folder walk; a target excluded by .kiroignore, ignore_patterns or include_extensions, a .pem, a node_modules/ file, a lock file, .git/config and .env stay as written; a read with no set never expands; a document with no include:: never iterates the set; ../ targets and outward symlinks are not read; URL, Antora id, {attribute}, escaped, indented, missing and absolute targets stay as written; a parent folder swapped for a symlink, kept or restored around the open, and a witness re-pointed before the fence, are refused, and so is a hardlinked target; a NUL in a target name keeps the directive; trailing whitespace after include::...[] still expands while leading whitespace keeps the line as text; a deduped includer records its targets with its claim, and drops that claim when a part is removed, also when the part shares its text or two includers share it, while a row outside the sweep keeps a shared claim; a sweep cancelled before it deletes the removed file still re-reads the includer next time; failed directives count toward the include cap; the depth, count and total-size caps hold; a real folder scan indexes a selected part and not a .kiroignore-excluded one; a part later excluded or deleted leaves its includer on the next sweep, also when the first sweep stops before the includer, and a selected name re-pointed after the walk at a rejected file is not inlined; an includer whose mtime is 0 drops a removed part (done or deduped, after a pause, and ahead of a newer file a chunk budget then defers), while an unchanged file with mtime 0 keeps the fast path; a part beyond max_files, an includer beyond max_files, a symlinked includer, a .md name linked to an includer and every level of a nested include are re-read when the part goes; removing a file nothing inlined re-reads nothing; a failed or skipped includer, and one whose re-read fails, lose their stale items, atomically; every stale includer loses its stale text in a sweep the chunk budget stops early, and one that sweep did not reach is still re-read after its part comes back; a retried (pending) deduped includer still releases its claim; a stale scanning includer keeps its retry budget and is retired; a record that does not parse forces a re-read; the source cascade drops its records; an includer that would fuzzy-match another folder's file is never merged, so removing its part and then deleting the other source leaves no untracked item; a legacy row with a null or empty text hash releases its claim on re-read and on deletion, and a legacy copy still keeps a shared claim.
  • test/test_knowledge_ingest_scan_off_loop.py: the pipeline stand-ins accept included.
  • test/test_knowledge.py: the AsciiDoc extension test pins the _read_asciidoc dispatch.
  • test/test_store_parts_bindings.py: the pinned signature of detach_source_location_by_hash now includes exclude_folder_paths: AbstractSet[str].
  • The .kiroignore, ignore_patterns, include_extensions and folder-scan tests fail against the previous reader (secret text inlined).
  • The capped-part, capped-includer, symlinked-includer, removal-elsewhere and waiting-includer tests fail against 2791b5ded81b, the folder-match version.
  • Related set: test/test_knowledge*.py, test/test_folder_watcher.py, test/test_pinned_fs*.py, test/test_store_parts*.py, test/test_folder_watch_handlers.py and test/test_agent_folder_dedupe.py, 1999 passed.

Manual verification

N/A — unit coverage sufficient: the tests drive the reader and real folder scans against real files on disk and a real store.

Related Issues

Part of #4699

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision fd17c7670f6a55daa498b389df4195cbbb378c5b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of fd17c7670f6a55daa498b389df4195cbbb378c5b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The include record and stale-includer sweep is a mechanism-level fix the description itself calls incomplete, and it is larger than the feature it protects.

Not justified as shipped

  • Item 2 — oversized: folder_include_targets, _retire_stale_include_text, re-read past max_files, and the bypass of the unchanged-file shortcuts handle removal only. The description says "the cause, that an includer's change key is its own mtime and hash, stays open for edits", and ## Not a goal excludes "Re-indexing an including document when only one of its included files changes".
    Subtraction: put each recorded target's walk mtime into the includer's change key. One check would then cover both removal and edit, and _retire_stale_include_text, the reread ordering and _KEEPS_BYTES_HASH could go.
    Clears when: the retire/reorder machinery is replaced by the change-key check, or a human amends the Goal to cover the removal-only shape.
  • Item 3 — rides along: the fuzzy dedup tier now skips any includer, only so that item 2 can release claims (if a.inlines or b.inlines: return None). It goes with item 2.
  • Item 4 — rides along: NULLIF(text_hash, '') in _OWNERSHIP_HASH_COL. No writer in src/ stores text_hash = '' (I searched src/kiro_crew/knowledge for that pattern and for NULLIF; the only hit is this line).
    Subtraction: drop NULLIF.
    Clears when: a writer of an empty text_hash is named, or the NULLIF is removed.

What this change ships

Inventory (6 items) — 3 justified

Intent: index an AsciiDoc book with the text of the local files it includes, so that searches find its content in context. ADDITION.

  1. .adoc files in a folder source are indexed with their local include:: targets inlined — justified
  2. An includer is re-read first, past the file cap, once one of its parts is removed or excluded — oversized
  3. Fuzzy dedup no longer pairs any document that includes others — rides along
  4. Folder-row claim lookups treat an empty text hash as missing — rides along
  5. Deleting a legacy folder row releases its dedup claim by its file hash — justified
  6. FileReader.read/ingest_file gain optional include_paths/included arguments — justified

[FIRST-PRINCIPLES-REVIEWED] fd17c76

Description read

[DESCRIPTION-READ] 93265560bdd36791fb937d841fc62c49ce7da25b086b6a1fcc698728569a53ad

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of fd17c7670f6a55daa498b389df4195cbbb378c5b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

An edited part never refreshes the copy inside its includer: change detection still keys on the includer's own mtime and hash. Text redacted from a part stays searchable indefinitely.

Watch

  • Edits are not tracked, only removal. The description says "editing a part … does not refresh the copy inside the documents that include it … That includes text redacted by an edit."
    • Cause: the includer's change key is still its own mtime and SHA-256. The new folder_include_targets record is checked only for targets that are missing from the walk.
    • Consequence: a user removes a secret or wrong instruction from chapter3.adoc. book.adoc still serves it, with no end date unless someone touches book.adoc. In the common AsciiDoc layout (a short top file stitching parts together) the top file rarely changes, so the stale copy can sit there for the life of the source.
    • The record already holds each includer's resolved targets, and the walk already has every target's mtime. Marking an includer stale when any recorded target's mtime is newer than the includer's last read would use the same retire-and-re-read path as removal. That is the "small follow-up" the description names.
    • Clears when: an includer is also marked stale when a recorded target's walk mtime is newer than its last read, with a test where a part is edited and the old text leaves the includer, or a maintainer explicitly accepts the redaction gap.

Suggestions

  • If edits feed the same staleness signal, "a recorded target changed" covers both removal and edit. That one rule may let you drop some of the special-case handling for removal in the retirement step and the dedup-claim release.

[DESIGN-REVIEWED] fd17c76

Description read

[DESCRIPTION-READ] 30c4f4999c04fc2a34a64bde34a24270aac44cb3a17e4474dfbc1dffaf970f19

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fd17c7670f6a55daa498b389df4195cbbb378c5b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fd17c76

Verdict parsed from the review's SHA-scoped output markers for commit fd17c7670f6a55daa498b389df4195cbbb378c5b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fd17c7670f6a55daa498b389df4195cbbb378c5b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fd17c7670f6a55daa498b389df4195cbbb378c5b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fd17c76

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fd17c7670f6a55daa498b389df4195cbbb378c5b: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/4699-asciidoc-include branch from 3a2f897 to 70f6605 Compare October 18, 2026 07:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: early refusal of a NUL-bearing include target in _AdocIncludes._load

  • span=80ccb364638c fixed: A NUL-bearing include target aborts the entire document's ingestion

Fixed in 70f6605: _load refuses a target containing '\x00' with the other early refusals, before any filesystem call, so the directive stays in the text as written.
Regression test_nul_target_stays_as_written reads a document with a literal NUL in an include target and asserts the whole text survives verbatim; it fails with ValueError without the fix.

@iamwhatever

iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: no include expansion for a document sitting directly in the system temp dir

  • span=b13c53569e8f fixed: the shared system temp directory became the include jail for uploaded .adoc files

Fixed in 70f6605: _read_asciidoc returns the text unexpanded when realpath(dirname(document)) equals realpath(tempfile.gettempdir()), read at call time; no usable temp dir counts as a match.
One reader-side check covers every staging caller: the dashboard upload route, artifact_ingest mkstemp and agent_source mkstemp all stage in that directory. A private subdirectory, or any normal folder, still expands.
Regression test places a document and a sibling in the temp dir and asserts the sibling's text is not pulled in, and that a project-folder document still expands; it fails without the fix.

@iamwhatever

iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: removed the included_files / unresolved_includes meta keys

  • span=36bbc743cc78 fixed: zero consumers for the new meta keys

Fixed in 70f6605: both meta keys and the resolved/unresolved counters that fed them are removed; a repository grep of src/ and website/src finds no reader.
Tests that asserted the keys now assert the exact indexed text, so cycle, refusal, depth, count and size coverage are unchanged.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: every included fragment is indexed twice, as its own file and inside its includer

Intended: the issue asks that included content be indexed as part of the referencing document, so a hit carries its context; the fragment's own entry stays because a folder source indexes every supported file it walks.
Duplicate retrieval is bounded: the two entries carry different text (the includer adds headings and sibling sections), and whole-document duplicates are already collapsed by content-hash dedup. Measuring retrieval quality on AsciiDoc corpora is outside this PR's goal.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: includes may be quietly refused on Windows and macOS when the kernel path differs from realpath in case or prefix

Only the jail ROOT must match exactly: a target whose own spelling differs in case resolves under root, the kernel path differs from realpath, and the fence judges the kernel path, which still starts with the root, so it is included.
Windows: realpath (GetFinalPathNameByHandle in 3.12) and fd_real_path give the same canonical spelling; fd_real_path strips the \?\ prefix; ntpath.commonpath and relpath compare case-insensitively.
macOS: /tmp vs /private/tmp resolves on both sides. The remaining case is a folder source registered with a hand-typed wrong-case root, where includes stay as text: fail-closed, never a wrong read.
Folding case in the jail test would approve a sibling that differs only in case on a case-sensitive APFS volume, a jail escape, so exact comparison is kept. macOS Tests (on demand) passed on this head.

@iamwhatever
iamwhatever force-pushed the fix/4699-asciidoc-include branch from 70f6605 to fe6fd55 Compare October 18, 2026 07:46
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main carried the fix for test/test_history_bindings.py::test_the_member_set_is_the_frozen_inventory (red on main at 9d088c8 and on this PR's Backend Tests shard 1, Linux and Windows; fixed by 943a4e5 / #21792). New head fe6fd55 on 943a4e5; that test and the 1708 related knowledge tests pass locally; CI re-running.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/4699-asciidoc-include branch from fe6fd55 to 792c5b1 Compare October 18, 2026 09:04
@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: include targets restricted to the folder scan's own screened file set

  • span=d8d99b2605bc fixed: an include:: target skips every filter the folder scan uses

Fixed in 792c5b1: an include expands only into a file the folder scan selected for that source. _walk records the exact resolved path it screened for each file it keeps (screened_out), the scan passes that set to ingest_file(include_paths=...) only for .adoc/.asciidoc, and the reader requires membership on both the pre-open path and the kernel path of the opened descriptor.
So .kiroignore, ignore_patterns, include_extensions, HARD_SKIP_DIRS, DEFAULT_IGNORE_GLOBS, the SUPPORTED suffix filter and is_sensitive_path hold for included text by construction; .pem/.key, node_modules/, lock files, .git/config and .env stay as written (parametrized test).
The set is the screened strings, never a second resolution, so a selected name re-pointed after the walk at a rejected file is not admitted (scan-level regression; fails with the secret inlined on a re-resolving build).
Maintainer-decided known limit: non-folder sources (single file, upload, artifact, agent doc) keep include:: lines as text; stated in the PR body.

@iamwhatever

iamwhatever commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • rebutted: check that open_fenced_for_read actually opens files on Windows and macOS, or includes are silently never expanded there

open_fenced_for_read is the shared pinned_fs read helper that other surfaces already use (artifacts, chat threads, hooks, the dashboard cron-script read, which has a Windows-only test); its descriptor-path lookup fd_real_path has native routes on Linux (/proc/self/fd), macOS (F_GETPATH) and Windows (GetFinalPathNameByHandleW).
A route that cannot read the descriptor path fails closed: the directive stays as text, never a wrong read. The include tests carry no platform skip besides symlink/hardlink availability, and macOS Tests (on demand) passed on a prior head of this PR.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: each included part is indexed twice, as its own file and inside its includer

Intended by the issue: included content is indexed as part of the referencing document so a hit carries its context, while the part keeps its own entry because the folder scan indexes every file it selects.
The two entries carry different text (the includer adds its headings and sibling sections), whole-document duplicates are already collapsed by content-hash dedup, and measuring retrieval quality on AsciiDoc corpora is outside this PR's goal.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: text redacted from a part stays searchable in every includer until the includer itself changes

Stated choice: the frozen "Not a goal", an accepted risk, including for redaction, in the spec and the PR body, with the cause and the small mtime follow-up named for the maintainer (span c3c9e15a418d).

@iamwhatever
iamwhatever force-pushed the fix/4699-asciidoc-include branch from 84c32fa to fd17c76 Compare October 18, 2026 19:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

mechanism: an includer is never paired by the fuzzy dedup tier, so its claim always lives under its own text hash

  • span=27a975dd11d9 fixed: a fuzzy-merged includer can no longer leave obsolete text untracked (fd17c76).

_match_reason in dedup.py now returns no match for the fuzzy tier when either document is a folder file with an include record (DocRef.inlines, set in enumerate_docs and _build_doc_for). The exact tier is unchanged: both hashes are equal there, so the retirement's release by the includer's own text hash already names the claim.
test_fuzzy_merged_includer_leaves_no_untracked_text runs your three steps: two same-name guides with different text and one shared embedding, a dedup sweep, the part removed and the includer's folder re-scanned, then the other source deleted. It asserts that every item the includer's source owns is named by one of its rows. It fails on 84c32fa (one untracked item) and passes now.
The wider gap is on main for any fuzzy-merged folder file that is later deleted: the collapse never records the hash it moved the claim to. That is filed as #21954 (deferred-finding), which cites this span and this PR.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

CI note: Backend Tests (Windows) (7) was red on 72e3d5e and 84c32fa in this PR's own tests. On 72e3d5e six legacy-plaintext tests failed: the fixture wrote its literal includer with write_text, which writes CRLF on Windows. On 84c32fa the remaining two, test_legacy_plaintext_copy_keeps_shared_claim[null|empty-literal_and_copy], failed for the same reason on the second literal copy (other/copy.adoc): its bytes hash no longer equals its text hash, so it does not count as a claimant. This branch owns it; the fixture now writes that copy's bytes too (fd17c76). Test-only, no product change.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=47288b752069 rebutted: edits are not tracked, only removal

Edit propagation is the frozen "Not a goal" ("Re-indexing an including document when only one of its included files changes"). The spec and the PR body state it as an accepted risk, including for redaction, name the cause left open (an includer's change key is its own mtime and hash), and name the small mtime follow-up the record makes possible. Answered on every earlier head (most recently c3c9e15a418d on 84c32fa); left to the maintainer at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=5aa7e62755b3 rebutted: the includer's change key is still its own mtime and hash

Edit propagation is the frozen "Not a goal" ("Re-indexing an including document when only one of its included files changes"). The spec and the PR body state it as an accepted risk, including for redaction, name the cause left open (an includer's change key is its own mtime and hash), and name the small mtime follow-up the record makes possible. Answered on every earlier head (most recently c3c9e15a418d on 84c32fa); left to the maintainer at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=df72e5800d3c rebutted: a part edited to remove a secret keeps serving it from an unchanged top file

Edit propagation is the frozen "Not a goal" ("Re-indexing an including document when only one of its included files changes"). The spec and the PR body state it as an accepted risk, including for redaction, name the cause left open (an includer's change key is its own mtime and hash), and name the small mtime follow-up the record makes possible. Answered on every earlier head (most recently c3c9e15a418d on 84c32fa); left to the maintainer at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=0ff367fa48e9 rebutted: the record and the walk already hold what an mtime check needs

Agreed that it would reuse the retire-and-re-read path; that is why the PR body names it as the small follow-up.
Edit propagation is the frozen "Not a goal" ("Re-indexing an including document when only one of its included files changes"). The spec and the PR body state it as an accepted risk, including for redaction, name the cause left open (an includer's change key is its own mtime and hash), and name the small mtime follow-up the record makes possible. Answered on every earlier head (most recently c3c9e15a418d on 84c32fa); left to the maintainer at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=1726f64ca5fa rebutted: mark an includer stale when a recorded target's mtime is newer than its last read

This span clears through its second branch: the maintainer accepts the redaction gap or asks for the check at review.
Edit propagation is the frozen "Not a goal" ("Re-indexing an including document when only one of its included files changes"). The spec and the PR body state it as an accepted risk, including for redaction, name the cause left open (an includer's change key is its own mtime and hash), and name the small mtime follow-up the record makes possible. Answered on every earlier head (most recently c3c9e15a418d on 84c32fa); left to the maintainer at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: an edited part never refreshes the copy inside its includer, so redacted text stays searchable

Stated choice: the frozen "Not a goal", an accepted risk, including for redaction, in the spec and the PR body, with the cause and the small mtime follow-up named for the maintainer (spans 47288b752069, 5aa7e62755b3, df72e5800d3c, 0ff367fa48e9, 1726f64ca5fa, 6f99492c08cc).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • answered: the record and the stale-includer sweep are a mechanism-level fix the description calls incomplete, and larger than the feature

The description calls it removal-only on purpose: removal takes text out of scope and must leave even without a successful re-read, while edits are the frozen "Not a goal" (span eb667c1b401f). The fuzzy skip is what keeps the claim release correct (span c72248bc3cff). Whether to widen the Goal is the maintainer's call at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=6f99492c08cc rebutted: one 'a recorded target changed' rule could cover removal and edits

An mtime rule would not remove the retire step: a waiting (failed/skipped) includer, a failed re-read and a deduped claim still need the stale text taken out without a successful read (answered on 65e469a, span 21bed7a89931).
Edit propagation is the frozen "Not a goal" ("Re-indexing an including document when only one of its included files changes"). The spec and the PR body state it as an accepted risk, including for redaction, name the cause left open (an includer's change key is its own mtime and hash), and name the small mtime follow-up the record makes possible. Answered on every earlier head (most recently c3c9e15a418d on 84c32fa); left to the maintainer at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=eb667c1b401f rebutted: the removal machinery is oversized for a fix that leaves edits open

Removal and edits differ in kind. A removed or excluded part is text the user took out of the source's scope, so it must leave even when no re-read succeeds; tests pin the waiting, failed and deduped cases (answered on 84c32fa, span 6f826d91297c). An edit leaves the text in scope and only stale, which is the frozen "Not a goal". This span clears through its second branch, a human decision on the Goal, which is the maintainer's at review.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=c72248bc3cff rebutted: skipping includers in the fuzzy tier rides along with the removal machinery

It is what keeps the claim release correct: a fuzzy collapse moves an includer's claim under a hash nothing records, which GPT showed leaves untracked text after the other source is deleted (span 27a975dd11d9 on 84c32fa). test_fuzzy_merged_includer_leaves_no_untracked_text fails without it. The wider main gap is #21954.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=f1a90bbc5517 rebutted: NULLIF(text_hash, '') has no writer

Agreed on the fact, as answered on 2791b5d, f6a3872 and 65e469a. It keeps the SQL lookup in step with the Python fallback and returns the same value as plain COALESCE for every real row. Left in this head because the PR is at its last push on this path; the maintainer can drop it at review with no other change.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at fd17c76: PR Readiness passed (all 103 checks green, including the Windows shards), GPT and Opus have no blocking finding on this head, and every design and first-principles item has a disposition.

For the maintainer:

  • Edit propagation is out of scope (the frozen "Not a goal") and stated as an accepted risk, including for redaction. The include record makes it a small follow-up: compare each recorded target's walk mtime with the includer's last read.
  • Removal-only shape: first-principles asks for a human decision on whether the Goal covers removal only (span eb667c1b401f).
  • Optional trims you can take or leave: NULLIF(text_hash, '') in _OWNERSHIP_HASH_COL has no writer, and the legacy-row bytes-hash fallback on deletion can come out with its tests.
  • Follow-up for main's wider fuzzy-dedup gap: Fuzzy dedup does not record the hash it moves a claim to, so a deleted fuzzy-merged file can leave untracked items #21954.
  • Main has moved since this head and touches docs/system-specs/modules/knowledge.md; a rebase may be needed before merge.

Not merging, approving or enabling auto-merge from this side.

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving per maintainer bulk readiness pass (readiness: passed, no failing lanes).

@iamwhatever
iamwhatever merged commit d0b2aed into main Oct 19, 2026
103 of 104 checks passed
@iamwhatever
iamwhatever deleted the fix/4699-asciidoc-include branch October 19, 2026 02:16
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants