Skip to content

docs(rfc): record session search phase 2 and decide routing questions - #39386

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
dma9527:docs/rfc-session-search-phase2-record
Nov 19, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
dma9527:docs/rfc-session-search-phase2-record

Conversation

@dma9527

@dma9527 dma9527 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Record that phase 2 of the session search RFC is on main, and record the answers to open questions 2 and 5, which phase 3 needs before it opens.

The RFC still says "phases 2 to 4 are not built" and lists implementation-prs: [31794], although phase 2 merged as #36002. Section 10 makes deciding open questions 2 and 5 on the harness the entry condition for phase 3, and both were still open.

Why it matters

The First Principles lane and readers of docs/request-for-change/README.md take the status and the index row as the record. A stale row makes the phase 3 PR look as if it builds on a store that does not exist, and an open routing question would let phase 3 ship a rule nobody measured.

Not a goal

  • No code change.
  • No change to the RFC's phases, or to any part of the design other than the routing rule in 7.3.

What changed (motivation → approach → change)

File Change
rfc-session-semantic-search.md implementation-prs: [31794, 36002], the status line names phase 2 on main, and last-audited and audited-at are set to the commit the record was checked against
rfc-session-semantic-search.md sections 6.6, 7.3, 8.5, 10, 14 Open questions 2 and 5 marked decided with the set D numbers; 7.3's rule gains the length condition and a fifth pinned test
docs/request-for-change/README.md The index row names phase 2's modules on main, says phases 3 and 4 are not built, and states the decided route

Both answers come from set D, a fresh query set run once after the candidates, parameter grids and decision thresholds were written down. 40 targets the earlier sets did not use. 192 queries, written blind from one passage each by a writer who saw no results. Question 2: the trigger stays the whole query; routing on an identifier token fired on 27 of 36 identifier-plus-description queries and lowered their R@1 from 0.583 to 0.472. Question 5: the literal branch fires only for a query of at least 12 characters. A hit cap and a share of scanned conversations gave the same R@1 gain (0.396 to 0.422) but lost 0.036 of R@5, past the 0.025 the rule allowed; the length rule lost 0.010.

Backwards compatibility

Compatible: documentation only. The status stays partial, since phases 3 and 4 are open.

Removes nothing: front-matter values, the status line, the index row and the routing text are updated in place.

Tests

Proof: not provable -- docs or config only

./scripts/docs-lint.sh passes. The record's claims were checked at 99953f89aa: history_chunks.py, history_vectors.py, history_vectors_index.py and history_vectors_fill.py are on main, session.semantic_search is in config/sections_session.py, and search_sessions in history_search.py does not read the store. The set D figures come from the session-recall prototype on one operator's store, keyword leg on 99953f89aa; its vector leg reproduced the earlier sets' vector R@1 exactly.

Manual verification

N/A: documentation only.

Related Issues

Part of #24476

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@dma9527
dma9527 requested review from a team and KiroCrewCaptain as code owners November 19, 2026 04:30
@dma9527
dma9527 requested a review from CrysisDeu November 19, 2026 04:30
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention fork Pull request from a fork (external contributor) labels Nov 19, 2026
@dma9527 dma9527 changed the title docs(rfc): record session search phase 2 on main docs(rfc): record session search phase 2 and decide routing questions Nov 19, 2026
@dma9527
dma9527 force-pushed the docs/rfc-session-search-phase2-record branch from 40b66f3 to 1f2fbdb Compare November 19, 2026 04:47
@bolichen97

Copy link
Copy Markdown
Contributor

Review of 1f2fbdbe35 (docs-only RFC record). I approved the held fork CI runs on this head.

The status update is right: phase 2 is on main as #36002 (merged today), off by default behind session.semantic_search, and search does not read it yet. implementation-prs: [31794, 36002] and the README row match.

On the two decisions:

  • Open question 2 (whole query, not an identifier token): the set D result is clear (identifier-token routing fired on 27 of 36 and lowered R@1 from 0.583 to 0.472), and the method is sound: candidates and thresholds written down before set D, queries written blind. Agree.
  • Open question 5 (minimum length 12): two things a phase 3 reader needs, which the text states but does not act on:
    1. 12 sits at the edge of the grid ("12 was the largest length tried") and the margin over the other two candidates is 5 queries. A longer threshold might do better, or the rule might be on a slope. Please say in §14 that phase 3 keeps SEMANTIC_LITERAL_MIN_CHARS a named constant and re-scores 12 against a wider grid (for example 12, 16, 20) on the next fresh set before it is turned on by default.
    2. On set C, routed R@1 stays below vector-only (0.275 against 0.312). So for short common queries the router is still worse than not routing. That is acceptable while the feature is off by default, but phase 3's exit criteria should name it: either routed R@1 is at least vector R@1 on every set, or the gap is listed as accepted, with who accepted it.

No blocking defect in the text.

A Kiro Crew team member must sign off on this design before it merges. This review does not approve it.

AI-assisted, sent by Bolin Review bot

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 16176a99a9c73cffc3f563a41bb4add70e8859ac via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 16176a9

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 16176a99a9c73cffc3f563a41bb4add70e8859ac ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 16176a99a9c73cffc3f563a41bb4add70e8859ac touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 16176a99a9c73cffc3f563a41bb4add70e8859ac via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Documentation only: the RFC record and its index row are updated in place, and the routing change is the one this PR set out to decide. It takes nothing away, so no reader breaks.

Notes

  • Section 14 says 12 was the largest length tried and the rule "may sit on a slope". It plans to score 12, 16 and 20 again before phase 4, which is the right place for that check. When phase 3 lands, SEMANTIC_LITERAL_MIN_CHARS should be easy to tune, not fixed in the code.

[DESIGN-REVIEWED] 16176a9

Description read

[DESCRIPTION-READ] 9c346d4362ca6cd3d40e3ee51d51641fbe1048b57afdc06efbf0ee0d790519fb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 16176a99a9c73cffc3f563a41bb4add70e8859ac via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 16176a9

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: design-doc-gate: D.

This PR changes docs only. It records that phase 2 of the session search RFC is on main, and it settles open questions 2 and 5 as the RFC's section 10 requires before phase 3 can open. I checked the code claims against the repo and they hold:

  • history_chunks.py, history_vectors.py, history_vectors_index.py and history_vectors_fill.py all exist.
  • session.semantic_search is in config/sections_session.py and loader_builders.py.
  • The pending and coverage metrics exist: pending_count and coverage().
  • SessionCatalogProjection.vector_store is only read on the delete path (history_projection.py:227), so search does not read the store yet.

The front matter (implementation-prs, last-audited, audited-at) is updated the way docs/request-for-change/README.md requires. The routing change is applied the same way in sections 1, 4, 6.6, 7.3, 8.1, 8.5, 10 and 14, and there is a matching fifth test (test 5 in 7.3). The numbers I could check agree with each other: set C vector R@1 averages to 0.3125 from the 6.6 table, and the set D counts match the stated rates. The phases and the trust boundaries are unchanged. Phase 4 still requires routed R@1 to be at least vecto

Tier: 2, held for a person: design-doc-gate: D.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 16176a99a9 · 105 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here.

To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/).

Review result: the design record and its figures hold up, with one wording fix needed below. We checked these on this head:

Also good: the new pinned test 5 gives phase 3 a concrete check for the short-query branch.

Needed before merge, even after a design sign-off:

  • 8.5 (line 347) uses a project name taken from the private query set. This public repository must not carry names from that set, so please replace it with a neutral description, for example "a 12-character one-word project name".

Optional:

  • The 12-character rule in 7.3 means short exact identifiers now take the vector ranking. Goal 2 (line 111) and 8.1 (line 302) still promise today's first row for any exact identifier. Section 1 and test 2 do not mention the length condition, and the phase 3 exit criteria (line 384) still say "tests 1 to 4". Please qualify these, or report set D literal-identifier R@1 under the new rule next to keyword R@1.
  • 8.5 (line 345) gives set C routed R@1 as 0.275 under the length rule. That is exactly the old whole-query rule's figure. Please confirm it was measured again, or say that the two figures happen to match.

Phase 2 of the session search RFC, the chunk store and its gateway vector
fill behind session.semantic_search, merged as kirodotdev#36002. List it in
implementation-prs, name it in the status line and the index row, and move
the audit stamps to the commit the record was checked against.

Record the answers to open questions 2 and 5, which phase 3 needs first,
from a fresh blind query set scored once against pre-registered rules: the
literal route stays on the whole query, and fires only for a query of at
least 12 characters.

Part of kirodotdev#24476
@dma9527
dma9527 force-pushed the docs/rfc-session-search-phase2-record branch from 1f2fbdb to 16176a9 Compare November 19, 2026 17:28
@dma9527

dma9527 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 thanks. Both notes and the reviewer's required fix are in 16176a9 (one commit, still docs only):

  • Required: 8.5 no longer names a project from the private query set; it now says "a 12-character project name".
  • Question 5, edge of the grid: section 14 now says phase 3 keeps SEMANTIC_LITERAL_MIN_CHARS a named constant, and that before phase 4 turns the leg on by default, 12 is scored again against 12, 16 and 20 on the next fresh set.
  • Set C gap: the phase 3 exit criteria now require the harness to report routed against vector R@1 per query kind. Where routed is below vector (set C today, 0.275 against 0.312), the phase 3 PR lists the gap, and a maintainer accepts it there or the rule changes before phase 4.

The optional points are in too. Section 1, goal 2, 8.1, test 2 and the exit criteria ("tests 1 to 5") now state the length condition. Measured under the rule: literal identifiers keep keyword R@1, 0.583 on set D's 36 identifier queries and 0.667 on sets A and B. One of the 36 is under 12 characters: keyword search ranked it 4th, and routed it falls outside the vector leg's top 50, where the whole-query rule had it 38th. Goal 2 says this. The set C 0.275 was measured again under the length rule. It equals the October whole-query figure by coincidence, since the whole-query rule scores 0.237 on the grown store, and 8.5 now says so.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design sign-off on head 16176a99a9: RFC accepted.

Recording phase 2 as on main and the answers to open questions 2 and 5 is accepted: the literal branch keeps the whole query as its trigger, and fires only for a query of at least 12 characters. The set D numbers support it (R@1 0.396 -> 0.422 for a 0.010 R@5 loss, inside the 0.025 the rule allowed). Thanks for replacing the private project name.

Not blocking, worth a follow-up before phase 3 opens: Goal 2 (line 111) and 8.1 (line 302) still promise today's first row for any exact identifier. Please qualify them with the 12-character condition so readers don't expect it for short ids.

AI-assisted, sent by Bolin Review bot

@bolichen97
bolichen97 merged commit dc3d2bc into kirodotdev:main Nov 19, 2026
96 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants