Repository navigation
docs(rfc): record session search phase 2 and decide routing questions - #39386
Conversation
40b66f3 to
1f2fbdb
Compare
|
Review of The status update is right: phase 2 is on main as #36002 (merged today), off by default behind On the two decisions:
No blocking defect in the text. A Kiro Crew team member must sign off on this design before it merges. This review does not approve it. AI-assisted, sent by Bolin Review bot |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Opus 5.5, fork) — ⏭️ skippedRevision |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Documentation only: the RFC record and its index row are updated in place, and the routing change is the one this PR set out to decide. It takes nothing away, so no reader breaks. Notes
[DESIGN-REVIEWED] 16176a9 Description read[DESCRIPTION-READ] 9c346d4362ca6cd3d40e3ee51d51641fbe1048b57afdc06efbf0ee0d790519fb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: design-doc-gate: D. This PR changes docs only. It records that phase 2 of the session search RFC is on main, and it settles open questions 2 and 5 as the RFC's section 10 requires before phase 3 can open. I checked the code claims against the repo and they hold:
The front matter ( Tier: 2, held for a person: design-doc-gate: D. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/). Review result: the design record and its figures hold up, with one wording fix needed below. We checked these on this head:
Also good: the new pinned test 5 gives phase 3 a concrete check for the short-query branch. Needed before merge, even after a design sign-off:
Optional:
|
Phase 2 of the session search RFC, the chunk store and its gateway vector fill behind session.semantic_search, merged as kirodotdev#36002. List it in implementation-prs, name it in the status line and the index row, and move the audit stamps to the commit the record was checked against. Record the answers to open questions 2 and 5, which phase 3 needs first, from a fresh blind query set scored once against pre-registered rules: the literal route stays on the whole query, and fires only for a query of at least 12 characters. Part of kirodotdev#24476
1f2fbdb to
16176a9
Compare
|
@bolichen97 thanks. Both notes and the reviewer's required fix are in 16176a9 (one commit, still docs only):
The optional points are in too. Section 1, goal 2, 8.1, test 2 and the exit criteria ("tests 1 to 5") now state the length condition. Measured under the rule: literal identifiers keep keyword R@1, 0.583 on set D's 36 identifier queries and 0.667 on sets A and B. One of the 36 is under 12 characters: keyword search ranked it 4th, and routed it falls outside the vector leg's top 50, where the whole-query rule had it 38th. Goal 2 says this. The set C 0.275 was measured again under the length rule. It equals the October whole-query figure by coincidence, since the whole-query rule scores 0.237 on the grown store, and 8.5 now says so. |
bolichen97
left a comment
There was a problem hiding this comment.
Design sign-off on head 16176a99a9: RFC accepted.
Recording phase 2 as on main and the answers to open questions 2 and 5 is accepted: the literal branch keeps the whole query as its trigger, and fires only for a query of at least 12 characters. The set D numbers support it (R@1 0.396 -> 0.422 for a 0.010 R@5 loss, inside the 0.025 the rule allowed). Thanks for replacing the private project name.
Not blocking, worth a follow-up before phase 3 opens: Goal 2 (line 111) and 8.1 (line 302) still promise today's first row for any exact identifier. Please qualify them with the 12-character condition so readers don't expect it for short ids.
AI-assisted, sent by Bolin Review bot
Problem / Motivation
Goal: Record that phase 2 of the session search RFC is on main, and record the answers to open questions 2 and 5, which phase 3 needs before it opens.
The RFC still says "phases 2 to 4 are not built" and lists
implementation-prs: [31794], although phase 2 merged as #36002. Section 10 makes deciding open questions 2 and 5 on the harness the entry condition for phase 3, and both were still open.Why it matters
The First Principles lane and readers of
docs/request-for-change/README.mdtake the status and the index row as the record. A stale row makes the phase 3 PR look as if it builds on a store that does not exist, and an open routing question would let phase 3 ship a rule nobody measured.Not a goal
What changed (motivation → approach → change)
rfc-session-semantic-search.mdimplementation-prs: [31794, 36002], the status line names phase 2 on main, andlast-auditedandaudited-atare set to the commit the record was checked againstrfc-session-semantic-search.mdsections 6.6, 7.3, 8.5, 10, 14docs/request-for-change/README.mdBoth answers come from set D, a fresh query set run once after the candidates, parameter grids and decision thresholds were written down. 40 targets the earlier sets did not use. 192 queries, written blind from one passage each by a writer who saw no results. Question 2: the trigger stays the whole query; routing on an identifier token fired on 27 of 36 identifier-plus-description queries and lowered their R@1 from 0.583 to 0.472. Question 5: the literal branch fires only for a query of at least 12 characters. A hit cap and a share of scanned conversations gave the same R@1 gain (0.396 to 0.422) but lost 0.036 of R@5, past the 0.025 the rule allowed; the length rule lost 0.010.
Backwards compatibility
Compatible: documentation only. The status stays
partial, since phases 3 and 4 are open.Removes nothing: front-matter values, the status line, the index row and the routing text are updated in place.
Tests
Proof: not provable -- docs or config only
./scripts/docs-lint.shpasses. The record's claims were checked at99953f89aa:history_chunks.py,history_vectors.py,history_vectors_index.pyandhistory_vectors_fill.pyare on main,session.semantic_searchis inconfig/sections_session.py, andsearch_sessionsinhistory_search.pydoes not read the store. The set D figures come from the session-recall prototype on one operator's store, keyword leg on99953f89aa; its vector leg reproduced the earlier sets' vector R@1 exactly.Manual verification
N/A: documentation only.
Related Issues
Part of #24476
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)