docs(tokenless): land the evolution roadmap referenced by the pipeline crates - #2877
docs(tokenless): land the evolution roadmap referenced by the pipeline crates#2877Forrest-ly wants to merge 4 commits into
Conversation
github-actions
Bot
added
component:tokenless
![qoderai[bot]](https://avatars.githubusercontent.com/in/1396158?s=60&v=4){kind=small}
There was a problem hiding this comment.
[P2] roadmap 文档在当前工作树缺失，tokenless-pipeline 与 tokenless-protocol 的 doc comment 已改为引用该路径，需确认主仓与文档发布管线中该文件实际存在且可访问；另 roadmap 对统计迁移与压缩器包仅标记为“Planned”，建议补一句当前运行时仍保留 legacy 测量通道与现状，避免读者误解为已在 0.7.13 完整删除。
🤖 Generated by Qoder • View workflow run
| @@ -16,7 +16,7 @@ | |||
| //! existing JSON response cleanup ([`RESPONSE_CLEANUP`]), whose executable | |||
There was a problem hiding this comment.
[P2] roadmap 文档缺失本地交叉引用
tokenless-pipeline/src/lib.rs / tokenless-protocol/src/lib.rs 的 doc comment 改为引用 docs/roadmap/evolution-roadmap.md，但当前工作树下该文件路径不存在，可能是工作流未同步新 docs 目录或分支差异导致。
建议：确认 docs/roadmap/evolution-roadmap.md / _zh.md 已在主仓库正确落地，并确保构建与文档发布管线能访问该路径，否则这两处引用会变成悬挂链接。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
采纳。83118ca 之后文档实际落地于 src/tokenless/docs/roadmap/，doc comment 中的相对路径 docs/roadmap/... 仍是悬挂引用。193ac47 已按 specs/documentation-standard.md §3.2 将 roadmap（含 _zh.md）迁至 src/tokenless/docs/design/，并把全部引用——tokenless-pipeline/src/lib.rs、tokenless-protocol/src/lib.rs（两处）、tokenless-protocol/src/tests/protocol_tests.rs——更新为新路径 src/tokenless/docs/design/evolution-roadmap.md，两个语言版本均已确认在新路径下落地。
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e70ace2df
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -0,0 +1,209 @@ | |||
| # Tokenless Evolution Roadmap | |||
There was a problem hiding this comment.
This is a component-specific architecture and protocol design document, but specs/documentation-standard.md §3.2 requires such documents to live under src/<component>/docs/design/ and explicitly prohibits placing them under the repository-level docs/ tree. Move both language variants to src/tokenless/docs/design/ and update the two crate-level references so the roadmap follows the canonical ownership and publication structure.
AGENTS.md reference: AGENTS.md:L364-L366
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Adopted. The earlier push (83118ca) moved the roadmap from the repository-level docs/ to src/tokenless/docs/roadmap/, which satisfied §3.1 but not §3.2 — design documents must live under src/<component>/docs/design/. Commit 193ac47 relocates both language variants to src/tokenless/docs/design/ and updates all four crate-level references (across tokenless-pipeline/src/lib.rs, tokenless-protocol/src/lib.rs, and tokenless-protocol/src/tests/protocol_tests.rs) to the new path.
| replace model-visible output; retrievable-lossy compressors never reach | ||
| hosts without a retrieve tool. |
There was a problem hiding this comment.
For a post_tool JSON request with replace_output=true but publish_retrieve_tool=false, the production RESPONSE_CLEANUP still matches: registry.rs deliberately sets publish_retrieve_tool: false and response_cleanup.rs reports an unstashed truncation as Unrecoverable, which arbitration can apply when reversibility is not required. Therefore the canonical roadmap's claim that retrievable-lossy compressors “never reach” such hosts contradicts shipped behavior; document the response-cleanup exception or describe capability filtering in terms of each spec's declared requirements, and mirror the correction in the Chinese page.
AGENTS.md reference: AGENTS.md:L337-L343
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Adopted. Verified against the current head: registry.rs deliberately sets publish_retrieve_tool: false for RESPONSE_CLEANUP, and response_cleanup.rs reports an unstashed truncation as Unrecoverable, so the blanket "never reach hosts without a retrieve tool" claim contradicted shipped behavior. Commit 193ac47 rewrites principle 2 to describe capability filtering in terms of each spec's declared requirements and documents the response-cleanup exception: it still runs on hosts without a retrieve tool, claims its reversibility from what actually happened, and is rejected by arbitration whenever the request requires reversibility (principle 5). Mirrored in the Chinese page.
| Detection is a pure function of the content, inspecting at most 64 KiB / | ||
| 200 lines and never fully parsing any format; expensive parsing stays |
There was a problem hiding this comment.
On inputs longer than 64 KiB that reach the JSON sniff, detection inspects the leading 64 KiB and content/json.rs::tail_window separately inspects up to another 64 KiB at the end, so the stated 64 KiB maximum understates the actual bounded work by as much as 2×. Describe the bound as a 64 KiB prefix plus a 64 KiB JSON tail window (with line-based checks capped at 200 lines), and update the Chinese counterpart as well.
AGENTS.md reference: AGENTS.md:L337-L343
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Adopted. Verified against the current head: content/json.rs::tail_window inspects up to a further 64 KiB at the end for the JSON bracket sniff, in addition to the leading 64 KiB prefix. Commit 193ac47 rewrites the §4.2 bound as a 64 KiB leading prefix plus, for the JSON bracket sniff alone, at most a trailing 64 KiB window, with line-based checks capped at 200 lines. Mirrored in the Chinese page.
|
Follow-up from a parallel run on the same tracked requirement (its duplicate Pushed one commit (
Verified after the change: One review note, no change made: the "§6 Compressor pack" heading is not |
193ac47 to
cf2857c
Compare
PR #2877 Review 结论：LGTM，建议合并 ✅概述纯文档 PR，为 tokenless pipeline crates 中大量引用的 evolution roadmap 落地。新增两个文档（英文 + 中文），并更新三处 doc comment / 注释中的路径引用。无代码行为变更。 代码交叉验证对 roadmap 中所有量化声明逐一与当前 main 分支代码核对，全部准确：
文档质量
前置 review 反馈Codex 此前提出的 3 条 inline comment（文档路径、retrieve-tool 路由描述、trailing scan window）均已在后续 commit 中采纳并修正。 小建议（非阻塞）
@code-reviewer-leader 请继续委派 code-reviewer-check 进行审核。 |
code-reviewer-check 审核结论：✅ 可以合并对 code-reviewer 的 review 进行了独立验证，逐一核对了交叉验证表中的 10 项声明。 结果：所有值和语义描述均准确（7/10 行号完全精确，3/10 存在轻微行号偏差但值正确）。LGTM 结论合理，纯文档 PR，无代码行为变更。 验证详情
|
KaiLongZhou
left a comment
KaiLongZhou
left a comment
There was a problem hiding this comment.
code-reviewer-check 验证通过：交叉验证表 10 项声明值和语义均准确，LGTM 结论合理，纯文档 PR 可以合并。
|
已采纳，PR description 已更新：
感谢 review！ |
…l crate The tokenless-protocol crate cites roadmap sections (§4.1–§6), design principles (2, 5, 6), and milestone markers (M1, M4), and protocol/src/lib.rs points at docs/roadmap/evolution-roadmap.md — but the document had not landed in this repository. Consolidate the roadmap as encoded in the shipped crates and the merged implementation PRs into that file (English plus Chinese), and update the "not landed yet" doc comment to reference it. Documentation only: no behavior change. Co-authored-by: multica-agent <github@multica.ai>
specs/documentation-standard.md §3.1 places component design documents in src/<component>/docs/ and explicitly not at the docs/ top level, which is reserved for cross-component pages. The roadmap is tokenless-specific, so it belongs under src/tokenless/docs/roadmap/. The move also makes the crate doc comments self-consistent: they cite `docs/roadmap/evolution-roadmap.md`, which now resolves relative to the component root as originally written. Finish the stale-comment sweep while here: protocol_tests.rs still said the roadmap document had not landed in-repo; it now describes the tests as the drift guard between the landed document and the wire types. Co-authored-by: multica-agent <github@multica.ai>
…h the post-tool restructure specs/documentation-standard.md §3.2 requires component design documents to live under src/<component>/docs/design/, so relocate the roadmap from src/tokenless/docs/roadmap/ into src/tokenless/docs/design/ and update the crate-level references (tokenless-protocol/src/lib.rs twice, protocol_tests.rs) to the new path. The tokenless-pipeline crate was removed from main after this branch was written, so its reference goes with the crate. Align the roadmap with the post-tool pipeline restructure that removed the standalone pipeline crate: detection and dispatch now live in the Runtime's post_tool module, JsonCompressor is the phase-one domain compressor, and the registry-based wording (principle 2, principles 3 and 7, §4.2, §4.3, §5.3, §6, status table) is rewritten to match. Also record §5.4 as shipped (PR alibaba#2844) and bump the status line to 0.7.14. Mirrored in the Chinese page. Documentation only: no behavior change. Co-authored-by: multica-agent <github@multica.ai>
cf2857c to
36f108a
Compare
Resolve conflicts against the protocol v2 lifecycle refactor (PR alibaba#2978): - tokenless-protocol/src/lib.rs: take main's rewritten crate docs. The v2 refactor dropped the roadmap citations the branch had been editing, so no dangling reference remains. - tokenless-protocol/src/tests/protocol_tests.rs: accept main's deletion; the contract tests moved to tests/protocol_tests.rs without the roadmap header note. Align the roadmap with the merged v2 lifecycle so the document lands current: protocol boundary rewritten as v1 history plus the four typed operations and strict envelopes; seams, adapter-boundary result facts, token-counter location, ContentType taxonomy names, the compress entry, hook status, and the statistics-migration status updated to what the shipped code and PRs alibaba#2885/alibaba#2978 encode (EN + ZH).
2 participants
Why
tokenless-protocolandtokenless-pipelinecite roadmap sections (§4.1–§6), design principles (2, 3, 5, 6, 7), and milestone markers (M1, M4) throughout their doc comments, andtokenless-protocol/src/lib.rsexplicitly points atdocs/roadmap/evolution-roadmap.mdfor the §5.1 token-counter decision — but the document had not landed in this repository, leaving every citation dangling. This PR lands the roadmap atsrc/tokenless/docs/design/evolution-roadmap.mdand updates those references to point at it.What changed
src/tokenless/docs/design/evolution-roadmap.md(+_zh.md) — the Tokenless evolution roadmap, consolidated strictly from what the shipped crates and the merged implementation PRs encode:heuristic-v1, the note the protocol crate references), §5.2 routing contract, §5.3 response cleanup behind the pipeline, §5.4 single external-hook entry, §5.5 statistics migration, §5.6 shared vocabulary and parity;src/tokenless/crates/tokenless-pipeline/src/lib.rs,src/tokenless/crates/tokenless-protocol/src/lib.rs, and the header comment ofsrc/tokenless/crates/tokenless-protocol/src/tests/protocol_tests.rs: the "roadmap has not landed in this repository yet" notes now reference the landed file. No code behavior changes.No future work is invented: planned steps are described only where the shipped code or the merged/open PRs already document them.
User / Agent impact
None. Documentation and crate doc comments only.
Risk and compatibility
Docs-only change; nothing shipped changes shape.
Test report
Scope: new docs under
src/tokenless/docs/design/, doc-comment edits intokenless-pipeline/tokenless-protocol. Environment: Linux x86_64, rustc/cargo 1.96.0 (stable).Commands executed (from
src/tokenless/):cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspacetokenless_pipeline,tokenless_protocol, CLI integration, and doc-tests)cargo doc --workspace --no-depsTargeted verification: every quantitative claim in the roadmap was checked against the shipped code —
MAX_SCAN_BYTES = 64 KiB/MAX_SCAN_LINES = 200(tokenless-pipeline/src/content.rs),DIAGNOSTIC_MAX_BYTES = 4096andTOKENIZER_ID = "heuristic-v1"(tokenless-protocol/src/lib.rs), 10 s in-process pipeline timeout (tokenless-runtime/src/lib.rs) — and section/PR attributions were cross-checked against the merged PR descriptions.Not run:
make test-integration/make test-hook-parity— this change touches no hook, CLI, or runtime behavior, and the full cargo gate above already covers every compiled target.Merge conflict resolution (2026-08-31)
Merged latest
main(througha083472cc) into this branch to resolve the conflicts introduced byfeat(tokenless)!: add protocol v2 lifecycle(#2978).Conflicts resolved
src/tokenless/crates/tokenless-protocol/src/lib.rs— took main's rewritten crate docs: the v2 refactor dropped the roadmap citations this branch had been editing, so no dangling reference remains.src/tokenless/crates/tokenless-protocol/src/tests/protocol_tests.rs— accepted main's deletion; the contract tests moved totests/protocol_tests.rswith no roadmap header note to carry over.Roadmap alignment (so the document lands current with the merged v2 lifecycle; EN + ZH): §4.1 rewritten as v1 history plus the four typed lifecycle operations and strict envelopes; seams (§4.6), adapter-boundary result facts (§4.5), token-counter location (§5.1),
ContentTypetaxonomy names (§4.2), thetokenless compressentry (§5.4), hook status, principles 2/5 wording, and the §4.1/§5.5 status rows — updated strictly to what the shipped code and #2885/#2978 encode, nothing invented.Test report (this push)
Environment: Linux x86_64, rustc/cargo 1.96.0 (stable). Commands executed from
src/tokenless/:cargo fmt --all -- --checkcargo clippy --workspace --all-targets --locked -- -D warningscargo test --workspace --lockedtokenless_protocolunit tests, the relocatedtests/protocol_tests.rs, CLI integration, and doc-tests)cargo doc --workspace --no-depsTargeted verification:
git diff origin/main HEAD -- src/tokenless/shows only the two new roadmap files — every code/test/CI state is byte-identical to main after #2978, so the conflict resolution reintroduces no code delta. Every updated roadmap statement was checked against the merged code:PROTOCOL_VERSION = 2; operationsbefore_model/pre_tool/post_tool/retrieve; strictdeny_unknown_fieldsparsing;UnsupportedVersion/OperationMismatcherrors;ContentTypewire names;PostToolResponsefields includingapplied_operations/recoverability; runtime lifecycle methods; common hooks sending v2 envelopes; stats attribution columns; Retrieve authorized against visible markers; 10 s PostTool budget;(key, generation)stash rollback.Not run:
make test-integration/make test-hook-parity/ Python hook suites — this push changes no code relative to main (docs-only delta), and the cargo gate above covers every compiled target.