Repository navigation
test: add reproducible harness containers and layered packet capture - #8
Merged
Merged
Conversation
适配本机没有安装、或被手工改过的 harness 时,测试环境既不干净、准备又繁琐;agent 还 得先装 CLI、抓包看请求头和请求体、再回到后台日志里翻出对应的那一条,中间浪费大量 token 和时间。把 harness 做成按锁文件构建的一次性容器、并把每一跳都留证,可以让 agent 直接启动一个干净的客户端并立刻拿到可断言的数据。 新增 testkit/:每个 harness 一个薄镜像(依赖由 bun.lock / uv.lock 锁定,工具链只按 官方 tag 取);统一 run.sh 契约让 runner 不认识任何 CLI 语法;L7 tap 出脱敏 NDJSON、 L4 tcpdump 共享 tap 的 netns 抓原始字节,两层靠 X-Test-Trace-Id 关联;mock 上游离线 确定,并严格校验会话历史,所以 resume 造假会直接失败。 Refs #2
批量用例只能覆盖"一个 prompt 一个回复"的形状,带 PTY 的交互式功能(旁会话、 分叉、TUI 斜杠命令)用它测不了;而手动起容器拿到的又是没配好的空容器,环境 变量、provider 配置、HOME 全得自己拼。把 run.sh 的前置准备和请求发送拆开, 就能在同一个干净环境里既跑批量用例、又交给人或 agent 手动驱动。 - run.sh 增加 MODE=shell|exec:做完 HOME/XDG 与 provider 配置准备后交出控制权 - 用 script -f 把整段终端录到 logs/terminal.log,实时 flush、退出码透传 - 新增 enter.sh 与 just testkit-shell / testkit-exec 两个入口 - 交互 shell 刻意不用 login shell:/etc/profile 会重置 PATH,把 harness 自己的 node_modules/.bin 丢掉,banner 刚推荐的命令会 command not found
Member
Author
|
#2 主要问题基本解决 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
对应 issue
Refs #2 ——「希望能够添加本项目统一的测试框架」。
matrix.json登记;just testkit-build构建cases.json驱动artifacts/testkit/runs/<run-id>/X-Test-Trace-Id打通 tap 与供应商侧;case-runs.jsonl记录每个用例的抓包行区间,可按用例切片不覆盖的部分
harness → tap → mock,验证的是测试台本身,不是插件逻辑。接真实栈只需扩CAPTURE_HOPS并复用deploy/compose.test.yaml。docker compose run --entrypoint bash),issue 里"可以直接使用的工具"目前只覆盖到批量用例这一层。reject判据已实现但 0 条用例,等网关接入后第一个"缺身份必须 400"的用例才会启用。entry一个 hop。验证
reply4/4:opencode 与 kimi 各 first → resume,回复逐字节匹配;resume 由 mock 的严格历史校验背书probe7/7:两个 harness 的身份供给与matrix.json声明一致capture13/13:NDJSON schema 完整、凭据全脱敏、SSE 识别为 streaming、tap 与供应商侧 trace 一致、pcap 可被 tcpdump 解析case-runs.jsonl的用例窗口覆盖全部抓包行,无遗漏无重叠探针做过负向验证:故意改错
matrix.json的session_identity后,断言以declared ... observed ...失败并 exit 1——不是只会通过的检查。说明
CI 按维护者意见不涉及 testkit,
check-matrix.sh与smoke.sh目前手动执行;harness 版本或 lockfile 漂移要等下次手动运行时才会暴露。