Skip to content

fix(sight): warn when data dir is on overlay storage - #2977

Merged
chengshuyi merged 1 commit into
alibaba:mainfrom
jfeng18:fix/sight/container-data-persistence
Aug 31, 2026
Merged

fix(sight): warn when data dir is on overlay storage#2977
chengshuyi merged 1 commit into
alibaba:mainfrom
jfeng18:fix/sight/container-data-persistence

Conversation

@jfeng18

@jfeng18 jfeng18 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Why

Deployed as a container, all observation data under /var/log/sysak/.agentsight is silently lost on every container restart — the directory lives in the container's writable layer and nothing in the repo told operators to mount a volume (#2826). Deterministic writable-layer semantics, reproduced 3x by the reporter. This also voids the 0.11.0 size-governance work in the container form: bounded retention presumes the data survives in the first place.

What changed

Docs (bilingual, per scoped AGENTS.md) — both deployment.md container/sidecar sections now carry a mandatory-persistence paragraph plus a reference sidecar YAML excerpt (hostPath at /var/log/sysak/.agentsight), and both data-and-storage.md retention sections cross-link to it. hostPath is recommended for host-scoped observation data; PVC only when data must follow the pod; emptyDir noted as surviving container restarts but not pod recreation.

Code — a startup warning closes the silent-loss gap: config::path_on_overlayfs resolves the storage directory against /proc/self/mounts (longest-prefix match, handles the kernel's \040 space escapes), and warn_if_data_dir_not_persistent logs once when it lands on overlay (the container writable layer). Wired into all three startup paths: AgentSight::new (after the config load, so the warning targets the effective directory), serve, and the trajectory-only collector path.

Not included: no standalone manifest artifact (the repo has no K8s-manifest convention or owner; the inline reference YAML covers the intent and adding a file later is additive if real demand shows up), no Dockerfile/entrypoint change.

Related issue

closes #2826

User / Agent impact

Operators deploying the container image without a volume now get an explicit warning at startup instead of silent data loss, plus documentation of the correct volume form. No behavior change for host/RPM installs (non-overlay → no warning).

Risk and compatibility

  • Public CLI, API, configuration, or documented behavior changed
  • Privileged or security-sensitive behavior changed
  • Cross-component contract changed
  • Migration or rollback guidance is needed

Low risk: one new warn-level log on a specific mount topology; no functional change. The Fixes: trailer is intentionally absent — this is a delivery-form gap, not a regression.

Validation

On ECS (Alinux, kernel 6.6, rustup 1.89.0 matching CI):

  • cargo fmt --all --check — pass
  • python3 scripts/check-arch-boundaries.py — PASS
  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo test --workspace --lib --bins --tests1935 passed, 0 failed, including the new test_path_on_overlayfs_parsing covering: overlay root, hostPath mount winning over the overlay root (longest-prefix), plain host, kernel \040 space escapes, and empty mount table.

Documentation and rollback

This PR is the documentation change (both languages updated together). Roll back by reverting the single commit; the warning disappears and the docs revert.

@github-actions github-actions Bot added component:sight src/agentsight/ scope:documentation ./docs/|./*.md|./NOTICE labels Aug 29, 2026

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 本次变更主要补足容器部署数据持久化告警与文档,整体实现符合 agentsight 约束,仅有少量范围澄清与平台一致性建议。


🤖 Generated by QoderView workflow run

Comment thread docs/user-guide/en/agent-observability/agentsight/deployment.md
Comment thread src/agentsight/src/config.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b84bd7e996

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agentsight/src/bin/cli/serve.rs Outdated
Comment thread src/agentsight/src/bin/cli/trace.rs Outdated
Comment thread docs/user-guide/en/agent-observability/agentsight/deployment.md

@chengshuyi chengshuyi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

方向对,实现也克制(150 行、纯函数拆出来可测、bilingual docs 同步、无 unwrap)。但有一个 CI 硬阻塞和两个真实缺陷,先修一轮。

阻塞

1. CI 挂了 —— 增量覆盖率 70% < 80%(Test agentsight

未覆盖行:

  • src/agentsight/src/config.rs:1400-1405 —— path_on_overlayfs 包装层完全没测,测试全打在 path_on_overlayfs_in
  • src/agentsight/src/config.rs:1412-1421 —— warn_if_data_dir_not_persistent 没测
  • src/agentsight/src/config.rs:1435 —— 畸形行的 continue 分支没测
  • src/agentsight/src/unified.rs:214 —— 调用点
  • src/agentsight/src/config.rs:1619 —— assert! 的自定义 message 是独立 coverage region,assert 通过时不执行,白扣一行

补两个 wrapper 的测试 + 去掉那句 assert message 就能过线(估算约 86%)。

2. config.rs 已 2470 行,本 PR 又加 62 行

AGENTS.md:「超过 2,000 行的文件在增加代码前必须先有拆分计划」。

更关键的是有更合适的落点:src/container.rs(154 行)已经在解析 /proc 做容器检测,而且结构完全同构 —— parse_container_id_from_cgroup(纯函数)+ extract_container_id(读 /proc 的薄壳)。把 path_on_overlayfs_in / path_on_overlayfs 搬过去,两个问题一起解决。

缺陷

3. trajectory-only 模式警告的目录是错的src/bin/cli/trace.rs:183

这里用 config.storage_base_path 告警,但该模式实际写库路径来自 resolve_trajectory_db_path()sibling_db_path("trajectories.db"),失败再退到 $HOME/.local/share/agentsight/。两者不是同一个目录。

而这个模式恰恰是给无特权容器沙箱用的,正是该告警最需要准的场景,现在报的是一个跟实际写入无关的路径。建议挪到 db_path 解析之后,用 db_path.parent()(跟 serve.rs 保持一致)。

4. 不解析符号链接 → 误报config.rs:1426

to_string_lossy() 直接做前缀匹配。运维把 /var/log/sysak 软链到挂了卷的 /data 是常见做法,这种情况会误报「数据会丢」。建议先 canonicalize,失败再退回原始路径(目录可能还不存在)。

5. 同长度挂载点的 tie-break 反了config.rs:1441

mount_point.len() > best_len 是「先出现的赢」。但 /proc/self/mounts 里同一挂载点上后挂的那个才是生效的(stacked mount、subPath 叠加)。应该用 >=

非阻塞

  1. doc comment 写「Warn once」,但函数本身没有去重机制,是靠调用点保证的。要么删掉 "once",要么加 std::sync::Once
  2. 三个调用点数据源不统一:tracestorage_base_pathserve--db 父目录。--db 指到别处时两条命令告警的是不同目录。
  3. 告警文案 "see the deployment guide" 偏虚,可以直接点出页面(Containers and sidecars / 容器与 Sidecar)。

已确认没问题的点

  • 锚点 deployment.md#containers-and-sidecars#容器与-sidecar 都对得上实际标题
  • emptyDir(tmpfs) 不触发告警,与文档表述一致;最长前缀匹配能让 hostPath 覆盖 overlay 根,测试也覆盖了
  • 非 Linux / 读不到 /proc/self/mounts 时返回 false,macOS 与裸机安全
  • 三个调用点不会重复告警:trace 的调用在 run_trajectory_only--no-ebpf)分支内,与 AgentSight::new 互斥
  • 无 CLI/API/配置结构变化,不需要 bump schema_versioncheck-arch-boundaries.py 与 Docs Lint 均通过

Container deployments silently lost all captured data on every
restart: the databases live in the writable layer and nothing told
operators to mount a volume. Document the mandatory volume in the
container/sidecar guidance (bilingual) and warn once at startup when
the effective storage directory resolves onto an overlay filesystem.

Assisted-by: Qoder:1.25.1
Signed-off-by: Jiangtian Feng <jiangtianf97@163.com>
@jfeng18
jfeng18 force-pushed the fix/sight/container-data-persistence branch from b84bd7e to 382a042 Compare August 30, 2026 15:42
@jfeng18

jfeng18 commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator Author

全部采纳并已修,逐项:

阻塞

  1. 覆盖率:补上 path_on_overlayfs wrapper 与 warn_if_data_dir_not_persistent 的测试 + 畸形行 continue 分支 fixture(overlayfs_malformed_line_is_skipped)+ 改写断言消除惰性求值 region。本地 cargo test --workspace 1943/0。
  2. 搬迁至 container.rs:与 parse_container_id_from_cgroup(纯函数)+ extract_container_id(/proc 薄壳)同构,config.rs 回到改动前规模。

缺陷
3. trajectory-only 告警挪到 resolve_trajectory_db_path() 之后,对 db_path.parent() 告警(与 serve 一致)——现在对准实际写入目录(共享目录或 $HOME fallback)。
4. path_on_overlayfscanonicalize(),目录不存在时退回原始路径——symlink 到挂卷场景不再误报。
5. tie-break 改 >=——同挂载点叠挂时后挂(生效的)赢,新增 overlayfs_stacked_same_mountpoint_last_wins 测试。

非阻塞
6. doc comment 删掉 "once"(调用点每进程一次,无内置去重)。
7. 数据源差异是有意的:trace 写整个 storage 目录(多个库)所以对 storage_base_path 告警;serve 只读 --db 指定的库,所以对该 db 的父目录告警——两条命令的告警都对准各自的写入/读取面。
8. 文案已点名章节("see the deployment guide, section "Containers and sidecars"")。

另吸收了 bot 的三条:serve 独立运行时 logger 未初始化导致 warn 被静默丢弃(已在告警前 apply_verbose())、PVC 随 Pod 生命周期的提示、以及 hostPath 目录权限(DirectoryOrCreate 出 0755 + sidecar 无 UMask=0077,文档已提示预建 chmod 700)。

ECS 门禁:fmt ✓ / arch-boundary PASS ✓ / clippy -D warnings 零告警 ✓ / 1943 passed 0 failed ✓

@chengshuyi chengshuyi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

上一轮 8 条意见已全部修复:代码移至 container.rs、trajectory-only 告警目录修正、canonicalize 防软链误报、tie-break 改 >=、wrapper 测试补齐(CI 覆盖率已绿)、doc comment 与告警文案修正。bot 意见(PVC 生命周期、overlayfs 范围、serve logger、hostPath 权限)也均有落实。LGTM。

@chengshuyi
chengshuyi merged commit 45f97d9 into alibaba:main Aug 31, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:sight src/agentsight/ scope:documentation ./docs/|./*.md|./NOTICE

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TestAgent][sight] container deployments silently lose all observation data on restart — no persistent storage for the DB directory

2 participants