Skip to content

fix(checkpoint): bound artifact member IO - #6781

Merged
loopx-agent merged 1 commit into
mainfrom
codex/checkpoint-bounded-io-20261018
Oct 18, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/checkpoint-bounded-io-20261018

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Growing declared research directories can exhaust a checkpoint capture window while the artifact owner serially inspects, hashes, copies and flushes each member. On a fixed complete 14,532-file / 44.8 MB inert directory, an alternating uninstrumented base/head/head/base comparison measured capture at 152.3–166.6 s → 28.7–28.9 s, with identical canonical source and artifact manifests.

This change bounds independent member IO to eight jobs, drains started work before failure cleanup, preserves the existing depth-first manifest order, and retains every source/copy/artifact check. It also hoists the per-component root inspection and indexes copied members instead of repeatedly searching the inventory. These are local helpers in the existing TypeScript artifact owner, with no new capability, provider or persisted vocabulary.

  • Outcome basis: reproduced capture-window failure; intended base: main.
  • Owning acceptance: reliable recovery/artifact integrity in the existing S2/S10 path. This is a bounded artifact-stage repair.

Author Declaration

Criterion Disposition Owner / evidence
Complete declared inventory with immutable byte/mode and source witnesses implemented checkpoint_artifacts.ts; independent filesystem inventory, nested ordering and copy-time ABA tests; full-size transport and ordinary capture/readback
Source movement, SQLite committed-WAL semantics, finite deadlines and exclusive publication implemented existing snapshot/publication owners retained; real filesystem, SQLite and Docker failure/recovery tests
Inert verification/materialization without imported execution authority implemented packaged CLI/Effect capture → verify → materialize → corruption rejection; actual Docker/native HTTP journey

Self-check covered the complete diff, unchanged manifest semantics, started-IO cleanup, source/copy validation, existing consumers and package contents. The related refactor is limited to indexing/invariant inspection within the existing verification phase.

Scope And Continuation

Complete within this artifact-stage scope. Capture remains explicit; ordinary heartbeat input, native opportunities, cooldown, source barriers and capture/sealing clocks retain their existing owners. There is no frontend/CLI/Lark configuration or interaction change.

Source handoff/export, actual late-point research admission, model evidence adoption and strategy outcomes retain their existing separate acceptance. The measurements below are one fixed-workload comparison and functional qualification, not p95, a sustained production SLO or a research-quality gain. Frozen research workers were not updated.

Validation

  • Tested revision: f9c7afc32e723923a25e9a8f62ba64ebc26f3113 (tested source bytes match the committed owner).
  • Run state: finished.
  • Input classes: synthetic; authorized_private_read_only. Private inputs, profiles and logs remain private.
Check kind Result Public-safe evidence / limitation
unit passed tests/control_plane_ts/checkpoint_artifacts.test.ts: 18 tests on macOS and Linux; nested/punctuation ordering, byte/mode preservation, ABA rejection and bounded drained cleanup. Characterization also passes on the unchanged base.
real_entrypoint / real_backend passed Checkpoint transport, capture, restore gate, provider fence and context tests: 96 passed; 11 optional real Codex Host variants skipped because no explicit native Host binary was bound. These skips do not qualify native session continuation.
integration passed Checkpoint worker/controller/archive/qualification/handoff families: 165 unique tests passed, including 34 real Docker writer-barrier cases and five Host-phase/next-opportunity cases using the compatible native framework environment. Grading in these fixtures is synthetic.
regression_parity passed Complete 14,532-file alternating base/head/head/base run on the same runtime and shared-volume fixture, with identical canonical manifests; median artifact capture 159.4 s → 28.8 s. Two samples per revision, not a percentile estimate.
real_entrypoint / real_backend passed Same complete directory through a prepared handoff, real Docker/native HTTP and the shared export/capture volume: capture 55.9 s under the original 90 s live / 30 s sealing bounds; inert materialization 15.0 s. All ordinary file bytes/modes verified; writer released; no model calls or benchmark grading. The same fixture also passed on local storage.
real_entrypoint passed Built wheel installed in an isolated Linux environment: packaged owner digest matches the candidate; CLI verification, capture/materialization, read-only mode and corruption rejection passed.
static passed Control-plane TypeScript typecheck, current-diff semantic advisory, bilingual RFC diff check and git diff --check. New helper/tree shape is local; no shared vocabulary added.
manual passed Public/private scan and source/runtime readback. Initial qualification setup errors (writer image executable, incomplete native dependencies and an unprepared handoff) were corrected; only successful final paths count above. Owned failed fixture containers were removed.

Visual Evidence

UI impact: none. No viewport, copy, navigation or interaction changed. The existing frontend bundle was built only to validate the wheel; no visual qualification is claimed.

Type of Change / LoopX Area

  • Bug fix
  • Refactoring (behavior-preserving verification/indexing seam)
  • Test update
  • Control plane runtime

Shared-authority RFC fixture impact

N/A: no authority-provider promotion, routing, migration or compatibility projection changed. The existing physical SQLite snapshot owner is reused.

Boundary Checklist

  • Public-safe diff/body; no credentials, raw research evidence, local paths or private links.
  • Existing owner reused; no duplicate benchmark, state or decision authority.
  • Scope limited to artifact IO and its durable regression tests.
  • No paired RFC changed; bilingual diff check passed.
  • UI impact recorded as none.
  • DCO sign-off included.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

研究者显式保存一个不断增长的工作目录时,需要在有限捕获窗口内保留全部文件和已提交数据库内容。
此前目录成员逐个读写,文件越多等待越长;现在最多八个独立成员同时处理,最终仍按原顺序生成完整清单,失败后先等在途写入结束再清理。
独立基头与当前实现对照验证了实际并发、完整字节和权限保持,以及源变化、失败清理、截止时间和数据库内容的拒绝或保留语义。
本次不扩大捕获范围、延长时限、调用真实模型或授予历史执行权,也不宣称安装版、跨平台持续运行或研究分数和长期效率已合格。
既有普通 Goal 打包界面、真实原生模型接续及持续生产和研究收益资格保持开放;作者的大目录实测没有被本轮受控对照替代。

不处理这一问题,大目录仍会把原本独立的磁盘等待串在一起,可能错过已冻结的捕获窗口;增加时限、漏掉文件或要求使用者手动拆目录都没有保留原任务。当前修改复用既有显式入口,不增加设置、重新输入或额外确认轮次。作者报告的 14,532 文件实测保留为作者证据;本轮独立验证的并发和完整性结果见下文,不据此推导生产时延承诺。

改动思路

完整清单、数据库快照和原子发布继续归既有 artifact owner;只把独立成员 IO 限为八个任务,并等待已启动任务结束,因此无需新增状态、配置或恢复 authority。
本 PR 交付既有显式检查点入口的 artifact 阶段修复;完整普通 Goal 界面、原生模型接续和长期研究效果仍由已有恢复与研究验收负责。

入口仍是显式 CLI/Host 经 runtime.checkpoint_artifacts 进入原模块。每一层目录的独立 IO 可以重叠,投影时仍按原深度优先顺序输出;复制后的查找只建立本地索引。原组件顺序、源前后比较、逐文件复制结果比较、数据库物理快照、最终截止时间和独占发布均保留。任一成员失败时,停止领取新项,并等待已开始的复制和 flush 返回,随后才由原 finally 清理 staging。八个任务是每次调用的成员并发上限,不是全机器文件描述符上限。

阶段 原实现 当前实现与保留条件
枚举、散列 逐项等待 每层最多八项 IO,最后还原同一完整 DFS 清单
复制、flush 逐项等待 独立成员重叠,失败先 drain,SQLite 仍走原 snapshot owner
发布前核对 每项线性寻找副本 复用根类型与 Map,仍核 size/hash/mode、源变化和原时限

该表是准确 head 的源码说明,执行证据来自下面的实际对照。相邻改进限于根类型复用和副本索引;没有理由另造调度器、设置或兼容协议。

具体改动

完整差异只有两个既有文件:生产代码 +80/-42,测试 +83。规范是原基头的显式研究检查点说明,revision 105c4bc15091afac832495167cb98a1389951a04;同时核对原路线图 S2/S10。其完整声明目录、源稳定性与原时限、新目的地独占发布、已提交 SQLite 和惰性接收方四组要求在当前头实现;原生模型、普通 Goal 打包界面与长期效用仍按规范保留为后续资格,未被本次局部修复关闭。

逐项映射:complete-declared-inventory 已实现,清单保持全部声明成员及原先的深度优先顺序。quiescent-source-and-exclusive-publication 已实现,源前后变化、复制期字节差异、原截止时间和已有目的地都仍阻止不合格发布。physical-sqlite-and-inert-receiver 已实现,提交到 WAL 的数据由原数据库快照封存,显式单独选择的 sidecar 保留原字节,接收方仍不激活执行。remaining-product-and-research-qualification 留在既有父级验收,当前没有宣称真实模型采用、安装版或长期收益完成。

关键代码讲解

  • mapMembers 最多启动八个 worker,按原索引保存结果;失败后等待所有已启动 visit 结束才抛出。
  • members 将 IO 前沿与清单投影分开。树节点只承载本次观察,最终栈遍历保留原 DFS;不能用完成顺序或完整相对路径排序替代它。
  • copyComponent 保留普通字节复制、只读文件 flush 和既有 SQLite snapshot。嵌套数据库 sidecar 与显式单独选中的 sidecar 继续区别处理;materialize 仍复制封存字节。
  • checkpointArtifacts 保留原 source/digest/逐副本比对与原子发布顺序;新增 Map 仅替代反复线性查找,没有给历史记录增加执行权。

新增三项测试覆盖嵌套与标点目录顺序、复制期 ABA(源只在复制时短暂变化后恢复)、失败后在途 IO 结束再清理。全部 18 项在当前头通过;把同一特征测试绑定不可变原 owner 后,18 项也通过,因此没有把“测试变绿”当作并发已发生的证明。

对主干的风险

独立实际文件系统对照使用同一 103 文件、109 项目录清单和混合权限,完整源清单、artifact 清单及整个 manifest 在 B/H 逐值相同,materialize 后逐文件 hash/mode 再读回。受控复制延迟下实际峰值为 B=1、H=8。另独立覆盖失败后 active=0 与无未发布 staging、嵌套复制期 ABA 拒绝、真实 SQLite 已提交 WAL 查询、显式 standalone WAL 原字节,以及操作已经开始并复制了 103 文件后仍因原截止时间拒绝发布。没有用合计数量代替逐成员证据。

首次私有夹具误要求失败场景启动八个文件,但首层实际是四文件和四目录,原断言因此失败。已按独立文件清单修正首层文件数量,保留八项上限、完整内容与 drain/清理断言;原脚本和失败记录保留,没有修改产品代码或仓库测试。受控计时只说明重叠机制,不是生产 SLO、p95 或真实研究收益。

真实 source-checkout CLI/Effect、File/SQLite、捕获、provider fence、恢复与上下文采用七个测试文件共 103 passed、12 skipped;跳过的是缺少显式原生 Host 二进制的可选变体,Host IO 合成边界不冒充真实模型或安装态。实际 8192 文件的大清单跨越普通 RPC 限制,并完成 verify/materialize 与损坏拒绝。TypeScript 检查、标准 premerge 的 4 项直接检查及 16 项选择检查、两个公开文件扫描均通过。wait_for_ci=false,没有查询、轮询或等待 GitHub CI。

必需的根 Python 模块预算仍是 1 failed、5 passed,146 > 145,具体为 workflow_skill_install.py。已在不可变 B 和准确 H 执行同一命令,除测试耗时外完整失败身份和详情一致;checker、预算、offender 的 blob 及完整顶层模块列表均相同。该失败与本次两个 TS 文件的因果路径无关,保留为原 #6768 owner 的独立合并门,不放宽预算。

我的整体评价

对 f9c7afc32e723923a25e9a8f62ba64ebc26f3113 给出代码层面的 APPROVE。这是可独立验证、可回滚的 artifact 阶段增量:并发机制实际生效,原完整性、数据库、惰性接收方和拒绝语义保留。长期工作的证据保存和后续采用机制保持,使用者沿既有入口完成同一任务且无额外交互轮次;真实模型的持续净收益尚未证明。新 Node/frontier 只属局部执行形状,复用既有 vocabulary 与单一 owner,没有新增共享状态或配置。

完整清单、数据库快照和原子发布继续归既有 artifact owner;只把独立成员 IO 限为八个任务,并等待已启动任务结束,因此无需新增状态、配置或恢复 authority。
本 PR 交付既有显式检查点入口的 artifact 阶段修复;完整普通 Goal 界面、原生模型接续和长期研究效果仍由已有恢复与研究验收负责。
原模块预算失败继续单独阻塞合并资格;本结论不授予合并、部署、安装或更新活动研究 worker 的权限。

English verdict: APPROVE — f9c7afc. Bounded member IO preserves complete DFS manifests, byte/mode and committed SQLite semantics, drained cleanup, original deadlines and inert authority. Independent B/H comparison measured peak1 vs8 with identical complete manifests;18owner tests,18baseline characterization and103real CLI/Effect/context tests passed,12optional native Host cases skipped. Standard premerge/typecheck passed. The unchanged146/145root-module failure remains a separate merge hold; no installed-model, production-SLO or research-gain claim.

@loopx-agent
loopx-agent merged commit b5a331d into main Oct 18, 2026
25 of 35 checks passed
@loopx-agent
loopx-agent deleted the codex/checkpoint-bounded-io-20261018 branch October 18, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant