Repository navigation
Archive the trace store weekly and pull it to the GPU box - #220
Conversation
Traces are kept forever, and the design requires a weekly off-machine copy (archive plus hash manifest) so a single bot host is never their only home. - archive.make_archive (scheduler, every 7 days) packs everything not yet archived into <state_dir>/archive/traces-<seq>-<time>.tar.gz. What was archived is tracked explicitly in archive/archived.json (path -> content hash), never by timestamps. Blobs are immutable and archived once; a record file that grew is archived again; a blob renamed into place mid-archive is in the next archive. Names lead with a monotonic sequence, so name order is creation (and restore) order even if the clock moves backwards, and they are never reused or overwritten. index.db is rebuilt from the records and never archived. A .sha256 manifest (the archive's hash, then one line per member) is written last and marks the archive complete. - The publisher, which already reaches the bot host over SSH, pulls every complete archive it does not have (kb publish, each round) and keeps it in $KB_PUBLISHER_STATE/archive only if the hash verifies. A mismatch is traced and rejected; transport failures are traced, never fatal. - Restoring means extracting the archives in name order into an empty trace root, then running TraceStore.rebuild_index; the test does exactly that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1249d5ed4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| dest.mkdir(parents=True, exist_ok=True) | ||
| (dest / f".{name}.tmp").write_bytes(data) | ||
| (dest / f".{name}.tmp").replace(dest / name) | ||
| (dest / f"{name}.sha256").write_text(manifest, encoding="utf-8") # last: marks it complete |
There was a problem hiding this comment.
When a remote archive exists but $KB_PUBLISHER_STATE/archive cannot be created or written—for example because that directory is read-only or a transient filesystem error occurs—these operations raise an uncaught OSError. Since Publisher.serve does not catch errors from sync_archives, the publisher exits and stops processing all future outbox actions; handle local persistence errors per archive and trace them instead.
Useful? React with 👍 / 👎.
| for name in names: | ||
| if (dest / f"{name}.sha256").exists(): | ||
| continue |
There was a problem hiding this comment.
If a previously pulled tarball is deleted or corrupted while its .sha256 sidecar remains, every subsequent round skips it solely because the sidecar exists. The GPU box then permanently lacks a valid off-machine copy even though the source archive remains available; require both files and verify the stored tarball before treating an archive as already pulled.
Useful? React with 👍 / 👎.
| data = path.read_bytes() | ||
| digest = _sha(data) | ||
| if archived.get(rel) != digest: | ||
| members.append((rel, data)) | ||
| archived[rel] = digest |
There was a problem hiding this comment.
On the initial archive or after a busy week, every unarchived prompt/output blob is read into and retained by members, after which the complete compressed tar is also built in a BytesIO. Memory therefore scales to roughly the raw weekly trace volume plus the archive, so a mature trace store can exhaust memory or make every archive attempt fail; write members directly to a temporary tar file and compute hashes while streaming.
Useful? React with 👍 / 👎.
Knowledge-service redesign follow-up: the design's weekly off-machine archive of the trace store (archive plus hash manifest).
archive.make_archive(scheduler, every 7 days) packs everything not yet archived intoarchive/traces-<seq>-<time>.tar.gz. A.sha256manifest is written last to mark it complete.archived.json(path → content hash), never by timestamps. Blobs are immutable and archived once; a record file that grew is archived again; a blob renamed into place mid-archive lands in the next archive.index.dbis rebuilt and never archived.TraceStore.rebuild_index. The tests do this, including after a clock rollback.Commit hook approved; full suite and SPEC freshness pass locally (rebased on #219).
🤖 Generated with Claude Code