Skip to content

fix(backup): distinguish runs from wall-clock timestamps - #12052

Merged
bolichen97 merged 2 commits into
mainfrom
fix/backup-run-identity-20260919
Sep 19, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
fix/backup-run-identity-20260919

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Backup run timestamps are wall-clock observations, not unique identities or monotonic order. Equal clock readings reproduced the Windows CI failure in TestALostRunWriteDoesNotReUploadForever::test_two_runs_in_the_same_second_are_distinguishable. More importantly, after an uploaded archive's local state write fails, timestamp comparisons can hide its newer run or acknowledge a different run, losing recovery metadata.

Why it matters

A completed upload must remain visible to the nightly scheduler and retain its bounded restore fingerprint while this process recovers local persistence. Merely advancing the test clock masks this production defect.

What changed (motivation → approach → change)

  • src/kiro_crew/apps/builtins/aws_control/backend/backup.py retains truthful at values and adds a process namespace/local sequence. A new sidecar-locked run always replaces its kind's prior state; only pending overlay/recovery uses recency comparisons. Its existing staging-prefix expression uses equivalent string concatenation so the unversioned Fast Gate parser can read the module; hashing, byte limits and supported runtime versions are unchanged.
  • Local recording, status reads and recovery share a process lock. State updates merge pending fingerprints before acknowledging the exact processed records. Pending fingerprints use the existing 200-entry bound; last-run storage remains one record per account/kind.
  • test/test_aws_control_backup.py covers equal/backwards clocks, prior-process/legacy records, failed reads/writes, exact acknowledgement, bounded fingerprints and barrier-synchronized recovery. Assertions that incorrectly equated timestamps with identity are replaced, not bypassed with a faster clock.
  • docs/system-specs/modules/aws-control.md records these guarantees and limits.

This is separate from the CI migration: no runner, workflow, UI, translation, permissions or upload-consent changes. No full-history collection or public unknown-order envelope.

Backwards compatibility

Compatible: existing run fields and legacy records remain readable; process/sequence metadata is additive. Cross-process/legacy recovery keeps the existing wall-time fallback, not a guarantee of global newest. New locked writes remain authoritative across restarts. Restarting before recovery still loses process-only metadata. The existing unresolved state-path sentinel recovery limitation is not redesigned.

Tests

  • Parent: 268 backup/application tests passed with imports verified against this worktree, including exact staging-name output for ASCII, Unicode and long basenames. Real Python 3.10/3.11 parse the whole module; the previously failing comment-history gate passes under Python 3.11. This is parser compatibility, not expanded supported runtime versions.
  • Equal/backwards identity/recovery selection: 36 cases passed in each of three fresh processes.
  • Negative controls: old production failed 13 regressions; timestamp-only acknowledgement failed; dropped fingerprint recovery failed 12 cases. The discarded candidate's authoritative-write guard failed 12 restart/legacy cases, and unconditional recovery failed four cases.
  • Parent isolated setuptools backend build passed; built backup source matched the candidate. Applicable backend static gates passed, including Black, isort, flake8 and mypy.
  • Independent GPT and Opus CI-contract reviews: no findings on 6976aeb7d241c97084d0e3e1ef9dddaee853829c. Focused independent verification of the syntax-only follow-up and three output cases: no findings on e7b1823eb10d44b1633147b6e0c9120101b50237.

Manual verification

Real local state files, injected persistence errors and thread barriers were exercised on Linux. Current-head CI run passed all eight Linux and eight native Windows backend shards, frontend tests, E2E, coverage and lint. Windows shard 7 ran on Windows Server 2025 / Python 3.12.10; its per-case progress artifact records all 36 identity/order/recovery/acknowledgement regressions passing with zero skips in that set. The full backup module recorded 110 passed and four existing archive-capability skips. Official readiness verification passed on e7b1823eb10d44b1633147b6e0c9120101b50237: 86 checks, zero running or failing, fresh GPT/Opus, no unresolved threads. Native macOS was not selected by the existing workflow; no live AWS upload/restore is claimed. No frontend source or UI changed.

Related Issues

Separate product fix for the backup failure exposed by PR #11880; that PR remains pure CI. Open-PR overlap scan found no backup implementation/test overlap; PR #7140 changes an unrelated section of the same specification.

Pattern harvest

Rule candidate: review-prompt
Pattern: distinguish observed wall time, operation identity, authoritative write order and exact persistence acknowledgement; a recency comparison is not a substitute for any of them.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing related tests pass and new tests added for new functionality
  • Parent self-review and independent GPT/Opus reviews completed
  • Documentation updated
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and chenmingwei23 September 19, 2026 17:07
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Distinguish same-process backup runs independently of equal or backwards wall clocks, and recover bounded failed-state-write records/fingerprints without acknowledging another run. Keep newly locked successful writes authoritative across process restarts.
Not a goal: No CI migration changes, UI/translation changes, full backup history, public unknown-order envelope, cross-process global ordering, live AWS changes, permission changes or automatic merge.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 19, 2026
@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of e7b1823eb10d44b1633147b6e0c9120101b50237 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the Windows CI shard is green at this head — the same-second/backwards-clock paths were only exercised on Linux with a mocked clock.

Suggestions

  • last_runs/uploaded_objects now wait on _run_lock, which _record_run holds across the cross-process sidecar file-lock wait and disk I/O; scoping reads to the in-memory overlay would keep status and due_for_nightly responsive while another install holds the file lock.

[DESIGN-REVIEWED] e7b1823

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of e7b1823eb10d44b1633147b6e0c9120101b50237 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e7b1823

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e7b1823eb10d44b1633147b6e0c9120101b50237: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of e7b1823eb10d44b1633147b6e0c9120101b50237 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the Windows shard on this head: the equal-clock defect is Windows-only and the author confirms no native Windows execution was run.

Not justified as shipped

  • Item 7 — rides along: "uses equivalent string concatenation so the unversioned Fast Gate parser can read the module" — a parser-compat fix separate from the run-identity defect, though declared and itself a real pre-3.12 SyntaxError fix.

What this change ships

Inventory (8 items) — 7 justified

Intent: FIX — keep a completed backup upload's last-run record and restore fingerprint correct when wall clocks tie/go backwards or a local state write fails.

  1. Two same-second or clock-backwards runs now select the correct last run — justified
  2. Run records persist additive process/sequence identity metadata — justified
  3. A run lost to a failed state write reaches disk on the next successful state update — justified
  4. Restore proof lists every pending upload's fingerprint, under the existing 200-entry bound — justified
  5. Acknowledgement clears only the exact persisted run; the equal-stamp eviction pin is replaced with the misacknowledgement defect it caused named and tested — justified
  6. Recording, status reads and recovery serialize on one in-process lock — justified
  7. Staging-prefix expression rewritten so Python 3.10/3.11 parse the module — rides along (declared parser-compat fix, separate defect from run identity)
  8. Spec section records the new run-identity and failed-write guarantees — justified

[FIRST-PRINCIPLES-REVIEWED] e7b1823

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed e7b1823eb10d44b1633147b6e0c9120101b50237 — this comment is updated in place on each push.

Review details

The single candidate is a deliberate liveness/perf tradeoff (read calls now serialize behind an in-process lock the author intentionally added — documented in the spec hunk), not a correctness defect: lock ordering is consistent (_run_lock → _unpersisted_lock everywhere, _run_lock is re-entrant), so no deadlock, no crash, no data loss. It is self-rated low and describes only a possible stall behind a fast atomic write. It does not clear the 80+ correctness bar. I examined the recovery/acknowledgement paths, _merge_pending/_forget_unpersisted equality clearing, and the fingerprint-migration path and found no grounded defect in the changed lines.

No findings.

[OPUS-REVIEWED] e7b1823

Verdict parsed from the review's SHA-scoped output markers for commit e7b1823eb10d44b1633147b6e0c9120101b50237.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e7b1823eb10d44b1633147b6e0c9120101b50237: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Read-side serialization — rebutted; span=9611dfdfd206

The wait is real, but removing the shared snapshot/acknowledgement lock alone reopens the gap where a reader loads old durable state, a writer persists and clears the overlay, and the reader then sees neither the pending record nor the new durable record.
Existing call sites offload these reads to threads. The protected section covers local JSON state and sidecar serialization, not archive building or AWS upload. It can still wait behind another process; no latency bound is claimed.
Keep the documented minimal consistency tradeoff rather than add a second snapshot/version protocol for this advisory. Current-head Opus independently classified it as a liveness/performance tradeoff, with no grounded correctness finding.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Staging-prefix parser compatibility — rebutted; span=84f22695822c

The one-line equivalent expression is required to execute this PR native CI: Fast Gate job105934163117 failed parsing the already-existing nested f-string and skipped every backend shard on6976aeb7d241c97084d0e3e1ef9dddaee853829c.
The follow-up preserves exact SHA-256 prefix and byte-bounded filename outputs; three ASCII/Unicode/long-name regressions and all268 backup/application cases pass. Actual Python3.10/3.11 parsing and the3.11 comment-history gate pass. Current e7b182 Fast Gate now has all12 jobs green and both native platform matrices running.
This is a declared, behavior-neutral validation prerequisite within the same file, not a new supported runtime, CI migration, or backup feature. A separate prerequisite PR adds coordination without reducing semantic risk.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Native Windows validation — fixed

Current-head Windows shard7 passed: https://github.com/kirodotdev/KiroCrew/actions/runs/35457951853/job/105936682974 . The log identifies Windows Server2025, Python3.12.10 and merge e7b182 into917b54.
Downloaded windows-pytest-progress-7 artifact10589416141 and checked per-case phase_end records: all36 identity/order/recovery/acknowledgement regression cases passed, zero failures or skips in that set. The backup module has110 passed and4 existing platform-skipped archive cases. The full shard reports13009 passed,895 skipped,11 xfailed.
Other platform shards and aggregate coverage/readiness are still pending; this evidence closes only the requested native backup validation, not overall readiness.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Native Windows provenance — fixed

https://github.com/kirodotdev/KiroCrew/actions/runs/35457951853/job/105936682974 passed on Windows Server2025/Python3.12.10 at the current head merge. Artifact10589416141 records all36 identity/recovery regressions passing with no skips in that set, including both same-time/backwards variants and all12 failed-read/write recovery variants.
This is first-hand current-head native Windows evidence, not the earlier Linux mocked-clock report. No live AWS round trip is claimed; unrelated remaining shards and overall readiness still need completion.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision labels Sep 19, 2026
@bolichen97
bolichen97 merged commit ee521b0 into main Sep 19, 2026
94 checks passed
@bolichen97
bolichen97 deleted the fix/backup-run-identity-20260919 branch September 19, 2026 17:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants