Repository navigation
fix(security-sync): serialize owner syncs and guard run progress - #7023
Conversation
Concurrent scheduled and manual runs for one owner overwrote the single runtime_state.sync_run progress slot, so every continuation found a mismatched runId, restarted at repo 0, exhausted the budget, and re-enqueued forever. last_synced_at never advanced and the queue saturated, so manual syncs were delivered after the reconciler had already failed them. Add a single-flight owner lease (runtime_state.sync_lease) and fence every sync_run write, clear, and freshness update by runId and chunkIndex. A completed-run marker stops a finished run being resurrected, a no-progress cap stops a run that never advances, and the queue consumer now acks denials/refusals without enqueueing a successor. Add diagnostics: alerts_disabled status and a redacted body excerpt, per-repo timing and budget, the lease lifecycle, and a per-command stalled-command log in the auto-analysis dispatcher.
Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Executive SummaryThe owner-sync lease/CAS design is sound and well covered by the new unit, fake-DB, and real-Postgres integration suites; the two findings are non-blocking scale/availability observations, not correctness defects. Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (11 files)
What was verified: lease claim takeover/fencing (including the expired-foreign-lease and same-run chunk branches), Fix these issues in Kilo Cloud Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0 Review guidance: REVIEW.md from base branch |
Summary
Fixes the Security Agent sync stall documented in
HANDOFF.md: one customer'slast_synced_atwas frozen at 2026-09-12 and manual syncs failed withCOMMAND_STALLED.Root cause.
agent_configs.runtime_state.sync_runwas a single per-owner progress slot. It was read with arunIdguard but written and cleared without one, so the 6-hourly scheduled run and manual runs for the same owner overwrote each other. Every continuation then found a mismatchedrunId, restarted at repo 0, exhausted the 8-minute budget, and re-enqueued forever. The resulting message storm saturated the queue, so a manual command's message was delivered after the accepted-timeout reconciler had already failed it.Changes
runtime_state.sync_lease = { runId, chunkIndex, expiresAt }. A foreign run is denied while the lease is live and enqueues nothing; a strictly older chunk of the holding run is denied asstaleChunkand acked.writeSyncRunProgress,clearSyncRunProgress, andadvanceOwnerSyncFreshnessare now conditionalUPDATE ... RETURNINGcompare-and-swaps fenced byrunIdandchunkIndex.completedReposis unioned so a late write cannot shrink the cursor; a swallowed freshness failure no longer reports success.last_completed_run_id) so a redelivery of a finished run is recognized and not resurrected.security-sync:SYNC_ALREADY_RUNNING,SYNC_SUPERSEDED,SYNC_NO_PROGRESS(failed, notno_op); the lease is released on the already-terminal path and on the final queue attempt.alerts_disablednow logs the GitHubhttpStatusand a redacted, boundedbodyExcerpt; per-repo start/finish with duration and remaining budget; budget exhaustion with last/next repo; the full lease lifecycle; and a per-commandsecurity_auto_analysis.command_stalledlog in the auto-analysis dispatcher.Verification
pnpm --filter cloudflare-security-sync test— 93 passed;typecheckclean;lint0/0.pnpm --filter cloudflare-security-sync test:integrationagainst a real Postgres 18 (pnpm test:db) — 9 passed, covering two-connection lease contention, the chunk fence, thecompletedReposunion, guarded write/clear/freshness, release fencing, and completion redelivery. This is the artifact that proves the CAS.pnpm --filter cloudflare-security-auto-analysis test— 194 passed; typecheck/lint clean.Limitations
COMMAND_STALLEDwhen its message is delayed past the running timeout — aligning the reconciliation timeouts (handoff fix-direction App Builder - Add session-based message pagination #3) is a deliberate follow-up.Studio201io/zmid-monorepostays skipped asalerts_disabled; the new status/body log is what will identify the real trigger next time.