Skip to content

fix(security-sync): serialize owner syncs and guard run progress - #7023

Merged
eshurakov merged 1 commit into
mainfrom
eshurakov/earnest-timber
Sep 30, 2026
Merged

eshurakov merged 1 commit into
mainfrom
eshurakov/earnest-timber

Conversation

@eshurakov

Copy link
Copy Markdown
Contributor

Summary

Fixes the Security Agent sync stall documented in HANDOFF.md: one customer's last_synced_at was frozen at 2026-09-12 and manual syncs failed with COMMAND_STALLED.

Root cause. agent_configs.runtime_state.sync_run was a single per-owner progress slot. It was read with a runId guard but written and cleared without one, so the 6-hourly scheduled run and manual runs for the same owner overwrote each other. Every continuation then found a mismatched runId, restarted at repo 0, exhausted the 8-minute budget, and re-enqueued forever. The resulting message storm saturated the queue, so a manual command's message was delivered after the accepted-timeout reconciler had already failed it.

Changes

  • Single-flight owner lease. runtime_state.sync_lease = { runId, chunkIndex, expiresAt }. A foreign run is denied while the lease is live and enqueues nothing; a strictly older chunk of the holding run is denied as staleChunk and acked.
  • Guarded progress. writeSyncRunProgress, clearSyncRunProgress, and advanceOwnerSyncFreshness are now conditional UPDATE ... RETURNING compare-and-swaps fenced by runId and chunkIndex. completedRepos is unioned so a late write cannot shrink the cursor; a swallowed freshness failure no longer reports success.
  • Completion marker (last_completed_run_id) so a redelivery of a finished run is recognized and not resurrected.
  • No-progress cap so a repo that keeps exhausting the budget and throwing cannot renew a chain forever.
  • Consumer dispositions in security-sync: SYNC_ALREADY_RUNNING, SYNC_SUPERSEDED, SYNC_NO_PROGRESS (failed, not no_op); the lease is released on the already-terminal path and on the final queue attempt.
  • Diagnostics (the second half of the ask): alerts_disabled now logs the GitHub httpStatus and a redacted, bounded bodyExcerpt; per-repo start/finish with duration and remaining budget; budget exhaustion with last/next repo; the full lease lifecycle; and a per-command security_auto_analysis.command_stalled log in the auto-analysis dispatcher.

Verification

  • pnpm --filter cloudflare-security-sync test — 93 passed; typecheck clean; lint 0/0.
  • pnpm --filter cloudflare-security-sync test:integration against a real Postgres 18 (pnpm test:db) — 9 passed, covering two-connection lease contention, the chunk fence, the completedRepos union, guarded write/clear/freshness, release fencing, and completion redelivery. This is the artifact that proves the CAS.
  • pnpm --filter cloudflare-security-auto-analysis test — 194 passed; typecheck/lint clean.

Limitations

  • The running Worker/queue path was not exercised end to end (no local queue binding or GitHub token); the closest boundary was the real-database CAS suite.
  • Residuals: a continuation delayed more than the 7-day lease TTL is taken over once (a restart, not a loop); a canceled invocation can hold the lease up to the TTL; a manual command can still COMMAND_STALLED when its message is delayed past the running timeout — aligning the reconciliation timeouts (handoff fix-direction App Builder - Add session-based message pagination #3) is a deliberate follow-up.
  • Studio201io/zmid-monorepo stays skipped as alerts_disabled; the new status/body log is what will identify the real trigger next time.

Concurrent scheduled and manual runs for one owner overwrote the single runtime_state.sync_run progress slot, so every continuation found a mismatched runId, restarted at repo 0, exhausted the budget, and re-enqueued forever. last_synced_at never advanced and the queue saturated, so manual syncs were delivered after the reconciler had already failed them.

Add a single-flight owner lease (runtime_state.sync_lease) and fence every sync_run write, clear, and freshness update by runId and chunkIndex. A completed-run marker stops a finished run being resurrected, a no-progress cap stops a run that never advances, and the queue consumer now acks denials/refusals without enqueueing a successor.

Add diagnostics: alerts_disabled status and a redacted body excerpt, per-repo timing and budget, the lease lifecycle, and a per-command stalled-command log in the auto-analysis dispatcher.
Comment thread services/security-sync/src/sync.ts
Comment thread services/security-auto-analysis/src/dispatcher.ts
@kilo-code-bot

kilo-code-bot Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

The owner-sync lease/CAS design is sound and well covered by the new unit, fake-DB, and real-Postgres integration suites; the two findings are non-blocking scale/availability observations, not correctness defects.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 2
Issue Details (click to expand)

SUGGESTION

File Line Issue
services/security-sync/src/sync.ts 168 7-day lease TTL: a crashed/canceled holder blocks an owner's scheduled and manual syncs for up to a week (acknowledged residual).
services/security-auto-analysis/src/dispatcher.ts 28 Per-command command_stalled logging is uncapped while reconcileStaleSecurityAgentCommands returns all rows, so the diagnostic scales with the storm.
Files Reviewed (11 files)
  • services/security-auto-analysis/src/dispatcher.test.ts
  • services/security-auto-analysis/src/dispatcher.ts - 1 issue
  • services/security-auto-analysis/src/logger.ts
  • services/security-sync/package.json
  • services/security-sync/src/index.test.ts
  • services/security-sync/src/index.ts
  • services/security-sync/src/sync.integration.test.ts
  • services/security-sync/src/sync.test.ts
  • services/security-sync/src/sync.ts - 1 issue
  • services/security-sync/vitest.config.ts
  • services/security-sync/vitest.integration.config.ts

What was verified: lease claim takeover/fencing (including the expired-foreign-lease and same-run chunk branches), writeSyncRunProgress completedRepos union with runId/chunkIndex CAS guards, the completed-redelivery and no-progress terminal arms, the re-claim vanish race, and consumer dispositions (SYNC_ALREADY_RUNNING/SYNC_SUPERSEDED/SYNC_NO_PROGRESS). No X to X regressions and no memory-leak vectors were found in the changed code; chunkIndex was already required by SecuritySyncMessageSchema, so deployed queue messages remain compatible.

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0

Review guidance: REVIEW.md from base branch main

@eshurakov
eshurakov merged commit 5958220 into main Sep 30, 2026
28 checks passed
@eshurakov
eshurakov deleted the eshurakov/earnest-timber branch September 30, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants