Skip to content

fix(skills): coordinate live and pending auto-skill slug claims - #12970

Merged
bolichen97 merged 1 commit into
mainfrom
fix/auto-skill-claim-race-10252
Sep 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/auto-skill-claim-race-10252

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation (1. What is the problem?)

Goal: Give the live and pending auto-skill namespaces one shared availability
test under one shared lock, so no claim on either side can strand or silently
drop a generated skill candidate.

Auto-skill slugs live in two places, the live tree at auto/<slug> and the
pending queue at auto/.pending/<slug>, but each side claimed a name while
consulting only its own half, and the sibling walk trusted a name it derived
without re-validating it. Three ways a generated skill candidate went missing:

A publisher could take a queued candidate's promotion destination.
create_auto_skill claimed auto/<slug> after checking only the live
directory. Staging's sibling allocation queues .pending/<slug>-2 when the
natural slug is under review; a later auto-publish (approval off, prose-only)
proposing the literal slug <slug>-2 then claimed auto/<slug>-2 live.
approve_pending_skill refuses a candidate whose live name is occupied
(live_exists), so the queued candidate became unapprovable for as long as
that directory stood, and TTL pruning deleted it unreviewed.

Exhausted sibling allocation reported success. With every suffix -2..-50
taken, stage_skill_candidate logged a warning and returned the natural
auto/<slug> name without persisting anything. The caller reads a truthy
return as staged: it logged the candidate and wrote a staged audit event, and
consolidation advanced its message offset regardless of per-candidate outcome.
The candidate was lost behind a success record.

A suffixed name could fall outside the canonical slug pattern. The pattern
_AUTO_NAME_PATTERN allows at most 64 characters, so appending -2 to a
63-character slug produces a 65-character name. list_pending_skills skips a
pending directory whose name is not canonical, and prune_pending walks that
same list, so such a claim is invisible to the queue, to the dashboard, and to
pruning alike. It can never be reviewed and is never cleaned up.

Why it matters (2. Why this issue matters to the user)

Every one of these paths destroys generated work silently, and the queue is the
only place a user ever sees it. Consolidation advances its message offset per
run, so a dropped candidate is not retried from the same sessions. The skill the
model extracted is simply gone. In the first and third paths the user is never
told anything at all; in the second the audit trail positively asserts the
candidate was staged, so the record disagrees with the disk and a user looking
for the skill has no way to find out why it is absent.

Not a goal

  • Rescuing the refused publish. When a publish and a queued candidate collide the
    publish loses and returns None, which its caller already audits as a
    rejection. Giving it a sibling live name auto/<slug>-N changes what
    publishing means and would put a name in the live tree that no queue entry
    points at. That belongs in its own issue.
  • Closing the two writers that reach auto/ outside this coordination. The
    crystallize builtin stages with raw file tools, and create_skill accepts an
    auto/-prefixed name from the dashboard handlers. Both are recorded in the
    owning spec as known gaps, and each needs a different change from this diff.
  • Raising the sibling ceiling of 50. It is now a real refusal instead of a silent
    success; 50 distinct candidates slugifying alike is an upstream dedupe problem,
    so the useful follow-up is surfacing the rejection rather than raising the
    number.
  • Making _is_pending_slug_safe enforce _AUTO_NAME_PATTERN. It guards
    traversal only, so the approve and dismiss handlers still accept a
    non-canonical slug the queue will not list. Nothing reaches that state through
    the staging path after this change.

What changed (3. How our fix solves it)

The root cause is that two namespaces forming one slug space were claimed
independently. The fix gives them one shared availability test,
_auto_slug_available, which both claim paths consult before allocating, and
one shared lock, _auto_slug_claim_lock, which they hold across the test and
the claim it authorizes.

  • create_auto_skill declines a slug held by a queued NEW candidate, so a
    publish cannot take that candidate's destination. One side of the collision
    has to lose, and the publish is the cheaper loss: it returns None, which the
    caller already audits as a rejection, whereas the queued candidate is
    immutable human-gated work that would disappear with no record at all.
    Consolidation advances its message offset whatever one candidate's outcome, so
    neither side is retried from the same sessions. The choice is which loss
    leaves a trail, and this PR does not claim the refused publish is re-derived.
  • stage_skill_candidate tests every name on its walk, so the queue only
    accepts a claim it can serve: a new candidate never lands on a slug whose live
    counterpart is occupied, and a suffixed name that breaks the pattern is never
    claimed.
  • Exhaustion returns None instead of a name. That is what makes the caller
    take its rejection branch, so a candidate that was not written is not
    recorded as staged.
  • The claim itself is the mkdir(exist_ok=False), not the availability test, on
    both sides. The test picks a name; the mkdir wins or loses, and a directory
    appearing in the window moves a staging walk to the next name and makes a live
    publish refuse rather than overwrite the other writer's content.

The two halves are not symmetric, so _auto_slug_available takes the claim
under test rather than reserving by name alone:

  • live needs auto/<slug> absent and no pending NEW candidate under that
    slug.
  • pending-new needs the pending directory absent and auto/<slug>
    unoccupied, since a queued candidate whose live name is taken is
    unapprovable.
  • pending-update needs only the pending directory absent. An update is queued
    under a name derived from its target (<target-slug>-update) and
    approve_pending_update promotes it over the live target named in its
    metadata, never consulting auto/<candidate-slug>. So a pending update
    reserves nothing in the live tree, and treating it as a reservation would hold
    auto/<target-slug>-update against an unrelated skill that slugifies to that
    name and drop it, since consolidation advances its offset whatever one
    candidate's outcome.

The kind is read from the candidate's metadata and the test fails closed:
metadata that is missing, unreadable, or silent about kind keeps the slug
reserved, so a candidate that may still be promoted to auto/<slug> cannot be
stranded by an unreadable file.

The lock is what closes the cross-namespace window an atomic mkdir cannot:
the two paths allocate in different directories, so each mkdir is safe on its
own while two processes sharing one skills home could each pass the other half's
test and then create their own directory. Staging holds it until .meta.json is
committed, because kind is the field a live publish reads: a publish seeing
the claimed directory before that file lands fails closed and refuses a name an
update candidate never reserves.

An unacquired lock is a REFUSAL, not a licence to claim uncoordinated.
_auto_slug_claim_lock yields whether it acquired, and all three claim paths
return None when it did not, because proceeding unlocked would reopen exactly
the window the lock exists to close. The acquire itself is
platform_compat.file_lock, which owns the platform matrix, the bounded wait and
the single-shot refusal for an on-loop caller; it is entered explicitly so the
acquire is the only failure that means unlocked, since one except OSError around
the whole block would read a failed write as a failed acquire. The ceiling is
overridden down to seconds because a claim's critical section is one directory
test plus a small write, so a holder that has not released by then is stuck. The
two acquire failures both refuse: the lock file cannot be opened, which is a home
that would fail the skill write too, or the lock is not won inside the ceiling. It
never raises, because None is what the callers audit as a rejection while an
exception would abort the consolidation pass, and the next pass reaches the same
code with the lock free. The lock file is a dot-prefixed plain file at the skills
root, which discovery skips, placed outside auto/ so taking it does not create
the auto namespace as a side effect of a refused claim.

restore_auto_skill is the third claimant on the live half and now obeys the
same rule. It moves an archived skill onto auto/<slug>, which strands a queued
candidate exactly as a publish would, and it previously checked only whether its
own destination existed. The invariant is now stated in the owning spec so a
future allocator reads it: the three auto-skill allocators test availability with
_auto_slug_available while holding _auto_slug_claim_lock.

Two writers reach auto/ without that coordination and are documented in the
spec as known gaps rather than covered cases, because closing either one is a
different change from this diff:

  • The crystallize builtin skill stages auto/.pending/<slug> with raw file
    tools, so it can write a pending directory after a live claim already stands.
    Closing it means giving crystallize a host path that calls
    stage_skill_candidate; no such path exists today, since the only in-tree
    caller is consolidation. Adding an agent-facing surface is outside this fix.
  • create_skill accepts an auto/-prefixed name from the dashboard prompt and
    discover handlers, since READONLY_SKILL_KEY_PREFIXES is only
    ("kiro-user/", "kiro-workspace/"). Closing it means refusing or routing those
    names at those two call sites.

Tests (4. What tests we did)

Fourteen new cases in test/test_skill_pending.py, each pinning one leg of the
root cause:

  • test_publish_cannot_occupy_a_queued_candidates_destination walks the
    reported sequence end to end: stage, sibling-queue a distinct candidate at
    -2, have a publisher propose that literal slug, and assert it is refused, no
    live directory appears, and the queued candidate still approves successfully.
  • test_staging_skips_a_sibling_whose_live_name_is_taken: the queue does not
    hand out an unapprovable name; the candidate lands on -3 and approves.
  • test_exhausted_pending_slugs_report_failure_not_success: all 50 names taken,
    staging returns None, and nothing is written under any claim.
  • test_overlong_sibling_slug_is_refused_not_orphaned: a 63-character slug
    whose -2 sibling exceeds the pattern; exactly one pending directory remains
    and it is listable.
  • test_update_candidate_may_share_its_live_targets_slug: the update path is
    not diverted by its live target.
  • test_pending_update_does_not_reserve_a_live_name: a queued update at
    topic-update does not block a distinct skill publishing to
    auto/topic-update, and the update still points at its own target.
  • test_pending_candidate_of_unknown_kind_still_reserves_its_slug: absent,
    malformed, new, and non-object metadata each keep the slug reserved.
  • test_both_claim_paths_hold_the_shared_slug_lock: both paths run their
    availability test at lock depth 1, and the lock file exists afterwards.
  • test_live_claim_refuses_a_directory_that_appears_under_it: a directory
    created in the window makes the live publish refuse, and the other writer's
    bytes survive. The Path.mkdir patch is scoped with
    pytest.MonkeyPatch.context() so nothing the fixtures installed is touched.
  • test_claim_without_the_lock_is_refused_not_attempted: with the acquire
    forced to fail, staging and publishing both return None and write nothing.
  • test_claim_lock_that_cannot_be_opened_refuses_without_raising: an unopenable
    lock file takes the same refusal path rather than raising.
  • test_pending_metadata_is_committed_under_the_claim_lock: at lock release the
    claimed directory and its .meta.json are both on disk, so no publish can read
    a claim whose kind has not landed.
  • test_restore_cannot_occupy_a_queued_candidates_destination: a restore is
    refused while a candidate of that slug is queued, the archive keeps its copy,
    and the candidate still approves.
  • test_claim_is_the_mkdir_not_the_availability_test forces the race by making
    the availability test approve an occupied name, and asserts the walk moves on
    while the candidate under review keeps its bytes.

test_approve_refuses_when_live_exists builds its live skill directly on disk.
Publishing through create_auto_skill over a queued slug is the behaviour this
change refuses; the guard it covers, approval's own live_exists refusal, still
applies to a live skill arriving by any other route, so the assertion is
unchanged.

Run locally with explicit -n0: test/test_skill_pending.py 43 passed,
test/test_skill_versioning.py 39 passed (the update path),
test/test_skill_pending_api.py + test_skill_pending_notify.py +
test_skill_update_flow.py 116 passed, test/test_skill_lifecycle.py 13 passed
(the restore path), and the skill selections in test/test_history.py 23 passed.
Gates: comment-history, black ratchet, sync-io-in-async, feature-map and
changelog-history all pass on the diff scope; flake8, isort and mypy are clean on
both changed Python files.

5. Any other suggestions on the work

The deliberate scope limits are listed under ## Not a goal above; the one
crystallize detail worth repeating is that a direct write could still reach the
non-canonical-slug state _is_pending_slug_safe does not refuse.

One review note needs no change. First Principles reported two ACP clientInfo
hunks as an undeclared rider and suspected a stale diff base, which is correct:
git diff origin/main...HEAD is exactly the two files above, with zero
clientInfo or acp matches. The hunks are main's own merged change appearing
in a two-dot diff against a moved base.

Pattern harvest

Rule candidate: review-prompt

Pattern: one name space with two independent allocators, where each tests only
its own half before claiming. The defect class has two faces, and this fix hit
both. First, a claim keyed by a name rather than by the destination approval
actually writes to: a pending update is stored under <target-slug>-update but
promoted to auto/<target-slug>, so reserving its storage name holds a live
name nothing will ever occupy. Second, an atomic create per namespace read as if
it made the cross-namespace pair atomic, which it cannot.

The review question to add: for every allocation, which path does approval
write to, does every other writer of that exact name run the same test, and do
the test and the claim sit inside one lock. Ask it wherever a store has a
staging area and a live area, which in this repository also covers pending
artifacts and staged monitors, not only skills.

Fixes #10252

@chenmingwei23
chenmingwei23 requested review from a team and pepmach September 23, 2026 01:59
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the shard-7 red really is main's #13005 before merging — Coverage Gate fails closed behind it and this PR cannot clear it itself.

[DESIGN-REVIEWED] 7a2cc4b

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The two declared uncoordinated writers (crystallize's raw-file staging, dashboard create_skill accepting auto/ names) can still recreate defect A — confirm their closure is tracked.

The three defects are provenanced by tests that fail on base; the cross-process premise for the lock is real (gateway, cli.py:781 and cli_server.py:2546 each build a SkillsLoader over one skills home); the lock reuses platform_compat.file_lock rather than inventing one; the two unfixed siblings are counted and declared in the spec with named closures; the spec updates in the same commit per AGENTS.md.

What this change ships

Inventory (9 items) — 9 justified

Intent: stop the auto-skill pipeline silently losing generated candidates when the live tree and pending queue claim the same slug independently — a FIX.

  1. Live auto-publish refuses a slug a queued NEW candidate will be promoted to — justified
  2. Staging's sibling walk skips names whose live counterpart is occupied — justified
  3. Exhausted sibling walk now reports failure instead of a fake staged name — justified
  4. Over-long suffixed sibling names are refused instead of orphaned invisibly — justified
  5. New lock file skills/.auto-slug-claim.lock; all claim paths refuse if unacquired within 5s — justified
  6. restore_auto_skill refuses a slug awaiting review, under the same lock — justified
  7. A lock-refused candidate is retried next pass (detection marker retracted via ClaimRefusal) — justified
  8. Live publish refuses, never overwrites, a directory appearing concurrently — justified
  9. Spec section records the one-slug-space invariant and two known uncoordinated writers — justified

[FIRST-PRINCIPLES-REVIEWED] 7a2cc4b

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7a2cc4b

Verdict parsed from the review's SHA-scoped output markers for commit 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 5.6 flagged blocking issues on 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94; Opus 5 adjudication downgraded every one of them to advisory. Adjudication downgraded all 1 blocking finding(s) to advisory: the remedy each one requires is disproportionate to the harm it prevents. Read them as advice, not as merge conditions.

This comment is updated in place on each push.

Review details

BLOCKING -- src/kiro_crew/history_consolidation.py:1819 -- Marker retraction never schedules the promised retry
self._last_skillgen_marker.pop(key, None)
lock refusal -> consolidation marks all messages consolidated -> idle checks require unconsolidated messages -> candidate is permanently lost
Anchor: a-refusal-is-not-a-commit
Fix: retry _process_auto_skills with the retained result before consolidation completes.
[BLOCK-MERGE-DOWNGRADED] 7a2cc4b
[GPT-REVIEWED] 7a2cc4b

Adjudication (Opus 5) — is blocking on each finding proportionate?

F1 — I opened history_consolidation.py:1819 and the code it calls into.

Conditions confirmed: the slug-claim lock refusal sets refusal.retryable, and the marker is popped at 1819; mark_consolidated then advances the offset to total (1560-1566), leaving zero unconsolidated messages; every consolidation entry point refuses without an unconsolidated tail (consolidate_session at 845, consolidate_now at 899). So the "retry" only occurs on a later pass, which requires a new appended message — accurate as the finding claims.

Recovery path (not none): any subsequent message re-triggers _consolidate → _run_skill_detection, which reads the FULL session window via derive_messages (1678) with the marker now absent (1694), and re-proposes the same candidate. Self-corrects on the next message; only a session that goes permanently silent loses it.

Harm rung — LOW. The lost item is a single speculative auto-skill candidate, an explicitly best-effort, human-review-gated proposal (1514, 1776-1780) whose prompt default is null; no user data is corrupted or lost, nothing irreversible, no governance/credential dimension. The real fix — an in-pass retry loop holding the retained result plus new coordination with the lock helper on the hot consolidation path, existing only to serve this one rare (lock contention + immediate permanent quiet) path — carries permanent maintenance load clearly exceeding a missed speculative candidate that self-heals on the next message.

[ADJUDICATION] 7a2cc4b total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/history_consolidation.py:1819 reason=disproportionate-remedy
[GPT-ADJUDICATED] 7a2cc4b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7a2cc4b064fd5d3bde3a0857d67a52bc9c6fbe94: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for cb7a4945ac53133f5ca5d0e1dbc5a71e25d37671; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cb7a4945ac53133f5ca5d0e1dbc5a71e25d37671: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 50133b8c86920c991b0bec11edcb8db52c581bf3; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 50133b8c86920c991b0bec11edcb8db52c581bf3: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/auto-skill-claim-race-10252 branch from 50133b8 to 987c6d3 Compare September 23, 2026 04:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/auto-skill-claim-race-10252 branch from 987c6d3 to 2e91f85 Compare September 23, 2026 04:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/auto-skill-claim-race-10252 branch from 2e91f85 to 45d72db Compare September 23, 2026 05:22
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 2e91f8532a9f710fc1fb7a451e73befc0e73b75c; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2e91f8532a9f710fc1fb7a451e73befc0e73b75c: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 45d72dbc790fff63beeb9369538c4c6ac7dbf091; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 45d72dbc790fff63beeb9369538c4c6ac7dbf091: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Evidence on the current fenced finding, for whoever decides its disposition. This is not an override, just the code reading behind it.

The finding at src/kiro_crew/skills.py yield False states that on claim-lock contention "the detection marker and consolidation offset advance -> candidate is never retried". Both halves of that premise do not hold in this tree, and the adjudicator reached the same conclusion (FLAG, not uphold).

The detection marker does not prevent a retry. _last_skillgen_marker is an in-memory dict[str, tuple[int, int]] on the consolidator holding (rotation_generation, message_count). The guard is an equality test that returns early only when nothing new has been appended. One further appended message changes the message count, so the pair differs and detection re-runs over the same content. Being in-memory, a gateway restart clears it outright and forces a fresh pass.

Skill detection never consumed the consolidation offset. The window is the tail of the whole session (all_messages[-_SKILL_DETECTION_WINDOW:]), recomputed each pass. The method's own docstring says it runs as its own LLM call precisely so the consolidation prompt stays tail-scoped, because widening that prompt would re-summarize already-consolidated messages into duplicate history and semantic entries.

That last point also prices the prescribed fix. "Prevent offset advancement" would hold back the offset that history, semantic and lesson extraction share, which reintroduces the duplicate-entry problem the decoupling exists to avoid.

On the residual itself. Reaching the drop needs two processes contending on one skills home AND the holder exceeding the 5 second ceiling over a critical section that is directory exists() tests plus three small local writes. The outcome is one review-gated candidate not staged, logged at WARNING and SEL-audited as rejected / creation_failed; the transcript is untouched.

On the anchor. The yield False this finding names is the refusal an earlier round of the same review required, in the same terms: make the claim paths return None unless the lock was acquired, because proceeding unlocked reopens the cross-namespace race that strands a queued candidate. Reverting it would restore that race, so this PR keeps the refusal.

Separately, Backend Tests shard 7 on both platforms is inherited from the base, not from this diff: test_crew_log_exc_info_sites.py flags new exc_info sites in crew_log/emit.py and crew_log/session_tree_projection.py, neither of which this PR touches and neither of which exists at its base. Tracked as #13005. Coverage Gate fails closed behind it (backend-test=failure).

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Round disposition for head 7e104db936bc9b8a6f79482267a13c028eacc15a (previous head 45d72dbc790fff63beeb9369538c4c6ac7dbf091).

This head is the branch rebased onto current main (75b52f117e), because main landed #13760 after the previous head and that PR touches the same three call sites this change threads a parameter through. The rebase resolutions are unions: #13760's publication guard is kept, and refusal is threaded beside it. Details under "Rebase" below.

The fenced finding F1 is closed in code. The prior disposition argued its premise; this one does not, because the mechanism the finding names now has a fix in the tree.

Which half of the prescribed fix I implemented

BLOCKING -- src/kiro_crew/skills.py:4888 -- Lock contention permanently drops generated skills
yield False
Fix: propagate a retryable result that prevents marker and offset advancement.

The marker half is implemented. The offset half is deliberately not, on the adjudication's own pricing of it: the finding's "prevent offset advancement" half "would hold back the shared consolidation offset that history/semantic/lesson extraction advance -- re-consolidating the same tail into duplicate entries, which is exactly why skill detection was decoupled from that offset." Implementing it would trade this residual for duplicate history, semantic and lesson entries.

What is in the tree is the remedy the adjudication itself described: "a retryable-outcome contract threaded from stage_skill_candidate through _process_auto_skills into marker accounting."

The mechanism was real, and the code's own documentation is the evidence

_auto_slug_claim_lock (src/kiro_crew/skills.py:4868) promises in its docstring, at 4883, that after a refused claim "the next consolidation pass reaches the same code with the lock free". The timeout constant's comment at 340 repeats it: "refusing early and letting the next consolidation pass retry beats waiting on a dead holder."

Nothing delivered that promise. _run_skill_detection wrote _last_skillgen_marker[key] = marker (src/kiro_crew/history_consolidation.py:1578) before staging ran, and the equality guard at 1488 skips any pass whose (rotation_generation, message_count) pair is unchanged. A session that goes quiet immediately after a stall therefore keeps the marker, and the documented retry never arrives.

This is narrower than the finding's "permanently drops": one further appended message still re-runs detection, as the prior disposition showed. But the recovery the code documents genuinely did not exist, and that is worth closing rather than arguing.

The fix

ClaimRefusal (skills.py:345) carries one new field, retryable (365), default False. Both claim paths the detection pass reaches set it for the lock refusal only, never for a name collision or an existing skill: create_auto_skill (4399) and stage_skill_candidate (5073). _run_skill_detection retracts its own marker when it sees the flag (history_consolidation.py:1606), so the next pass re-judges the same unchanged session. The shared consolidation offset is untouched.

It is an out-parameter rather than a raise or a new return type, for two reasons this tree forces. _auto_slug_claim_lock documents that it never raises, so raising would break its contract. And the existing marker tests patch _process_auto_skills with a Mock, whose truthy return value would otherwise fake a refusal on every pass.

Claim-lock sites, enumerated

Three call sites take the lock. Two are reachable from _process_auto_skills and both set the flag: create_auto_skill:4396 and stage_skill_candidate:5070. The third, restore_auto_skill:4753, has no caller anywhere in src/, takes no refusal, and is not on the detection path; it returns None to its own caller, so a marker retraction has no meaning there.

Pins

Two tests, both failing before the fix and passing after.

Test Before After
test_history.py::TestSkillDetectionFullWindow::test_claim_lock_stall_reopens_detection_on_the_next_pass 1 failed pass
test_skill_pending.py::test_claim_lock_refusal_is_reported_as_retryable 1 error (ImportError) pass

The history pin's pre-fix failure is the assertion it was written for, not an incidental error:

E  AssertionError: a claim-lock refusal must not leave a marker that suppresses the retry the lock helper documents
E  assert 'dashboard:chat-claim-stall' not in {'dashboard:chat-claim-stall': (0, 6)}

The skills pin errors before the fix because it must construct the new class: ImportError: cannot import name 'ClaimRefusal' from 'kiro_crew.skills'. That is the honest shape for a test of a new symbol. The "before" runs restored only the two source files, from copies taken outside the repo and verified by sha256 in both directions.

Scoped local run:

python -m pytest -q -n 0 \
  test/test_history.py::TestSkillDetectionFullWindow \
  test/test_skill_update_flow.py \
  test/test_skill_pending.py

78 passed, exit 0. Two pre-existing test doubles of _process_auto_skills needed the new parameter (fake_process in test_history.py, the fake loader in test_skill_update_flow.py); the whole test tree was swept and there are no others.

Backend Tests shard 7 and Coverage Gate belonged to the base

Backend Tests (3.12, 7) and Backend Tests (Windows) (7) failed on the previous head with one test, byte-identical on both platforms:

FAILED test/test_crew_log_exc_info_sites.py::test_the_exc_info_sites_left_in_crew_log_are_exactly_the_vetted_ones

That test is a static AST walk over src/kiro_crew/crew_log/*.py, which this PR does not touch. Measured with zero PR content: on main's own tip at the moment that CI run started (1080c09af6), 1 failed with the same assertion text; on current main (75b52f117e), 1 passed. Same command, same interpreter, both worktrees clean, so the pass is the base fix landing rather than a collection miss.

It was a main-red incident, filed as #13005 sixteen minutes before that CI run and closed as completed by #13007. Coverage Gate failed closed behind it; its own annotations are backend-test=failure -- failing closed. and name no test of their own.

Rebase

The previous head did not merge with current main. #13760 (cda1606fd1) added _skill_publication_guard to _process_auto_skills, _stage_skill_update, and the two staging call sites inside them -- the same functions and the same calls this change threads refusal through. Eight conflict hunks, seven in history_consolidation.py and one in test_history.py, every one a union rather than a choice: the guard and its commit_state are kept exactly as #13760 wrote them, and refusal is added as a further keyword-only parameter.

One test-side fix was needed beyond the merge. The new pin's double did not accept #13760's guard_publication keyword and raised TypeError; it now takes **_kwargs, the same shape #13760 gave its own double in the same file, so the double pins the refusal contract without freezing the rest of the signature.

Verified on the rebased tree: the scoped suite plus the shard-7 test give 81 passed, and #13760's own tests in test/test_transcript_derivation_seam.py give 35 passed. flake8 and isort are clean on all four touched source files.

A second instance of the same invariant, deliberately left for a follow-up

#13760's guard yields None on TranscriptBusy or TranscriptWithheld, and the caller then returns without staging while the detection marker still stands. That is the same shape as this finding: a drop caused by a transient condition leaving a marker that suppresses the retry.

It is not closed here, and not because it was missed. TranscriptBusy is transient and should be retryable; TranscriptWithheld means the transcript became restricted, which is not transient and must not be retried indefinitely. The guard collapses both into one None, so the caller cannot tell them apart, and giving them different answers means changing the contract #13760 introduced. That belongs in its own change against that feature rather than in a rebase resolution.

@chenmingwei23
chenmingwei23 force-pushed the fix/auto-skill-claim-race-10252 branch from 45d72db to 7e104db Compare September 27, 2026 07:46
@chenmingwei23
chenmingwei23 requested a review from a team as a code owner September 27, 2026 07:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
The live tree and the pending queue are two halves of one auto-skill slug
space, but each side claimed a name while consulting only its own half, and
the sibling walk trusted a derived name it never re-validated. Three ways a
candidate went missing:

A publisher could take a queued candidate's promotion destination.
create_auto_skill claimed auto/<slug> checking only the live directory, so an
auto-publish proposing a slug that staging had already queued as a sibling
took that name. approve_pending_skill then refused the candidate with
live_exists for as long as the live directory stood, and TTL pruning deleted
it unreviewed.

Exhausted sibling allocation reported success. With every suffix -2..-50
taken, stage_skill_candidate returned the natural auto/<slug> name without
persisting anything. The caller read the truthy return as staged, logged it
and wrote a staged audit event, and consolidation advanced its message
offset, so the candidate was lost behind a success record.

A suffixed name could fall outside the canonical slug pattern. Appending -N
to a long slug can exceed the pattern's length, and list_pending_skills skips
a non-canonical directory while prune_pending walks that same list, so such a
claim was invisible to the queue, the dashboard and pruning alike.

_auto_slug_available is the one availability test both claim paths consult:
it checks the pending queue, the live tree, and the slug pattern before any
allocation. create_auto_skill declines a slug that is awaiting review, since
a publish slug is re-derivable while a queued candidate is immutable,
human-gated work.

The reservation is scoped to the claim under test. A pending UPDATE candidate
is queued under a name derived from its target and is promoted to the live
target named in its metadata, so its own queue slug is never a live
destination; reserving by name alone would hold auto/<target-slug>-update
against an unrelated skill that slugifies to that name and drop it, because
consolidation advances its offset whatever one candidate's outcome. So
_auto_slug_available takes the claim kind (live, pending-new, pending-update)
and the live test reads the queued candidate's kind. That test fails closed:
metadata that is missing, unreadable, or silent about kind keeps the slug
reserved.

The two claim paths allocate in different directories, so an atomic mkdir
makes each half safe on its own but cannot make the cross-namespace pair
safe. Both paths test and claim inside one advisory lock at the skills root,
and the live claim uses mkdir(exist_ok=False) so a platform that cannot lock
still refuses a directory another writer owns. Staging holds that lock until
.meta.json is committed, because kind is the field a live publish reads: a
publish seeing the claimed directory before that file lands fails closed and
refuses a name an UPDATE candidate never reserves.

Every acquire failure proceeds unlocked instead of raising, because both
callers contract to return None so the caller audits a rejection, and an
escaping error would convert a recorded refusal into the silent loss this
coordination exists to stop.

A stalled claim lock also reopens skill detection, not just its log. The lock
refuses rather than proceeding unlocked, and both its docstring and the
timeout constant's comment promise that the next consolidation pass retries;
nothing delivered that promise. Skill detection records a
(rotation_generation, message_count) marker BEFORE staging runs and skips a
pass whose pair is unchanged, so a candidate refused by the lock was not
re-judged until a further message changed the count or a gateway restart
cleared the marker. The two refusals were also indistinguishable at the
caller: a bare None covers an invalid slug, an oversized procedure and an
exhausted sibling walk, which are properties of the candidate and final, as
well as an unacquired lock, which is a property of the moment.

ClaimRefusal carries that one bit. A claim path sets retryable only for the
lock, and the detection pass retracts its own marker when it sees one, so the
next pass re-judges the same unchanged session and reaches the claim with the
lock free. It is a mutable out-parameter rather than a raise or a new return
type on purpose: the lock helper documents that it never raises because an
escaping error would abort a consolidation pass mid-way, and every caller
that does not care keeps reading str | None.

The shared consolidation offset is left advanced deliberately. History,
semantic and lesson extraction all consume it, so rewinding it to re-attempt
one skill candidate would re-summarize an already-consolidated tail into
duplicate entries, which is why skill detection was decoupled from that
offset to begin with. The marker is the only state the retry rewinds.

Fixes #10252
@chenmingwei23
chenmingwei23 force-pushed the fix/auto-skill-claim-race-10252 branch from 7e104db to 7a2cc4b Compare September 27, 2026 14:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on behalf of the maintainers (sweep 2026-09-27): readiness passed, required checks green on 7a2cc4b, no blocking review threads, security posture unchanged. Approving and merging.

@bolichen97
bolichen97 merged commit e7d6d90 into main Sep 27, 2026
103 of 105 checks passed
@bolichen97
bolichen97 deleted the fix/auto-skill-claim-race-10252 branch September 27, 2026 16:50
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

auto-skills: live/pending slug claims are uncoordinated (lost candidates on collision + false-success on exhaustion)

2 participants