Skip to content

fix(memory): keep 0.7.0 pre-release member tasks and schedules working - #16269

Merged
bolichen97 merged 1 commit into
mainfrom
fix/memory-v2-upgrade-compat
Oct 4, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/memory-v2-upgrade-compat

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An install upgraded from 0.7.0-insider.1 to .5 keeps its tasks and its crew members' schedules working without the operator having to edit any files.

Two kinds of records written by those pre-release builds stop working on today's build:

  • Task registry. For a member task, runs.json holds a {"task_id": ..., "private_payload": true} row. Today's reader refuses the whole registry over that row. TaskRunner restores no runs and fences every snapshot write, and this repeats on every restart, so the operator has no way out.
  • Member schedules. A member schedule holds a member alias and a memory store, but no execution_context. Every fire raises memory_unavailable before dispatch, so the job fails forever and the dashboard's session registry refuses the job's member scope.

Why it matters

Anyone who ran a 0.7.0 pre-release with crew members hits this on upgrade. The task runner stays fenced, so no task state persists across restarts. Member schedules never run again, and nothing tells the operator why or what fixes it.

Not a goal

  • Running a pre-release member task again. Its payload lived in a hidden sidecar this build does not read, so the row is left out, not resumed.
  • Restoring member memory snapshots in the pre-release layout. That is a separate PR, fix(memory): restore member snapshots from the 0.7.0 pre-release layout #16253, and the two share no code.
  • Running a pre-identity schedule that cannot be attributed (deleted member, unowned store, lost declaration). It fails like any other memory_unavailable refusal, and its error names the repair.
  • Changing the default memory store or any global-memory path. Only crew-member records of the pre-release shape are touched.

What changed (motivation → approach → change)

Task registry. The registry is refused only because of rows this build cannot run. workflow_memory.read_task_snapshot now picks out rows of exactly the pre-release shape. TaskRunner._load_runs leaves them out, restores the other runs, and logs one warning naming their task ids. Writes are not fenced, so the next snapshot rewrites the registry without those rows. The hidden payload sidecar is not touched. Any other private_payload row still refuses restore, as before.

Member schedules. A pre-identity schedule can be attributed once the store upgrade has given its store an owner. Attributing it again on every fire would give readers a record they could disagree on. So cron_service/identity.migrate_legacy_member_schedules captures each such schedule once. It runs from the gateway's memory worker (slack/gateway_runtime/memory_lifecycle.py), after repair_legacy_member_stores and before the scheduler arms. The capture is a compare-and-set under cron_store_lock that writes execution_context, the permanent member_id, and the template. A named agent_id is kept and becomes the captured template, as for a schedule created today, since 0.7.0-insider.5 already ran that agent. Before each schedule is captured, the cron:<id> session record the old build left is brought into agreement by a compare-and-set, because dispatch refuses a record that disagrees. If that step fails, the schedule stays uncaptured and the next start retries it. Attribution shares one helper, attribute_legacy_member in src/kiro_crew/execution_context.py, with the existing chat backfill, and nothing is guessed. Dispatch never derives an uncaptured schedule. resolve_cron_memory raises a typed LegacyScheduleRefused, a memory_unavailable error that names the repair and is recorded as an ordinary failed run.

Backwards compatibility

Compatible: every record main accepts is read exactly as before. Only rows and records that main already refuses (the pre-release task row, the uncaptured member schedule) are handled differently.

Reader: src/kiro_crew/taskrunner.py:TaskRunner._load_runs -- release -- restores the remaining rows with writes unfenced; test_legacy_private_reference_is_left_out_and_writes_are_not_fenced
Reader: src/kiro_crew/workflow_memory.py:read_task_snapshot -- release -- any other private_payload row refuses as before; test_any_other_private_payload_row_still_refuses_the_registry
Reader: src/kiro_crew/cron_service/store.py:_job_from_record -- cron -- reads the captured record like any new schedule; test_pre_identity_member_schedules_are_captured_once
Reader: src/kiro_crew/dashboard/handlers/_shared.py:_cron_execution_from_registry -- crew page -- resolves the captured member; test_pre_identity_member_schedules_are_captured_once
Reader: GET /api/crons and cron_list (member_id, agent_id) -- crew page -- report the permanent id and the captured template, as for a schedule created today

Tests

  • test/test_workflows_startup_recovery.py: a pre-release row is left out, the other runs restore, writes are not fenced, the warning names the task id, and a restart with the same row is fine. Any other private_payload shape still refuses the registry.
  • test/test_execution_context.py: attribution by alias or by id. Each refusal names its repair: another member, an unattributed store, a deleted member, a lost declaration, a bad slug, a string version. V1 stores are left alone. The capture runs once, is idempotent, rewrites member_id, keeps a named agent_id as the captured template, reconciles the old cron:<id> session record before capture and retries after a failed write, shares one attribution helper with the chat backfill, is readable by the dashboard registry, and never raises.
  • test/test_cron_gateway_integration.py: a refused pre-identity schedule is an ordinary failed run whose error names the repair. Other refusals before dispatch still raise. An upgraded member schedule naming an agent fires twice through the real callback.
  • test/test_workflows_private_diagnostics.py: passes the now-required legacy_references.
  • test/test_member_memory_upgrade.py: the memory worker captures schedules after the store upgrade and before restores.

Manual verification

N/A: the unit tests build the 0.7.0-insider record shapes directly (checked against the v0.7.0-insider.1 and .5 tags) and drive the real TaskRunner restore, the real capture over a crons.json, and the gateway cron callback.

Related Issues

Follow-up to #12103, which upgraded pre-identity member stores.

Pattern harvest

Not generalizable: these are one-time readers for the record shapes of one pre-release range, and each one is keyed on that exact shape.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): learn-cron-dashboard, taskrunner and memory-skills-hooks specs
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 21:40
@bolichen97
bolichen97 requested a review from CrysisDeu October 2, 2026 21:40
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check before merge: the next snapshot permanently drops the pre-release task rows from runs.json and orphans their hidden sidecar. Make sure that loss is intended.

What this change ships

Inventory (8 items) — 8 justified

Intent: an operator upgrading from 0.7.0-insider.1–.5 keeps their task runner and their crew members' schedules working without editing files. This is a FIX.

  1. A task registry holding pre-release member-task rows now restores the other runs instead of refusing the whole registry — justified
  2. The next snapshot writes the registry without those rows, so their tasks can never resume — justified
  3. A startup warning names the task ids that were left out — justified
  4. At gateway start, before the scheduler arms, each pre-identity member schedule gets its execution captured once — justified
  5. The cron:<id> session record the old build left is rewritten to agree with the capture — justified
  6. A schedule left uncaptured fails each fire with an error that names its repair, not the generic "no canonical execution context" — justified
  7. The chat backfill and the schedule capture now share one attribution helper (attribute_legacy_member) — justified
  8. For upgraded schedules, /api/crons and cron_list report the member's permanent id and keep the named agent_id — justified

[FIRST-PRINCIPLES-REVIEWED] 2632d56

Description read

[DESCRIPTION-READ] 3712a3c97e5cdb07f49db87d5985de45ea838c4e64e8722cb7d6fdbf6982e49d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a CronService write made before the memory barrier (a next_run update, for example) re-reads crons.json first, so it can't overwrite the capture.

[DESIGN-REVIEWED] 2632d56

Description read

[DESCRIPTION-READ] be32d8825e1ab922fb7e8542bf88b74762a123ab1c83ff0a2567149d929ad942

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2632d56

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2632d56

Verdict parsed from the review's SHA-scoped output markers for commit 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2632d564f55bfcfb15a5d4ba3f618ba46fd05fc2: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 3, 2026 20:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026
@bolichen97
bolichen97 force-pushed the fix/memory-v2-upgrade-compat branch from 575b1b0 to e6af73f Compare October 4, 2026 07:10
@bolichen97

bolichen97 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=314f81430d52 legacy quarantine blocks the gateway event loop

fixed in e6af73f: _load_runs only records the legacy rows; TaskRunner.settle_legacy_task_references runs quarantine_legacy_task_references via asyncio.to_thread.
Snapshot writes stay fenced while the rows are pending, on a flag separate from other recovery fences.
The gateway awaits the settlement before _init_dashboard / _init_api_server, so no task mutation can land in the fenced window.

@bolichen97

bolichen97 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c13229b22004 migration discards the selected agent

fixed in e6af73f: migrate_legacy_member_schedules keeps a named agent_id unchanged and captures it as execution.template_id, as bind_cron_memory does.
Test test_pre_identity_member_schedules_are_captured_once is parametrized over no agent and a named agent.

@bolichen97
bolichen97 force-pushed the fix/memory-v2-upgrade-compat branch from e6af73f to 2c47dd0 Compare October 4, 2026 10:55
@bolichen97

Copy link
Copy Markdown
Contributor Author

fixed in 2c47dd0: a squash onto a newer main had reverted test/test_loop_watchdog.py by accident; the file is no longer in the diff.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=254692be9072 Item 6 — undeclared watchdog hunk rides along

fixed in 2c47dd0: the test/test_loop_watchdog.py hunk is removed; main's #16688 version is kept.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=6ba7d5709aea Item 2 — zero consumers: runs.json.legacy-private-tasks.json

fixed in 2c47dd0: the set-aside file and its mechanism are removed (quarantine_legacy_task_references, _read_legacy_task_references, settle_legacy_task_references, the write fence, the gateway await).
TaskRunner._load_runs leaves the pre-release rows out and logs their task ids; writes are not fenced; the hidden payload sidecar is untouched.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=ab196165ee52 The set-aside only reaches one of the two TaskRunner construction sites

fixed in 2c47dd0: the set-aside and its write fence are removed, so no construction site (gateway or cli_server) needs a settle step; _load_runs alone leaves the rows out.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 4, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=8e9902e958a9 named agent_id clashes with the existing cron: session record

fixed in 2c47dd0: reproduced on both legacy record shapes; migrate_legacy_member_schedules now reconciles the cron: record with the capture by compare-and-set (_reconcile_legacy_cron_session), never guessing a member.
Regression: test_an_upgraded_member_schedule_naming_an_agent_fires fires twice through the real cron callback with the named agent.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@bolichen97
bolichen97 force-pushed the fix/memory-v2-upgrade-compat branch from 2c47dd0 to 3d2d71e Compare October 4, 2026 15:23
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=c13229b22004 Keep partial schedule captures recoverable

fixed in 3d2d71e: migrate_legacy_member_schedules now reconciles each schedule's cron: session record BEFORE capturing that schedule, under cron_store_lock.
A reconcile that raises leaves the schedule uncaptured (no execution_context), so the next start retries; a retry after a failed crons.json write accepts the already-reconciled session.
Regressions: test_session_write_failure_leaves_the_schedule_uncaptured_for_retry, test_schedule_write_failure_retries_an_already_reconciled_session.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=0b43cb5dcf92 Item 3 — duplicate of _backfill_legacy_member_record

fixed in 3d2d71e: the five attribution steps live once in execution_context.attribute_legacy_member; both _backfill_legacy_member_record and legacy_member_cron_execution call it.
Each caller keeps its own outcome (backfill returns None, cron raises LegacyScheduleRefused with the same reason and remedy); test_the_schedule_capture_and_the_chat_backfill_share_one_attribution pins it.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=7b4a9ff21415 Item 7 — oversized prevented-run handling

fixed in 3d2d71e: the except LegacyScheduleRefused branch, legacy_refusals_alerted, and the model.py / cron.py doc changes are removed; slack/gateway.py, cron.py and model.py match main.
The refusal is now an ordinary memory_unavailable failed run naming its repair (test_a_refused_legacy_member_schedule_is_an_ordinary_failed_run).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=db7c9f0a4d60 Item 8 — one consumer, generalized

fixed in 3d2d71e: read_task_snapshot's legacy_references is now a required keyword argument and the None branch is gone.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@bolichen97
bolichen97 force-pushed the fix/memory-v2-upgrade-compat branch from 3d2d71e to 805a2d3 Compare October 4, 2026 15:54
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=c13229b22004 Keep transient session read failures retryable

fixed in 805a2d3: _reconcile_legacy_cron_session now raises OSError when the cron: session record is unreadable, so migrate_legacy_member_schedules leaves that schedule uncaptured and the next start retries.
Regression: test_session_read_failure_leaves_the_schedule_uncaptured_for_retry (first migrate captures nothing; second binds the named agent).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@bolichen97
bolichen97 force-pushed the fix/memory-v2-upgrade-compat branch from 805a2d3 to 8db1259 Compare October 4, 2026 17:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026
…es working

Leave pre-release private task references out of the restored registry, and capture pre-identity member schedules once at start, keeping any named agent and reconciling the old cron session record first.
@bolichen97
bolichen97 force-pushed the fix/memory-v2-upgrade-compat branch from 8db1259 to 2632d56 Compare October 4, 2026 21:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked this at head 2632d564f5.

Attribution never guesses a member. attribute_legacy_member (execution_context.py:934-965) requires the store's owner_member_id to be exactly one configured member, the record to name that member's alias or id, and that member to still resolve to the same store. Each failed step raises its own LegacyMemberAttributionRefused subclass, so a pre-identity schedule is never routed into another member's private store or into Global. resolve_cron_memory (cron_service/identity.py:121-125) refuses any legacy member schedule that has not been captured yet instead of deriving one at dispatch, so the dispatch path fails closed too. The session write-back CAS checks memory_store, agent, persistent mode and app before touching a cron:<id> record.

Idempotent and lock-consistent. Already-captured records are skipped, and a retry after a session write that landed but a crons.json write that did not falls into the current == execution branch. The migration runs under cron_store_lock, the same .crons.lock CronService uses, before the scheduler arms.

Mutation checks. Removing each of these reddens a named test: the named not in (alias, owner_member_id) check (test_legacy_member_schedule_naming_another_member_is_refused and 4 others), the store-binding check (...[rebound-LegacyOwnerBoundElsewhere]), the skip-if-captured check (test_pre_identity_member_schedules_are_captured_once), the uncaptured-dispatch refusal (test_an_uncaptured_legacy_member_schedule_never_dispatches), each of the four CAS conditions (test_a_session_the_capture_cannot_vouch_for_is_left_as_it_is[another-store|another-member|app-owned|restricted]), and the exact legacy task-row shape (test_any_other_private_payload_row_still_refuses_the_registry[row3]). The four related test files pass (268) at head; the full suite is left to CI, which is green.

Non-blocking follow-ups

  1. The title says member tasks keep working, but legacy private_payload task rows are dropped from recovery and have to be recreated (taskrunner.md says so). Worth rewording the title/commit so main's history matches the behaviour.
  2. In migrate_legacy_member_schedules, only legacy_member_cron_execution and the reconcile call have per-record try. An unexpected exception elsewhere for one record makes the outer except skip capture for the whole batch. It is fail-closed (dispatch refuses and asks for a restart), but wrapping each record would stop one bad entry from holding back the rest.
  3. The migration runs before apply_pending_member_restores, so a member restored by a pending restore gets its schedules captured only on the second start. Fail-closed; worth one line in the spec.

@bolichen97
bolichen97 merged commit 3d3a752 into main Oct 4, 2026
99 checks passed
@bolichen97
bolichen97 deleted the fix/memory-v2-upgrade-compat branch October 4, 2026 22:09
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants