Skip to content

migrate-task-source-ref-identity: the ordering note says "safe either way", but dropping the legacy index before the deploy removes the old code's race backstop #1959

Open

Description

@lilyshen0722

backend/scripts/migrate-task-source-ref-identity.ts (landed in #1876, merged as e6a82522) carries this header note:

Ordering is safe either way: before the deploy it creates an index the deployed code never consults; after the deploy it only shortens the window in which a mismatched-title create is refused.

The "either way" is not right, and the reason is that the sentence reasons about the index the script creates while passing over the one it drops.

The script does, unconditionally when the legacy index is present:

await Task.collection.dropIndex(LEGACY_INDEX);   // podId_1_sourceRef_1_partial
result.droppedLegacy = true;
// ...
await Task.createIndexes();

The pre-#1876 code very much does consult the dropped index:

  • backend/routes/tasksApi.ts:263 — "sourceRef is an idempotency key (a unique partial index backs it)".
  • :340 — the 11000 recovery matches it by literal name: duplicate.message?.includes('podId_1_sourceRef_1_partial').
  • :344 — "The pre-check can lose a race to another request." The findOne-then-create is racy by construction, and the unique index is the backstop that converts the lost race into an idempotent replay.

Measured

Real Mongo (mongodb-memory-server), inserting {podId:'p1', sourceRef:'ask-1'} twice:

index differing title same title
legacy (podId, sourceRef) unique 11000 — two rows impossible 11000
pair (podId, sourceRef, title) unique both rows insert 11000, and the old sourceRefIndexCollision predicate still matches (keyPattern.podId===1 && keyPattern.sourceRef===1 && keyPattern.taskId===undefined)

Consequence

Running the migration before the new code is deployed removes the only constraint preventing two rows per (podId, sourceRef). The still-live old code then resolves findOne({podId, sourceRef}) against two rows and adopts an arbitrary one — the silent-wrong-row behaviour #1876 exists to eliminate, reintroduced unguarded, because in the differing-title case no 11000 is raised at all so the recovery path never runs.

Bounding it honestly: this needs a race — two concurrent creates sharing a sourceRef with differing titles. Sequential creates hit the findOne pre-check and adopt the existing row (the old bug, not new corruption). Same-title races remain safe.

The two orderings therefore have non-comparable worst cases:

  • after the deploy — a loud, named 503 task_source_ref_index_migration_pending on a narrow case for a bounded window. No data written.
  • before the deploy — silent duplicate rows, arbitrarily resolved from then on.

Suggested fix

  1. Correct the header note: say the drop is the load-bearing step, that the deployed code's race recovery depends on the dropped index, and that the migration should run after the deploy.
  2. Same correction wherever the runbook repeats it.
  3. Optionally have the script refuse to drop unless it can see the pair-index-aware code is live, so the ordering cannot be got wrong by hand.

Operationally, -- --dry first and read legacyIndexPresent: if it is already false there is nothing to drop and the ordering is genuinely moot.

Found while gating #1958. Not a defect in #1876's code — only in its ordering guidance, which is what an operator running this by hand will read.

Activity

  1. lilyshen0722 commented on Sep 27, 2026

    @lilyshen0722
    ContributorAuthor

    Verified from source (not a re-run of your Mongo table — this is the code side, which is where the ordering claim turns, and I have now read both files at the right revs).

    Confirmed, and one refinement that sharpens the bounding.

    1. backend/scripts/migrate-task-source-ref-identity.ts drops podId_1_sourceRef_1_partial unconditionally when present, before createIndexes(). The header note reasons only about the index the script creates, so it never sees the drop. Your read is right.

    2. The pre-fix(tasks): key create on (sourceRef, title), so a second ask cannot adopt the first ask's row (TASK-063) #1876 predicate (4e60f240:332-341) has two disjuncts: keyPattern{podId:1, sourceRef:1, taskId:undefined}, or the literal string podId_1_sourceRef_1_partial. After the drop, the literal branch can never fire — but the keyPattern branch matches the PAIR index too for a same-title collision. So the regression is exactly the differing-title case you name, and nothing more: same-title races still become idempotent replays through the pair index. That makes your "same-title races remain safe" line load-bearing on the keyPattern branch rather than incidental, which is worth a sentence in the corrected note — an operator reading "the backstop is gone" might otherwise widen the rollback.

    3. Deployed-code rev for the window: dev is running pre-fix(tasks): key create on (sourceRef, title), so a second ask cannot adopt the first ask's row (TASK-063) #1876 today (the last deploy failed on the rollout gate), so the pre-deploy direction is not hypothetical right now. Agreed on after the deploy, -- --dry first, read legacyIndexPresent.

    Record-keeping: my earlier pod hand-off to lily-shen (msg 74831) said the ordering was "safe before/after deploy" — I reproduced the script note without checking what it dropped. TASK-171 is corrected to AFTER-the-deploy and points here. The note on main is the thing an operator reads, so this issue is the right home for the fix; no separate action needed from me unless you want the refuse-to-drop guard written, which I have offered on the thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions