Backfill offeringId onto offering-contained documents [CLUE-643] by scytacki · Pull Request #2980 · concord-consortium/collaborative-learning

scytacki · 2026-08-25T22:31:40Z

A repair script only — no product code changes, and nothing here has been run against a database in
apply mode.

The problem

isInClassUnitContainer (src/models/document/document-axes.ts) identifies a document's container by
the absence of offeringId. So an offering-contained document that has no offeringId on its
Firestore metadata reads as belonging to the class's copy of the unit — the wrong container axis.

71,399 documents across authed and demo are in that state. This script recovers the value from
the realtime database, where the document's own metadata node has carried it all along, and writes
it back.

What's here

scripts/metadata-repair/backfill-document-offering-id.ts  the sweep
scripts/lib/document-metadata-lookup.ts                   the shared realtime-database lookup
scripts/find-documents-missing-metadata.ts                switched to the shared lookup

find-documents-missing-metadata.ts is the existing repair for metadata documents missing their
unit, investigation and problem. This PR only switches it to the shared lookup. The one change
in what it does: a numeric offeringId in the realtime database, which it used to pass on, is now
treated as absent.

The sweep sits beside the other CLUE-643 repair scripts, and scripts/metadata-repair/README.md
now covers it as step 5 of the runbook. It uses those scripts' isRtdbAddressable and
resolveDatabaseUrl rather than its own copies.

Documents are scanned by collection group, one type at a time, using the containerType: "offering"
types from document-kinds.ts. Each candidate is classified into exactly one bucket, and only the
resolved bucket is written — update({ offeringId }), batched, with the written count
incremented only after a commit resolves. update rather than set with merge, so a document
deleted between the scan and the commit is not recreated holding only an offeringId. A batch that
fails with NOT_FOUND is retried one document at a time, and the deleted document is counted as
deletedDuringRun. A failed run prints its whole partial result, including written and the
per-type and per-space counts.

"publication" appears in the type list because that is the problem publication's stored value;
ProblemPublication in document-types.ts is the constant's name, not its value. Both "group" and
"axes" appear because the generic axes type is mid-rename, and accepting both is what lets this
sweep and backfill-group-document-axes.ts run in either order.

What the production census found (2026-09-23)

A full dry run of this branch against production scanned 112,528 documents. It wrote nothing, and
every document landed in exactly one bucket.

root	spaces	docs	resolved	alreadySet	noMetadataNode	skippedClassWide
`authed`	5	96,291	69,501	26,586	43	161
`demo`	426	3,637	1,898	1,720	8	11
`dev`	3,829	4,743	588	4,059	92	4
`qa`	2,657	7,857	22	5,658	1,104	1,073
total		112,528	72,009	38,023	1,247	1,249

Buckets not shown were 0 everywhere: nodeWithoutOfferingId, unusableDocument, unknownSpace,
keyNotRtdbSafe and lookupError.

By type:

type	resolved	alreadySet	noMetadataNode	skippedClassWide
`problem`	54,306	34,104	1,212	—
`planning`	3,239	2,447	11	—
`publication`	14,169	713	11	—
`supportPublication`	295	—	13	—
`axes`	—	759	—	1,249
`group`	—	—	—	—

A 99.93% recovery rate across real data (authed and demo): 71,399 documents to write, and 51
that cannot be recovered. Two further recovery sources were considered (walking the offering tree, and
a fuzzy match against the portal API), and neither is worth building for fifty-odd documents. "Report
and leave alone" is the policy, and it is what the script does.

nodeWithoutOfferingId is 0. No realtime-database node was written without an offering; the only
failure mode is the node being absent entirely.

These numbers match the August 20 census for everything that needs a write. The differences are
elsewhere:

The generic axes rename has run. Every generic document is now stored as axes, and none as
group.
qa has fewer spaces and documents, since its data is periodically purged.
keyNotRtdbSafe is 0. The two documents it found in August were deleted by hand. They were 2021
supports whose curriculum-authored keys contain a ..

Safety

Dry run by default, APPLY=1 to write, mode printed at startup. TYPES= limits the scan to a subset
of types and PAGE_SIZE= tunes the query page, so a large environment can be sampled before a full
sweep is committed to. A run that dies prints its whole partial result and attaches it to the error,
so it still says what it found and which writes landed.

Idempotency works differently here than in a create-only script: a second run reclassifies the
documents the first one wrote as alreadySet rather than rewriting them.

Requires a deployed index

The collection-group queries need the single-field COLLECTION_GROUP index on type, declared in
firestore.indexes.json. It was deployed to staging and production on 2026-08-13, so neither
environment needs anything done. A new environment would need it created before this script runs at
all — the first query fails outright without it. Diff against the deployed indexes before deploying:
an environment may carry indexes absent from the file, which a --force deploy would delete.

Sequencing

No app code, no rules change, no client behavior change, so this needs no drain, no deploy window and
no place in the release chain.

It does need to run after the two repairs in #2977. This script finds a document's
realtime-database node through its context_id, so a wrong one makes the document look unrecoverable;
and rows that do not exist yet cannot be scanned at all.

That predicts something worth checking when the repairs run: the 43 authed noMetadataNode
documents should shrink, because some of them are likely the documents whose context_id names the
wrong class. If the count does not move, the two populations are disjoint and that is worth knowing
too.

Against CLUE-604's axes backfill it is order-independent, because this script accepts either side of
that rename.

Testing

50 unit tests across 2 suites, against a mock Firestore and a mock realtime database. Worth a look
in review: the bucket classification table, the pagination boundary, the deleted-during-run retry
(only NOT_FOUND is retried one document at a time; any other failure stops the run),
and the keyNotRtdbSafe bucket — a key containing ., #, $, [, ], / or an ASCII control
character cannot appear in a realtime-database path, and a lookup on one throws rather than returning
empty.

Not in this PR

Running it. APPLY=1 needs a separate go-ahead, after the repairs in #2977 have been applied.

🤖 Generated with Claude Code

…CLUE-643] Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The scan is ordered by document id, so two documents belonging to the same user essentially never land in the same page and a per-user cache would hit approximately never. Getting the clustering that would make caching pay off means ordering by uid, which needs a new composite collection-group index. Also record that the extracted lookup must avoid import.meta so Jest can load it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

] A transport error is not evidence that a document has no offering, so it gets its own census bucket rather than joining the two "not found" ones. The extracted lookup throws and each caller decides what to do. Also pin the classification order: class-wide is decided before alreadySet, so a class-wide document that wrongly carries an offeringId is still reported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ll [CLUE-643]

…-643] Adds the paginated collection-group scan that census-tallies every offering-contained document by outcome, type, and space, with RTDB lookups run concurrently per page. Nothing is written yet; dryRun is honored and written stays 0 until a later task drains the resolved lookups into batched writes. Also adds a Jest moduleNameMapper rule stripping a relative import's trailing `.js` extension so ts-jest's default resolver can find the sibling `.ts` file. scripts/ is an ESM package (`"type": "module"`), so its local imports use the `.js` extension per Node's ESM rules, but ts-jest never transpiles that extension away for value imports, and Jest's resolver treats an explicit extension as literal. This is the first script under test that imports another script module as a value rather than only a type, so the gap was previously unexercised.

] Jest resolves only the extensionless specifier to the sibling .ts file, so the tested script uses that form. tsx resolves either, verified against the module. Reverts the moduleNameMapper entry, which rewrote every relative .js import in the repo to fix a scripts-only resolution problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…d report on crash [CLUE-643] Applies the final code review's fix wave to backfill-document-offering-id.ts: derive the Realtime Database URL from the service account instead of hardcoding it, so the script cannot read one environment's offerings and write them onto another's documents; count `written` only after a batch commits so a failed run cannot over-report what landed; print the census in a `finally` and log per-type progress so a crash still leaves a record of how far the run got; and reject a non-string offeringId in the shared RTDB lookup so a legacy numeric value can't silently fail the app's string-typed Firestore query. Adds tests for the new type check, the default dry-run behavior, and per-bucket accounting in the mixed-bucket test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Corrects three claims that implementation made untrue: per-type queries are not independently resumable, group documents are exempt from the uid derivation rather than from repair, and the order-by uses the "__name__" string so the module needs no runtime firebase-admin import. Adds an "Operating the run" section for the properties that matter to whoever points this at production data: the database URL is derived from the credential rather than named, `written` counts committed writes so a dying run under-reports rather than over-reports, a failed run still prints its census, and there is no resume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…E-643] The two Firebase projects do not share an RTDB host pattern: production is collaborative-learning-ec215.firebaseio.com, staging is collaborative-learning-staging-default-rtdb.firebaseio.com. Deriving the URL from the project id produced a wrong host for staging, so the script now looks it up and fails loudly on an unknown project rather than guessing. DATABASE_URL overrides. Also correct the index claim, which a staging run disproved: the collection group index on documents.type is declared in firestore.indexes.json, but that file was reconciled against production, so each environment needs it actually deployed. The first query fails outright without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nown [CLUE-643] A staging census scanned 209 documents, of which 68 were qa and 46 dev. Those roots are keyed by user id rather than portal, so they matched no space and were labelled "unknown" — while being counted under buckets that say nothing is wrong, because they all short-circuited as alreadySet or skippedClassWide first. A partition document that actually needed repair would have reported as an unrecognized path shape, which reads as an anomaly rather than scratch data. They now get their own bucket, asked before every other question because it is a matter of scope rather than a property of the document. unknownSpace is left meaning what it says. The space label now travels with the classification, so the path is parsed once instead of once per document at each of two sites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…LUE-643] TYPES limits the scan to a subset, for a cheap first look before committing to a full sweep; PAGE_SIZE tunes the query page. Both are parsed and validated before anything connects, and an unrecognized type name throws rather than scanning nothing — a typo would otherwise produce a clean empty census that reads exactly like "this type has no problems". The run header now echoes types, page size and mode, so what a run did is recoverable from its log. Also correct the index claim again: the collection group index on documents.type is deployed in neither staging nor production. It was added to the file alongside backfill-group-document-axes.ts, which needs the same index and has not been run. It is a prerequisite for the sweep in every environment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLUE-643] It was created in staging and production on 2026-08-13, and indexes persist, so CLUE-604 inherits a satisfied prerequisite rather than a step to repeat. Kept as a note because the symptom is otherwise baffling: without it the first query fails outright, identically for both sweep scripts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLUE-643] The qa and dev partitions are no longer skipped. Both stores derive their root from the same getRootId, so the Firestore root id is also the RTDB root id; the only missing piece was the portal segment, which sampling production settles as "localhost" for dev and "qa" for qa. `test` takes an arbitrary portal and still cannot be derived, so it reports as an unknown space rather than being guessed at — production has no test documents. A production census also found curriculum supports keyed like "2.2 Initial Challenge Support 1". A "." is illegal in an RTDB path, so those lookups throw and were counted as lookupError, which reads as transient when the failure is permanent. They now get their own bucket, decided before the lookup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

112,900 documents scanned. Across real data 71,399 need an offeringId and 53 cannot get one — a 99.93% recovery rate from the single source this script uses, which settles the question the spec left open: do not build the offering-tree or fuzzy portal sources. Also records what the per-root split shows and the totals hide: 88% of the unrecoverable residue is qa, whose RTDB side delete-qa-user-data.ts purges while leaving the Firestore metadata behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Scanning all 146,105 documents in the collection group confirmed they were the only two with an RTDB-illegal path segment, and both were deleted on 2026-08-20, so keyNotRtdbSafe now reports zero and the real-data residue is 51 rather than 53. Worth recording that neither carried a `unit`, so isInClassUnitContainer could never have misread them — removing them was housekeeping rather than a fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The backfill was scoped under CLUE-610 on the theory that shipping the axes work without it would break something for users. It would not: the affected documents never carry `concurrent`, so they cannot reach the container guard that reads `offeringId`'s absence, and every group document -- the only population that can reach it -- already has one. It belongs with the other repairs of the Firestore metadata mirror, which run their own sweeps rather than waiting on CLUE-604's release gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

] no-duplicate-imports treats a type-only import of the same module as a second import, and a bracket inside a character class needs no escape. Five lint errors, all of which would have failed the build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ffering-id-backfill

codecov · 2026-08-25T22:36:44Z

Codecov Report

❌ Patch coverage is 86.59218% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.36%. Comparing base (9a951a5) to head (ed506e9).
⚠️ Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
...s/metadata-repair/backfill-document-offering-id.ts	84.41%	24 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (9a951a5) and HEAD (ed506e9). Click for more details.

HEAD has 22 uploads less than BASE

Flag BASE (9a951a5) HEAD (ed506e9)

cypress-regression 15 0

cypress 7 0

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2980       +/-   ##
===========================================
- Coverage   86.82%   72.36%   -14.46%     
===========================================
  Files        1031     1034        +3     
  Lines       59032    59201      +169     
  Branches    15733    15775       +42     
===========================================
- Hits        51253    42841     -8412     
- Misses       7757    16324     +8567     
- Partials       22       36       +14

Flag	Coverage Δ
cypress	`?`
cypress-regression	`?`
cypress-smoke	`40.80% <ø> (ø)`
jest	`60.54% <86.59%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cypress · 2026-08-25T22:39:01Z

collaborative-learning Run #20437

Run Properties: Passed #20437 • ed506e9165: docs: bring three settled questions up to date in the backfill design [CLUE-643]

Project	`collaborative-learning`
Branch Review	`CLUE-643-document-offering-id-backfill`
Run status	`Passed #20437`
Run duration	`03m 26s`
Commit	`ed506e9165: docs: bring three settled questions up to date in the backfill design [CLUE-643]`
Committer	`Scott Cytacki`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
View all changes introduced in this branch ↗︎

Copilot

🟡 Changes recommended

Input deduplication, RTDB key validation, failure reporting, and stale policy documentation need correction before execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a repair workflow to recover missing Firestore offeringId values from Realtime Database metadata.

Changes:

Adds a dry-run-first, paginated backfill script with batched writes.
Extracts and tests the shared metadata lookup.
Documents the repair design, census, and sequencing.

File summaries

File	Description
`scripts/backfill-document-offering-id.ts`	Implements the census and repair.
`scripts/backfill-document-offering-id.test.ts`	Tests classification, pagination, and writes.
`scripts/lib/document-metadata-lookup.ts`	Provides the shared RTDB lookup.
`scripts/lib/document-metadata-lookup.test.ts`	Tests lookup outcomes and failures.
`scripts/find-documents-missing-metadata.ts`	Uses the extracted lookup.
`docs/superpowers/specs/2026-08-13-clue-643-document-offering-id-backfill-design.md`	Documents the repair design and findings.

Review details

Suppressed comments (1)

docs/superpowers/specs/2026-08-13-clue-643-document-offering-id-backfill-design.md:402

This final section contradicts the resolved decision at lines 303–307: the production census has already established that additional recovery sources are not worthwhile and that unresolved documents should be reported and left alone. Update this section so readers do not treat the policy as pending.

## Open question, deliberately deferred

What to do about documents whose `offeringId` cannot be recovered. Resolving it needs the
production dry run's numbers — specifically the split between `noMetadataNode` and
`nodeWithoutOfferingId`, and how the residue distributes across spaces and types. The options are
building the offering-tree source, building the fuzzy portal-API match, accepting a documented
residue, or deciding the affected documents are dead and should be deleted. Choosing now would be
guessing.

Files reviewed: 6/6 changed files
Comments generated: 4
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…UE-643] The realtime database also rejects U+0000-U+001F and U+007F in a path. A key carrying one passed the check, so the lookup threw and the document was counted as a transient lookupError rather than as permanently keyNotRtdbSafe. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

TYPES=problem,problem scanned and counted every problem document twice, and in apply mode could queue the same document again in the second pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

After a failed query or commit, the finally block printed only the totals, and main()'s final JSON never ran. So an interrupted apply did not say how many writes had committed, or where. The full result is now logged on failure and attached to the error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The write policy and the closing section still called the policy for unrecoverable documents an open question, although the production census settled it: report them and leave them alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…-643] Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…t stays deleted [CLUE-643] set with merge writes whether or not the document exists, so a document deleted between the scan and the commit was recreated holding only an offeringId. update refuses a missing document on the server. Because that refusal fails the whole batch, the batch is retried one document at a time, and a deleted document is counted as deletedDuringRun and logged instead of ending the run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ffering-id-backfill

…nd share its helpers [CLUE-643] The backfill is the last step of the metadata repair, so it now lives beside the other repair scripts, and their README is its runbook. It imports isRtdbAddressable and resolveDatabaseUrl from the repair's shared modules instead of keeping its own copies, and its duplicate isRtdbAddressable tests are dropped in favor of the ones beside the shared function. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🔵 Needs a closer look

A production repair affecting roughly 72,000 metadata records warrants final human validation despite its safeguards and tests.

Review effort: Balanced
Findings: 3 · 2

Open (5)

If a Firestore query or batch commit fails, finally runs but main() never reaches the final…
TYPES is described as a subset, but duplicate valid entries are accepted. For example,…
Realtime Database keys also reject ASCII control characters U+0000–U+001F and U+007F. For a segment…
Fix backwards description of offering container detection · New
Correct reversed isInClassUnitContainer guard semantics · New

Resolved since last review (1)

This policy text is stale: the production census above and the PR description have already selected…

The script header and design doc said isInClassUnitContainer identifies the offering container by the absence of offeringId. It is the reverse: a missing offeringId marks the class-unit container, and offeringId is the only positive marker of the offering container. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

dougmartin

Looks good 👍

The dry-run default is safe, and nothing I found in the apply path could write a wrong offeringId, write to the wrong document, or recreate a deleted one. The failing codecov checks don't block this: patch is main(), and project comes from missing Cypress uploads on HEAD.

A few non-blocking notes (take or leave). The README one is worth doing before APPLY=1, since the operator relies on that section:

scripts/metadata-repair/backfill-document-offering-id.ts (line 270): the comment says "Firestore's 400-write limit", but the limit is 500, and the sibling scripts say so. Reword to "Batched at 400, well below Firestore's 500-write limit". Optionally, share kBatchSize from scripts/metadata-repair/lib/ across all three scripts.
scripts/metadata-repair/backfill-document-offering-id.ts (lines 35-37): the import comment explains leaving off "the .js extension that the other scripts here use", but the siblings already import ./lib/... without an extension. Delete the comment.
scripts/metadata-repair/backfill-document-offering-id.ts (line 291) and scripts/metadata-repair/backfill-document-offering-id.test.ts (lines 445-460): deleting the if (!isNotFound(err)) throw err; guard leaves the suite green, so a non-NOT_FOUND failure could be retried one document at a time with nothing catching it. In the partial-result test, assert db.batches.length === 2 after the failure. Also add a case where the per-document retry hits a non-NOT_FOUND error, to cover line 298.
scripts/metadata-repair/backfill-document-offering-id.test.ts (lines 8-20): the type-list test compares kOfferingContainedTypes to a hand-copied literal, so it can't catch drift from the kind registry. Have the test import the constants from src/models/document/document-types.ts. Ideally, also check against the kinds whose containerType is "offering", plus AxesDocument for the rename.
scripts/metadata-repair/backfill-document-offering-id.ts (lines 77-95): getSpaceFromFirestorePath re-derives the authed/demo root mapping that resolveSpace already encodes in scripts/metadata-repair/lib/rtdb-document-index.ts (lines 46-63). Build on resolveSpace and add only the /classes suffix and the qa/dev handling, or give resolveSpace an option that accepts qa/dev.
scripts/metadata-repair/backfill-document-offering-id.test.ts (lines 198-212) and scripts/lib/document-metadata-lookup.test.ts (lines 6-23): makeRtdb is identical in both files. Move it into a shared test helper next to document-metadata-lookup.ts.
scripts/metadata-repair/backfill-document-offering-id.test.ts (lines 10-15, 24): these comments restate the JSDoc on kOfferingContainedTypes and the inline comment at script line 80. Delete them.
scripts/metadata-repair/backfill-document-offering-id.ts (lines 15-21): "deployed to staging and production on 2026-08-13" is a process record that will go stale. Keep only the requirement: the collection-group index on type, plus the --force caution about deleting undeclared indexes.
docs/superpowers/specs/2026-08-13-clue-643-document-offering-id-backfill-design.md (lines 92, 107-111, 135-136, 284-324): several passages are stale or contradicted.
- Line 92 still treats the unrecoverable-document policy as open, though lines 339 and 405 settle it.
- Lines 107-111 predict group documents will read as noMetadataNode/nodeWithoutOfferingId, but the census found them all already set.
- Lines 135-136 say "behavior is unchanged", but the lookup now returns a status and treats "" and non-string values as absent.
- The 2026-08-20 census tables disagree with the PR's 2026-09-23 numbers. Trim them to a two- or three-line summary of what the census decided.
docs/superpowers/specs/2026-08-13-clue-643-document-offering-id-backfill-design.md (lines 3-6, 115-131): the Branch/Jira process lines and the "Neither existing script is the right host" argument are process history. Drop the Branch line, and compress that section to one sentence.
scripts/metadata-repair/README.md (lines 116-119, 127-128): line 128 says "both repairs" print their counts on failure, but the README now covers a third writer that does too. Lines 116-119 promise one line per bucket for each space, but the backfill prints a finished <type> line for each type and puts the per-space counts only in the final or partial-result JSON. Update both lines.
PR description ("What's here") and scripts/find-documents-missing-metadata.ts: the description calls this file "a small census used while designing this". It's the existing repair for missing unit, investigation and problem fields, and this PR only switches it to the shared lookup. Reword that line.

…LUE-643] All three scripts defined their own kBatchSize of 400, and the backfill's comment called 400 Firestore's limit. It is 500; 400 is the margin the other two scripts already described. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Drops the index's deployment date, which records past events rather than the requirement, and a comment about import extensions that no longer described the sibling scripts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…-643] Removing either isNotFound guard left the suite green. The partial-result test now asserts no retry batches follow a hard failure, and a new test covers a per-document retry that fails for another reason. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…CLUE-643] The expected list was a hand-copied literal, so it could not catch a type value drifting from the app. It now uses the constants from document-types.ts. Comments that restated the constant's documentation are gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…veSpace [CLUE-643] getSpaceFromFirestorePath re-derived the authed and demo root mapping that resolveSpace already encodes. It now calls resolveSpace, appends /classes, and keeps only the qa and dev mapping that resolveSpace deliberately refuses. New tests cover qa, dev and test paths directly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…E-643] makeRtdb was defined identically in both test files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…LUE-643] The README said every run prints one line per space and that both repairs attach their counts on failure. The backfill reports per type as it goes and per space only in its final or partial-result JSON, and three scripts now attach their counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…CLUE-643] The design still called the unrecoverable-document policy open, predicted group documents would be unrecoverable, and said the shared lookup changed no behavior. The census settled the first two, and the lookup now returns a status and skips an empty or non-string offeringId. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

scytacki · 2026-09-23T18:08:12Z

@dougmartin Thanks. I addressed these in the commits since your review:

Batch comment: kBatchSize now lives in scripts/metadata-repair/lib/firestore-batch.ts, shared by all three scripts, and its comment says 400 is well below Firestore's 500 limit.
Import comment: deleted.
isNotFound guards: the partial-result test now asserts db.batches.length === 2 after a hard failure. A new test covers a per-document retry that fails with something other than NOT_FOUND. I deleted each guard in turn and confirmed the matching test fails without it.
Type-list test: now built from the constants in src/models/document/document-types.ts. I left out the check against the kind registry's containerType: "offering" entries, because it would tie a script test to the axis-profile machinery.
getSpaceFromFirestorePath: now calls resolveSpace for authed and demo, and keeps only the qa/dev mapping that resolveSpace refuses. New tests cover qa, dev and test paths.
makeRtdb: moved to scripts/lib/document-metadata-lookup-test-helpers.ts.
Restating test comments: deleted.
Index deployment date: removed from the script header. The requirement and the --force caution stay.
Design doc: I fixed the three contradictions: the policy for unrecoverable documents, the prediction about group documents, and "behavior is unchanged". The last now spells out the lookup changes. For find-documents-missing-metadata.ts the one visible change is that a numeric offeringId is now skipped. I left the 2026-08-20 census tables and the process history as they are; the PR description has the current census.
README: step 5's output is now described separately (a finished <type> line per type, per-space counts only in the final or partial-result JSON), and the failure note now covers every script that writes.
PR description: find-documents-missing-metadata.ts is now described as the existing repair that switched to the shared lookup.

scytacki and others added 20 commits August 20, 2026 09:48

docs: design the offeringId backfill for offering-contained documents […

661d3fa

…CLUE-643] Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

refactor: share the RTDB offeringId lookup between scripts [CLUE-643]

0ef55c9

feat: classify offering-contained documents for the offeringId backfi…

92b4413

…ll [CLUE-643]

feat: write recovered offeringIds and add the CLI entry point [CLUE-643]

f22faeb

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into CLUE-643-document-o…

cf24173

…ffering-id-backfill

scytacki mentioned this pull request Aug 25, 2026

Repair Firestore document metadata against the realtime database [CLUE-643] #2977

Merged

github-actions Bot temporarily deployed to development August 25, 2026 22:38 Inactive

scytacki requested a balanced review from Copilot September 7, 2026 18:47

Copilot started reviewing on behalf of scytacki September 7, 2026 18:47 View session

Copilot AI reviewed Sep 7, 2026

View reviewed changes

scytacki and others added 3 commits September 22, 2026 17:24

fix: scan a type named twice in TYPES only once [CLUE-643]

ed5d8c8

TYPES=problem,problem scanned and counted every problem document twice, and in apply mode could queue the same document again in the second pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

scytacki and others added 3 commits September 22, 2026 17:25

docs: drop a reference to an implementation plan from a comment [CLUE…

ebe85c8

…-643] Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions Bot temporarily deployed to development September 22, 2026 21:50 Inactive

scytacki and others added 2 commits September 22, 2026 18:43

Merge remote-tracking branch 'origin/master' into CLUE-643-document-o…

9c1c713

…ffering-id-backfill

scytacki requested a balanced review from Copilot September 22, 2026 23:32

Copilot started reviewing on behalf of scytacki September 22, 2026 23:33 View session

github-actions Bot temporarily deployed to development September 22, 2026 23:36 Inactive

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Comment thread docs/superpowers/specs/2026-08-13-clue-643-document-offering-id-backfill-design.md Outdated

Comment thread scripts/metadata-repair/backfill-document-offering-id.ts Outdated

github-actions Bot temporarily deployed to development September 23, 2026 14:00 Inactive

scytacki requested a review from dougmartin September 23, 2026 14:55

dougmartin approved these changes Sep 23, 2026

View reviewed changes

scytacki and others added 8 commits September 23, 2026 14:02

test: share the RTDB stand-in between the two lookup test suites [CLU…

ed106f2

…E-643] makeRtdb was defined identically in both test files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions Bot temporarily deployed to development September 23, 2026 18:15 Inactive

scytacki merged commit bd9eba6 into master Sep 23, 2026
11 of 13 checks passed

scytacki deleted the CLUE-643-document-offering-id-backfill branch September 23, 2026 18:49

dougmartin mentioned this pull request Sep 23, 2026

Backfill canonical pointers for group documents [CLUE-694] #3010

Merged

Navigation Menu

Backfill offeringId onto offering-contained documents [CLUE-643] - #2980

Backfill offeringId onto offering-contained documents [CLUE-643]#2980
scytacki merged 37 commits into
masterfrom
CLUE-643-document-offering-id-backfill

scytacki commented Aug 25, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

cypress Bot commented Aug 25, 2026 •

edited by atlassian Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

dougmartin left a comment

Uh oh!

scytacki commented Sep 23, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

scytacki commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

What's here

What the production census found (2026-09-23)

Safety

Requires a deployed index

Sequencing

Testing

Not in this PR

Uh oh!

codecov Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cypress Bot commented Aug 25, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

collaborative-learning Run #20437

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

dougmartin left a comment

Choose a reason for hiding this comment

Uh oh!

scytacki commented Sep 23, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

scytacki commented Aug 25, 2026 •

edited

Loading

codecov Bot commented Aug 25, 2026 •

edited

Loading

cypress Bot commented Aug 25, 2026 •

edited by atlassian Bot

Loading