Skip to content

Prepare shared curriculum source-order import compatibility #445

Description

@alexeygrigorev

Goal

Prepare the DTC import boundary for the two reviewed nullable source-order fields in community-base C5.4, preserving the current visible course experience, source positions, stable records, replay metadata and fail-closed field coverage. Keep the existing tagged package dependency.

Read first

  • _docs/specs/04-courses-and-cohorts.md: preservation of learner workflows, shared curriculum and stable identities.
  • _docs/specs/09-migration-rollout-roadmap.md: atomic, repeatable imports and retained learner data.
  • _docs/architecture/course-platform-shared-apps-mapping.md: SharedModule/SharedLesson field decisions and explicit package defaults.
  • _docs/runbooks/production-data-migration.md, Step 9 (P6): mapped-family equality, refusal before writes and controlled target access.
  • _docs/PROCESS.md, _docs/ci/change-selective-ci.md, coding-standard.md.
  • Accepted compatibility precedent #443, including its actual-model evidence and final scope; preserve accepted security remediation Remediate the locked PyJWT advisory blocking required quality checks #444.
  • community-base #306, remaining implementation contract, and consumer sequencing clarification.

Current evidence and precise owner

Reviewed DTC commit: cdcd1ca87f0de0a7d19ac8da8f9b744503d1ad73.

scripts/prod/import_shared_course_platform.py owns the compatibility decision:

  • _mapping() maps courses.SharedModule to cb_curriculum.Module and courses.SharedLesson to cb_curriculum.Unit (currently lines 529–606).
  • Both families copy site position into target sort_order; cohort placement positions have their separate existing owner.
  • _refuse_mapping_drift() inspects actual concrete fields and rejects unclassified source or target fields before any family write (currently lines 262–297).
  • Existing module/unit import dictionaries omit target-only metadata (currently lines 1252–1323). Do not alter those dictionaries or create an ordinal from site positions.

The package contract proposes nullable internal source_sibling_position on Module and Unit, separate from public/Studio sort_order and cohort placements. These concrete fields/migration are not present in the inspected package default-branch model snapshot at grooming. This issue is anticipated compatibility work, not a claim of a current production failure or future-model test success.

Artifact prerequisite and non-circular delivery sequence

Grooming is complete, but implementation is on hold until the package owner supplies an exact reviewable #306 candidate artifact containing:

  1. Module and Unit model definitions for nullable source_sibling_position, an additive migration, and explicit fresh-row None defaults.
  2. The real ordering/projection contract showing that null source positions preserve the prior sort-order/tie behavior for non-authored imported rows.
  3. An exact package commit SHA, migration name and worktree/artifact reference suitable for a package-owned disposable P16 consumer test. A mutable branch name or design comment alone is insufficient.

The package candidate need not be merged or released. Do not make package #306 merge a dependency of DTC #445: its final consumer gate itself needs this DTC compatibility correction. The sequence is concrete candidate → bounded DTC implementation → pinned DTC gates plus package-owned actual-candidate proof → independent tester and PM → focused DTC commit/integration → package's final P16 against the resulting DTC default branch → package merge/release through its own process.

If the supplied candidate changes field names, null defaults or legacy ordering behavior, stop and amend this contract before implementing. Never broaden the default allowlist or overwrite metadata to conceal that difference.

There is no source-level dependency on development recovery #442 / AWS #58. Those remain separate operational gates; this issue authorizes no workaround, additional reset or deployed import. Site process still governs integration and post-push observation.

Field decisions

DTC source family Package target field Fresh row Replay
SharedModule Module.source_sibling_position None; no source ordinal exists Preserve the existing target value, null or non-null
SharedLesson Unit.source_sibling_position None; no source ordinal exists Preserve the existing target value, null or non-null

DTC's site position continues to copy exactly to sort_order; these values are not reinterpreted as authored source ordinals. Slugs, parent/module relations, cohort placements, copied Markdown/HTML and learner state keep their current import semantics. A site source position edit on replay updates the existing copied sort_order as before while preserving target-only source metadata.

An explicitly named default absent from the older pinned package is inert under the existing guard. Every other unclassified actual concrete source/target field remains an error. Do not add version checks or a general unknown-field exemption.

Scope

  1. Add only source_sibling_position to the existing package-default sets for the two named mappings. Leave import writes, identity matching, transaction boundaries, permission/target checks and drift detection unchanged.
  2. Extend the existing bounded scripts/tests/test_import_shared_course_platform_defaults.py owner with cohesive curriculum guard/default/replay tests. Reuse the existing synthetic family graph and helpers; retain Declare authored-homework defaults in the shared-course import mapping #443 homework tests. This module is already in PRODUCTION_IMPORT_PYTHON and strict mypy, so no new registry, pyproject, seal or lock change is needed for this scope. If a separate test owner becomes necessary, return to PM for a justified bounded amendment before editing registrations.
  3. Add the two decisions to _docs/architecture/course-platform-shared-apps-mapping.md, recording absent-field compatibility, null creation values, unchanged copied positions/ordering, and replay preservation.
  4. Work in a new isolated DTC worktree at an explicitly recorded committed main revision. Shared dirty main and other agents' worktrees remain read-only. Announce exact ownership before source edits.

Coding standard: new functions at most 30 lines and the existing bounded test module at most 300 lines; no prohibited control flow. The importer and _mapping() are already oversized (2,178 and 690 lines at the reviewed baseline). A narrow cohesion exception permits only the formatter-required declaration-line growth caused by adding the two exact set members (at most nine lines across the existing importer/function). Record measured before/after file and function line counts and demonstrate unchanged executable write/guard logic in both engineer and tester reports. No executable control flow or write function may grow. Do not compress unrelated code to offset formatting, split the large importer arbitrarily, or introduce an unrelated refactor.

Non-goals

No package pin/dependency/lock change, model/migration in DTC, UI/template/view/route/API change, package source-order implementation, mixed-source adoption, new schema inference, source conversion, project-reference mapping, general compatibility shim, guard weakening, deployed import, database reset, production-data access or manual deployment. There is no analogous AISL field-default issue in this scope.

Acceptance criteria

  • AC1: The exact concrete package candidate/model/migration and null-ordering contract are recorded before implementation, matching the artifact prerequisite above. No candidate-merge prerequisite is introduced.
  • AC2: Only the two named Module/Unit target defaults are added; existing write dictionaries, identity matching, transaction and target/access logic are unchanged. No ordinal is synthesized from a DTC position, title or slug.
  • AC3: Against the unchanged tagged dependency, anticipated absent fields are inert and the existing dry-run/apply/count/replay/refusal suites pass. Dependency declarations, lock bytes and dependency seals remain unchanged from the recorded baseline.
  • AC4: In the package-owned disposable test using actual migrated candidate models, fresh imported Modules and Units have source_sibling_position=None. Nontrivial site positions remain exact target sort_order values; legacy order/tie behavior, copied slugs/content/HTML/relationships and mapped-family equality remain correct.
  • AC5: In that same real-model context, replay preserves Module, Unit, CohortModule and UnitProgress primary keys and relationships; deliberately set non-null source ordinals on both Module and Unit survive replay. A site position change still updates only its historically copied sort_order while target metadata and learner progress remain intact. Family counts stay equal with no duplicate rows.
  • AC6: An unrelated concrete target field on each of Module and Unit raises named MappingCoverageDrift before the first family write. Existing unknown-source-field refusal and Declare authored-homework defaults in the shared-course import mapping #443's reviewed homework defaults/retention continue to pass. No blanket allowlist or guard change is made.
  • AC7: Documentation explains both exact defaults and their source gaps, old-tag compatibility, unchanged public position/order and replay semantics. No UI, route, access, schema, pin or deployed-data change exists.
  • AC8: The existing registered test owner remains within coding-standard limits and passes unchanged adoption-coverage/lint/format/strict-type gates. Importer growth is limited to the documented declaration-only exception; all other source owners remain untouched.
  • AC9: Final engineer and independent tester reports bind exact DTC base/head/patch, package candidate SHA/migration, relevant input hashes, plan/graph digests and exhaustive component dispositions. Candidate-only tests skipped on the old tag demonstrably execute and pass on real candidate models. No required failed, skipped or pending evidence remains before separate PM acceptance.

Verification scenarios

Pinned site environment

  • Run existing scripts.tests.test_import_shared_course_platform and extended scripts.tests.test_import_shared_course_platform_defaults through the maintained uv-backed Django test environment.
  • Guard tests may simulate concrete metadata to prove recognized-versus-unknown field handling. Metadata mocks do not satisfy real creation/replay AC4–AC5.
  • Exercise unrelated fields separately on Module and Unit and assert the model/field diagnostic and zero family writes. Retain the existing source-field refusal tests.
  • Run unchanged ci/tests/test_adoption_gate_coverage.py, source/dependency guards and exact byte seals; independently confirm pyproject/lock are unchanged.
  • Generate the final versioned plan from frozen source and follow its selected gates without manual narrowing. Backend browser minimum is smoke; obey any broader generated tier. No render impact means screenshots are explicitly not applicable, not pending.

Actual candidate models under package-owned P16

  • The package owner creates a disposable DTC checkout at the recorded base, overlays the exact frozen DTC patch and links the exact Redirect legacy /podwiki/* URLs to the canonical /wiki/* routes #306 package candidate under community-base D15/P16. Never link the original site worktree or commit branch/path sources.
  • Apply the real candidate migration in an owned synthetic test database. Execute both importer modules; all new candidate-only tests must run, not skip.
  • Include more than one published module/unit with distinctive noncontiguous positions and permitted ties so a default-order assertion cannot pass accidentally on a one-row fixture. Use the candidate's actual legacy/null ordering owner to prove the fallback contract; do not invent a second sorter in the site importer.
  • Cover fresh defaults, unchanged copied values/order/counts, replay with deliberately changed target ordinals, source position updates, stable IDs/placement/progress links, and refusal before writes.
  • Record exact package SHA, DTC base/patch and relevant file hashes, migration, commands/counts, zero candidate-only skips and retained log digests. Restore the disposable tagged dependency afterward. Use synthetic data only.
  • This focused candidate proof enables DTC acceptance. It does not replace the package's later full consumer comparison against DTC default main after integration, nor any real-data adoption/rehearsal gate.

Required handoffs

SWE implements only after the candidate artifact prerequisite is met, then freezes uncommitted source and posts evidence. Separate tester independently verifies final source/plan and real-model proof. Separate PM accepts. Original SWE commits; orchestrator performs isolated no-ff integration and assigns sole site on-call. Package on-call separately owns final P16. No DTC pull request is created.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Important follow-upcoursesArea: coursesenhancementNew feature or requestintegrationArea: integration

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions