DTC: adopt repository-derived course hierarchy and homework units

[alexeygrigorev]

Goal

Adopt the generic course repository hierarchy and YAML-backed homework contract from community-base#306 (C5.4) in DataTalks.Club. The site's code adapts its course/cohort records and policies to the shared graph instead of re-parsing or flattening curriculum locally. Keep the DTC public routes, APIs, supported features and visible UI behavior unchanged.

Development data boundary

Use the D5.3/D5.1/D5.2 boundary in docs/plan/phase-5.md as merged in PR #322 (3449643dc5061676586911efb4a9c0c7d3e7a2c6). Before implementation, identify the exact target and choose fresh or in-place adoption. The owner-authorized fresh exception is limited to an explicitly disposable DTC development logical database; #440's immediate target is dtc_website_dev. It does not authorize production, AISL, Relay or shared RDS resets. Historical IDs and rows from a discarded development schema need not survive; mark their preservation checks Not applicable: authorized fresh DTC development target. In-place adoption retains lossless migration and development-copy rehearsal. In both modes, stable authored identity must preserve rows and learner links created after import across repeated sync and source-directory moves, and every supported behavior remains available.

Milestone ownership

This remains the full adoption acceptance milestone. D5.3a proves the source/policy contract;
D5.1 adopts storage and activates the single shared-parser-backed source writer; D7.3 completes
six-source registration/proof and generic reader retirement; D5.3b integrates site projection. Account for every original #436 criterion with their linked evidence and a
green selected development deploy. Source-contract completion alone does not finish D5.3.
Approved scratch/branch sources can prove development acceptance; live authored conversion and
source-ref cutover remain D7.4, with its existing freeze and human-review requirements.

The original scope and acceptance criteria below remain required. D5.3a owns source/policy proof; #414 owns storage, selected-mode data evidence and activation of the single shared-parser-backed scheduled/webhook writer; D7.3 completes six-source registration/proof and generic reader retirement; D5.3b owns site projection and deployed reader parity. The shared package must be tagged before site changes; no branch/path pin, duplicate parser or generic-template replacement is permitted. At D5.1 deployment, source updates must work against accepted package storage through approved compatible development refs; no legacy jobs may write old/disconnected tables, mirror writes or invoke a second parser. D7.4 alone switches final live authored refs under its original freezes. Existing module-local previous/next policy remains unchanged.

Scope

1. Register the community-base course parser and projection for DTC course sources. Where existing sources express child sections inside manifest lists, convert them to the package's physical directory convention as part of source migration; module.yaml directories define modules and their parentage.
2. Use homework.yaml plus companion homework.md for a structured homework unit where the course tree authors it. Keep stable unit/question IDs in YAML and instruction prose in Markdown. Keep cohort-specific assignment manifests and bindings separate; adapt those to package coursework bindings without confusing them with the course-tree unit manifest.
3. Remove duplicate local module/submodule parsing and hierarchy projection after the package contract covers them. Re-scope open Sub-modules phase B: importer and parser support #398/Sub-modules phase C: module page and cohort-flow rendering #399 to retain only DTC-owned cohort placement/binding behavior. Share generic parsing, projection and behavior helpers, not public templates or visible UI. Public templates, design system, page styling and route placement stay DTC-owned under decision D18; no second generic directory parser or local title/slug/path hierarchy exceptions.
4. Keep DTC-owned course access, cohort flow, registration and URL compatibility through thin adapters. Resolve project-to-module references from source identity/path rather than course slug or module title conditionals. Preserve source order, project references, existing route/API shapes, permission and empty/error behavior, Studio behavior, and desktop/mobile presentation.
5. Fresh mode: import authored source identities into the new target, then prove repeat import and a directory move with unchanged content_id retain imported course/module/unit identity and synthetic homework, project, review, submission, answer, progress and certificate links created after import. Do not require discarded historic development IDs or learner rows to survive. In-place mode: preserve existing course identities and homework, project, review, submission, answer, progress and certificate rows through cutover, with development-copy proof. Both modes retain _docs/compatibility/course-route-contracts.json and existing public routes, rather than changing canonical paths or requiring new redirects.

Acceptance criteria

• A flat course and a nested course fixture import in physical repository order; one-unit modules remain modules and are not flattened by a site rule.
• A YAML homework directory with its Markdown companion imports as one unit; stable unit/question IDs, cohort bindings and learner state created after fresh import survive repeated import and a source-directory move without duplicate rows. In-place mode additionally preserves pre-existing identities and learner rows.
• DTC-owned cohort flow and homework binding semantics work through thin adapters; malformed hierarchy fails validation before a DB constraint error.
• The generic parser/projection and reusable behavior helpers come from community-base. DTC public templates and styling remain site-owned, with no course-specific hierarchy flattening, title/slug/path exceptions or duplicate recursive parser. Rendered public UI and Studio behavior remain consistent with existing contracts.
• Existing public and compatibility route contracts and API shapes pass unchanged, with valid cohort selection, access decisions and project references.
• Package checks and affected DTC tests pass; report package, AISL and DTC results separately. Fresh-mode historical-row preservation is marked not applicable, while repeat-sync/new-identity preservation is proven; in-place mode reports the lossless preservation checks.

Dependencies and related work

• Depends on D5.3a, D5.3b and D7.3. Their prerequisites retain C5.4 (community-base#306), C7.12b (community-base#347), C5.3 and all tagged-package/donor/AISL compatibility gates. Full completion requires all three milestones done in community-base docs/plan/STATUS.md; source-contract success alone is insufficient.
• D5.3a supplies the source/policy prerequisite for D5.1 Map DTC course platform data to the shared apps #414 (D5.1 shared course-model adoption). Full D5.3 blocks D5.2 Freeze weekend: DTC courses cutover and self-paced mode #415 (D5.2 cutover) and D7.4 (live authored-source conversion/cutover).
• Coordinate existing submodule work in Sub-modules phase B: importer and parser support #398/Sub-modules phase C: module page and cohort-flow rendering #399 with the package contract before either duplicates generic parser/projection logic.

Sequencing references

• Source/policy proof: D5.3a / #446.
• Storage and single source-writer activation: D5.1 / #414.
• Reader projection: D5.3b / #447.
• Full preservation acceptance: D5.3 / #436.
• Coordinated course cutover: D5.2 / #415.
• Plan sequencing correction: community-base #348; its phase-5/phase-7 and generated STATUS update must merge before these revised dependencies govern implementation. No milestone is completed by this grooming.
• Package conversion semantics: C7.12b / community-base #347. Package capability, donor compatibility and site adoption remain separate gates.

[alexeygrigorev]

Adoption-readiness review against C5.4

Read-only review used package PR #346 head 6b2522dc036f1b1e1dbe045d0462816278fa06f5 and DTC main 058956428f19aaeaa292c980afac75e92132b841. No source conversion, model change or test was performed by this review.

The smallest candidate owner replacement is current-curriculum parsing in content_sync/course_repository_v2.py: _root_module_dirs (20 lines) and _parse_module (131 lines), using public curriculum.parsers.read_courses, course_collections and parse_course. That is 151 gross candidate lines; required policy/metadata mapping is not yet measured, so no net reduction is claimed.

Keep DTC transport, resource budgets, ingest diagnostics, cohort/archive metadata, homework bindings, assets and transaction/replay protection. ReadResult.documents and document values must preserve site metadata absent from the generic graph, including summaries and lesson code declarations. Do not call CourseParser.upsert as a parser substitute: it writes package models.

Source and conversion gates

DTC schema 2 lists units in module.yaml; shared input uses content.yaml, physical hierarchy and unit front matter. Conversion must preserve existing identities, explicit published slugs, manifest-list order, ignores, README separation, video/code assets, archives and policy metadata. The package default slug strips numeric prefixes, so relying on inferred slugs can change existing public routes.

A separate package converter correction is now tracked as community-base #347 (C7.12b). It addresses mixed-directory refusal and semantic preservation for homework directories/bindings, legacy list order and section metadata. Its capability evidence will not itself establish live DTC conversion or retirement of legacy-format support.

Preserve the visible contract

courses/views/shared_course.py::shared_lesson_view navigates among published lessons within the current module. Generic package projection neighbors traverse the selected tree. A direct substitution changes previous/next behavior. After model adoption, use DTC policy/URL handling to preserve publication filtering, module-local navigation, cohort query context, canonical paths, homework panels and missing-target behavior. Generic package templates/views are not substitutes for the existing pages.

curriculum_flow._shared_flow also has distinct homework-placement semantics and omits unmatched homework. It is not equivalent to generic syllabus traversal.

Sequencing to clarify before implementation

There is a practical milestone conflict: D5.3 currently requires nested import/rendering, but live SharedModule/SharedLesson are flat and CourseTree is bound to package Module/Unit/CohortModule. D5.1 owns that model/caller cutover while depending on D5.3. The current dependency graph has no explicit cycle, but implementation ordering needs a separately named prerequisite capability/source milestone and later model-dependent rendering acceptance. A source-only slice must not be reported as completion of full D5.3.

Keep C5.3 and donor gates intact. D7.3 owns course-reader replacement; D7.4 owns actual source conversion/cutover. The authorized disposable DTC development target may be fresh, but subsequent sync/moves must preserve all newly established learner links. #442 / AWS #58 remains the operational recovery gate, with no further reset authorized by this review.

[alexeygrigorev]

Grooming Complete

Preserved all original scope and six acceptance criteria. #436 remains full aggregate adoption acceptance: #446 proves source/policy, #414 adopts storage and activates the single writer, D7.3 completes source registration/proof and reader retirement, and #447 integrates reader projection. Full completion still gates #415 and final live-source conversion D7.4; approved scratch/branch development evidence does not claim live conversion complete.

Freshness check found no intervening edits to #414/#415/#436. The corrected plan draft was independently reviewed at patch SHA c7646fdfbdd51a07860a4bc9a7b35267c2e0bd0a1f2789ef8d7950affd232acb; baseline/proposed plan checks passed at 180/183 issues with all 180 existing statuses/evidence links preserved. Plan publication is tracked by community-base#348. No implementation, tests, task-status advancement, data operation, package release or site deployment is performed or authorized by this grooming. Open product questions: none within this bounded sequencing correction.