306 Repository-derived curriculum hierarchy and YAML-backed homework units

CHANGELOG.md

@@ -6,6 +6,12 @@
   accepted/draft snapshots. Add a configurable Learning in Public link editor that preserves
   autosave, supports an optional blank state, and falls back to a plain textarea without JavaScript.
 
+## 0.5.13
+
+- #306: derive recursive mixed course hierarchy from repository directories and import structured
+  YAML homework units with Markdown prose companions. Reparenting keeps Unit, Homework, Question,
+  Answer and Submission identities; course-tree scoring answers stay out of learner projections.
+
 ## 0.5.11
 
 - #301: add a six-state learner homework descriptor and a read-only helper for rendering the same

community_base/__init__.py

@@ -1 +1 @@
-__version__ = "0.5.12"
+__version__ = "0.5.13"

community_base/content_sync/FORMAT.md

@@ -158,10 +158,11 @@ provenance set on those two models.
 - A tree is expressed by directories and by nothing else. `parent:` keys do not exist.
 - In a docs collection: a directory is a node and must contain `index.md`; leaves are `NN-slug.md`
   files; maximum depth four below the collection root.
-- In a course: a module is a directory holding `module.yaml`; a submodule is a directory holding
-  `module.yaml` inside a module directory; maximum two module levels
-  (`curriculum.source.validate_module_tree`); a module directory holds either submodule directories
-  or unit files, never both, apart from `README.md` and asset directories.
+- In a course: a module is a directory holding `module.yaml`. Module directories may nest to any
+  depth and may contain direct units beside child module directories. Direct units and child
+  modules share one sibling order; every sibling must have a unique order, from its numeric name
+  prefix or an explicit `sort_order`. Missing or duplicate sibling orders are errors. `README.md`,
+  `homework.md` companions and asset directories are supporting files, not siblings.
 - A wiki collection is flat: one directory of `slug.md` files; subdirectories are an error, apart
   from asset directories.
 - Cohorts are not part of the tree; they are placements (section 3.8, course).
@@ -352,8 +353,9 @@ Layout, with `<course>` the collection path (`.` for a single-course repository,
 <course>/NN-<module>/NN-<unit>.md
 <course>/NN-<module>/images/...                   assets
 <course>/NN-<module>/code/...                     never synced, referenced by unit `code`
-<course>/NN-<module>/NN-<submodule>/module.yaml   optional second level
-<course>/NN-<module>/NN-<submodule>/NN-<unit>.md
+<course>/NN-<module>/NN-<child-module>/module.yaml recursive module directory
+<course>/NN-<module>/NN-homework/homework.yaml    structured homework unit
+<course>/NN-<module>/NN-homework/homework.md      homework page prose, required
 <course>/cohorts/<identifier>/cohort.yaml
 <course>/cohorts/<identifier>/README.md           cohort notes or archive notice, optional
 <course>/cohorts/<identifier>/homework/<module-slug>/homework.yaml
@@ -403,6 +405,12 @@ under `extra` and stay DTC-read. `cohorts`, `current_cohort`, `urls`, `schema_ve
 syllabus. Set it on the first top-level module in a section. It is a presentation label; it does
 not affect module ordering, access or progress.
 
+The directory tree is the module hierarchy at every depth. A module may mix child module
+directories, ordinary Markdown units and structured homework unit directories. Those direct units
+and child modules occupy one ordered sibling sequence. Their numeric directory/file prefixes are
+orders unless `sort_order` is written explicitly; each parent's sibling orders must be present and
+unique.
+
 No `units` list, no `schema_version`, no `bonus`, no `ignore`. The overview is `README.md`.
 
 Unit document `NN-<unit>.md`
@@ -420,6 +428,33 @@ The body is the lesson. A `kind: homework` unit body is the instructions page; t
 assignment is the cohort's `homework.yaml`. `is_homework`, `is_preview`, `access`, `prev_url` and
 `next_url` do not exist.
 
+Structured course-tree homework unit directory `NN-<slug>/homework.yaml` and `homework.md`
+
+The directory is one unit in its parent module's sibling order. `homework.yaml` uses the normal
+unit core keys (`content_id`, `title`, and `slug`; order comes from the numeric directory prefix or
+an explicit `sort_order`) plus these fields:
+
+| Key | Type | Required | Default |
+|---|---|---|---|
+| `due_at` | ISO datetime with offset | yes | none |
+| `form` | mapping of form flags and `learning_in_public_cap` | no | field defaults |
+| `final_fields` | list of `{key, label, type, required}` | no | `[]` |
+| `questions` | ordered question list | yes | none |
+
+Question `type` is `multiple_choice`, `checkboxes`, `free_form` or `free_form_long`. Every
+question has a UUID `content_id`, stable authored `id` slug, `prompt`, and optional `points`
+(default `1`) and `step_label`. Choice questions carry ordered `{id, label}` `options`; free-form
+questions carry an `answer_type` (`any`, `float`, `integer`, `exact_string` or `contains_string`).
+Course-tree homework alone accepts `correct` in the existing scoring format (one-based option
+indices for choice questions). The importer carries it into the existing scoring field and
+learner-facing curriculum projections do not include it. Answer sealing follows when the shared
+keyring is provisioned.
+
+The required `homework.md` companion is the unit's prose body and is stored as the unit's homework
+content; it is not a cohort assignment instruction document. This convention does not change
+cohort manifests below: their answers remain encrypted envelopes and plaintext `correct` remains
+invalid there.
+
 `cohorts/<identifier>/cohort.yaml`. The identifier is the directory name and is not repeated inside
 the file. It follows the slug pattern (`2026`, `self-paced`, `4`).
 

community_base/content_sync/check.py

@@ -132,6 +132,7 @@ def main(argv: list[str] | None = None) -> int:
 
 def _check_dialect(item: ParsedDocument, headings: list[tuple[int, str, str]], diagnostics) -> None:
     title = item.data.get("title")
+    body_source = item.raw.body_path or item.raw.path
     for number, line in _code_free_lines(item.body):
         located = item.body_line + number
         for pattern, rule, message, severity in _DIALECT_RULES:
@@ -140,7 +141,7 @@ def _check_dialect(item: ParsedDocument, headings: list[tuple[int, str, str]], d
                 continue
             diagnostics.append(
                 Diagnostic(
-                    item.raw.path,
+                    body_source,
                     "/body",
                     rule,
                     f"{message}: {match.group(0).strip()[:60]}",
@@ -154,7 +155,7 @@ def _check_dialect(item: ParsedDocument, headings: list[tuple[int, str, str]], d
         for message in _check_embed(line):
             diagnostics.append(
                 Diagnostic(
-                    item.raw.path,
+                    body_source,
                     "/body",
                     "4.1",
                     message,
@@ -166,7 +167,7 @@ def _check_dialect(item: ParsedDocument, headings: list[tuple[int, str, str]], d
         if level == 1 and text.strip().lower() == title.strip().lower():
             diagnostics.append(
                 Diagnostic(
-                    item.raw.path,
+                    body_source,
                     "/body",
                     "4.1",
                     "the body repeats the title as a leading H1; the renderer strips it",

community_base/content_sync/documents.py

@@ -180,6 +180,7 @@ def record(self) -> dict[str, Any]:
             "kind": self.kind,
             "part": self.part.name,
             "source_path": self.raw.path,
+            "body_source_path": self.raw.body_path or self.raw.path,
             "parent": self.raw.parent,
             "path": self.path,
             "slug": self.slug,
@@ -531,9 +532,13 @@ def _node_at(tree: DirNode, path: str) -> DirNode | None:
 
 
 def _read_collection(
-    repository: Repository, collection: Collection, diagnostics: list[Diagnostic]
+    repository: Repository,
+    collection: Collection,
+    diagnostics: list[Diagnostic],
+    *,
+    collection_root: DirNode | None = None,
 ) -> list[ParsedDocument]:
-    node = _node_at(repository.tree, collection.path)
+    node = collection_root or _node_at(repository.tree, collection.path)
     if node is None and (repository.root / collection.path).is_dir():
         # A collection root the repository holds but whose every file `ignore`
         # hides, or which is empty, is an empty collection and not a missing
@@ -588,6 +593,12 @@ def _read_item(
         diagnostics.append(locate(raw.path, problem))
     if problems:
         return None
+    if raw.body_path:
+        body, body_line, body_problems = _load_companion_markdown(repository, raw.body_path)
+        for problem in body_problems:
+            diagnostics.append(locate(raw.body_path, problem))
+        if body_problems:
+            return None
     if isinstance(content, Mapping):
         data = dict(content)
         for problem in check_item_keys(content, part):
@@ -616,7 +627,7 @@ def _read_item(
         sort_order=_item_sort_order(raw, data),
         required_level=0,
         path=slug,
-        is_document=_is_document(raw.path),
+        is_document=_is_document(raw.path) or raw.body_path is not None,
     )
 
 
@@ -846,6 +857,18 @@ def _load_document(repository: Repository, rel: str) -> tuple[Any, str, int, lis
     return data, body, closing + 1, []
 
 
+def _load_companion_markdown(repository: Repository, rel: str) -> tuple[str, int, list[Problem]]:
+    """Read prose-only Markdown paired with a structured manifest item."""
+
+    try:
+        text = repository.read_text(rel)
+    except UnicodeDecodeError:
+        return "", 0, [Problem(WHOLE_FILE, "3.2", "must be UTF-8")]
+    if "\r\n" in text:
+        return "", 0, [Problem(WHOLE_FILE, "3.2", "must use LF line endings")]
+    return text, 0, []
+
+
 def one_line(error: Exception) -> str:
     """An exception message on one line, for a diagnostic."""
 

community_base/content_sync/kinds/base.py

@@ -97,6 +97,11 @@ class RawItem:
     name: str
     parent: str | None = None
     contributes_slug: bool = True
+    # Composite items may keep their machine-readable fields in one file and
+    # their Markdown prose in a sibling companion. The layout still emits one
+    # item, keyed by ``path``; this path is the body source for rendering and
+    # relative references.
+    body_path: str | None = None
 
 
 @dataclass(frozen=True, slots=True)

community_base/content_sync/kinds/course.py

@@ -33,6 +33,21 @@
         "docs_url": KeySpec("url"),
         "faq_url": KeySpec("url"),
         "hashtag": KeySpec("hashtag"),
+        # Kept source-relative so the curriculum parser can resolve the
+        # authored course hierarchy rather than guessing from a title or
+        # site-owned route.
+        "projects": KeySpec(
+            "object_list",
+            item_keys={
+                "slug": KeySpec("slug", required=True),
+                "title": KeySpec("string", required=True, max_length=200),
+                "module_path": KeySpec("string", required=True),
+                "cohort_key": KeySpec("slug"),
+                "submission_due_at": KeySpec("datetime", required=True),
+                "review_due_at": KeySpec("datetime", required=True),
+                "peer_review_count": KeySpec("integer"),
+            },
+        ),
         "testimonials": KeySpec(
             "object_list",
             item_keys={
@@ -52,6 +67,9 @@
     keys={
         "syllabus_section": KeySpec("string", max_length=255),
         "is_bonus": KeySpec("boolean", default=False),
+        # Existing course sources call the same generic module flag `bonus`.
+        # The graph and persistence API use the clearer `is_bonus` name.
+        "bonus": KeySpec("boolean"),
         "available_after_days": KeySpec("integer"),
     },
 )
@@ -71,6 +89,7 @@
         ),
         "session_position": KeySpec("integer"),
         "is_bonus": KeySpec("boolean", default=False),
+        "available_after_days": KeySpec("integer"),
         "code": KeySpec(
             "object_list",
             item_keys={
@@ -81,6 +100,56 @@
     },
 )
 
+HOMEWORK_UNIT = PartSpec(
+    name="homework_unit",
+    shape=SHAPE_MANIFEST,
+    keys={
+        "due_at": KeySpec("datetime", required=True),
+        "is_bonus": KeySpec("boolean", default=False),
+        "available_after_days": KeySpec("integer"),
+        "form": KeySpec(
+            "mapping",
+            item_keys={
+                **{name: KeySpec("boolean") for name in FORM_FLAGS},
+                "learning_in_public_cap": KeySpec("integer"),
+            },
+        ),
+        "final_fields": KeySpec(
+            "object_list",
+            item_keys={
+                "key": KeySpec("slug", required=True),
+                "label": KeySpec("string", required=True),
+                "type": KeySpec("choice", choices=("text", "url", "textarea")),
+                "required": KeySpec("boolean", default=False),
+            },
+        ),
+        "questions": KeySpec(
+            "object_list",
+            required=True,
+            item_keys={
+                "content_id": KeySpec("uuid", required=True),
+                "id": KeySpec("slug", required=True),
+                "type": KeySpec("choice", choices=QUESTION_TYPES, required=True),
+                "prompt": KeySpec("markdown", required=True),
+                "points": KeySpec("integer", default=1),
+                "step_label": KeySpec("string"),
+                "options": KeySpec(
+                    "object_list",
+                    item_keys={
+                        "id": KeySpec("slug", required=True),
+                        "label": KeySpec("string", required=True),
+                    },
+                ),
+                "answer_type": KeySpec("choice", choices=ANSWER_TYPES),
+                # The source course repository still owns historical scalar
+                # answers. This key exists only for course-tree homework units;
+                # cohort homework manifests continue to require envelopes.
+                "correct": KeySpec("string"),
+            },
+        ),
+    },
+)
+
 COHORT = PartSpec(
     name="cohort",
     shape=SHAPE_MANIFEST,
@@ -159,6 +228,7 @@
         "course": COURSE,
         "module": MODULE,
         "unit": UNIT,
+        "homework_unit": HOMEWORK_UNIT,
         "cohort": COHORT,
         "homework": HOMEWORK,
     },