Consolidate homework answer encryption on the shared package

[alexeygrigorev]

Consolidate homework answer encryption on the shared package

Status: pending
Tags: courses, security, enhancement, P1
Depends on: None
Blocks: —

Reporter context

The owner asks to unify and simplify community-base, AISL and DTC while preserving every feature and the current UX, with no visible UI changes. Use focused changes in an isolated DTC worktree because shared checkouts have other owners. The cross-repository inventory identified the 490-line courses/homework_answer_crypto.py as a duplicate of the shared answer-crypto owner already present in DTC's pinned community-base release.

Normative references

• _docs/specs/04-courses-and-cohorts.md, especially preserved learner homework behavior and the rule to characterize scoring behavior before deduplication.
• _docs/specs/07-security-privacy-operations.md for bounded failures and privacy.
• _docs/architecture/app-boundaries.md and AGENTS.md for direct domain ownership and the prohibition on legacy runtime shims.
• _docs/ci/change-selective-ci.md and _docs/specs/10-verification-strategy.md for versioned, change-selective verification.
• community-base v0.5.10 community_base.coursework.answer_crypto for the existing implementation contract.

Scope

Delete the duplicated courses/homework_answer_crypto.py implementation. Retarget all proven live and test imports directly to community_base.coursework.answer_crypto, including courses/homework_answer_checks.py, courses/homework_answer_resolution.py, and their tests. The existing DTC pyproject.toml and uv.lock pin community-base to v0.5.10, whose crypto module has the same implementation plus validate_source_envelope; this issue needs no package release or dependency bump. Before deleting, inventory all repository Python import forms, aliases, test imports, patch targets, and literal/dynamic references, then repeat the check on the final diff. Do not leave a compatibility facade or copy classes/exceptions locally.

Keep the DTC keyring configuration lookup, answer-resolution boundary, source option-ID mapping, free-form answer conversion, answer-check handling, homework models, and persisted envelopes in place. Preserve exact object identity for HomeworkAnswerKeyring, HomeworkAnswerCryptoError and its validation, unavailable-key, and decryption subclasses across every remaining caller. Change the deterministic test patch from the deleted site's secrets.token_bytes path to the shared module's path; keep its known AES-GCM/HKDF envelope vector meaningful. Existing encrypted answers must decrypt with the shared implementation, and newly encrypted answers must remain compatible with the prior implementation.

Characterize and preserve strict keyring parsing and rotation, canonical authenticated context, scalar and choice payloads, envelope validation and size bounds, malformed/tampered ciphertext failures, unknown key IDs, context mismatch, redacted error text, and failure handling in scoring. Do not compare independently randomized ciphertext for equality. If the existing tests leave a specific behavior gap, add one authoritative behavior test before changing the import owner and prove it passes before and after.

Non-goals

• No visible UI, template, route, API, access, scoring-policy, feature, or schema change.
• No model migration, stored-data rewrite, keyring configuration change, or new crypto algorithm/version.
• No package-code change, new abstraction, legacy runtime shim, or unrelated dependency pin update.
• No shared-main edit, commit, merge, push, or pull request in this grooming/implementation-review stage.

Acceptance criteria

• A final-revision caller inventory covers all three Python import forms, aliases, tests, scripts, patch targets, and dynamic references; no use of the deleted site module remains.
• courses/homework_answer_crypto.py is deleted, all callers use the package owner directly, and the report measures actual net application lines removed separately from tests or moved code.
• Class and exception identity is single-owner: DTC resolution/checks and package encrypt/decrypt paths use the same HomeworkAnswerKeyring and crypto exception classes.
• Existing and new scalar/choice envelopes interoperate across old and shared implementations; the deterministic envelope vector, key rotation, malformed/oversized envelope rejection, unknown-key behavior, authenticated-context mismatch, and tamper failures retain their exact contract.
• DTC's configured keyring source, answer-resolution mapping, scoring behavior and safe failure handling stay unchanged; no secret or answer value is added to logs or error output.
• The authoritative crypto and answer-resolution tests pass before and after; the old secrets.token_bytes patch is retargeted to the shared owner without weakening the vector assertion.
• No page, API, template, route, model, migration, stored envelope, feature, or visible UX changes.
• make lint, make lint-advisory, focused tests, and the versioned _docs/ci/change-selective-ci.md verification plan pass. Engineer and Tester report exact base/head, graph and plan digests, every component's rerun/reused/skipped/not_applicable disposition, exact commands/counts, and the graph-selected backend browser tier (smoke when the focused backend profile applies); any full-profile fallback follows the plan.
• Independent Tester verifies the uncommitted isolated worktree and records screenshot evidence as not_applicable if the graph confirms no render impact. Shared main remains untouched; no commit, merge, push, or PR occurs before the owner-authorized handoff.

Browser regression scenario

Scenario: learner submits source-managed homework with an encrypted answer

Given: an enrolled learner sees a homework question whose answer is stored as an encrypted source envelope
When: the learner submits an answer through the existing homework flow
Then: the response and score follow the same existing policy, and the learner sees no plaintext source answer, key detail, or new error message

The cryptographic boundary cases belong in focused unit/service tests. This backend-only ownership change introduces no new page or browser behavior.

Repository and operations scenarios

• A previously stored envelope decrypts with the shared owner using the existing configured keyring and the exact course/homework/question context.
• An envelope with a wrong context, unavailable key, malformed fields, or tampered ciphertext fails in the same bounded class and does not leak its secret input.

---

Blocked by: (none)

[alexeygrigorev]

Grooming Complete

Scope: delete the 490-line DTC crypto duplicate, retarget the two production importers and the crypto/resolution tests directly to the pinned package owner, and update the deterministic secrets.token_bytes patch target. No legacy facade. DTC's v0.5.10 pin already contains the equivalent implementation plus one additional envelope-validation helper, so there is no package-release dependency.

Acceptance protects class/exception identity, keyring configuration, known envelope vector, old/new envelope interoperability, context authentication, key rotation, bounded malformed/unknown-key/tamper failures, answer resolution, and safe scoring behavior. The implementation agent must complete an all-import-form and dynamic-reference inventory on the actual deletion revision; the planning inventory alone is insufficient.

Verification: baseline and post-edit focused crypto/resolution tests; blocking lint and advisory; the versioned change-selective plan with exact digests and dispositions; and the graph-selected backend browser tier. One existing learner browser regression and two service/operations scenarios are specified. No render change means screenshots are not_applicable only if the plan confirms it. No open product question or dependency remains. Keep the isolated worktree uncommitted and shared main untouched for independent Tester and PM gates.

[alexeygrigorev]

Software Engineer Report

Issue: #438, course-crypto-consolidation isolated worktree. Implementation is uncommitted and unpushed. HEAD/base and local plan head: cfc21a90b0b9a856dcfe7c6385a6bb39c8974ed0. The deletion is staged only so git ls-files inventories the proposed tree correctly; four modified caller/test files remain unstaged.

Changed files: deleted courses/homework_answer_crypto.py; switched courses/homework_answer_checks.py, courses/homework_answer_resolution.py, courses/tests/test_homework_answer_crypto.py, and courses/tests/test_homework_answer_resolution.py to community_base.coursework.answer_crypto; retargeted the deterministic randomness patch; added decryption of the known stored AES-GCM/HKDF envelope vector. No package, dependency, configuration, schema, route, template, scoring, or UI change. pyproject.toml and uv.lock still pin community-base v0.5.10. A direct diff of the old and installed package crypto modules found only the package's additional validate_source_envelope helper. git diff HEAD --stat: 5 files, 18 insertions, 508 deletions. Net application deletion: 490 lines; the two production import rewrites are line-neutral. No code was moved.

Final P17 inventory: all repository Python import forms, aliases, literal and dynamic module references, and patch targets were searched. No Python or runtime use of courses.homework_answer_crypto or the deleted filename remains. The only literal old filename in tracked source is the historical ci/adoption_baseline.txt inventory, which also lists the surviving test filename and is not a module caller. courses/tests/test_moved_profile_field_readers.py dynamically lists indexed production files with git ls-files; the first full run found it reading the unstaged deletion and failed 1 of 4,146 tests. Staging only the approved deletion made that inventory accurate; its focused 3 tests passed, and the new full run passed. No test-harness suppression was added.

Pre-edit baseline: uv run --frozen pytest courses/tests/test_homework_answer_crypto.py courses/tests/test_homework_answer_resolution.py -q: 23 passed, 45 subtests passed. After adding the known-envelope decryption assertion against the old owner, the same 23 tests and 45 subtests passed. After direct package imports and deletion, the same 23 tests and 45 subtests passed. Explicit Django runtime identity checks passed for the production callers' keyring, crypto exceptions, decrypt_answer, and parse_keyring against the package objects. The deterministic known ciphertext vector remains asserted; the known envelope decrypts, and scalar/choice, rotation, malformed/oversized, context mismatch, tamper, unknown-key, answer resolution, and safe scoring cases pass.

Versioned verification: VERIFY_ISSUE=438 uv run --frozen python scripts/ci.py verification-plan; VERIFY_ISSUE=438 VERIFY_WORKTREE=course-crypto-consolidation uv run --frozen python scripts/ci.py verification-run; uv run --frozen python scripts/ci.py verification-evidence-check; uv run --frozen python scripts/ci.py verification-report-check. The final plan's graph SHA-256 is 557fbababfd8ff983fa88aa29cdef88833c52667878ee72ff3493ae9c66d0ed1, policy version 5, report plan SHA-256 dcefcc5ac575b1f84921a942c5aa8ff3582d188640456d53ced542cebd908811, plan file SHA-256 9e75ff8dd136964794b70d836cf96fea49927c65e13e00bc13de54e15a4be6c5, and verification state SHA-256 4c9d08f875012dcebce0d5a5bbfa57144e92e73448a508af830d29e9ac677409. Direct node: app.courses. Downstream: django.accounts, django.api, django.content, django.content_sync, django.core, django.courses, django.management_api, django.studio, django.studio_courses. Risk test_infrastructure selects full profile; browser tier is smoke; render impact is false.

Disposition buckets: rerun: selector (1 assertion), evidence validation (670/670 tests), quality contract (670/670 embedded CI tests, lint/format/type/migration/Django checks passed), full Django (4,146 tests; 4,140 passed, 6 skipped, 0 failed), smoke Playwright (30/30, 0 retries), container (13 assertions, 0 failed). reused: none. skipped: none. not_applicable: screenshots, because render inputs are unchanged and the graph lists no required captures. Final engineer report verdict: success. This repository has no make lint or make lint-advisory targets; equivalent maintained commands uv run --frozen python scripts/ci.py lint and format-check passed independently, and the full quality contract passed in the plan.

Artifacts: .tmp/verification/verification-plan.json, .tmp/verification/verification-report.json, .tmp/verification/evidence/, and .tmp/issue-438/ baseline/post/final run logs. Independent Tester and PM gates remain pending; no commit, push, merge, or PR was made.

[alexeygrigorev]

QA Review — PASS

Independent tester gate on the uncommitted course-crypto-consolidation worktree. HEAD/base/head: cfc21a90b0b9a856dcfe7c6385a6bb39c8974ed0. The final status remains one staged deletion (courses/homework_answer_crypto.py) and four unstaged import/test rewrites; no source file was changed by QA. git diff HEAD --stat: five files, 18 insertions, 508 deletions. Net application removal is 490 lines; production import rewrites are line-neutral.

Code and contract review: The deleted 490-line implementation and the installed pinned community-base v0.5.10 module are byte-for-byte identical except for the package's additional validate_source_envelope helper. pyproject.toml and uv.lock retain v0.5.10. The final repository search found no Python, literal/dynamic, alias, or patch-target reference to the deleted module; only historical ci/adoption_baseline.txt names the old path. Django runtime identity checks passed for HomeworkAnswerKeyring, HomeworkAnswerCryptoError, HomeworkAnswerKeyUnavailable, decrypt_answer, and parse_keyring in answer resolution, the crypto error in answer checks, and validation/decryption subclass inheritance. Existing keyring lookup, answer mapping, free-form conversion, scoring catch, models, routes, templates, and API remain untouched. The deterministic ciphertext vector remains an exact assertion and is also decrypted as a prior stored envelope. Tests cover scalar/choice payloads, rotation, strict keyring parsing, malformed/oversized envelopes, unknown key, context mismatch, tamper, error redaction, and scoring behavior. No new logging or secret output was found. The pre-change and characterized-before-import focused logs each show 23 tests and 45 subtests passed; my independent post-change focused run had the same count.

Plan: VERIFY_ISSUE=438 VERIFY_OUTPUT_DIR=.tmp/issue-438/tester-verification VERIFY_CONSUMER=tester VERIFY_BASE_SHA=cfc21a90b0b9a856dcfe7c6385a6bb39c8974ed0 VERIFY_HEAD_SHA=cfc21a90b0b9a856dcfe7c6385a6bb39c8974ed0 uv run --frozen python scripts/ci.py verification-plan. The engineer and tester plans differ only in created_at; their verification-state SHA-256 is the same, 4c9d08f875012dcebce0d5a5bbfa57144e92e73448a508af830d29e9ac677409. Graph SHA-256 557fbababfd8ff983fa88aa29cdef88833c52667878ee72ff3493ae9c66d0ed1, policy 5, tester canonical plan SHA-256 ea561bff074543b8cbc35161ebb7f495120bb1eb44dd21d6e3920a930b0ade4a, tester plan-file SHA-256 0cad585288f6dc0e1a04ec348b26816b9123ee0fa327bec645d033e894322552. Direct node app.courses; nine downstream Django labels. test_infrastructure risk selects full Django and smoke Playwright. Render impact false, no captures; screenshots not_applicable.

Executed independently:

• uv run --frozen pytest courses/tests/test_homework_answer_crypto.py courses/tests/test_homework_answer_resolution.py -q: 23 passed, 45 subtests passed.
• VERIFY_ISSUE=438 VERIFY_PLAN=.tmp/issue-438/tester-verification/verification-plan.json VERIFY_EVIDENCE_DIR=.tmp/issue-438/tester-verification/evidence VERIFY_REPORT=.tmp/issue-438/tester-verification/verification-report.json VERIFY_CONSUMER=tester VERIFY_WORKTREE=course-crypto-consolidation VERIFY_PRODUCER_ROLE=tester VERIFY_PHASE=tester uv run --frozen python scripts/ci.py verification-run: exit 0. Selector: one assertion; evidence validation: 670/670; quality contract: 670/670 with lint, format, type, migration and Django checks; full Django: 4,146 executed, 4,140 passed, six skipped, zero failed; smoke Playwright: 30/30, zero reruns/quarantine; container: 13 assertions, zero failed. The repository has no make lint or make lint-advisory targets; the maintained quality contract ran the equivalent blocking checks.
• VERIFY_PLAN=.tmp/issue-438/tester-verification/verification-plan.json VERIFY_EVIDENCE_DIR=.tmp/issue-438/tester-verification/evidence VERIFY_REPORT=.tmp/issue-438/tester-verification/verification-report.json VERIFY_CONSUMER=tester VERIFY_PHASE=tester uv run --frozen python scripts/ci.py verification-evidence-check and verification-report-check: both exit 0.

Tester report .tmp/issue-438/tester-verification/verification-report.json: verdict success, file SHA-256 f0582d2c5700cb69a88b42e1d895b5f5d5feb504b7963aae648e8bac29c27a47. Artifacts and envelopes are under .tmp/issue-438/tester-verification/evidence/. Four buckets: rerun selector, evidence_validation, quality, django, playwright, container; reused none; skipped none; not_applicable screenshots. Each rerun has local tester origin and an independently validated output/artifact digest. Final source/status check still has the same five-file uncommitted diff and exact staged deletion.

Evidence integrity note: An initial tester runner invocation was stopped during evidence validation because verification-run requires explicit VERIFY_PLAN and VERIFY_EVIDENCE_DIR even when VERIFY_OUTPUT_DIR is set. That setup-aborted attempt wrote to the default engineer evidence directory. It replaced the historical selector files (ci-selection.json, selector-plan.json, selector-environment.json, selector-result.json, selector-evidence.json) and changed evidence_validation-output.log. The saved engineer report binds selector evidence ID c3d408035719f3a182d65a0e70593de26a63a7be8a8cdf4e67f6cd74ef575a4d, while the current default selector ID is 2ea2a386a23b7b50c2f308cc43e8976ae04c148ba65025a8298f45fc0fc249e0. The old validation-log SHA-256 was b2bd503edd4309a427e6647a8a59afcdf8b1f283f7957117f9e3a6c1c83c632a; current is fd619d9e10edab042575d7771117933309ba0278987725e4569af93bd4527f1d. Read-only validation of the saved engineer report now fails with artifact_digest_mismatch. No historical envelope is reused or claimed valid for this QA gate. The separate tester plan/evidence/report above were produced afresh and pass their own validation.

Acceptance: all nine automated criteria verified. The learner submission scenario is covered at the existing scoring boundary by focused tests and the graph-selected smoke browser tier; no render input changed. PM user acceptance remains the next independent gate. No commit, push, merge, or PR was made.

[alexeygrigorev]

Product Acceptance

Verdict: ACCEPT.

The five-file uncommitted diff removes the 490-line DTC crypto duplicate and routes both production callers and both test modules to the existing pinned community-base v0.5.10 implementation. It adds no compatibility shim, changes no dependency pin, and leaves the site's keyring lookup, answer mapping, and scoring catch in place. The deterministic envelope vector is still asserted exactly and is also decrypted as a prior stored envelope. From the learner's perspective, homework submission, scoring, safe failures, and answer privacy retain their existing behavior. No route, API, template, page copy, model, schema, or rendered state changed.

I reviewed the independent QA PASS and its fresh tester evidence at .tmp/issue-438/tester-verification/. The validated report has verdict success, verification-state SHA-256 4c9d08f875012dcebce0d5a5bbfa57144e92e73448a508af830d29e9ac677409, and report-file SHA-256 f0582d2c5700cb69a88b42e1d895b5f5d5feb504b7963aae648e8bac29c27a47. All six required components were rerun; none was reused or skipped. Focused crypto/resolution tests passed 23 tests and 45 subtests; full Django passed 4,140 of 4,146 with six skips; smoke Playwright passed 30; quality and evidence contracts each passed 670 checks; container passed 13 checks. Screenshots are not_applicable because the graph records no render impact.

The issue's make lint/make lint-advisory wording predates this DTC checkout's maintained command interface. This repository has no such targets; the tester ran its blocking versioned quality contract, which includes lint, formatting, typing, migration, and Django checks. The tester also disclosed that an aborted setup run invalidated the earlier engineer artifact. Acceptance relies only on the separate, fresh tester plan and validated report; no historical digest was repaired or reused.

All automated acceptance criteria are satisfied. The staged deletion remains staged because repository checks inspect tracked-file state. The branch remains uncommitted and isolated. No merge or push is authorized by this review.

[alexeygrigorev]

On-call post-push result for merge f143a19b7d5ef7b713c7b8139d01619f333961b6 (implementation 8065533d, Closes #438): development deployment is red. I reopened this issue; it is not complete.

• CI run 36522190258 succeeded for that exact SHA, including ci-gate, quality, container, Django (3,753 tests; 16 skipped and 1 expected failure), and Playwright smoke (30 passed). CI classified screenshots as not applicable because render inputs were unchanged.
• Deploy Dev run 36522190231 passed test, the exact-SHA verify-ci gate, and image publish, then failed in deploy (job 109258739450).
• The one-off migration task exited 21 in phase schema_migration; ECS reported EssentialContainerExited with no container reason category. The deploy controller explicitly reported that services were not changed. It uploaded a redacted recovery receipt artifact.

The CI log identifies the failing phase but does not expose the underlying manage.py migrate exception. An authorized development-operations investigator needs to inspect the bounded migration task diagnostic in the deploy job and its named development migration log stream, determine whether this is a code/migration or environment failure, and route the fix through the issue's engineer, independent tester, and PM gates. Re-run and observe development deployment for the exact fixed SHA before closing. No production promotion was attempted.

[alexeygrigorev]

On-call follow-up on the failed development migration: I attempted read-only retrieval of the exact development migration log stream named by the deploy controller. The configured profile that reaches the target account received AccessDeniedException: its cmp-alert-investigator role has no identity-based permission for logs:GetLogEvents on /ecs/website-dev/migration. No migration log content was returned, so the underlying manage.py migrate exception and narrow fix are still unknown.

Read-only comparison of implementation commit 8065533d against its parent found changes only to the homework crypto module/imports and their tests. It contains no model, migration, schema, dependency pin, or deployment changes. This does not rule out a runtime interaction with the development database. The next diagnostic step is for an authorized development log reader to extract and redact the exception from the stream reported in failed deploy job 109258739450, then hand that evidence to engineering. I did not access credentials or database data, or run a migration manually.