Skip to content

Align development deployment validation with the reviewed Relay runtime contract #442

Description

@alexeygrigorev

Goal

Make the supported DTC development deployment accept and preserve the reviewed Relay runtime configuration from aws-infra #58 while retaining exact target boundaries. Restore development through the ordinary deployment path after the separately owned infrastructure prerequisites are satisfied. Preserve every course feature and UI.

Evidence and current state

  • Exact source audited: 19e7393489b7158c30bae39f93915a1219f2c757.
  • Authorized reset run 36601136596 completed schema migration, then failed relay_schedule_sync with exit 22. Both website services are stopped. No second database reset is part of this issue.
  • The failed migration:53 definition lacks the Relay URL and secret references. aws-infra #58 already has accepted source providing them; live activation was not performed. Do not duplicate its Terraform implementation.
  • deploy/update_task_definition_image.py:98-99 invokes validate_source_workload(). That calls _secrets() in deploy/task_definitions.py:194-218, which currently permits exactly DATABASE_URL and DJANGO_SECRET_KEY. Correct four-secret definitions from Preserve graduates, certificates, and historical Wrapped records #58 therefore fail before registration.
  • The active updater preserves unrelated environment entries and existing secret references. It does not fetch or synthesize missing runtime credentials.
  • deploy/deploy_website.sh:155-177 reads web/worker source definitions from the services' selected task-definition ARNs; only migration uses the latest family revision. Registering new web/worker family revisions alone does not repair those sources.

Read first and normative ownership

  • _docs/PROCESS.md: separate engineer, tester, PM and on-call gates; no pull requests.
  • _docs/specs/08-aws-development-terraform.md: Relay client boundary, secret confidentiality and separate application/infrastructure permissions. Its opening sandbox physical identifiers are historical; do not revive or widen that retired target.
  • _docs/runbooks/release-deployments.md: supported controller, current targets, exact service-source selection and release gates.
  • deploy/deployment_targets.py: single closed target-definition owner. Current target is website-development, URL https://dev.datatalks.club, namespace website-dev, region eu-west-1, account 387546586013. Production remains independently scoped.
  • _docs/ci/change-selective-ci.md: frozen verification plan and evidence classification.
  • Current coding-standard.md, including the reviewed local version in the shared checkout if it has not landed on main. Read that checkout only; do not copy unrelated work.
  • aws-infra main/dev/README.md, Website Relay runtime contract, main/dev/website_service.tf shared task locals and main/dev/iam.tf execution policy: operator activation ownership.

Scope

  • Give the supported active development promotion gate one authoritative, target-scoped secret-reference contract matching aws-infra Preserve graduates, certificates, and historical Wrapped records #58.
  • Require exactly one each of DATABASE_URL, DJANGO_SECRET_KEY, RELAY_API_KEY, and RELAY_WEBHOOK_SECRET for active development source tasks.
  • Keep existing database/Django reference checks. Relay references must be the development integrations secret in the selected region/account/namespace, with the exact corresponding JSON selector:
    • arn:aws:secretsmanager:eu-west-1:387546586013:secret:website-dev/integrations-<six-character AWS suffix>:RELAY_API_KEY::
    • arn:aws:secretsmanager:eu-west-1:387546586013:secret:website-dev/integrations-<six-character AWS suffix>:RELAY_WEBHOOK_SECRET::
  • Both Relay entries must identify the same integrations secret container. Derive target identifiers through the existing target owner; do not create a second target registry or shell literals.
  • Preserve source RELAY_BASE_URL and both selectors through image promotion for web, worker and migration. Do not guess a private endpoint or rewrite a configured URL. Live URL/configuration correctness remains the infrastructure/runtime prerequisite below.
  • Preserve production's existing exact secret contract and every existing role, family, account, architecture, command, single-container and immutable-image guard.
  • Add a concise recovery handoff to the supported release runbook. No workflow permission expansion is needed.

Non-goals

  • No UI, template, route, feature, notification-policy, package-pin, database-model or schema change.
  • No second reset, migration/Relay bypass, ad hoc secret injection, credential retrieval/logging, Terraform apply, or production mutation under this source issue.
  • No new main/dev apply workflow and no widening of the sandbox Terraform workflow.
  • No broad rewrite or deletion of the retained manual normalizer/release machinery. If a shared helper is extracted, preserve existing callers' distinct contracts; expand changes only for a demonstrated dependency and report it before broadening scope.
  • No progress on frozen answer-check consolidation Investigate shared homework scoring consolidation without behavior changes #439 until development recovery is green.

Dependencies and ownership

Source implementation is ready: aws-infra #58 already supplies the accepted reference schema. Live activation is an external dependency, not a reason to block local implementation or claim recovery from local tests.

  • Engineer: isolated worktree, source/tests/runbook only, frozen uncommitted handoff.
  • Independent tester: verification plan, contract/security boundary and regression evidence.
  • PM: source acceptance after tester pass; independently record outstanding live criteria.
  • Authorized infrastructure operator: existing Preserve graduates, certificates, and historical Wrapped records #58 activation and stopped-service source reconciliation.
  • Orchestrator/on-call: ordinary exact-main deployment after accepted source and operator prerequisites; one CI/deploy observer.

Source acceptance criteria

  • A valid development definition with the four exact references passes the real active module CLI for web, worker and migration; the resulting registration document retains the URL and exact secret entries while updating only the existing owned release fields/command normalization.
  • Missing, duplicate or unexpected secret names; wrong region/account/namespace/container; malformed references; missing, swapped or incorrect JSON selectors; added stage/version selectors; and mismatched Relay container ARNs are refused. No source credential value is printed. Failure occurs before registration of the offending definition, task launch or service promotion; an earlier successfully validated family need not be retrospectively unregistered.
  • Valid production definitions still pass unchanged. Production rejects development Relay references and additional secret names. Existing target, role, family, architecture, container, command and image rejection tests remain effective.
  • The real deploy shell integration fixture models all three corrected development definitions, proves preserved Relay configuration at registration, uses service-selected web/worker revisions, and runs the ordinary migration/schedule/template sequence with reset absent. A rejected source cannot launch migration or promote services. No database reset command is introduced.
  • New tests run in the applicable repository/deployment verification selection. Existing oversized test files are not simply expanded to hide new coverage outside the runner.
  • New handwritten functions are at most 30 lines and files at most 300 lines. Existing oversized modules/functions do not grow; any necessary extraction follows one cohesive validation responsibility and preserves callers. No ternaries or filtered/nested comprehensions are added. Any unavoidable exception is justified explicitly.
  • Engineer and independent tester record exact base/head/worktree state, plan and graph digests, commands/counts and evidence dispositions. Follow the generated verification plan, blocking lint and applicable advisory report; do not narrow a required full gate. PM accepts only after tester pass.
  • The release runbook records the operational handoff below, with source acceptance clearly distinguished from live recovery.

Verification scenarios

  1. Extend coverage at the existing owners: TaskDefinitionImageUpdateTests in core/tests/test_cmp_style_deployment.py provides production and real dev module-CLI examples; ci/tests/test_deploy_release_verification.py provides the real shell/fake-AWS transport harness and service-source selection coverage. Prefer cohesive separate test modules/helpers where size limits require them; update discovery if necessary.
  2. Use inert synthetic ARNs and fake AWS only for local contract tests. At least one authoritative test must fail on the old two-secret validator and pass on the fix. Reuse existing negative guards where they already prove unchanged behavior.
  3. Browser/screenshots: this is deployment control-plane code with no render impact. Product-page screenshot evidence is not_applicable, with that reason in the plan/report. Run the repository-required backend Playwright smoke tier and any additional graph-selected gates; do not invent course visual changes to justify screenshots.
  4. No cloud access, infrastructure mutation or secret-value read is required to complete source acceptance.

Separately owned live recovery prerequisites and evidence

These criteria remain unchecked until their named owner supplies actual evidence. Source acceptance may proceed first; use Refs #442 and leave the issue open while live criteria remain outstanding.

  • [OPERATOR] Complete Preserve graduates, certificates, and historical Wrapped records #58's non-printing secret-shape preflight and review a fresh saved plan from the accepted main/dev revision, using its existing approved operator path. No authorized main/dev apply workflow currently exists; the sandbox workflow must remain scoped to sandbox.
  • [OPERATOR] Reconcile the README restriction limiting activation to task-definition/IAM changes with the necessary web/worker service-pointer updates. The current Terraform resources point those services at its task definitions, but DTC deploy reads their selected revisions. Explicitly review the pointer changes, safe image references and live drift; do not silently bypass the README restriction. Keep both desired counts at zero during preparation. Stop for unrelated state, IAM, networking, database, replacement or service-count changes.
  • [OPERATOR] Provide redacted evidence that all three selected deployment sources have the reviewed Relay URL/selectors and required execution access. Secret content and remote state must not enter issue comments/artifacts. Registering family revisions without fixing stopped service pointers is insufficient.
  • [ON-CALL] After accepted DTC source is merged and exact-SHA CI passes and operator prerequisites are complete, run the ordinary deploy-dev.yml path with confirm_dev_schema_reset EMPTY. Never dispatch another schema reset for this recovery. A push-triggered attempt before operator readiness is not recovery success and must remain failed/blocked.
  • [ON-CALL] Record exact successful run/SHA/image, migration/schedule/template completion, service/runtime health and readiness at https://dev.datatalks.club, then the Preserve graduates, certificates, and historical Wrapped records #58-required deployed sync_relay_schedules --dry-run no-diff and jobs_ingress_selftest OK evidence, without secrets. Failure keeps recovery open; no success record or resumed Investigate shared homework scoring consolidation without behavior changes #439 on partial evidence.

Related: #438, #440, aws-infra #58. #439 remains held.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Must-have or release-blockingbugSomething isn't workinginfraArea: infraoperationsArea: operations

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions