Skip to content

WEB-002B — Restore production after unexpected Netlify preview promotion #617

Closed
Closed
WEB-002B — Restore production after unexpected Netlify preview promotion#617
Labels
area:webWeb application and hostingpriority:P0Launch blocker or urgent security risksize:XSOwner decision or very narrow changetype:reliabilityReliability and recovery

Description

@daliu
daliu
opened on Aug 8, 2026

Outcome

Restore the last independently verified bounded Netlify artifact after an unexpected deploy-preview promotion placed a backend-dependent current-main frontend on runmprc.com without its release marker or verified Firebase release.

Parent: #460
Related feature release: #504 / #507
Prior verified containment: #473

Incident evidence

At the 2026-08-07 audit:

  • Netlify identifies production as deploy 6a6f038b9701b00008066000, a deploy preview for PR OAUTH-001A2K — Bound Strava OAuth token JSON responses before parsing #615 head 165934dee32ecc1daaafb596bab16160ff52e6b0 manually published on 2026-08-05.
  • That head has tree 3683a9ccb982b6a2bb1c67a8e0fd27a86f38bcc9, identical to current main b94622bd42775d84704d5fc05ee87da81f5a76e0.
  • The public bundles expose the Account profile-photo/searchability controls and /admin/member-directory People finder.
  • The release marker path falls through to SPA HTML instead of returning the required JSON provenance record.
  • The required directory callable endpoints return platform 404s; no directory Functions, Rules, or index deployment/readback exists.
  • PR OAUTH-001A2K — Bound Strava OAuth token JSON responses before parsing #615 explicitly recorded the preview as unpublished, and the ordinary exact-main Netlify attempt failed unpublished.

This is a split frontend/backend production state, not a successful feature release.

Exact containment scope

  • Change only the Netlify production pointer by restoring ready deploy 6a6dc9ea588b0c0008036312.
  • Require its exact marker after restoration:
    • control 9ad6837756cdd409d296009fde5082eeeae5c059
    • source 39ab8649df411262c8109a3c81a57bc38f1e168b
    • tree d76b496cdc5e79015bc048cb961758abbe88b9ce
    • 60 files
    • artifact SHA-256 07b10c7d5ff176a1ad893d7549b07042312df35f4a4126e8765824ca94eaefb8
  • Do not rebuild, merge, move a Git ref, alter the inactive release manifest, change Netlify configuration, change DNS, deploy Firebase, touch provider settings, sign in to the app, or read/write production member/payment/event data.
  • Preserve the current published deploy ID, hashes, and timestamps in this issue before restoration so the action remains reversible and auditable.

Invariant and failure cases

  • Production must not expose backend-dependent profile or People-finder UI until the corresponding Rules, indexes, Functions, authorization/privacy decisions, repair path, staging proof, and live readbacks are approved and complete.
  • Stop if the target deploy is not ready and atomically restorable, its recorded source/tree/artifact differs, Netlify presents a build/configuration action instead of a pointer restoration, a newer production action appears, or any public verification differs from the exact WEB-002A — Publish bounded Shop and Events permissions containment #473 artifact.
  • A failed verification does not authorize Firebase, DNS, configuration, data, or broad source changes.

Required verification

  • Netlify identifies deploy 6a6dc9ea588b0c0008036312 as the published production deploy.
  • /.well-known/run-mprc-release.json returns JSON with the exact control/source/tree/file-count/digest above and appropriate no-store/content-type headers.
  • The live asset manifest and JavaScript match the restored deploy and do not contain the directory/profile release UI or callable names.
  • Signed-out /shop, /events, /events/calendar, /account, and /admin/member-directory deep routes load at phone and desktop widths; Shop retains the static pickup catalog and Events/Calendar retain the fixed retry-later containment without raw provider errors.
  • Firebase, Rules, indexes, Functions, DNS, provider configuration, account state, payment state, and production data remain unchanged/unverified.
  • Re-read Netlify after the checks to prove no later attempt replaced the restored deploy.

Documentation impact

The rollback restores the exact production state already documented by #473. Record the unexpected promotion, restoration, exact evidence, approver, and residual release blockers in this issue. Do not reword current-behavior officer guidance unless post-restoration readback shows it differs.

Officer impact: Restore the documented static Shop and contained Events/Calendar website. Remove the prematurely published profile-photo and People-finder frontend until its backend, privacy, authorization, and protected-release gates are complete.

Officer documentation: None before restoration — this restores the exact #473 current behavior already described in OFFICER_START_HERE.md and docs/officers/; update those guides only if the verified result differs.

Deployment evidence: The incident audit and rollback evidence must separately identify Netlify, runmprc.com, Firebase, outside-provider configuration, and production behavior. No green source/CI result may be treated as proof of a live backend.

Activity

  1. daliu
    added
    priority:P0Launch blocker or urgent security risk
    type:reliabilityReliability and recovery
    area:webWeb application and hosting
    size:XSOwner decision or very narrow change
    status:readyDependencies complete and ready to claim
    status:in-progressClaimed and actively being implemented
    and removed
    status:readyDependencies complete and ready to claim
    on Aug 8, 2026
  2. daliu
    self-assigned this
    on Aug 8, 2026

  3. daliu commented on Aug 8, 2026

    @daliu
    daliu
    on Aug 8, 2026
    ContributorAuthor

    CLAIMED by /root at 2026-08-08T06:52:26Z. No source branch exists because this incident owns no repository file or Git ref.

    Exact ownership is only the Netlify production pointer from unexpected published deploy 6a6f038b9701b00008066000 to already-built, ready deploy 6a6dc9ea588b0c0008036312, plus provider/public readback and this issue's evidence. Source, merge state, release manifest, Netlify configuration, DNS, Firebase, Rules, indexes, Functions, outside-provider settings, app sign-in, account/member/payment/event state, and production data are immutable.

    Independent technical reviews are GO:

    • target deploy is ready, production-context, contains no Netlify or edge Functions, and exposes exact WEB-002A — Publish bounded Shop and Events permissions containment #473 control/source/tree/file-count/artifact provenance;
    • its public root, marker, and bundle hashes are independently reproducible;
    • directory/profile UI and callable identifiers are absent;
    • Netlify documents restoration as publishing the existing artifact without rebuilding.

    Current preserved incident evidence:

    • displaced/current deploy: 6a6f038b9701b00008066000, context deploy-preview, PR OAUTH-001A2K — Bound Strava OAuth token JSON responses before parsing #615 head 165934dee32ecc1daaafb596bab16160ff52e6b0, published 2026-08-05T13:44:43.799Z;
    • current/public tree: 3683a9ccb982b6a2bb1c67a8e0fd27a86f38bcc9, identical to exact main b94622bd42775d84704d5fc05ee87da81f5a76e0;
    • current root and false-marker SHA-256: addae38deda751ce2948e9933a14343b9af7824bbc4fa568765b0f0bae9f3597;
    • current bundle: /static/js/main.b796b13a.js, SHA-256 a495df529699be41def092a413079d633822312b444620055f6f0dae4c65478b, with directory feature identifiers present;
    • target root SHA-256: 2978a162d55409fae343e9595871413b7d5887884d5a6158a8b2123537823b94;
    • target marker SHA-256: 60552b724766751acb843acf1b70238f215fbae4c32ee008672e3e3a268cb604;
    • target bundle /static/js/main.2a1afaac.js: 836,108 bytes, SHA-256 0397d8dd2716884fc92e293d6082992edfaeae751bcd058093a7ee953c8a8ff0.

    UNDO / roll-forward: the displaced deploy and bytes remain preserved in Netlify and in the exact current-main tree, but a one-click preview-to-production roll-forward is not claimed. Restoration of the newer frontend must use a fresh protected production release after the Firebase/backend, privacy, authorization, staging, repair, and live-readback gates are complete.

    STOP before production mutation: repository recovery procedure requires platform-owner approval plus one named backup officer. The repository owner explicitly authorized deployment in the current Codex task, but no backup officer approval has been named. The signed-in Netlify detail page for the target also does not currently expose a Publish deploy action; it shows the exact successful production artifact and deploy log only. Do not use rebuild, retry, configuration, DNS, or an undocumented API path as a substitute.

    No production or repository state was changed by this claim. The claim remains active while the named approval/control-path gap is resolved.

  4. daliu commented on Aug 8, 2026

    @daliu
    daliu
    on Aug 8, 2026
    ContributorAuthor

    CONTROL-PATH AUDIT — 2026-08-08T07:08:01Z

    No intervening state change: Netlify still publishes deploy-preview 6a6f038b9701b00008066000; the false-marker/root SHA-256 remains addae38deda751ce2948e9933a14343b9af7824bbc4fa568765b0f0bae9f3597; #617 has no new approval or release action.

    The missing supported control is now explained by access, not by target provenance:

    • Netlify's public site metadata identifies account jeffchang5, display name Jeffrey Chang's team.
    • The current signed-in session can read this project's deploy list and logs.
    • The same session receives Page not found — You might not have permissions to see this page for Project configuration.
    • Team Members is likewise unavailable.
    • Exact target deploy 6a6dc9ea588b0c0008036312 remains successful/ready, but its detail page exposes no Publish deploy action in this session.
    • Current official Netlify documentation says a rollback uses Publish deploy on a successful deploy detail page, without rebuilding, and that Developers can manage site deploys while an Internal Builder cannot publish to production.

    Therefore the supported handoff is concrete: a Team Owner for Jeffrey Chang's team must either perform the exact reviewed Publish deploy restoration or grant a named backup officer Developer access to this project and approve that exact action. Do not use rebuild/retry, DNS/configuration, token sharing, or an undocumented API workaround.

    This identifies the correct owner/workspace and access class, but it is not itself backup-officer approval. No production, repository, Firebase, provider-configuration, sign-in, or data state changed. #617 remains claimed and stopped before mutation.

  5. daliu commented on Aug 8, 2026

    @daliu
    daliu
    on Aug 8, 2026
    ContributorAuthor

    THIRD CONSECUTIVE AUDIT — BLOCKED / CLAIM RELEASED at 2026-08-08T07:10:46Z

    The exact blocking condition is unchanged for a third resumed goal turn:

    • no named backup officer has approved the exact WEB-002B — Restore production after unexpected Netlify preview promotion #617 pointer restoration;
    • the current Netlify session remains read-only and cannot access Team Members or Project configuration;
    • exact target 6a6dc9ea588b0c0008036312 remains ready and independently approved, but Publish deploy is not available to this session;
    • production still publishes preview 6a6f038b9701b00008066000 with the same missing-marker SPA bytes;
    • no protected Firebase/directory release authority or live backend proof appeared.

    The /root Netlify-pointer claim is RELEASED so a Team Owner or approved project Developer can take over without collision. #617 is owner-blocked, not technically blocked: a Team Owner for Jeffrey Chang's team must record the named backup approval and either perform the exact reviewed Publish deploy action or grant that approved operator project Developer access.

    Do not copy credentials, rebuild/retry, alter configuration or DNS, deploy Firebase from a laptop, sign in to the app, or touch production data as a substitute. Once approval/access exists, reclaim with a fresh timestamp, re-read production and the target immediately, perform only the exact pointer restoration, and run the recorded marker/hash/header/deep-route checks.

    No repository, Netlify production, Firebase, provider-configuration, account, payment, event, or production-data state changed during this blocker audit.

  6. daliu
    added
    status:blocked-ownerRequires an owner or external decision
    on Aug 8, 2026
  7. daliu
    removed their assignment
    on Aug 8, 2026
  8. daliu
    removed
    status:in-progressClaimed and actively being implemented
    on Aug 8, 2026

  9. daliu commented on Aug 8, 2026

    @daliu
    daliu
    on Aug 8, 2026
    ContributorAuthor

    APPROVAL RECORDED / CLAIM RESUMED — 2026-08-08T07:44:36Z

    Dave Liu explicitly approved the #617 recovery in the current Codex task. /root now reclaims only the already-reviewed Netlify production-pointer restoration from published preview 6a6f038b9701b00008066000 to exact ready deploy 6a6dc9ea588b0c0008036312, plus provider/public readback and incident evidence.

    This approval does not widen scope. Source, Git refs/merges, release manifest, Netlify configuration, DNS, Firebase, Rules, indexes, Functions, outside-provider settings, app sign-in, account/member/payment/event state, and production data remain immutable.

    Before mutation, re-read current published deploy, target state/provenance, all newer issue comments, and the supported Netlify control. Proceed only if the target remains exact and the UI explicitly offers Publish deploy for that existing artifact without rebuild/retry/configuration. If the current Netlify role still lacks that control, stop and retain the exact Team Owner/Developer handoff; do not substitute credentials, hidden API calls, or broader actions.

  10. daliu
    added
    status:in-progressClaimed and actively being implemented
    and removed
    status:blocked-ownerRequires an owner or external decision
    on Aug 8, 2026
  11. daliu
    self-assigned this
    on Aug 8, 2026

  12. daliu commented on Aug 8, 2026

    @daliu
    daliu
    on Aug 8, 2026
    ContributorAuthor

    APPROVAL READBACK / ACCESS STOP — 2026-08-08T07:45:59Z

    Approval is recorded and the immediate pre-action readback passed:

    The approved action still cannot be executed by the current Netlify session. The exact successful target page exposes no Publish deploy control, project navigation remains limited to Deploys and Agent runs, and Project configuration remains unavailable for this role. Approval does not create provider permission.

    No hidden API call, credential handling, rebuild/retry, configuration, DNS, Firebase, or data action was attempted. The claim remains active while a Team Owner for Jeffrey Chang's team either grants the current operator project Developer access or signs an authorized Team Owner/Developer session into the browser. After that external access change, re-read production and target again before one exact Publish deploy action.

    No production or repository state changed.

  13. daliu commented on Aug 13, 2026

    @daliu
    daliu
    on Aug 13, 2026
    ContributorAuthor

    FINAL COMPLETE / CLOSED / CLAIM RELEASED — 2026-08-13T05:32Z

    The recovery outcome is now independently verified. This audit did not perform or attribute the Netlify mutation: Netlify's public site record shows that exact reviewed deploy 6a6dc9ea588b0c0008036312 became the published production deploy at 2026-08-09T18:55:25.440Z. Public metadata does not identify the operator, so this closure records an observed restoration, not a claim that this Codex turn performed it.

    Exact provider truth:

    • site deploy ID: 6a6dc9ea588b0c0008036312
    • state/context: ready / production
    • commit: 9ad6837756cdd409d296009fde5082eeeae5c059
    • public marker: control 9ad6837756cdd409d296009fde5082eeeae5c059, source 39ab8649df411262c8109a3c81a57bc38f1e168b, tree d76b496cdc5e79015bc048cb961758abbe88b9ce, 60 files, digest 07b10c7d5ff176a1ad893d7549b07042312df35f4a4126e8765824ca94eaefb8
    • root SHA-256: 2978a162d55409fae343e9595871413b7d5887884d5a6158a8b2123537823b94
    • marker SHA-256: 60552b724766751acb843acf1b70238f215fbae4c32ee008672e3e3a268cb604
    • bundle /static/js/main.2a1afaac.js: SHA-256 0397d8dd2716884fc92e293d6082992edfaeae751bcd058093a7ee953c8a8ff0

    Public verification:

    • marker is HTTPS 200, application/json, Cache-Control: no-store,max-age=0, nosniff, and HSTS;
    • /shop, /events, /login, /shop/mprc-hat, /events/calendar, /account, and /admin/member-directory load through the restored SPA;
    • phone-width Shop shows the bounded in-person Hat / Jacket / Treasurer cash-or-Venmo truth;
    • Events and Calendar show the fixed accessible retry-later outcome without raw provider text;
    • signed-out Account redirects to Login;
    • /admin/member-directory is absent/404;
    • the exact live bundle contains zero matches for member-directory, listDirectoryMembers, updateMemberDirectoryProfile, profilePhoto, or searchableByMembers.

    Current repository main is the later source-only SHA 3af585077d665d7720b35b5b89505b1d90d9a007; exact-main CI 31670041767 passed all five jobs, but that later main is not the published website and is not claimed live.

    No repository edit, Git ref, rebuild, retry, Netlify configuration, DNS, Firebase, Rules, indexes, Functions, sign-in, provider configuration, payment/event/member record, or production-data action occurred during this closure audit. Firebase and other provider surfaces remain unchanged/unverified. The exact Netlify-pointer claim is RELEASED, the assignee/status are cleared, and #617 is closed. Future publication work must begin with a fresh claim and re-read this production marker before any preview or release action.

    Officer impact: the documented #473 static Shop and contained Events/Calendar website is restored; the prematurely published profile/directory frontend is no longer public.

    Officer documentation: None — public behavior exactly matches the already documented #473 artifact.

    Deployment evidence: Netlify production and public route/marker/bundle readbacks are verified above; no Firebase or other provider deployment is claimed.

  14. daliu
    closed this as completedon Aug 13, 2026
  15. daliu
    removed their assignment
    on Aug 13, 2026
  16. daliu
    removed
    status:in-progressClaimed and actively being implemented
    on Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:webWeb application and hostingpriority:P0Launch blocker or urgent security risksize:XSOwner decision or very narrow changetype:reliabilityReliability and recovery

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions