Skip to content

fix(aws-control): report the last retention sweep's result - #19783

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
fix/12491-retention-sweep-result
Oct 13, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
fix/12491-retention-sweep-result

Conversation

@cruisercohen

@cruisercohen cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The aws-control backup status read says how each kind's last retention sweep ended, refusals included, so a sweep refused every night no longer looks identical to a working one.

A retention sweep can end in several terminal states: retention off or its setting unreadable, authorization refused, a listing it would not trust, authorization withdrawn before deletion, the keep count withdrawn mid-sweep, cleanup failed, or a declined sessions sweep. Each one reaches only the SEL audit event and a log line. GET /backup/{account} serves the configured retentionKeep and nothing about whether any sweep acted on it. A successful sweep cannot say what it retired either.

Why it matters

An operator enables retention, the S3 bill does not fall, and nothing in the app says why. Authorization withdrawn and cleanup failed are both states an operator can fix once they know about them, and today they are invisible from the endpoint where retention is configured.

Not a goal

What changed (motivation → approach → change)

The status read now carries retentionLastSweep: per kind, how the last sweep ended.

Field Meaning
result successful, refused (an authorization gate decided against it) or failed (anything else, including a credential error at a gate)
skipped the sweep's own fixed reason text, never an exception message
retired, versions the same counts the audit event carries
at when the record was written

Every terminal state reached only the audit trail because nothing wrote a record on those paths. The precedent for a fix already exists: retentionUnclaimed persists a stamped per-kind record in backup.json and serves it from the same read.

The sweep in backup_parts/retention.py now calls one best-effort recorder, _record_last_sweep, on every terminal path. The sessions decline in backup.py calls it too. Unlike the unclaimed pair, it writes on the refusing paths, because the refusal is the fact worth publishing. The refused/failed split is the audit event's own, so the two never disagree about one ending. A later sweep replaces the kind's record, so a success never outlives the refusals after it. The stamp is taken under the state lock, so the record a reader sees is never older than the one it replaced. A failed write never fails the backup. routes.py serves the record, and docs/system-specs/modules/aws-control.md documents it.

Backwards compatibility

Compatible: one new key in the state document and one new field on the status read. Existing fields, audit events and sweep behaviour are unchanged, and an install with no record reads {}.

Removes nothing: the diff only adds a record, a reader and a response field.

Tests

  • test_aws_control_backup_retention.py: every terminal state publishes its result and skipped, including retention off, an unusable count, both authorization gates (a gate decision vs. a credential error), an untrusted listing, the count withdrawn, switched off or garbled mid-sweep, cleanup failed, and a half-finished purge with its erased count. Also covered: a refusal replacing an earlier success, per-kind isolation, absence before any sweep, a corrupted record, a failed state write, and a recorder overtaken at the lock.
  • test_aws_control_backup.py: a declined sessions sweep replaces a stale success.
  • test_aws_control_routes.py: the status read serves retentionLastSweep beside retentionKeep, per account, on the unpolled half.
  • test_aws_control_backup_composition_contract.py: the new names on the facade.

Each recorder call site was removed one at a time, and each removal turns at least one of these tests red.

Manual verification

N/A — unit coverage sufficient: the change is a state write and a read of local state, and every terminal path is driven through the fake drive.

Related Issues

Fixes #12491

Pattern harvest

Rule candidate: review-prompt
Pattern: a terminal outcome of a background operation, a refusal above all, written only to the audit log and logs and never to the status read where that operation is configured

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…s read

A retention sweep ends in one of several terminal states -- retention off or
its setting unreadable, authorization refused, an untrusted listing,
authorization withdrawn before deletion, the keep count withdrawn mid-sweep,
cleanup failed, a declined sessions sweep, or a normal finish -- and every one
of them reached only the SEL audit event and a log line. GET
/backup/{account} served the configured keep count and nothing about whether
any sweep had acted on it, so a sweep refused every night looked identical to
a working one.

Persist the last sweep's ending per kind under retention_last_sweep in
backup.json and serve it as retentionLastSweep:
{kind: {result, skipped, retired, versions, at}}. result is successful,
refused or failed, split the way the audit event splits them (a credential
failure at a gate is failed, a gate decision is refused); skipped is the
sweep's own fixed reason text, never an exception message; retired and
versions are the audit event's counts; at is taken under the state lock, so
the record a reader sees is never older than the one it replaced. Unlike
retentionUnclaimed it is written on every
terminal path, the refusing ones included, and a later sweep replaces the
kind's record, so a success never outlives the refusals after it. The write
is best-effort and never fails the backup, like the recorders beside it.

Fixes #12491
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that conversations.skipped (the source of the sessions decline's skipped text) is always fixed reason text and never exception-derived, since it is now served over HTTP.

[DESIGN-REVIEWED] 3ca5fdf

Description read

[DESCRIPTION-READ] fea3a231a1ac2ca903d5450ccef856296e81b39b1775c8875c392f65d94271da

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, confirm #12491 asks for an HTTP-only field. Nothing in the repo reads retentionLastSweep, so its only reader is an operator calling the endpoint.

What this change ships

Inventory (8 items) — 8 justified

Intent: let an operator see from the backup status read whether each kind's retention sweep is actually working or refused every night. It is framed as a FIX, but it is an ADDITION that closes the observability gap #12491 reports.

  1. The backup status read gains retentionLastSweep, which gives per kind the result, the skip reason, what was retired, the versions erased and when — justified
  2. Refusals, untrusted listings, a withdrawn keep count and cleanup failures now reach the status read, not only the audit trail — justified
  3. A declined sessions sweep now replaces an older success with failed — justified
  4. A shutdown-time gate refusal reads failed, not refused, matching the audit event — justified
  5. backup.json gains a persisted retention_last_sweep record per account and kind — justified
  6. The record's timestamp is taken under the state lock, so a later write never carries an earlier time — justified
  7. The shutdown refusal sentence is now one shared constant instead of a literal string — justified
  8. The aws-control spec documents the new field and record — justified

[FIRST-PRINCIPLES-REVIEWED] 3ca5fdf

Description read

[DESCRIPTION-READ] 1ac6e96f85b8cba22e9c2736fe79f60da413eed86e457016a1ee292c13e6124a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3ca5fdf

Verdict parsed from the review's SHA-scoped output markers for commit 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3ca5fdf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3ca5fdf96ac40d59b350a62283f2401e04f3d7eb: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author
  • On a credential error, skipped reads "authorization refused" while result reads failed

rebutted, kept as shipped: the two fields answer different questions. skipped is the sweep's own existing reason text and names WHICH step stopped it (the authorization step); result says whether that was a decision (refused) or a fault (failed).
That reason text is the sweep's established value for both gate exits, asserted by the existing unaudited-authorization tests, and the audit event splits the same pair the same way. Renaming it for one path would add a second spelling of the same stop.

@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=1106a63d745f Item 4: decline audit payload carried the reason in skipped

fixed in c936b12: the dict passed to the audit call keeps skipped empty as before, and only the published last-sweep record gets {**declined, "skipped": reason}.
The existing declined-sweep audit test now asserts the audit payload's skipped is empty, so the description's "audit events are unchanged" holds and is pinned.

@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=470f4d344069 Pin the clock before asserting strict timestamp order

fixed in c936b12: the overtaken-recorder test now pins the clock through the module's dt binding to instants that advance one second per read, and keeps the strict comparison.
Reverting the production fix (stamp taken before the lock) still turns the test red on the pinned clock, so it proves lock order, not clock resolution.

@cruisercohen
cruisercohen force-pushed the fix/12491-retention-sweep-result branch from c936b12 to 4dd311e Compare October 13, 2026 10:25
@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=0149c5065ada Item 3: _authorization_ending restated the RuntimeError check

fixed in 4dd311e: the unfiled-authorization audit helper now returns early on _authorization_ending(exc) == "refused", so isinstance(exc, RuntimeError) appears once in the retention module, inside _authorization_ending.
Fold note: the branch is at the two-commit limit, so this change was folded into the second commit only (c936b12 -> 4dd311e); the first commit is untouched. The fold changes that one call in the retention module and its helper's docstring, nothing else.

@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=55b8e652ef07 Suggestion: wrap each audit call and its last-sweep record in one end-of-sweep helper

rebutted as disproportional for this fix: the exits pair two audit helpers with different contracts. _audit_unfiled_authorization deliberately files nothing for a gate decision, because the gate filed its own denied event, while _audit_retention always files. One wrapper would have to carry that branch and both error spellings.
The record follows the shape the module already uses for its two measurement recorders, which also sit beside the audit call. Every current exit is pinned: removing any single recorder call turns at least one test red.

@cruisercohen
cruisercohen force-pushed the fix/12491-retention-sweep-result branch from 4dd311e to 1a783d6 Compare October 13, 2026 10:46
@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=6ce889e48032 Gateway shutdown at an authorization gate published refused while the gate audits failed

fixed in 1a783d6: a gate refusal is published as refused only when the app is not stopping. The teardown refusal is the one the gate files as failed, and it is raised exactly when the stop signal is set, so it is now published as failed at both gates.
The already-filed check stays a RuntimeError type test, written once and shared by the audit helper, so the shutdown refusal is still filed exactly once. A new test runs the real gate to its stop check at each gate and fails without the fix.
Fold note: the branch is at the two-commit limit, so this was folded into the second commit only (4dd311e -> 1a783d6). The first commit is untouched.

@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=e43851d6586b Suggestion: one end-of-sweep helper for the audit call and the last-sweep record

rebutted as disproportional for this fix, as on the prior head: the exits pair two audit helpers with different contracts. One of them deliberately files nothing for a gate decision, because the gate filed it, while the other always files. A single wrapper would have to carry that branch and both error spellings.
The record follows the module's existing shape for its two measurement recorders, which also sit beside the audit call. Every current exit is pinned: removing any one recorder call turns at least one test red.

…s with the audit

The overtaken-recorder test compared two live clock reads strictly, which two reads in one Windows clock tick cannot satisfy; it now reads a pinned clock that advances one second per read. The declined sessions sweep's audit payload keeps skipped empty as before; only the published last-sweep record carries the reason. The already-filed check for gate refusals is written once and shared by the audit helper and the published record. The gate's teardown refusal, the one it files as failed, is told apart by its own named text, so it is published as failed while an access decision made during a shutdown is still published as refused.
@cruisercohen
cruisercohen force-pushed the fix/12491-retention-sweep-result branch from 1a783d6 to 3ca5fdf Compare October 13, 2026 11:05
@cruisercohen

cruisercohen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the stop-flag read added on the prior head, replaced by the teardown refusal's own named text

  • span=6ce889e48032 Disabling the app mid-sweep: the stop flag relabels the gate's denied refusal as failed

fixed in 3ca5fdf: the stop flag is no longer read. The teardown refusal is told apart by its own text, which is now one named constant in the gate module that both the gate and the retention module read, so the sentence has one spelling.
Every gate exit and what is now published, matching what the gate audits:
live account mismatch, app disabled, consent no longer holds, grant withdrawn, grant names another account: gate files denied, published refused.
teardown refusal (the stop check, last in the gate): gate files failed, published failed.
anything the gate raises that is not a RuntimeError (expired credential, STS error): the unfiled-authorization helper files failed, published failed.
The unattended-grant checks never run for a sweep, whose payload kind is None. New test: app disabled plus stop flag set before the second gate gives a denied audit and a refused record; it fails on the stop-flag version.
Fold note: at the two-commit limit, folded into the second commit only (1a783d6 -> 3ca5fdf); the first commit is untouched.

@cruisercohen
cruisercohen marked this pull request as ready for review October 13, 2026 11:37
@cruisercohen
cruisercohen requested review from a team and KiroCrewCaptain as code owners October 13, 2026 11:37
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 13, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 13, 2026 12:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (9 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the aws-control retention sweep now records how each kind's last sweep ended (successful, refused or failed) and the backup status read serves it, so a sweep refused every night no longer looks like a working one. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/aws-control.md.

@KiroCrewCaptain
KiroCrewCaptain merged commit 19216a1 into main Oct 13, 2026
104 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/12491-retention-sweep-result branch October 13, 2026 12:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws-control retention: six terminal sweep states publish nothing to the status read, so a permanently refused sweep looks identical to a working one

2 participants