diff --git a/docs/adr/0323-decommission-deletes-owned-storage-as-journaled-effects.md b/docs/adr/0323-decommission-deletes-owned-storage-as-journaled-effects.md index a2db62596..67a862d62 100644 --- a/docs/adr/0323-decommission-deletes-owned-storage-as-journaled-effects.md +++ b/docs/adr/0323-decommission-deletes-owned-storage-as-journaled-effects.md @@ -2,6 +2,8 @@ ## Status +> **Decision 4 retained/partition-row lifetime superseded by [0341](0341-terminal-decommission-ledger-reclamation.md)** (2026-11-04); physical cleanup and other decisions remain unchanged. + Accepted (2026-11-03), issue #1848 (piece 2 of #1229). Implements ADR 0192 on the journaled decommission of ADR 0320, whose effect keys, kinds and retain path stand. Consumes ADR 0313 Decision 6's ledger vocabulary: provision writes `virtual_disk` rows today, and rows #1846 diff --git a/docs/adr/0341-terminal-decommission-ledger-reclamation.md b/docs/adr/0341-terminal-decommission-ledger-reclamation.md new file mode 100644 index 000000000..e219f12d3 --- /dev/null +++ b/docs/adr/0341-terminal-decommission-ledger-reclamation.md @@ -0,0 +1,53 @@ +# 0341 — Reclaim terminal decommission ledger rows after preserving provenance + +## Status + +Accepted for issue #1890; operator approved the provenance-preserving rule on 2026-11-04. +Supersedes ADR0323 Decision4's retained/partition-row lifetime only; its physical cleanup, +effect vocabulary and retain mode remain unchanged. Design review and scope audit completed. + +## Context + +Terminal delete_owned cleanup leaves partition and retained-item rows consuming capacity. +Its result normally records retained items, but unreadable rows currently report only kind +and reason, losing even available partial identity if the row is removed. Failed operations +may have no fresh result; abandoning an operation explicitly leaves its ledger untouched. + +## Decision + +1. On a fresh terminal completed/failed delete_owned transition, reclaim the guarded + system/partition's partition, virtual_disk and installer_media rows only with a matching + applied delete_lpar effect and a complete freshly persisted storage result covering the + remaining storage rows. State, result and ledger deletion share one SQLite transaction. +2. Unreadable retained entries gain optional ledger_identity, their original JSON identity + object without value coercion or discarded keys. Distinct unreadable records survive + independently. Reconstruction matches readable disks by kind/VIOS/group/name and media + by kind/VIOS/name, with normalized UUIDs and exact names; a deleted disk must not hide + a retained same-name disk in another group. Existing readable projections and + deleted/pending shapes remain unchanged. +3. Never use an older result retained by COALESCE as freshness evidence. Abandoned, + nonterminal, retain, dry-run, unproven, missing/oversized-result and historical cases + keep existing semantics; unknown ledger kinds are not garbage-collected. +4. Existing managed-system/partition ownership is the boundary, including rows whose + producer operation has expired. Other partitions/systems are untouched; no attribution + columns, database migration or producer reconstruction is introduced. +5. Existing result-size and 30-day retention limits stand. Operators own longer archival, + retained physical storage removal and historical/manual recovery. No live HMC writes + are introduced by reclamation. + +## Consequences + +Eligible rows stop consuming capacity while retained storage remains recorded for the +operation's existing lifetime. A missing complete bounded result conservatively leaves rows. +The result schema admits one optional nested object; old results remain readable. A failed +transaction preserves previous state/result/ledger together for existing recovery paths. + +## Considered & rejected + +- **Delete before saving the result.** judgment: a crash could destroy the remaining record. +- **Treat any terminal outcome as proof.** verified: store.operator_abandon explicitly + preserves ledger, and engine._run can end failed with no result (base e00b81241). +- **Keep unreadable rows indefinitely.** judgment: incomplete closure; the operator chose + preserving their original identity and applying the same eligibility rule. +- **Add an attribution schema or general ledger collector.** judgment: neither is needed + for the existing system/partition ownership boundary or this issue's terminal transition. diff --git a/docs/workflow/specs/2026-11-04-terminal-decommission-ledger-reclamation-design.md b/docs/workflow/specs/2026-11-04-terminal-decommission-ledger-reclamation-design.md new file mode 100644 index 000000000..1b767aeab --- /dev/null +++ b/docs/workflow/specs/2026-11-04-terminal-decommission-ledger-reclamation-design.md @@ -0,0 +1,145 @@ +# Terminal decommission ledger reclamation + +## Problem and authority + +Issue #1890 identifies partition and retained-resource rows that consume capacity after +`delete_owned` has deleted their partition. ADR 0323 deliberately leaves those rows. +The operator approved the provenance-preserving rule on 2026-11-04, recorded in +[the approval](https://github.com/randomparity/hmcpctl/issues/1890#issuecomment-6647344871) +and frozen in [scope](https://github.com/randomparity/hmcpctl/issues/1890#issuecomment-6647359549). +[ADR 0341](../../adr/0341-terminal-decommission-ledger-reclamation.md) records the decision. + +## Global Constraints + +Python >=3.11; hosted verification targets amd64 on Python 3.11, 3.12, 3.13 and 3.14. +No new dependency, database schema, migration, public facade, or HMC operation. +Use the existing managed-system UUID plus partition UUID ownership namespace. +Retained physical storage deletion, retain-mode changes, old-history rewrite, and general +ledger garbage collection are outside this issue and operator-owned. +Abandonment, nonterminal states, retain mode, dry runs, unproven deletion, missing or +oversized fresh results, and historical terminal records keep existing semantics. +Existing 30-day operation-result retention remains; longer archival is operator-owned. + +## Success + +- C1: a fresh completed/failed `delete_owned` termination with matching applied partition + deletion and complete fresh storage provenance removes its partition/retained-resource + rows, freeing their contribution to capacity; no other system/partition rows change. +- C2: each unreadable retained entry preserves its original JSON identity losslessly in + optional `ledger_identity`; readable entries retain their existing shape. +- C3: state/result/reclamation form one SQLite write transaction. A failing result/state + write or ledger delete leaves the previous state/result/ledger together recoverable. +- C4: the Global Constraints' unchanged cases are not retroactive cleanup entry points. +- C5: persisted status and reconstructed decommission results retain storage provenance + on completed, failed-after-delete, and repeat reads, including distinct unreadable rows. + +## Architecture and data flow + +Extend existing owners; no ownership move or new finalizer framework. `store.set_state` +continues to own persisted terminal transitions. `engine._end` already wraps it in +`store.write_transaction`; admission/operator abandonment use that transaction too. +`decommission_storage.owned_items` remains the owner of projected storage identities. +`decommission_operation` carries the cleanup pass result to the engine; its normal and +HTTP-412 failure paths already do so. No caller signature change is needed. + +Add one private store helper called by `set_state` for an eligible transition, before +releasing the transaction. Read the previous operation row to exclude a terminal replay; +use the incoming `result_json`, never the COALESCE-preserved previous result, as proof of +freshness. Eligibility requires tool `hmc_decommission_lpar`, recorded request mode +`delete_owned`, nonempty guarded system/partition, requested state terminal and outcome +completed/failed, and that operation's `delete_lpar` effect with kind `lpar.delete`, +status applied, target `lpar:` and identity `lpar_uuid` matching the +partition. UUID identity comparisons are case-insensitive. Open effects prohibit cleanup. + +A complete fresh storage result is the successfully serialized cleanup pass result with +`storage` object, `deleted`, `retained`, and `pending` arrays, and empty `pending`. +The bounded internal producer is `decommission_body.run`: only after `clean_storage` +returns does it construct this result. Exceptions before that return supply no complete +new storage pass. The store must also verify remaining storage rows have retained +provenance: a readable row matches its existing canonical projected identity in retained; +an unreadable row matches a retained entry with the same kind, reason unreadable_record, +and exact `ledger_identity` JSON value. Duplicate readable rows may share one entry as +ADR0323 already deduplicates those resources; distinct unreadable identities must not be +merged. Missing coverage makes the transition ineligible for reclamation, not a reason +to drop evidence or invent a retention reason. Partition rows need the applied delete +proof, not a storage entry. Only existing kinds partition, virtual_disk and installer_media +are reclaimed; unknown kinds retain their prior behavior rather than becoming general GC. + +Keep this comparison aligned with `owned_items` projection: disk identity is vios_uuid, +vg_uuid and disk_name; media identity is vios_uuid and media_name. UUIDs are normalized +as the producer already does; name comparison remains exact. Extra readable identity +fields are ignored under ADR0323, not newly exposed. No producer-operation join is used; +rows whose producing operation expired remain reclaimable in the same partition scope. +Use bound SQL parameters and delete only the selected kind/system/partition rows in the +same transaction as `set_state`. Preserve existing per-item removal after applied physical +storage deletes; it remains necessary during pauses and resumes before terminalization. + +For unreadable rows, add optional `ledger_identity` containing the original identity +object, including unknown keys and non-string JSON values without coercion. Widen only +the retained-entry value annotation to admit that object and propagate it through cleanup, +body serialization and result reconstruction. Existing readable, deleted and pending shapes +and legacy results remain accepted. Preserve each unreadable occurrence and avoid normal +item-key deduplication/filtering of those entries. Readable reconstruction filtering in +`same_item` and its `_storage` consumer uses the complete canonical identity: disk +(kind, normalized vios_uuid, normalized vg_uuid, exact disk_name), media +(kind, normalized vios_uuid, exact media_name). A deleted disk in one volume group must +not hide a retained namesake in another group on the same VIOS. This corrects an existing +direct-consumer defect required by C5 without changing result fields or adding a second +identity convention. No raw identity is added to warnings. +The existing MAX_RESULT_BYTES check applies after this addition: no truncation or larger +bound. An oversized successful body result pauses; an oversized carried failure result +uses existing failed/no-new-result behavior. Neither path reclaims ledger rows. + +## Alternatives + +- Recommended: extend the existing terminal transaction with explicit proof and coverage + checks. It preserves atomicity without another lifecycle or persisted marker. +- Reclaim in the cleanup body: rejected because its result has not yet been durably saved. +- Leave unreadable rows indefinitely: operator declined this limited closure in favor of + preserving original available identity in the result; no exception is silently added. + +## Failure model + +Actors and deployments: +- Local CLI/MCP operation workers and status readers sharing the private SQLite store. +- Supported Python runtimes; deterministic tests mock the external HMC boundary only. +Invariants and assets at stake: +- Retained storage provenance, recoverable operation state, capacity, and other partitions. +- Fresh matching delete proof; lossless unreadable identities; atomic terminal visibility. +Accepted failure classes: +- Approved ineligible cases retain rows and their capacity cost; safety precedes reclamation. +- Unknown ledger kinds remain unchanged; they are not the existing resource kinds of #1890. +- Result expiry after 30 days is existing policy, not permanent archival. +Covered elsewhere: +- Physical cleanup and longer archival: operator. HMC write authorization: existing tools. +- Corrupt/unreadable private stores: existing store refusal and operator recovery, ADR0190. + +## Threat model + +Boundary inventory: +- Added: unreadable ledger identity appears inside an authenticated operation result. +- Existing: persisted operation/effect/result data controls local ledger deletion. +Actor model: +- Trust the local credential holder and existing journal producers, not HMC text as code. +- A local adversary with write access to the private store is outside ADR0190's trust model. +Control per boundary: +- Existing private-store checks, operation status authorization, bounded JSON serialization; + identity stays structured JSON, with no evaluation, warning interpolation or public posting. +- Eligibility and row-coverage checks plus parameterized SQL bound the deletion target; + BEGIN IMMEDIATE serializes writers. Existing execution ownership excludes competing workers. +Explicitly out of scope: +- New authorization, storage deletion, historical GC or protection from an owner editing SQLite. + +## Validation + +Use the transient plan's focused-test inventory for C1-C5. Cover real SQLite commit/rollback +and second-connection visibility with deterministic synchronization; no sleep-based races. +Inject a SQLite delete trigger failure and verify all three persisted surfaces roll back. +Prove tests bite by disabling the scoped delete and by dropping ledger_identity in controlled +faults, observing failures, then restoring source. Cover duplicate rows, other systems and +partitions, mixed-case UUIDs, shared retained resources, pruned producers and capacity reuse. +Exercise the real engine/decommission body with only HMC calls mocked, including HTTP412, +resume, oversize fresh results and repeat result reconstruction. Readable results stay stable. +Run focused tests, lint/type checks, generated-doc regeneration if its schema output changes, +then final `just verify` and `uv run --no-sync prek run --all-files`. Hosted matrix supplies +Python3.12-3.14 evidence. No live-system mutations are authorized or needed. diff --git a/src/hmcpctl/operations/logical/store.py b/src/hmcpctl/operations/logical/store.py index 86a37a002..78b5be6ae 100644 --- a/src/hmcpctl/operations/logical/store.py +++ b/src/hmcpctl/operations/logical/store.py @@ -575,6 +575,12 @@ def set_state( ) -> None: _require_vocabulary(state, get_args(OperationState), "state") _require_vocabulary(outcome, (*get_args(OperationOutcome), None), "outcome") + if ( + state == "terminal" + and outcome in ("completed", "failed") + and result_json is not None + ): + _reclaim_decommission_ledger(conn, operation_id, result_json) now = _clock() conn.execute( "UPDATE operations SET state = ?, outcome = ?, warnings_json = ?," @@ -592,6 +598,104 @@ def set_state( ) +def _retained_identity(kind: str, identity: Mapping[str, Any]) -> dict[str, Any] | None: + """ADR0323's readable projection; unreadable rows require lossless ADR0336 evidence.""" + name_field = "disk_name" if kind == "virtual_disk" else "media_name" + fields = ( + ("vios_uuid", "vg_uuid", name_field) + if kind == "virtual_disk" + else ("vios_uuid", name_field) + ) + if ( + not all(isinstance(identity.get(key), str) for key in fields) + or not identity[name_field] + ): + return None + return { + key: identity[key].lower() if key.endswith("_uuid") else identity[key] + for key in fields + } + + +def _retained_row(row: sqlite3.Row, retained: list[Any]) -> bool: + kind, identity = row["kind"], json.loads(row["identity_json"]) + projected = _retained_identity(kind, identity) + for entry in retained: + if ( + not isinstance(entry, dict) + or entry.get("kind") != kind + or not entry.get("reason") + ): + continue + if projected is not None: + if _retained_identity(kind, entry) == projected: + return True + elif entry.get("reason") == "unreadable_record" and json.dumps( + entry.get("ledger_identity"), sort_keys=True + ) == json.dumps(identity, sort_keys=True): + return True + return False + + +def _reclaim_decommission_ledger( + conn: sqlite3.Connection, operation_id: str, result_json: str +) -> None: + """Reclaim only newly terminal, proven decommission rows with fresh provenance.""" + row = get_operation(conn, operation_id) + arguments = json.loads(row["request_json"])["arguments"] + if ( + row["state"] == "terminal" + or row["tool"] != "hmc_decommission_lpar" + or arguments.get("storage_cleanup") != "delete_owned" + or arguments.get("dry_run") + or not row["system_uuid"] + or not row["partition_uuid"] + or len(result_json.encode("utf-8")) > MAX_RESULT_BYTES + ): + return + partition = row["partition_uuid"].lower() + effects = load_effects(conn, operation_id) + deleted = effects.get("delete_lpar") + if ( + deleted is None + or deleted.kind != "lpar.delete" + or deleted.status != "applied" + or deleted.target.lower() != f"lpar:{partition}" + or str((deleted.identity or {}).get("lpar_uuid", "")).lower() != partition + or any(e.status in ("intended", "uncertain") for e in effects.values()) + ): + return + result = json.loads(result_json) + storage = result.get("storage") if isinstance(result, dict) else None + if ( + not isinstance(storage, dict) + or not all( + isinstance(storage.get(key), list) + for key in ("deleted", "retained", "pending") + ) + or storage["pending"] + ): + return + rows = conn.execute( + "SELECT operation_id, key, kind, identity_json FROM ledger" + " WHERE lower(system_uuid) = ? AND lower(partition_uuid) = ?" + " AND kind IN ('partition', 'virtual_disk', 'installer_media')", + (row["system_uuid"].lower(), partition), + ).fetchall() + if not all( + r["kind"] == "partition" or _retained_row(r, storage["retained"]) for r in rows + ): + return + if not conn.in_transaction: + raise RuntimeError( + "terminal decommission reclamation requires a write transaction" + ) + conn.executemany( + "DELETE FROM ledger WHERE operation_id = ? AND key = ?", + [(r["operation_id"], r["key"]) for r in rows], + ) + + def interrupt_orphan(conn: sqlite3.Connection, operation_id: str) -> None: """Mark a running record with no live worker interrupted, as lock acquisition does.""" conn.execute( diff --git a/src/hmcpctl/operations/lpar/decommission.py b/src/hmcpctl/operations/lpar/decommission.py index d790db175..d2ca69fda 100644 --- a/src/hmcpctl/operations/lpar/decommission.py +++ b/src/hmcpctl/operations/lpar/decommission.py @@ -193,7 +193,7 @@ class DecommissionStorage(TypedDict): """What ``delete_owned`` deleted, retained (with a reason) and has yet to delete.""" deleted: tuple[dict[str, str], ...] - retained: tuple[dict[str, str], ...] + retained: tuple[dict[str, Any], ...] pending: tuple[dict[str, str], ...] diff --git a/src/hmcpctl/operations/lpar/decommission_operation.py b/src/hmcpctl/operations/lpar/decommission_operation.py index ec40d58cd..44881de17 100644 --- a/src/hmcpctl/operations/lpar/decommission_operation.py +++ b/src/hmcpctl/operations/lpar/decommission_operation.py @@ -379,7 +379,11 @@ def _storage( deleted = deleted_from_effects(record.effects) gone = {same_item(item) for item in deleted} earlier = (latest or {}).get("retained", ()) - retained = [i for i in earlier if same_item(i) not in gone] + retained = [ + i + for i in earlier + if i.get("reason") == "unreadable_record" or same_item(i) not in gone + ] return { "deleted": deleted, "retained": tuple(retained), diff --git a/src/hmcpctl/operations/lpar/decommission_storage.py b/src/hmcpctl/operations/lpar/decommission_storage.py index 45ceb4205..ffbd122d0 100644 --- a/src/hmcpctl/operations/lpar/decommission_storage.py +++ b/src/hmcpctl/operations/lpar/decommission_storage.py @@ -136,7 +136,7 @@ def _item(kind: str, identity: Mapping[str, Any]) -> OwnedItem | None: def owned_items( system_uuid: str, partition_uuid: str -) -> tuple[tuple[OwnedItem, ...], tuple[dict[str, str], ...]]: +) -> tuple[tuple[OwnedItem, ...], tuple[dict[str, Any], ...]]: """The partition's ledgered items, and those retained because another partition's entry names them too (ADR 0323 Decision 1). No store means nothing is owned.""" partition = partition_uuid.lower() @@ -152,13 +152,19 @@ def owned_items( ) mine: dict[OwnedItem, None] = {} elsewhere: set[OwnedItem] = set() - unreadable: list[dict[str, str]] = [] + unreadable: list[dict[str, Any]] = [] for kind, (_operation, lpar, identity) in rows: item = _item(kind, identity) ours = (lpar or "").lower() == partition if item is None: if ours: - unreadable.append({"kind": kind, "reason": "unreadable_record"}) + unreadable.append( + { + "kind": kind, + "reason": "unreadable_record", + "ledger_identity": identity, + } + ) continue if ours: mine[item] = None @@ -368,7 +374,7 @@ async def clean_storage( partition_uuid: str, *, authorized_vios: frozenset[str], -) -> tuple[tuple[dict[str, str], ...], str | None]: +) -> tuple[tuple[dict[str, Any], ...], str | None]: """Delete the partition's deletable owned items; return the retained ones, and the failure to end with when a delete got HTTP 412 (ADR 0323 Decision 3) after every item was tried. Any other HMC error propagates with its effect open.""" @@ -446,8 +452,8 @@ def deleted_from_effects(effects: Iterable[EffectRecord]) -> tuple[dict[str, str def same_item(entry: Mapping[str, Any]) -> tuple[Any, ...]: """What identifies one item across report entries, whatever else they carry.""" - name = entry.get("disk_name", entry.get("media_name")) - return entry.get("kind"), entry.get("vios_uuid"), name + item = _item(entry.get("kind", ""), entry) + return () if item is None else (item.kind, item.vios_uuid, item.vg_uuid, item.name) def pending_from_effects(effects: Iterable[EffectRecord]) -> tuple[dict[str, str], ...]: diff --git a/tests/lpar/test_decommission_operation.py b/tests/lpar/test_decommission_operation.py index f9c9e28f8..8610b55b5 100644 --- a/tests/lpar/test_decommission_operation.py +++ b/tests/lpar/test_decommission_operation.py @@ -808,3 +808,72 @@ def test_a_resume_after_the_delete_keeps_the_first_runs_inventory(hmc): assert result.blast_radius["lpar_name"] == "aix-prod" assert not any("inventory is unavailable" in w for w in result.warnings) assert len(result.storage["deleted"]) == 1 + + +@pytest.mark.parametrize("failed", [False, True]) +def test_terminal_reclamation_preserves_unreadable_and_readable_namesakes(hmc, failed): + _owned_disks(hmc, "data", "fail" if failed else "extra") + other_group = "77777777-7777-7777-7777-777777777777" + malformed = [ + {"disk_name": "partial", "extra": [None, 7, {"flag": True}]}, + {"disk_name": "partial", "extra": {"second": False}}, + ] + with store.session() as conn, store.write_transaction(conn): + for index, identity in enumerate( + [ + {"vios_uuid": VIOS_UUID, "vg_uuid": other_group, "disk_name": "data"}, + *malformed, + ] + ): + store.add_ledger( + conn, + "expired", + key=f"kept{index}", + kind="virtual_disk", + system_uuid=SYSTEM_UUID, + partition_uuid=LPAR_UUID, + identity=identity, + ) + store.add_ledger( + conn, + "expired", + key="partition", + kind="partition", + system_uuid=SYSTEM_UUID, + partition_uuid=LPAR_UUID, + identity={"lpar_uuid": LPAR_UUID}, + ) + original = hmc.get_volume_group.return_value + hmc.get_volume_group.side_effect = lambda _vios, group: ( + {"Resource": {"VirtualDisks": ""}} if group == other_group else original + ) + if failed: + hmc.delete_virtual_disk.side_effect = [None, HMCError("changed", 412)] + record = _submit(hmc, arguments=OWNED) + assert record.outcome == ("failed" if failed else "completed") + for observed in (record, _submit(hmc, arguments=OWNED)): + retained = result_from_record(observed, delete_owned=True).storage["retained"] + assert [ + r["ledger_identity"] for r in retained if r["reason"] == "unreadable_record" + ] == malformed + assert any(r.get("vg_uuid") == other_group for r in retained) + with store.session() as conn: + assert conn.execute("SELECT count(*) FROM ledger").fetchone()[0] == 0 + + +@pytest.mark.parametrize("failed", [False, True]) +def test_oversized_terminal_provenance_keeps_ledger(hmc, monkeypatch, failed): + _owned_disks(hmc, "data") + if failed: + hmc.delete_virtual_disk.side_effect = HMCError("changed", 412) + else: + hmc.get_volume_group.return_value = {"Resource": {"VirtualDisks": ""}} + monkeypatch.setattr(store, "MAX_RESULT_BYTES", 1) + record = _submit(hmc, arguments=OWNED) + assert (record.state, record.outcome) == ( + ("terminal", "failed") if failed else ("paused", "needs_attention") + ) + assert record.result is None + assert _statuses(record)["delete_lpar"] == "applied" + with store.session() as conn: + assert conn.execute("SELECT count(*) FROM ledger").fetchone()[0] == 1 diff --git a/tests/unit/test_logical_store.py b/tests/unit/test_logical_store.py index c6924d09f..4779c02e1 100644 --- a/tests/unit/test_logical_store.py +++ b/tests/unit/test_logical_store.py @@ -731,3 +731,229 @@ def test_write_transaction_rolls_back_body_failure(): assert caught.value is primary assert not conn.in_transaction assert conn.execute("SELECT * FROM example").fetchall() == [] + + +def _decommission_ledger(conn): + operation = _oid(80) + _seed(conn, operation) + identity = {"vios_uuid": "A", "vg_uuid": "B", "disk_name": "data"} + result = { + "storage": { + "deleted": [], + "pending": [], + "retained": [ + { + "kind": "virtual_disk", + "vios_uuid": "a", + "vg_uuid": "b", + "disk_name": "data", + "reason": "referenced", + } + ], + } + } + with store.write_transaction(conn): + conn.execute( + "UPDATE operations SET tool = ?, request_json = ? WHERE operation_id = ?", + ( + "hmc_decommission_lpar", + json.dumps({"arguments": {"storage_cleanup": "delete_owned"}}), + operation, + ), + ) + store.set_partition(conn, operation, "sys", "part") + store.record_intent(conn, operation, "delete_lpar", "lpar.delete", "lpar:part") + store.record_outcome( + conn, operation, "delete_lpar", "applied", {"lpar_uuid": "part"} + ) + for key, kind, system, partition in ( + ("p", "partition", "sys", "part"), + ("d", "virtual_disk", "SYS", "PART"), + ("duplicate", "virtual_disk", "sys", "part"), + ("other", "virtual_disk", "sys", "other"), + ("system", "virtual_disk", "other", "part"), + ("future", "unknown", "sys", "part"), + ): + store.add_ledger( + conn, + "pruned-producer", + key=key, + kind=kind, + system_uuid=system, + partition_uuid=partition, + identity=identity, + ) + return operation, json.dumps(result) + + +@pytest.mark.parametrize("outcome", ["completed", "failed"]) +def test_terminal_decommission_reclaims_scoped_rows(outcome, monkeypatch): + with store.session() as conn: + operation, result = _decommission_ledger(conn) + with store.write_transaction(conn): + store.set_state(conn, operation, "terminal", outcome, result_json=result) + assert {r[0] for r in conn.execute("SELECT key FROM ledger")} == { + "other", + "system", + "future", + } + assert store.read_record(conn, operation).result == json.loads(result) + monkeypatch.setattr(store, "STORE_CAPACITY", 4) + _seed(conn, _oid(81)) # capacity freed; the three unrelated rows remain + + +@pytest.mark.parametrize( + "change", + [ + "terminal", + "retain", + "tool", + "no_partition", + "missing", + "kind", + "target", + "identity", + "uncertain", + "open", + "absent_result", + "stale", + "oversized", + "coverage", + "pending", + "malformed", + "abandoned", + "paused", + "dry_run", + ], +) +def test_ineligible_decommission_preserves_ledger(change): + with store.session() as conn: + operation, result = _decommission_ledger(conn) + state, outcome = "terminal", "completed" + if change in {"terminal", "retain", "tool", "no_partition", "dry_run"}: + updates = { + "terminal": ("state", "terminal"), + "retain": ("request_json", '{"arguments":{}}'), + "tool": ("tool", "other"), + "no_partition": ("partition_uuid", None), + "dry_run": ( + "request_json", + '{"arguments":{"storage_cleanup":"delete_owned","dry_run":true}}', + ), + } + field, value = updates[change] + conn.execute( + f"UPDATE operations SET {field} = ? WHERE operation_id = ?", + (value, operation), + ) + elif change in {"kind", "target", "identity", "uncertain"}: + field, value = { + "kind": ("kind", "wrong"), + "target": ("target", "lpar:other"), + "identity": ("identity_json", '{"lpar_uuid":"other"}'), + "uncertain": ("status", "uncertain"), + }[change] + conn.execute(f"UPDATE effects SET {field} = ?", (value,)) + elif change == "missing": + conn.execute("DELETE FROM effects") + elif change == "open": + store.record_intent(conn, operation, "other", "write", "target") + elif change in {"absent_result", "stale"}: + if change == "stale": + conn.execute("UPDATE operations SET result_json = ?", (result,)) + result = None + elif change == "oversized": + result = json.dumps( + {"padding": "x" * store.MAX_RESULT_BYTES, **json.loads(result)} + ) + elif change in {"coverage", "pending", "malformed"}: + payload = json.loads(result) + payload["storage"]["retained"] = ( + [] if change == "coverage" else payload["storage"]["retained"] + ) + if change == "pending": + payload["storage"]["pending"] = [{"kind": "virtual_disk"}] + if change == "malformed": + payload["storage"] = None + result = json.dumps(payload) + elif change == "abandoned": + outcome = "abandoned" + elif change == "paused": + state, outcome = "paused", "needs_attention" + before = list(conn.execute("SELECT * FROM ledger")) + with store.write_transaction(conn): + store.set_state(conn, operation, state, outcome, result_json=result) + assert list(conn.execute("SELECT * FROM ledger")) == before + + +def test_terminal_reclamation_rolls_back_with_state_result(): + with store.session() as conn, store.session() as reader: + operation, result = _decommission_ledger(conn) + conn.execute( + "CREATE TRIGGER reject_reclaim BEFORE DELETE ON ledger BEGIN SELECT RAISE(ABORT, 'fault'); END" + ) + with ( + pytest.raises(sqlite3.IntegrityError, match="fault"), + store.write_transaction(conn), + ): + store.set_state( + conn, operation, "terminal", "completed", result_json=result + ) + assert store.read_record(reader, operation).state == "running" + assert store.read_record(reader, operation).result is None + assert reader.execute("SELECT count(*) FROM ledger").fetchone()[0] == 6 + conn.execute("DROP TRIGGER reject_reclaim") + with store.write_transaction(conn): + store.set_state( + conn, operation, "terminal", "completed", result_json=result + ) + assert store.read_record(reader, operation).state == "running" + assert reader.execute("SELECT count(*) FROM ledger").fetchone()[0] == 6 + assert store.read_record(reader, operation).state == "terminal" + assert store.read_record(reader, operation).result == json.loads(result) + assert reader.execute("SELECT count(*) FROM ledger").fetchone()[0] == 3 + + +@pytest.mark.parametrize("covered", [True, False]) +def test_terminal_reclamation_requires_media_and_unreadable_provenance(covered): + with store.session() as conn: + operation, result = _decommission_ledger(conn) + malformed = {"media_name": "partial", "raw": [None, False, {"n": 5}]} + media = {"vios_uuid": "V", "media_name": "image"} + for key, identity in (("bad", malformed), ("media", media)): + store.add_ledger( + conn, + "expired", + key=key, + kind="installer_media", + system_uuid="sys", + partition_uuid="part", + identity=identity, + ) + payload = json.loads(result) + payload["storage"]["retained"].extend( + [ + { + "kind": "installer_media", + "vios_uuid": "v", + "media_name": "image", + "reason": "referenced", + }, + { + "kind": "installer_media", + "reason": "unreadable_record", + "ledger_identity": malformed if covered else {}, + }, + ] + ) + with store.write_transaction(conn): + store.set_state( + conn, + operation, + "terminal", + "completed", + result_json=json.dumps(payload), + ) + assert conn.execute("SELECT count(*) FROM ledger").fetchone()[0] == ( + 3 if covered else 8 + )