Skip to content

fix(ops-mission-control): count and compact duplicate ledger lines in hygiene - #21787

Merged
bolichen97 merged 1 commit into
mainfrom
fix/ledger-hygiene-dedupe-21715
Oct 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/ledger-hygiene-dedupe-21715

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The ledger hygiene pass reports how many duplicate ledger lines it collapsed and rewrites the file without them.

POST /ledger/hygiene always returned deduped: 0. When a git merge left two copies of the same entry in the ledger file, the pass said there were no duplicates and left both lines on disk.

Why it matters

The hygiene SOP tells the agent the pass dedupes and returns deduped: N, so a merged team ledger looked clean when it was not. The duplicate lines stayed in the file and in the synced team repo until some unrelated decay or prune forced a rewrite.

Not a goal

What changed (motivation → approach → change)

Symptom: deduped was always 0 and merged duplicate lines were never removed.

Root cause: hygiene read the ledger through read_entries_for_update, which already merges duplicate ids with _reconcile. Every id it saw was unique, so its own merge loop never ran and "before minus after" was always 0.

Change: in backend/ledger.py, the strict read moves into _read_counted_for_update, which also counts the duplicate lines it collapsed (valid lines minus unique ids; a malformed line is skipped, not counted). read_entries_for_update wraps it unchanged. Hygiene reports that count as deduped and rewrites the file when it is non-zero. The unreachable merge loop is gone. The ledger-hygiene.md SOP step 1 now says what deduped counts. The POST /ledger/hygiene handler already counts deduped toward changed, so a compacting pass now writes its ledger_hygiene audit row and is reported.

Backwards compatibility

Compatible: the response keys and types are the same; deduped is now the real count, and a ledger with merge duplicates gets one rewrite to one line per entry with the same merged values reads already returned.

Removes nothing: the deleted merge loop in _hygiene_locked could never run, and reads, merges and the route contract are unchanged.

Tests

In tests/test_store_and_gate_ledger.py:

  • test_hygiene_counts_and_compacts_duplicate_lines: two entries, every line duplicated; hygiene reports deduped: 2, the file has 2 lines and 2 unique ids, and a second pass reports 0. Red on origin/main (0 != 2).
  • test_hygiene_compaction_keeps_the_merged_counters: two raw lines of one id with different counters and fingerprints; the rewritten line keeps the max use_count, the max miss_count and both fingerprints. Red on origin/main (0 != 1).
  • test_hygiene_does_not_count_a_malformed_line_as_a_duplicate: a broken line and a blank line give deduped: 0 and no rewrite.

tests/test_ledger_index.py and test_a_failed_mutation_read_refuses_instead_of_truncating patch _read_counted_for_update, the read hygiene now starts from, so the race and refused-read tests still drive hygiene.

Determinism: both test files ran 5 times in a row serially and once under -n 2 in reversed file order, all green. The ops-mission-control tests (1121 passed) and the local gate's related backend set (22515 passed) are green.

Manual verification

N/A — unit coverage sufficient: the change is a pure file-level ledger pass and the tests drive it on a real file under an isolated KIROCREW_HOME.

Related Issues

Closes #21715

Pattern harvest

Rule candidate: review-prompt
Pattern: a second dedupe/merge pass over data an upstream reader already deduplicated, so its counter is always zero.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 955c76f2e51211065ddebba44746658ca39fe2d7 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 955c76f2e51211065ddebba44746658ca39fe2d7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 955c76f

Description read

[DESCRIPTION-READ] ed48537144b06a72bf715f2e380afa804f28e359f47362300b334d2bc24c97a1

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 955c76f2e51211065ddebba44746658ca39fe2d7 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (5 items) — 5 justified

Intent: let operators trust ledger hygiene to report and remove duplicate ledger lines left by git merges. FIX (Closes #21715; new tests fail on base).

  1. POST /ledger/hygiene now reports the real deduped count instead of always 0 — justified
  2. A hygiene pass whose only change is duplicates now rewrites the file with one line per entry, using the existing if dupes_removed or … write path — justified
  3. The second merge loop in _hygiene_locked, which could never run, is deleted; _reconcile is now the only merge rule — justified
  4. New private _read_counted_for_update returns the read plus a collapsed-line count; one caller, not generalized — justified
  5. SOP step 1 now says what deduped counts and that a non-zero count rewrites the file — justified

[FIRST-PRINCIPLES-REVIEWED] 955c76f

Description read

[DESCRIPTION-READ] 9a282637abe97042548235e280389143b2e9f95cebe7124c482b7e4cacfbed4d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 955c76f2e51211065ddebba44746658ca39fe2d7 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 955c76f

Verdict parsed from the review's SHA-scoped output markers for commit 955c76f2e51211065ddebba44746658ca39fe2d7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 955c76f2e51211065ddebba44746658ca39fe2d7: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 955c76f2e51211065ddebba44746658ca39fe2d7 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 955c76f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 955c76f2e51211065ddebba44746658ca39fe2d7: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests (Windows) (1) is main's, not this PR's: test/test_history_bindings.py::test_the_member_set_is_the_frozen_inventory fails on main too, and #21790 fixes it. This diff touches only the ops-mission-control ledger, its tests and the hygiene SOP. I will rebase onto main once #21790 merges.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97]

outcome: rebutted (no change). before keeps its main meaning, the number of unique entries the pass started from, so before - after is the prune count, as it was.
The SOP's step 1 lists only deduped, decayed, demoted and pruned, never before or after, so no SOP sentence describes before - after. The new SOP text already says deduped counts collapsed duplicate lines.
Adding SOP text about before would document a field the SOP does not ask the agent to read, which is out of this PR's goal.

@bolichen97

bolichen97 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Correction: the main red is issue #21790, and its fix is PR #21792. The other red, test_dashboard_session_meta.py::test_closed_session_answers_its_row_by_slot_key_or_stem on Windows (a PermissionError on a transcript rename), is unrelated to this diff and is recorded as a flake in #21800. I will rebase once #21792 merges, which re-runs every shard.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 18, 2026
… hygiene

The hygiene pass read entries through the reconciling reader, so every id
was already unique and its own dedupe loop could never merge anything:
`deduped` was always 0 and duplicate lines from a git merge stayed on disk.

The strict read now also returns how many duplicate lines it collapsed.
Hygiene reports that as `deduped`, rewrites the file when it is non-zero,
and drops the unreachable merge loop so `_reconcile` stays the only merge.

Closes #21715
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased onto main 943a4e5, which carries the #21792 fix for the history-bindings red (120efb2 -> 955c76f). Same single commit, no code change. CI re-running.

@bolichen97
bolichen97 force-pushed the fix/ledger-hygiene-dedupe-21715 branch from 120efb2 to 955c76f Compare October 18, 2026 07:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 18, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: ledger hygiene now counts duplicate lines collapsed by the read and compacts them, reusing the one reconcile merge; tests added.

@bolichen97
bolichen97 merged commit dae48ce into main Oct 18, 2026
104 checks passed
@bolichen97
bolichen97 deleted the fix/ledger-hygiene-dedupe-21715 branch October 18, 2026 09:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ops-mission-control: ledger hygiene never dedupes, so deduped is always 0

2 participants