Skip to content

[M11] Engine introspection snapshot and single compaction step (M11.1) - #90

Merged
ssoufii merged 1 commit into
mainfrom
auto/m11.1-engine-introspection-snapshot
Nov 16, 2026
Merged

ssoufii merged 1 commit into
mainfrom
auto/m11.1-engine-introspection-snapshot

Conversation

@ssoufii

@ssoufii ssoufii commented Nov 16, 2026

Copy link
Copy Markdown
Owner

ledgerlog.introspect gives a tool (the M11 web demo first) one read-only, JSON-shaped view of the engine's internals, and gives it a way to run one compaction step on the calling thread.

What changed

  • src/ledgerlog/introspect.py: snapshot(engine) returns plain JSON types describing the WAL (offset, framed length, the CRC32 the framing carries, kind, key, value byte length, expiry), the active and frozen memtables in key order (value, tombstone flag, absolute expiry, skip list tower level), every committed SSTable (sequence, name, format version, record count, size, key range, sparse index entry count, bloom bit count, bits set, hash count) and the compaction plan's tiers and policy. compact_once(engine) is the JSON shape of the engine's one-step compaction. Keys and values are decoded as UTF-8 with errors="backslashreplace", documented as one-way and for reading, with real byte lengths reported beside the text.
  • LedgerLog.compact_once(): merges the plan's next ready tier with merge_tier and retires its sources with swap_in_merged_table, under the flush lock, on the caller's thread, and returns a CompactionRun (source sequences, output sequence, record counts in and out, records dropped).
  • LedgerLog.active_memtable, Memtable.entries_with_levels() and SkipList.items_with_levels(): the tower heights and the live memtable's records, so a UI never reads a private attribute.
  • WalRecord.checksum: the CRC32 read out of the record's framing, not recomputed.
  • docs/ROADMAP.md: ticks the M11 introspection bullet. story_contexts.md: the plain-language note for this story.

Acceptance criteria

  • snapshot(engine) returns plain JSON types describing the WAL records, the active and frozen memtable entries in key order with tower level and tombstone flag, every SSTable, and the compaction plan's tiers
  • compact_once(engine) runs at most one tier merge synchronously using the existing compaction code and reports source sequences, output sequence and records dropped, or None when there is nothing to merge
  • tests/test_introspect.py asserts the snapshot against the engine's state after puts, deletes, a flush and a compaction, and that json.loads(json.dumps(snapshot(engine))) round-trips unchanged
  • A test drives put, delete, flush, snapshot and compact_once on a flush_in_background=False engine with threading.Thread.start monkeypatched to raise
  • Two snapshots in a row are equal, every probed key reads the same before and after a snapshot, and the directory is byte for byte unchanged by one

Decisions recorded

Nobody was available to ask, so two conservative calls:

  1. Where a merged table lands. Position in sstables is age, and age decides which copy of a key wins, so a merged table has to sit above every table the merge read and below every table it did not. Size tiers group by size, not by age, so that slot is not always free: a table outside the tier can sit inside the tier's sequence range, and the gap above the tier's newest member can be empty. compact_once works the slot out before it starts (_merged_sequence_for), tries each ready tier in the plan's order, and returns None if none of them can be merged safely rather than publishing a table that would shadow newer values with older ones. A tier skipped this way is usually mergeable after the next compaction frees sequence numbers. Lifting the restriction properly needs a per-table sequence token in the footer, which is an on-disk format change this story rules out.
  2. Retired handles. The merged table is published before the sources' handles are closed, so a get starting after the swap never touches them; a get already walking an older snapshot of the table list can raise instead of answering. That is stated in the docstring, with why raising beats a read that silently skips a table it cannot open.

What ran locally

  • python -m pytest: 1237 passed (43 new), Python 3.11 and 3.13 both installed here, run on 3.13
  • ruff check src tests and ruff format --check src tests: clean
  • No Playwright: web/ does not exist yet (that is M11.2), so the CI web job is a no-op for this branch

Closes #79


Generated by Claude Code

ledgerlog.introspect.snapshot() describes the log, the active and frozen
memtables, every committed SSTable and the current size tiers as plain JSON,
with keys and values decoded as UTF-8 using backslashreplace so a snapshot can
cross a worker boundary and be read by a person. Memtable entries carry the
height of the skip list node holding them, through new public accessors on
SkipList and Memtable, so a tool can draw the structure without reaching into
private attributes. WalRecord now reports the CRC32 its framing carries.

LedgerLog.compact_once() merges one ready tier on the calling thread, which is
what the browser demo needs since Pyodide cannot start a thread, and reports
the sources, the output and how many records the merge reclaimed. It chooses
the output's sequence number so that the age ordering of the table list
survives the merge, and declines to merge a tier whose output would have
nowhere safe to sit rather than risk shadowing a newer table with older values.

Claude-Session: https://claude.ai/code/session_01QWNkuNaLsrxqkKrLQixfS5

Co-Authored-By: Claude <noreply@anthropic.com>
@ssoufii
ssoufii merged commit 09a47eb into main Nov 16, 2026
9 checks passed
@ssoufii
ssoufii deleted the auto/m11.1-engine-introspection-snapshot branch November 16, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[M11] Engine introspection snapshot and single compaction step (M11.1)

2 participants