Repository navigation
[M11] Engine introspection snapshot and single compaction step (M11.1) - #90
Merged
Merged
Conversation
ledgerlog.introspect.snapshot() describes the log, the active and frozen memtables, every committed SSTable and the current size tiers as plain JSON, with keys and values decoded as UTF-8 using backslashreplace so a snapshot can cross a worker boundary and be read by a person. Memtable entries carry the height of the skip list node holding them, through new public accessors on SkipList and Memtable, so a tool can draw the structure without reaching into private attributes. WalRecord now reports the CRC32 its framing carries. LedgerLog.compact_once() merges one ready tier on the calling thread, which is what the browser demo needs since Pyodide cannot start a thread, and reports the sources, the output and how many records the merge reclaimed. It chooses the output's sequence number so that the age ordering of the table list survives the merge, and declines to merge a tier whose output would have nowhere safe to sit rather than risk shadowing a newer table with older values. Claude-Session: https://claude.ai/code/session_01QWNkuNaLsrxqkKrLQixfS5 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ledgerlog.introspectgives a tool (the M11 web demo first) one read-only, JSON-shaped view of the engine's internals, and gives it a way to run one compaction step on the calling thread.What changed
src/ledgerlog/introspect.py:snapshot(engine)returns plain JSON types describing the WAL (offset, framed length, the CRC32 the framing carries, kind, key, value byte length, expiry), the active and frozen memtables in key order (value, tombstone flag, absolute expiry, skip list tower level), every committed SSTable (sequence, name, format version, record count, size, key range, sparse index entry count, bloom bit count, bits set, hash count) and the compaction plan's tiers and policy.compact_once(engine)is the JSON shape of the engine's one-step compaction. Keys and values are decoded as UTF-8 witherrors="backslashreplace", documented as one-way and for reading, with real byte lengths reported beside the text.LedgerLog.compact_once(): merges the plan's next ready tier withmerge_tierand retires its sources withswap_in_merged_table, under the flush lock, on the caller's thread, and returns aCompactionRun(source sequences, output sequence, record counts in and out, records dropped).LedgerLog.active_memtable,Memtable.entries_with_levels()andSkipList.items_with_levels(): the tower heights and the live memtable's records, so a UI never reads a private attribute.WalRecord.checksum: the CRC32 read out of the record's framing, not recomputed.docs/ROADMAP.md: ticks the M11 introspection bullet.story_contexts.md: the plain-language note for this story.Acceptance criteria
snapshot(engine)returns plain JSON types describing the WAL records, the active and frozen memtable entries in key order with tower level and tombstone flag, every SSTable, and the compaction plan's tierscompact_once(engine)runs at most one tier merge synchronously using the existing compaction code and reports source sequences, output sequence and records dropped, orNonewhen there is nothing to mergetests/test_introspect.pyasserts the snapshot against the engine's state after puts, deletes, a flush and a compaction, and thatjson.loads(json.dumps(snapshot(engine)))round-trips unchangedsnapshotandcompact_onceon aflush_in_background=Falseengine withthreading.Thread.startmonkeypatched to raiseDecisions recorded
Nobody was available to ask, so two conservative calls:
sstablesis age, and age decides which copy of a key wins, so a merged table has to sit above every table the merge read and below every table it did not. Size tiers group by size, not by age, so that slot is not always free: a table outside the tier can sit inside the tier's sequence range, and the gap above the tier's newest member can be empty.compact_onceworks the slot out before it starts (_merged_sequence_for), tries each ready tier in the plan's order, and returnsNoneif none of them can be merged safely rather than publishing a table that would shadow newer values with older ones. A tier skipped this way is usually mergeable after the next compaction frees sequence numbers. Lifting the restriction properly needs a per-table sequence token in the footer, which is an on-disk format change this story rules out.getstarting after the swap never touches them; agetalready walking an older snapshot of the table list can raise instead of answering. That is stated in the docstring, with why raising beats a read that silently skips a table it cannot open.What ran locally
python -m pytest: 1237 passed (43 new), Python 3.11 and 3.13 both installed here, run on 3.13ruff check src testsandruff format --check src tests: cleanweb/does not exist yet (that is M11.2), so the CIwebjob is a no-op for this branchCloses #79
Generated by Claude Code