Repository navigation
feat: keyed upsert for large loads - #33
Merged
Merged
Conversation
create_note listed the note directory twice per note and checked prefixes with a linear search, so a loop of creates grew faster than the square of its size. A batch now lists the store once and claims each id from an in-memory set. create_note runs as a one-note batch. Repo::keyed_batch opens a batch keyed on one frontmatter field, such as url. KeyedBatch::upsert creates the note for a new record, updates the note for a changed record, and writes nothing for an unchanged record. A loaded note carries a record mapping with separate digests for content and metadata. A metadata change keeps the body and its review stamps. A body changed on disk is not overwritten: the upsert reports a conflict until the body equals the source text or the mapping is removed. A note without the mapping is never taken over. The source removes only the fields, tags and links that it supplied. Two records with one title get distinct slugs: the bare slug when it is free, otherwise a suffix from the key value's digest. A new slug is cut to 100 bytes at a word.
A change to only the title, the tags, the links, or the provenance spec now has its own test, so a field dropped from a digest fails the suite. A record that was updated reruns as unchanged, in the same batch and in a new one. A standing conflict leaves the file's mtime as it was.
A record revised twice updates both times, and a body taken from the source accepts the next revision without a conflict. The new spec is written when a person takes the source body. Tags, links and fields that the source added in one run go when it drops them in the next. Field order does not change a record, and an update moves the updated timestamp.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
create_notelisted the note directory twice for each note and checked prefixes with a linear search, so a loop of creates grew faster than the square of its size.Repo::keyed_batch(key)returnsKeyedBatch. It lists the store once, reads each note once, and claims new IDs from an in-memory set.create_notenow lists once per call.KeyedBatch::upsert(NoteRecord)matches frontmatter key values. It returnsUpserted::Created(id),Updated(id),Unchanged(id), orConflict { id, reason }.ConflictReasonisNotLoadedorBodyChanged.NoteRecordcontains title, body, provenance, tags, links, and fields. Fields are a YAML mapping and must contain the selected key with a string value.Stored notes
A frontmatter
recordmapping tracks digests and source-supplied metadata.An unchanged record writes nothing. Metadata changes keep the body, provenance spec, and stamps. Content changes replace an untouched body and its spec and remove stamps.
A body changed on disk is never overwritten. The upsert reports a conflict and still applies any metadata change. While the conflict stands, a run with no metadata change writes nothing.
To resolve
BodyChanged, set the body to the new source text. The next upsert accepts it and keeps its stamp. Alternatively, removerecordto keep the hand text. A matching note withoutrecordreturnsNotLoaded.The source removes only fields, tags, and links it supplied before. Personal additions stay.
Caller requirements
Use one writer per run. Each write replaces the file at its path. Normalize key values first. Values match byte for byte. Deleted loaded notes return on the next run.
DuplicateKeyfails only that record. An unparseable note or malformedrecordmapping failskeyed_batchwithUnreadableNote { id, reason }.New IDs use the bare slug when free and a key hash suffix on collision. New slugs stop at 100 bytes at a word.