Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ Please choose versions by [Semantic Versioning](http://semver.org/).
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.

## Unreleased

- feat: add `pkg/index`, a shared metadata-only page index (frontmatter, name, path, mtime and pre-parsed Open Questions per page, never the body) with the ported stat-diff refresh contract, lazy on-disk body reads and an optional bolt cache (store format version 2)

## v0.196.0

- fix: `goal set` now writes the flag as an unquoted YAML boolean and records `flag_set_by` / `flag_set_at` in the same write via `--by`, matching `task set`
Expand Down
151 changes: 151 additions & 0 deletions docs/page-index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# Page index

`pkg/index` keeps the metadata of each vault folder's pages in memory so a list
read can be answered without opening a vault file. It implements vault-cli's
`storage.PageStorage`, so it plugs in exactly where vault-cli reads a folder:
filtering, sorting and blocked-state logic stay in vault-cli's list operation
(`ops.NewListOperation(idx)` works unchanged). Single files are read through the
`PageReader` seam, which delegates to vault-cli's own
`storage.PageStorage.ReadPage`, so one parse path and one symlink-out-of-vault
guard serve both the index and vault-cli's own folder listing.

One immutable snapshot is held per key — a vault root plus a vault-relative
pages dir. A published snapshot is never mutated; concurrent readers share the
same `*Entry` and `*domain.Page` pointers. **Published entries and pages are
read-only for every consumer**: a consumer that needs to change one copies it
first, because mutating a shared page corrupts every other reader's view.

The package imports no glog and no prometheus. Warnings go to `log/slog` by
default (`SetWarnf` redirects them per index), and counters go through the
injected `Metrics` interface (`FileRead(reason)`,
`ChangedPagesUnbounded(reason)`); a nil `Metrics` records nothing.

## Metadata-only records

- Every indexed page keeps its full frontmatter map (every key), its name, its
file path and its modification date exactly as vault-cli's folder listing
produces them — and an **empty** `Content`.
- The body is read by the `PageReader` and dropped before the page is
published. No body is held in memory and none is written to the store.
- `ListPages` over a folder equals `storage.NewPageStorage(nil).ListPages` over
the same folder with only `Content` zeroed: same set, same order, same
frontmatter, path and modification time — including files without
frontmatter, invalid YAML, non-`.md` files, subdirectories, symlinks out of
the vault, broken symlinks, in-vault symlinks and non-ASCII names.

## Entries and open questions

- `ListEntries` serves the same snapshot as `ListPages` — same order, same page
pointers — where each `Entry` adds the page's `OpenQuestions` as
`[]domain.OpenQuestion{Index, Text}`, projected from
`storage.ParseOpenQuestions` (`Text` is the item's question, without any
recorded answer).
- `OpenQuestions` is never nil: a page without an `Open Questions` section has
an empty slice.
- Every retained `Text` is a copy (`strings.Clone`) that shares no memory with
the body, so a retained question never pins a body in memory.
- An open-question parse error fails that read like any read error.

## Lazy body reads

- `ReadPage(ctx, vaultPath, pagesDir, name)` takes the bare base name (without
`.md`) and always reads the file from disk through the `PageReader`,
returning the full page including `Content`.
- It never caches the body, never mutates or republishes the snapshot and never
moves the key's revision. A name that is empty, contains a path separator or
`..` is rejected without a read.

## Staleness bounds

- A warm read never touches disk. `ListPages` / `ListEntries` return the
published snapshot without any filesystem access.
- `Rescan` refreshes every known key once per `RescanInterval` (50 s) plus the
1 s poll granularity plus one stat-diff — under 60 s. Each storage call is
bounded by the rebuild timeout (2 min), so the worst case is the rescan
interval plus that timeout. The bound holds for every change to a file's
presence, size, modification time or status-change time.
- The first read after a mark resolves it before serving: a per-file mark
re-reads exactly the marked files, a folder-level mark runs one stat-diff, and
every concurrent reader waits on that same work. Marking a key the index has
never seen is a no-op: its first read builds it anyway.
- Concurrent cold reads of one key share a single build.
- An event-driven (`RefreshFile`) or rescan-driven (`Refresh`) read never
blocks `ListPages`: a reader is served from the current snapshot.
- `ForceReload` is the only path that ignores fingerprints: the next read of
every key re-reads every file.
- A failed listing keeps serving the previous snapshot and logs the error with
the key; the mark stays pending, so the next read retries. A failed first
build returns an error.
- With a store, a key is hydrated from it and resolves exactly one stat-diff
before it is served; the stored records are never served as written.

## Incremental updates

- `RefreshFile(ctx, key, filename)` re-reads only that file and lists nothing.
The file's current on-disk state decides the outcome: present and readable
replaces or inserts its entry at its filename-ordered position, absent or
unreadable removes it. A filename that is empty, contains a path separator or
`..`, or does not end in `.md` is rejected and nothing is read.
- Every file read records a fingerprint — size, modification time and
status-change time — from a stat that follows symlinks, including files that
end up excluded.
- `Refresh` and the rescan are a **stat-diff**: list the folder, compare each
entry's fingerprint with the one recorded at its last read (inequality, not
"newer"), and re-read only what differs or is new. A body-only edit changes
the fingerprint, so it is re-read and republished with a new page pointer and
named by `ChangedPagesSince`.
- An unchanged stat-diff reads nothing, publishes nothing, keeps the snapshot
and the recorded fingerprint set in place, and allocates nothing beyond the
one folder listing it performs. An excluded file is warned about once per
fingerprint change.
- `MarkDirty(keys...)` marks a folder stale; the next read resolves it by one
stat-diff. `MarkFileDirty(key, id)` marks one file stale only when the id is
exact (`[[`/`]]` stripped, no separator or `..`, a byte-equal page name in the
snapshot, `<name>.md` present); anything else widens to a folder-level mark.
- `Revision(key)` advances on every publication and mark. `ChangedPagesSince`
returns the `.md` filenames changed after a revision, with `ok == false` for
an unknown key, a revision ahead of the key, a revision outside the retained
change log (64 advances), a folder-level mark, a forced reload, or the key's
first publication.

## Key derivation

A key is `NewKey(vault path, folder)`, which applies `filepath.Clean` to both
parts so equivalent paths produce identical keys. A consumer derives the read
side and the event side from the same vault config entry; a mismatch silently
disables event refresh and leaves the folder on its rescan cadence.

## Store format version 2

An optional bolt store (`NewPageIndexWithStore`) persists the index as a cache,
so a restart does not re-parse unchanged files. It is never a source of truth.

- **Record.** One record per indexed file: filename, has-page flag, full
frontmatter, file metadata, Open Questions and fingerprint. **No body.** An
excluded file stores its fingerprint and no page.
- **Frozen names**, byte-identical to vault-ui's v1 store: meta bucket
`vault-ui-page-index-meta`, identity key `identity`, identity JSON fields
`StoreFormat` / `ParserVersion`, entry bucket name
`vault-ui-page-index\x00<VaultPath>\x00<PagesDir>`. A v1 file is therefore
recognised and discarded as a format mismatch, entry buckets deleted.
- **Identity.** `StoreFormat` is `2`; `ParserVersion` is the vault-cli module
version from build info (the dependency entry, or the main module when
vault-cli is itself main, else `(unknown)`).
- **Discard reasons.** On open, a store whose identity is missing or empty,
damaged, of another store format, or of another parser version is discarded:
its entry buckets are deleted and one warning names the path and the reason —
`store format mismatch (stored N, current 2)`, `parser version mismatch`,
`store file missing or empty`, or `damaged`. Format is checked before parser
version. Opening waits at most 5 s for the file lock; any open or reconcile
failure yields a discarded store that serves nothing and never fails
construction.
- **Writes.** Each publication writes exactly its changed and vanished entries
in one transaction, outside the index mutex. A failed write leaves the
previous content intact and logs one warning.
- **Location.** `DefaultStorePath(app)` returns
`<user cache dir>/<app>/page-index.bolt`; `NewBoltStore` creates the
directory with mode 0700. The store never lives inside a vault.
- **Codec.** Frontmatter is encoded through `FrontmatterMap.RawMap()` and pages
are rebuilt with `domain.NewPage`: a JSON codec on `domain.Page` would drop
every frontmatter key. Decoding never preallocates from stored counts, and
any decode failure discards the store.
6 changes: 4 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,23 @@ module github.com/bborbe/vault-cli
go 1.27.4

require (
github.com/bborbe/boltkv v1.17.0
github.com/bborbe/collection v1.21.7
github.com/bborbe/cqrs v0.6.16
github.com/bborbe/errors v1.6.7
github.com/bborbe/kafka v1.29.14
github.com/bborbe/kv v1.21.30
github.com/bborbe/log v1.7.9
github.com/bborbe/notification v0.7.13
github.com/bborbe/run v1.11.12
github.com/bborbe/time v1.27.28
github.com/bborbe/validation v1.5.10
github.com/fsnotify/fsnotify v1.10.1
github.com/google/uuid v1.6.0
github.com/onsi/ginkgo/v2 v2.34.0
github.com/onsi/gomega v1.44.0
github.com/spf13/cobra v1.10.2
go.etcd.io/bbolt v1.5.0
golang.org/x/sys v0.50.0
golang.org/x/term v0.48.0
gopkg.in/yaml.v3 v3.0.1
Expand All @@ -26,10 +30,8 @@ require (
github.com/Masterminds/semver/v3 v3.5.0 // indirect
github.com/bborbe/http v1.26.34 // indirect
github.com/bborbe/k8s v1.15.1 // indirect
github.com/bborbe/kv v1.21.30 // indirect
github.com/bborbe/math v1.4.20 // indirect
github.com/bborbe/parse v1.12.7 // indirect
github.com/bborbe/run v1.11.12 // indirect
github.com/bborbe/sentry v1.10.8 // indirect
github.com/bborbe/strimzi v1.8.25 // indirect
github.com/beorn7/perks v1.0.1 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ github.com/IBM/sarama v1.61.1 h1:I59MWPHQUWqJNdRpsDUcbeCriog8SxjQaPfHNWxidEg=
github.com/IBM/sarama v1.61.1/go.mod h1:dITlGHIiCQL/maGtBfDHNMDvyWgC9Ww//8pmlsU3RUs=
github.com/Masterminds/semver/v3 v3.5.0 h1:kQceYJfbupGfZOKZQg0kou0DgAKhzDg2NZPAwZ/2OOE=
github.com/Masterminds/semver/v3 v3.5.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/bborbe/boltkv v1.16.8 h1:L+YyepgvRjYKkDg9CSQJvOweXdCcKiqsl40g+CHXWJQ=
github.com/bborbe/boltkv v1.16.8/go.mod h1:IjPczouQRUnWcGbIBXQMyXoyDic0qgLL9HmFQxBLkGo=
github.com/bborbe/boltkv v1.17.0 h1:IRfO5oBznAuGzVYz7HrJjPPKWuUj4Df0yLcI9/ZL+jw=
github.com/bborbe/boltkv v1.17.0/go.mod h1:RyD+DM/KROJrbEseBRUpIguQV4UepTwNFN+0rVHko9c=
github.com/bborbe/collection v1.21.7 h1:VNbMotiaTZy4b4Mk9gZ5UrDXeYiywJGP95Jhp+dqXZ8=
github.com/bborbe/collection v1.21.7/go.mod h1:nMOfUE29djizqSTytT+OGvl41NvFfU1HpIjRnpl8Zco=
github.com/bborbe/cqrs v0.6.16 h1:wwEnswzU6jbfJMPp1qlPnkvhxusVV92zEkQiE/MmvS8=
Expand Down
119 changes: 119 additions & 0 deletions mocks/index-directory-lister.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading