Repository navigation
feat: S3 sync, local mirror, post parsing and render - #2
Merged
Merged
Conversation
Several lines had dropped words or mojibake, so the Russian copy no longer matched SPEC.md. Restored them from the English text.
Classify rejects unsafe keys and ignores . and _ paths. Writes are atomic (temp file + rename), files 0644, the state file 0600.
goldmark with GFM and chroma classes, then bluemonday. Assets links map to /assets/, external links get rel=noopener noreferrer, heading ids and file-name slugs are transliterated.
minio-go lists and fetches with If-Match, applies size and object limits, removes deleted objects and rebuilds the post index from disk. S3 errors retry twice; with no local mirror a failed first sync is fatal.
/reindex checks REINDEX_ALLOW (client IP via TRUSTED_PROXIES) and the Bearer token in constant time. /healthz is ready once an index exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 2: S3 sync.
What
internal/mirror: checks bucket keys (rejects absolute paths,\, NUL, control characters and empty segments; ignores.and_paths) and mirrors them intoVAULT_DIRthroughos.Root. Writes are atomic (temp file + rename). Files are 0644, dirs 0755, and the state fileVAULT_DIR/.hyperhack.jsonis 0600.internal/content: frontmatter parser (all SPEC fields, both Obsidian date formats, size checked before YAML) and Markdown render (goldmark + GFM + chroma classes, then bluemonday).assets/…maps to/assets/…, external links getrel="noopener noreferrer", and Russian file names and headings are transliterated. Builds a post index with slug-conflict handling.internal/syncer: minio-go client (path-style, HTTPS,If-Matchon GET). Polls everyPOLL_INTERVALand fetches only new or changed objects. AppliesMAX_POST_BYTES,MAX_ASSET_BYTESandMAX_OBJECTS, removes deleted objects, and re-renders only changed posts. S3 errors get 3 attempts with WARN logs, then ERROR.POST /reindex:REINDEX_ALLOW(client IP viaTRUSTED_PROXIES), then a constant-time Bearer token check. It waits for the sync and returns204,401,403or503./healthzis ready once an index exists. With S3 down at startup the service serves from the mirror; with no mirror, a failed first sync exits 1.S3_ENDPOINTwith a path is rejected.Tests
FuzzParse(frontmatter + render, which also checks the output has no unsafe HTML) andFuzzClassify./reindexCIDR, XFF and token cases./reindexreturns 401 and 204, and an update and a delete are picked up;Not done here
RATE_LIMIT_REINDEX: comes with the rate limiter in stage 5.loading=lazy: stages 3 and 4.