Skip to content

feat: S3 sync, local mirror, post parsing and render - #2

Merged
Mimic890 merged 6 commits into
mainfrom
feat/s3-sync
Nov 3, 2026
Merged

Mimic890 merged 6 commits into
mainfrom
feat/s3-sync

Conversation

@Mimic890

@Mimic890 Mimic890 commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Stage 2: S3 sync.

What

  • internal/mirror: checks bucket keys (rejects absolute paths, \, NUL, control characters and empty segments; ignores . and _ paths) and mirrors them into VAULT_DIR through os.Root. Writes are atomic (temp file + rename). Files are 0644, dirs 0755, and the state file VAULT_DIR/.hyperhack.json is 0600.
  • internal/content: frontmatter parser (all SPEC fields, both Obsidian date formats, size checked before YAML) and Markdown render (goldmark + GFM + chroma classes, then bluemonday). assets/… maps to /assets/…, external links get rel="noopener noreferrer", and Russian file names and headings are transliterated. Builds a post index with slug-conflict handling.
  • internal/syncer: minio-go client (path-style, HTTPS, If-Match on GET). Polls every POLL_INTERVAL and fetches only new or changed objects. Applies MAX_POST_BYTES, MAX_ASSET_BYTES and MAX_OBJECTS, removes deleted objects, and re-renders only changed posts. S3 errors get 3 attempts with WARN logs, then ERROR.
  • POST /reindex: REINDEX_ALLOW (client IP via TRUSTED_PROXIES), then a constant-time Bearer token check. It waits for the sync and returns 204, 401, 403 or 503.
  • /healthz is ready once an index exists. With S3 down at startup the service serves from the mirror; with no mirror, a failed first sync exits 1.
  • S3_ENDPOINT with a path is rejected.
  • SPEC + SPEC.ru: stage 2 decisions and a Decision log entry. A separate commit repairs SPEC.ru.md, which had dropped words and mojibake in about 20 lines.

Tests

  • Unit tests for every package. Fuzz tests: FuzzParse (frontmatter + render, which also checks the output has no unsafe HTML) and FuzzClassify.
  • An XSS corpus checked with an HTML tokenizer, path traversal keys, a symlink escape from the mirror, and the /reindex CIDR, XFF and token cases.
  • A minio adapter test against a fake S3 TLS server.
  • Smoke test against Garage v2.1.0 behind TLS, with the container run as in compose (read-only, uid 10001, no caps):
    • first sync;
    • /reindex returns 401 and 204, and an update and a delete are picked up;
    • S3 down with a mirror: ready, ERROR after retries, recovers when S3 is back;
    • S3 down without a mirror: exit 1;
    • wrong secret: neither the key nor the secret appears in the logs;
    • SIGTERM exits 0.

Not done here

  • RATE_LIMIT_REINDEX: comes with the rate limiter in stage 5.
  • Reading time, image width/height and loading=lazy: stages 3 and 4.

Several lines had dropped words or mojibake, so the Russian copy no
longer matched SPEC.md. Restored them from the English text.
Classify rejects unsafe keys and ignores . and _ paths. Writes are
atomic (temp file + rename), files 0644, the state file 0600.
goldmark with GFM and chroma classes, then bluemonday. Assets links
map to /assets/, external links get rel=noopener noreferrer, heading
ids and file-name slugs are transliterated.
minio-go lists and fetches with If-Match, applies size and object
limits, removes deleted objects and rebuilds the post index from disk.
S3 errors retry twice; with no local mirror a failed first sync is fatal.
/reindex checks REINDEX_ALLOW (client IP via TRUSTED_PROXIES) and the
Bearer token in constant time. /healthz is ready once an index exists.
@Mimic890
Mimic890 merged commit 5f83c84 into main Nov 3, 2026
4 checks passed
@Mimic890
Mimic890 deleted the feat/s3-sync branch November 3, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant