From af4c233cf71e7e0c605a4f45e55467ace8c213e7 Mon Sep 17 00:00:00 2001 From: Mykhailo Chalyi Date: Mon, 26 Oct 2026 01:54:14 +0000 Subject: [PATCH] chore(knowledge): adopt OKF knowledge bundle and bashkit repo conventions Move specs/ to an OKF v0.2 knowledge/ bundle (foundations, formats, security, runtimes, operations) with frontmatter, indexes, log and a maintenance contract. Port check_okf.py from bashkit and run it plus okf-lint in CI and just check-okf. Rewrite AGENTS.md in bashkit style with a knowledge map and maintenance invocation contract; add the maintain skill and give all skills frontmatter. Audit workflows: top-level read-only permissions and no persisted checkout credentials outside the release tag push. Run examples in CI. Correct README, SECURITY, getting-started and binding docs against the code. --- .agents/skills/maintain/SKILL.md | 41 +++ .agents/skills/process-issues/SKILL.md | 10 +- .agents/skills/ship/SKILL.md | 172 ++++++---- .claude/commands/maintain.md | 55 +-- .github/pull_request_template.md | 20 ++ .github/workflows/ci.yml | 25 ++ .github/workflows/coverage.yml | 5 + .github/workflows/js.yml | 5 + .github/workflows/nightly.yml | 5 + .github/workflows/publish-js.yml | 5 + .github/workflows/publish-python.yml | 2 + .github/workflows/python.yml | 5 + AGENTS.md | 142 ++++++-- CHANGELOG.md | 7 + CONTRIBUTING.md | 17 +- Cargo.lock | 26 +- README.md | 95 +++++- SECURITY.md | 13 +- crates/markkit-eval/Cargo.toml | 2 +- crates/markkit/src/docx.rs | 2 +- docs/getting-started.md | 23 +- justfile | 52 ++- .../formats}/format-support.md | 20 +- .../formats}/html-extraction.md | 28 +- knowledge/formats/index.md | 4 + .../foundations}/architecture.md | 19 +- knowledge/foundations/index.md | 4 + .../ir-spec.md => knowledge/foundations/ir.md | 16 + knowledge/index.md | 16 + knowledge/knowledge-contract.md | 69 ++++ knowledge/log.md | 7 + knowledge/operations/documentation.md | 41 +++ {specs => knowledge/operations}/eval.md | 20 ++ knowledge/operations/examples.md | 61 ++++ knowledge/operations/index.md | 8 + knowledge/operations/maintenance.md | 140 ++++++++ .../operations}/release-process.md | 21 +- knowledge/operations/testing.md | 60 ++++ knowledge/runtimes/index.md | 4 + knowledge/runtimes/node-package.md | 61 ++++ knowledge/runtimes/python-package.md | 63 ++++ knowledge/security/index.md | 4 + .../security/security-model.md | 21 +- {specs => knowledge/security}/threat-model.md | 32 +- scripts/check_okf.py | 319 ++++++++++++++++++ scripts/tests/test_agent_skills.py | 35 ++ scripts/tests/test_check_okf.py | 247 ++++++++++++++ specs/documentation.md | 26 -- specs/examples-spec.md | 67 ---- specs/js-package.md | 33 -- specs/maintenance.md | 35 -- specs/python-package.md | 34 -- specs/testing.md | 43 --- 53 files changed, 1840 insertions(+), 447 deletions(-) create mode 100644 .agents/skills/maintain/SKILL.md create mode 100644 .github/pull_request_template.md rename {specs => knowledge/formats}/format-support.md (90%) rename {specs => knowledge/formats}/html-extraction.md (89%) create mode 100644 knowledge/formats/index.md rename {specs => knowledge/foundations}/architecture.md (68%) create mode 100644 knowledge/foundations/index.md rename specs/ir-spec.md => knowledge/foundations/ir.md (90%) create mode 100644 knowledge/index.md create mode 100644 knowledge/knowledge-contract.md create mode 100644 knowledge/log.md create mode 100644 knowledge/operations/documentation.md rename {specs => knowledge/operations}/eval.md (54%) create mode 100644 knowledge/operations/examples.md create mode 100644 knowledge/operations/index.md create mode 100644 knowledge/operations/maintenance.md rename {specs => knowledge/operations}/release-process.md (74%) create mode 100644 knowledge/operations/testing.md create mode 100644 knowledge/runtimes/index.md create mode 100644 knowledge/runtimes/node-package.md create mode 100644 knowledge/runtimes/python-package.md create mode 100644 knowledge/security/index.md rename specs/security.md => knowledge/security/security-model.md (69%) rename {specs => knowledge/security}/threat-model.md (90%) create mode 100644 scripts/check_okf.py create mode 100644 scripts/tests/test_agent_skills.py create mode 100644 scripts/tests/test_check_okf.py delete mode 100644 specs/documentation.md delete mode 100644 specs/examples-spec.md delete mode 100644 specs/js-package.md delete mode 100644 specs/maintenance.md delete mode 100644 specs/python-package.md delete mode 100644 specs/testing.md diff --git a/.agents/skills/maintain/SKILL.md b/.agents/skills/maintain/SKILL.md new file mode 100644 index 0000000..4f5015b --- /dev/null +++ b/.agents/skills/maintain/SKILL.md @@ -0,0 +1,41 @@ +--- +name: maintain +description: Analyze, fix, validate, and ship Markkit maintenance through green CI and merge. Trigger on run maintenance, maintain, maintainace, or maintaiance; use analysis-only mode only when explicitly requested. +user_invocable: true +metadata: + internal: true +--- + +# Markkit maintenance + +A request to run maintenance authorizes the full outcome: analyze the repository, +fix findings, validate, push, open a PR, resolve review/CI failures, and squash-merge +when every required check is green. A local commit or a list of follow-up issues +is not completion. Explicit analysis-only requests omit edits and shipping. + +Read `knowledge/operations/maintenance.md` and execute its checklist, then use the +`ship` skill. Sync from latest `origin/main` before editing. Split independent +fixes into separate small PRs (one for structure/docs, one per behavior fix). + +Keep findings in the active pass and fix root causes. A large diff, missing audit, +failed local check, or long build is work to complete, not a reason to defer. +Never weaken tests, resource limits, or `deny.toml` policy to ship an upgrade. +If the newest upstream version breaks a security contract, establish that, keep +the newest safe version with a tested and documented pin, and finish the pass. + +Audit CI definitions, not only results: run the Workflow credential audit in +`knowledge/operations/maintenance.md` against `.github/workflows/` (top-level +`permissions`, `persist-credentials: false`, secrets via step `env:` only, no +`pull_request_target`/`workflow_run`/`issue_comment`, no `${{ github.event.* }}` +inside `run:`). A finding there is a fix in this pass. + +Smoke test real-world shaped input, not only fixtures: run the CLI on indented, +multi-block HTML and on each office format, and read the output. + +Only stop for a missing permission, unavailable service, or user decision that +cannot be resolved within the task. Report evidence and continue independent work. +Never merge red CI or claim success while a required check is unfinished. + +Finish by appending a factual entry to the Pass record in +`knowledge/operations/maintenance.md` and a `knowledge/log.md` entry. The final +response names the merged PRs, key fixes, verification, and accepted constraints. diff --git a/.agents/skills/process-issues/SKILL.md b/.agents/skills/process-issues/SKILL.md index b3d0af7..9038091 100644 --- a/.agents/skills/process-issues/SKILL.md +++ b/.agents/skills/process-issues/SKILL.md @@ -1,3 +1,11 @@ +--- +name: process-issues +description: Resolve open GitHub issues for Markkit, one shipped PR per issue. Trigger when user says process issues, work through issues, resolve issues, or handle open issues. +user_invocable: true +metadata: + internal: true +--- + Process open GitHub issues and resolve them via merged pull requests. ## Qualifying Issues @@ -29,7 +37,7 @@ One issue = one PR. No bundling. No skipping. ### 4. Ship - Use the `/ship` skill for: - - Spec updates + - Knowledge updates (`knowledge/`) - Quality gates - PR creation - CI verification diff --git a/.agents/skills/ship/SKILL.md b/.agents/skills/ship/SKILL.md index f0be729..77cffb5 100644 --- a/.agents/skills/ship/SKILL.md +++ b/.agents/skills/ship/SKILL.md @@ -1,61 +1,111 @@ -Run the full ship flow: verify quality, ensure test coverage, update -artifacts, smoke test, then push, create PR, address review comments, -and merge only when CI is green, all threads resolved, and user confirms. - -## Phases - -### 1. Pre-flight -- Verify on a feature branch (not main) -- Check for uncommitted changes -- Fetch latest main and rebase if needed - -### 2. Test Coverage -- Identify changed code paths -- Ensure tests exist for: - - Positive cases (happy path) - - Negative cases (errors, edge cases) - - Security scenarios (malicious input) -- Add missing tests - -### 3. Artifact Updates -- Update specs if behavior changed -- Update CHANGELOG.md for user-facing changes -- Update documentation for API changes - -### 4. Simplification -- Remove dead code introduced by changes -- Reduce duplication -- Simplify where possible without changing behavior - -### 5. Security Review -- Check for injection risks in new parsing code -- Verify resource limits are enforced -- Review input validation - -### 6. Smoke Test -- Run `just test` for full test suite -- Run `just lint` for code quality -- Build CLI and verify basic extraction works - -### 7. Quality Gates -- `cargo fmt --all -- --check` -- `cargo clippy --workspace -- -D warnings` -- `cargo deny check` - -### 8. Push & PR -- Push branch to remote -- Create or update PR with summary -- Wait for CI to pass - -### 9. Review & Merge -- Check for unresolved review comments — address or reply to each -- Ensure all review threads are resolved -- Confirm no changes-requested reviews remain outstanding -- Ask the user whether to merge now or wait for reviewer feedback -- Only merge when: CI green, no unresolved threads, no pending change requests - -**IMPORTANT**: Phases 2-5 (tests, artifacts, simplification, security review, -smoke testing) are the quality core — do NOT skip them. - -**IMPORTANT**: Never auto-merge without confirming with the user. PRs should -have time for human review before merging. +--- +name: ship +description: Run the full Markkit ship flow — verify quality, ensure test coverage, update artifacts, smoke test, push, create PR, and merge when CI is green. Trigger when user says ship, ship it, fix and ship, asks to push and merge a branch, or runs maintenance. +user_invocable: true +metadata: + internal: true +--- + +Run the full ship flow: verify quality, ensure test coverage, update artifacts, +smoke test, then push, create PR, and merge when CI is green. + +Implements the Pre-PR Checklist in `AGENTS.md`. Execute ALL phases, not just push/merge. + +## Arguments + +- `$ARGUMENTS` - Optional: what is being shipped (PR title/body context, scopes checks) + +## Phase 1: Pre-flight + +1. Confirm not on `main` +2. Detached `HEAD` with task changes: create a branch first +3. Uncommitted changes belong to the task: keep going. Unrelated: stop and tell the user +4. `git config user.name`/`user.email` are human (see `AGENTS.md` Commit Attribution) + +## Phase 2: Test Coverage + +Review `git diff origin/main...HEAD` and `git log origin/main..HEAD`: + +1. Identify every changed code path +2. `just test`; fix failures +3. Add missing tests: + - **Positive**: happy path, valid input + - **Negative**: invalid input, errors, boundaries, limits + - **Security**: parsing, limits, or URL fetching changes add tests per + `knowledge/security/threat-model.md` and update its Test Coverage map + - **Golden fixtures**: HTML conversion changes add `tests/fixtures/html/.{html,md}` +4. `just test` green + +## Phase 3: Artifact Updates + +Skip what isn't affected. + +1. **Knowledge** (`knowledge/`): update affected concepts in the same change; add a `knowledge/log.md` entry; `just check-okf` +2. **Threat model**: new input surface or limit → new/updated entry, keep `T*`/`S*` ids stable +3. **AGENTS.md**: new commands, knowledge concepts, workflow changes +4. **README.md / docs/**: user-facing behavior, flags, API +5. **Binding READMEs**: exported function changes in `markkit-python` / `markkit-js` +6. **CHANGELOG.md**: Unreleased entry for user-facing changes + +## Phase 3b: Simplification + +Duplication, deep nesting, dead code, unclear names, needless abstraction. If changed, back to Phase 2. + +## Phase 3c: Security Review + +1. Untrusted input validated and bounded before use (size, depth, counts) +2. No unbounded allocation driven by attacker-controlled headers or sizes +3. Errors don't leak raw input or internal paths +4. No code execution paths (scripts, macros, formulas, entities) +5. `unsafe` justified and sound + +Fix, add regression tests, update the threat model. + +## Phase 3d: Design Quality + +Real domain types over stringly-typed interfaces; specific error variants; no +compatibility shims (fix call sites); no premature generics; visibility protects invariants. + +## Phase 4: Smoke Testing + +1. CLI: `just run extract --file ` on realistic input (indented HTML, office docs); read the output +2. Examples: `just examples` +3. Bindings changed: build them (`maturin develop` / `napi build`) and call the changed function + +## Phase 5: Quality Gates + +```bash +git fetch origin main && git rebase origin/main +just pre-pr +``` + +Fix root causes; never bypass hooks or weaken tests/audit policy. + +## Phase 6: Push and PR + +```bash +git push -u origin +``` + +Create or update the PR: + +- **Title**: conventional commit style +- **Body**: `.github/pull_request_template.md`; functional change and impact, not a file walkthrough +- **Evidence**: Before / After (CLI output, test results); say so when nothing observable changed + +## Phase 7: CI and Merge + +- Wait for CI with bounded checks; elapsed time alone is not a blocker +- Failure: read logs, fix root cause, push, wait again +- All required checks green and review threads resolved: squash-merge, confirm it merged +- **NEVER** merge red CI + +## Phase 8: Post-merge + +Report the merged PR URL. + +## Rules + +- Phases 2-4 are the quality core; never skip them +- "Fix and ship": implement first, then ship +- Partial issue coverage: `Part of #N`, not `Closes #N` diff --git a/.claude/commands/maintain.md b/.claude/commands/maintain.md index 40cf3ac..0ffa26d 100644 --- a/.claude/commands/maintain.md +++ b/.claude/commands/maintain.md @@ -1,49 +1,10 @@ -Run pre-release maintenance checks on the markkit codebase. +Analyze, fix, and ship Markkit maintenance. "Run maintenance" (including +"maintainace" and "maintaiance") requests the complete shipped outcome. -Each section below is an outcome to achieve, not a script to follow. -Use whatever tools and approaches make sense to verify and fix. +Read `.agents/skills/maintain/SKILL.md`, execute the checklist in +`knowledge/operations/maintenance.md`, then run `.agents/skills/ship/SKILL.md` +through green CI and squash-merge. `$ARGUMENTS` may scope the maintenance area; +an explicit analysis-only request is the only exception to fixing and shipping. -Fix issues as you find them — don't just report. Use conventional -commit messages for any fixes. - -Sections (run all unless $ARGUMENTS specifies a subset): - -## 1. Dependencies -- Run `cargo update` and verify tests pass -- Check for outdated dependencies -- Review Dependabot PRs if any - -## 2. Security -- Run `cargo audit` for vulnerabilities -- Run `cargo deny check` for license compliance -- Review any new advisories - -## 3. Tests -- Run full test suite: `just test` -- Check coverage isn't regressing -- Identify any `#[ignore]` tests that should be re-enabled - -## 4. Documentation -- Verify README is current -- Check rustdoc builds without warnings -- Ensure specs match implementation - -## 5. Examples -- Verify examples compile and run -- Update examples for any API changes - -## 6. Specs -- Review specs/ for accuracy -- Update implementation status - -## 7. Simplification -- Look for dead code, unused dependencies -- Identify duplication opportunities -- Reduce complexity where possible - -## 8. CI Health -- Check recent GitHub Actions runs -- Verify nightly builds are passing -- Review any flaky tests - -Deliver a summary of findings and actions taken. +Do not replace fixes, audits, or validation with deferral issues or stop at a +local commit. Preserve security contracts, fix CI failures, never merge red checks. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..8036f3c --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,20 @@ +## What changed +Describe the change functionally — what behavior changes and its impact on users +or callers. Lead with outcomes; don't walk through code locations, the diff shows +where and how. Keep any code-level notes short and specific. + +## Why +Problem or motivation. + +## Before / After +Show the effect with evidence. Include before and after whenever behavior changes — +CLI/API output, logs, metrics, or screenshots for UI (attach working screenshots +when possible). For changes with no observable behavior (pure refactor, docs), say so. + +## Risk +- Low / Medium / High +- What can break + +## Checklist +- [ ] Tests added or updated +- [ ] Backward compatibility considered diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4e6c7c3..647799b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,12 +11,17 @@ env: CARGO_TERM_COLOR: always RUSTFLAGS: -Dwarnings +permissions: + contents: read + jobs: lint: name: Lint runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable with: components: rustfmt, clippy @@ -29,12 +34,25 @@ jobs: run: cargo doc --workspace --no-deps env: RUSTDOCFLAGS: -Dwarnings + # OKF v0.2 conformance of knowledge/. okf-lint enforces the spec; + # check_okf.py enforces bundle-local conventions it does not cover. + # Keep the version and --max-line-length in lockstep with the justfile. + - name: Install okf-lint + run: cargo install okf-lint --version 0.1.1 --locked + - name: Check knowledge bundle (OKF) + run: | + python3 scripts/check_okf.py knowledge + okf-lint knowledge --max-line-length 10000 + - name: Test repo scripts + run: python3 -m unittest discover -s scripts/tests -p 'test_*.py' audit: name: Audit runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - name: Install cargo-audit run: cargo install cargo-audit @@ -50,6 +68,8 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - name: Run tests @@ -63,6 +83,11 @@ jobs: echo "HTTP client leaked into the docx,spreadsheet feature set" exit 1 fi + - name: Run examples + run: | + for ex in basic custom_selectors configuration ir_access; do + cargo run -q -p markkit --example "$ex" + done check: name: Check diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index cd3d84b..1fae950 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -6,12 +6,17 @@ on: pull_request: branches: [main] +permissions: + contents: read + jobs: coverage: name: Code Coverage runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - name: Install cargo-tarpaulin diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml index 42a7e8e..22de698 100644 --- a/.github/workflows/js.yml +++ b/.github/workflows/js.yml @@ -12,12 +12,17 @@ on: - "crates/markkit-js/**" - "crates/markkit/**" +permissions: + contents: read + jobs: build: name: Build Node.js bindings runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - uses: actions/setup-node@v4 with: diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 35161b5..5dd890a 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -5,12 +5,17 @@ on: - cron: "0 4 * * *" workflow_dispatch: +permissions: + contents: read + jobs: nightly: name: Nightly checks runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@nightly with: components: rustfmt, clippy diff --git a/.github/workflows/publish-js.yml b/.github/workflows/publish-js.yml index 6fb33e8..d189da2 100644 --- a/.github/workflows/publish-js.yml +++ b/.github/workflows/publish-js.yml @@ -8,12 +8,17 @@ on: required: false default: "true" +permissions: + contents: read + jobs: publish: name: Publish npm package runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - uses: actions/setup-node@v4 with: diff --git a/.github/workflows/publish-python.yml b/.github/workflows/publish-python.yml index 1b82c0c..1e8441a 100644 --- a/.github/workflows/publish-python.yml +++ b/.github/workflows/publish-python.yml @@ -19,6 +19,8 @@ jobs: version: ${{ steps.metadata.outputs.version }} steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - uses: actions/setup-python@v5 with: diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml index 4c33715..f11ef5c 100644 --- a/.github/workflows/python.yml +++ b/.github/workflows/python.yml @@ -12,12 +12,17 @@ on: - "crates/markkit-python/**" - "crates/markkit/**" +permissions: + contents: read + jobs: build: name: Build Python bindings runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + persist-credentials: false - uses: dtolnay/rust-toolchain@stable - uses: actions/setup-python@v5 with: diff --git a/AGENTS.md b/AGENTS.md index a5451f9..97fa39f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,33 +1,127 @@ -# Markkit – Coding Agent Guidance +## Coding-agent guidance -Minimal tokens. Telegraph style. Drop filler. +### Style -## Quick Reference +Telegraph. Drop filler/grammar. Min tokens. -- Build: `just build` -- Test: `just test` -- Lint: `just lint` -- Pre-PR: `just pre-pr` -- Specs: `specs/` directory +### Critical Thinking -## Rules +Fix root cause. Unsure: read more code; if stuck, ask w/ short options. Unrecognized changes: assume other agent; keep going. If causes issues, stop + ask. -- Fetch `origin/main` and rebase before editing -- Small PRs, incremental changes -- Conventional Commits: `type(scope): description` -- Real human git identity only -- No Claude session links in commits/PRs -- Write failing test first, then fix -- Security tests for parsing changes -- Run `just pre-pr` before submitting +### Principles -## Crate Map +- Always work on top of latest remote main: fetch `origin/main`, rebase (or recreate worktree) before editing +- Important decisions as comments on top of file +- Code testable, smoke testable, runnable locally +- Small, incremental PR-sized changes +- Write failing test before fixing bug +- All input is untrusted: parsing changes need security tests + +### Knowledge + +`knowledge/` is the canonical OKF bundle and persistent project memory (replaces the old `specs/`). Read relevant knowledge before changing behavior; update it in the same change when decisions, behavior, constraints, threats, tests, or operations change. Rules: `knowledge/knowledge-contract.md` (`just check-okf` enforces OKF v0.2). + +| Knowledge | Description | +|------|-------------| +| knowledge-contract | Knowledge maintenance rules + OKF v0.2 conformance rules | +| foundations/architecture | Extraction pipeline, design principles, crate structure, builder config | +| foundations/ir | Canonical IR between parsers and the markdown renderer | +| formats/html-extraction | HTML element mapping, stripping, whitespace, depth limits | +| formats/format-support | Formats, bytes-in API, detection order, DOCX + spreadsheet mapping | +| security/security-model | One-page threat/invariant summary | +| security/threat-model | Threats T1-T6, mitigations, test map, resource limits, invariants S1-S11 | +| runtimes/python-package | PyO3 `markkit` module, maturin, PyPI | +| runtimes/node-package | NAPI-RS `@everruns/markkit`, npm, binding parity table | +| operations/testing | Test layout, categories, CI jobs | +| operations/documentation | Doc trees and what each must cover | +| operations/examples | `examples/` contents and requirements | +| operations/maintenance | Maintenance invocation contract + checklist + workflow credential audit | +| operations/release-process | Agent-driven release, crates.io + PyPI + npm | +| operations/eval | LLM-as-judge eval design (harness is a stub) | + +### Documentation + +- User docs: `README.md`, `docs/` +- API reference: rustdoc in `crates/markkit/src/lib.rs` and modules +- Binding READMEs (`crates/markkit-python/README.md`, `crates/markkit-js/README.md`) ship to PyPI/npm, keep them in sync with exports +- Durable engineering knowledge: `knowledge/`, never `docs/` + +### Maintenance invocation + +"Run maintenance" (incl. "maintainace" / "maintaiance") = analyze, fix, validate, ship through green CI and squash-merge. Use the `maintain` and `ship` skills; checklist in `knowledge/operations/maintenance.md`. Don't stop at a report, local commit, or deferral issues. Analysis-only needs an explicit request. Preserve security contracts; never merge red CI. + +### Cloud Agent Setup + +```bash +./scripts/init-cloud-env.sh # Install just + gh +just build +``` + +### Local Dev + +```bash +just --list # All commands +just build # Build +just test # cargo test --workspace +just lint # clippy -D warnings +just check # fmt + clippy + test + knowledge + scripts + examples +just check-okf # knowledge/ OKF conformance +just examples # Run every example +just bench # Criterion benches (crates/markkit/benches) +just pre-pr # Pre-PR checks +``` + +### Crate Map | Crate | Purpose | |-------|---------| -| `markkit` | Core extraction library | -| `markkit-cli` | CLI tool | -| `markkit-bench` | Benchmarks | -| `markkit-eval` | LLM evaluation | -| `markkit-python` | Python bindings | -| `markkit-js` | Node.js bindings | +| `markkit` | Core extraction library (HTML, DOCX, spreadsheets) | +| `markkit-cli` | `markkit` CLI | +| `markkit-bench` | Benchmark harness (stub; real benches are criterion in `markkit`) | +| `markkit-eval` | LLM evaluation harness (stub) | +| `markkit-python` | Python bindings (PyO3), module `markkit` | +| `markkit-js` | Node.js bindings (NAPI-RS), package `@everruns/markkit` | + +### Rust + +- Stable Rust, pinned in `rust-toolchain.toml` +- `cargo fmt`, `cargo clippy --workspace --all-targets -- -D warnings` +- Licenses/advisories: `cargo deny check` (`deny.toml`), `cargo audit` +- Format deps stay feature-gated; `docx,spreadsheet` build must not link reqwest/hyper (CI checks) + +### Pre-PR Checklist + +1. `just pre-pr` (fmt, clippy, test, check-okf, script tests, examples, deny) +2. Tests cover positive and negative cases +3. Security tests if change touches parsing, limits, or URL fetching (see `knowledge/security/threat-model.md`) +4. Rebase on main: `git fetch origin main && git rebase origin/main` +5. Update knowledge if behavior or durable facts change; add a `knowledge/log.md` entry +6. `CHANGELOG.md` Unreleased entry for user-facing changes +7. CI green before merge; resolve all PR comments + +### CI + +- GitHub Actions; `ci.yml` aggregate `Check` job gates merge +- **NEVER merge when CI is red.** No exceptions. + +### Commits + +[Conventional Commits](https://www.conventionalcommits.org): `type(scope): description` + +Types: feat, fix, docs, refactor, test, chore, ci + +- Updates to `knowledge/` and `AGENTS.md`: `chore` type +- NEVER add links to Claude sessions in PR body or commits + +### Commit Attribution + +All commits attributed to the real human user, never a bot/agent identity, no AI `Co-authored-by` trailers, no "generated by" text. Verify `git config user.name`/`user.email` are human before committing; if missing or bot-like, set from `$GIT_USER_NAME`/`$GIT_USER_EMAIL` (`.claude/hooks/fix-git-identity.sh` does this at session start). If those are missing too, stop and ask. + +### PRs + +Squash and Merge. Use `.github/pull_request_template.md`. Center the description on functional change and impact; add Before / After evidence (CLI output, test results) whenever behavior changes. + +- Prefer small, shippable PRs; split independent changes +- When asked for separate PRs, don't bundle + +See `CONTRIBUTING.md` for details. diff --git a/CHANGELOG.md b/CHANGELOG.md index 219fca7..dc5afbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +### Changed + +- Design docs moved from `specs/` to an OKF v0.2 knowledge bundle in `knowledge/`, validated in CI. +- CI workflows declare read-only default permissions and do not persist checkout credentials. +- CI runs every example; `just bench` now runs the criterion benchmarks. +- README, SECURITY and binding docs corrected: Python module is `markkit`, URL fetching has no private-network blocking yet. + ## [0.1.1] - 2026-10-22 ### Security diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b5e2271..3c1f58f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,9 +23,10 @@ just test 1. Fork the repository 2. Create a feature branch: `git checkout -b feat/your-feature` -3. Make your changes -4. Run `just pre-pr` to validate -5. Submit a pull request +3. Write a failing test, then make your changes +4. Update `knowledge/` and `CHANGELOG.md` if behavior changes +5. Run `just pre-pr` to validate +6. Submit a pull request using the template; PRs are squash-merged ## Code Quality @@ -51,11 +52,15 @@ We use [Conventional Commits](https://www.conventionalcommits.org/): - Cover positive and negative scenarios - Security tests for input parsing changes - Run `just bench` for performance-sensitive changes +- Run `just examples` when the public API changes -## Specs +## Knowledge -The `specs/` directory contains architecture and design documentation. New -features should either comply with existing specs or propose changes. +`knowledge/` is the project's design knowledge in [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) +format: architecture, IR, format mappings, threat model, and processes. New +features either follow the existing knowledge or update it in the same PR. +`just check-okf` validates the bundle; rules are in +[knowledge/knowledge-contract.md](knowledge/knowledge-contract.md). ## Questions? diff --git a/Cargo.lock b/Cargo.lock index 3b7beee..c851243 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -260,9 +260,9 @@ checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" [[package]] name = "cc" -version = "1.9.0" +version = "1.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877bf67ac8e52449eb2f9e98a344ea163beb70dd1f9eebe3cc5357dcfeaee4df" +checksum = "86e055fa6e3e40ad6187ba25b6855eeb635d5271f8e0f0e1e9a5826b8d75bea0" dependencies = [ "find-msvc-tools", "jobserver", @@ -2549,9 +2549,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "smallvec" -version = "1.16.3" +version = "1.16.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b" +checksum = "d0d5d4b1981b8bbbb7aed204cae5726d683d5d6ac7e74ec54122ea28f315e7c8" [[package]] name = "socket2" @@ -2954,9 +2954,9 @@ checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" [[package]] name = "typed-path" -version = "0.12.4" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "54ec2755506471f54baf5e4bb5499d8e510cc1caf271270e778a736bd2d7336c" +checksum = "9ce898823d8cd0f028662d2f349694b1af0a5cc9be83f1d213a00d3147445e1e" [[package]] name = "typenum" @@ -3020,9 +3020,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.28.0" +version = "1.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cc1186384beb7dd8eedea376413fd654937285ea6c9cfbb928dc3043ea4b606" +checksum = "4a0e2d11620c50695fe12bbbfde96b3477fe606e7284c22d505c12ce51214514" dependencies = [ "js-sys", "wasm-bindgen", @@ -3158,9 +3158,9 @@ dependencies = [ [[package]] name = "webpki-root-certs" -version = "1.0.10" +version = "1.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "58ec80e73fd8baf57bf0d9615554c75348bda990d6d5bbeab964dd068319fecb" +checksum = "e49d0e65abc00c142e8f9428da991da9f1a250c39be97459901eb12b9b49f62a" dependencies = [ "rustls-pki-types", ] @@ -3392,16 +3392,18 @@ dependencies = [ [[package]] name = "zip" -version = "9.0.5" +version = "9.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e04976da629d3b04204ba8105b396a193af8ad2583b2c79bfb3ab8af580a11e" +checksum = "518b847fd864440c379748ad684c449f293af68978a485a4ff16f7c901c1f7c4" dependencies = [ "crc32fast", "flate2", + "getrandom 0.4.3", "indexmap", "memchr", "nix", "typed-path", + "windows-sys 0.61.2", "zopfli", ] diff --git a/README.md b/README.md index 47939c6..dc343aa 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,38 @@ # Markkit -Extract clean markdown from any format. +[![CI](https://github.com/everruns/markkit/actions/workflows/ci.yml/badge.svg)](https://github.com/everruns/markkit/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) +[![Crates.io](https://img.shields.io/crates/v/markkit.svg)](https://crates.io/crates/markkit) +[![docs.rs](https://img.shields.io/docsrs/markkit)](https://docs.rs/markkit) +[![PyPI](https://img.shields.io/pypi/v/markkit.svg)](https://pypi.org/project/markkit/) +[![npm](https://img.shields.io/npm/v/@everruns/markkit.svg)](https://www.npmjs.com/package/@everruns/markkit) +[![Repo: Agent Friendly](https://img.shields.io/badge/Repo-Agent%20Friendly-blue)](https://github.com/everruns/markkit/blob/main/AGENTS.md) -Markkit is a Rust library and CLI tool for converting HTML, PDF, DOCX, spreadsheets, and other document formats into well-structured markdown. It provides native bindings for Python and Node.js. +Extract clean markdown from any format. Written in Rust. + +Markkit converts HTML, Word documents and spreadsheets into well-structured +markdown, built for feeding untrusted documents to LLMs and agents. It ships as +a Rust library, a CLI, and native bindings for Python and Node.js. ## Features -- **HTML** – Convert web pages and HTML documents to markdown -- **DOCX** – Convert Word documents (headings, lists, tables, links, emphasis) -- **Spreadsheets** – Convert XLSX/XLSM, XLS and ODS workbooks to markdown tables -- **PDF** – Planned (currently reported as unsupported) -- **URL fetching** – Fetch and extract from URLs directly -- **Configurable** – CSS selectors for include/exclude, resource limits -- **Safe** – Input sanitization, resource limits, no code execution +- **HTML** – Web pages and HTML documents; nav, header, footer, scripts, styles and forms stripped by default +- **DOCX** – Headings, lists, tables, links, emphasis +- **Spreadsheets** – XLSX/XLSM, XLS and ODS workbooks to markdown tables, one section per sheet +- **Format detection** – Content type, then file extension, then magic bytes +- **URL fetching** – Fetch and extract in one call (`http_client` feature) +- **Configurable** – CSS include/exclude selectors, link/image preservation, resource limits +- **Safe by default** – No code execution (scripts, macros, formulas), bounded decompression, + depth and size limits, [threat model](knowledge/security/threat-model.md) with a test per mitigation +- **PDF** – Planned (currently an explicit `UnsupportedFormat` error, never an empty success) + +| Format | Status | Feature | +|---|---|---| +| HTML / XHTML | Supported | `html` (default) | +| DOCX | Supported | `docx` | +| XLSX, XLSM, XLS, ODS | Supported | `spreadsheet` | +| PDF | Planned | `pdf` | +| DOC, PPTX, XLSB | Detected, rejected explicitly | – | ## Quick Start @@ -57,6 +77,9 @@ markkit extract https://example.com --include "article" --exclude "nav,footer" # JSON output with metadata markkit extract https://example.com --json + +# From stdin +cat page.html | markkit extract - ``` ### Python @@ -64,7 +87,7 @@ markkit extract https://example.com --json ```python import markkit -markdown = await markkit.extract_html("

Hello

") +markdown = markkit.extract_html_sync("

Hello

") markdown = await markkit.extract_url("https://example.com") ``` @@ -73,8 +96,8 @@ markdown = await markkit.extract_url("https://example.com") ```javascript const { extractHtml, extractUrl } = require('@everruns/markkit'); -const markdown = await extractHtml('

Hello

'); -const markdown = await extractUrl('https://example.com'); +const fromHtml = await extractHtml('

Hello

'); +const fromUrl = await extractUrl('https://example.com'); ``` ## Installation @@ -98,12 +121,16 @@ cargo install markkit-cli pip install markkit ``` +Wheels: Linux x64, CPython 3.12. Other platforms build from the sdist (needs Rust). + ### Node.js ```bash npm install @everruns/markkit ``` +Native binary: Linux x64 (glibc). + ## Configuration ```rust @@ -143,16 +170,50 @@ Office-document limits are configurable on the builder: `max_archive_entries`, `max_archive_entry_size`, `max_archive_total_size`, `max_spreadsheet_cells`, `max_sheet_rows`, `max_sheet_cols`, `include_hidden_sheets`. +## Security + +Markkit treats every input as untrusted. HTML scripts, styles, forms, iframes and +SVG are stripped; office containers are decompressed under measured byte limits; +spreadsheet grids are pre-scanned so sparse cells cannot force huge allocations; +XML entities are never expanded; macros and formulas are never executed; hidden +sheets are skipped by default. URL fetching does not yet block private networks, +so embedders fetching attacker-chosen URLs should fetch themselves and call +`extract_document` (build with `default-features = false`). + +See the [threat model](knowledge/security/threat-model.md) and the +[security policy](SECURITY.md) for reporting vulnerabilities. + +## Documentation + +- [Getting started](docs/getting-started.md) +- API reference: [docs.rs/markkit](https://docs.rs/markkit) +- Design knowledge (architecture, IR, formats, threat model, processes): [`knowledge/`](knowledge/index.md) +- [Examples](examples/): `cargo run -p markkit --example basic` + ## Development ```bash -just build # Build workspace -just test # Run all tests -just lint # Run clippy -just pre-pr # Full pre-PR validation -just bench # Run benchmarks +just build # Build workspace +just test # Run all tests +just lint # Run clippy +just check # fmt, clippy, tests, knowledge, examples +just pre-pr # Full pre-PR validation +just bench # Criterion benchmarks ``` +Coding agents: start with [AGENTS.md](AGENTS.md). + +## Contributing + +Bug reports, feature requests and questions are welcome as +[issues](https://github.com/everruns/markkit/issues). For code, see +[CONTRIBUTING.md](CONTRIBUTING.md). + +## Ecosystem + +Markkit is part of the [Everruns](https://everruns.com) ecosystem, alongside +[Bashkit](https://github.com/everruns/bashkit). + ## License MIT License. See [LICENSE](LICENSE) for details. diff --git a/SECURITY.md b/SECURITY.md index bac4de5..fb4d01c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -27,13 +27,14 @@ Please include: ## Security Architecture -Markkit processes untrusted input (HTML, PDF, DOCX, etc.) and converts it to markdown. Key security considerations: +Markkit processes untrusted input (HTML, DOCX, spreadsheets) and converts it to markdown. Key security considerations: -- **Input sanitization**: All parsed content is sanitized before output -- **Resource limits**: Configurable limits on input size and processing time -- **No code execution**: Extracted content is treated as data, never executed -- **Network isolation**: URL fetching is opt-in with configurable allowlists -- **Memory budgets**: Configurable memory limits to prevent DoS via large documents +- **No code execution**: scripts, styles, forms, iframes and SVG are stripped; macros and formulas are never executed; XML entities are never expanded +- **Resource limits**: input size, HTML/XML nesting depth, archive entry count and decompressed size, spreadsheet grid cells, rows and columns +- **Explicit failure**: malformed input returns errors, never panics; unsupported formats fail explicitly +- **URL fetching**: enabled by the default `http_client` feature; private-network blocking and allowlists are not implemented yet, so fetch attacker-chosen URLs yourself and build with `default-features = false` + +Full analysis: [threat model](knowledge/security/threat-model.md). ## Scope diff --git a/crates/markkit-eval/Cargo.toml b/crates/markkit-eval/Cargo.toml index d904ace..9ae387e 100644 --- a/crates/markkit-eval/Cargo.toml +++ b/crates/markkit-eval/Cargo.toml @@ -1,5 +1,5 @@ # LLM evaluation harness for markkit extraction quality -# See specs/012-eval.md for design +# See knowledge/operations/eval.md for design [package] name = "markkit-eval" diff --git a/crates/markkit/src/docx.rs b/crates/markkit/src/docx.rs index 143372b..cf2795f 100644 --- a/crates/markkit/src/docx.rs +++ b/crates/markkit/src/docx.rs @@ -8,7 +8,7 @@ //! need a small subset of WordprocessingML: paragraphs, runs, hyperlinks, //! numbering, tables, styles and core properties. //! -//! Best-effort mapping (see specs/format-support.md): +//! Best-effort mapping (see knowledge/formats/format-support.md): //! - Headings: `Title`/`HeadingN` style ids or names (`heading n`, resolved through //! `basedOn`), or `outlineLvl`; levels > 6 clamp to 6. //! - Lists: `numPr` (direct or from the paragraph style) + numbering.xml `numFmt`; diff --git a/docs/getting-started.md b/docs/getting-started.md index b207219..35dbc5a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -9,11 +9,11 @@ Add markkit to your `Cargo.toml`: markkit = "0.1" ``` -For additional format support: +For DOCX and spreadsheets: ```toml [dependencies] -markkit = { version = "0.1", features = ["all_formats"] } +markkit = { version = "0.1", features = ["docx", "spreadsheet"] } ``` ## Basic Usage @@ -38,6 +38,18 @@ async fn main() -> anyhow::Result<()> { } ``` +## Documents from bytes + +```rust +use markkit::{DocumentHint, Markkit}; + +let markkit = Markkit::builder().build(); +let bytes = std::fs::read("report.docx")?; +let hint = DocumentHint { content_type: None, file_name: Some("report.docx") }; +let doc = markkit.extract_document_sync(&bytes, hint)?; +println!("{}", doc.markdown); +``` + ## Configuration ```rust @@ -51,6 +63,7 @@ let markkit = Markkit::builder() ## Next Steps -- See `specs/002-html-extraction.md` for HTML conversion details -- See `specs/003-format-support.md` for supported formats -- See `specs/005-security.md` for security considerations +- [HTML extraction](../knowledge/formats/html-extraction.md): element mapping and stripping rules +- [Format support](../knowledge/formats/format-support.md): formats, detection, DOCX and spreadsheet mapping +- [Threat model](../knowledge/security/threat-model.md): security boundaries and limits +- [Examples](../examples/): runnable with `cargo run -p markkit --example ` diff --git a/justfile b/justfile index 739986b..85d67fd 100644 --- a/justfile +++ b/justfile @@ -3,6 +3,10 @@ set dotenv-load := false +# Keep in lockstep with the okf-lint step in .github/workflows/ci.yml +okf_lint_version := "0.1.1" +okf_lint_max_line_length := "10000" + # Default recipe: list all available commands default: @just --list @@ -17,8 +21,8 @@ build: build-release: cargo build --workspace --release -# Run all checks (format, lint, test) -check: fmt-check lint test +# Run all checks (format, lint, test, knowledge, scripts, examples) +check: fmt-check lint test check-okf test-scripts examples # Format code fmt: @@ -46,6 +50,19 @@ test-all: test-one name: cargo test --workspace -- {{name}} +# Run repository script tests (scripts/tests) +test-scripts: + python3 -m unittest discover -s scripts/tests -p 'test_*.py' + +# Run every example (CI runs this too) +examples: + #!/usr/bin/env bash + set -euo pipefail + for ex in basic custom_selectors configuration ir_access; do + echo "── example: $ex" + cargo run -q -p markkit --example "$ex" + done + # ─── Run ───────────────────────────────────────────────────── # Run the CLI with arguments @@ -62,13 +79,17 @@ extract-file path: # ─── Benchmarks ────────────────────────────────────────────── -# Run benchmarks +# Run criterion benchmarks for the core crate bench *args: + cargo bench -p markkit --bench extraction {{args}} + +# Run the markkit-bench harness (stub, not yet implemented) +bench-harness *args: cargo run -p markkit-bench --release -- {{args}} # ─── Eval ──────────────────────────────────────────────────── -# Run LLM evaluation harness +# Run the LLM evaluation harness (stub, not yet implemented) eval *args: cargo run -p markkit-eval --release -- {{args}} @@ -89,7 +110,7 @@ vet: # ─── Release ──────────────────────────────────────────────── # Pre-PR checklist -pre-pr: fmt lint test deny +pre-pr: fmt lint test check-okf test-scripts examples deny @echo "All pre-PR checks passed" # Check current version @@ -100,7 +121,26 @@ version: publish-dry-run: cargo publish -p markkit --dry-run -# ─── Docs ─────────────────────────────────────────────────── +# ─── Docs & Knowledge ─────────────────────────────────────── + +# okf-lint covers the OKF spec rules; check_okf.py covers bundle-local +# conventions it does not enforce (see knowledge/knowledge-contract.md). +# okf-lint is pinned in CI and skipped here when absent: just install-okf-lint +# Validate the knowledge/ OKF v0.2 bundle +check-okf: + #!/usr/bin/env bash + set -euo pipefail + python3 scripts/check_okf.py knowledge + if command -v okf-lint >/dev/null; then + okf-lint knowledge --max-line-length {{ okf_lint_max_line_length }} + echo "okf-lint {{ okf_lint_version }}: knowledge conforms to OKF v0.2" + else + echo "okf-lint not installed — skipping (just install-okf-lint)" + fi + +# Install the pinned upstream OKF spec linter +install-okf-lint: + cargo install okf-lint --version {{ okf_lint_version }} --locked # Build documentation docs: diff --git a/specs/format-support.md b/knowledge/formats/format-support.md similarity index 90% rename from specs/format-support.md rename to knowledge/formats/format-support.md index e55ef6f..7e3a825 100644 --- a/specs/format-support.md +++ b/knowledge/formats/format-support.md @@ -1,3 +1,15 @@ +--- +type: Subsystem Design +title: Format Support +description: Supported and planned formats, bytes-in API, format detection, DOCX and spreadsheet mapping. +tags: + - markkit + - formats + - docx + - spreadsheet + - detection +--- + # Format Support ## Supported Formats @@ -101,8 +113,14 @@ All formats enforce: - Maximum processing time (default: 60s) – configuration only, enforcement TBD - Maximum output size (prevent markdown explosion) – bounded by the caps below -Office formats additionally enforce (see `specs/threat-model.md`): +Office formats additionally enforce (see [Threat Model](../security/threat-model.md) T6): - `max_archive_entries` (10 000), `max_archive_entry_size` (64 MiB decompressed), `max_archive_total_size` (256 MiB decompressed) - `max_spreadsheet_cells` (4 000 000) for the dense grid calamine builds for XLS/ODS - XML nesting depth 256 for DOCX parts + +## See also + +- [HTML Extraction](html-extraction.md), the Tier 1 path +- [IR Specification](../foundations/ir.md), the shared output of every parser +- [Threat Model](../security/threat-model.md), T6 malicious office documents diff --git a/specs/html-extraction.md b/knowledge/formats/html-extraction.md similarity index 89% rename from specs/html-extraction.md rename to knowledge/formats/html-extraction.md index ffb70a3..1da7924 100644 --- a/specs/html-extraction.md +++ b/knowledge/formats/html-extraction.md @@ -1,3 +1,13 @@ +--- +type: Subsystem Design +title: HTML Extraction +description: HTML to Markdown conversion covering element mapping, stripping, whitespace normalization, and depth limits. +tags: + - markkit + - html + - extraction +--- + # HTML Extraction ## Overview @@ -29,12 +39,6 @@ HTML from the web: malformed markup, complex layouts, embedded scripts, etc. | `/` | `**text**` | | `/` | `~~text~~` | -## Content Extraction Strategies - -- **CSS selectors**: User-provided include/exclude selectors -- **Readability**: Article content detection heuristic -- **Full page**: Convert everything (for known-good HTML) - ## Complete Element Mapping ### Block Elements @@ -135,9 +139,9 @@ providing defense-in-depth. ## Content Extraction Strategies -- **CSS selectors**: User-provided include/exclude selectors -- **Readability**: Article content detection heuristic -- **Full page**: Convert everything (for known-good HTML) +- **CSS selectors** (implemented): include/exclude selectors, with the defaults below +- **Full page** (implemented): no include selector converts the whole body minus excludes +- **Readability** (planned): article-content detection heuristic, not implemented ## Edge Cases @@ -151,3 +155,9 @@ providing defense-in-depth. - Unclosed/mismatched tags: html5ever auto-repairs per HTML5 spec - Empty containers: all block-level containers with no content are elided - HTML entities: decoded by html5ever before reaching parser + +## See also + +- [IR Specification](../foundations/ir.md), the nodes HTML maps onto +- [Format Support](format-support.md), how HTML is detected among other formats +- [Threat Model](../security/threat-model.md), T1 injection and T2 nesting limits diff --git a/knowledge/formats/index.md b/knowledge/formats/index.md new file mode 100644 index 0000000..f9c3b35 --- /dev/null +++ b/knowledge/formats/index.md @@ -0,0 +1,4 @@ +# Formats + +* [HTML Extraction](html-extraction.md) - HTML to Markdown conversion covering element mapping, stripping, whitespace normalization, and depth limits. +* [Format Support](format-support.md) - Supported and planned formats, bytes-in API, format detection, DOCX and spreadsheet mapping. diff --git a/specs/architecture.md b/knowledge/foundations/architecture.md similarity index 68% rename from specs/architecture.md rename to knowledge/foundations/architecture.md index bbf3178..1451347 100644 --- a/specs/architecture.md +++ b/knowledge/foundations/architecture.md @@ -1,3 +1,13 @@ +--- +type: Architecture +title: Markkit Architecture +description: Extraction pipeline, design principles, crate structure, and builder configuration. +tags: + - markkit + - architecture + - pipeline +--- + # Architecture ## Overview @@ -42,4 +52,11 @@ Via `MarkkitBuilder`: - Processing time limits - CSS selectors for include/exclude - Link and image preservation -- Code block extraction +- Office-document limits (archive entries/sizes, sheet rows/cols, grid cells, hidden sheets) + +## See also + +- [IR Specification](ir.md), the normalized AST every parser emits +- [Format Support](../formats/format-support.md), detection and per-format parsers +- [HTML Extraction](../formats/html-extraction.md), the primary conversion path +- [Threat Model](../security/threat-model.md), why every input is untrusted diff --git a/knowledge/foundations/index.md b/knowledge/foundations/index.md new file mode 100644 index 0000000..7571dab --- /dev/null +++ b/knowledge/foundations/index.md @@ -0,0 +1,4 @@ +# Foundations + +* [Markkit Architecture](architecture.md) - Extraction pipeline, design principles, crate structure, and builder configuration. +* [IR Specification](ir.md) - Canonical intermediate representation that decouples format parsers from the Markdown renderer. diff --git a/specs/ir-spec.md b/knowledge/foundations/ir.md similarity index 90% rename from specs/ir-spec.md rename to knowledge/foundations/ir.md index 722d6da..a0eaeab 100644 --- a/specs/ir-spec.md +++ b/knowledge/foundations/ir.md @@ -1,3 +1,13 @@ +--- +type: Interface Contract +title: IR Specification +description: Canonical intermediate representation that decouples format parsers from the Markdown renderer. +tags: + - markkit + - ir + - pipeline +--- + # IR Specification ## Overview @@ -130,3 +140,9 @@ Roundtrip guarantee: `serialize -> deserialize` produces `==` document. Tested in `ir::tests::ir_roundtrip_json` and `html_integration::ir_serialization_roundtrip`. All types derive: `Debug`, `Clone`, `PartialEq`, `Serialize`, `Deserialize`. + +## See also + +- [Markkit Architecture](architecture.md), where the IR sits in the pipeline +- [HTML Extraction](../formats/html-extraction.md), the HTML parser that builds it +- [Examples](../operations/examples.md), `ir_access.rs` walks it diff --git a/knowledge/index.md b/knowledge/index.md new file mode 100644 index 0000000..6132cc0 --- /dev/null +++ b/knowledge/index.md @@ -0,0 +1,16 @@ +--- +okf_version: "0.2" +--- + +# Markkit Knowledge + +* [Knowledge Maintenance Contract](knowledge-contract.md) - Rules for maintaining the Markkit knowledge bundle and its OKF conformance. +* [Update Log](log.md) - Chronological history of changes to this bundle. + +# Domains + +* [foundations/](foundations/) - Extraction pipeline architecture and the intermediate representation. +* [formats/](formats/) - Per-format parsing: HTML extraction, format detection, DOCX and spreadsheet mapping. +* [security/](security/) - Security model and detailed threat model for untrusted input. +* [runtimes/](runtimes/) - Published language bindings for Python and Node.js. +* [operations/](operations/) - Testing, documentation, examples, maintenance, release, and eval. diff --git a/knowledge/knowledge-contract.md b/knowledge/knowledge-contract.md new file mode 100644 index 0000000..f7fb888 --- /dev/null +++ b/knowledge/knowledge-contract.md @@ -0,0 +1,69 @@ +--- +type: Playbook +title: Knowledge Maintenance Contract +description: Rules for maintaining the Markkit knowledge bundle and its OKF conformance. +tags: + - markkit + - knowledge + - okf + - process +--- + +# Knowledge Maintenance Contract + +`knowledge/` is Markkit's canonical [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) +bundle and persistent project memory. It replaces the former `specs/` directory. + +## Maintenance rules + +- Treat this knowledge as part of the implementation, not historical documentation. +- Before changing behavior, read the relevant concepts and follow their decisions, or update them in the same change. +- When code changes a documented behavior, design decision, invariant, limitation, threat, test strategy, or process, update the affected concept in the same pull request. +- Record decisions not recoverable from code. Link to source and tests rather than duplicating volatile details. +- Keep stable identifiers (`T1`..`T6`, `S1`..`S11`); never renumber them. +- User-facing guides stay in `docs/`; API reference stays in rustdoc. +- Say what is true now: a stub is a stub, a planned mitigation is planned. + +## OKF conformance rules + +The bundle declares `okf_version: "0.2"` in the bundle-root [`index.md`](index.md). + +- Every `.md` file except the reserved `index.md` and `log.md` is a **concept** and starts with YAML frontmatter containing a non-empty `type`, plus `title` and a single-line `description`. +- `index.md` files carry **no** frontmatter, except the bundle root, which may carry only `okf_version`. +- `index.md` bodies are link lists under headings: `* [Title](path) - description`, reusing each concept's `description` verbatim. +- Every `index.md` lists its own directory: concepts beside it and subdirectories under it, nothing deeper. +- `log.md` entries are grouped under `## YYYY-MM-DD` headings, newest first. +- Links between concepts are relative and must resolve; links inside code spans and fenced blocks are text. +- Every concept links to at least one other concept, conventionally in a closing `## See also`. +- Bundle documents are referenced as relative markdown links, never as `knowledge/...` repository paths inside the bundle. + +## Layout + +| Directory | Holds | +|---|---| +| [`foundations/`](foundations/) | Pipeline architecture, IR | +| [`formats/`](formats/) | HTML extraction, format detection, office mappings | +| [`security/`](security/) | Security model, threat model | +| [`runtimes/`](runtimes/) | Python and Node.js packages | +| [`operations/`](operations/) | Testing, docs, examples, maintenance, release, eval | + +Only this contract and `log.md` sit at the bundle root. Directory conveys domain; +`type` conveys kind. Types in use: `Architecture`, `Subsystem Design`, +`Interface Contract`, `Package Design`, `Threat Model`, `Test Strategy`, `Playbook`. + +## Enforcement + +`scripts/check_okf.py knowledge` validates these rules (ported from bashkit, no +third-party dependencies). `just check-okf` runs it plus the pinned upstream +`okf-lint` when installed; CI runs both in the lint job. Regression tests live in +`scripts/tests/test_check_okf.py` (`just test-scripts`). + +```console +$ python3 scripts/check_okf.py knowledge +knowledge: OKF v0.2 conformant (15 concepts, 6 index files, 1 log file) +``` + +## See also + +- [Maintenance](operations/maintenance.md), the periodic pass that re-checks the bundle +- [Documentation Architecture](operations/documentation.md), the doc trees this bundle is not diff --git a/knowledge/log.md b/knowledge/log.md new file mode 100644 index 0000000..1afc07f --- /dev/null +++ b/knowledge/log.md @@ -0,0 +1,7 @@ +# Markkit Knowledge Update Log + +## 2026-10-26 + +* **OKF migration**: `specs/` became this OKF v0.2 bundle, grouped into `foundations/`, `formats/`, `security/`, `runtimes/`, `operations/`. Every concept gained frontmatter and a See also section; `just check-okf` and CI enforce conformance. See [Knowledge Maintenance Contract](knowledge-contract.md). +* **Accuracy pass**: [Python Package](runtimes/python-package.md) now names the shipped `markkit` module; [Examples](operations/examples.md) lists the four real examples; [Evaluation Harness](operations/eval.md) is marked as a stub; [Threat Model](security/threat-model.md) T3 records that `http_client` is a default feature and SSRF mitigations are still open. +* **Maintenance contract**: [Maintenance](operations/maintenance.md) adopts the analyze, fix, ship invocation contract and a workflow credential audit. diff --git a/knowledge/operations/documentation.md b/knowledge/operations/documentation.md new file mode 100644 index 0000000..97b6d78 --- /dev/null +++ b/knowledge/operations/documentation.md @@ -0,0 +1,41 @@ +--- +type: Playbook +title: Documentation Architecture +description: Where public docs, rustdoc, and the knowledge bundle live and what each must cover. +tags: + - markkit + - docs +--- + +# Documentation Architecture + +## Trees + +| Tree | Audience | Holds | +|---|---|---| +| `README.md` | Users | Pitch, features, quick start per surface, install, feature flags | +| `docs/` | Users | Guides (`getting-started.md`) | +| rustdoc (`crates/*/src/lib.rs`) | Rust users | API reference, feature flags, examples | +| `crates/markkit-python/README.md`, `crates/markkit-js/README.md` | Binding users | Shipped as the PyPI / npm package readme | +| `knowledge/` | Contributors, agents | OKF bundle: design decisions, threats, processes | +| `AGENTS.md` | Coding agents | Commands, rules, knowledge map | +| `CHANGELOG.md` | Everyone | Keep a Changelog, Unreleased section first | + +Durable engineering knowledge goes in `knowledge/`, not `docs/`. User-facing guides +link to GitHub-rendered knowledge only where no user doc exists. + +## Requirements + +- All public API items have rustdoc; `cargo doc --workspace --no-deps` with + `RUSTDOCFLAGS=-Dwarnings` is clean (CI lint job) +- README feature list, feature-flag table, and binding snippets match the code +- Binding READMEs match exported functions (see [Python Package](../runtimes/python-package.md), + [Node.js Package](../runtimes/node-package.md)) +- User-facing changes get a `CHANGELOG.md` Unreleased entry +- New features include a usage example (see [Examples](examples.md)) +- Knowledge changes follow the [Knowledge Maintenance Contract](../knowledge-contract.md) + +## See also + +- [Knowledge Maintenance Contract](../knowledge-contract.md), rules for `knowledge/` +- [Maintenance](maintenance.md), the documentation section of each pass diff --git a/specs/eval.md b/knowledge/operations/eval.md similarity index 54% rename from specs/eval.md rename to knowledge/operations/eval.md index f6a4019..1af0b2c 100644 --- a/specs/eval.md +++ b/knowledge/operations/eval.md @@ -1,5 +1,20 @@ +--- +type: Test Strategy +title: Evaluation Harness +description: LLM-as-judge extraction quality evaluation with its methodology, metrics, and providers. +tags: + - markkit + - eval + - llm +--- + # Evaluation Harness +## Status + +Not implemented. `crates/markkit-eval` is a CLI stub (`--provider`, `--dataset`) +that prints a placeholder and exits 0. This document is the design it will follow. + ## Overview `markkit-eval` measures extraction quality using LLM-as-judge @@ -23,3 +38,8 @@ to compare markkit output against reference extractions. - Anthropic (Claude) - OpenAI (GPT) + +## See also + +- [Testing Strategy](testing.md), deterministic tests that eval complements +- [HTML Extraction](../formats/html-extraction.md), the behavior being scored diff --git a/knowledge/operations/examples.md b/knowledge/operations/examples.md new file mode 100644 index 0000000..6005845 --- /dev/null +++ b/knowledge/operations/examples.md @@ -0,0 +1,61 @@ +--- +type: Playbook +title: Examples +description: Runnable Rust examples in examples/, what each demonstrates, and their requirements. +tags: + - markkit + - examples +--- + +# Examples + +## Structure + +``` +examples/ + basic.rs # Default HTML to markdown extraction + custom_selectors.rs # include/exclude CSS selectors + configuration.rs # How builder options change output + ir_access.rs # Parse to IR, inspect nodes, render separately +``` + +Examples live at the workspace root and are registered as `[[example]]` targets +of the `markkit` crate (`crates/markkit/Cargo.toml`, `required-features = ["html"]`). +Run one with `cargo run -p markkit --example `, all with `just examples`. + +## Examples + +### basic.rs + +- `Markkit::builder().build()` with defaults +- `extract_html_sync` on an inline HTML string +- Print the markdown + +### custom_selectors.rs + +- `include_selector("article")` targets main content +- `exclude_selector(".sidebar")` strips boilerplate +- Print the filtered markdown + +### configuration.rs + +- Compare default output with `preserve_links(false)` and `preserve_images(false)` +- Show how each option changes the rendered markdown + +### ir_access.rs + +- `parse_html()` returns the IR `Document` (see [IR Specification](../foundations/ir.md)) +- Walk `Document.nodes`, match on `Node` variants +- Render with `MarkdownRenderer` to show parse and render are separate stages + +## Requirements + +- No network or file access; inline HTML samples only +- Output to stdout, exit 0 on success +- CI runs every example (`just examples` in the Test job), so a broken example fails the build +- A new example is added to `crates/markkit/Cargo.toml`, this document, and the `examples` recipe + +## See also + +- [Documentation Architecture](documentation.md), where examples fit among the doc trees +- [Testing Strategy](testing.md), the CI jobs that run them diff --git a/knowledge/operations/index.md b/knowledge/operations/index.md new file mode 100644 index 0000000..0747777 --- /dev/null +++ b/knowledge/operations/index.md @@ -0,0 +1,8 @@ +# Quality and operations + +* [Testing Strategy](testing.md) - Test categories, fixtures, property and security tests, and CI expectations. +* [Documentation Architecture](documentation.md) - Where public docs, rustdoc, and the knowledge bundle live and what each must cover. +* [Examples](examples.md) - Runnable Rust examples in examples/, what each demonstrates, and their requirements. +* [Maintenance](maintenance.md) - Maintenance invocation contract and the dependency, security, test, docs, and CI checklist. +* [Release Process](release-process.md) - Versioning, agent-driven release flow, and publication to crates.io, PyPI, and npm. +* [Evaluation Harness](eval.md) - LLM-as-judge extraction quality evaluation with its methodology, metrics, and providers. diff --git a/knowledge/operations/maintenance.md b/knowledge/operations/maintenance.md new file mode 100644 index 0000000..9b21de8 --- /dev/null +++ b/knowledge/operations/maintenance.md @@ -0,0 +1,140 @@ +--- +type: Playbook +title: Maintenance +description: Maintenance invocation contract and the dependency, security, test, docs, and CI checklist. +tags: + - markkit + - maintenance +--- + +# Maintenance + +## Invocation contract + +"Run maintenance", "maintain", and misspellings such as "maintainace" and +"maintaiance" mean **analyze, fix, and ship**: local validation, push, PR, fix +CI and review findings, then squash-merge with every required check green. +A report, a local commit, or a set of deferral issues is not a completed pass. +Only an explicit analysis-only request narrows this. + +The `maintain` skill (`.agents/skills/maintain/`; `.claude/skills` symlinks there) +and the `/maintain` command implement this contract; the `ship` skill completes it. +Preserve security contracts; never weaken limits, tests, or audit policy to ship. + +## When to run + +- Before every minor or major release +- Monthly during patch-only periods +- After large feature merges + +## Checklist + +### Dependencies + +- Direct dependencies at latest versions, including major upgrades: bump + `[workspace.dependencies]`, build, test, fix API changes +- `cargo update` after bumps; `Cargo.lock` is tracked and audited +- `cargo audit` and `cargo deny check` clean; every `deny.toml` ignore still + needed and still justified +- `cargo tree --duplicates` reviewed; no unused dependencies +- Format dependencies stay behind their feature (`docx`, `spreadsheet`, `http_client`); + CI asserts the `docx,spreadsheet` build links no reqwest/hyper +- Dependabot (`.github/dependabot.yml`) covers cargo and github-actions weekly; + review and merge its open PRs + +### Security + +- [Threat Model](../security/threat-model.md) covers every input surface and feature +- Every mitigated threat has a test listed in its Test Coverage block +- New parsers or limits add a threat entry and tests +- Status of open threats (T3 SSRF) is stated honestly + +### Tests + +- `just test` green; no new `#[ignore]` without a reason +- Property and security suites green +- Coverage (`coverage.yml`) reviewed for large uncovered paths + +### Documentation + +- `README.md` features, feature flags, quick starts match the code +- Binding READMEs and [Python Package](../runtimes/python-package.md) / + [Node.js Package](../runtimes/node-package.md) match exported functions +- `docs/` examples compile against the current API +- rustdoc builds clean +- `CHANGELOG.md` Unreleased covers every user-facing change since the last release + +### Examples + +- `just examples` runs every example (also in CI) +- [Examples](examples.md) lists exactly the examples that exist + +### Knowledge + +- `just check-okf` green ([Knowledge Maintenance Contract](../knowledge-contract.md)) +- Each concept's status matches reality (stubs called stubs, planned called planned) +- Durable decisions made since the last pass are recorded; [Update Log](../log.md) updated + +### Code quality + +- `cargo fmt`, `cargo clippy -D warnings` clean +- No stale `TODO`, dead code, or commented-out code +- Duplication and needless abstraction removed + +### Binding parity + +- Python and Node.js expose the same functions; gaps listed in + [Node.js Package](../runtimes/node-package.md) § Binding parity + +### Agent configuration + +- `AGENTS.md` / `CLAUDE.md` accurate: commands work, knowledge table lists every concept +- `.claude/skills` still symlinks to `.agents/skills` (`just test-scripts` checks) + +### CI health + +- Latest CI on `main` green; any failure blocks the rest of the pass +- Nightly green for the past week; failures >2 consecutive days are fixed in this pass +- No `continue-on-error` without a cited open issue + +#### Workflow credential audit + +Audit `.github/workflows/` itself, not only run results: + +- **Checkout credentials**: every `actions/checkout` sets `persist-credentials: false` + unless the job authenticates a `git`/`gh` call (only `release.yml` pushes a tag) +- **Default permissions**: every workflow declares top-level `permissions: contents: read`; + write scopes are opt-in per job +- **Secrets**: only in the step that needs them, through `env:`; never via `$GITHUB_ENV` +- **Triggers**: no `pull_request_target`, `workflow_run`, or `issue_comment` +- **Untrusted interpolation**: no `${{ github.event.* }}` or `${{ github.head_ref }}` + inside a `run:` block; pass through `env:` + +## Findings and external blockers + +Fix findings in the active pass. A genuine external blocker (upstream bug, missing +permission) or an explicit scope split may become a tracked issue with evidence; +size alone is not a blocker. Never mark a pass complete or merge with red CI. + +## Pass record + +### 2026-10-26 + +Migrated `specs/` to the OKF `knowledge/` bundle with `just check-okf` in CI; +added the maintain skill; workflow credential audit (top-level permissions and +`persist-credentials: false` everywhere except the release tag push); examples now +run in CI; docs corrected (Python module name, example list, eval/bench stubs, +`http_client` is default and T3 SSRF is open). `cargo update` applied (6 semver +bumps); `cargo audit` and `cargo deny check` clean apart from the two allowed +unmaintained warnings, both pulled only by the stub `markkit-bench`. + +Findings shipped as separate PRs in the same pass: unused dependencies (core +`async-trait`, `futures`, `regex`, `unicode-segmentation`, `encoding_rs`, `base64`, +`mime`, `lol_html`; most stub-crate deps) and pending major upgrades; indented HTML +losing paragraph breaks between blocks (smoke test of `examples/basic.rs`). + +## See also + +- [Release Process](release-process.md), what follows a pre-release pass +- [Threat Model](../security/threat-model.md), the security section's source of truth +- [Testing Strategy](testing.md), the suites the pass runs diff --git a/specs/release-process.md b/knowledge/operations/release-process.md similarity index 74% rename from specs/release-process.md rename to knowledge/operations/release-process.md index e7c8219..247e902 100644 --- a/specs/release-process.md +++ b/knowledge/operations/release-process.md @@ -1,3 +1,12 @@ +--- +type: Playbook +title: Release Process +description: Versioning, agent-driven release flow, and publication to crates.io, PyPI, and npm. +tags: + - markkit + - release +--- + # Release Process ## Versioning @@ -43,9 +52,15 @@ The initial npm artifact supports Linux x64/glibc only, and the current Python wheel publisher targets Linux x64/CPython 3.12 only. Other platforms and interpreters are not claimed as available; a future native release can expand coverage. -Both publishers default to a credential-free dry run. Actual publication requires -an existing publisher credential before building; missing credentials fail without -printing values. Do not create credentials or alter permissions to complete a release. +Both publishers default to a credential-free dry run. npm publication requires the +`NPM_TOKEN` secret and fails before building when it is missing, without printing +values; Python publishes through PyPI Trusted Publishing (OIDC), with no stored token. Do not create credentials or alter permissions to complete a release. Before upload, npm installs and loads its packed tarball (loader plus native binary), and Python checks metadata and imports the built wheel in a clean virtual environment. Both builds must preserve the patched Cargo.lock. + +## See also + +- [Maintenance](maintenance.md), the pre-release pass +- [Python Package](../runtimes/python-package.md), wheel scope +- [Node.js Package](../runtimes/node-package.md), npm scope diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md new file mode 100644 index 0000000..77e5e90 --- /dev/null +++ b/knowledge/operations/testing.md @@ -0,0 +1,60 @@ +--- +type: Test Strategy +title: Testing Strategy +description: Test categories, fixtures, property and security tests, and CI expectations. +tags: + - markkit + - testing +--- + +# Testing Strategy + +## Test Layout + +| Location | Covers | +|---|---| +| `crates/markkit/src/**` `#[cfg(test)]` | Unit: detection, element conversion, builder, error paths | +| `crates/markkit/tests/html_integration.rs` | End-to-end HTML to markdown | +| `crates/markkit/tests/html_fixtures.rs` | Golden files: `tests/fixtures/html/.html` vs `.md` | +| `crates/markkit/tests/html_security.rs` | XSS payloads, nesting, oversized input | +| `crates/markkit/tests/html_proptest.rs` | Property tests: no panic, no script/style, determinism, limits | +| `crates/markkit/tests/docx_integration.rs`, `spreadsheet_integration.rs` | Office mapping | +| `crates/markkit/tests/document_detection.rs` | Format detection order and explicit unsupported errors | +| `crates/markkit/tests/document_security.rs` | Zip bombs, grid amplification, XML entities, malformed containers | +| `crates/markkit/tests/common/` | In-memory DOCX/XLSX/XLS/ODS fixture builders | +| `crates/markkit-cli/tests/cli_integration.rs` | CLI flags and output | +| `crates/markkit/benches/extraction.rs` | Criterion throughput benchmarks | + +Office fixtures are generated in code (`tests/common`), not committed binaries, so +each security test states exactly which bytes it feeds the parser. + +## Test Categories + +- **Unit**: detection, element conversion, builder, error paths +- **Integration**: HTML and office pipelines end to end, CLI +- **Property** (`proptest`): arbitrary input never panics, output never contains + `