Close the Python 4.7 parity gaps in the Rust binary

AGENTS.md

@@ -30,10 +30,10 @@ Prefer the CLI when the connected MCP server is still `python3 -m wikifier.mcp.s
 
 - `Cargo.toml` keeps an empty `[dependencies]` list. Do not add a crates.io crate, a Python package, or a Node module.
 - Do not hand-edit `library.md`, `file_health.md`, `file_health.json`, or `pending_updates.md`. `update-maps` rewrites `library.md`. Health and pending change only through the commands. They take `.wikifier_staging/.lock`.
-- `init` copies `index.html` only when it is missing, and it does not clobber an existing path list. `diagnostics.html` stays in this repo. `index.v2.html` and `diagnostics.v2.html` are comparison pages and are not deployed.
+- `init` copies `index.html` only when it is missing (`--no-copy` skips it), and it does not clobber an existing path list. It also creates `exclude_patterns.txt`, empty health, pending, and library files, `.wikifier/config`, `journal/`, and `Logged_issues/` when those paths are absent. `diagnostics.html` stays in this repo. `index.v2.html` and `diagnostics.v2.html` are comparison pages and are not deployed.
 - This repo's map roots are `src/` and `tests/` (`map_paths.txt`). `monitored_paths.txt` is the extra watch list for docs and `wikifier.sh`.
-- The import scan is comment-masked regex, plus named JS/TS re-exports to depth 8 (`parser: "regex+reexport"`). An indented Python import is conditional. `import()` is dynamic. Rust resolves `mod`, `crate::`, `self::`, `super::`, and package or lib names from the root `Cargo.toml` and from `[workspace] members` (globs included, `-` to `_`). `crate::` stays in that file's crate (`tests/*.rs`, `src/bin/*.rs`, and each workspace member). A package or lib name points at that member's library, which is how `src/main.rs` and `tests/*.rs` show up as dependents. The accuracy gate is `accuracy_fixtures_match_frozen_edges` (the frozen layouts, including a Cargo crate). Do not describe the scan as a CPython `ast` walk, and do not claim it has been run on Babylon.js. It does not read tsconfig paths, package exports, or npm workspaces.
-- `mark_green` checks Python, JS/TS, and Rust public names against the wiki. Rust names are unrestricted `pub` items, `pub` methods and associated items on a public type, and trait-impl items for a public type. `pub(crate)`, `pub(super)`, `pub(self)`, and `pub(in ...)` stay out. Go, C/C++, C#, Java, and Markdown come back `unchecked`. That is expected.
+- The import scan is comment-masked regex, plus named JS/TS re-exports to depth 8 (`parser: "regex+reexport"`). An indented Python import is conditional. `import()` is dynamic. JS/TS resolution reads `tsconfig.json` `paths`, `package.json` `exports`, and npm `workspaces` (`*` directory globs). Rust resolves `mod`, `crate::`, `self::`, `super::`, and package or lib names from the root `Cargo.toml` and from `[workspace] members` (globs included, `-` to `_`). `crate::` stays in that file's crate (`tests/*.rs`, `src/bin/*.rs`, and each workspace member). A package or lib name points at that member's library, which is how `src/main.rs` and `tests/*.rs` show up as dependents. The accuracy gate is `accuracy_fixtures_match_frozen_edges` (the frozen layouts, including a Cargo crate). Do not describe the scan as a CPython `ast` walk, and do not claim it has been run on Babylon.js.
+- `mark_green` checks Python, JS/TS, and Rust public names against the wiki. Python names come from a tokenizer (`__all__`, top-level `def`/`class`, `.pyi`). JS/TS names are exports after comments and strings are masked. Rust names are unrestricted `pub` items, `pub` methods and associated items on a public type, and trait-impl items for a public type. `pub(crate)`, `pub(super)`, `pub(self)`, and `pub(in ...)` stay out. Go, C/C++, C#, Java, and Markdown come back `unchecked`. That is expected. Wiki lookup is the sibling `.wiki.md`, then `.md`, then `docs/wiki`, `docs`, `wiki`, `documentation`, and `.wiki`.
 - `wikifier serve` binds `127.0.0.1:8787`. Do not bind port 8080.
 - Do not set the project root to a parent that contains several repos.
 - Git: honor hooks. Do not use `--no-verify`. Do not commit `.cursor/`, `target/`, or gitignored leftovers (`wikifier/*.pyc`, `build/`, `dist/`, `.venv-build/`).

CHANGELOG.md

@@ -29,7 +29,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `heal-stubs --dry-run` counts the stubs it would mark and does not write health. `seed-health` and `seed-source-hashes` return `HealthError` when the health write fails.
 - `init DIR` uses that directory as the project root unless `--target` is set. `verify-wikis DIR` limits the check to that file or directory. `search-journal --file` keeps only that path.
 - The CI empty-dependency check reads `Cargo.toml` with `awk`. The workflow does not run Python.
-- The live daemon actions are `start`, `stop`, `status`, and `run`.
+- Daemon actions are `start`, `stop`, `status`, `run`, `logs`, `restart`, `install-service`, and `uninstall-service`. The loop checks every 30 seconds and refreshes maps at start and then at most every 600 seconds. `start` refuses only readiness `blocked` unless `--force`.
+- `init --no-copy` skips the dashboard. `init` writes `exclude_patterns.txt`, empty health, pending, and library files, `.wikifier/config`, `journal/`, and `Logged_issues/` when missing.
+- Wiki lookup falls back from `<file>.wiki.md` through `docs/wiki`, `docs`, `wiki`, `documentation`, and `.wiki`. Python and JS/TS public names ignore comments and strings. Python `__all__` may span lines. `.pyi` is checked.
+- `record-changes --reasons` accepts a JSON object or `PATH=REASON` lines. `health --summary` omits the entry list. `list-paths` honors `--depth`, `--recursive`, and `--max-files`. `cycles` includes severity and a reverse-dependency ranking. `--text` prints prose; the default remains JSON.
+- `suggest-next` emits `review_low_confidence_edges` when a non-dynamic resolved edge is below 0.65. `seed-source-hashes` records hashes on Green rows and does not change status. `heal-stubs` defaults to a 350-character wiki. `update-maps` seeds health rows only for paths in `monitored_paths.txt` when that file lists real paths.
+- Pending bullets use `- path: message`.
 - This repo's import-map roots are `src/` and `tests/`.
 - The import scan remains comment-masked regex plus named JS/TS re-exports to depth 8 (`parser: "regex+reexport"`). It is not a CPython `ast` walk, and it has not been run on Babylon.js. The accuracy gate is `accuracy_fixtures_match_frozen_edges`, now including a Cargo crate layout.
 - Rust `mod`, `crate::`, `self::`, `super::`, and the root `Cargo.toml` package or lib name resolve to files. Hyphens in the package name become underscores. `crate::` stays inside the file's crate (`tests/*.rs` and `src/bin/*.rs` are separate crates). The package or lib name points at the library, so binaries and integration tests are dependents of the library modules they import.

CLAUDE.md

@@ -60,7 +60,7 @@ Pipeline: **scan → parse → resolve → cache → health → artifacts**, imp
 
 - `src/api.rs` — the library surface: `session_bootstrap`, `check_changes`, `prepare_edit`, `suggest_next_actions`, `record_change`, `record_changes`, `record_deletion`, `mark_green`, `verify_wikis`, `update_maps` / `update_maps_run`, `dependency_report`, `load_dependents`, `cycles`, `init_project`, `serve_loop`, `monitor_loop`, `daemon_ctl`. `check_changes` watches mapped files, Green files, and `monitored_paths.txt`. `record_changes` reads `git status` (git is optional).
 - `src/scope.rs` — project root, `map_paths.txt`, `monitored_paths.txt`, excludes, and the file walk. A missing configured map directory is a scope error. Three or more child projects make the root a container and the walk stops.
-- `src/parse.rs` — comment-masked regex scans for Python, JS/TS, Rust, Go, C/C++, C#, and Java. Named JS/TS re-exports are followed to depth 8 (`via_barrel`). An indented Python import is `conditional`. `import()` is `dynamic`. Rust resolves `mod`, `crate::`, `self::`, `super::`, and package or lib names from the root `Cargo.toml` and `[workspace] members` (`-` to `_`, globs included). `crate::` stays in the file's crate; a member name points at that member's library. This is not a CPython `ast` parser and it does not read tsconfig paths, package `exports`, or npm workspaces. The accuracy gate is `accuracy_fixtures_match_frozen_edges`.
+- `src/parse.rs` — comment-masked regex scans for Python, JS/TS, Rust, Go, C/C++, C#, and Java. Named JS/TS re-exports are followed to depth 8 (`via_barrel`). An indented Python import is `conditional`. `import()` is `dynamic`. JS/TS resolution reads `tsconfig.json` `paths`, `package.json` `exports`, and npm `workspaces`. Rust resolves `mod`, `crate::`, `self::`, `super::`, and package or lib names from the root `Cargo.toml` and `[workspace] members` (`-` to `_`, globs included). `crate::` stays in the file's crate; a member name points at that member's library. This is not a CPython `ast` parser. The accuracy gate is `accuracy_fixtures_match_frozen_edges`.
 - `src/cache.rs` + `src/cache_commit.rs` — text index at `.wikifier_staging/wk_cache` (per-file payload, `by_src`, `by_dst`, notes, barrels). One commit per `update-maps`. Size and mtime skip unchanged files. `--full` rehashes. `src/legacy_cache.rs` imports an empty cache from `import_cache.sqlite` (the `sqlite3` CLI) or `import_cache.json`.
 - `src/artifacts.rs` — `file_health.json` (schema version 2) plus the `file_health.md` view, pending lines, and the journal. Identical journal retries within 120 seconds are skipped. `mark_green` stores `public_symbols`, `wiki_hash`, and `source_mtime_ns`, and refuses a Python or JS/TS wiki that misses the current public names, still mentions a removed name in backticks, or is unchanged after the public surface changed.
 - `src/maps_out.rs` — `library.md` (a file tree, a Mermaid block capped at 80 edges, and the dependency table) and sharded folder cards at `.wikifier_staging/maps/_root.json` and `maps/{prefix}/_index.json`.

README.md

@@ -32,7 +32,7 @@ Works from small scripts to large monorepos. **Deep import/include maps** (zero-
 | C# | `.cs` | `using` namespaces |
 | Java | `.java` | `import` / `import static` |
 
-Health/journal still work for any monitored path. The non-Python parsers are pragmatic regex (not full cargo/`go.mod`/classpath/`-I` resolution). On huge monorepos, split scope deliberately:
+JavaScript and TypeScript resolution also reads `tsconfig.json` `paths`, `package.json` `exports`, and npm `workspaces`. Health/journal still work for any monitored path. The non-Python parsers are pragmatic regex (not full cargo/`go.mod`/classpath/`-I` resolution). On huge monorepos, split scope deliberately:
 
 | File | Surface |
 |------|---------|
@@ -135,6 +135,25 @@ Warm **0-dirty** re-runs after **4.6.7** (same machine class; scoped; candidates
 
 Residual floor on large scopes is mtime/stat + live count under MapScope (not full JSON re-walk). Sub-100ms is not a hard SLA on every 1k+ tree.
 
+## Benchmarks
+
+Python 4.7 at `24812ef` versus this Rust release build, 5 runs, median milliseconds (min–max). Machine: Intel Xeon, 4 cores, 15.6 GiB RAM, Linux 6.12.94+, Python 3.12.3, rustc 1.83.0. Cold deletes `.wikifier_staging` first. Warm is the next `update-maps`. Full notes, peak RSS, and commands: [`scripts/benchmark_results.md`](scripts/benchmark_results.md). Reproduce with `scripts/benchmark_parity.sh`.
+
+| Command | small Python | small Rust | medium Python | medium Rust | large Python | large Rust |
+|---|---:|---:|---:|---:|---:|---:|
+| startup (`--version`) | 63 (61–66) | 2 (1–2) | same pair | same pair | same pair | same pair |
+| cold update-maps | 123 (121–126) | 8 (7–13) | 167 (163–168) | 62 (59–73) | 12675 (12540–13094) | 3142 (3114–3305) |
+| warm update-maps | 69 (67–72) | 4 (3–5) | 73 (72–74) | 4 (4–5) | 422 (417–433) | 246 (243–253) |
+| session-bootstrap | 66 (64–67) | 2 (1–2) | 65 (64–67) | 2 (1–2) | 270 (269–278) | 21 (21–21) |
+| check-changes | 64 (62–65) | 3 (2–3) | 70 (69–75) | 5 (5–6) | 1232 (1217–1266) | 74 (73–76) |
+| health --summary | 60 (60–62) | 2 (1–2) | 60 (59–63) | 1 (1–2) | 109 (108–114) | 8 (8–9) |
+| verify-wikis | 60 (60–61) | 2 (1–2) | 62 (60–67) | 1 (1–2) | 106 (103–108) | 8 (7–8) |
+| cycles | 60 (60–61) | 2 (1–2) | 61 (60–63) | 2 (2–2) | 63 (61–64) | 54 (53–55) |
+
+Resolved file edges after a warm map: small 2 and 2, medium (this repo) 44 Python and 98 Rust, large (Django `django/` + `tests/`, 2939 Python files) 9074 and 9074. Rust cold `update-maps` peak RSS was 3384 / 4424 / 40012 kbytes; Python was 27544 / 28392 / 97564 kbytes. The release binary is 2197456 bytes.
+
+Small is three modules under `src/pkg`. Medium is `src/` and `tests/`. Large is a shallow clone of Django.
+
 ### Parser accuracy
 
 `cargo test` checks internal edges on the frozen accuracy layouts:
@@ -163,13 +182,13 @@ Tests: `cargo test` (std-only Rust; accuracy fixtures, cache lookups, and the ag
 | `wikifier verify-wikis [dir]` | Re-check every Green wiki against its code (exit 1 on failures) |
 | `wikifier record-deletion <file> "reason"` | Mark removed paths 🔴, drop them from the graph, prune barrel refs |
 | `wikifier dependencies <file> [--full]` / `dependents <file>` | What a file imports (compact; `--full` for per-edge confidence records) / who imports it (JSON) |
-| `wikifier suggest-next` | Next actions (🔴/actionable 🟡 only; `--json` for `actions[]`) |
+| `wikifier suggest-next` | Next actions (JSON `actions[]`; `--text` for prose). Includes low-confidence edges when any exist |
 | `wikifier update-maps [--directory=src/] [--max-files=N]` | Rebuild graph + human `library.md` + agent folder maps (warm 0-dirty is fast; honors `map_paths.txt`) |
 | `wikifier cache-status` | `wk_cache` backend and mapped file count |
-| `wikifier health [--summary\|--json]` | Health matrix (machine-friendly flags) |
+| `wikifier health [--summary\|--format text]` | Health matrix. Default is JSON. `--summary` is counts only. `--text` or `--format text` is prose |
 | `wikifier validate` | Missing wiki rows + ghost paths |
-| `wikifier cycles [--json]` | Circular deps + break hints |
-| `wikifier heal-stubs [--dry-run]` | Promote Initial stubs that now have a real wiki |
+| `wikifier cycles [--no-analysis] [--max-items N]` | Circular deps, break hints, severity, and who is most depended on |
+| `wikifier heal-stubs [--dry-run]` | Promote Initial stubs whose wiki is at least 350 characters |
 | `wikifier monitor` / `daemon` | Background maintenance (`WIKIFIER_DAEMON_MAPS=0` for check-only) |
 | `wikifier serve` | Localhost dashboard with Run/Stop |