Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Launchers: bash needs LF, cmd.exe needs CRLF (labels misparse in LF-only batch files).
*.sh text eol=lf
*.bat text eol=crlf
*.ps1 text eol=crlf
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AGENTS.md

Entry point for any coding agent working in this repository, or using Wikifier on another tree. The day-to-day contract is `skills/run.md` (Agent Protocol v0.7, package 5.0.0). Read that file before substantive work. This file says what is live, what is history, and which rules are easy to miss.
Entry point for any coding agent working in this repository, or using Wikifier on another tree. The day-to-day contract is `skills/run.md` (Agent Protocol v0.7, package 5.1.0). Read that file before substantive work. This file says what is live, what is history, and which rules are easy to miss.

## What this program is

Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md.wiki.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@

**Role:** Entry point for any coding agent in this repository.

It points at `skills/run.md` (protocol v0.7, package 5.0.0) and names the live Rust binary, the daily loop, and the Python-era records kept under `Findings/archive/`. `pip install wikifier` is the archived Python 4.7 package. Map roots stay `src/` and `tests/`. The import scan stays comment-masked regex plus named re-exports. Rust also resolves `crate::`, `self::`, `super::`, root crate names, and `[workspace] members`, so `src/main.rs` and `tests/*.rs` are dependents of the library modules they import. `mark-green` checks Rust `pub` names, impl methods, and trait-impl items.
It points at `skills/run.md` (protocol v0.7, package 5.1.0) and names the live Rust binary, the daily loop, and the Python-era records kept under `Findings/archive/`. `pip install wikifier` is the archived Python 4.7 package. Map roots stay `src/` and `tests/`. The import scan stays comment-masked regex plus named re-exports. Rust also resolves `crate::`, `self::`, `super::`, root crate names, and `[workspace] members`, so `src/main.rs` and `tests/*.rs` are dependents of the library modules they import. `mark-green` checks Rust `pub` names, impl methods, and trait-impl items.
32 changes: 32 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [5.1.0] - 2026-11-05

Agent-facing fixes found by driving the MCP server like an agent on real repositories, Windows fixes, and a speed pass. Protocol stays v0.7.

**MCP clients, note:** `session_bootstrap` and `get_project_status` now list at most 100 files unless the call passes `limit` (`files_total` gives the full count, `limit: 0` lists every file), and `update_maps` leaves out the edge list unless `edges: true`. Orient with `map_index` and `list_paths`. The CLI defaults for both commands are unchanged.

### Fixed
- Windows: `wikifier.bat` passes the binary's exit code through. It read `%ERRORLEVEL%` inside a parenthesized block, so every failure exited 0.
- Windows: health baselines hash CRLF as LF (`content_sha256_prefixed`). A `core.autocrlf` checkout no longer flips every Green file to dirty (51 files in this repo).
- A file argument written with `\` separators or as an absolute path under the root names the same row as `src/a.rs`. Before, an absolute path returned empty lookups and a backslash path created a second health row.
- `prepare_edit`, `get_dependencies`, `get_dependents`, and `get_file_wiki` return `FileNotFound` with a `list_paths` hint for a path that is neither on disk nor in the map, instead of `success: true` with empty fields.
- MCP `tools/call` with a missing required argument (for example `file_path` instead of `file`) returns `InvalidArguments` naming the accepted and received arguments. Before, the tool ran with an empty path and reported success.
- `health --limit N` (and MCP `limit`) caps the entry list and adds `entries_total`. The argument was advertised and ignored.
- `pending_updates.md` no longer says "(no pending items …)" above a list of items. The note is dropped when the first item arrives and comes back when the last one is removed. This was carried over from the Python package.
- A `record-change` pending item reads `- path: <reason> — refresh the wiki, then mark-green` instead of the Python wording `LLM/agent edit — <reason>`.

### Performance
Same outputs: on a 587-file Cargo workspace the cache files, `library.md`, folder cards, health hashes, and the `update-maps`, `cycles`, `dependencies --full`, `dependents`, `check-changes`, and `session-bootstrap` output were byte-identical before and after (CLI, timestamps aside). Warm MCP calls there, median of 5 on Windows 11: `check_changes` 353 → 53 ms, `update_maps` 708 → 50 ms, `get_cycles` 317 → 39 ms, `session_bootstrap` 33 → 16 ms and 21.7 → 4.7 KB. Cold `update-maps` (CLI, runs interleaved on a loaded machine, median of 3) went from 18.7 s to 6.4 s.
- Derived cache files (`by_src`, `by_dst`, `payload`, `notes`, barrel records) are written on several threads through a temp file and a rename, so a reader never sees a torn file, and are no longer synced. `wk_cache/sizes.tsv` records each file's length; `update-maps` compares it with one directory listing per kind and treats a missing, resized, or unreadable file as a miss: the source is parsed again or the `by_dst` record is rewritten from the index. Without `sizes.tsv` every file is read and checked once. Lookups that meet a bad record answer from `index.tsv`. On this Windows machine the sync change made no measurable difference to a cold map (median 11.9 s with batched syncs, 12.5 s without, same copy, interleaved, machine at about 80% load): the remaining cost is creating about 2,000 small files.
- `file_health.json`, `file_health.md`, the journal, and `pending_updates.md` are now replaced through a synced temp file and a rename. They were plain overwrites with no sync.
- A full map scans each file once; the second JS/TS/Rust resolution pass reuses those scans. Import line numbers come from one substring search per distinct spec instead of a line-by-line scan per import. JS/TS barrel follow-through builds its sorted scan table once instead of once per imported name.
- `cycles` and `dependencies --full` read edges from `index.tsv` instead of one `by_src` file per mapped file, and `cycles` loads edge flags only for files inside a cycle. `library.md` reads `by_src` records without the notes files.
- `update-maps` keeps `library.md` and the folder cards when the cache commit changed nothing.
- `check-changes` and health seeding take a mapped file's content hash from its cache row when size and mtime match (the rule `update-maps` already used), and read the file otherwise or when that hash is not the baseline. Health rows are found by path through a map, not a scan per file.
- The file walk takes directory and file types from the directory entry instead of two `stat` calls per entry.

### Changed
- `init` seeds `map_paths.txt` and `monitored_paths.txt` with `src/` when it exists, otherwise with each top-level directory that holds source (a Cargo workspace's `crates/`, a Python package directory, `tests/`). Before, a project without `src/` hit a scope error on the first `update-maps`.
- MCP `update_maps` omits the edge list unless `edges: true`; `edges_persisted` still gives the count. On a 587-file Cargo workspace the list was about 280 KB. The schema now lists `full`, `max_files`, and `edges`; `health` lists `limit` and `summary`.
- `.gitattributes` keeps `*.sh` LF and `*.bat` / `*.ps1` CRLF on every checkout.
- `session-bootstrap --limit N` (MCP `limit`) caps the `files` list and adds `files_total`. Over MCP the default cap is 100; `limit: 0` lists every file. `map_index` and `list_paths` are the way to orient on a large tree.

## [5.0.0] - 2026-11-03

First release of the std-only Rust rewrite. `[dependencies]` is empty: no crates.io crates and no Python runtime. The binary implements the Python 4.7 agent contract. Protocol v0.7 stays. `pip install wikifier` still installs that Python 4.7 package.
Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

Wikifier is a **zero-dependency, agent-to-agent codebase wiki**: it gives AI agents a token-efficient, queryable map of a codebase (health matrix, dependency graph, per-file wiki summaries) so they can look things up instead of re-reading full sources. The human layer (`index.html` dashboard) is strictly secondary — a read-only viewer of the same artifacts. It is deliberately NOT a general human documentation tool.

**This repo dogfoods itself.** Wikifier runs on its own source, and you are expected to follow its agent protocol when working here (see below). `AGENTS.md` is the entry point for any coding agent. The authoritative contract is `skills/run.md` (Agent Protocol v0.7, package 5.0.0). Read it before substantive work.
**This repo dogfoods itself.** Wikifier runs on its own source, and you are expected to follow its agent protocol when working here (see below). `AGENTS.md` is the entry point for any coding agent. The authoritative contract is `skills/run.md` (Agent Protocol v0.7, package 5.1.0). Read it before substantive work.

**USE WIKIFIER FIRST (non-negotiable in this repo, preferred on any managed target):** At session start call `session_bootstrap` (MCP or `wikifier session-bootstrap`). Prefer Core daily tools (`list_core_tools`): bootstrap, `check_changes`, `prepare_edit`, `suggest_next_actions` (json `actions[]`), `record_change`, `mark_green`. Look up via map/wiki/deps before re-reading full sources. Selective work only (🔴 / actionable 🟡). Never skip `record_change` after edits.

Expand Down Expand Up @@ -56,12 +56,12 @@ wikifier mcp # MCP server on stdio

## Architecture

Pipeline: **scan → parse → resolve → cache → health → artifacts**, implemented in Rust under `src/` (version in `Cargo.toml`, currently **5.0.0**). The Python package is no longer in this repository. `pip install wikifier` still installs Python 4.7. CLI (`src/main.rs`) and MCP (`src/mcp.rs`) both call `src/api.rs`. Every function returns a JSON object with `success`. Operational failures return `{"success": false, "error", "error_type"}`.
Pipeline: **scan → parse → resolve → cache → health → artifacts**, implemented in Rust under `src/` (version in `Cargo.toml`, currently **5.1.0**). The Python package is no longer in this repository. `pip install wikifier` still installs Python 4.7. CLI (`src/main.rs`) and MCP (`src/mcp.rs`) both call `src/api.rs`. Every function returns a JSON object with `success`. Operational failures return `{"success": false, "error", "error_type"}`.

- `src/api.rs` — the library surface: `session_bootstrap`, `check_changes`, `prepare_edit`, `suggest_next_actions`, `record_change`, `record_changes`, `record_deletion`, `mark_green`, `verify_wikis`, `update_maps` / `update_maps_run`, `dependency_report`, `load_dependents`, `cycles`, `init_project`, `serve_loop`, `monitor_loop`, `daemon_ctl`. `check_changes` watches mapped files, Green files, and `monitored_paths.txt`. `record_changes` reads `git status` (git is optional).
- `src/scope.rs` — project root, `map_paths.txt`, `monitored_paths.txt`, excludes, and the file walk. A missing configured map directory is a scope error. Three or more child projects make the root a container and the walk stops.
- `src/parse.rs` — comment-masked regex scans for Python, JS/TS, Rust, Go, C/C++, C#, and Java. Named JS/TS re-exports are followed to depth 8 (`via_barrel`). An indented Python import is `conditional`. `import()` is `dynamic`. JS/TS resolution reads `tsconfig.json` `paths`, `package.json` `exports`, and npm `workspaces`. Rust resolves `mod`, `crate::`, `self::`, `super::`, and package or lib names from the root `Cargo.toml` and `[workspace] members` (`-` to `_`, globs included). `crate::` stays in the file's crate; a member name points at that member's library. This is not a CPython `ast` parser. The accuracy gate is `accuracy_fixtures_match_frozen_edges`.
- `src/cache.rs` + `src/cache_commit.rs` — text index at `.wikifier_staging/wk_cache` (per-file payload, `by_src`, `by_dst`, notes, barrels). One commit per `update-maps`. Size and mtime skip unchanged files. `--full` rehashes. `src/legacy_cache.rs` imports an empty cache from `import_cache.sqlite` (the `sqlite3` CLI) or `import_cache.json`.
- `src/cache.rs` + `src/cache_commit.rs` — text index at `.wikifier_staging/wk_cache` (per-file payload, `by_src`, `by_dst`, notes, barrels). One commit per `update-maps`. Size and mtime skip unchanged files. `--full` rehashes. Cache files are derived: they are renamed into place but not synced, `wk_cache/sizes.tsv` records their lengths, and a missing, resized, or malformed file is a miss that the next `update-maps` re-parses. Health, journal, and pending writes are synced. `src/legacy_cache.rs` imports an empty cache from `import_cache.sqlite` (the `sqlite3` CLI) or `import_cache.json`.
- `src/artifacts.rs` — `file_health.json` (schema version 2) plus the `file_health.md` view, pending lines, and the journal. Identical journal retries within 120 seconds are skipped. `mark_green` stores `public_symbols`, `wiki_hash`, and `source_mtime_ns`, and refuses a Python or JS/TS wiki that misses the current public names, still mentions a removed name in backticks, or is unchanged after the public surface changed.
- `src/maps_out.rs` — `library.md` (a file tree, a Mermaid block capped at 80 edges, and the dependency table) and sharded folder cards at `.wikifier_staging/maps/_root.json` and `maps/{prefix}/_index.json`.
- `src/project_lock.rs` — advisory lock `.wikifier_staging/.lock`, taken before the cache lock, re-entrant per thread, 30 second wait, stolen when the recorded pid is dead.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md.wiki.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@

**Role:** Repo guidance for an agent working in this tree.

Hard constraints: empty Cargo dependencies, no hand-edits of `library.md` or the health and pending files, and the human dashboard stays secondary. The Python-era plan logs live under `Findings/archive/`. It points agents at `AGENTS.md` and `skills/run.md` (protocol v0.7, package 5.0.0). The architecture section names the Rust modules under `src/`, the library file tree, and this repo's map roots `src/` and `tests/`. The parse note includes Rust `crate::`, `self::`, `super::`, Cargo.toml crate names, `[workspace] members`, and `rust_public_symbols` for wiki checks. JSON failures use `success: false`.
Hard constraints: empty Cargo dependencies, no hand-edits of `library.md` or the health and pending files, and the human dashboard stays secondary. The Python-era plan logs live under `Findings/archive/`. It points agents at `AGENTS.md` and `skills/run.md` (protocol v0.7, package 5.1.0). The architecture section names the Rust modules under `src/`, the library file tree, and this repo's map roots `src/` and `tests/`. The parse note includes Rust `crate::`, `self::`, `super::`, Cargo.toml crate names, `[workspace] members`, and `rust_public_symbols` for wiki checks. JSON failures use `success: false`.
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "wikifier"
version = "5.0.0"
version = "5.1.0"
edition = "2021"
description = "Wikifier — zero-dependency agent-to-agent codebase wiki."
license = "MIT"
Expand Down
Loading
Loading