Two new commands, three fixes, and a benchmark that killed a feature - #2
Merged
Merged
Conversation
`impact <file>` seeded its traversal from the File node and walked inbound CALLS. Call edges attach to symbols, never to files, so a file argument could only ever return "no dependants found" — while `preflight` on the same file, using `symbols_in_file`, correctly reported 199. Two commands contradicting each other about one file is worse than either being silent, because an agent acts on the answer it was given. Seed a file's symbols alongside the file itself, and follow inbound IMPORTS so file-level importers are reported next to symbol-level callers. Symbol arguments are untouched. A file every module imports has hundreds of dependants, so the renderer leads with the count and shows the nearest few as evidence rather than spending an agent's budget to say "a lot". The corpus-wide `impact >= preflight` invariant is satisfiable by IMPORTS edges alone, so the test also pins what actually broke: a file's affected list must contain a caller, which fails without this change.
…ready does `classify_phrase` tested the subject with a raw substring match while the polarity test on the next line used the word-boundaried `contains_word`. The asymmetry read `must-revalidate` as the `validation` subject — `validate` sits inside `revalidate` — with `must` supplying the polarity, so an HTTP cache header was mined as a business rule at 0.97 confidence. `contains_word` already falls back to substring for multi-word needles like "credit limit" and for non-ASCII ones, so the multilingual corpus is unaffected. Django's mined rules go 180 -> 152. All 30 losses were inspected: every one is a test method name, a mangled documentation snippet, or the same bug in French via premise/remise. Two genuine rules are newly gained. Known and deliberately out of scope: the `validation` subject lacks the inflections `approval` has, so prose like "the service validates the order" is now missed where the loose match caught it incidentally. Restoring those inflections would also restore the test-name noise this removes; that is a tuning decision about the subject table, not part of this bug.
`prompt()` hardcoded "(ERPNext)" with no parameter, so every model-in-the-loop run against Medusa, OFBiz and OpenMRS told the model it was working on ERPNext while asking about a different codebase in a different language. It does not obviously favour one arm — every condition shared the same wrong name, and the prompt-differs-only-in-context test still holds — but it is a validity defect in three published tables. The name now comes from `TaskSet::repository`. The three affected reports carry a dated note stating the defect and its scope rather than being silently regenerated from runs that were not redone.
`reify verify` — a post-flight check that reads an agent's diff and reports what the patch missed — is only worth building if the call graph can support it. This measures that before the feature is written. For each qualifying merged commit the parent tree is indexed, one file's only hunk is withheld, and the truncated patch goes to the checker. The same commit then goes through complete, where every finding is a false positive by construction. That negative control is what stops the metric rewarding a checker that simply shouts. The checker is the shipped graph query reached through `query::impact`, not a new one, so this measures the substrate the decision needs. The pre-registered condition — recall below 0.25 or false alarms above 0.1 per commit means do not build — was written into metrics.rs before the first run. It fires on all three repositories. Recall is 0.50 on Rust, 0.10 on Python and 0.40 on Go, but false alarms run 4.4 to 23.5 per already-complete commit. It fails on noise, not blindness: a CALLS edge says a caller exists, not that the caller needed changing, and nothing in the graph separates a changed signature from an edit inside a body. No rewrite around that edge removes it. `reify verify` is therefore not built. 116s, no model, no network. AGENTS.md records the verdict so it is not rebuilt without beating these numbers first.
The README disagreed with itself in ways that undercut the measurement discipline it is arguing for. - Three different answers for its own index timings. The install section said 4.6s/0.7s where the measured table says 4.2s/0.49s. Unified to the measured numbers. - It said `init` appends "a six-line block", then showed a four-line block and called it "the same four lines". Neither matched what the tool writes. It now shows the actual constant, including the two lines the paraphrase dropped. - Shell completions were documented twice, forty lines apart. - The quickstart and Install repeated the same commands and the same uninstall explanation; Install now carries only what the quickstart does not. - The optimisation war stories are compressed to a paragraph pointing at the changelog, keeping the `git log -L` lazy-fetch story because it is load-bearing for the privacy claim rather than for speed. Adds the verify-benchmark section, and records that `impact` now takes a file. Net 48 lines shorter. Every measurement section — Medusa's no-win row, "where it doesn't work", the p-value section — is untouched.
Nothing told anyone whether Reify would help before they invested in it. The README FAQ answered it informally and `benchmarks/REPORT-medusa.md` already shows the advantage can be zero on a real repository. `reify doctor` makes that measured instead of anecdotal, and runs before there is an index — the whole point is deciding before committing to the tool. The signals were fitted against the four repositories in `benchmarks/`, not invented. Two hypotheses were tested and dropped: size inverts (OFBiz wins biggest at 1,364 code files, Medusa ties at 11,821), and so does the obvious reading of "history and file naming share vocabulary" — pooled corpus-wide, Medusa scores 0.43 against OFBiz's 0.38. Two signals do fit all four outcomes, each explaining a different failure: repo grep margin commit focus subject->path OFBiz +58 0.96 0.80 ERPNext +48 0.98 0.85 OpenMRS +9 0.98 0.48 Medusa 0 0.84 0.79 Commit focus is the share of commits touching few enough files that their subject says anything about them; Medusa is the only measured repository where it falls away and the only one where Reify won nothing. The mechanism is already load-bearing elsewhere: `History::co_changes` discards commits touching more than 20 files for the same reason, so `doctor` uses the same threshold rather than a second definition of a sweeping commit. No suitability score. `docs/metrics.md` forbids printing a number that cannot be defined, and a weighted blend of heuristics tuned on four repositories is exactly that. Named signals, measured values, a plain verdict, and — where the answer is no or marginal — what would change it. The verdict floor is 200 commits, which is where a 95% Wilson interval around Medusa's 0.84 lies wholly below the threshold; at 50 it straddles it, so a shorter history is reported as short rather than condemned. Verified on all four benchmarked repositories, which land where their reports say they should, plus django (yes) and a two-commit scratch repository (too small). No new dependency; the offline guarantee is untouched.
`reify init --write-agent-instructions` handled one file, chosen from a list four filenames long. `reify install` detects which agents this repository is actually configured for and wires each one in its own format and location, shows its plan before writing, and is fully reversed by `reify uninit`. It installs the shell-command instruction block, not MCP. That is the existing reasoned position in docs/integration/claude-code.md — an MCP server's tool schemas are re-sent on every turn of every session, and for a tool whose purpose is reducing context, a per-turn tax to deliver it is self-defeating — and nothing here overturns it. Every agent this can detect runs shell commands, so level 0 is right for all of them. `--mcp` is the deliberate opt-in and says what it costs before it writes anything. Two rules do most of the work: Detection requires evidence *in the repository*. `~/.cursor` says the user has Cursor installed, not that this repository is worked on with it; creating a `.cursorrules` on that basis is exactly the guess the command must not make. Home directories are read, reported as corroboration, and never acted on alone — an agent seen only there is listed so its absence from the plan is explained. Every detection prints what it rests on. Everything written stays inside the repository. A machine-wide MCP registration cannot be undone by a per-repository `uninit` without breaking every other repository relying on it, so the MCP entries written are the repository-scoped ones and a client with no such config gets the block instead, with the reason stated. Config merging is textual, not a serde round-trip: re-serialising sorts a user's keys and drops their formatting, and the file is theirs. Adding the entry to a hand-written `.mcp.json` changes exactly one line, an unrelated server's env block survives byte for byte, removal restores the original exactly, and the result is parsed before it is written so a bad splice fails loudly. A config that exists but does not parse is reported and skipped — never overwritten. `uninit` derives its removal targets from the same table `install` plans from, so a new agent cannot be added to one without appearing in the other, and a test asserts it. The MCP surface stays at six tools: these are operator commands and belong to the CLI.
The quickstart still taught `init --write-agent-instructions`, the hand-wiring path, and skipped the question `doctor` exists to answer. It now runs the two commands in the order someone actually needs them: should you use this, then wire it. The Install section gains the same pair, with what `install` detects, that an agent present on the machine but not configured in the repository is reported rather than written to, and that nothing outside the repository is touched. Both blocks were run as written against a scratch repository before being documented.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight commits: two new commands, three correctness fixes, one new benchmark, and two documentation passes. The benchmark is the important one — it was built to decide whether a feature was worth writing, and it decided no.
A feature measured, then not built
reify verify— a post-flight check reading an agent's diff and reporting what the patch missed — looked like the strongest thing on the roadmap.reify-bench verify-evalmeasures whether the call graph can actually support it: withhold one hunk from a real merged commit, ask the graph what the patch missed, then run the complete commit through the same query, where every finding is a false positive by construction. That negative control is what stops the metric rewarding a checker that simply shouts.The condition was written into
metrics.rsbefore the first run: recall below 0.25, or false alarms above 0.1 per commit, means do not build.It fails on noise, not blindness. A
CALLSedge says a caller exists; it does not say the caller needed changing, and nothing in the graph separates a changed signature from an edit inside a body. No rewrite around that edge removes it. Counting only findings the complete commit does not also produce, recall drops to 0.00 / 0.05 / 0.15 — most apparent hits were files the checker would have named anyway.Model-free, deterministic, 116s for three repositories.
AGENTS.mdrecords the verdict so it is not rebuilt without beating those numbers first.Two new commands
reify doctor— should this repository use Reify at all, answered before there is an index. Grounded in the four-repository benchmark's own measured predictor rather than invented heuristics: what separated the repositories where Reify helped from the one where it did not is whether commit subjects speak the vocabulary the code is named in. Candidate signals were validated against all four repositories, and commit focus is the only one that isolates Medusa; it reusesHistory::co_changes' existing 20-file threshold rather than inventing a second definition.It is willing to say no. A 2-line repository gets
TOO SMALLand nothing else is measured. This repository getsMARGINAL— at 53 commits the intervals straddle the thresholds, so a verdict either way would be an artefact of the sample. No suitability score: named signals, measured values, and what would change a no. Every run discloses that it is a heuristic fitted to four repositories and points atreify-benchfor measuring a specific one.reify install— detect the agents configured here and wire each one. It keepsdocs/integration/'s existing position: level 0 by default,--mcpopting into the per-turn cost with a warning before it writes.Detection requires repository evidence. An earlier build used home directories as triggers and created
.cursorrulesandAGENTS.mdin a repository that had neither, purely because the machine has~/.cursor; home evidence now corroborates and is reported, never acted on alone. Nothing outside the repository is written, which drops Codex/Windsurf MCP registration — their configs are machine-wide and a per-repouninitcould not safely undo one — so those get the instruction block with the reason stated.Config merging is textual rather than a serde round-trip: adding to a hand-written
.mcp.jsonchanges exactly one line, an unrelated server'senvblock and the file's formatting survive byte-for-byte, anduninitrestores the original byte-identically. Asserted in tests.Three correctness fixes
reify impact <file>returned nothing whilereify preflighton the same file reported 199 dependants. It seeded from the File node and walked inboundCALLS, but call edges attach to symbols. Two commands contradicting each other about one file is worse than either being silent, because an agent acts on the answer it was given.validatesits insiderevalidate,mustsupplied the polarity. The polarity test one line away was already word-boundaried. Django's rules go 180 → 152; all 30 losses were inspected individually and every one is a test name, a mangled doc snippet, or the same bug in French via premise/remise.(ERPNext), so three published tables told the model it was working on ERPNext while asking about Medusa, OFBiz or OpenMRS. Those reports carry a dated note rather than being regenerated from runs that were not redone.Documentation
The README disagreed with itself: three different answers for its own index timings, a "six-line block" that showed four lines and matched neither the tool's output nor itself, completions documented twice, and the quickstart duplicating Install. Net 48 lines shorter, and the quickstart now leads with
doctorandinstall— both blocks run as written against a scratch repository before being documented.Every measurement section is untouched, including Medusa's no-win row and the end-to-end p-value section.
Test plan
cargo test --workspace— 426 passingcargo clippy --workspace --all-targetscleanreify doctoron this repository, django, and a 2-line scratch repository: three distinct, defensible verdictsreify installplan → apply → rerun →uninit, with an unrelated.mcp.jsonserver surviving byte-identicallyreify impact <file>reports 208 againstpreflight's 199, with a regression test that fails without the fixreify-bench verify-evalreproduces the table above in 116s with no model