Add: R (.r/.R) language support via bespoke extract_r#1
Open
Sirhan1 wants to merge 42 commits into
Open
Conversation
…raphify-Labs#1870) The Graphify-Labs#1757 batch-scoping followup built the per-chunk allowlist by reading FileSlice.rel, which does not exist (a FileSlice carries its parent file in .path). So every chunk containing a sliced oversized document leaked the FileSlice object into the allowlist, save_semantic_cache raised TypeError on Path(FileSlice), and the best-effort except swallowed it: extraction finished but those chunks were never checkpointed, so a re-run or a crash/rate-limit resume re-billed them. Resolve each unit through the canonical unit_path() helper so a slice maps to its parent file. Adds a regression test that slices a real oversized .md and asserts the checkpoint writes without swallowing a TypeError. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…the unreleased 0.9.16 section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ify-Labs#1873) Non-anchored patterns from a nested .gitignore/.graphifyignore were matched against root-relative paths first, so a nested ignore file's patterns leaked outside its directory. In the wild, .hypothesis/.gitignore (a bare "*" auto-written by the hypothesis library) ignored the entire repository and detect() returned 0 files. Per gitignore semantics, patterns from A/.gitignore apply only to paths under A. Match every pattern against the path relative to its own anchor (the anchor dir itself exempt — an ignore file governs its directory's contents, not the directory), and skip patterns whose anchor does not contain the target. Regression introduced with nested-ignore support in 8a5287a (Graphify-Labs#1206). tests/test_detect.py: 143 passed, including the existing nested-ignore and nested-negation tests plus two new regressions. Fixes Graphify-Labs#1873 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hangelog for Graphify-Labs#1873/Graphify-Labs#1880 Graphify-Labs#1880 is the update-layer symptom of the Graphify-Labs#1873/Graphify-Labs#1887 nested-ignore subtree-scoping regression (fixed in fb4d452, @Alwyn93): a nested broad `.gitignore` zeroed update's re-scan, so it built 0 nodes and the shrink-guard refused to overwrite. Adds a _rebuild_code regression test asserting update sees the real files (and still scopes the nested ignore correctly) so this can't recur at the update layer. Records both regressions in the unreleased 0.9.16 changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-Labs#1865) A full graphify update evicted every edge whose source_file was a re-extracted source, including LLM semantic edges (e.g. semantically_similar_to) from Markdown docs that also have an AST extractor. The AST pass cannot regenerate those edges, so they were silently lost while their concept nodes survived — node eviction is provenance-aware via _origin (Graphify-Labs#1116), edge eviction was not. Tag AST-extracted edges with the same _origin=ast marker nodes already carry, and scope rebuild-driven edge eviction to that tier: re-extraction replaces a source's AST edges, while its semantic edges survive until a semantic re-extraction supersedes them. Deletion-driven eviction stays provenance-blind, so edges of deleted or excluded sources are still purged regardless of tier. Edges from graphs built before this change lack the marker; a stale AST edge from a file changed exactly once between the old and new version can linger until its source is deleted — the same migration trade-off the Graphify-Labs#1116 node marker made.
…the unreleased 0.9.16 section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…about real loss Node IDs are <source-path>_<entity>, so a doc that merely references an entity mints the entity's own ID and collides with the entity's node by construction. The pre-dedup pass kept whichever arrived first, so chunk order decided whether an entity kept its own attributes or a passing cross-reference's, and the warning that fired (Graphify-Labs#1504) told the user a same-name-different-directory clash had lost their data — while the one drop that really is lossy, two labels for one ID from the same file, was silent because the warning was gated on source_file differing. The survivor is now the node whose source_file is the file its ID encodes (any trailing slice of the path, so absolute, repo-relative, and pre-Graphify-Labs#1504 bare-stem IDs all resolve), falling back to first-seen. Reporting follows what the drop actually costs: a cross-reference folding into the node it references loses nothing (edges are keyed by ID and rewire to the survivor) and is silent; a same-file relabel notes the discarded label; two files that both encode the ID are distinct entities and still WARNING, now stating the ID-scheme cause rather than a filename clash. Refs Graphify-Labs#1851
…phify-Labs#1851 followup) Two tweaks on top of @bchan84x's Graphify-Labs#1852: 1. The definer heuristic decided the survivor pairwise, so when several same-file nodes co-defined one ID the surviving label still depended on arrival order (the exact 3-node case Graphify-Labs#1851 reports). Choose the survivor by a total order (definer first, then shorter/canonical label, then lexically) via a min over _collision_rank, so it is order-independent. Direction preserves Graphify-Labs#1504 (lexically-first source path wins). 2. _defines_id now also recognizes a bare file-level node whose id is exactly the slugified path (nid == prefix), not only <path>_<entity>. Adds an order-independence test over all 6 permutations of definer + same-file relabel + cross-file reference, and a bare-file-node test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s#1886) --exclude was honored only by the initial extract scan; update/watch/hook rebuilds re-ran detect() without it and silently re-indexed excluded paths. extract now writes the patterns to a graphify-out/.graphify_build .json sidecar, and _rebuild_code reads and re-applies them on every rebuild (layered after the ignore files, so they still win). Pre-existing graphs without the sidecar behave as before. Adds a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1883) `module.function()` where `module` is imported produced no calls edge, while bare-name calls did. The member call was captured, but the shared cross-file pass skips all member calls and _resolve_python_member_calls only handled capitalized class receivers, so a lowercase module receiver fell through. Add a module arm: a lowercase receiver naming a module imported into the caller's file resolves to the single callable that module contains. Keyed on stable node ids (imports edge source is the caller file node; contains maps file -> children), not source_file strings, so it holds under the cache_root id-remap relativization. Also drop the no-classes early return that skipped the whole resolver when a corpus had functions but no class methods. Guards: only modules imported into the caller's own file match (so self/obj/local instances never match), and a single-definition god-node guard on the callable. Adds a positive test and a false-edge negative. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bs#1837) Graphify-Labs#1837 (update silently fails to discover new files) was the same nested-gitignore scoping bug (Graphify-Labs#1873/Graphify-Labs#1880), already fixed. The existing test only does a single build; add the build -> add new file+dir -> update -> discovered sequence the report describes, with a nested `*` scratch dir as a scoping guard. No production change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_kotlin_collect_type_refs had no equivalent of _JAVA_BUILTIN_TYPES / _PYTHON_ANNOTATION_NOISE / _GO_PREDECLARED_TYPES, so every Boolean/String/ Int/List/... type annotation in a .kt file became a real graph node/edge. Unrelated files sharing only these language-level types were getting merged into the same community by cluster-only. Add _KOTLIN_BUILTIN_TYPES (kotlin.* scalars, collections, throwables - deliberately not Android/JVM framework types like Context/View/Bundle, matching how _JAVA_BUILTIN_TYPES doesn't filter framework types either) and check it plus _JAVA_BUILTIN_TYPES (Kotlin freely references java.util.Date/UUID/etc) at the three call sites that previously appended unconditionally. Deliberately excludes "Result" from the filter list even though kotlin.Result exists - it collides with the very common user-defined sealed-class name (confirmed against this repo's own sample.kt fixture, which defines its own Result<T>). Verified on a ~400-file Android/Kotlin project: 3693->3239 nodes (-12%), 6481->5221 edges (-19%), with a previously merged 5-file grab-bag community cleanly splitting out an unrelated screen-dimension utility.
…ify-Labs#1893 in the unreleased 0.9.16 section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…abs#1899) Completes Graphify-Labs#1789. Two residual leaks into committed graph.json: (a) Out-of-root reference targets (a .csproj ProjectReference, .sln project, or bash `source` pointing outside the scan root) kept an absolute source_file and an absolute-derived id, because the relativization post-passes only handled paths under root and silently left out-of-root ones absolute. They now get a portable walk-up relative source_file and an `ext_`-namespaced id (basename fallback for far-outside or cross-drive targets), with edge endpoints remapped. (b) A symbol whose name normalizes to nothing (minified `$`, a JSONC `"//"` key) collapsed _make_id(stem, name) to the bare absolute file stem, leaking the path and colliding with the file node. Guard at the three mint sites (json_config keys, engine function names) to skip these no-signal symbols. Adds regression tests: an out-of-root ProjectReference stays portable (no scan-path in any id/source_file/edge), and a minified `$` is dropped while the real function survives. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1890) A semantic chunk can return a clean, non-empty response that omits some of the documents it was given. Those docs vanished from the graph with no node, no warning, and no cache/manifest stamp, so they were silently re-dispatched (and typically re-omitted) on every run. extract_corpus_ parallel now diffs the dispatched file set against the source_files that returned, records the gap in merged["uncovered_files"], and prints a loud warning naming the omitted files. Smallest visibility guard; routing docs through the deterministic extractor for a guaranteed file node is a separate, larger change. Adds a regression test: a chunk that omits odd-numbered docs is caught and warned, not silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… classify .skill files as documents (Graphify-Labs#1901) - pg_introspect: the psycopg-missing ImportError suggested pip install 'graphify[postgres]', but the PyPI package is graphifyy (double-y) - detect: add .skill (Markdown with YAML frontmatter agent files) to DOC_EXTENSIONS so they are no longer dropped as unclassified - extract: route .skill through extract_markdown for the structural quick-scan Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ms (Graphify-Labs#1900) Full-sentence non-English queries picked wrong BFS seeds because _QUERY_STOPWORDS was English-only: in a mostly-English code corpus, fillers like wie/die/das/funktioniert are rare, get high IDF weight, and out-seed the real keyword (~175x score collapse). Extend the frozenset with a curated German set plus trimmed French/ Spanish/Portuguese/Italian question/filler words, diacritics intact. English-collision words (war, bald, comment, come, son, sin, con, pour, des) are deliberately omitted; die/hat are kept since the all-stopword fallback and unfiltered find_node protect English use. Filtering stays in _query_terms, so the per-term seed guarantee in _pick_seeds composes unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s#1896) Re-exporting into an existing vault left notes for nodes that dropped out of the graph, and rewriting the manifest to only this run's files disowned those orphans - so a returning node's stale note became permanently unwritable. Before rewriting the manifest, delete stale = owned - written - skipped. This only ever touches files graphify itself wrote (foreign files go to _skipped, never the manifest), and each path is containment-checked against the vault dir to defuse a corrupt/hostile manifest with ../ entries. The manifest rewrite then correctly drops the pruned files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix Graphify-Labs#1907: _hooks_dir parsed .git/config with a strict configparser, which rejects git-legal duplicate keys/sections (VS Code writes such configs) and printed a spurious "could not read core.hooksPath" warning on every hook command. Ask git itself (rev-parse --git-path hooks) as the primary resolution instead; it handles core.hooksPath, includeIf and linked worktrees, keeps the Windows-path and multiline/NUL guards, and surfaces git's own stderr when the config is genuinely corrupt. The .git/hooks final fallback is unchanged. Fix Graphify-Labs#1902: hook install never registered the graph.json union merge driver that README and CHANGELOG 0.7.0 document. install() now sets merge.graphify.name/driver via git config (pinning sys.executable with the same shell-safe allowlist the hook scripts use, falling back to the graphify launcher) and idempotently appends the graph.json merge=graphify line to .gitattributes without clobbering other entries. uninstall() mirrors the removal, and status()/install() report the merge-driver state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1895) The Graphify-Labs#1757 cache guard refuses to write a cache entry for a node whose source_file resolves to a real corpus file that was not dispatched, but the node itself still flowed into merged["nodes"] and landed in graph.json (plus any edges/hyperedges built on it). extract_corpus_ parallel now filters the merged result right before the Graphify-Labs#1890 dispatched-vs-returned reconciliation: a node is dropped when its source_file resolves (against root, same normalization as Graphify-Labs#1890) to an existing file (.is_file(), mirroring the Graphify-Labs#1757 condition) outside the dispatched set. Non-file source_files (concepts, model-invented anchors) pass through untouched. Edges whose endpoint and hyperedges whose member is a dropped node id go with it, plus any edge/hyperedge itself attributed to an undispatched real file. One summary warning names the offending files and merged["out_of_scope_dropped"] records the count. Running before the reconciliation keeps covered/uncovered reflecting the post-filter graph. Regression tests: a chunk over A.md+C.md returning a stray B.py node loses the stray (and its edge/hyperedge) while sibling and concept attributions survive; a clean run records a zero count and no warning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hify-Labs#1897) The Graphify-Labs#933 manifest filter built its extracted-set from node/edge source_file values, which are root-relative on a fresh extraction, and compared them against files_by_type entries, which are absolute (from detect()). The raw string membership test therefore never matched, so every freshly-extracted semantic doc was dropped from the manifest and re-queued as changed on the next run; only code files and cache-replayed docs got stamped. The filter now lives in _stamped_manifest_files(), which resolves BOTH sides against the scan root before the membership test — the same Path/is_absolute/resolve normalization the Graphify-Labs#1890 reconciliation uses in graphify.llm. Genuinely omitted zero-node docs still have no source_file entry and stay unstamped, preserving the intentional Graphify-Labs#933 re-queue behavior. Regression tests: a CLI extract with a mocked corpus extractor returning root-relative source_files lands the doc in manifest.json with a non-empty semantic_hash while a zero-node doc stays out; a unit test covers relative (fresh) and absolute (cache-hit) source_file shapes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bs#1897 Graphify-Labs#1902 Graphify-Labs#1907 Graphify-Labs#1896 Graphify-Labs#1900 Graphify-Labs#1901 Graphify-Labs#1906) Bumps to 0.9.17 (unreleased) and records the batch implemented via Fable subagents in isolated worktrees, integrated onto v8: out-of-scope node drop, manifest stamping, merge-driver registration, hooksPath configparser fix, obsidian prune, multilingual query stopwords, .skill classification, and the postgres package-name hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ot misreported as deleted (Graphify-Labs#1908 Graphify-Labs#1909) Two coupled excluded-vs-deleted fixes: Graphify-Labs#1909 — incremental extract's prune set was derived from the manifest alone (manifest - corpus), so a file that became excluded without ever being manifest-listed (every pre-Graphify-Labs#1897 graph) kept its stale nodes in graph.json forever. The prune set is now also derived from the existing graph's own node source_files reconciled against the post-exclude detect corpus (_stale_graph_sources), restricted to in-root paths; out-of-root --include/symlinked entries and remote (://) sources are never pruned. Relative source_files are anchored against both the scan root and the --out root (the Graphify-Labs#555/Graphify-Labs#1899 relativized form). The --no-cluster incremental early exit never runs build_merge, so an exclusion-only change now prunes the raw graph.json in place instead. Graphify-Labs#1908 — save_manifest retained any prior row whose file still existed on disk, so an excluded-but-alive file survived as a permanent phantom that detect_incremental reported as deleted on every run. Full-scan callers (extract's saves, watch._rebuild_code's saves) now pass the RAW detect corpus via a new scan_corpus parameter and in-root rows outside it are dropped; the corpus is deliberately not the Graphify-Labs#933 stamp-filtered files dict, so failed-chunk/omitted-doc rows and --code-only doc rows survive. Subset saves (changed_paths hooks, Graphify-Labs#917) keep the seeding default. detect_incremental now splits manifest rows that left the scan into deleted_files (gone from disk) and excluded_files (alive but out of scan), mirroring the watch-side Graphify-Labs#1795 distinction, and the extract summaries report the two separately. Ordering matters: extract's cleanup of newly-excluded nodes previously worked only through the Graphify-Labs#1908 conflation, so the graph-source prune lands together with the manifest split to avoid regressing Graphify-Labs#1909.
…des (Graphify-Labs#1910) tree-sitter-sql parses PL/pgSQL CREATE FUNCTION statements (OUT/INOUT params, tagged dollar quotes, PERFORM/:= body statements) as ERROR nodes, and the dispatch loop had no branch for them, so the functions were silently dropped from the graph. Handle ERROR nodes inside walk() (they can nest inside a merged create_function during multi-statement error recovery) and dispatch top-level ERROR statements to it. The branch regex-scans the raw node text for every CREATE [OR REPLACE] FUNCTION/PROCEDURE, mirroring the existing fb_proc_or_trigger and has_error fallbacks, with a name class that keeps schema-qualified names (exposed.important_function) whole. The PL/pgSQL body is not scanned for FROM/JOIN references to avoid junk reads_from targets, and node ids match the clean create_function branch so seen_ids dedups consistently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…phify-Labs#1910 in unreleased 0.9.17 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.cjs was half-registered: the language maps in build.py and extract.py already routed it to the JS grammar, but it was missing from CODE_EXTENSIONS, the extractor _DISPATCH, _LANG_FAMILY, and the JS resolution/cache suffix sets — so .cjs sources (Electron main/preload scripts, CommonJS escape hatches in "type": "module" packages) were silently skipped during a build: classified as non-code and never handed to the JS extractor. Add .cjs alongside .mjs in the six lists that gate/route JS files (detect.py, extract.py _DISPATCH, analyze.py _LANG_FAMILY, extractors/models.py cache-bypass, extractors/resolution.py resolve exts, cli.py _HOOK_SOURCE_EXTS), with regression locks mirroring the .mts/.cts fix (Graphify-Labs#1607). Real-world impact: an Electron app whose ~2000-line main.cjs backend was invisible went from 201 to 294 nodes and 256 to 441 edges on rebuild. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…odes (Graphify-Labs#1915) _rebuild_code fed Markdown/doc files (.md/.mdx/.qmd/.skill) into the AST quick-scan while _reconcile_existing_graph preserved the same docs' semantic (LLM) nodes, so every semantic-backed doc was represented twice (heading nodes + concept nodes) and the graph bloated ~4x vs the CLI `graphify . --update` path. Semantic now supersedes AST per doc source: before choosing extract_targets, read the existing graph's source identities that carry semantic doc nodes (non-_origin=="ast", gated on file_type=="document" so pre-Graphify-Labs#1865 marker-less graphs aren't misread) and exclude those docs from the quick-scan in both the full-rebuild and changed_paths branches. They stay in code_files, so the Graphify-Labs#1795 fail-closed corpus check and the shrink accounting still cover them — a previously-bloated graph self-heals on the next full rebuild (the AST ownership rule drops the stale heading nodes) without the shrink guard refusing the smaller write. On incremental rebuilds the excluded docs never enter rebuilt/node-evicted identities, mirroring Graphify-Labs#1865's tier-scoped edge rule at the node level, so a doc's semantic nodes and edges survive. Docs with no semantic layer (and brand-new docs) keep the no-LLM quick-scan structure from #09b33b7.
… cache (Graphify-Labs#1916) save_semantic_cache groups nodes/edges/hyperedges by their own source_file and the write loop skips a group whose path is a ghost (not .is_file(), silently) or out-of-scope per the Graphify-Labs#1757 guard — but an edge or hyperedge in an ALLOWED group that references a node id from a skipped group was still written verbatim, so on replay (check_semantic_cache) it dangled forever. The Graphify-Labs#1895 filter does not cover this: it cleans the in-memory merged result, while the checkpoint writes the cache BEFORE it runs and replay bypasses it entirely. Fix at the cache layer (the authoritative write path): before the write loop, compute the node ids belonging to groups that will be skipped — mirroring BOTH skip branches — minus ids also defined in a group that will be written (duplicate-attribution nodes must not be over-pruned). Each written group then drops edges whose source/target is a skipped id and hyperedges whose member list intersects them (whole-hyperedge drop, mirroring Graphify-Labs#1895). Pruning runs on the incoming result only, so with merge_existing=True (the llm.py checkpoint path) the prior cached entry's valid edges survive the union untouched. Everything is gated on allowed_source_files being provided, so unscoped callers stay byte-identical. Complementary hardening in build_from_json: hyperedges were copied into G.graph["hyperedges"] verbatim without member validation, so a dangling hyperedge reached graph.json even from a live (non-cache) extraction. Members are now remapped via the same normalization the pairwise-edge loop uses and pruned when they still don't resolve; a hyperedge with no surviving member is dropped whole with a stderr warning. (Single-member hyperedges are legal in this codebase — per-file flows in the Graphify-Labs#1574 tests — so the drop threshold is zero survivors, not two.) Tests: scoped save with an edge to an out-of-scope real file, the same with a ghost source_file, whole-hyperedge drop, unscoped save preserved byte-identically (raw cache entry compared), merge_existing keeping the prior slice's valid edges, and build_from_json pruning a dangling hyperedge member / dropping an all-dangling hyperedge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ce (Graphify-Labs#1894) `graphify extract --mode deep` over a warm tree was a silent no-op, for three stacked reasons: 1. The semantic cache ignored mode: deep runs were served standard-mode entries (and vice versa). check_semantic_cache/save_semantic_cache now take `mode` (default None, byte-identical when omitted so older installed callers keep working) and map it to a namespaced kind — cache/semantic/ for None, cache/semantic-{mode}/ otherwise. The per-chunk checkpoint in llm.extract_corpus_parallel and the extract / cache-check consumers thread the run's mode through (cache-check grows --mode/--deep). cached_files, clear_cache, and prune_semantic_cache sweep BOTH namespaces; prune uses the same live-hash set for both (liveness is content-based, mode-independent) so semantic-deep/ can't regrow the Graphify-Labs#1527 unbounded-orphan problem and inherits the files_by_type-derived exclusion gating for free. 2. extract had no --force — the flag was silently swallowed by the parser's unknown-arg fallthrough. It is now real (plus GRAPHIFY_FORCE env parity with `update`): force disables the incremental gate so detection is a full scan and skips the semantic cache READ so every semantic file re-dispatches, while the post-run save and manifest stamping still happen. 3. The incremental gate dispatched zero files on a warm unchanged tree before the cache was ever consulted, so namespacing alone couldn't fix the repro. In deep+incremental runs the semantic pass now widens to the full live doc/paper/image set from detect_incremental's files_by_type (already exclusion-filtered, Graphify-Labs#1908/Graphify-Labs#1909) and lets the mode-namespaced cache decide hits/misses, with a loud count line so the first deep run's full re-dispatch is visible. Skill-side threading of mode is deliberately deferred to PR-2; mode defaults keep generated skills byte-compatible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1915 Graphify-Labs#1912 in unreleased 0.9.17 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add _score_query() producing combined ranking + per-term singleton winners in a single graph traversal. _pick_seeds consumes precomputed winners instead of rescoring each term via _score_nodes. Behavior byte-identical to the legacy T+1 path. Supersedes the rescoring loop from Graphify-Labs#1596 while preserving its per-term coverage guarantee; folds in coverage scaling from Graphify-Labs#1724 and multiplicity penalty from Graphify-Labs#1832. Orthogonal to the trigram prefilter from Graphify-Labs#1431. Benchmark (full sweep in Graphify-Labs#1889): 1k-100k nodes, 1-10 terms: 1.43x-2.43x median speedup Single-term: no regression (1.70x-1.89x improvement) Traversals: T+1 -> 1 regardless of term count Tests: 3221 passed, 3 skipped. Ruff clean. Graphify graph updated. Refs Graphify-Labs#1889
…score_query API; bench newline PR Graphify-Labs#1918 replaced _pick_seeds(terms=) with best_seed_by_term= from _score_query. Update the Graphify-Labs#1900 German-stopword seed test to the new single-traversal API and add the missing trailing newline in the (manual, non-collected) query-scoring benchmark. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…raphify-Labs#1917) An id whose canonical stem contains its legacy stem as a prefix (parent dir name == file stem, e.g. .claude/CLAUDE.md) re-matched the legacy branch every build and grew another stem segment, defeating the same_topology/no_change short-circuits and churning graph.json + clustering on every zero-delta update. Skip an id that already carries its canonical stem (mirrors graph_has_legacy_ids); genuine legacy migration still applies. Also records the Graphify-Labs#1889/Graphify-Labs#1918 query-scoring perf work in the changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e diagnostic (Graphify-Labs#1925 Graphify-Labs#1920 Graphify-Labs#1923 Graphify-Labs#1922) - Graphify-Labs#1925: a missing manifest.json no longer degrades `extract --code-only` into a full scan that discards the committed semantic layer. An existing graph.json is a sufficient incremental baseline (detect_incremental treats an absent manifest as "all new / none deleted"), so out-of-scope doc/paper/ image nodes are preserved while genuinely deleted sources still evict. - Graphify-Labs#1920: _stamped_manifest_files now counts hyperedge output, so a doc whose only chunk output is a hyperedge is stamped instead of re-extracted forever. - Graphify-Labs#1923: new namespace/use-aware PHP resolver (mirrors the Java resolver, runs before the unique-name rewire) so App\Models\Page and an imported Filament\Pages\Page stay distinct — no more false inherits/imports edge. - Graphify-Labs#1922: detect() records ignored files/dirs in a new `ignored` diagnostic field (the nested-ignore scoping bug itself shipped in 0.9.16 / Graphify-Labs#1873). Regression tests added for each; full suite 3325 passed, 3 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a new bespoke R extractor under graphify/extractors/, following the Julia/Elixir precedent. Handles the five function-assignment forms (<-, <<-, =, ->, ->>), nested function scoping, top-level vs function-level call attribution, member ($/@) and package-qualified (::/:::) raw_calls, pipes (|>/%>%), static library/require/requireNamespace imports, and static source() cross-file linkage. Unqualified bare calls route through the shared raw_calls resolver with an R language-family guard that blocks binding to non-R definitions. Wires .r into _DISPATCH/_EXTRA_FOR_EXTENSION/_LANG_FAMILY_BY_EXT and Rscript into _SHEBANG_DISPATCH; .R reuses the existing case-insensitive suffix fallback. Ships under optional [r] extra via tree-sitter-language-pack (no maintained tree-sitter-r wheel fits the per-language model); the pack may fetch/cache the R grammar on first use (no source uploaded, R not executed). Fixes Graphify-Labs#1689 for .R (no longer a no-AST-extractor case) and reuses the existing Graphify-Labs#1745 install-extra warning with pip install "graphifyy[r]". Adds tests/fixtures/sample.r + 3 cross-file fixtures, 22 R tests in test_languages.py following the DM optional-grammar skip-when-unavailable convention, and 2 dispatch/missing-dep tests in test_extract.py. R package metadata (DESCRIPTION/NAMESPACE/exports) and package-qualified call resolution are deferred to a follow-up (Graphify-Labs#9).
Sirhan1
force-pushed
the
feat/r-language-support
branch
from
July 15, 2026 21:19
a9f2400 to
f3acf2d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test PR on fork (the real target is upstream Graphify-Labs/graphify:v8 — see PR creation attempt in conversation). This PR exists to surface the diff; the actual review UI will be opened upstream.
What
Adds R (
.r/.R) language support via a new bespokeextract_rundergraphify/extractors/, following the Julia (dd8235e) and Elixir (3d5da60) precedent. Scoped strictly to.Rcode structure + call topology — package metadata and package-qualified call resolution are deferred to a follow-up.Why
R was the motivating case for the
#1689"no AST extractor" warning (.r/.Ris inCODE_EXTENSIONSbut had no dispatch entry). This PR wires a real extractor and reuses the existing#1745install-extra warning path for the missing-dependency case (pip install "graphifyy[r]").Named functions in R are assignments whose right- or left-hand side is an anonymous
function_definition, so the generic extractor is not a good fit. A bespoke extractor undergraphify/extractors/matches the established home for language-specific extractors (not theextract.pygod node).No maintained
tree-sitter-rwheel fits the repo's per-language wheel model, so the grammar ships viatree-sitter-language-pack>=1.12,<2under the optional[r]extra. The pack may fetch/cache the R grammar on first use; no source is uploaded and R itself is not executed.How
.r→_DISPATCH(.Rreuses the existing case-insensitive suffix fallback);.r→_EXTRA_FOR_EXTENSION(names the[r]extra in#1745);.r→_LANG_FAMILY_BY_EXTas the"r"family (blocks cross-language binding);Rscript→_SHEBANG_DISPATCH(Rscriptwas already in_SHEBANG_CODE_INTERPRETERS, so R-shebang scripts were already detected as code — they now dispatch instead of hitting#1689). No new node types, relations, resolver interfaces, or package-ownership semantics;CODE_EXTENSIONS,_CASE_INSENSITIVE_EXTS, and the analysis family map unchanged.extract_r()emits the existingnodes/edges/raw_callscontract:<-,<<-,=,->,->>), scoped IDs for nested fns beneath the enclosing fn.=used as a named call argument.$/@receiver calls and::/:::package-qualified calls becomeraw_callswithis_member_call=True, so the shared resolver won't bind them by method name.|>,%>%) traverse normally so contained calls are captured.EXTRACTED; unresolved bare calls go toraw_callsfor the shared cross-file resolver, which resolves a unique cross-file candidateEXTRACTEDwith import evidence orINFERREDwithout, leaving ambiguous names unresolved.library(pkg),require(pkg),requireNamespace("pkg")with a literal identifier or string →importsedges;pkg::fn/pkg:::fndouble as import evidence forpkg. Dynamic expressions (call args, variables) are skipped.source("helper.r"): only a static string ending in.r(case-insensitive), resolved against the caller file's directory (mirrors the Bash static-sourcepolicy). Emitsimports_fromonly when the target exists on disk. URLs, connections, variables, computed paths, and missing files are skipped.Tests
tests/fixtures/sample.r+ 3 cross-file fixtures (r_other.r,r_extra.r,r_caller.r).tests/test_languages.pyfollowing the DM optional-grammar skip-when-unavailable convention (@_needs_rfor real-grammar tests; dispatch/missing-dep tests stay network-free via monkeypatchedsys.modules).tests/test_extract.py(test_extract_warns_when_r_extra_missing,test_extract_no_warning_when_r_extra_installed).test_extract.pytests that asserted.R/.rtriggered#1689now use.ejs(the canonical no-extractor case), since.Ris now dispatched.Real-grammar tests skip when
tree-sitter-language-packis absent. Tests must not download a parser implicitly; missing-dependency and dispatch tests are network-free.Verification
test_label_communities_batches_when_over_batch_sizeis order-dependent and fails on a clean stashed tree too — unrelated to R).ruff checkclean on touched files.pyrightclean ongraphify/extractors/r.py(0 errors with the venv pythonpath; the import-resolution noise ontree_sitter_language_pack/tree_sitteris repo-wide pre-existing and affects every extractor identically).graphify update .run to refresh the graphify-out graph.Scope deferred to follow-up (Graphify-Labs#9)
R package metadata (
DESCRIPTIONDCF ingestion,NAMESPACEimports/exports), package-qualified call resolution once package-to-symbol ownership exists, S3/S4/R6 semantics,.Rmdchunk extraction, NSE/formula handling, higher-order callback inference,renv.lock, and runtime evaluation.