Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
20 changes: 20 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Keep the image build context lean. Runtime corpus is mounted, not copied.
.git
target
example
**/node_modules
dashboard/dist/node_modules
dashboard/node_modules
**/.rgctl
**/.rgctl-diff
docs
openspec
rgctl-tests
fuzz
.cursor
.claude
.idea
.vscode
*.log
*.profraw
*.profdata
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
.claude/
.opencode/
.agent
.reports
# OpenSpec change proposals (local only — do not commit)
openspec/
.scratch/
Expand Down
44 changes: 41 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

**Your goal when contributing here:** preserve ingest scale, query correctness, memory discipline, and deterministic artifacts under `.rgctl/` — not add convenience at the cost of Tokio blocking, whole-repo clones, or ungated cold regressions.

> **Looking for how to *use* rgctl on another codebase?** Install skills (`rgctl install --skill --with-commands`) or copy [docs/agents/USER_AGENTS_TEMPLATE.md](docs/agents/USER_AGENTS_TEMPLATE.md) into *that* repo’s `AGENTS.md`. See [docs/guides/agent-commands.md](docs/guides/agent-commands.md).
> **Looking for how to *use* rgctl on another codebase?** Install skills (`rgctl install --skill`) or copy [docs/agents/USER_AGENTS_TEMPLATE.md](docs/agents/USER_AGENTS_TEMPLATE.md) into *that* repo’s `AGENTS.md`. See [docs/guides/agent-skill.md](docs/guides/agent-skill.md).

---

Expand All @@ -16,12 +16,14 @@
- **Parallel ingest:** Per-file plugin extraction runs on the discover worker pool. Do not replace with a serial whole-repo walk when parallel ingest exists.
- **Streaming commits:** Emit symbols/relations file-by-file; avoid unbounded `Vec<Relation>` / whole-repo ASTs before commit (`rgctl-extraction` spill patterns).
- **Clone hygiene:** Prefer `&[u8]` / `Cow` / borrows in tree-sitter walkers; `Vec::with_capacity` when sizes are known; no `unwrap()` in library paths.
- **Ingest hot path:** Follow **Ingest hot-path practices** below (no per-symbol heap strings, hash/prep once on workers, spill scratch reuse, tracker mapping without re-scan).
- **Typed graph:** Respect `EdgeType` / node kinds; do not invent ad-hoc string edges for hot paths.
- **Artifacts:** Session data lives in `{repo}/.rgctl/`. Warm caches invalidate wall-time claims.
- **Constrained discover (opt-in):** Prefer `rgctl discover . --with-limits max-mem-mb=4096,threads=1` (or `RGCTL_WITH_LIMITS`) in containers / cgroups — do **not** change default desktop discover for memory. Soft RSS tripwire at ~95%; smaller spill sort-runs and stream channel when `max-mem-mb` is set. Container smoke: `./scripts/run-container-with-limits-smoke.sh` (`tests/Containerfile`, mounts `example/linux`).
- **Features:** Default semantic embedder is compiled **vocab**. Do not require ONNX / Python ML unless behind an explicit feature (e.g. `semantic-onnx` / code-daemon + Git LFS).
- **OpenSpec language work:** Still cite [openspec/changes/_shared/starting-context.md](openspec/changes/_shared/starting-context.md) (pointer here); follow the sections below.
- **Grammar bumps:** When you bump a tree-sitter grammar pin, update that language’s `*-ast-coverage.json` (and add the language to `rgctl-ast-coverage::bundled_specs` for new languages). Unit tests hard-fail the same drift; `cargo check -p rgctl-languages` warns (`RGCTL_AST_COVERAGE_STRICT=1` fails). The website `/docs/languages/` pages are generated from those JSON files — do not maintain parallel tables under `docs/languages/`.

- **Releases:** Follow **Releases** below (and [docs/releasing.md](docs/releasing.md)). Do not hand-edit dozens of crate `version =` lines.
---

## Context & architecture
Expand All @@ -44,6 +46,25 @@ Applies to all extraction / language / discover hot-path work (and OpenSpec `*-e
3. **Streaming** — incremental graph commit; match extraction spill/channel patterns.
4. **Idiomatic Rust** — `Result` + `thiserror`; follow `rgctl-lang-java` / `rgctl-extraction` conventions.

### Ingest hot-path practices

Rules distilled from linux cold-discover work (`index_extract` / pass-1 / spill / `save_tracker`). Breaking these usually shows up as Gate A wall or RSS regressions — treat O(files)×O(symbols) heap work as a bug.

| Practice | Do | Don't |
|----------|----|-------|
| **No heap strings per symbol** | Pass `&str` / slices into pass-1 (`add_symbol_with_prep`); borrow file bytes | `String::from` / `to_string()` for every symbol body or path key on the merge thread |
| **Prep on workers** | Compute line offsets, BLAKE3 `code_hash`, token bloom in `SymbolPass1Prep` on extract workers | Re-walk source / re-hash on the sequential pass-1 thread |
| **Hash once** | Set `FileExtraction.file_hash` from bytes already in memory; thread through `StreamStats` / `PipelineStats` into `FileTracker::index_files_with_mapping` | Re-`fs::read` + BLAKE3 all files in `save_tracker` after extract already hashed them |
| **Empty-tracker short-circuit** | When `file_hashes.json` is empty, `detect_changes` marks all paths **added** without hashing (keeps ChangeSet non-empty so a stale snapshot is not reused) | Hash the whole tree twice on cold discover (detect + index) |
| **Normalize / map once per file** | `begin_file_batch` → push into `active_tracker_ids`; flush once in `end_file_batch`; accumulate mapping at commit | Per-symbol `HashMap::get_mut` / `normalize_path_str(...).into_owned()`; full mmap node scan/sort just to rebuild file→node ids |
| **Reuse tree-sitter parsers** | Call `rgctl_plugin_helpers::parse_source` (thread-local `Parser` per Rayon worker); never `Parser::new()` per file on the extract hot path | Fresh `Parser::new` + `set_language` inside `extract_*` / `parse` for every file (linux: ~71k×) |
| **Spill alloc reuse** | `SegmentedSpill` scratch `Vec` + `bincode::serialize_into`; keep sort runs at `DEFAULT_SORT_RUN_BYTES` (256 MiB) unless profiling says otherwise | Fresh `bincode::serialize` → new `Vec<u8>` per node/edge; shrinking sort runs without a cold gate |
| **CodeIndex bodies off by default** | Default discover: no body-storing `CodeIndex` (no multi-GB `code_index.json`); nodes still get `code_hash` from prep | Attach a full CodeIndex on the cold path “for convenience” |

When adding extract or graph-commit code, ask: *does this allocate or re-read once per symbol/file on the sequential merge thread?* If yes, move it to workers or reuse an existing buffer/key.

Stage meanings and current linux notes: [docs/internal/profile.md](docs/internal/profile.md).

### Cold profile (mandatory for scale / perf claims)

1. **Release binary only:** `cargo build --release --bin rgctl`
Expand Down Expand Up @@ -138,10 +159,27 @@ Baselines and notes: [docs/internal/profile.md](docs/internal/profile.md#snapsho
| [CONTRIBUTING.md](CONTRIBUTING.md) | Setup, tests, PR norms |
| [docs/contributor-checklist.md](docs/contributor-checklist.md) | Language / feature checklist |
| [docs/guides/semantic-search.md](docs/guides/semantic-search.md) | Embedders (if touching semantic) |
| [docs/releasing.md](docs/releasing.md) | Version bump + GitHub Release tags |
| [openspec/changes/_shared/starting-context.md](openspec/changes/_shared/starting-context.md) | OpenSpec pointer (canonical policy is this file) |

---

## Releases

When asked to cut or bump a release, use the lockstep tooling — full detail: [docs/releasing.md](docs/releasing.md).

| Rule | Detail |
|------|--------|
| **One version** | SSOT is `[workspace.package] version` in root `Cargo.toml`. Crates use `version.workspace = true`. Do **not** sed/`version =` across every crate by hand. |
| **Bump TOMLs only** | `./scripts/bump-version.sh patch` (or `minor` / `major` / `X.Y.Z`). Syncs workspace version, `[workspace.dependencies]` path pins, and README release links. |
| **Bump + tag + push** | `cargo release patch --workspace` (dry-run), then `--execute` when the user wants commit/tag/push. Config: [`release.toml`](release.toml) (`shared-version`, `publish = false`, tag `v{{version}}`). Needs a **clean** git tree. |
| **Tools** | `cargo install cargo-edit cargo-release --locked` if missing. |
| **GitHub Release** | Pushing `v*` runs [`.github/workflows/release.yml`](.github/workflows/release.yml) (binaries). Add `docs/releases/vX.Y.Z.md` for curated notes. |
| **No crates.io** | `publish = false` — do not `cargo publish` unless the user explicitly asks to enable it. |
| **Commits / tags / push** | Only when the user explicitly requests them (same standing rule as other git ops). Prefer preparing the bump + release notes and stopping for the user to commit/sign/tag if they GPG-sign locally. |

---

## Build and day-to-day commands

```bash
Expand All @@ -159,7 +197,7 @@ cargo build --release

Code-daemon / ONNX weights: `git lfs pull` when using that embedder feature.

Dogfood fixtures: `rgctl-tests/` (e.g. ecommerce-*). Consumer agent pack: `rgctl install --skill --with-commands --tools cursor`.
Dogfood fixtures: `rgctl-tests/` (e.g. ecommerce-*). Consumer agent pack: `rgctl install --skill --tools cursor`.

---

Expand Down
9 changes: 8 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,13 @@ Full map: [docs/Code_structure.md](docs/Code_structure.md)

---

## Releasing (version bump)

Lockstep workspace version via `[workspace.package]` + `version.workspace = true`.
See **[docs/releasing.md](docs/releasing.md)** for `./scripts/bump-version.sh` and `cargo release`.

---

## Adding or improving a language / feature

Use the hub checklist for path choice, test matrices, and pre-PR commands:
Expand All @@ -103,7 +110,7 @@ Tier 1 depth (Layers A–F): [docs/tier-1-language-support.md](docs/tier-1-langu

- **User-facing:** `docs/Introduction.md`, `docs/user-guide.md`, `docs/dashboard-user-guide.md`
- **Agents (contribute to rgctl):** root [`AGENTS.md`](AGENTS.md) (starting-context, profiles/tests/benches) · [`docs/json-api.md`](docs/json-api.md)
- **Agents (use rgctl elsewhere):** [`docs/guides/agent-commands.md`](docs/guides/agent-commands.md) · [`docs/agents/USER_AGENTS_TEMPLATE.md`](docs/agents/USER_AGENTS_TEMPLATE.md) · [`docs/agent-recipes.md`](docs/agent-recipes.md)
- **Agents (use rgctl elsewhere):** [`docs/guides/agent-skill.md`](docs/guides/agent-skill.md) · [`docs/agents/USER_AGENTS_TEMPLATE.md`](docs/agents/USER_AGENTS_TEMPLATE.md) · [`docs/agent-recipes.md`](docs/agent-recipes.md)
- **Accuracy:** keep CLI examples aligned with `dashboard/scripts/validate-guide-cli-gbuilder.sh` where possible

---
Expand Down
11 changes: 9 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ resolver = "2"

[workspace.package]
edition = "2024"
rust-version = "1.88"
rust-version = "1.99"
version = "0.4.17"

[workspace.dependencies]
rgctl-plugin-api = { path = "crates/rgctl-plugin-api", version = "0.4.17" }
Expand Down Expand Up @@ -95,6 +96,12 @@ rgctl-lang-groovy = { path = "crates/rgctl-lang-groovy", version = "0.4.17" }
rgctl-ast-coverage = { path = "crates/rgctl-ast-coverage", version = "0.4.17" }
rgctl-languages = { path = "crates/rgctl-languages", version = "0.4.17" }
tree-sitter = "0.25"
# OSV schema parse (no client); OpenVEX types for later VEX emission
osv = { version = "0.3", default-features = false, features = ["schema"] }
openvex = "0.1.1"
walkdir = "2"
zip = { version = "2", default-features = false, features = ["deflate"] }
thiserror = "1"

[workspace.lints.rust]
unsafe_code = "warn"
Expand Down Expand Up @@ -124,7 +131,7 @@ expect_used = "warn"

[package]
name = "rgctl"
version = "0.4.17"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
authors = ["rgctl Contributors"]
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@

[![Docs](https://img.shields.io/badge/docs-shaaf.dev%2Frgctl-2563eb?style=flat-square&logo=readthedocs&logoColor=white)](https://shaaf.dev/rgctl)
[![Website](https://img.shields.io/github/actions/workflow/status/sshaaf/rgctl/website.yml?branch=main&style=flat-square&label=website)](https://shaaf.dev/rgctl)
[![Rust](https://img.shields.io/badge/rust-1.88%2B-orange?style=flat-square&logo=rust)](https://www.rust-lang.org/)
[![Rust](https://img.shields.io/badge/rust-1.99%2B-orange?style=flat-square&logo=rust)](https://www.rust-lang.org/)
[![Platforms](https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-555?style=flat-square)](https://github.com/sshaaf/rgctl/releases/latest)
[![tree-sitter](https://img.shields.io/badge/parser-tree--sitter-brightgreen?style=flat-square)](https://tree-sitter.github.io/tree-sitter/)
[![Agents](https://img.shields.io/badge/agents-Cursor%20%7C%20Claude%20%7C%20Codex-111827?style=flat-square)](https://shaaf.dev/rgctl/docs/guides/agent-commands/)
[![Agents](https://img.shields.io/badge/agents-Cursor%20%7C%20Claude%20%7C%20Codex-111827?style=flat-square)](https://shaaf.dev/rgctl/docs/guides/agent-skill/)
[![Tier 1](https://img.shields.io/badge/languages-14%20Tier%201-8b5cf6?style=flat-square)](https://shaaf.dev/rgctl/docs/languages/)

[![C](https://img.shields.io/badge/C-A8B9CC?style=flat-square&logo=c&logoColor=black)](docs/languages/README.md)
Expand Down Expand Up @@ -39,7 +39,7 @@ rgctl -f json blast-radius MyService
rgctl -f json gql 'MATCH (a:Function)-[:CALLS]->(b) RETURN a,b LIMIT 20'

# Use with your favorite LLM agent
rgctl install --skill --with-commands --tools cursor,claude,codex,agents
rgctl install --skill --tools cursor,claude,codex,agents
```

https://github.com/user-attachments/assets/15ec6d91-f716-4cbd-a873-e982ba3c6dca
Expand All @@ -57,7 +57,7 @@ https://github.com/user-attachments/assets/15ec6d91-f716-4cbd-a873-e982ba3c6dca
rgctl --version
```

**Or build from source** (Rust **1.88+**):
**Or build from source** (Rust **1.99+**):

```bash
git clone https://github.com/sshaaf/rgctl.git
Expand Down Expand Up @@ -97,13 +97,13 @@ Always prefer **`-f json`** for agents and scripts ([JSON API](docs/json-api.md)

## Use with coding agents

Install the bundled pack (skills + slash commands) into your IDE tooling:
Install the bundled pack (skills) into your IDE tooling:

```bash
rgctl install --skill --with-commands --tools cursor,claude,codex,agents
rgctl install --skill --tools cursor,claude,codex,agents
```

Then: **discover once → query with `-f json`**. See [Agent commands](docs/guides/agent-commands.md).
Then: **discover once → query with `-f json`**. See [Agent pack](docs/guides/agent-skill.md).
For *your* application repo, optionally paste [USER_AGENTS_TEMPLATE.md](docs/agents/USER_AGENTS_TEMPLATE.md) as `AGENTS.md`.

---
Expand Down
2 changes: 1 addition & 1 deletion agent-pack/agents/registry.toml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Agent adapters (OpenSpec-aligned paths). See docs/guides/agent-commands.md
# Agent adapters (OpenSpec-aligned paths). See docs/guides/agent-skill.md
# command_style: hyphen | colon | dollar | slash
# command_extension: md (default), prompt, prompt.md, toml
# global_skills / global_commands: reserved for future -g path wiring (install uses agent_dir under prefix today)
Expand Down
6 changes: 3 additions & 3 deletions agent-pack/manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ workflows:
title: Data flow and slices
- id: search
title: Semantic and structural search
- id: gql
title: Graph query language
- id: migrate
title: Migration roadmap
- id: kantra
title: Konveyor Kantra rules
- id: gate
title: CI and policy gates
- id: vuln
title: OSV triage and deps check
meta_skills:
- id: rgctl
title: rgctl router
title: rgctl
38 changes: 0 additions & 38 deletions agent-pack/out/agents/host-agents/skills/rgctl-discover/SKILL.md

This file was deleted.

35 changes: 0 additions & 35 deletions agent-pack/out/agents/host-agents/skills/rgctl-flow/SKILL.md

This file was deleted.

36 changes: 0 additions & 36 deletions agent-pack/out/agents/host-agents/skills/rgctl-gate/SKILL.md

This file was deleted.

35 changes: 0 additions & 35 deletions agent-pack/out/agents/host-agents/skills/rgctl-gql/SKILL.md

This file was deleted.

Loading