Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions PERFORMANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ The reproducible real-repository harness, operator commands, correctness gates,
and optional explicit Graphify comparison are documented in
[`benchmarks/performance/README.md`](benchmarks/performance/README.md).

The [2026-10-02 real-repository comparison](benchmarks/agent_query/reports/2026-10-02-main-vs-graphify/REPORT.md)
retains indexing, query, CPU and memory observations for Compass 0.4.1 main
and Graphify 0.9.74 across five languages. It discloses the busy shared host,
different extraction scopes, storage/clustering configurations and every
failed observation. This diagnostic capture does not establish a controlled
performance ranking or replace the qualification baseline below.

## Baseline policy

A benchmark records:
Expand Down
8 changes: 8 additions & 0 deletions benchmarks/agent_query/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@ question/evidence matrix, including ask, communities, clusters, and god nodes.
The audit report distinguishes completed checks from surfaces still awaiting
source or design-quality judgments.

The [2026-10-02 comparison](reports/2026-10-02-main-vs-graphify/REPORT.md)
records Compass main `b7b7fa81` (0.4.1) against Graphify 0.9.74 on these five
language panels, with JSON/SQLite and base/native-Leiden configurations kept
separate. It retains 120 builds, 1,020 query workflows, source oracles, raw
transcripts and path-normalized method snapshots. These selected development
inputs and shared-host timings are diagnostic evidence; text recall and the
post-capture directional audit have distinct scoring requirements.

Five suites share the harness:

| Suite | Questions | Shape |
Expand Down
237 changes: 237 additions & 0 deletions benchmarks/agent_query/reports/2026-10-02-main-vs-graphify/REPORT.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"protocol": "Ten existing source-reviewed caller/callee tasks, identical question text and 2000-token budget; Compass ask versus Graphify query. Three repetitions, all four variants. These reuse source facts from the structured panel; do not count them as independent source labels.",
"registeredAt": "2026-10-02T18:10:13.794695+00:00",
"registeredBeforeAnyQueryExecution": true,
"suite": "${COMPASS_CHECKOUT}/benchmarks/agent_query/suite_ask.toml",
"suiteSha256": "ea1306f24b1f612f78f5d4e579df21d07ac8537dc49bbb403be1eddce71110e4"
}

Large diffs are not rendered by default.

2,412 changes: 2,412 additions & 0 deletions benchmarks/agent_query/reports/2026-10-02-main-vs-graphify/data/captures.jsonl

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"afterInitialGraphCaptureBeforeCensusScoring": true,
"at": "2026-10-02T18:12:32.505321+00:00",
"censusScriptSha256": "17beeed9c6932c94146462312cfcf24c1d27a637c4f23e4094e62444a5c96086",
"reason": "Graphify clustered AST nodes omit kind/type in their published schema, including normal function nodes. The scorer must not reject a field solely for an absent optional kind. Require the same exact source file, identifier line, name, unique declaration, and direct containment by the exact source type. Allow absent kind or field/property/attribute; reject contradictory explicit kinds. Source files and expected declarations are unchanged."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"buildObservations": 120,
"compassHead": "b7b7fa819b127464566430cc454fd2085d5d5b62",
"queryObservations": 1020,
"queryTimeouts": 1,
"scriptsSha256": {
"census.py": "17beeed9c6932c94146462312cfcf24c1d27a637c4f23e4094e62444a5c96086",
"directional_audit.py": "cdb4d0785bbd9f9e5ab9ffae332e65e38dcb4acff07aa05c21b02524e1d5d5fb",
"evaluate.py": "e09529a5ccdbec34b87d1fe64207c5ad6eb6b3c78304c36dee5af4077dcd2aff",
"source_audit.py": "4397599684a69e133b65b25846f53c9f1b3c7f750a4e26edc3187c953f6870a3",
"summarize.py": "7c83ecb22f3f8efd066bc543f8a9b43cd5736f53b8927ab42a1f80e10047368b",
"supplement.py": "0d4dd7f466b2096cf6327770e8ac7551af37fa475260e9bb01e7ee6bac4b7c88",
"transitions.py": "f0fef0559231be6236cba9590374501108bdc4d33bfa3aaf128661aab0948bcd",
"verify_evidence.py": "9af80d5dd34f6487ee6c180e3ca239346efec9d78bb80bf1861257dab4b2851e"
},
"sourceVerification": "All five pinned clean corpus HEADs, suite source digests and before-build call-oracle source digests passed.",
"verification": "Graph digests, query graph identities, binary identities, both Graphify package source trees and benchmark helper identities unchanged.",
"verifiedAt": "2026-10-02T19:07:08.029053+00:00",
"verifiedFrozenGraphs": 120
}
Loading
Loading