Skip to content

feat: add trustworthy change evidence across hosts - #477

Merged
Helweg merged 2 commits into
mainfrom
feat/trustworthy-change-evidence
Oct 28, 2026
Merged

Helweg merged 2 commits into
mainfrom
feat/trustworthy-change-evidence

Conversation

@Helweg

@Helweg Helweg commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Deliver the approved agent-reliability work across the shared operations, human CLI, MCP, OpenCode, Pi, and omp:

  • Explicit source freshness: index_status({ checkFreshness: true }) / ocbi status --check-freshness --json, separate from catalog readiness and coordinator activity. Default status remains cheap; inspection does not refresh indexes.
  • Source-linked pre-edit tests: opt-in codebase_edit_context({ includeTests: true }), reusing AST-backed JS/TS direct-target and one-hop caller discovery. Complete budgeted citations and selected-only structured metadata; static links are not runtime coverage.
  • Current indexed working-tree impact: pr_impact({ workingTree: true }) / ocbi impact --working-tree --json for staged, unstaged, and untracked files, with explicit refresh requirements, bounded local Git status, provenance, and UNKNOWN for incomplete graph evidence.

Correctness and review

  • Resolve canonical native symbol IDs, then match exact target paths and declaration identity. Large declarations start at their signature rather than a ranked middle-body slice; missing source is withheld. Cap source blocks at 512 tokens to preserve room for relationships/tests. Production and evaluation share the same per-indexer lookup.
  • Keep selection-hook test references metadata-only, count retained evidence, reserve final heading/truncation overhead for closed whole-line source blocks, and propagate cancellation after empty lookups.
  • Read structural architecture files from the SQLite catalog without requiring vectors.
  • Reject concurrent Git/source/index changes, including database republication with unchanged source bytes/catalog counts. Reject invalid UTF-8 paths; retain only local rename endpoints for nested projects.
  • Native-backed regressions cover real source/citation identity, CRLF/Unicode, tight token budgets, scope, freshness, host surfaces, and working-tree graph transitions. An independent pre-review control reproduced incorrect suffix-file source selection and undetected concurrent graph publication.
  • Documentation and Unreleased changelog updated. No version bump, storage migration, installed-extension modification, merge, or release.

Verification

  • npm run build, then npm run build:ts for final TypeScript-only repairs; Rust source unchanged.
  • npm run typecheck
  • npm run lint
  • Final full coverage suite with short temporary roots: TMPDIR=/tmp npx vitest run --coverage — 3,082 passed, 7 skipped; 154 files passed / 1 skipped on macOS ARM64, Node 26.12.1. The initial full gate also ran npm run test:run.
  • npm run smoke:package — both package identities; strict standalone and SDK-interoperability declarations, ESM/CJS loading, CLI aliases and MCP initialize/tools-list.
  • Actual final built CLI and stdio MCP scenario: cached/current/stale/refreshed freshness, stale-impact rejection, indexed dirty-tree symbols and transitive callers, source-linked metadata-only test references, preserved HEAD and Git status.
  • Final built stdio MCP canonical-source scenario: exact target beats a same-named suffix file, a fragmented 240-line declaration starts at its signature, test references remain metadata-only, and missing source receives no source credit or fake body.
  • Final built stdio MCP minimum-budget scenario: 16/64/96-character identities with actual native callers/callees, CRLF/Unicode sources and 128-token responses retain closed whole-line source blocks, and source credit matches the visible body.
  • Final isolated pre-edit evaluation, unchanged budget: Hit@5/MRR@10/graph-neighbor recall 1.0; response average 478, p95 707.7, max 734 tokens; p95 latency 213.1 ms. Inspected every query against the retained baseline:
Query Baseline tokens Final tokens Target hit retained Expected graph neighbor retained
Resolver + caller 425 471 Yes Yes
Large evaluator target 360 734 Yes; now starts at canonical signature Not requested
Unresolved fallback 229 229 Yes Not requested

Linux-only invalid-byte filename coverage cannot run on macOS/APFS and is left to Linux CI. The pre-edit gate caught and drove repair of declaration-source loss and excessive source-block size; failed reports were preserved, and thresholds/baselines were not relaxed. These deterministic-mock evaluations validate harness/routing behavior, not semantic embedding quality. CI must verify the final PR revision on supported Node/platform jobs before merge.

The initial CI matrix exposed source-fence clipping during final response fitting and a path-length-dependent graph-omission fixture. Both were repaired with failing-before/passing-after source regressions and a caller identity that explicitly exceeds the small response budget. The corrected revision must complete fresh CI; this is not an unchanged rerun.

Release Notes

Category: feature. Semver: minor. This PR is for review and CI only; do not merge or publish as part of this task.

@Helweg Helweg added feature New feature or capability semver:minor New feature (minor bump) labels Oct 28, 2026
@Helweg

Helweg commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Validation complete for 3d74979. All 15 PR checks are SUCCESS, including Node 22/24 CI, all six platform/path jobs, CodeQL, and release labels. Fresh CI: https://github.com/Helweg/open-codebase-index/actions/runs/40674544143 . The initial red run was repaired in code, not retried unchanged. Local final acceptance: 3,082 tests passed with coverage and short temporary roots; both packed package identities passed; actual built stdio MCP verified closed whole-line source and correct credit at 128 tokens with long identities and real graph neighbors. The unchanged pre-edit gate passed all three queries with Hit@5/MRR/graph-neighbor recall 1.0 and p95 response size 707.7 tokens; all per-query deltas were inspected. Mock evaluation is routing/harness evidence, not semantic embedding-quality evidence. Review workers are settled/released. PR remains OPEN; no merge, tag, publication, or installed omp-extension change.

@Helweg
Helweg merged commit 47bf78b into main Oct 28, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or capability semver:minor New feature (minor bump)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant