🧭 JEV Integration Evaluator

Find where an AI decision point would actually help your codebase — and prove it before you trust it.

A CompleteTech LLC agent skill and Python CLI

---

What is this?

JEV answers small, structured questions — "which of these five known actions fits this situation?" — instead of generating free text. Many codebases have spots where a brittle keyword check or heuristic makes exactly that kind of judgment call.

This toolkit scans a repository, finds those spots, scores them, and walks you through testing whether JEV actually does better — with real experiments, not vibes. It never runs your code, never calls the network, and never changes anything unless you explicitly approve an exact, reviewed patch.

LLMs generate. JEV classifies, selects, and evaluates. Deterministic code enforces. Instrumentation measures. Experiments decide whether JEV stays.

🛡️ Safe by default

📖 Read-only	Scans parse source; they never import or execute your code
🔌 Offline	No API key, no network — until you explicitly authorize and budget it
⏸️ Runtime off	Generated adapters ship disabled, behind flags, with fallback
✍️ Approved changes only	Edits require your review and an exact plan-digest approval
🧪 Evidence-gated	Synthetic or demo results can never justify adoption

Tip

"No useful JEV placement found" is a successful result, not an error. The tool is built to say no.

🚀 Quick start

Needs Python 3.10+. Runtime dependencies are PyYAML, jsonschema, and packaging, plus tomli on Python 3.10.

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
python -m pip install -e ".[test]"

Scan a repository and review what it found:

python -m jev_integration_evaluator scan --repo /path/to/repo --out ./analysis --depth STANDARD
python -m jev_integration_evaluator onboard --inventory ./analysis/jev-opportunities.json

The scan writes ranked opportunities, an architecture map, integration and experiment plans, a risk analysis, and machine-readable JSON/CSV — into ./analysis, outside your repo. The results report says "no experiment was run" until you actually run one.

For native Windows source-bound discovery, use a local NTFS drive path and a new report path outside the target (PowerShell):

python -m jev_integration_evaluator repository-discovery `
  'C:\src\my-project' --out 'C:\jev-reports\capabilities.json'

See the platform matrix for Windows versions, link handling, UNC outcomes and path limits.

For a proposed native Windows Python console delivery, the read-only source and package-input checks inspect reviewed files on local NTFS. A separate offline native API checkpoint has a locally observed Windows 11 package/install, supervised normal-console path, and retained installed-version cutover and rollback under one run ID. Local Windows 11 CPython 3.10/3.13/3.14 installed fixture runs have retained private receipts. Windows Server 2022 build 20348 also passed the native 31-case off-mode preparation, package/install, supervised console and retained generation suite on CPython 3.10.11 and 3.13.15 with zero skips; see the PR #106 merge run. The later 38-case suite also passed controlled launch/install interruption and owned installer-process death on both Server interpreters, with zero skips in the PR #110 merge run. Broader source-filesystem and connected qualification remain pending.

A finite independent-package owner composes separately installed Alpha and queue distributions through a third source-reviewed installed console. Its public source/package/install APIs and template packages-owner-*, owner-* and connected-packages-installed-bind commands bind exact member and owner origins to one durable task ledger. The dual stopped-generation transfer retains both placements through upgrade and signed retained rollback. Normal entrypoint claims, shared budgets and closed-task replay are offline software gates; real provider, combined treatment, canary/active and benefit criteria remain open. Source rollback does not restart an installed workload.

The separate JavaScript recipe C template catalog validates reviewed ESM, CommonJS and TypeScript source through trusted external tooling and materializes source-bound planner inputs on Linux. An offline Node package/install adapter checkpoint now stages an owned off-mode generation through separate exact approvals. Its component tests stub the upstream JS source verifier, while a separate native test exercises the real source-bound ESM/CommonJS/TypeScript install and normal command with external effect files. A read-only Node delivery descriptor binds exact installation and observation inputs. A separate offline Node supervisor now records an owned process, checks independent ready/effect paths, and supports bounded stop, disable and retained generation selection under exact scopes. Independent synthetic installed ESM, TypeScript and CommonJS journeys each exercise two reviewed fixture versions, distinct raw host effects, exact upgrade and retained rollback. The ESM journey resumes an interrupted first launch under the same session run ID and refuses source drift before upgrade. The CommonJS journey also resumes an interrupted upgrade selection under the same session run ID. The ESM fixture materializes, applies, verifies, packages, installs and launches through an exact branch-built installed evaluator CLI. An installed connected owner protocol adds an exact private descriptor, typed JEV client, independently authenticated grant callbacks, durable single-owner budget, and finite CLI launch. Its offline shadow fixture is synthetic protocol evidence; live provider and observed canary/active qualification remain pending, and live benefit is unknown. The separate installed synthetic selected-effect profile adds explicit source-owned cancellation, exact offline receipts and one durable owner shared by two installed placements. Its fixture commands remain synthetic; real provider authority and observed activation gates are unchanged.

The recipe lifecycle matrix records, for every Python recipe A–M and the separate JS/TS recipe C entry, which lifecycle stages a named offline synthetic test exercises and which remain pending or unsupported. The six-use-case source matrix pins offline host contracts for registered tools, graph identity, retrieval, completion, claim support and safe retention. The five new fixture oracles are tested offline. A shared fixture wheel is also installed and invoked through its normal off-mode console on Linux CPython 3.13. A separate L graph host, D retrieval host, E completion host, E installed shadow protocol, M claim support host and H retention host now have separate source-bound offline package/install/supervised normal-console effects, versioned upgrade and owned rollback. All five use-case hosts also have fresh source-bound bounded-loop console bindings and owned edits carried through their documented installed normal-command journeys. These are synthetic off-mode results for the declared host shapes; other recipe profiles, provider modes and benefit remain pending.

The separate M connected shadow profile uses the same deterministic citation consumer with two stable tasks and one durable ledger. Its installed qualification and review are pending.

The separate L installed synthetic graph shadow protocol checks two host-owned SQLite task effects through the same reviewed L source binding. Its added malformed-response and actual-timeout schedule awaits fresh installed qualification. The separate D installed synthetic shadow protocol checks an offline local TLS fixture through the same reviewed D source binding. External provider operation, connected upgrade and measured benefit remain pending.

The finite H connected retention profile requires an explicit /prune choice and private authority references. Its candidate verification schedule checks independent retained bytes, pinned provenance, timeout events and between-task revocation. Fresh installed qualification, external provider operation, connected upgrade and benefit remain pending.

Want to see everything working first? Run the bundled offline demo:

python -m jev_integration_evaluator scan --repo examples/coding-agent --out demo/agent
python examples/run_demo.py --out demo/runtime
python -m pytest -q && python scripts/validate_package.py

Warning

Every model response in examples/ is synthetic. Demos prove the tools work; they never prove a live model helps.

🗺️ How it works

flowchart LR
    A["🔍 Scan"] --> B["🧑‍⚖️ Review"] --> C["🎯 Select"] --> D["🛠️ Implement"] --> E["🧪 Experiment"] --> F{"⚖️ Decide"}
    F -->|"keep"| G["Deploy<br/>(your call, not the tool's)"]
    F -->|"modify · disable ·<br/>needs more evidence"| A
    B -.-> H["🚫 No useful placement<br/>(a valid answer)"]

Loading

Each stage rechecks source identity and requires its own explicit authorization — no stage unlocks the next one automatically.

📚 Learn more

I want to…	Read
Follow the full staged workflow	SKILL.md
Answer intake questions once, up front	ONBOARDING.md
Discover and review candidates in a repo	references/repository-discovery-v1.md
Check qualified platforms and filesystem limits	references/platform-support.md
Understand the 13 A–M placement patterns	references/placement-patterns.md
Pick a placement to prepare	references/experimental-selection.md · references/placement-selection-v1.md
Generate and apply a reviewed code change	references/executable-integrations.md
Render versioned source-bound planner inputs	references/template-catalog-v1.md
Bind recipe C to a reviewed Python console command	references/template-python-entrypoint-v1.md
Generate a bounded method, async, or fixed-positional Python adapter runtime	references/python-adaptation-runtime-v1.md
Build and install a verified Python host in an owned offline environment	references/template-installation-v1.md
Modify JavaScript/TypeScript hosts	references/javascript-typescript-backend-v1.md
Run rigorous before/after experiments	references/experimental-methodology.md · references/lifecycle-and-evidence.md
Inspect synthetic coding-agent completion and retention studies	examples/coding-agent/README.md
Read the frozen issue #44–#49 study outcomes	reports/
Operate multiple placements with budgets and canaries	references/operational-evidence-v1.2.md
Connect to the live JEV API	Live integration below · references/security-and-privacy.md
Coordinate template delivery through reviewed PRs and merge	references/template-delivery-orchestration-prompt.md
Supervise an offline installed Python console and its owned session	references/template-delivery-session-v1.md
See what each release validated	CHANGELOG.md · validation/

Every command also documents itself: python -m jev_integration_evaluator --help.

🔌 Going live

Local scans and demos need no key. To deliberately send approved, minimized decision evidence to the real API (/v1/systemone, pinned to jev-1.13.0), set TYPESAFE_API_KEY and pass explicit network and budget flags:

python -m jev_integration_evaluator replay --input approved-decisions.jsonl --allow-network \
  --max-calls 5 --max-total-cost 0.01 --cost-upper-bound-per-call 0.002 \
  --out approved-replay-results.json

No price is silently assumed, and a model answer is never treated as permission to act — your host policy stays in charge.

Reviewed Python hosts can use the opt-in HostRuntimeLifecycle for connected shadow, canary, or active operation. The host supplies an exact expiring egress grant, a credential reference, a trusted digest-authentication callback, and a durable single-process budget/effect ledger. Canary and active also require raw observed holdouts, a frozen paired study, and exact runtime and deployment receipts; the component recomputes those gates before startup. See the mode and authority table. Call HostRuntimeLifecycle.qualification_inputs() for the redacted inputs needed by an independent live qualification. Offline fixture tests do not establish real provider connectivity, operational safety, or measured benefit.

For the separately reviewed installed Alpha console, or one finite D, H, L, M or E use-case console, a stopped generation transfer can carry one durable ledger and run ID between two versions and back under distinct issuer-signed grants. A pending child needs explicit receipt reconciliation before launch, and the original run cutoff remains binding. The installed two-version tests use only local synthetic TLS protocols. The registered-tool template quickstart orders the whole recipe C journey with exact commands, expected receipts and a support matrix in which provider operation, canary, active and measured benefit stay pending.

⚠️ Honest limits

• No live-model benefit has been measured for this build. All validation to date uses synthetic fixtures and mocked HTTP.
• Static analysis can't see every dynamic call, callback, or generated file. A partial scan is not proof a repo has no opportunities.
• Code changes are supported for bounded, documented Python shapes (and one JS/TS shape); everything else is analysis-only and says so.
• The generated Python adaptation runtime supports normal installed commands in off and offline synthetic shadow for its reviewed regular-package shapes. Connected async/canary/active use and a general multi-statement rewriter are not qualified.
• Native Windows discovery is qualified for local NTFS drive paths on Windows 11 and Windows Server 2022. Windows Server 2022 CI ran the required discovery fixtures on CPython 3.10 and 3.13 with real symlinks and zero skips. UNC/network roots and broader runtime/session workflows have separate limits; see the platform matrix.
• The test runner is not a sandbox; use an isolated environment for untrusted projects.

Full boundaries, trust assumptions, and validation records: references/ and validation/.

📜 License

MIT for code and authored docs — see LICENSE and BRANDING.md. No affiliation with TypeSafe is claimed.

---

_{🧭 jev-integration-evaluator · CompleteTech LLC · Evidence over assertion, always.}