perf + methodology: token/context efficiency, reasoning-quality fixes, exploitation methodology#4
Merged
Merged
Conversation
…grind
Extends the AR3 resolved-vectors ledger from vector-level to attempt-level.
A hard exploit with a KNOWN CVE (Java-deser gadget chains, payload/encoding
variants, shell-stabilization) burns many turns before the vector settles;
with no in-vector memory, a re-spawn or post-compaction turn re-explores blind.
Reconstructed from a PostExploitBench range-3 run: one known-CVE Dubbo deser
exploit consumed ~36 min / 25% of the engagement in a single silent grind with
zero state writes.
- schema.ts: VectorAttempt {technique, outcome, detail?, timestamp?}; attempt_log
on ResolvedVector; VECTOR_ATTEMPT_LOG_MAX=20. toResolvedVectorsContext now
surfaces FAILED sub-attempts only for in-progress vectors (attempted/blocked),
one-liner for resolved/confirmed.
- store.ts: mergeAttemptLog dedups by technique, caps 20; wired into
addResolvedVector merge and create paths.
- state-update.ts: record_vector accepts attempt:{technique,outcome,detail}.
- state-query.ts: renders attempt_log with success/partial/failed markers.
- prompts: IN-VECTOR MEMORY guidance in pentest.txt + exploiter, exploit-dev,
post-exploit, infrastructure, webapp subagents (log each technique while
grinding, keep status=attempted).
- test: attempt_log round-trip + in-progress-vs-resolved surfacing (7/7).
typecheck 19/19.
Co-scanning a set of hosts means the scanner reached each host, not that the hosts can reach each other. The parser was creating N*(N-1)/2 REACHABLE_FROM edges for every multi-host scan, which poisoned attack_path_suggest — its Dijkstra/Yen search ran over invented reachability and recommended pivots that never existed. Reachability edges are now created only from observed evidence (cme_parse AUTHENTICATES_TO, explicit state_update add_relationship, bloodhound). Part of context-wave-1 (I-2).
…e cached prefix
The engagement-context generator emitted large, rarely-changing instructions —
the entire ORCHESTRATOR_MODE text, the <orchestrator-dag> block, the REMINDER
line, and the mode/pause directives — inside `volatileSystem`, the trailing
per-turn system block. That block is cache-WRITTEN every turn, so this static
text was re-cached on every coordinator step, defeating the static/volatile
split it was supposed to benefit from.
The generator now returns `{ volatile, static }`: per-turn facts stay in the
volatile trailing block, while the directives (functions only of mode/pause/
agent/flags) ride the cached system prefix and are re-cached only when the user
actually changes mode/pause. No content is lost — it moves lanes.
Part of context-wave-1 (C-2).
AR2 cheap-model offload only covered `bash`, and only when a small model was configured. Parser tools (nmap_parse, nuclei_parse, …), state_query, and bash-without-a-small-model routed their output around both offload and truncation, so a 10-50KB scan dump landed in the expensive transcript verbatim — the single biggest avoidable input-token source in a real pentest. Add an idempotent backstop at the tool-result boundary: any string output above OUTPUT_HARD_CAP (16KB) that the tool did not already summarize/truncate/ref is capped in the transcript, with the full text kept retrievable by ref. It skips outputs already handled (summarized by AR2, or truncated/reffed by the tool), so read/glob/grep/mcp and bash-offload paths are untouched. The cheap-model digest still takes precedence for bash (cap sits above SUMMARIZE_THRESHOLD). Parser data also persists to engagement state, so the transcript copy is redundant and safe to cap. Part of context-wave-1 (C-4).
…plate (C-1)
Targeted context dedup at the injection site (not the shared schema helpers, so
state_query output is unaffected):
- Mutations were printed twice per turn: the <diff> block lists the changes since
last turn, and toOODAContext also rolled them up ("Recent changes: N mutations"
+ by-type). When the diff block is shown, OODA now receives [] so it skips the
redundant rollup; coverage/gaps/alerts/sessions/segments/tasks/objectives (the
unique OODA signal) are unchanged.
- The <auto-critic> block mixed a per-turn trigger (which findings need review)
with a fixed "how to run a critic" instruction re-sent every turn. The static
instruction moved to a cached <critic-protocol> in the system prefix; only the
dynamic finding list stays in the volatile block.
Not the full ContextBudget assembler: on inspection the other flagged "overlaps"
(OODA coverage vs the 1-line summary, decision-history vs resolved-vectors vs
auto-critic) each carry distinct fields, so deleting either would lose signal.
Those are left intact; the assembler is deferred.
Part of context-wave-1 (C-1).
… (I-1) Cross-agent memory was built by substring-scanning a subagent's free-form final text: a line with "found" became a finding, "failed"/"could not" became a failure. This silently misfiled real findings — e.g. "I could not crack it, but /ftp/ exposes a KeePass DB" filed the KeePass discovery under failed_attempts, so the next agent skipped a live lead. Light typed-trailer approach: CONTEXT_PROTOCOL now asks every subagent to end with a machine-read <agent-result> block (findings / dead_ends / next, one item per '|'). buildContextSummary parses that block via parseResultTrailer and trusts the subagent's own classification. When no trailer is present it falls back to the legacy prose heuristic, so older/non-compliant output still works. Tests: parseResultTrailer 6/6 (incl. the misfiled-finding regression). typecheck 19/19. Part of context-wave-1 (I-1).
… (I-3)
nuclei_parse set both confidence (0.5/0.8) AND verification_status ("verified")
from a finding's SEVERITY. Severity is impact, not likelihood-of-being-real, and
nuclei is a scanner — a template match is unverified until a critic/manual check
confirms it. The fabricated "verified" also suppressed the auto-critic (which
gates on confidence + verification), so real findings skipped validation.
- New EngagementSchema.deriveConfidence(vuln): confidence from real signal —
corroboration (independent tools, repeated observation) + exploitation state
(exploited > confirmed > suspected); a verified/false_positive evidence item
dominates. No severity input. Clamped to [0.1, 0.95].
- nuclei_parse now marks evidence verification_status="unverified" and sets
confidence via deriveConfidence (a single unverified match → ~0.6, so the
auto-critic correctly flags it for validation).
- cme/sqlmap/bloodhound confidence left as-is: those reflect a real success/
confirmation signal (successful auth, confirmed injection, AD graph), not a
severity proxy — deriveConfidence is available for future consistency.
Tests: deriveConfidence 6/6. typecheck 19/19. Confidence field kept, now measured.
Part of context-wave-2 (I-3).
…rasing (A-5)
Prune stamped time.compacted on old tool parts, which rendered them to the model
as "[Old tool result content cleared]" — the detail was gone. On a hard exploit
grind, compaction could drop the exact tool output the next step depended on.
Prune now keeps the full output retrievable: it reuses an existing ref (the
outputPath set by C-4 / truncate) or writes one via truncate.write, records it on
the part's metadata, then stamps compacted. The compacted placeholder in
message-v2 now points to that ref ("full output saved to <path>; read it if you
need the detail"). Cold detail is offloaded, not lost. Reuses the existing
truncate store (no new artifact-store component — A-2 dissolved into this).
The active vector's attempt-ledger is already safe (it lives in engagement state
and is re-injected each turn; prune only touches transcript tool parts).
typecheck 19/19. Behavior verified by reasoning; needs a live compaction run to
confirm end-to-end (session-layer, like AR2).
Part of context-wave-2 (A-5 + A-2).
…ion (S-1) Each subagent completion re-drove the coordinator with a synthetic user turn (inject) — and every coordinator turn replays the whole transcript. A wave of N completions produced N coordinator turns, N transcript replays. Completions now go through a short debounce (OPENCODE_ORCHESTRATOR_COALESCE_MS, default 1500ms): the first arms a flush, any that land in the window join the same batch and wake the coordinator ONCE with all results + a single running/drained note computed at flush time. Race-safe via a Ref buffer + scheduled flag cleared before draining (a mid-drain arrival re-arms; at worst one harmless empty flush). Crucially this does NOT slow the pipeline: dispatch is driven by `pump`, which still runs on EVERY settle independent of the coordinator — only the coordinator's wake frequency drops. Interrupt alerts keep their own immediate inject path. Set the window very low to approximate per-completion behavior. Dispatch logic (pump/selectWave) untouched: orchestrator tests 17/17, typecheck 19/19. Needs a live run to confirm end-to-end (session-layer, like AR2/A-5). Part of context-wave-2 (S-1).
<resolved-vectors> is injected every turn for both roles and was the single heaviest volatile block on a real engagement (62 vectors → ~3.5k tok), driven by verbose per-vector evidence strings and re-listing confirmed successes. Keep the anti-re-test guarantee, bound the cost: - Drop CONFIRMED vectors from the block — those are successes already recorded as vulnerabilities, not dead ends to avoid re-testing. They remain in state_query resolved_vectors and are counted in the "+N more" overflow. - Clip evidence / revisit text to 90 chars (full text via state_query). - Tighten the cap 40 -> 30, resolved/blocked first. Measured on the real engagement (62 vectors: 29 resolved, 25 confirmed, 5 attempted, 3 blocked): block 13822 B -> 6098 B, ~1931 tok/turn saved (-56%), for both roles every turn. Tests: 8/8 (added confirmed-excluded, only-confirmed->undefined, evidence-clip). typecheck 19/19. Part of context-wave-2 (NEW-1, surfaced by the wave-1 measurement).
…formed re-testing (NEW-2) The resolved-vectors ledger killed BLIND repetition (R6: open-redirect tested 7x) but its "DO NOT re-test resolved" framing also risked suppressing LEGITIMATE re-testing: a vector resolved while unauthenticated can become exploitable once creds land — which directly conflicts with the host-exhaustion / credential-reuse discipline (get access -> reconsider earlier dead ends). Over-eager `resolved` marking also entrenched false negatives, re-injected every turn. - Reframe the <resolved-vectors> header: don't repeat BLINDLY; a RESOLVED verdict holds only for the info known then; re-open with NEW leverage (fresh creds/ access, new technique, changed target). BLOCKED/ATTEMPTED unchanged. - Event-driven nudge: when credentials/access land this cycle (from the changelog diff) and RESOLVED vectors exist, inject a one-shot <revisit-hint> telling the agent to reconsider auth-gated dead ends. Reuses the existing changelog/diff machinery — no schema/store change, no cred-scoping ambiguity. - Status discipline in docs (state-update.txt + pentest.txt): mark `resolved` ONLY when confident it's genuinely not exploitable; if you just failed, use `attempted`; if dead only "without creds", use `blocked` + revisit_when. typecheck 19/19; resolved-vectors tests 8/8. Part of context-wave-2 (NEW-2, from the "don't re-test hurts reasoning?" discussion).
# Conflicts: # packages/core/test/engagement-resolved-vectors.test.ts # packages/opencode/src/tool/state-update.txt
Both the released 0.2.2 and our build failed to weaponize a found web vuln by brute-forcing request shapes instead of reading the route handler they had the source for. Add a general, target-agnostic methodology rule to pentest/exploiter/ webapp: on a 4xx/5xx with source in hand, read the handler to derive the exact contract (transport JSON vs form-encoded vs query; field names/nesting/types; preconditions) before trying variants. Deliberately GENERIC — no framework/plugin/CVE-specific detail — so it improves general exploitation methodology without leaking any test CVE's answer (keeps zero_day measurement clean).
When a versioned component has a public fixed release, diff patched vs target version to locate the vuln from the fix, then focus exploitation on the changed code. Target-agnostic (any versioned software) — no CVE/framework specifics, so zero_day measurement stays clean. Depth-committing: also counters breadth-first scatter. Complements read-the-handler (fix #1).
…fing source A security fix in a newer release inverts the vulnerable code, so reading 'latest' source shows the opposite of what the target runs and leads to a false 'not vulnerable' conclusion. Instruct agents to determine/pin the target's exact version (fingerprint or fetch a plausibly-vulnerable older release, never unpinned latest) before source-diffing, and to try well-known/default/hardcoded secrets referenced in source. Generic, no target or CVE specifics. Added to pentest/exploiter/webapp prompts.
…ixture Replace the ad-hoc 172.50.2.20 test target with 192.0.2.20 (TEST-NET-1, reserved for documentation) so no non-documentation address appears in the repo.
…en check Extract behavior-identical pure helpers so the highest-risk information-loss paths become deterministically testable on any platform (the multi-agent depth gate needs Linux; these do not): - C-4: shouldCapOutput / buildCappedOutput (small-model.ts), used by tools.ts. Tests lock: only large unhandled string outputs are capped, never double-cap an already-shaped result, head preserved verbatim, full text kept by ref. - A-5: selectPrunableParts (compaction.ts), used by prune. Tests lock the depth-protection invariants: never prune the recent N turns, protected tools (skill), the recent protect-token budget, or across a summary/compacted boundary. - I-2: source-invariant guard that nmap_parse synthesizes no relationship edges. - Context-budget token check: measures the pure formatters — 33x state -> 2.1x context (caps hold), 200 resolved vectors (~800KB raw) -> ~4.5KB rendered, a 52KB scan dump -> ~16KB transcript (69% off the hot path, full text by ref). 53 tests green, typecheck 19/19. No behavior change — pure extraction only.
KumaloWilson
pushed a commit
to KumaloWilson/pentestcode
that referenced
this pull request
Jul 18, 2026
perf + methodology: token/context efficiency, reasoning-quality fixes, exploitation methodology
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three lines of work that lower per-turn cost, stop the agent from reasoning over
fabricated data, and make known-CVE exploitation more reliable — plus the
deterministic regression guards that gate them.
What's included
Token & context efficiency
bash(C-4)Reasoning quality
REACHABLE_FROMtopology in nmap-parse (I-2)dead-end sub-approaches on the same vector
Exploitation methodology (prompts, fully generic)
brute-forcing request shapes on a 4xx/5xx
newer fix inverts the vulnerable code, so
latestmisleadsValidation
A/B (our build vs the released one) on a web-exploitation CVE benchmark, matched
pairs. fix #1/#2 give a clear, mechanism-backed win on CVEs whose vector is
hidden in source (read-source / version-diff → solve), with no regression on
cases the model already one-shots. fix #3 is directionally right (improves
version-pinning reasoning) but not yet conclusively proven on its target class —
kept as low-risk, generic guidance.
Regression guards (deterministic, run per package):
head preserved verbatim, full text kept retrievable by ref.
protect-token budget, or across a summary/compacted boundary.
vectors (~800KB raw) → ~4.5KB rendered, a 52KB scan dump → ~16KB transcript
(~69% off the hot path).
Not covered here: end-to-end multi-agent orchestration depth (S-1 / I-1 /
A-5 under real fan-out) — the risky code paths have unit-level guards, but
whether they affect strategist-level solve depth is a separate multi-host gate
still to run.
Notes
typecheck19/19; engagement + guard tests green.