Skip to content

Improve dump fidelity, system performance, security, and recovery - #39

Merged
glmcdona merged 8 commits into
mainfrom
glmcdona-process-dump-hardening
Oct 3, 2026
Merged

glmcdona merged 8 commits into
mainfrom
glmcdona-process-dump-hardening

Conversation

@glmcdona

@glmcdona glmcdona commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

Follow-up to #38, based on merged main revision 4763c7e.

  • Preserve virtual section/image sizes independently of packed file size, omit zero-filled raw tails without discarding runtime-initialized data, and align raw sizes/offsets correctly.
  • Put reconstructed imports in a separate .pdimp section when header space permits; otherwise explicitly extend the last section with a warning.
  • Remove obsolete 60,000 KiB section truncation; recover terminal/high-address imports; clear unmapped certificates/stale checksums; relocate debug payload file offsets.
  • Restore original IATs using complete PE32/PE64 pointers and validated source/destination ranges, preserving captured bytes when lookup tables are absent or malformed.
  • Preserve intentional repeated imports at different locations: every reference retains its own loader fixup. No deduplication pass was added.
  • Improve extended-header parsing, export discovery, raw-address dumping, import summaries, module enumeration, database concurrency, and system-dump completion.
  • Optimize reconstruction and parallel entrypoint recovery using measured phase/allocation profiles, without changing established PE/import fingerprints.
  • Add opt-in live-content and controlled-performance benchmarks, native regression cases, portable analyzer tests, and documentation. Machine-specific reports and memory payloads are not included.
  • Close an output-directory reparse race and prevent artifact archive paths from crossing into privileged release jobs; pin Actions and add release provenance checks.
  • Add experimental, default-off -reexec preparation and controlled original/copy/dump execution measurements. Improve loader and runtime startup without changing normal forensic output or disabling mitigations.

Reconstruction regressions and fixes

The first reconstruction revision (40e31c9) reproduced ten failing PE32/PE64 cases on unchanged production code: sparse data with/without imports, sparse RVAs, unaligned import RVAs, and valid large sections. Two more cases reproduced missing terminal imports. The initial suite had 58 passing cases, including 32 varied layout/byte-preservation checks within the layout-matrix cases.

A fixture containing a 4 MiB mostly-zero virtual section now produces at most 12 KiB without imports / 16 KiB with imports, retaining the full virtual extent and initialized bytes. Non-import-reconstructed PE and standalone import-table fingerprints are unchanged. Import-reconstructed fixture goldens intentionally changed in that initial revision to reflect the new .pdimp layout.

The content investigation (94a3434) reproduced PE64 IAT restoration copying DWORDs instead of complete QWORDs, zero OriginalFirstThunk values copying DOS-header bytes into the IAT, and truncated tables being partially rewritten. Full-section tests cover PE32/PE64 restoration, missing lookup tables, truncated source/destination ranges, overlapping/shared thunks, and loader-style rebinding of repeated references. That phase increased the suite to 66 cases. Generated fixture binaries are never executed.

Further pipeline review (868369e)

Reviewed first-party reconstruction, streams, export/import discovery, process/module enumeration, output, databases, and monitoring paths. Added 16 native cases, bringing that revision to 82; ten functional cases were demonstrated failing before their fixes.

  • Section tables: honor SizeOfOptionalHeader instead of assuming the standard C++ header size. Extended PE32/PE64 optional headers retain their actual sections and bytes, with/without import reconstruction.
  • Exports: include ordinal-only and page-aligned exports; exclude zero EAT holes and forwarder-string addresses; reject malformed metadata without publishing partial results. Preserve legitimate loaded redirects outside the exporting image. Live comparisons exposed WOW64 USER32 redirects, so an initially overstrict target-range restriction was removed and preservation coverage added. Forwarder strings themselves are not resolved as code.
  • Raw-address reconstruction: fix the oversized generation argument preventing headerless -a dumps, check generation success, and correct EXE/DLL flags and architecture diagnostics. Self-process cases verify both generated architectures and captured raw bytes.
  • Import summaries: include full pointers at the range boundary, count libraries, and distinguish ordinal identities. Affected loose-code hashes can change; README documents regenerating clean databases when relying on them. Per-location fixups remain independent.
  • Module enumeration: replace the fixed 2,048-handle assumption with bounded resizing/retries. Cover 4,097 handles, continually growing lists, and API failures. Limits are eight attempts and 1,048,576 handles; failures are reported.
  • Database reliability: initialize all options, synchronize lookups with insertions/rehashing, and release owned database paths. A four-worker test verifies all three sets through 76,800 insertions each.
  • Traversal/completion: stop hash scans at the region boundary, preserve loose regions adjacent to modules, skip uncommitted regions, and finish the final system-dump scan even if all workers have exited.

Initial live reconstruction comparison

The initial investigation used a 515-process inventory, dumping accessible main executables with saved before/after builds. Pairing requires the same PID, creation time, original path/hash, and import mode. It does not cover every DLL/private allocation.

Initial paired metric Imports disabled Imports enabled
Comparable processes 329 329
Structural failures before / after 223 / 0 223 / 0
Lost executable-code blocks before / after 0 / 0 0 / 0
Newly introduced dump failures 0 0
Aggregate output bytes before 9,920,569,344 10,269,224,960
Aggregate output bytes after 9,915,600,896 10,264,256,512

The inventory separately recorded 144 unavailable processes, 38 exited/replaced/inaccessible processes, one missing original, one image over the 256 MiB limit, and two exclusions after Defender quarantined earlier output. Spotify was not running, so its exact reported symptom could not be tested live. One additional target was quarantined during the deeper investigation; subsequent sweeps exclude all three without bypassing protection.

Deep section-content comparisons

The IAT-fix phase selected one process per executable path. It produced 73 analyzable images per mode, with 72 pairs against 40e31c9. Layout-mismatched original files are excluded from authoritative IAT and memory-fidelity totals.

IAT-fix paired metric (94a3434) Imports disabled Imports enabled
Paired images 72 72
Matching-layout pairs 71 67
Incorrect original-IAT slots before / after 18,167 / 0 17,353 / 0
Pairs with independent memory witnesses 71 67
Stable unexplained witness differences after 0 0
Newly introduced dump failures 0 0

The pipeline-review phase paired 73 images per mode, with matching-layout subsets of 72 without imports / 68 with imports. It found no new dump failures, structural failures, lost code blocks, original-IAT mismatches, missing original imports, or stable unexplained witness differences. Import entries increased 279,864 → 291,272, across 11 images, with no decreases or export-table warnings. These live counts can vary with runtime activity and do not prove that every aggressive match is a necessary import.

Pipeline-review content metric Imports disabled Imports enabled
Matching-layout images 72 68
Original section bytes compared 1,101,033,627 1,088,059,482
.text bytes compared 673,607,382 663,325,106
Unexplained .text differences 0 0
Resource payloads compared 2,817 2,711
Changed resource payloads 0 0
Original-IAT mismatches / missing imports 0 / 0 0 / 0
Stable unexplained witness differences 0 0

Comparisons align section RVAs, normalize HIGHLOW/DIR64 relocations and rewritten debug file offsets, inspect complete IAT pointers against original lookup values, and hash resource payloads. Independent read-only pagewise reads distinguish changing live bytes from stable bytes differing from the reconstruction. Intentional IAT/debug rewrites are excluded from witness comparisons, not their dedicated checks. Unsupported relocation types, unreadable pages, analyzer errors, inaccessible/quarantined output, and mismatched layouts are reported separately, never treated as successful fidelity checks.

Small non-code differences faithfully captured from running memory are not reconstruction loss. Loaded/on-disk layout differences can make original-file comparisons inconclusive; an apparent resource loss in one such image was not established data loss. Last-section extension can also prevent a layout match. Snapshots are non-atomic and cannot detect a byte changing and changing back between reads. Dumps are analysis artifacts, not guaranteed runnable or signed copies. Raw dumps are removed after comparison; private reports contain counts/hashes and local metadata, not memory payloads.

Performance review and optimizations (9788b66)

Added an opt-in native benchmark and Python comparison runner. Profiling exposed millions of per-import allocations, redundant full-image copies, repeated export-name copies, and an exclusive database lock acquired for each candidate entrypoint byte. The last bottleneck made parallel recovery substantially slower than single-worker recovery.

  • Store import records contiguously, with geometric capacity growth and inline descriptors/thunks. Reconstruction borrows names from its immutable export list; string-based APIs still copy/own caller strings. Every location and the serialized ordering/format remain intact.
  • Use one filtered export lookup instead of set lookup + map lookup + deep copy. Transfer parsed export ownership without recopying strings, preserving duplicate-address precedence and stable record addresses.
  • Pack directly from captured bytes when imports are disabled, zero only the newly added import tail, and scan zero tails a machine word at a time.
  • Reuse the dumper's existing process handle while preserving module names and borrowed-handle lifetime. Avoid export discovery when both imports and loose chunks are disabled.
  • Share immutable, cached entrypoint snapshots between workers. Invalidate on entrypoint changes/clear, retain old snapshots safely for in-flight scans, skip empty databases, and filter impossible short hashes before lookup. No per-byte shared lock and no per-image database copy. Clean-hash deduplication lookups remain synchronized; input sets are passed by const reference.

Reproducible measurements

Compared saved 868369e production sources against the final code using the same harness and Release flags, with 16 jobs, 1/2/4/8 workers, five alternating trials per combination. Separate allocation-instrumented runs used eight jobs, 1/4 workers, and three alternating trials. The allocation hook exists only in the test host and cannot be combined with ASan.

Median worker processing elapsed milliseconds, before → after (not sums across workers):

Workload 1 worker 4 workers 8 workers
Dense 4 MiB 291.57 → 287.60 319.27 → 334.43 317.64 → 301.25
Sparse 32 MiB 566.04 → 213.46 215.28 → 99.94 162.15 → 84.65
Import-heavy, 65,536 fixups 729.79 → 316.17 318.58 → 211.66 316.29 → 256.44
Missing entrypoint, nonmatching database 1,068.11 → 145.94 5,226.80 → 82.37 12,277.27 → 81.45
Missing entrypoint, empty database 1,068.76 → 100.29 5,524.67 → 74.31 12,646.17 → 75.19
Export discovery in test process 215.16 → 118.10 67.26 → 42.13 45.82 → 31.57

Single-worker reconstruction phases alone improved 601.45 → 166.11 ms for imports and 396.69 → 46.41 ms for sparse images. C++ allocations for eight import-heavy jobs fell 3,670,384 → 272; export discovery fell 1,118,856 → 396,576. Requested allocation bytes decreased, rather than merely consolidating larger allocations. Eight-worker sparse peak working set fell 578,727,936 → 310,276,096 bytes (about 46%).

Dense-output throughput remains I/O-bound/noisy despite reduced reconstruction CPU work: its four-worker median was slightly slower, and individual eight-worker candidate samples ranged from 247.72 to 378.07 ms. General clean-database query stress is essentially unchanged. More workers do not universally help. Removing entrypoint serialization increases simultaneous in-flight images: eight-worker nonmatching-entrypoint peak working set rose from about 54.5 MB to 79.6 MB despite lower per-image allocation. These tradeoffs are recorded rather than claiming uniform speedups.

The harness reports capture/reconstruction/write/cleanup service sums, worker elapsed time, thread-teardown-inclusive wall time, CPU cycles/coarse CPU time, and peak working set. Setup, filename reservation, thread creation, output verification, and deletion are outside the timed intervals. Filesystem caching, security scanning, scheduling, CPU frequency, and memory bandwidth still affect results; this is not an isolated full-system -system scheduler benchmark. No protection was disabled. Further speculative traversal/scheduling rewrites were not made without evidence of benefit and preserved coverage.

Every serialized benchmark job retained its baseline size and CRC32, normalizing only the generated PE64 fixture's ASLR-dependent ImageBase. Database queries independently assert the expected hit count; export discovery has no serialized fingerprint and is covered by native semantic tests plus live comparisons.

Final fidelity check after optimization

The final executable again paired 72 live images in each import mode with 868369e, with 71 / 67 matching-layout witness pairs. Aggregate output sizes were unchanged: 1,272,950,784 bytes without imports; 1,300,660,224 with imports. There were no new dump failures, structural failures, lost code blocks, original-IAT mismatches, missing original imports, changed resource payloads, unexplained .text bytes, or stable unexplained memory-witness differences. Matching subsets compared 673,570,956 / 663,288,680 .text bytes and 2,816 / 2,710 resource payloads. Unavailable/changed processes, missing originals, safety-limit exclusions, and known security-software exclusions remain explicitly reported.

Added ten native cases for moved/borrowed import equivalence, export ownership/alias precedence, cached-snapshot reuse/invalidation/concurrency/lifetime, file-based database updates, PE32/PE64 weak/strong entrypoint selection, and borrowed process-handle identity/lifetime. Added portable performance-parser/comparator cases. That revision's suite was 92 native + 18 Python tests.

Local validation before the final security pass

  • Win32 Debug: 92/92 native tests
  • Win32 Release: 92/92 native tests
  • x64 Debug: 92/92 native tests
  • x64 Release: 92/92 native tests
  • x64 AddressSanitizer: 92/92 native tests; runtime instrumentation verified
  • 18 portable analyzer/performance-runner tests
  • Alternating normal/instrumented performance comparisons and unchanged serialized fingerprints
  • Final live deep-content comparisons in both import modes
  • Whitespace checks and exact identity of benchmarked/current x64 application at that revision
  • Benchmark-created raw output cleanup; pre-existing temporary files left untouched

Final native and GitHub Actions security pass (e70c0cc)

Two additional source-supported vulnerabilities were identified and fixed. Locations below refer to the pre-fix code. No exploit PoCs were created or executed.

# Severity File Lines Vulnerability Confidence
1 🟡 MEDIUM pd/dump_output.h 21–29, 81–91 Intermediate junction metadata can change after parent validation, redirecting elevated new-file creation 9/10
2 🟡 MEDIUM .github/workflows/deploy-create-releases.yml (also both other publishers) 40–47 Untrusted artifact ZIP extraction can overwrite code executed in the write-enabled publisher 9/10

Native output-path race

Holding directories without delete sharing prevents renaming/replacement but does not prevent an in-place reparse metadata change. The earlier final full-path CreateFileA could follow an intermediate junction introduced after its validation. This requires an attacker-writable, initially empty directory, an elevated dumper, and a successful race; existing files still could not be overwritten. The defect also exists in merged 4763c7e.

Parent traversal and final creation now use root-relative NtCreateFile operations with OBJ_DONT_REPARSE; the verified root and parent handles remain open through writing. Final creation retains FILE_CREATE, exclusive sharing, synchronous writes, and handle-based incomplete-file cleanup. Native failures propagate through RtlNtStatusToDosError, with no weaker retry. Added ntdll.lib consistently for application and test builds.

Three new native cases cover ordinary nested output, invalid native names/roots, and mocked reparse/access/parameter failures. The mock asserts the exact root-relative name, creation/directory flags, sharing, single-call behavior, and error propagation. Existing file-collision, raw-dump, content, and serialization cases remain unchanged. No live junction exploit was used, and UNC/network-share integration was not exercised.

Privileged artifact boundary

Review of the resolved actions/download-artifact@v4 implementation (d3f86a106a0bac45b974a628896c90dbdf5c8093) found automatic extraction using a filename guard that strips leading traversal but lacks canonical destination containment: guard, write path. A hostile build producer controls its uploaded ZIP, not merely the path specified in the trusted upload step. Selecting that source for publication authorizes binary bytes, not execution inside the publisher holding contents: write. A fork PR alone does not enter the publication path.

All three publishers now check out only trusted preparation code from github.workflow_sha, not the manually selected build SHA. The helper lists artifacts from the current repository/run, requires both unique unexpired architecture artifacts, and accepts exactly one regular pd.exe entry per archive. It validates metadata, CRC, and 64 MiB archive/uncompressed limits, then copies data into fixed exclusively created output names. It never extracts archive-controlled paths or runs downloaded binaries. Signed HTTPS blob redirects receive no repository token; additional redirects and API errors fail closed.

Added ten portable cases for same-run pagination/name selection, duplicate/missing/expired/oversized artifacts, entry metadata, benign compressed/stored roundtrips, overwrite refusal, invalid archives, bounded reads, tag identity, HTTP failures, redirect credential isolation, and end-to-end two-binary manifest preparation. Tests use benign data and mocks, not hostile archive payloads.

Additional release hardening and review conclusions

  • Pin checkout, upload-artifact, and setup-msbuild to resolved official full SHAs; disable checkout credential persistence. Pinning alone would not have fixed the extractor issue.
  • Keep PR/build jobs read-only and publishers separate. Only publishers receive contents: write and the actions: read needed by the REST artifact reader. No pull_request_target, privileged workflow_run consumer, inherited build secrets, cross-run artifact selection, or cache-consumption path was found.
  • No additional shell-expression injection or reusable-workflow permission escalation was identified. Manually choosing a fetchable full commit SHA remains an authorized maintainer capability, not an implicit main-only policy.
  • Reject existing version tags that resolve to a different commit, including annotated tag dereferencing. Serialize publication per release across manual/tag publishers, and serialize Develop updates without cancelling an upload midway. Automatic main builds check freshness before publication; manual development dispatch remains intentional.
  • Publish provenance.json containing the selected source commit, run/artifact IDs, sizes, and SHA-256 hashes. This is an asset-integrity record, not source-to-binary attestation. Develop retains its historical unmoved tag; its notes and manifest identify the current asset commit. GitHub multi-asset uploads are not transactional: interrupted uploads can leave a partial set, so both hashes must match the manifest.

Final security validation

  • 95/95 native tests in each Win32/x64 Debug/Release configuration
  • 95/95 x64 AddressSanitizer tests, with the ASan runtime and native dependency verified
  • 28/28 portable tests (18 existing analyzer/performance tests plus 10 release tests)
  • Existing PE/import fingerprints and reconstruction assertions unchanged
  • All workflows pass actionlint 1.7.7
  • Trusted preparation script successfully read both real artifacts from prior CI run 36980358932, with independently verified manifest hashes; neither executable was run, and downloaded binaries were removed
  • Whitespace checks and clean committed workspace

The publisher's preparation boundary was tested locally; no release was created, overwritten, or manually dispatched for validation. CI build matrices run on the new PR revision. The review is not a guarantee that no further vulnerabilities exist.

Reconstructed application execution (b30b441)

Added default-off -reexec, guided by failures from newly launched, debugger-owned benign children. Ordinary forensic dumping remains unchanged; preparation does not execute an image and does not restore a process checkpoint.

  • Captured GS cookies caused Microsoft images to fail before entrypoint with 0xc000007b. Restoring guard-function slots did not help; restoring the compiler's cookie bootstrap sentinel did. Zero was not an equivalent replacement. Validate load-config version/ranges and writable pointer-sized storage, then serialize the PE32/PE64 sentinel without modifying captured source bytes or weakening mitigations.
  • Aggressive scanning mistook x86 instruction bytes for exported function pointers. Additional normal imports also targeted delay-IAT sections that Windows protects before normal binding. Restrict new fixups to originally file-backed writable/non-executable data, excluding the GS cookie and sections containing delay IATs. Keep existing imports and independent repeated eligible fixups; preserve default aggressive analysis behavior.
  • Initialized test hosts retained stale CRT heap pointers in original zero-fill storage. Reset only writable/non-executable tails defined by the captured original raw/virtual section sizes, retaining initialized file-backed data. Cover pure BSS, cookies in BSS, occupied-header fallback, both import modes and architectures, and invalid bounds. This assumes original section layout and intentionally discards captured state; it is inappropriate for faithful unpacked-data analysis.

Controlled results

The opt-in Python runner observes the original application, an untouched staged disk-copy control and the reconstructed image using the same architecture-matched native debugger probe. It restores its entrypoint breakpoint before capture, uses kill-on-close jobs for its own children, and records loader/entrypoint/GUI/fixture milestones, exceptions, hashes and exit codes. Entry/fixture-marker captures suspend the child through a debug event; GUI-idle capture is live. Dump time is excluded. Optional explicitly selected MUI resources are staged beside both copies; no original executable code is substituted into the dump. Failed or changed controls are inconclusive, and absent output hashes never count as equality.

Two trials on Windows 11 build 22631, six application images, both import modes and relevant capture phases produced 40 matched baseline/prepared pairs against e70c0cc: 34 gained milestones, six were unchanged, zero lost milestones. Every candidate reached its entrypoint.

Application / capture Candidate result Runs
Frozen x86/x64 native test hosts, entrypoint and initialized-CRT marker Threaded fixture completed, exit zero, identical 14-byte output 16
x64 sort.exe, entrypoint Exit zero, exact expected sorted 30-byte output 4
x64 findstr.exe /?, entrypoint with matching MUI Exit zero, identical 2,426-byte help output 4
x86/x64 Notepad, entrypoint with matching MUI GUI input-idle, matching original and staged-copy controls 8
x86/x64 Notepad, GUI-idle capture Improved from loader rejection to entrypoint, then fail-fast 0xc0000409, parameter 5 8

Both original and staged-copy output hashes matched all candidate runs; empty GUI output is not treated as functional proof. Input-idle is startup progress, not full editor functionality. The remaining initialized Notepad failure is documented, not hidden by resetting arbitrary globals or disabling ASLR/DEP/CFG/delay-IAT protections. Heaps, handles, TLS, initialized file-backed globals and external state remain outside this mode's restoration scope.

The execution harness is not a security sandbox. It requires explicit acknowledgement, must only run trusted benign bounded workloads, and never runs automatically in CI or the normal unit suite. Reports and diagnostic payloads remain private; benchmark captures are cleaned up and the six retained investigation executables were removed. No exploit PoCs or security-software bypasses were used.

Reexecution validation

  • 107/107 native tests in each Win32/x64 Debug/Release configuration, including 12 new reconstruction cases
  • 107/107 x64 AddressSanitizer tests, with runtime instrumentation verified
  • 38/38 portable tests, including ten reexecution harness/comparison cases
  • Existing default PE/import fingerprints unchanged
  • All 40 paired comparisons checked against the final strict control validation; expected sorted bytes verified independently
  • Additional final-probe paired smoke runs for initialized x86 host and x64 Notepad startup
  • Missing-application diagnostics retain Windows error 2; no owned benchmark children remain
  • Benchmarked/current x64 dumper identity verified, whitespace checks passed, changes committed and pushed

glmcdona and others added 2 commits October 1, 2026 22:53
Preserve virtual layout while packing initialized section data, isolate reconstructed imports, and repair alignment, boundary imports, and file-only metadata. Add regression matrices and an opt-in local process comparison harness.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Validate complete import lookup tables before restoring pointer-sized IAT entries. Preserve missing, truncated and overlapping tables, and retain independent repeated import fixups. Add deep section, relocation, resource and independently witnessed memory comparisons to the opt-in benchmark.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona glmcdona changed the title Fix sparse PE reconstruction and add live process dump benchmarks Fix sparse PE reconstruction and IAT fidelity with live byte comparisons Oct 2, 2026
Honor extended optional headers, recover ordinal and page-aligned exports without treating forwarder strings as addresses, and preserve live redirected exports. Fix raw-address generation, image type flags, terminal import summaries and module snapshot growth. Synchronize hash database reads and initialize dump options. Correct region boundaries and system-dump completion. Add 16 regressions and document behavior changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona glmcdona changed the title Fix sparse PE reconstruction and IAT fidelity with live byte comparisons Fix PE dump reconstruction fidelity and pipeline quality Oct 2, 2026
Profile reconstruction phases and allocation churn, share immutable entrypoint snapshots, preserve per-location imports with compact borrowed-name records, reuse process handles and transfer export ownership. Add controlled before/after benchmarks and regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona glmcdona changed the title Fix PE dump reconstruction fidelity and pipeline quality Fix PE dump fidelity, quality, and performance Oct 2, 2026
Use root-relative no-reparse native file operations and fail closed. Copy only validated binary bytes from current-run artifacts in trusted publishers; pin Actions, verify version-tag identity, serialize publication, and record asset provenance.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona glmcdona changed the title Fix PE dump fidelity, quality, and performance Fix PE dump fidelity, performance, and release security Oct 2, 2026
Restore validated GS cookie bootstrap values and PE-defined writable zero-fill state, restrict speculative imports to safe original data ranges, and add controlled original/copy/dump execution comparisons with native and portable regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona glmcdona changed the title Fix PE dump fidelity, performance, and release security Improve dump fidelity, performance, security, and reexecution Oct 2, 2026
Parallelize bounded file database generation and add reproducible tail scheduling, database and entrypoint corpus comparisons. Preserve output and hash equivalence while avoiding weak-only recovery guesses.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona glmcdona changed the title Improve dump fidelity, performance, security, and reexecution Improve dump fidelity, system performance, security, and recovery Oct 2, 2026
@glmcdona

glmcdona commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner Author

Module-level system scheduling, entrypoint recovery, and database throughput

Added in 990a932. Existing fidelity, security, performance and opt-in reexecution work is retained.

System worker pool

-system now shares fixed persistent workers across process scans, export parsing, individual modules and loose-code jobs. Idle workers can assist the last process rather than waiting on a process-owned thread. Waiting producers help only leaf jobs, avoiding both thread starvation and recursively retained process contexts. No nested thread pools; -t remains the worker bound and -nt uses one worker.

Process handles, module snapshots and export tables remain alive until their leaf jobs drain. Export aliases merge in original snapshot order. Leaf batches are bounded at 256 jobs per producer; file database work also drains every 256 queued files. One region walk discovers both MZ candidates and executable heaps; hidden/headerless discovery, per-region limits and chunk selection remain. System dumping drains all initial work before one final new-PID pass. Database hashing no longer mutates import-reconstruction options.

Final-build isolated measurements, three alternating trials:

Workload Serial/before Four workers 16 workers
Last-process dump + hash, imports/chunks on 523.02 ms 342.58 ms 320.87 ms
Last-process dump + hash, imports/chunks off 373.07 ms 183.16 ms 165.31 ms
Last-process hashing, chunks on 119.37 ms 70.08 ms 67.29 ms
-db add, 621 staged PE files 500.65 ms 240.28 ms 221.62 ms

The one-worker database candidate took 478.35 ms. All three database sets were identical. The first baseline trial was cold (10.94 s), retained in the report and not presented as a CPU speedup. The system fixture models tail imbalance, not full-machine throughput; filesystem/security scanning and scheduling affect timings, and 16 workers are not universally faster.

All 96 final timed system samples retained identical output filenames/bytes and all three hash sets. An earlier 96-run measurement, 24 generated-header/reexecution option samples and six Win32 integration samples also retained equivalence. Eight option combinations remained byte-identical against the saved pre-change application (78 normal or 156 generated images).

Entrypoint recovery

Evaluated seven policies on 621 SHA-256-unique local system/application images; 577 had suitable nonzero executable entrypoints (263 PE32, 314 PE64). The corpus is read only, never executed.

Production recovery now preserves valid executable entrypoints (including RVA 0x2000 and low-alignment images), scans executable storage, requires a full opcode hash, uses deterministic earliest exact-prefix/full matches regardless of verbosity, and never overwrites the entrypoint on a weak prefix alone. When exact-prefix matching fails, bounded x64 runtime-function metadata provides a unique-full-match fallback. PE64 runtime metadata parsing also works in Win32 builds. -nep and forced -eprec remain supported; unsuccessful recovery retains the prior RVA with a warning.

Leave-one-image-out result Legacy Selected production policy
Correct 323 357
Wrong 252 11
Unresolved 2 209

Both policies recovered 577/577 when the image's own signatures were present. Exact-only matching produced 316 correct / 7 wrong / 254 unresolved held-out results: runtime-function fallback adds 41 correct recoveries at the cost of four additional wrong guesses. Requiring uniqueness for every exact match lost seven known-image recoveries; x86 call-target scanning added no correct recoveries and was omitted. These are heuristic signatures, not perfect recovery. Leave-one-image-out retains signatures shared by other files and is not leave-one-family-out validation.

Integration and validation

  • 115 native cases pass in Win32/x64 Debug/Release and x64 ASan; 41 portable tests pass.
  • New tests exercise worker lifetime/concurrency bounds, failure propagation/draining, final-process parallelism, final PID discovery, adjacent-heap preservation, recursive/filtered parallel file databases, and entrypoint selection/malformed/ambiguous metadata.
  • A real private-database -db genquick -t 4 -nh run completed (3,237 clean / 484 full-entrypoint / 121 short-entrypoint hashes). A subsequent default -system -t 4 run completed with imports and loose chunks enabled, producing 507 captures (~231 MiB). Captures were removed after recording hashes. Existing access restrictions and the 256 MiB image bound remain in force.
  • Full -db gen over all Windows/user/program folders was not run end-to-end; its live-hashing and recursive file-worker components, filters and all three set outputs were exercised separately.
  • Reproducible runners and methodology are documented in README. Private reports/manifests stay outside the repository; no process-memory payloads or exploit PoCs are included.

Both final CI matrices passed against this commit: push and pull request.

Share the version across the banner and Windows resources; document all flags, aliases, defaults, database effects and reconstruction caveats. Exit after help and exercise CLI documentation and built metadata in every CI configuration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@glmcdona

glmcdona commented Oct 3, 2026

Copy link
Copy Markdown
Owner Author

Updated in ad5b188: source/banner version 3.0.0 and Windows file/product version 3.0.0.0 now share pd/version.h. Refreshed help and README with every parsed flag and alias, actual defaults and database effects, module-level scheduling, conservative entrypoint recovery, and -reexec caveats. Corrected -a examples, regex syntax, %USERPROFILE%, recursion scope, and all-three-database clean/ignore behavior. Help now exits without running a requested database or dump command. Added CLI/documentation/version regression checks to each CI configuration; all four local builds passed 115 native and 47 Python tests. README includes an unreleased 3.0.0 changelog. No release/tag was created.

@glmcdona
glmcdona merged commit 7b81e74 into main Oct 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant