Repository navigation
Improve dump fidelity, system performance, security, and recovery - #39
Conversation
Preserve virtual layout while packing initialized section data, isolate reconstructed imports, and repair alignment, boundary imports, and file-only metadata. Add regression matrices and an opt-in local process comparison harness. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Validate complete import lookup tables before restoring pointer-sized IAT entries. Preserve missing, truncated and overlapping tables, and retain independent repeated import fixups. Add deep section, relocation, resource and independently witnessed memory comparisons to the opt-in benchmark. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Honor extended optional headers, recover ordinal and page-aligned exports without treating forwarder strings as addresses, and preserve live redirected exports. Fix raw-address generation, image type flags, terminal import summaries and module snapshot growth. Synchronize hash database reads and initialize dump options. Correct region boundaries and system-dump completion. Add 16 regressions and document behavior changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Profile reconstruction phases and allocation churn, share immutable entrypoint snapshots, preserve per-location imports with compact borrowed-name records, reuse process handles and transfer export ownership. Add controlled before/after benchmarks and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use root-relative no-reparse native file operations and fail closed. Copy only validated binary bytes from current-run artifacts in trusted publishers; pin Actions, verify version-tag identity, serialize publication, and record asset provenance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Restore validated GS cookie bootstrap values and PE-defined writable zero-fill state, restrict speculative imports to safe original data ranges, and add controlled original/copy/dump execution comparisons with native and portable regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Parallelize bounded file database generation and add reproducible tail scheduling, database and entrypoint corpus comparisons. Preserve output and hash equivalence while avoiding weak-only recovery guesses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Module-level system scheduling, entrypoint recovery, and database throughputAdded in 990a932. Existing fidelity, security, performance and opt-in reexecution work is retained. System worker pool
Process handles, module snapshots and export tables remain alive until their leaf jobs drain. Export aliases merge in original snapshot order. Leaf batches are bounded at 256 jobs per producer; file database work also drains every 256 queued files. One region walk discovers both MZ candidates and executable heaps; hidden/headerless discovery, per-region limits and chunk selection remain. System dumping drains all initial work before one final new-PID pass. Database hashing no longer mutates import-reconstruction options. Final-build isolated measurements, three alternating trials:
The one-worker database candidate took 478.35 ms. All three database sets were identical. The first baseline trial was cold (10.94 s), retained in the report and not presented as a CPU speedup. The system fixture models tail imbalance, not full-machine throughput; filesystem/security scanning and scheduling affect timings, and 16 workers are not universally faster. All 96 final timed system samples retained identical output filenames/bytes and all three hash sets. An earlier 96-run measurement, 24 generated-header/reexecution option samples and six Win32 integration samples also retained equivalence. Eight option combinations remained byte-identical against the saved pre-change application (78 normal or 156 generated images). Entrypoint recoveryEvaluated seven policies on 621 SHA-256-unique local system/application images; 577 had suitable nonzero executable entrypoints (263 PE32, 314 PE64). The corpus is read only, never executed. Production recovery now preserves valid executable entrypoints (including RVA
Both policies recovered 577/577 when the image's own signatures were present. Exact-only matching produced 316 correct / 7 wrong / 254 unresolved held-out results: runtime-function fallback adds 41 correct recoveries at the cost of four additional wrong guesses. Requiring uniqueness for every exact match lost seven known-image recoveries; x86 call-target scanning added no correct recoveries and was omitted. These are heuristic signatures, not perfect recovery. Leave-one-image-out retains signatures shared by other files and is not leave-one-family-out validation. Integration and validation
Both final CI matrices passed against this commit: push and pull request. |
Share the version across the banner and Windows resources; document all flags, aliases, defaults, database effects and reconstruction caveats. Exit after help and exercise CLI documentation and built metadata in every CI configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Updated in ad5b188: source/banner version 3.0.0 and Windows file/product version 3.0.0.0 now share pd/version.h. Refreshed help and README with every parsed flag and alias, actual defaults and database effects, module-level scheduling, conservative entrypoint recovery, and -reexec caveats. Corrected -a examples, regex syntax, %USERPROFILE%, recursion scope, and all-three-database clean/ignore behavior. Help now exits without running a requested database or dump command. Added CLI/documentation/version regression checks to each CI configuration; all four local builds passed 115 native and 47 Python tests. README includes an unreleased 3.0.0 changelog. No release/tag was created. |
Summary
Follow-up to #38, based on merged
mainrevision4763c7e..pdimpsection when header space permits; otherwise explicitly extend the last section with a warning.-reexecpreparation and controlled original/copy/dump execution measurements. Improve loader and runtime startup without changing normal forensic output or disabling mitigations.Reconstruction regressions and fixes
The first reconstruction revision (
40e31c9) reproduced ten failing PE32/PE64 cases on unchanged production code: sparse data with/without imports, sparse RVAs, unaligned import RVAs, and valid large sections. Two more cases reproduced missing terminal imports. The initial suite had 58 passing cases, including 32 varied layout/byte-preservation checks within the layout-matrix cases.A fixture containing a 4 MiB mostly-zero virtual section now produces at most 12 KiB without imports / 16 KiB with imports, retaining the full virtual extent and initialized bytes. Non-import-reconstructed PE and standalone import-table fingerprints are unchanged. Import-reconstructed fixture goldens intentionally changed in that initial revision to reflect the new
.pdimplayout.The content investigation (
94a3434) reproduced PE64 IAT restoration copying DWORDs instead of complete QWORDs, zeroOriginalFirstThunkvalues copying DOS-header bytes into the IAT, and truncated tables being partially rewritten. Full-section tests cover PE32/PE64 restoration, missing lookup tables, truncated source/destination ranges, overlapping/shared thunks, and loader-style rebinding of repeated references. That phase increased the suite to 66 cases. Generated fixture binaries are never executed.Further pipeline review (
868369e)Reviewed first-party reconstruction, streams, export/import discovery, process/module enumeration, output, databases, and monitoring paths. Added 16 native cases, bringing that revision to 82; ten functional cases were demonstrated failing before their fixes.
SizeOfOptionalHeaderinstead of assuming the standard C++ header size. Extended PE32/PE64 optional headers retain their actual sections and bytes, with/without import reconstruction.-adumps, check generation success, and correct EXE/DLL flags and architecture diagnostics. Self-process cases verify both generated architectures and captured raw bytes.Initial live reconstruction comparison
The initial investigation used a 515-process inventory, dumping accessible main executables with saved before/after builds. Pairing requires the same PID, creation time, original path/hash, and import mode. It does not cover every DLL/private allocation.
The inventory separately recorded 144 unavailable processes, 38 exited/replaced/inaccessible processes, one missing original, one image over the 256 MiB limit, and two exclusions after Defender quarantined earlier output. Spotify was not running, so its exact reported symptom could not be tested live. One additional target was quarantined during the deeper investigation; subsequent sweeps exclude all three without bypassing protection.
Deep section-content comparisons
The IAT-fix phase selected one process per executable path. It produced 73 analyzable images per mode, with 72 pairs against
40e31c9. Layout-mismatched original files are excluded from authoritative IAT and memory-fidelity totals.94a3434)The pipeline-review phase paired 73 images per mode, with matching-layout subsets of 72 without imports / 68 with imports. It found no new dump failures, structural failures, lost code blocks, original-IAT mismatches, missing original imports, or stable unexplained witness differences. Import entries increased 279,864 → 291,272, across 11 images, with no decreases or export-table warnings. These live counts can vary with runtime activity and do not prove that every aggressive match is a necessary import.
.textbytes compared.textdifferencesComparisons align section RVAs, normalize HIGHLOW/DIR64 relocations and rewritten debug file offsets, inspect complete IAT pointers against original lookup values, and hash resource payloads. Independent read-only pagewise reads distinguish changing live bytes from stable bytes differing from the reconstruction. Intentional IAT/debug rewrites are excluded from witness comparisons, not their dedicated checks. Unsupported relocation types, unreadable pages, analyzer errors, inaccessible/quarantined output, and mismatched layouts are reported separately, never treated as successful fidelity checks.
Small non-code differences faithfully captured from running memory are not reconstruction loss. Loaded/on-disk layout differences can make original-file comparisons inconclusive; an apparent resource loss in one such image was not established data loss. Last-section extension can also prevent a layout match. Snapshots are non-atomic and cannot detect a byte changing and changing back between reads. Dumps are analysis artifacts, not guaranteed runnable or signed copies. Raw dumps are removed after comparison; private reports contain counts/hashes and local metadata, not memory payloads.
Performance review and optimizations (
9788b66)Added an opt-in native benchmark and Python comparison runner. Profiling exposed millions of per-import allocations, redundant full-image copies, repeated export-name copies, and an exclusive database lock acquired for each candidate entrypoint byte. The last bottleneck made parallel recovery substantially slower than single-worker recovery.
Reproducible measurements
Compared saved
868369eproduction sources against the final code using the same harness and Release flags, with 16 jobs, 1/2/4/8 workers, five alternating trials per combination. Separate allocation-instrumented runs used eight jobs, 1/4 workers, and three alternating trials. The allocation hook exists only in the test host and cannot be combined with ASan.Median worker processing elapsed milliseconds, before → after (not sums across workers):
Single-worker reconstruction phases alone improved 601.45 → 166.11 ms for imports and 396.69 → 46.41 ms for sparse images. C++ allocations for eight import-heavy jobs fell 3,670,384 → 272; export discovery fell 1,118,856 → 396,576. Requested allocation bytes decreased, rather than merely consolidating larger allocations. Eight-worker sparse peak working set fell 578,727,936 → 310,276,096 bytes (about 46%).
Dense-output throughput remains I/O-bound/noisy despite reduced reconstruction CPU work: its four-worker median was slightly slower, and individual eight-worker candidate samples ranged from 247.72 to 378.07 ms. General clean-database query stress is essentially unchanged. More workers do not universally help. Removing entrypoint serialization increases simultaneous in-flight images: eight-worker nonmatching-entrypoint peak working set rose from about 54.5 MB to 79.6 MB despite lower per-image allocation. These tradeoffs are recorded rather than claiming uniform speedups.
The harness reports capture/reconstruction/write/cleanup service sums, worker elapsed time, thread-teardown-inclusive wall time, CPU cycles/coarse CPU time, and peak working set. Setup, filename reservation, thread creation, output verification, and deletion are outside the timed intervals. Filesystem caching, security scanning, scheduling, CPU frequency, and memory bandwidth still affect results; this is not an isolated full-system
-systemscheduler benchmark. No protection was disabled. Further speculative traversal/scheduling rewrites were not made without evidence of benefit and preserved coverage.Every serialized benchmark job retained its baseline size and CRC32, normalizing only the generated PE64 fixture's ASLR-dependent ImageBase. Database queries independently assert the expected hit count; export discovery has no serialized fingerprint and is covered by native semantic tests plus live comparisons.
Final fidelity check after optimization
The final executable again paired 72 live images in each import mode with
868369e, with 71 / 67 matching-layout witness pairs. Aggregate output sizes were unchanged: 1,272,950,784 bytes without imports; 1,300,660,224 with imports. There were no new dump failures, structural failures, lost code blocks, original-IAT mismatches, missing original imports, changed resource payloads, unexplained.textbytes, or stable unexplained memory-witness differences. Matching subsets compared 673,570,956 / 663,288,680.textbytes and 2,816 / 2,710 resource payloads. Unavailable/changed processes, missing originals, safety-limit exclusions, and known security-software exclusions remain explicitly reported.Added ten native cases for moved/borrowed import equivalence, export ownership/alias precedence, cached-snapshot reuse/invalidation/concurrency/lifetime, file-based database updates, PE32/PE64 weak/strong entrypoint selection, and borrowed process-handle identity/lifetime. Added portable performance-parser/comparator cases. That revision's suite was 92 native + 18 Python tests.
Local validation before the final security pass
Final native and GitHub Actions security pass (
e70c0cc)Two additional source-supported vulnerabilities were identified and fixed. Locations below refer to the pre-fix code. No exploit PoCs were created or executed.
pd/dump_output.h.github/workflows/deploy-create-releases.yml(also both other publishers)Native output-path race
Holding directories without delete sharing prevents renaming/replacement but does not prevent an in-place reparse metadata change. The earlier final full-path
CreateFileAcould follow an intermediate junction introduced after its validation. This requires an attacker-writable, initially empty directory, an elevated dumper, and a successful race; existing files still could not be overwritten. The defect also exists in merged4763c7e.Parent traversal and final creation now use root-relative
NtCreateFileoperations withOBJ_DONT_REPARSE; the verified root and parent handles remain open through writing. Final creation retainsFILE_CREATE, exclusive sharing, synchronous writes, and handle-based incomplete-file cleanup. Native failures propagate throughRtlNtStatusToDosError, with no weaker retry. Addedntdll.libconsistently for application and test builds.Three new native cases cover ordinary nested output, invalid native names/roots, and mocked reparse/access/parameter failures. The mock asserts the exact root-relative name, creation/directory flags, sharing, single-call behavior, and error propagation. Existing file-collision, raw-dump, content, and serialization cases remain unchanged. No live junction exploit was used, and UNC/network-share integration was not exercised.
Privileged artifact boundary
Review of the resolved
actions/download-artifact@v4implementation (d3f86a106a0bac45b974a628896c90dbdf5c8093) found automatic extraction using a filename guard that strips leading traversal but lacks canonical destination containment: guard, write path. A hostile build producer controls its uploaded ZIP, not merely the path specified in the trusted upload step. Selecting that source for publication authorizes binary bytes, not execution inside the publisher holdingcontents: write. A fork PR alone does not enter the publication path.All three publishers now check out only trusted preparation code from
github.workflow_sha, not the manually selected build SHA. The helper lists artifacts from the current repository/run, requires both unique unexpired architecture artifacts, and accepts exactly one regularpd.exeentry per archive. It validates metadata, CRC, and 64 MiB archive/uncompressed limits, then copies data into fixed exclusively created output names. It never extracts archive-controlled paths or runs downloaded binaries. Signed HTTPS blob redirects receive no repository token; additional redirects and API errors fail closed.Added ten portable cases for same-run pagination/name selection, duplicate/missing/expired/oversized artifacts, entry metadata, benign compressed/stored roundtrips, overwrite refusal, invalid archives, bounded reads, tag identity, HTTP failures, redirect credential isolation, and end-to-end two-binary manifest preparation. Tests use benign data and mocks, not hostile archive payloads.
Additional release hardening and review conclusions
contents: writeand theactions: readneeded by the REST artifact reader. Nopull_request_target, privilegedworkflow_runconsumer, inherited build secrets, cross-run artifact selection, or cache-consumption path was found.Developupdates without cancelling an upload midway. Automatic main builds check freshness before publication; manual development dispatch remains intentional.provenance.jsoncontaining the selected source commit, run/artifact IDs, sizes, and SHA-256 hashes. This is an asset-integrity record, not source-to-binary attestation.Developretains its historical unmoved tag; its notes and manifest identify the current asset commit. GitHub multi-asset uploads are not transactional: interrupted uploads can leave a partial set, so both hashes must match the manifest.Final security validation
actionlint1.7.736980358932, with independently verified manifest hashes; neither executable was run, and downloaded binaries were removedThe publisher's preparation boundary was tested locally; no release was created, overwritten, or manually dispatched for validation. CI build matrices run on the new PR revision. The review is not a guarantee that no further vulnerabilities exist.
Reconstructed application execution (
b30b441)Added default-off
-reexec, guided by failures from newly launched, debugger-owned benign children. Ordinary forensic dumping remains unchanged; preparation does not execute an image and does not restore a process checkpoint.0xc000007b. Restoring guard-function slots did not help; restoring the compiler's cookie bootstrap sentinel did. Zero was not an equivalent replacement. Validate load-config version/ranges and writable pointer-sized storage, then serialize the PE32/PE64 sentinel without modifying captured source bytes or weakening mitigations.Controlled results
The opt-in Python runner observes the original application, an untouched staged disk-copy control and the reconstructed image using the same architecture-matched native debugger probe. It restores its entrypoint breakpoint before capture, uses kill-on-close jobs for its own children, and records loader/entrypoint/GUI/fixture milestones, exceptions, hashes and exit codes. Entry/fixture-marker captures suspend the child through a debug event; GUI-idle capture is live. Dump time is excluded. Optional explicitly selected MUI resources are staged beside both copies; no original executable code is substituted into the dump. Failed or changed controls are inconclusive, and absent output hashes never count as equality.
Two trials on Windows 11 build 22631, six application images, both import modes and relevant capture phases produced 40 matched baseline/prepared pairs against
e70c0cc: 34 gained milestones, six were unchanged, zero lost milestones. Every candidate reached its entrypoint.sort.exe, entrypointfindstr.exe /?, entrypoint with matching MUI0xc0000409, parameter 5Both original and staged-copy output hashes matched all candidate runs; empty GUI output is not treated as functional proof. Input-idle is startup progress, not full editor functionality. The remaining initialized Notepad failure is documented, not hidden by resetting arbitrary globals or disabling ASLR/DEP/CFG/delay-IAT protections. Heaps, handles, TLS, initialized file-backed globals and external state remain outside this mode's restoration scope.
The execution harness is not a security sandbox. It requires explicit acknowledgement, must only run trusted benign bounded workloads, and never runs automatically in CI or the normal unit suite. Reports and diagnostic payloads remain private; benchmark captures are cleaned up and the six retained investigation executables were removed. No exploit PoCs or security-software bypasses were used.
Reexecution validation