Skip to content

Fix query deduplication and domain scope; reduce fingerprint memory - #213

Merged
egorsmkv merged 1 commit into
mainfrom
fix/python-audit-url-fingerprints
Oct 8, 2026
Merged

egorsmkv merged 1 commit into
mainfrom
fix/python-audit-url-fingerprints

Conversation

@egorsmkv

@egorsmkv egorsmkv commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Adding Request.params previously collapsed unrelated repeated query keys: ?tag=a&tag=b plus page=2 became ?tag=b&page=2. This changed the outgoing request and gave it the same fingerprint as a different URL. Preserve repeated/blank values while replacing only explicitly overridden keys.

Also fix allowed_domains normalization for bare/bracketed IPv6, IPv6 URLs with ports, and DNS names with a trailing dot. Add component and engine regression tests and document the accepted forms.

Hash C-contiguous memoryviews backed by immutable bytes directly instead of copying their payload. Keep snapshots for mutable backing buffers (including read-only views over bytearray) and conversion for non-contiguous views. Public signatures and dependencies remain unchanged.

Validation on CPython 3.14.7:

  • Full pytest suite: 380 passed, 73 skipped. Includes available local HTTP/parser integration tests. The first full run hit a full filesystem; the successful rerun used --basetemp=/dev/shm/silkworm-audit-pytest-3ee32dcc8281.
  • Ruff formatting over src/examples/tests and Ruff lint over src/examples plus changed tests pass.
  • Full Pyright: 0 errors, 0 warnings.
  • All source files parse with Python 3.11 grammar; other interpreter runtimes were not tested locally.
  • Before the fix, 10 new URL regression cases failed; they now pass.

Fingerprint measurement: seven fresh processes per version and case, alternating order; warm-up, GC disabled for timing, tracemalloc measured separately after payload construction. For an immutable 8 MiB memoryview, median time was 4.985 ms before versus 4.528 ms after (about 9% lower); process ranges were 4.928–5.112 ms versus 4.482–4.598 ms. Peak additional traced Python memory fell from 8,388,732 to 794 bytes. Small-payload timing ranges overlap. These are shared-environment fingerprint microbenchmarks, not whole-crawl throughput or native allocation measurements.

Fingerprints intentionally change for the previously mishandled repeated-key requests; ordinary buffer fingerprints retain their digest.

@egorsmkv
egorsmkv merged commit fc987bf into main Oct 8, 2026
23 checks passed
@egorsmkv
egorsmkv deleted the fix/python-audit-url-fingerprints branch October 8, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant