Repository navigation
Fix query deduplication and domain scope; reduce fingerprint memory - #213
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding Request.params previously collapsed unrelated repeated query keys: ?tag=a&tag=b plus page=2 became ?tag=b&page=2. This changed the outgoing request and gave it the same fingerprint as a different URL. Preserve repeated/blank values while replacing only explicitly overridden keys.
Also fix allowed_domains normalization for bare/bracketed IPv6, IPv6 URLs with ports, and DNS names with a trailing dot. Add component and engine regression tests and document the accepted forms.
Hash C-contiguous memoryviews backed by immutable bytes directly instead of copying their payload. Keep snapshots for mutable backing buffers (including read-only views over bytearray) and conversion for non-contiguous views. Public signatures and dependencies remain unchanged.
Validation on CPython 3.14.7:
Fingerprint measurement: seven fresh processes per version and case, alternating order; warm-up, GC disabled for timing, tracemalloc measured separately after payload construction. For an immutable 8 MiB memoryview, median time was 4.985 ms before versus 4.528 ms after (about 9% lower); process ranges were 4.928–5.112 ms versus 4.482–4.598 ms. Peak additional traced Python memory fell from 8,388,732 to 794 bytes. Small-payload timing ranges overlap. These are shared-environment fingerprint microbenchmarks, not whole-crawl throughput or native allocation measurements.
Fingerprints intentionally change for the previously mishandled repeated-key requests; ordinary buffer fingerprints retain their digest.