Add adaptive extraction, typed anti-bot mitigation, and identity profiles - #1
Open
satvik007 wants to merge 19 commits into
Open
Add adaptive extraction, typed anti-bot mitigation, and identity profiles#1satvik007 wants to merge 19 commits into
satvik007 wants to merge 19 commits into
Conversation
… control, engine filter, compare helper (Track D / S4) - conformance scenario (new): adversarial-HTML HtmlProcessingMode parity gate. Corpus covers adoption-agency misnesting, foster parenting, template/noscript, RAWTEXT/RCDATA/plaintext apparent anchors, multiple/invalid <base>, entity-encoded hrefs (incl. legacy no-semicolon and an entity-encoded colon), fragments, traversals, SVG/MathML/foreignObject, an XHTML-content-type page, and a non-UTF-8 page. Ground truth is a simulated DOM-path crawl (same scraper version, mirrored resolve/strategy/unique-key semantics); trials must reproduce the discovered-URL/request-ID digest (server-relative, port-independent) plus the server hit-set/off-host gates. 12 unit tests pin per-construct DOM behavior; end-to-end smoke validates against a real millipede crawl. - millipede-adaptive engine variant: explicit Adaptive row for the gate; bails with a clear error until Track A lands (TODO(track-a) markers carry the exact one-line opt-ins for the engine and the headline row). - phase timing (opt-in via SPIDER_BENCH_PHASES): handler/accounting/ extract/enqueue wall-time sums per millipede trial, emitted into samples.jsonl and a residual-CPU decomposition section in summary.md; absent otherwise. - compressed-identity scenario: compressed corpus served identity-encoded (identical bodies/checksums/gates) to isolate gzip CPU (B2 control). - orchestrate --engines filter (composes with --scenario/--quick; skips Go/npm preparation when external engines are filtered out). - compare subcommand: before/after per-(scenario, C, engine) medians with deltas over two samples.jsonl files. - server: *.xhtml paths served as application/xhtml+xml (pre-rendered bodies cannot carry per-page metadata). - docs: disclosure + diagnostics section in benchmarks-vs-crawlers.md; README run/diagnostics updates; rustfmt normalization of the package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add HtmlProcessingMode { EagerDom (default), Adaptive } to HtmlKind.
EagerDom is bit-identical to previous releases, including eager parse
timing in execute(). Adaptive defers the scraper DOM behind
SynchronizedHtml's new lazy representation (bytes::Bytes source +
OnceLock<Mutex<scraper::Html>>; public API unchanged, parse-exactly-once
under concurrent first touch) and dispatches default-selector link
discovery per extract() call:
- EagerDom, custom selectors, XHTML content types, non-UTF-8 bodies,
and already-materialized DOMs take the DOM path (previous code,
verbatim; DOM-reuse keeps books/hn-style DOM-first handlers on
ADR-0005's d2 path with zero streaming passes);
- everything else runs a synchronous lol_html pass (new stream.rs)
that never crosses an .await and never fails a request — any
rewriter error falls back to the DOM path.
Parity contract (hard gate): streaming produces the DOM path's resolved
URL set, unique keys, and RequestIds, or statically routes to DOM.
Enforced by htmlize::unescape_attribute entity decoding (attribute-mode
legacy rules, only when the raw value contains '&') plus U+0000 -> U+FFFD
tokenizer parity; first-base[href]/invalid-base semantics mirroring
extract.rs; RequiresDom aborts on <select>/<frameset> subtrees (html5ever
drops their illegal children); pinned-construct fixtures (template,
noscript/raw-text containers, SVG/MathML/foreignObject/CDATA, adoption
agency, foster parenting, duplicate attributes); two differential
proptests (256 cases each); and frontier-determinism tests across four
handler shapes in both modes. Adaptive parse time is attributed to
request_handler_timeout — documented and tested.
Deps: lol_html =2.8.1 moves dev->prod in millipede-html; htmlize 1.1
(MIT OR Apache-2.0, unescape feature) and bytes added; cargo deny and
cargo +1.85 check clean. Public-API baselines regenerated for millipede
and millipede-html (additive only; bench-only internals are doc(hidden)
and excluded).
Benches (link_extraction, new adaptive_extraction groups at
4/32/256 KiB): streaming_cold beats dom_cold 3.7-4.2x; dom-first
dispatch within noise of dom_warm; selector_bench d2 within +/-2.4% of
the ADR-0005 baseline. Full gate results recorded in ADR-0009.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…/ S2) Check the enqueue strategy before serializing candidate URLs, thread a single owned URL string through the filtered-candidates/seen/request_urls checks, and skip the post-transform unique-key recompute when no transform is configured. Adds a fixture-matrix test that diffs the optimized pipeline against a test-only pre-optimization reference across strategies, filters, limits, transforms, and duplicate paths, plus a criterion benchmark (enqueue_pipeline) to measure the allocation-diet win. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate success-path snapshot cloning on subscriber presence (failure path and failed_handler untouched); move Arc<Request> through the pipeline instead of cloning, recovering ownership via Arc::unwrap_or_clone; slim LeasedIdentity while preserving mark_handled's stored-identity semantics; merge dedup/handled tracking into a single HashMap<String, DedupEntry> in millipede-storage-memory; keep reclaim/abandon/renew re-queueing lease.request unchanged; cache the no-proxy HTTP client to skip the per-hop mutex and mem::take response headers on both buffered and streaming paths. Adds engine_overhead/queue_ops bench coverage and a new client_overhead bench (no-proxy, redirect chain, buffered vs streaming, large headers). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Post-merge integration: activate Track A's HtmlProcessingMode::Adaptive in the spider-bench millipede-adaptive engine, run the conformance scenario as an EagerDom/Adaptive pair, and pin spider-bench's lockfile to lol_html 2.8.1 to match millipede-html's new production pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings D-1..D-4: fold stored RequestIds and per-page strategy-skip counts into the conformance frontier digest so ID-derivation or off-host divergence between EagerDom and Adaptive fails the trial; refuse --engines filters that silently degrade the conformance pair to single mode (explicit single-engine runs stay possible, loudly disclaimed); rework the phase table as an honest cumulative handler wall-time decomposition instead of an unsound residual-CPU subtraction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing divergence Review finding A-1 (critical) investigated empirically and refuted: scraper::Html::select iterates the ego_tree arena in node-creation order (token order), not document order, so html5ever's table foster parenting never reorders extraction relative to the streaming pass. Instead of a routing change, harden the contract: differential tests and proptests now compare ordered deduplicated frontier sequences (mirroring enqueue's dedupe-before-limit), with regression fixtures for fostered base/anchor, .limit() truncation prefixes, adoption-agency clones, and an adversarial table battery — these fail loudly if a scraper upgrade ever switches to document-order traversal. ADR-0009 restates the ordered contract and the formatting-clone tolerance. Also gate the handler-timeout overrun test to debug profile where its 200 ms parse-budget assumption holds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dual-mode conformance gate passed 5/5 trials with identical frontier digests, satisfying the plan's flip criterion. Headline Engine::Millipede rows now opt into HtmlProcessingMode::Adaptive (library default remains EagerDom); conformance trials keep this arm on EagerDom so the gate always compares the two modes. Disclosure updated in docs/benchmarks-vs-crawlers.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
navigator.platformvaluesReview fixes
BlockFailuremetadataVerification
cargo fmt --all -- --checkgit diff --checkcargo test --workspacecargo test --workspace --all-targetsalso passed all library, integration, real-browser, and benchmark targets until the existingmillipede-httpclient_overheadbenchmark oversized-header case exceeded the HTTP message-head limit.