You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HttpComponentsResolver drops partition router request options and cannot use AsyncRetriever #1181
HttpComponentsResolver builds its retriever with the pre-#707 kwarg name, which causes two problems on current main (d8d8e6f):
Partition router request options are silently dropped.create_http_components_resolver passes only stream_slicer= to the retriever factory. Since chore: remove declarative stream #707, create_simple_retriever takes the partition router as partition_router (used as the request_options_provider), and swallows stream_slicer into **kwargs. So a request_option declared on the resolver retriever's partition router is never sent. Before chore: remove declarative stream #707 the same router was used as the request options provider, so this is a regression.
AsyncRetriever cannot be used as the resolver retriever, although the schema allows it. Discovery fails with KeyError: 'requester' at concurrent_declarative_source.py:894, which forces use_cache onto retriever.requester. Past that line, the resolver iterates its own partition router instead of the AsyncRetriever's job slicer, and would treat the slice log message as component values.
Reproduction (problem 1)
A DynamicDeclarativeStream whose HttpComponentsResolver.retriever is a SimpleRetriever with a ListPartitionRouter (values: ["p1"], request_option: {inject_into: request_parameter, field_name: p}). On main the resolver request goes to https://api.test.com/list without ?p=p1. Passing partition_router= next to stream_slicer= (the pattern create_default_stream uses) sends ?p=p1.
Impact
Problem 1 affects only resolvers whose retriever partition router declares a request_option. No monorepo connector does (source-airtable, source-google-sheets and source-hubspot use HttpComponentsResolver without one), but Builder-only manifests cannot be checked. Fixing it changes the resolver's HTTP requests for those manifests, so it should ship as its own change with a release note.
Problem 2 blocks async APIs from resolving dynamic streams.
Precedent
Found while fixing #766 (the same kwarg mismatch in create_dynamic_schema_loader), in PR #974. That PR intentionally leaves the resolver unchanged.
Activity