Skip to content

HttpComponentsResolver drops partition router request options and cannot use AsyncRetriever #1181

Closed

Description

@bazarnov

Summary

HttpComponentsResolver builds its retriever with the pre-#707 kwarg name, which causes two problems on current main (d8d8e6f):

  1. Partition router request options are silently dropped. create_http_components_resolver passes only stream_slicer= to the retriever factory. Since chore: remove declarative stream #707, create_simple_retriever takes the partition router as partition_router (used as the request_options_provider), and swallows stream_slicer into **kwargs. So a request_option declared on the resolver retriever's partition router is never sent. Before chore: remove declarative stream #707 the same router was used as the request options provider, so this is a regression.
  2. AsyncRetriever cannot be used as the resolver retriever, although the schema allows it. Discovery fails with KeyError: 'requester' at concurrent_declarative_source.py:894, which forces use_cache onto retriever.requester. Past that line, the resolver iterates its own partition router instead of the AsyncRetriever's job slicer, and would treat the slice log message as component values.

Reproduction (problem 1)

A DynamicDeclarativeStream whose HttpComponentsResolver.retriever is a SimpleRetriever with a ListPartitionRouter (values: ["p1"], request_option: {inject_into: request_parameter, field_name: p}). On main the resolver request goes to https://api.test.com/list without ?p=p1. Passing partition_router= next to stream_slicer= (the pattern create_default_stream uses) sends ?p=p1.

Impact

  • Problem 1 affects only resolvers whose retriever partition router declares a request_option. No monorepo connector does (source-airtable, source-google-sheets and source-hubspot use HttpComponentsResolver without one), but Builder-only manifests cannot be checked. Fixing it changes the resolver's HTTP requests for those manifests, so it should ship as its own change with a release note.
  • Problem 2 blocks async APIs from resolving dynamic streams.

Precedent

Found while fixing #766 (the same kwarg mismatch in create_dynamic_schema_loader), in PR #974. That PR intentionally leaves the resolver unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    communityPRs and issues from community contributors

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions