Repository navigation
feat(mcp): add a Fusion search backfill tool and correct the search tool descriptions - #1396
Conversation
|
Note 📝 PR Converted to Draft More info...Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy. As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page. To skip draft status in future PRs, please include |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@johnny/mcp-fusion-backfill-tool' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@johnny/mcp-fusion-backfill-tool'PR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful ResourcesCommunity SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
Code coverage overviewLanguages: Python Python / code-coverage/pytest-no-creds
View coverage of the 10 most impacted files.
Python / code-coverage/pytest
View coverage of the 10 most impacted files.
Python / code-coverage/pytest-unit
Line coverage data for the release-candidate/v1 branch is not yet available. View coverage of the 10 most covered files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
Aaron ("AJ") Steers (aaronsteers)
left a comment
Aaron ("AJ") Steers (aaronsteers)
left a comment
There was a problem hiding this comment.
Approved with some comments. Nothing blocking.
…-world, and add it to the trace fixture
c041983
into
release-candidate/v1
2 participants
Summary
Adds
request_cloud_search_index_backfill, an MCP tool that asks Airbyte Cloud to backfill a connection's search index. It also fixes the search tool descriptions that no longer match Sonar.Turning search on for a connection indexes only data from syncs that run afterwards. Data already in the destination is not searchable until someone requests a backfill, and until now no tool could do that.
New API
CloudConnection.request_search_index_backfill(streams=None, backfill_start_time=None)callsPOST {config_api_root}/connections/{connectionId}/backfill, which isPOST /api/v1/connections/{connectionId}/backfillin Cloud. Cloud forwards it to Sonar's/api/cloud/v1/connections/{connection_id}/backfill.{"streams": [...]?, "start_time": "<RFC 3339 with offset>"?}. Cloud rejects unknown fields and an emptystreamslist with a 422, and returns a 409 when search indexing is not enabled for the connection.{"streams": [{"stream_name", "outcome": "requested"|"running"|"failed", "run_id", "backfill": {...}}]}, parsed intoExternalSearchBackfillRequestResult.backfill_start_timeis rejected before any request, because Cloud requires an offset.request_cloud_search_index_backfill(connection_id, streams?, backfill_start_time?). It is a write tool: not read-only, not idempotent, not destructive. It is closed-world, like the sync tools, because it starts index work rather than returning external data. Like the other Fusion tools it setsexternal_access=True. Likerun_cloud_syncit setspipeline_change=False, because it does not change the connection's configuration.What the tool description says, and where each claim comes from (Sonar
main)FusionBackfillRequesterinbackfill.py).streamsqueues one run that covers every stream selected in the connection. The response shows it as a single entry with an emptystream_name(ALL_STREAMS,FusionBackfillRequested).backfill_start_timekeeps only rows whose cursor field (a top-level timestamp or date) is at or after it. Streams without such a cursor are backfilled in full. Omitting it backfills the whole history (sonar#6954).running, so a retry is safe.FusionBackfillController._descriptor). Snowflake OAuth and PAT are in open sonar#6959, so the description does not mention them.get_cloud_search_status. An empty table completes with 0 records (sonar#6944).Description and model changes on existing tools
ExternalSearchHit.entity_data: null values are not stored; a keyword-only hit leaves out fields longer than 4096 characters or split across chunksExternalSearchHit.contextandmax_context_chars: the keyword context is centered on the first query term, with…where text was cutexecute_external_search_query: says where dropped fields went and points to the direct-connector and SQL tools for their full valuesExternalSearchBackfillStatus: addsfailed_atandcancelled_at, lists every status (starting,running,finalizing,complete,failed,cancelled), and documents whatbackfill_start_timemeansget_cloud_search_status: explains whenbackfillis null and how it moves through its states, and points to the backfill toolNot changed: Cloud drops these fields today
Cloud's
FusionSearchControllerkeeps only the search-status fields listed in its OpenAPI spec. Several new Sonar fields therefore never reach PyAirbyte, and these descriptions do not claim them yet:error_codeanderror(nothing_to_backfill,destination_unsupported,system_error, ...), from sonar#6944. Cloud shows onlystatus: failedandfailed_at. Follow-up: add these fields to the platform's search-status schema, then name the codes in the status tool.held_recordson index entries, from sonar#6937. Same fix needed in the platform schema.error_codeand specific details, from sonar#6926. These reach PyAirbyte once airbyte-platform-internal#19962 merges.gaps,publication_held,stats_errorandsegments, and each stream's and source'sgaps. Cloud's status schema does not list them either.All of these depend on Cloud passing the fields through. Once it does, the status model and tool description can document them.
Follow-ups
needs_backfillflag to search status: per connection, plus a per-destination list. When it merges and Cloud forwards it, update the status tool description to tell agents to request a backfill when the flag is set."") from search status. Until it merges, status can list a stream with an empty name after an all-streams request.start_timestill says "rows extracted at or after this time". It should be updated to the cursor-field meaning from sonar#6954.Test plan
uv run pytest tests/unit_tests/test_cloud_direct_execute.py tests/unit_tests/test_mcp_cloud.py:streamsandstart_timenames, offset kept,{}when no argumentsrunningfailed_atparsedtests/unit_tests/fixtures/arg_trace_plan.json) includes the new toolruff format --check .,ruff check .,pyrefly checkgit merge-tree).