test: fix flaky bulk-job CI by raising poll timeout to 300s#19
Draft
brendancsmith wants to merge 3 commits into
Draft
test: fix flaky bulk-job CI by raising poll timeout to 300s#19brendancsmith wants to merge 3 commits into
brendancsmith wants to merge 3 commits into
Conversation
The enhance bulk-job functional tests hit the live Diffbot API (cassettes are gitignored, so record_mode="new_episodes" records fresh every run). A 60s poll ceiling in test_bulkjob_status and test_bulkjob_coverage_report is often too short for Diffbot to finish the job, causing a timeout that cascades into every downstream bulk-job assertion. Raise both poll timeouts to 300s to give the job room to complete.
Explain in the cassettes .gitignore that the functional suite runs against the live Diffbot API by design, so cassettes must not be committed to force deterministic CI. Prevents a future contributor from "fixing" bulk-job flakiness by pinning recordings instead of the poll timeout.
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
| Duplication | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
The bulkjob_coverage_report poll only checked TIMEOUT after a non-error response, so while the report was still generating (repeated 400s) the loop could run past the 300s ceiling until the CI job itself timed out. A non-400 ClientResponseError was also swallowed, spinning the loop with no sleep and no visible failure. Check the elapsed time at the top of every iteration so the timeout applies uniformly, and re-raise any non-400 error instead of retrying. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GFjHh9GV3WyGspDxB2rbLc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The enhance bulk-job functional tests (
tests/functional/clients/test_enhance_client.py) fail intermittently in CI —test_bulkjob_statusreports "Bulk job status check did not complete in time", which cascades into every downstream bulk-job assertion (test_bulkjob_results,test_single_bulkjob_result,test_bulkjob_coverage_report,test_bulkjob_stop).Root cause: these tests hit the live Diffbot API by design — VCR cassettes are gitignored and
record_mode="new_episodes"records fresh every run. The 60s poll ceiling intest_bulkjob_statusandtest_bulkjob_coverage_reportis often too short for Diffbot to finish the job, so the poll times out. This is why a matrix leg can pass on 3.12 while failing on 3.11.Change
cassettes/.gitignorethat live functional tests are intentional and cassettes must not be committed to force deterministic CI.Verification
CI on this PR runs the functional suite against the live API — the check here is the verification that the 300s timeout resolves the flakiness.