Skip to content

fix(ci): stop flaky sleep test and compat evaluation errors - #677

Open
jariy17 wants to merge 1 commit into
mainfrom
fix/ci-sleep-mock-isolation
Open

jariy17 wants to merge 1 commit into
mainfrom
fix/ci-sleep-mock-isolation

Conversation

@jariy17

@jariy17 jariy17 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Fixes two CI failures seen on #674 and #675. Neither was caused by those PRs.

1. Flaky test_zero_delay_skips_sleep

Fails with Expected 'sleep' to not have been called. Called 898 times.

Why:

  • Importing ragas starts a background thread that calls time.sleep(1) in a loop. RAGAS_DO_NOT_TRACK doesn't stop it.
  • Our tests patch paths like middleware.time.sleep. That patches the shared time module, so every thread sees the mock.
  • If the ragas thread wakes while a test holds the patch, the mock returns instantly, the loop spins, and the mock records hundreds of calls.

Fix: tests/conftest.py stops the ragas thread after the first test that imports ragas.

2. Compat (evaluation) errors

test_third_party_adapters.py needs OpenAI and Braintrust API keys. The compat job doesn't have them, so its 16 tests always error. This PR skips them in that job; the secure integ workflow still runs them.

Follow-up: add skip guards to the test file itself so this ignore can be removed.

Testing

  • Unit tests: 3565 passed. 4 memorystore typing tests fail locally with or without this change.
  • The ragas thread is gone at the end of the run with the fix, and still running without it.
  • Compat evaluation collects 37 tests instead of 53. The 16 dropped are the ones that errored.

@github-actions github-actions Bot added the size/s PR size: S label Sep 23, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 23, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 23, 2026
- Stop ragas' background telemetry thread in tests. It calls
  time.sleep(1) in a loop, and patching time.sleep in a test patches it
  for every thread, so the mock picked up hundreds of stray calls.
- Skip test_third_party_adapters.py in the compat job. It needs OpenAI
  and Braintrust API keys that the job doesn't have.
@jariy17
jariy17 force-pushed the fix/ci-sleep-mock-isolation branch from 94188e6 to a7a7925 Compare September 23, 2026 18:49
@jariy17 jariy17 changed the title fix(ci): isolate patched time.sleep and skip keyless compat tests fix(ci): stop flaky sleep test and compat evaluation errors Sep 23, 2026
@jariy17
jariy17 marked this pull request as ready for review September 23, 2026 18:55
@jariy17
jariy17 requested a review from a team September 23, 2026 18:55
@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Sep 23, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 23, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants