Repository navigation
Sync Cohere client blocks async event loop on every LLM call #1
Description
What
The CohereClient class at src/services/cohere_client.py:23 instantiates cohere.ClientV2 (the synchronous Cohere SDK client). All methods — generate(), generate_json(), and embed() — are declared async and awaited by callers, but the underlying .chat() and .embed() calls execute synchronously on the main asyncio event loop thread.
Why
Every LLM/embedding call freezes the entire FastAPI server for the duration of the Cohere API request (typically 1-10 seconds). Under concurrent requests, the server becomes effectively single-threaded — all other requests queue behind the blocking call. This makes the service unable to handle concurrent users and defeats the purpose of using FastAPI/async.
Scope
In scope:
- Replace
cohere.ClientV2withcohere.AsyncClientV2in the Cohere client - Update all method calls to use
awaiton the async client methods - Verify the async client has the same API surface
Out of scope:
- Changing the prompt structure or LLM parameters
- Adding caching or rate limiting
- Changing the FastAPI application structure
Acceptance Criteria
-
CohereClientusescohere.AsyncClientV2instead ofcohere.ClientV2 - All
generate(),generate_json(), andembed()methods are truly async - The FastAPI server can handle concurrent LLM requests without blocking
-
ruff check src/ tests/passes -
pytestpasses
Technical Context
File to modify:
src/services/cohere_client.py— classCohereClient(line 23)
Current code:
self._client = cohere.ClientV2(api_key=self._api_key)Fix:
self._client = cohere.AsyncClientV2(api_key=self._api_key)The AsyncClientV2 has the same method signatures but returns awaitable coroutines. All callers already await the results, so no caller changes should be needed.
Callers:
src/services/generation.py— course generationsrc/services/quiz_generator.py— quiz generationsrc/services/feedback_engine.py— feedback generationsrc/services/embedding.py— embedding generationscripts/build_index.py— knowledge base index builder
Activity
Aycode01 commentedon Jul 20, 2026
can i work on this
Adamsonoladipupo commentedon Jul 20, 2026
Can I work on this please?
Okorie2000-code commentedon Jul 20, 2026
Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.
Okorie2000-code commentedon Jul 20, 2026
Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.
Okorie2000-code commentedon Jul 20, 2026
Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.
Okorie2000-code commentedon Jul 20, 2026
Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.
Okorie2000-code commentedon Jul 20, 2026
Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.
Okorie2000-code commentedon Jul 20, 2026
Hi maintainer,
would love to work on this issue, i have already reviewed the description and will submit a clean PR.
kindly assign.
Joeloo1 commentedon Jul 22, 2026
Hi maintainer,
I'd like to work on this issue.
My approach is to replace
cohere.ClientV2withcohere.AsyncClientV2, update the client methods to useawaitfor all Cohere API calls, and verify that the async API remains compatible with the existing implementation. I'll also validate that concurrent requests no longer block the FastAPI event loop, ensure all affected callers continue to work correctly, and runruff checkandpytestbefore submitting the PR.I'd appreciate it if this issue could be assigned to me. Thanks!
grantfox-oss commentedon Jul 22, 2026
🦊 GrantFox — @Joeloo1 has been assigned to this issue as part of the Official Campaign | FWC26 campaign!
Next steps:
Closes #1)Good luck! Track your progress on GrantFox.
fix: resolve pre-existing lint errors to pass CI
grantfox-oss commentedon Jul 24, 2026
🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign | FWC26 campaign!
@Joeloo1's PR #15 was approved and merged by @DeFiVC.
🏆 @Joeloo1: You earned 40 FoxPoints for this contribution! Your current tier: Builder (1,212 total points). Track your full progress on GrantFox.
👏 Great work, @Joeloo1! Keep contributing to ChainLearnOfficial.