[AkhileshNegi]

Issue

Closes #1019

Summary

Fast evaluations and high-priority LLM jobs shared one Celery default queue and one worker pool, with no cap on concurrent fast-eval chunk tasks. A 500-item run fans out to 10 chunk tasks (chunk size 50), which can occupy every worker slot — x-max-priority only reorders queued-not-yet-claimed messages, it can't preempt a task a worker already started, so a high-priority LLM job (priority 9) waits behind eval chunks (priority 6) until a slot frees.

This adds a second RabbitMQ queue (evaluations) for the actual burst source — run_evaluation_fast_chunk + run_evaluation_fast_aggregate — consumed by a dedicated worker pool, so the isolation is physical instead of priority-advisory:

• backend/app/celery/celery_app.py: DEFAULT_QUEUE/EVALUATIONS_QUEUE constants, second Queue(...) entry (x-max-priority=10, distinct routing key).
• backend/app/celery/tasks/job_execution.py: only the two fast-eval tasks move to queue=EVALUATIONS_QUEUE; every other task stays on default. Module docstring rewritten for the two-queue topology.
• docker-compose.yml / .dev.yml / .staging.yml: existing celery_worker pinned to -Q default, new celery_worker_eval service added with -Q evaluations --concurrency=${CELERY_EVAL_WORKER_CONCURRENCY:-2}.
• .env.example: new CELERY_EVAL_WORKER_CONCURRENCY knob.
• .claude/conventions/celery.md, docs/wiki/services.md: docs updated for the two-queue setup.

Note: a Celery worker with no -Q flag auto-subscribes to every declared queue, so declaring the queue alone (without a worker pinned to it) achieves nothing — this is why an earlier attempt at multiple queues (#964) was reverted, and why this PR pairs the queue with dedicated compose workers.

Production (ECS) still needs a manual companion change, not covered by this PR — ECS task defs aren't managed in this repo. The existing ECS worker(s) currently consume default with no -Q flag; until they're repointed (-Q default on the existing task def(s), plus a new task def running -Q evaluations), a queue-less worker there will silently also drain evaluations, and the split won't take effect in prod. See PR description history / ask reviewer for the ECS runbook if not tracked elsewhere yet.

Checklist

• Ran docker compose up locally and verified queue isolation (celery -A app.celery.celery_app inspect active_queues on each worker container reports exactly one, distinct queue name).
• No new test needed — Celery/broker wiring, not new branching logic in application code.

Notes

Reviewer: production rollout needs the ECS task-def change (see Summary) applied in the same deploy window, or the split has no effect in prod. No GitHub issue tracks that yet — happy to file one if wanted.

[github-actions]

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ 00fb3876 · generated by oasdiff}

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Source mode: Auto
• Repository: ProjectTech4DevAI/kaapi-backend
• Branch: enhancement/evaluation-worker
• File: .coderabbit.yaml
• Commit: da5d34a
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: 26d30f73-9533-41ca-93ce-6d07a1e5aaf5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

🎉 This PR is included in version 1.9.0-main.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀