Skip to content

feat(queue): priority lists and * wildcard for queue:work - #15

Merged
3m1n3nc3 merged 2 commits into
mainfrom
feat/queue-priority-wildcard
Oct 20, 2026
Merged

3m1n3nc3 merged 2 commits into
mainfrom
feat/queue-priority-wildcard

Conversation

@3m1n3nc3

@3m1n3nc3 3m1n3nc3 commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Summary

ark queue:work can only work one queue, and a bare worker silently ignores jobs dispatched to any other queue. This caught Arkyc in production: its single PM2 worker drained only default while every job went to a named queue, so sessions stalled in processing and no webhook was ever delivered.

--queue now accepts:

ark queue:work                      # unchanged: the connection's default queue only
ark queue:work --queue=webhook,ocr  # priority list: ocr runs only while webhook is empty
ark queue:work --queue=*            # every queue on the connection
ark queue:work --queue=webhook,*    # webhook first, then anything

A bare queue:work is deliberately unchanged. Making it work every queue would break split deployments where a worker must not take jobs it can't run.

Changes

  • Worker: runNextJob() and daemon({ queue }) take a string or a priority-ordered list; ANY_QUEUE ('*') is exported.
  • QueueWorkCommand: parses --queue into a deduplicated list; the startup line says what it drains.
  • QueueContract.popAny() backs the wildcard. The base implementation works the default queue, so custom drivers behave as before until they override it.
    • database: the oldest due job across all queues (the existing query minus the queue filter), FIFO by arrival, so a busy queue can't starve the rest. The reserved job keeps its own queue name.
    • redis: discovers queues with SCAN (never the blocking KEYS), including queues holding only delayed jobs, and works them round-robin. RedisClient gains scan, which ioredis already has.
  • Docs: the queue guide and README cover all three --queue forms and the "every queue needs a worker" rule. The jobs guide warns that a bare worker ignores onQueue(...) jobs. The CHANGELOG has an Unreleased entry.
  • A separate style: commit holds unrelated pnpm format output (6 files, formatting only).

Tests

12 new tests in packages/queue/tests/multi-queue.test.ts:

  • Worker: priority order, skipping empty queues, never touching unlisted queues, * draining all, and webhook,* ordering.
  • database popAny: run against a fake table that applies the real filters. Checks there's no queue filter and the reserved job keeps its own queue.
  • redis popAny: run against a fake client. Checks discovery across two SCAN pages, round-robin order, and a queue holding only delayed jobs being found once due.

tests/MemoryQueue.ts now keeps jobs per queue (unnamed pushes and pops still use default), so the existing worker tests pass unchanged. All 24 queue tests pass.

`ark queue:work` could only work one queue, and a bare worker silently
ignores jobs dispatched to any other queue. `--queue` now accepts:

- a comma-separated priority list (`--queue=webhook,ocr,default`): each
  loop takes the first job found walking the list;
- `*` for every queue on the connection, combinable with names to
  prioritize some (`--queue=webhook,*`).

A bare `queue:work` still works only the default queue.

`QueueContract.popAny()` backs the wildcard. The database driver takes
the oldest due job across all queues (FIFO, so no queue starves); the
redis driver discovers queues with SCAN (never KEYS) and works them
round-robin. The base implementation falls back to the default queue,
so drivers without it behave as before.
@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.01299% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
packages/queue/tests/MemoryQueue.ts 78.78% 7 Missing ⚠️
packages/queue/src/Contracts/QueueContract.ts 0.00% 2 Missing ⚠️
packages/console/src/prepare/BuildInterfaces.ts 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@3m1n3nc3
3m1n3nc3 merged commit c1f053c into main Oct 20, 2026
4 checks passed
@3m1n3nc3
3m1n3nc3 deleted the feat/queue-priority-wildcard branch October 21, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant