Repository navigation
core: background job queue for chain submission and long-running work - #71
Merged
joelpeace48-cell merged 1 commit intoAug 25, 2026
Merged
Conversation
…ing work Adds a small file-backed durable job queue (src/queue) for work that takes too long to run inline in a request handler: transaction submission/confirmation polling, export generation, indexer catch-up, and webhook fan-out. - Durable: every state transition is written to disk (write-temp then rename) so pending/active jobs survive a process restart. - Idempotent handlers: a job id that has already been observed to complete is never re-dispatched, including across a simulated crash-and-restart against the same durable file — covered by a test that runs the same job through two separate JobQueue instances. - Retry with exponential backoff (jittered, bounded) up to a per-job maxAttempts, after which the job moves to a dead-letter status that is queryable via deadLetterQueue(). - metrics() exposes queue depth, oldest-pending age, dead-letter count, and failure rate. Closes FinesseStudioLab#54.
Contributor
|
@Codeinx Nice job |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Submitting a Soroban transaction and waiting for confirmation takes several seconds. Doing that inside a request handler ties up a connection, times out under load, and loses the work entirely if the process restarts mid-flight.
What this adds
New module:
src/queue/JobQueue+JobStore) backed by a JSON snapshot file, written with write-to-temp-then-rename on every mutation so a crash mid-write never leaves a half-written file behind. Chosen over BullMQ/Redis to keep this repo's dependency footprint and CI/test setup unchanged — the acceptance criteria call for jobs to survive a restart, not for a specific backend.type, so callers plug in their own).maxAttemptsis exhausted (deadLetterQueue()).metrics()exposes queue depth, oldest-pending age, dead-letter count, and failure rate.Testing
src/queue/queue.test.tscovers:run()only executes onceJobQueueinstances against the same durable file, verifying the handler does not re-run on the second instancenpm run lint,npm run build,npm test, andnpm audit --audit-level=highare all clean on this branch.Scope note
This PR only adds
src/queue/**(new files) — it does not touchpackage.json,src/routes/v1/index.ts, or any other file shared with the other two PRs open against this issue tracker (#44,#58), so it should apply cleanly regardless of merge order.Closes #54.