[tcely]

TODO

• Services for the consumer workers
• Enqueue startup tasks in ready

Repository

https://github.com/RealOrangeOne/django-tasks-db

Backends

1. default
2. huey

Queues

1. database
2. filesystem
3. network
4. limited
5. default

A worker service for the huey backend. Another worker service for the limited queue of the default backend. Multiple worker services for the remaining queues of the default backend.

Services

Bundle: django-tasks-consumers

1. django-tasks-default
2. django-tasks-net-limited
3. django-tasks-huey (uses the run_huey command)

Flags: -v 2 --no-reload --interval 20 --max-tasks 100 --queue-name '*' --exclude-queues limited

Worker

usage output

usage: manage.py db_worker [-h] [--queue-name [QUEUE_NAME]]
                           [--exclude-queues [EXCLUDE_QUEUES]]
                           [--interval [INTERVAL]] [--batch]
                           [--reload | --no-reload]
                           [--backend [BACKEND_NAME]]
                           [--no-startup-delay]
                           [--max-tasks [MAX_TASKS]]
                           [--worker-id [WORKER_ID]] [--version]
                           [-v {0,1,2,3}] [--settings SETTINGS]
                           [--pythonpath PYTHONPATH] [--traceback]
                           [--no-color] [--force-color]
                           [--skip-checks]

Run a database background worker

options:
  -h, --help            show this help message and exit
  --queue-name [QUEUE_NAME]
                        The queues to process. Separate multiple with a comma.
                        To process all queues, use '*' (default: 'default')
  --exclude-queues [EXCLUDE_QUEUES]
                        Queues to exclude. Separate multiple with a comma.
  --interval [INTERVAL]
                        The interval (in seconds) to wait, when
                        there are no tasks in the queue, before
                        checking for tasks again (default: 1)
  --batch               Process all outstanding tasks, then exit.
                        Can be used in combination with --max-                               tasks.
  --reload, --no-reload
                        Reload the worker on code changes.
                        Not recommended for production as tasks may not
                        be stopped cleanly (default: DEBUG)
  --backend [BACKEND_NAME]
                        The backend to operate on (default: 'default')
  --no-startup-delay    Don't add a small delay at startup.
  --max-tasks [MAX_TASKS]
                        If provided, the maximum number of tasks
                        the worker will execute before exiting.
  --worker-id [WORKER_ID]
                        Worker id. MUST be unique across worker
                        pool (default: auto-generate)
  --version             Show program's version number and exit.
  -v, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't
                        provided, the DJANGO_SETTINGS_MODULE
                        environment variable will be used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Display a full stack trace on CommandError exceptions.
  --no-color            Don't colorize the command output.
  --force-color         Force colorization of the command output.
  --skip-checks         Skip system checks.

[tcely]

usage: manage.py run_huey [-h] [--logfile FILE] [--huey-verbose] [--quiet]
                          [--simple] [--workers WORKERS]
                          [--worker-type {thread,greenlet,process}]
                          [--delay SECONDS] [--max-delay SECONDS]
                          [--backoff FACTOR] [--health-check-interval SECONDS]
                          [--disable-health-check] [--flush-locks]
                          [--extra-locks EXTRA_LOCKS] [--max-tasks COUNT]
                          [--shutdown-timeout SECONDS]
                          [--graceful-signal {INT,TERM}]
                          [--scheduler-interval SCHEDULER_INTERVAL]
                          [--no-periodic] [-A] [--version] [-v {0,1,2,3}]
                          [--settings SETTINGS] [--pythonpath PYTHONPATH]
                          [--traceback] [--no-color] [--force-color]
                          [--skip-checks]

Run the queue consumer

options:
  -h, --help            show this help message and exit
  --logfile, -l FILE
  --huey-verbose, -V    verbose logging (includes DEBUG statements)
  --quiet, -q           minimal logging
  --simple, -S          simple logging format (time message)
  --workers, -w WORKERS
                        number of worker threads/processes (default=1)
  --worker-type, -k {thread,greenlet,process}
                        worker execution model (thread, greenlet, process). Use
                        process for CPU-intensive workloads, and greenlet for
                        IO-heavy workloads. When in doubt, thread is the safest
                        choice.
  --delay, -d SECONDS   minimum time to wait when polling queue (default=.1)
  --max-delay, -m SECONDS
                        maximum time to wait when polling queue (default=10)
  --backoff, -b FACTOR  factor used to back-off polling interval when queue is
                        empty (default=1.15, must be >= 1)
  --health-check-interval, -c SECONDS
                        minimum time to wait between worker health checks
                        (default=10)
  --disable-health-check, -C
                        disable health check that monitors worker health,
                        restarting any worker that crashes unexpectedly.
  --flush-locks, -f     flush all locks when starting consumer.
  --extra-locks, -L EXTRA_LOCKS
                        additional locks to flush, separated by comma.
  --max-tasks, -M COUNT
                        restart worker after it has executed COUNT tasks
  --shutdown-timeout, -t SECONDS
                        seconds to wait for workers to finish on graceful
                        shutdown before interrupting them (default=wait forever)
  --graceful-signal, -g {INT,TERM}
                        signal that triggers graceful shutdown, INT or TERM
                        (default=INT). The other signal interrupts running
                        tasks.
  --scheduler-interval, -s SCHEDULER_INTERVAL
                        Granularity of scheduler in seconds.
  --no-periodic, -n     do NOT enqueue periodic tasks
  -A, --disable-autoload
                        Do not autoload "tasks.py"
  --version             Show program's version number and exit.
  -v, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Display a full stack trace on CommandError exceptions.
  --no-color            Don't colorize the command output.
  --force-color         Force colorization of the command output.
  --skip-checks         Skip system checks.