What happened?
We have a db_worker that occasionally exits when MySQL has a brief DNS or network hiccup.
These are the two errors I have seen against Azure Database for MySQL:
OperationalError: (2005, "Unknown server host '<server>.mysql.database.azure.com' (-5)")
OperationalError: (2013, "Lost connection to server at 'handshake: reading initial communication packet', system error: 110")
The traceback ends up in Worker.run() while entering exclusive_transaction(), then goes through transaction.atomic(), ensure_connection(), and MySQLdb.Connect().
The worker handles an OperationalError when the database is locked, but re-raises other connection errors. That causes the management command to exit and leaves the process manager to restart it.
In these cases the connection fails before a task is claimed, so there is no task result to mark as failed or retry.
What I expected
A short connection failure during the idle polling loop should not stop the worker. It would be helpful if the worker closed the unusable connection, waited with a bounded backoff, and tried again while still responding to shutdown signals.
Errors that happen while an actual task is running should keep their current behavior.
Test idea
Mock get_new_connection() or the polling transaction so it raises OperationalError once and succeeds on the next attempt, then check that the worker stays alive and resumes polling.
Environment
- django-tasks-db 0.12.0
- Django 6.0.5
- Python 3.12.13
- mysqlclient 2.2.8
- MySQL
CONN_MAX_AGE=0db_worker --interval 1 (non-batch, no reload)
This has happened only a few times across multiple application containers and database endpoints, so it looks like a transient infrastructure issue rather than a persistent database configuration problem.
What happened?
We have a
db_workerthat occasionally exits when MySQL has a brief DNS or network hiccup.These are the two errors I have seen against Azure Database for MySQL:
The traceback ends up in
Worker.run()while enteringexclusive_transaction(), then goes throughtransaction.atomic(),ensure_connection(), andMySQLdb.Connect().The worker handles an
OperationalErrorwhen the database is locked, but re-raises other connection errors. That causes the management command to exit and leaves the process manager to restart it.In these cases the connection fails before a task is claimed, so there is no task result to mark as failed or retry.
What I expected
A short connection failure during the idle polling loop should not stop the worker. It would be helpful if the worker closed the unusable connection, waited with a bounded backoff, and tried again while still responding to shutdown signals.
Errors that happen while an actual task is running should keep their current behavior.
Test idea
Mock
get_new_connection()or the polling transaction so it raisesOperationalErroronce and succeeds on the next attempt, then check that the worker stays alive and resumes polling.Environment
CONN_MAX_AGE=0db_worker --interval 1(non-batch, no reload)This has happened only a few times across multiple application containers and database endpoints, so it looks like a transient infrastructure issue rather than a persistent database configuration problem.