Skip to content

fix(status): refresh on restart and periodically, not only at init - #6

Merged
impuls42 merged 1 commit into
mainfrom
fix/refresh-status-hooks
Jul 30, 2026
Merged

impuls42 merged 1 commit into
mainfrom
fix/refresh-status-hooks

Conversation

@impuls42

Copy link
Copy Markdown
Member

Closes #5.

The problem

The retained topic was read in exactly one place — init_machine. Anything that cleared the machine's shared state left the page at UNKNOWN with an empty status text until someone printed or the pods restarted. Both routes were seen on the cluster:

  • a manual restart from the Admin Center
  • a Redis restart, which drops every machine:* key at once, because this deployment runs Redis without persistence by design

An empty status text is itself the tell: no code path in this driver produces one, so it means the driver never ran, not that it ran and found nothing.

The fix, and a correction

The framework already offered both hooks and this driver implemented neither:

hook when
restart_machine manual restart from the Admin Center
ping_machines periodically, when MACHINE_PING_ENABLED is set
def ping_machines(self):
    """...called periodically by a background task if the setting
       'MACHINE_PING_ENABLED' is active."""

Confirmed live: MACHINE_PING_ENABLED = True, and it defaults to True.

That corrects the issue. #5 said a periodic refresh "would cover the idle case too, but the machine framework does not obviously offer a hook for it." It does, I had not looked, and it turns out to be the part that actually matters — it is what recovers the page after a cache wipe when nobody is printing, which is the case the issue was really about.

ping_machines guards each machine separately so one broker problem does not stop the others refreshing. _refresh_status already turns an unreachable broker into an UNKNOWN status rather than an exception, so the guard is for everything else.

What is deliberately unchanged

print_labels. With AWAIT_RESULT_S=0 the page sits at PRINTING after a dispatch, and the periodic refresh now corrects that within a ping interval. Reading the retained topic immediately after dispatch would be worse — the agent has not yet published anything about that job, so it would report the pre-print state and look like nothing happened.

Testing

34 tests pass, ruff clean. The hooks themselves are three lines each delegating to _refresh_status, and live in driver.py, which has no unit tests here by design — InvenTree is deliberately not a test dependency of this package, which is why the testable logic sits in status.py. Keeping them thin is what makes that acceptable.

Verification is on the cluster after the deploy chain: restart the machine and confirm the page repopulates rather than blanking, and confirm it recovers on its own after a Redis restart.

Closes #5. The status was read from the retained topic in exactly one place --
init_machine -- so anything that cleared the machine's shared state left the page
showing `UNKNOWN` with an empty status text until someone printed or the pods
restarted. Two ways to hit that, both seen on the cluster: a manual restart from the
Admin Center, and a Redis restart, which drops every `machine:*` key at once because
this deployment runs Redis without persistence by design.

The framework already offered both hooks and this driver implemented neither:

  restart_machine   called on a manual restart
  ping_machines     called periodically when MACHINE_PING_ENABLED is set, which is
                    on by default and confirmed on here

That last one is the real fix -- it is what recovers the page after a cache wipe with
nobody printing, which is the case the issue was actually about. The issue text
claimed the framework "does not obviously offer a hook" for periodic work; that was
wrong, and it is corrected there.

ping_machines guards each machine separately so one broker problem does not stop the
others refreshing. _refresh_status already turns an unreachable broker into an
UNKNOWN status rather than an exception, so the guard is for everything else.

print_labels is deliberately unchanged. With AWAIT_RESULT_S=0 the page sits at
PRINTING after a dispatch, and the periodic refresh now corrects that within a ping
interval -- which is better than reading the retained topic immediately, when the
agent has not yet published anything about the job.
@impuls42

Copy link
Copy Markdown
Member Author

Both findings verified rather than assumed. Neither needs a code change, and one of them I could check where you couldn't — machine isn't importable outside InvenTree, so I read it from the running pod.

Hook ordering — confirmed correct. BaseMachineType.restart():

def restart(self):
    """...This will first reset the machines state (errors, status, status_text)
       and then call the drivers restart function."""
    self.restart_required = False
    self.reset_errors()
    self.set_status(self.default_machine_status)   # -> UNKNOWN (101)
    self.set_status_text('')                       # -> ''
    self.driver.restart_machine(self)              # <- the hook, after the reset

The reset happens first and the hook after, so _refresh_status writes survive. It also confirms the original diagnosis in #5: the status=101 text='' I saw was InvenTree clearing the state, with nothing to put it back — those two lines are literally where the empty string came from.

Ping-vs-PRINTING flicker — real, narrow, and accepted. You are right that a ping landing between publish() and the agent's first status publish reads the pre-print retained state. The window is about two seconds: measured on the cluster, the agent publishes printing at +2.2s and settles to idle at +12.9s after dispatch. So a ping has to land inside that window to show CONNECTED briefly, and the next ping corrects it either way.

That is the same trade-off as leaving a stuck PRINTING on the page indefinitely with AWAIT_RESULT_S=0, and I would rather have a two-second race than a permanently wrong status. Worth stating explicitly rather than leaving implicit, which is what your comment prompted — thanks.

On the asymmetry you flagged as fine: agreed, and to say it out loud — ping_machines guards per machine because it iterates, while restart_machine and init_machine are single user-initiated actions where an exception belongs to the framework's own error handling.

@impuls42
impuls42 merged commit 94dd547 into main Jul 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Status is only refreshed at init, so it goes blank after any cache loss or machine restart

1 participant