You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A render task that never returned (eg some text locking up the renderer) held its thread forever; once every
thread was stuck, nothing else was drawn, on surfaces or in the ui.
This adds some logging of the failures, and attempts to recover as gracefully as we can. This includes quarantining the failed draws, and when necessary restarting companion to kill the stalled threads.
The failures are now reported to the ui, so this should be very noticable when this happens, and will hoepfully result in better bug reports instead of confusion over all buttons not drawing.
As we run the drawing in worker-threads, we can't kill one of them directly without restarting all of companion. And we don't want to just keep spawning more, as the threads could be stuck with high cpu and will leak memory
Summary by CodeRabbit
New Features
Added a render-stall banner showing affected renders, their location or description, and available text.
Quarantined renders display a placeholder instead of repeatedly blocking rendering, and can be retried from the banner.
The banner reports scheduled or manual restart status, with options to restart immediately or postpone a scheduled restart.
Rendering now recovers from stalled tasks by replacing worker pools. Lost render threads can trigger a scheduled restart, with safeguards against repeated restart loops.
…drawing
A render task that never returned (eg some text locking up the renderer) held its thread forever; once every
thread was stuck, nothing else was drawn, on surfaces or in the ui. Each task now has a timeout, after which its
thread is terminated and replaced, and the button and its draw style are logged so the cause can be found.
process.exit waits for every thread to stop, which a render thread stuck in native code never does, so quitting
(or the launcher going away) left the process running forever. After a stall it is killed instead.
…stalling again
A stalled draw stalled again at each size it was shown at and each time it was redrawn, until every render thread
was stuck. It is now quarantined for the rest of the run, and drawn as a placeholder.
…ds which are stuck for good
A stalled thread which can't be terminated (eg stuck in native code) took a place in the pool for good. Its pool is
now replaced, so drawing carries on with new threads, and if it doesn't stop in time it is counted as lost.
…stalls to the ui
Once a pool's worth of render threads are stuck for good, Companion restarts itself a minute later, which can be
postponed. The stalled draws stay quarantined across the restart, so if it has to restart twice within 10 minutes
anyway, the restart is left to the user. The stalled draws and any restart are reported to the ui.
…restart to recover
A banner lists the draws which stalled and are shown as a placeholder, with their text and a link to their page, and
can try them again. When too many render threads are stuck it counts down to the restart, which can be postponed or
done now.
The render pipeline now detects stalled worker tasks, replaces stalled pools, and quarantines affected draws. A monitor tracks lost threads and restart history. The web UI displays quarantined renders and provides retry and restart controls.
Render tasks use per-method timeouts. RenderPool replaces pools after stalls, retries queued tasks, and tracks terminations and lost threads. GraphicsController uses the pool and exposes its metrics. Tests cover execution, timeouts, pool replacement, and lost-thread accounting.
Quarantine stalled draws shared-lib/lib/Model/RenderStall.ts, companion/lib/Data/Cache.ts, companion/lib/Graphics/RenderQuarantine.ts, companion/lib/Graphics/Controller.ts, companion/test/Graphics/RenderQuarantine.test.ts
The quarantine stores draw keys and stall details, persists entries for the current build, and applies a placeholder to quarantined styles. GraphicsController records stalled render details and regenerates renders when quarantine changes. Tests cover key generation, persistence, and placeholder application.
The monitor publishes render status and schedules restarts based on lost-thread counts and restart history. Registry exposes its router and passes the stalled-thread state to process-exit handling. The process callback uses SIGKILL when render threads may be stuck. Tests cover scheduling, cancellation, postponement, and manual restart status.
Render stall banner and controls webui/src/App.tsx, webui/src/Layout/RenderStallBanner.tsx, webui/src/Layout/RenderStallBannerView.tsx, webui/src/Layout/RenderStallBanner.css, webui/src/Layout/__tests__/RenderStallBannerView.test.tsx
The connected, unlocked app displays render-stall status. The banner lists quarantined renders and provides retry and restart actions, including postponement for scheduled restarts. Tests cover empty, quarantined, scheduled, and manual states.
Merge Risk: Moderate · up to fe5bc
The new stall recovery has gaps. Retrying a stalled button may not redraw it. Stalled presets may not show their placeholder. Lost-thread counting can trigger a restart that is not needed. An automatic restart requested during startup can be lost. Fix these before merging; the manual restart control remains available as a workaround.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title "feat: tracing of stalled render threads" matches the main change. The PR adds detection, logging and reporting of render tasks that do not return, plus quarantine, pool replacement, and res…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title "feat: tracing of stalled render threads" matches the main change. The PR adds detection, logging and reporting of render tasks that do not return, plus quarantine, pool replacement, and restart recovery. The title covers the stall tracing and reporting part, but not the quarantine, pool replacement, or restart recovery parts. It is a clear and specific summary of the primary change, and it follows a conventional commit style.
Finishing Touches
Fix failing CI checks
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
A worker stalls; the pool renews, A careful record marks the draw. Lost threads count, and clocks take note, A banner shows the next safe step. Retry or wait; the status stays in view.
pushed = false and initial = [] make new parameters optional. Remove the defaults and pass false or [] explicitly at each affected call site. This also makes each test's initial state explicit.
As per coding guidelines, “New function parameters are required, not optional,” and arguments must be threaded through all call sites.
Also applies to: 36-36
Source: Coding guidelines
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: ecdf89e8-8788-463f-9155-818a5b996bcb
Commits
Reviewing files that changed from the base of the PR and between 38099f7 and fe5bcd6.
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Please invalidate cached renders before retrying quarantined draws.
When a static button stalls, its original ImageResult remains in #renderLRUCache. ImageResult.drawNative also retains the rejected promise for that size and format. (raw.githubusercontent.com)
When retry removes quarantine, regeneration selects the original cache key and reuses that failed result. No new worker task runs. The retry endpoint only removes the quarantine entry. Please evict the affected cached renders before regeneration, so retry creates a fresh ImageResult. (raw.githubusercontent.com)
Minor · Functional Correctness · Quick win
Include presets in quarantine-triggered regeneration.
The renderQuarantinechanged listener calls triggerRegenerateAll, which runs #regenerateAll. That method redraws only populated page locations. A preset has no page location, so a quarantine change does not redraw it. A non-clock-sensitive preset whose draw stalled can keep its stalled render, and it does not show its placeholder until another event invalidates it.
Quarantine changes should also redraw presets, so each affected preset shows its placeholder or its restored draw. Keep the existing button regeneration as it is.
The reason will be displayed to describe this comment to others. Learn more.
Major · Stability & Availability · Heavy lift
Please count stalled workers that are still running.
If four tasks time out, stalledThreads becomes four. If three workers then exit and one remains blocked, entry.exited stays false because pool.terminate() waits for every worker. This line reports four lost threads instead of one. workerpool 10.0.3 aggregates all worker termination promises. (raw.githubusercontent.com)
The count can remain inflated indefinitely and cause an unnecessary restart when it reaches the monitor's budget. Please track individual stalled-worker exits and remove recovered workers from the count without waiting for the whole pool. The monitor schedules restart directly from this count. (raw.githubusercontent.com)
The reason will be displayed to describe this comment to others. Learn more.
Minor · Stability & Availability · Quick win
A stall-triggered restart can be dropped when startup has not finished.
ready() starts the render regeneration before it awaits initInstances(). If the render pool reports enough stuck threads and startup is still pending when the 60-second restart countdown fires, exit() returns early because #isReady is still false. The monitor's timer clears itself but leaves its scheduled restart in place. Later stall reports do not reschedule it, so the restart never runs automatically. Only a manual restart from the UI, which is available after startup, recovers the state.
Consider either deferring the restart until #isReady is set, or clearing the monitor's restart state when the request is dropped so a later stall report can schedule it again. Either approach keeps the restart from being lost.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A render task that never returned (eg some text locking up the renderer) held its thread forever; once every
thread was stuck, nothing else was drawn, on surfaces or in the ui.
This adds some logging of the failures, and attempts to recover as gracefully as we can. This includes quarantining the failed draws, and when necessary restarting companion to kill the stalled threads.
The failures are now reported to the ui, so this should be very noticable when this happens, and will hoepfully result in better bug reports instead of confusion over all buttons not drawing.
As we run the drawing in worker-threads, we can't kill one of them directly without restarting all of companion. And we don't want to just keep spawning more, as the threads could be stuck with high cpu and will leak memory
Summary by CodeRabbit