Skip to content

feat: tracing of stalled render threads - #4622

Open
Julusian wants to merge 8 commits into
mainfrom
feat/render-stall-recovery
Open

Julusian wants to merge 8 commits into
mainfrom
feat/render-stall-recovery

Conversation

@Julusian

@Julusian Julusian commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

A render task that never returned (eg some text locking up the renderer) held its thread forever; once every
thread was stuck, nothing else was drawn, on surfaces or in the ui.

This adds some logging of the failures, and attempts to recover as gracefully as we can. This includes quarantining the failed draws, and when necessary restarting companion to kill the stalled threads.
The failures are now reported to the ui, so this should be very noticable when this happens, and will hoepfully result in better bug reports instead of confusion over all buttons not drawing.

As we run the drawing in worker-threads, we can't kill one of them directly without restarting all of companion. And we don't want to just keep spawning more, as the threads could be stuck with high cpu and will leak memory

Summary by CodeRabbit

  • New Features
    • Added a render-stall banner showing affected renders, their location or description, and available text.
    • Quarantined renders display a placeholder instead of repeatedly blocking rendering, and can be retried from the banner.
    • The banner reports scheduled or manual restart status, with options to restart immediately or postpone a scheduled restart.
    • Rendering now recovers from stalled tasks by replacing worker pools. Lost render threads can trigger a scheduled restart, with safeguards against repeated restart loops.

…drawing

A render task that never returned (eg some text locking up the renderer) held its thread forever; once every
thread was stuck, nothing else was drawn, on surfaces or in the ui. Each task now has a timeout, after which its
thread is terminated and replaced, and the button and its draw style are logged so the cause can be found.
Once per run for each button and text, as a stalled draw stalls again at each size and redraw.
process.exit waits for every thread to stop, which a render thread stuck in native code never does, so quitting
(or the launcher going away) left the process running forever. After a stall it is killed instead.
…stalling again

A stalled draw stalled again at each size it was shown at and each time it was redrawn, until every render thread
was stuck. It is now quarantined for the rest of the run, and drawn as a placeholder.
…date

A draw which stalls would stall again after a restart, so the quarantine is kept in the cache, for the build it
stalled on.
…ds which are stuck for good

A stalled thread which can't be terminated (eg stuck in native code) took a place in the pool for good. Its pool is
now replaced, so drawing carries on with new threads, and if it doesn't stop in time it is counted as lost.
…stalls to the ui

Once a pool's worth of render threads are stuck for good, Companion restarts itself a minute later, which can be
postponed. The stalled draws stay quarantined across the restart, so if it has to restart twice within 10 minutes
anyway, the restart is left to the user. The stalled draws and any restart are reported to the ui.
…restart to recover

A banner lists the draws which stalled and are shown as a placeholder, with their text and a link to their page, and
can try them again. When too many render threads are stuck it counts down to the restart, which can be postponed or
done now.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Walkthrough

The render pipeline now detects stalled worker tasks, replaces stalled pools, and quarantines affected draws. A monitor tracks lost threads and restart history. The web UI displays quarantined renders and provides retry and restart controls.

Changes: Render stall recovery

Layer / File(s) Summary
Render task execution and stall detection
companion/lib/Graphics/RenderPool.ts, companion/lib/Graphics/RenderTaskTimeout.ts, companion/lib/Graphics/Controller.ts, companion/test/Graphics/RenderPool.test.ts, companion/test/Graphics/RenderTaskTimeout.test.ts, companion/test/Graphics/helpers/StallingThread.mjs
Render tasks use per-method timeouts. RenderPool replaces pools after stalls, retries queued tasks, and tracks terminations and lost threads. GraphicsController uses the pool and exposes its metrics. Tests cover execution, timeouts, pool replacement, and lost-thread accounting.
Quarantine stalled draws
shared-lib/lib/Model/RenderStall.ts, companion/lib/Data/Cache.ts, companion/lib/Graphics/RenderQuarantine.ts, companion/lib/Graphics/Controller.ts, companion/test/Graphics/RenderQuarantine.test.ts
The quarantine stores draw keys and stall details, persists entries for the current build, and applies a placeholder to quarantined styles. GraphicsController records stalled render details and regenerates renders when quarantine changes. Tests cover key generation, persistence, and placeholder application.
Lost-thread restart monitoring
companion/lib/Graphics/RenderStallMonitor.ts, companion/lib/Registry.ts, companion/lib/UI/TRPC.ts, companion/lib/main.ts, companion/test/Graphics/RenderStallMonitor.test.ts
The monitor publishes render status and schedules restarts based on lost-thread counts and restart history. Registry exposes its router and passes the stalled-thread state to process-exit handling. The process callback uses SIGKILL when render threads may be stuck. Tests cover scheduling, cancellation, postponement, and manual restart status.
Render stall banner and controls
webui/src/App.tsx, webui/src/Layout/RenderStallBanner.tsx, webui/src/Layout/RenderStallBannerView.tsx, webui/src/Layout/RenderStallBanner.css, webui/src/Layout/__tests__/RenderStallBannerView.test.tsx
The connected, unlocked app displays render-stall status. The banner lists quarantined renders and provides retry and restart actions, including postponement for scheduled restarts. Tests cover empty, quarantined, scheduled, and manual states.
Merge Risk: Moderate · up to fe5bc

The new stall recovery has gaps. Retrying a stalled button may not redraw it. Stalled presets may not show their placeholder. Lost-thread counting can trigger a restart that is not needed. An automatic restart requested during startup can be lost. Fix these before merging; the manual restart control remains available as a workaround.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "feat: tracing of stalled render threads" matches the main change. The PR adds detection, logging and reporting of render tasks that do not return, plus quarantine, pool replacement, and res…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "feat: tracing of stalled render threads" matches the main change. The PR adds detection, logging and reporting of render tasks that do not return, plus quarantine, pool replacement, and restart recovery. The title covers the stall tracing and reporting part, but not the quarantine, pool replacement, or restart recovery parts. It is a clear and specific summary of the primary change, and it follows a conventional commit style.

Finishing Touches

Fix failing CI checks

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A worker stalls; the pool renews,
A careful record marks the draw.
Lost threads count, and clocks take note,
A banner shows the next safe step.
Retry or wait; the status stays in view.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Nitpick comments: 1

companion/test/Graphics/RenderQuarantine.test.ts: 1

13-13: Trivial · Maintainability & Code Quality · Quick win

Please make both fixture parameters required.

pushed = false and initial = [] make new parameters optional. Remove the defaults and pass false or [] explicitly at each affected call site. This also makes each test's initial state explicit.

As per coding guidelines, “New function parameters are required, not optional,” and arguments must be threaded through all call sites.

Also applies to: 36-36

Source: Coding guidelines

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ecdf89e8-8788-463f-9155-818a5b996bcb
Commits

Reviewing files that changed from the base of the PR and between 38099f7 and fe5bcd6.

Files selected for processing: 20

  • companion/lib/Data/Cache.ts
  • companion/lib/Graphics/Controller.ts
  • companion/lib/Graphics/RenderPool.ts
  • companion/lib/Graphics/RenderQuarantine.ts
  • companion/lib/Graphics/RenderStallMonitor.ts
  • companion/lib/Graphics/RenderTaskTimeout.ts
  • companion/lib/Registry.ts
  • companion/lib/UI/TRPC.ts
  • companion/lib/main.ts
  • companion/test/Graphics/RenderPool.test.ts
  • companion/test/Graphics/RenderQuarantine.test.ts
  • companion/test/Graphics/RenderStallMonitor.test.ts
  • companion/test/Graphics/RenderTaskTimeout.test.ts
  • companion/test/Graphics/helpers/StallingThread.mjs
  • shared-lib/lib/Model/RenderStall.ts
  • webui/src/App.tsx
  • webui/src/Layout/RenderStallBanner.css
  • webui/src/Layout/RenderStallBanner.tsx
  • webui/src/Layout/RenderStallBannerView.tsx
  • webui/src/Layout/__tests__/RenderStallBannerView.test.tsx

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

this.setMaxListeners(0)

// Draw a newly quarantined draw as its placeholder, or one which is allowed again as itself
this.renderQuarantine.on('changed', () => this.triggerRegenerateAll())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Please invalidate cached renders before retrying quarantined draws.

When a static button stalls, its original ImageResult remains in #renderLRUCache. ImageResult.drawNative also retains the rejected promise for that size and format. (raw.githubusercontent.com)

When retry removes quarantine, regeneration selects the original cache key and reuses that failed result. No new worker task runs. The retry endpoint only removes the quarantine entry. Please evict the affected cached renders before regeneration, so retry creates a fresh ImageResult. (raw.githubusercontent.com)


Minor · Functional Correctness · Quick win

Include presets in quarantine-triggered regeneration.

The renderQuarantine changed listener calls triggerRegenerateAll, which runs #regenerateAll. That method redraws only populated page locations. A preset has no page location, so a quarantine change does not redraw it. A non-clock-sensitive preset whose draw stalled can keep its stalled render, and it does not show its placeholder until another event invalidates it.

Quarantine changes should also redraw presets, so each affected preset shows its placeholder or its restored draw. Keep the existing button regeneration as it is.

this.#logger.error(
`${entry.stalledThreads} stalled render threads didn't stop within ${this.#lostThreadGraceMs / 1000}s, so are stuck`
)
this.#setLostThreads(this.lostThreads + entry.stalledThreads)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Stability & Availability · Heavy lift

Please count stalled workers that are still running.

If four tasks time out, stalledThreads becomes four. If three workers then exit and one remains blocked, entry.exited stays false because pool.terminate() waits for every worker. This line reports four lost threads instead of one. workerpool 10.0.3 aggregates all worker termination promises. (raw.githubusercontent.com)

The count can remain inflated indefinitely and cause an unnecessary restart when it reaches the monitor's budget. Please track individual stalled-worker exits and remove recovered workers from the count without waiting for the whole pool. The monitor schedules restart directly from this count. (raw.githubusercontent.com)

Comment thread companion/lib/Registry.ts
// Once a whole pool's worth of threads are stuck, the stuck ones are probably slowing everything else
this.graphics.renderPoolSize,
new CachedRenderStallRestartHistory(this.#data.cache.defaultTableView),
() => this.exit(true, true)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Quick win

A stall-triggered restart can be dropped when startup has not finished.

ready() starts the render regeneration before it awaits initInstances(). If the render pool reports enough stuck threads and startup is still pending when the 60-second restart countdown fires, exit() returns early because #isReady is still false. The monitor's timer clears itself but leaves its scheduled restart in place. Later stall reports do not reschedule it, so the restart never runs automatically. Only a manual restart from the UI, which is available after startup, recovers the state.

Consider either deferring the restart until #isReady is set, or clearing the monitor's restart state when the request is dropped so a later stall report can schedule it again. Either approach keeps the restart from being lost.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant