Skip to content

feat: buffer connection logs and flush them on failure - #374

Open
fioan89 wants to merge 17 commits into
mainfrom
feature/flight-recorder
Open

fioan89 wants to merge 17 commits into
mainfrom
feature/flight-recorder

Conversation

@fioan89

@fioan89 fioan89 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Up until today Toolbox discards plugin messages below its active log level, so connection failures can leave diagnostic logs without the preceding context unless verbose logging is enabled beforehand.

CoderLogger now captures filtered messages in a private flight recorder backed by a fixed-capacity ring buffer. Entries retain their original timestamp, severity, message, and exception details. Both string and lazy logging methods participate in recording, so future uses of lazy methods cannot silently bypass capture.

The connectionLogBufferSize setting defaults to 1000 entries, accepts values from 0 to 10000, and clears and disables recording at 0. Plugin settings expose changes through Kotlin flows, allowing the recorder to resize without a restart. Shrinking discards the oldest entries, while growing preserves existing entries.

Connection lifecycle events and failures trigger replay into the Toolbox log files. Per-environment support bundle collection also flushes the recorder before export. Replay preserves chronological order and original timestamps and levels, groups up to 100 entries per call, and uses a severity accepted by the current Toolbox logging level. Replayed entries are removed from memory. Logging set to Off prevents both capture and replay.

An important note is that the Toolbox API upgrade to 1.15.88839 raises the minimum supported Toolbox version to 3.9 but the change is required in order to have access to the logging level configured in TBX.

The plugin targets Toolbox API 1.13.87111 instead of the required
1.15.88839 release.

This commit updates the shared API version so the plugin dependencies
and packaged manifest consistently target 1.15.88839.
The plugin has no setting for a connection log buffer limit, so users
cannot configure how many diagnostic entries a buffer should retain.

This commit adds a General settings field backed by the
connectionLogBufferSize key. The settings store defaults to 1000 entries
and accepts whole numbers from 0 through 10000, with 0 representing
disabled buffering. Invalid stored values fall back to the default,
while the settings page rejects invalid input before saving.

The README describes the setting, and tests cover persistence, range
validation, default handling, and reloading the field.
@linear-code

linear-code Bot commented Oct 20, 2026

Copy link
Copy Markdown

JET-4

The plugin settings store exposes current values without a way for
other components to observe changes.

This commit adds typed observation that emits the current effective
value and subsequent distinct values. Subscribers can cancel collection
independently, and slow collectors receive the latest state.

The settings store serializes writes and observer reads, publishing
changes after successful storage updates. TLS settings use value
equality to suppress duplicate notifications. Existing consumers retain
their current behavior.

Tests cover initial values, duplicate suppression, failed writes,
cancellation, multiple subscribers, and concurrent updates.
Currently, Toolbox discards plugin logs below the active log level, which
removes detail needed to diagnose connection failures.

This commit adds a private flight recorder to CoderLogger for messages
supplied as strings. The recorder uses Toolbox's filtering decision to
retain each message with its original level, timestamp, and exception
details in a bounded ring buffer. It observes the plugin's buffer capacity
setting, preserves the newest records when shrinking, and clears the
buffer when recording is disabled.
Currently, CoderLogger delegates lambda logging overloads directly to
Toolbox, bypassing the flight recorder. The plugin does not use these
overloads yet, but introducing a call later can silently lose filtered
messages.

This commit overrides every lambda logging overload in CoderLogger,
including exception variants, so future calls participate in recording.
FlightRecorder evaluates captured messages once and reuses their text,
while leaving the original suppliers unevaluated when capture is
unnecessary.
Currently, Toolbox disables every log level when logging is off, so the
flight recorder treats those messages as filtered and retains them.

This commit makes FlightRecorder check Toolbox's effective log level
before capturing either string or lambda messages. When the level is off,
the recorder leaves buffered entries untouched and does not evaluate
message suppliers.
The flight recorder retains suppressed log entries but has no way to replay
them into the Toolbox log, so the captured context cannot be written on
demand.

CoderLogger now exposes a flush operation, and FlightRecorder replays entries
from oldest to newest in batches of up to 100. The replay preserves original
UTC timestamps and severity in each message, uses a level the Toolbox logger
currently persists, and removes entries after their batch is logged. Logging
at OFF leaves buffered entries untouched and produces no replay output.

The CoderLogger test suite exercises replay formatting, batching, log levels,
buffer resizing, and lazy messages through controllable mocked dependencies.
Connection lifecycle messages do not replay the flight recorder, so the
Toolbox log can omit the suppressed context leading up to a disconnect or
connection-affecting failure.

The remote environment and provider now flush after the selected lifecycle
messages. The REST client, workspace filter, and action failure handler also
flush at their corresponding triggers. Each flush follows its log message
so filtered trigger messages appear in the replay, and environment-specific
flushes retain the available session ID.

Replay tests cover action failure context and filtered status updates through
CoderLogger. The REST tests configure the Toolbox logger's effective level
and filter explicitly so the new flush calls use realistic logging behavior.
Support bundle export leaves suppressed log entries in memory, so exported
Toolbox logs can miss the context recorded before collection.

CoderSupportBundleCollector now flushes the flight recorder through
CoderLogger before collecting diagnostics. The replay uses the support bundle
reason and current session ID to preserve the connection context in the log.

The regression test exercises a real CoderLogger and verifies that buffered
entries are replayed before bundle collection and removed afterwards.
The README describes the buffer limit but does not explain when recorded logs
are replayed or where to find them, leaving connection diagnostics difficult
to collect and interpret.

The flight recorder documentation now covers configurable capacity, live
resizing, log levels, replay triggers and formatting, and Toolbox log
locations. The settings instructions identify the action menu in the Coder
page header. The collection guidance also explains that per-environment log
collection requests a recorder flush and Coder support bundle, while general
Toolbox diagnostics collection does not explicitly request either.
The release notes omit the flight recorder and list an outdated Toolbox
minimum, while the support bundle requirements do not identify both Coder
components.

The README and changelog now identify Toolbox 3.9 as the minimum supported
version and specify Coder server and CLI 2.10.0 or newer for support bundle
collection. The release notes also summarize the flight recorder and explain
that Toolbox logs and eligible local SSH logs remain available when a Coder
support bundle cannot be generated.
The connection log buffer description wraps across several lines in Toolbox's
narrow settings window.

The settings page now uses a 46-character description that names the flight
recorder and retains the guidance to set its capacity to zero to disable it.
The localization catalog uses the same text.
@fioan89
fioan89 marked this pull request as ready for review October 22, 2026 19:52
CoderRemoteEnvironment logs each action-menu refresh with an unreadable status object, which adds noise to the flight recorder.

This commit removes that debug message because the existing status-transition logs already provide readable workspace and agent details.
CoderRemoteEnvironment flushes the flight recorder after every status update, so routine updates drain diagnostic context even when reachability does not change.

This commit limits the status-update flush to transitions from reachable to unreachable using the state published to Toolbox. Status messages continue to be logged for every update, while buffered context remains available through other transitions. Regression tests cover all four reachability transition combinations, and the README describes the narrower trigger.
CoderRemoteProvider flushes the flight recorder whenever the environment list changes, so initial discovery and routine additions drain diagnostic context.

This commit removes that flush while preserving SSH reconfiguration. Removed environments with an SSH session continue to flush during disposal, and the README now reflects the remaining replay triggers.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Automatic replay can persist sensitive process arguments, while concurrent capture ordering and initial setup-failure replay remain unresolved.

3 open issues
What changed in this PR

Adds a configurable flight recorder to preserve filtered plugin logs and replay diagnostic context during connection lifecycle events, failures, and support-bundle collection.

Changes:

  • Adds bounded log buffering, batched replay, and live capacity updates through observable settings.
  • Integrates replay triggers and adds recorder, settings, and export tests.
  • Documents diagnostic behavior and raises the minimum supported Toolbox version to 3.9.
File Description
src/​test/​kotlin/​com/​coder/​toolbox/​views/​CoderSettingsPageTest.kt Tests buffer-size validation, saving, and reloading.
src/​test/​kotlin/​com/​coder/​toolbox/​views/​ActionTest.kt Updates logger construction.
src/​test/​kotlin/​com/​coder/​toolbox/​util/​ConnectionMonitoringServiceTest.kt Supplies logger settings.
src/​test/​kotlin/​com/​coder/​toolbox/​util/​CoderProtocolHandlerTest.kt Supplies logger settings.
src/​test/​kotlin/​com/​coder/​toolbox/​store/​CoderSettingsStoreTest.kt Tests buffer-size defaults and persistence.
src/​test/​kotlin/​com/​coder/​toolbox/​store/​CoderSettingsObservationTest.kt Tests settings observation and concurrent writes.
src/​test/​kotlin/​com/​coder/​toolbox/​session/​SessionIdRegistryTest.kt Updates logger settings and coroutine scope.
src/​test/​kotlin/​com/​coder/​toolbox/​sdk/​CoderRestClientTest.kt Models effective logging levels.
src/​test/​kotlin/​com/​coder/​toolbox/​feed/​IdeFeedManagerTest.kt Updates logger setup and coroutine scope.
src/​test/​kotlin/​com/​coder/​toolbox/​feed/​IdeFeedManagerOfflineTest.kt Updates logger setup and coroutine scope.
src/​test/​kotlin/​com/​coder/​toolbox/​diagnostics/​CoderSupportBundleCollectorTest.kt Tests replay before support export.
src/​test/​kotlin/​com/​coder/​toolbox/​diagnostics/​CoderLoggerTest.kt Supplies logger settings.
src/​test/​kotlin/​com/​coder/​toolbox/​diagnostics/​CoderLoggerFlightRecorderTest.kt Tests capture, resizing, replay, and triggers.
src/​test/​kotlin/​com/​coder/​toolbox/​CoderRemoteProviderTest.kt Supplies logger settings.
src/​test/​kotlin/​com/​coder/​toolbox/​CoderRemoteEnvironmentTest.kt Supplies logger settings.
src/​test/​kotlin/​com/​coder/​toolbox/​cli/​EnsureCLITest.kt Uses a concrete coroutine scope.
src/​main/​resources/​localization/​defaultMessages.po Adds buffer-setting interface text.
src/​main/​kotlin/​com/​coder/​toolbox/​views/​NewEnvironmentPage.kt Flushes on workspace-filter changes.
src/​main/​kotlin/​com/​coder/​toolbox/​views/​CoderSettingsPage.kt Exposes and validates buffer capacity.
src/​main/​kotlin/​com/​coder/​toolbox/​views/​CoderPage.kt Flushes after action failures.
src/​main/​kotlin/​com/​coder/​toolbox/​store/​StoreKeys.kt Adds the buffer-capacity persistence key.
src/​main/​kotlin/​com/​coder/​toolbox/​store/​CoderSettingsStore.kt Persists capacity and publishes settings changes.
src/​main/​kotlin/​com/​coder/​toolbox/​settings/​ReadOnlyCoderSettings.kt Defines observable settings and capacity limits.
src/​main/​kotlin/​com/​coder/​toolbox/​sdk/​CoderRestClient.kt Adds replay triggers for selected failures.
src/​main/​kotlin/​com/​coder/​toolbox/​diagnostics/​FlightRecorder.kt Implements ring buffering and batched replay.
src/​main/​kotlin/​com/​coder/​toolbox/​diagnostics/​CoderSupportBundleCollector.kt Flushes before collecting diagnostics.
src/​main/​kotlin/​com/​coder/​toolbox/​diagnostics/​CoderLogger.kt Captures string and lazy logs through the recorder.
src/​main/​kotlin/​com/​coder/​toolbox/​CoderToolboxExtension.kt Constructs the shared settings-aware logger.
src/​main/​kotlin/​com/​coder/​toolbox/​CoderToolboxContext.kt Reuses an existing Coder logger when provided.
src/​main/​kotlin/​com/​coder/​toolbox/​CoderRemoteProvider.kt Adds provider lifecycle replay triggers.
src/​main/​kotlin/​com/​coder/​toolbox/​CoderRemoteEnvironment.kt Adds environment lifecycle replay triggers.
README.md Documents configuration, replay, diagnostics, and compatibility.
gradle/​libs.versions.toml Upgrades the Toolbox plugin API.
CHANGELOG.md Records the feature and compatibility requirements.

Review effort: Balanced

Comment thread src/main/kotlin/com/coder/toolbox/diagnostics/FlightRecorder.kt Outdated
Comment thread src/main/kotlin/com/coder/toolbox/diagnostics/FlightRecorder.kt Outdated
textError ?: "",
ex,
)
context.logger.flush("action_failed $description", sessionId)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flight recorder’s automatic error-triggered flushes are intended for failures that affect one or more SSH connections. This is consistent with the VS Code extension’s approach to connection-failure replay and maybe @EhabY and @aqandrew can back me up on this.

This connect screen can happen one of the 4 scenarios:

  1. During fresh setup when there are no environments or SSH sessions yet, so an authentication or CLI setup failure cannot interrupt an existing connection.

  2. After logout when the plugin cleans up its state, removes the existing environments, and flushes the recorder before presenting the setup wizard again.

  3. When an expired API token requires signing in again, the same cleanup sequence (mentioned at 2) removes the existing environments and flushes diagnostics before returning to the wizard.

  4. When a URI targets deployment 2 while the plugin is already connected to deployment 1. The existing deployment stays active while the new setup runs. A failure during deployment 2’s authentication or CLI setup does not itself terminate deployment 1’s SSH connections. Once the new setup succeeds, the plugin cleans up the previous deployment and flushes its diagnostics before installing the replacement.

I don’t think this handler needs an additional flush for the intended SSH-connection scope.

The SSH process search logs full command lines for every visited descendant and for the matching process. These arguments can contain credentials, and the flight recorder can persist them during automatic replay at INFO.

This commit removes per-process command logging and limits the match diagnostic to the PID and workspace/agent name. A regression test covers token-bearing matching and unrelated processes and verifies that automatic disconnect replay retains the match diagnostic without exposing command arguments.
FlightRecorder captures timestamps before message preparation and insertion, so concurrent producers can replay entries with timestamps that disagree with their insertion order.

This commit captures timestamps when prepared entries are accepted into the buffer, under the insertion lock. Message suppliers and exception formatting remain outside the lock so their execution does not block other producers. This keeps the timestamp semantics in sync with Coder's VS Code extension, where timestamps also mark acceptance of the prepared entry.

The README documents acceptance order, and two-producer regression tests cover lazy evaluation, exception formatting, replay order, and capacity-one eviction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants