Log why the 9p transport went away - #5
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the transport dies, every in-flight request is failed with
EIOand the reason is discarded:A clean end of stream isn't distinguished from an error, and neither is logged. So a caller sees an unexplained
EIOand there is nothing on the client side saying the transport went away, let alone why.That isn't hypothetical. In codedown CI this surfaces as an editor read failing with
hGetBufSome: hardware fault (I/O error), reported to the user as "is it a valid Jupyter notebook?". Establishing that it was a transport death took ruling out the 9p server from its own logs — diod recorded no read errors at all, only laterclunk: invalid fidcomplaints — and then reasoning backwards throughtransact's fourEIOpaths. The client knew the answer at the time and threw it away.The change
Distinguish the two ways the pump can end, and log before clearing the waiters:
Some(Err(e))->transport error: {e}None->peer closed the streamThen log at
errorwith the reason, the number of in-flight requests, and what they were — T-message name, tag and age, the same information the stall watchdog already reports. If nothing was in flight it logs atwarninstead, since no caller is about to see anEIO.Logging happens before
pending.clear(), so the in-flight list is still there to report.No behaviour change: the same waiters are dropped and
transport_goneis still signalled.Note
This makes the failure diagnosable, it doesn't make it recover. Worth considering separately: a request outstanding when the transport dies could be retried on the reconnected session rather than failed, so callers don't have to treat
EIOas normal. That's a bigger change — 9p fids are per-connection, so a retry has to re-walk rather than resend, and theclunk: invalid fidnoise suggests stale fids already cross reconnects today.