Skip to content

tls: teardown skips its flush when the application left one in flight - #235

Merged
MDA2AV merged 1 commit into
mainfrom
fix/tls-disposal-flush-guard
Sep 19, 2026
Merged

MDA2AV merged 1 commit into
mainfrom
fix/tls-disposal-flush-guard

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Sep 19, 2026

Copy link
Copy Markdown
Owner

Fixes #234.

What was happening

TlsConnectionDualPipe.DisposeAsync flushes the connection unconditionally:

_writer.Complete();
await _conn.FlushAsync();

That is load-bearing — a handler that wrote its response and never flushed has nothing else to carry it out (WriterContractTests covers exactly that shape). But against a flush the application left in flight, it met the one-flush-at-a-time guard in TcpConnection.FlushAsync and threw out of a finally, arriving as:

[r0] connection handler faulted: FlushAsync already in progress.

The _closed early-out cannot save it: the read side is what calls MarkClosed(), and it runs in the finally after this flush. TLS-only, as reported — TcpConnectionDualPipe has no disposal at all, and HopDuplexPipe drains its send pump before closing.

The call, and why it went this way

The guard is not stylistic and it stays. The parked caller waits on _flushSignal, one ManualResetValueTaskSourceCore with room for a single continuation — a second flush is not something the connection can serve, and an application flushing twice concurrently has a bug it needs to hear about.

What was wrong is the caller. Disposal is the connection's own flush, not the application's, so it should not be tripping a guard aimed at application misuse. FlushAsync keeps the throw; FlushIfIdleAsync returns instead, and disposal calls that.

Skipping sends nothing because there is nothing to send — a property of the write path, not a hope. While _flushInProgress is set every GetSpan/GetMemory/Advance/Write is refused (TcpConnection.Write.cs), so nothing can have entered the slab since the in-flight flush armed, and that flush snapshotted everything already there. The in-flight send owns the whole slab; a second flush would compute target == 0 and return anyway.

Considered and rejected: coalescing — returning the outstanding ValueTask to the second caller — which cannot work, since that hands a second awaiter the same token and the source throws on the second OnCompleted. And quiescing (waiting the in-flight flush out before disposal's own) is the only ordering that would actually deliver bytes staged during it, but the thing that releases a flush parked on a peer that stopped draining is the MarkClosed underneath it, so an unbounded wait turns this fault into a teardown hang.

Test

tls writer: disposal does not throw while the connection's flush is in flight parks a real flush the way the existing contract test does — a peer that stops draining, 256 KB at a time — and stages nothing before tearing down. That is the production shape: every application write was flushed, so Complete has nothing to commit and the disposal's own flush is the only call left that can throw. It asserts non-vacuity both ways (a flush genuinely in flight, and nothing staged).

Proven by reverting the fix:

FAIL  tls writer: disposal does not throw while the connection's flush is in flight:
      disposal threw against a flush the application left in flight:
      InvalidOperationException: FlushAsync already in progress.
141 passed, 1 failed, 5 skipped, 7 pending

With it: 142 passed, 0 failed, 5 skipped, 7 pending. Unit, Http, Chaos, File and E2E green. Tls/OpenSsl -0.6%, Tls/OpenSslPipes +1.6% at 2 reactors — inside the noise floor, as expected for a teardown-path change.

Not covered

The same disposal still throws a line earlier, out of Complete, when plaintext is staged: the commit reaches TcpConnection.GetSpan, which refuses it, and Complete catches only IOException. That is already tracked as the pending test directly above the new one, and it wants its own decision — those bytes cannot be delivered from that state regardless, since anything appended to the slab during an in-flight flush is zeroed by CompleteFlush, so the choice is to drop them quietly or to quiesce first and accept the hang risk. Left out of this PR deliberately.

TlsConnectionDualPipe.DisposeAsync flushes the connection unconditionally,
because a handler that wrote its response and never flushed has nothing else
to carry it out. Against a flush the application left in flight that call met
TcpConnection.FlushAsync's one-flush-at-a-time guard and threw out of a
finally, surfacing as "[r0] connection handler faulted: FlushAsync already in
progress." - reported on a long-lived TLS websocket, a few times per run (#234).

The guard is right and stays. The parked caller waits on _flushSignal, a single
ManualResetValueTaskSourceCore that holds one continuation, so a second flush
is not something the connection can serve; an application that flushes twice
concurrently needs to hear about it. What is wrong is the caller: disposal is
the connection's own flush, not the application's, and it has no business
tripping a guard aimed at application misuse.

So FlushAsync keeps the throw and grows FlushIfIdleAsync beside it, which
returns instead. Skipping sends nothing because there is nothing to send: while
_flushInProgress is set every GetSpan/GetMemory/Advance/Write is refused, so
nothing can have entered the slab since the in-flight flush armed, and that
flush snapshotted everything already there. The in-flight send owns the whole
slab; a second flush would compute target == 0 and return anyway.

The regression test parks a real flush the way the existing contract test does
- a peer that stops draining - and stages nothing before tearing down, which is
the production shape: every application write was flushed, so Complete has
nothing to commit and the disposal's own flush is the only call left that can
throw. Without the fix it fails with the reported message.

Not covered: the same disposal throws a line earlier, out of Complete, when
plaintext IS staged - GetSpan refuses it and Complete catches only IOException.
That one is already tracked as the pending test above this one, and needs its
own call about whether those bytes are dropped or waited for, since anything
appended to the slab during an in-flight flush is zeroed by CompleteFlush.

TLS suite 142 passed / 0 failed / 7 pending; Unit, Http, Chaos, File and E2E
green. Tls/OpenSsl -0.6%, Tls/OpenSslPipes +1.6% at 2 reactors - inside noise.
@MDA2AV
MDA2AV merged commit 0ec7bcf into main Sep 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TLS teardown flushes into an in-flight application flush and throws "FlushAsync already in progress"

1 participant