If we are using MSG WAITALL, Do we need the flush logic on short flushing?

[MDA2AV]

investigate whether we haev redundant logic to deal with short flushes while MSG WAITALL is active

[MDA2AV]

Investigated. The logic is not redundant — it is reachable with MSG_WAITALL on, and it is the only correct path for kTLS. Recommend keeping it.

Measured on 6.14.0-37-generic by counting every send CQE and every resubmit through the partial branch, driving Playground/Tcp/Big at an 8 MiB body, 4 reactors, wrk -t8 -c64 -d12s.

Is the branch taken under load?

send shape	send CQEs	partial resubmits
plain SEND + MSG_WAITALL	25,873	0
plain SEND, flag cleared	29,060	3,196 (11.0%)
SEND_ZC + MSG_WAITALL	27,890	0
SENDMSG (segmented) + MSG_WAITALL	30,548	0

So io_uring honours MSG_WAITALL for all three shapes we submit, not just plain SEND — worth knowing, since SubmitSendMsg and the SEND_ZC branch pass the same flags and neither is obviously covered. Under ordinary load the branch is dead code for a flagged connection. The cleared-flag row is the control: the workload genuinely produces short sends, 11% of completions.

But it is still reachable with the flag on

A peer that resets mid-body, 1,200 connections (read 64 KB, then SO_LINGER {1,0} + close):

send-cqes=2 partial-resubmits=1 send-errors=1
| all-cqes=402 stale=399 stale-with-bytes=399

402 send CQEs arrived. 399 were discarded by the stale guard because the recv side had already torn the connection down and cleared the table slot — and all 399 carried res > 0, a short byte count rather than an error. So MSG_WAITALL does not convert a mid-transfer failure into an error return; io_uring reports the bytes it managed to transfer, exactly as a bare send() would.

Of the three CQEs that reached a still-live connection, one took the partial branch. With the flag on.

That is the case that makes deleting it unsafe: without the resubmit, a short send on a live connection falls through to CompleteFlush() with WriteHead < WriteInFlight, which declares a truncated response fully sent. Rare, silent, and a correctness bug rather than a performance one.

kTLS does not use the flag at all

TlsService.cs:1210 clears SendOpFlags after the kTLS TX handoff, and says why:

kTLS rejects MSG_WAITALL, so switch this connection's sends to plain (the reactor still loops on short sends).

Every kTLS-TX connection therefore runs the no-flag column above. Under the same RST abuse with the flag cleared, the branch ran on 99.74% of completions (391 of 392). For those connections the loop is not a fallback, it is the mechanism.

Caveats

• One kernel, 6.14. The README floor is 6.1, and I could not test whether io_uring honours MSG_WAITALL for send there. If it does not, the flag is silently ignored on supported kernels and the loop is load-bearing for every connection, not just kTLS — which is a third reason not to remove it, and worth confirming before anyone revisits this.
• The kTLS conclusion rests on the code path plus the flag-cleared measurements; the tls module is not loaded on this box, so no kTLS connection was driven directly.

Related

Benchmarked clearing the flag entirely (#230) while I was here: no measurable throughput cost at 2 B, 64 KiB or 1 MiB bodies — every delta inside the run-to-run spread. So if the flag is ever dropped to get per-chunk send progress, this logic becomes the whole send path rather than something to delete.