feat: truth under load — coalescing links, grace, ESI, restarts, pipe (tracing P3)

[jensens]

Summary

P3 of OpenTelemetry tracing, per the (herein synced) design spec: the traces tell the truth under load.

• Truth fixtures first: seven new VSL recordings from real varnish:8.0.2 (coalesce, grace/bgfetch, esi, restart, retry, pipe, synth) via an upgraded rig with a traffic-shaping backend, catalogued in testdata/NOTES.md — every span-builder change was written against recorded bytes, and the fixtures corrected the plan more than once (restart is a cross-transaction affair; ESI subrequests get no traceparent propagation; Hit payloads come in 4- and 6-field shapes).
• Recursive span building: fetch spans per retry attempt (nested under the first attempt's span), bgfetch children, ESI child request spans carrying the parent's trace through the tree (varnish.minted_trace_mismatch where the wire couldn't), restart counting with an honest end-timestamp fallback, statuses via last-record truth.
• The Linker (bounded LRU+TTL fetch-identity cache, single-goroutine): every hit links to the fetch that created its object; varnish.coalesced=true on genuine time overlap; restarted requests are stitched across top-level VSL groups into the original's trace (varnish.restart_continuation); every unresolvable correlation is a visible attr + counter, never silence. Skip counters split: unsampled vs genuinely unusable; pipe no longer counts as data loss.
• Links and timings on the wire: through the ReadOnlySpan adapter, the E2E sink (links + start/end nanos), and vinylprobe (-link-span-id, -min-duration).
• E2E (suite: full, this PR carries e2e-full): coalescing (two backgrounded probes on one slow cold URL, asserted via varnish.coalesced with a 3-round fresh-path retry against the VSL group-emission ordering race), grace/bgfetch (short-TTL traffic poll), and the spec's long-promised honest-timing assertion (-min-duration 1500ms on a 2s-delayed fetch).
• Two pre-existing defects flushed out: import esi; referenced a vmod that does not exist in varnish 8 (removed; proven via varnishd -C; ESI is core via the new do_esi-on-Surrogate-Control gate) — and varnishParameters never reaches varnishd -p at all, which defers the ESI E2E to varnishParameters never reach varnishd, and the ESI import gate referenced a nonexistent vmod #109. The restart-then-miss mismatch-flag refinement is Tracer: restart-then-miss leaves backend spans orphaned without the minted_trace_mismatch flag #110.

Verification

Full battery green at HEAD: lint, unit (incl. 8 Linker tests, race-clean), envtest, helm-unittest 59/59, both boundary/label checks, zero generated-file drift. Every task passed spec+quality review; the final whole-branch review's findings are fixed in the last two commits and re-reviewed (the coalescing retry loop was executed under both bash and dash against mocked commands).

Note for reviewers

The tracing-truth suite runs here for the first time (needs this PR's e2e-full label). The final review recommends letting the full suite run 2–3 times before merging — the coalescing scenario is inherently racy and the retry is new. Budget arithmetic is documented inside each step.

Assisted-by: Claude Fable 5

[jensens]

Update: the coalescing race is now fixed in the tracer, not just retried around it

The first e2e-full run failed the coalescing assertion in all three retry rounds. Investigation (local rig reproducing the E2E's exact traffic shape, 10 rounds against real varnish 8.0.2, then the real Linker replayed over each recording):

• Coalescing itself worked every time: 10/10 rounds produced exactly one fetch for two concurrent requests.
• varnishlog -g request emitted the initiator's group first in 5/10 rounds and the waiter's first in the other 5 — a clean coin flip.
• The Linker matched that split exactly: initiator-first → varnish.coalesced=true; waiter-first → varnish.coalesced_unlinked, permanently, because the hit was resolved against the fetch cache the instant its group arrived.

So three independent rounds failing was ~12.5% bad luck on a real 50/50, not a new bug — but retrying around a coin flip is not truth. The Linker now carries a bounded parking lot: a coalescing waiter whose originating fetch is not yet known is parked (keyed by that fetch's vxid) instead of being emitted degraded, and is released with its link and varnish.coalesced as soon as the initiator's group arrives. Both orderings now produce identical spans.

Bounds and honesty, because a parking lot that can grow or forget silently would be worse than the race:

• Parking is gated on the waiter's own Timestamp Waitinglist: record, so only genuine race waiters park. An ordinary warm hit whose origin fetch has aged out of the cache degrades immediately, exactly as before — it is not a race and must not be delayed.
• 256 sets max (overflow releases the oldest, degraded), 10s age limit, and a Flush() on shutdown so nothing is dropped when the process exits. Every release-without-resolution still sets varnish.coalesced_unlinked and counts vinyl_tracer_link_cache_misses_total; vinyl_tracer_hits_parked_total counts the race itself.
• The batcher now runs on its own context, cancelled only after the supervisor returns and the flush is enqueued. Without that, the shutdown flush landed in a channel nobody was reading any more — the spans it exists to save were being lost, silently, without even incrementing the drop counter.

The E2E retry loop stays as belt-and-braces for residual pathologies, and its description now says what the linker guarantees.

Known residual: cmd/tracer/main.go has no testable seam, so the new shutdown regression test guards the pattern rather than main's own wiring. Extracting main's body into a run(ctx) error would close that; it is not part of this PR.

Assisted-by: Claude Fable 5