FEATURE-ci-darwin-header-deadline: Defer native diagnostic output until suite shutdown

Capture bounded typed metadata with per-event monotonic timestamps in one compiled-library buffer, preserving errno and ordinary-build elision. Flush only after all integration fixtures stop, and record notify/signal/ack state for the observed no-deadline handoff. Keep the same exact archive overlay, assertions and budgets. Native live tracing passed without establishing a cause; this remains a diagnostic checkpoint. RED/GREEN deferred-output contract, enabled/disabled shared-library checks, actual archive overlay, extracted-source syntax, workflow syntax and changed-file lint passed. Await one reviewed native execution before any behavior repair.

Author: etr

.github/workflows/v3-package-validation.yml

@@ -115,6 +115,7 @@ jobs:
                      CPPFLAGS='-DHTTPSERVER_DIAG_HEADER_DEADLINE',
                      LDFLAGS='-Wl,-rpath,' + str(provider / 'lib'))
           receipt = dict(kind='diagnostic-only; not complete package validation', tls='no', linkage='shared',
+                         trace_mode='bounded memory; stderr emitted after all suite fixtures stop',
                          revision=command(['git', 'rev-parse', 'HEAD'], checkout, env).strip(),
                          compiler=command([compiler, '--version'], checkout, env),
                          provider_header_sha256=hashlib.sha256((provider / 'include/openssl/opensslv.h').read_bytes()).hexdigest())

specs/tasks/FEATURE-ci-darwin-header-deadline.md

@@ -31,6 +31,26 @@ LLVM toolchain and `-O0 -g`. Checkout versus extracted archive/execution context
 is now an observed distinction; it is not an established cause. The CA issue is
 queued separately and is outside this task.
 
+## First native diagnostic result and remaining uncertainty
+
+Native diagnostic run `39974555288`, job `120485650697`, passes all four existing
+cases with zero failures using the exact c0 archive plus the live stderr trace
+overlay. The header case takes 303.347 ms. Its actual timer sequence 8 fires,
+applies result 0, resumes, matches the current deadline, enforces timeout,
+releases the connection and closes the native socket successfully near 300 ms.
+A snapshot with no deadline occurs between timer application and watchdog
+resumption; a later host wait receives a wake. The original f91/c0 untraced
+archive failures remain unexplained. Passing under tracing is neither a fix nor
+proof that the failure is independent of this path.
+
+Source review of that handoff finds registration retirement/reset under the
+backend mutex, followed by notify; notify and wake acknowledgment share that
+mutex. Snapshots borrow handles; dispatch-held leases end before the host returns
+to waiting. No violated race invariant or decisive forced-interleaving regression
+was established from this successful trace. The next checkpoint reduces trace
+perturbation and records actual notify/signal/ack states to distinguish missing
+wake from retained lease when a failure is observed.
+
 ## Action items
 
 - [x] Preserve and read prior/current evidence without repeating the closed Linux probe.
@@ -42,20 +62,30 @@ queued separately and is outside this task.
   jobs explicitly on manual dispatch.
 - [x] Check enabled/disabled trace compilation, bounded output, errno preservation,
   ordinary-build argument elision, workflow syntax and changed-file lint.
-- [ ] Execute the native diagnostic and retain actual installed-library provenance.
+- [x] Execute the first native diagnostic and retain actual installed-library provenance.
+- [x] Prepare bounded in-memory capture with deferred output and explicit event timestamps.
+- [ ] Execute one reviewed buffered-trace native diagnostic against the same exact archive.
 - [ ] Establish cause from native evidence, then write and observe a failing
   regression before applying a narrow repair.
 - [ ] Remove all temporary trace code/header and workflow edits before final merge.
 - [ ] Run task-relevant final checks and Groundwork validation/finalization.
 
 ## Checkpoint scope and checks
 
-`HTTPSERVER_DIAG_HEADER_DEADLINE` enables temporary stderr records, capped at 64
-per call site. Correlation uses connection IDs, operation pointers/sequences,
-socket descriptors and registration generations. Time points are converted with
-`duration_cast<nanoseconds>`; header durations are explicit milliseconds. No
-request bytes, credentials or peer content are logged. Ordinary builds erase
-trace argument evaluation.
+`HTTPSERVER_DIAG_HEADER_DEADLINE` enables temporary typed metadata capture,
+capped at 64 records per call site and 2,048 records overall. Each record stores
+an explicit monotonic `event_ns`, a static format and copied scalar/pointer
+values in fixed storage in the compiled library. Capture performs no formatting,
+stderr writes, allocation or application mutex locking. The installed test and
+library use that same buffer. Only after all four test fixtures have stopped and
+the suite has completed does the test call flush to format and emit stderr.
+Correlation uses connection IDs, operation pointers/sequences, socket descriptors
+and registration generations; new notify/signal/ack metadata covers the observed
+wake/retirement handoff. Time points use `duration_cast<nanoseconds>`; header
+durations are explicit milliseconds. No request bytes, credentials or peer
+content are retained. Capture and flush preserve errno; ordinary builds erase
+trace argument evaluation, storage and diagnostic symbols. A discarded compile
+branch retains format checking without executing stderr writes during capture.
 
 Manual dispatch runs macOS only, preserving its selected compiler/runtime,
 provider setup and artifact upload. It builds and installs TLS-off libraries at
@@ -96,7 +126,16 @@ workflow extraction/overlay block passes, all eight original overlay files match
 the exact base, and the private trace header is present. All eight affected
 translation units also pass diagnostic-enabled syntax/format checks using only
 the extracted source/include paths. Workflow syntax passes again; only workflow provenance and this task record change in the follow-up.
-Darwin execution and the original required gate remain incomplete.
+The buffered capture contract was tested RED/GREEN: live stderr output first
+fails the deferred-output assertion; the new buffer passes. Enabled/disabled
+checks link the actual io_operation implementation as a shared library and prove
+no output before flush, 64/site caps across concurrent producers, errno
+preservation, typed field formatting, positive per-event timestamps, one buffer
+shared by executable/library events, and no diagnostic symbols or argument
+evaluation in ordinary builds. The updated literal archive overlay and
+workflow/test-preservation checks pass; both enabled/disabled extracted-source
+syntax checks and changed-file lint pass. Buffered native execution and the
+original required gate remain incomplete.
 
 The caller reviews/publishes only this task branch and dispatches the existing
 workflow, then resumes this same worktree with actual native evidence. No root

src/detail/io_external_adapter.cpp

@@ -94,10 +94,13 @@ void io_poll_backend::notify() {
     bool failed = false;
     {
         std::lock_guard<std::mutex> lock(mu_);
+        V3_DEADLINE_TRACE("notify backend=%p pending=%d failed=%d\n", static_cast<void*>(this), wake_pending_, wake_failed_);
         if (wake_failed_ || wake_pending_) return;
         if (wake_.signal()) {
+            V3_DEADLINE_TRACE("signal backend=%p result=1\n", static_cast<void*>(this));
             wake_pending_ = true;
         } else {
+            V3_DEADLINE_TRACE("signal backend=%p result=0\n", static_cast<void*>(this));
             wake_failed_ = true;
             active_ = false;
             failed = true;
@@ -109,8 +112,10 @@ void io_poll_backend::notify() {
 }
 
 void io_poll_backend::acknowledge_wake() {
+    V3_DEADLINE_TRACE("acknowledge backend=%p pending=%d failed=%d\n", static_cast<void*>(this), wake_pending_, wake_failed_);
     if (!wake_.acknowledge()) wake_failed_ = true;
     wake_pending_ = false;
+    V3_DEADLINE_TRACE("acknowledged backend=%p failed=%d\n", static_cast<void*>(this), wake_failed_);
 }
 
 server::interest_snapshot io_poll_backend::interests() const {

src/detail/io_operation.cpp

@@ -27,12 +27,45 @@
 
 #include "httpserver/detail/io_operation.hpp"
 
+#if defined(HTTPSERVER_DIAG_HEADER_DEADLINE)
+#include <algorithm>
+#include <array>
+#endif
 #include <exception>
 #include <memory>
 #include <utility>
 
 #include <httpserver/detail/deadline_trace.hpp>
 
+#if defined(HTTPSERVER_DIAG_HEADER_DEADLINE)
+namespace httpserver::detail {
+namespace {
+constinit std::array<deadline_trace_record, 2048> deadline_records{};
+constinit std::atomic<std::size_t> deadline_record_count{0};
+}  // namespace
+
+deadline_trace_record* deadline_trace_reserve(long long monotonic_ns) noexcept {  // NOLINT(runtime/int)
+    const auto index = deadline_record_count.fetch_add(1, std::memory_order_relaxed);
+    if (index >= deadline_records.size()) return nullptr;
+    auto& record = deadline_records[index];
+    record.monotonic_ns = monotonic_ns;
+    return &record;
+}
+
+void deadline_trace_flush() noexcept {
+    const int saved_errno = errno;
+    const auto count = std::min(deadline_record_count.load(std::memory_order_relaxed), deadline_records.size());
+    for (std::size_t index = 0; index < count; ++index) {
+        const auto& record = deadline_records[index];
+        if (!record.ready.load(std::memory_order_acquire)) continue;
+        std::fprintf(stderr, "V3_DEADLINE event_ns=%lld ", record.monotonic_ns);  // NOLINT(build/include_what_you_use)
+        record.write(record);
+    }
+    errno = saved_errno;
+}
+}  // namespace httpserver::detail
+#endif
+
 namespace httpserver {
 namespace detail {
 

src/httpserver/detail/deadline_trace.hpp

@@ -2,26 +2,66 @@
 #ifndef SRC_HTTPSERVER_DETAIL_DEADLINE_TRACE_HPP_
 #define SRC_HTTPSERVER_DETAIL_DEADLINE_TRACE_HPP_
 
-// Temporary native Darwin diagnostic; each call site emits at most 64 records.
+// Temporary native Darwin diagnostic; each site captures at most 64 records.
 #if defined(HTTPSERVER_DIAG_HEADER_DEADLINE)
 #include <atomic>
 #include <cerrno>
 #include <chrono>
+#include <cstddef>
 #include <cstdio>
+#include <new>
+#include <tuple>
+#include <type_traits>
 
 namespace httpserver::detail {
 using deadline_trace_id = unsigned long long;  // NOLINT(runtime/int)
 inline long long deadline_trace_ms(std::chrono::milliseconds time) { return time.count(); }  // NOLINT(runtime/int)
 inline long long deadline_trace_ns(std::chrono::steady_clock::time_point time) {  // NOLINT(runtime/int)
     return std::chrono::duration_cast<std::chrono::nanoseconds>(time.time_since_epoch()).count();
 }
+
+struct deadline_trace_record {
+    long long monotonic_ns = 0;  // NOLINT(runtime/int)
+    void (*write)(const deadline_trace_record&) = nullptr;
+    alignas(std::max_align_t) std::byte payload[128]{};
+    std::atomic<bool> ready{false};
+};
+
+// One buffer in the compiled library also receives the installed test's events.
+deadline_trace_record* deadline_trace_reserve(long long monotonic_ns) noexcept;  // NOLINT(runtime/int)
+void deadline_trace_flush() noexcept;  // Call only after all producers have stopped.
+
+template<typename... Args>
+void deadline_trace_capture(long long monotonic_ns, const char* format, Args... values) noexcept {  // NOLINT(runtime/int)
+    struct payload_type {
+        const char* format;
+        std::tuple<Args...> values;  // NOLINT(build/include_what_you_use)
+    };
+    static_assert(sizeof(payload_type) <= sizeof(deadline_trace_record::payload));
+    static_assert(alignof(payload_type) <= alignof(std::max_align_t));
+    static_assert(std::is_trivially_destructible_v<payload_type>);
+    if (auto* record = deadline_trace_reserve(monotonic_ns)) {
+        new (record->payload) payload_type{format, {values...}};
+        record->write = [](const deadline_trace_record& entry) {
+            const auto& payload = *std::launder(reinterpret_cast<const payload_type*>(entry.payload));
+            if constexpr (sizeof...(Args) == 0) {
+                std::fputs(payload.format, stderr);  // NOLINT(build/include_what_you_use)
+            } else {
+                std::apply([&](auto... args) { std::fprintf(stderr, payload.format, args...); }, payload.values);  // NOLINT(build/include_what_you_use)
+            }
+        };
+        record->ready.store(true, std::memory_order_release);
+    }
+}
 }  // namespace httpserver::detail
 
 #define V3_DEADLINE_TRACE(...) do { \
     static std::atomic<unsigned> trace_count{0}; \
     if (trace_count.fetch_add(1, std::memory_order_relaxed) < 64) { \
         const int trace_errno = errno; \
-        std::fprintf(stderr, "V3_DEADLINE " __VA_ARGS__); /* NOLINT(build/include_what_you_use) */ \
+        const auto trace_ns = httpserver::detail::deadline_trace_ns(std::chrono::steady_clock::now()); \
+        if constexpr (false) { std::fprintf(stderr, __VA_ARGS__); } /* NOLINT(build/include_what_you_use) */ \
+        httpserver::detail::deadline_trace_capture(trace_ns, __VA_ARGS__); \
         errno = trace_errno; \
     } \
 } while (false)

test/integ/external_http1_e2e_test.cpp

@@ -276,4 +276,7 @@ LT_END_AUTO_TEST(reconnect_replayed_callbacks_and_host_driven_drain)
 
 LT_BEGIN_AUTO_TEST_ENV()
     AUTORUN_TESTS()
+#if defined(HTTPSERVER_DIAG_HEADER_DEADLINE)
+    httpserver::detail::deadline_trace_flush();
+#endif
 LT_END_AUTO_TEST_ENV()