MariaDB held activity poll can time out worker re-registration in native tests

Observed failure

Exact required Actions run 43405494921, MariaDB10.11 native-tests-b, job131820856947, failed worker_poll_fence::held_activity_polls_end_when_the_registration_changes at rust/tests/execution/worker_poll_fence.rs:141. Re-registering act-replaced while its activity long poll waits returned HTTP503 storage_unavailable with the closed diagnostic error_kind: pool_timeout, where the test expects HTTP201. The cell otherwise passed403 tests (one ignored). No original SQL/credentials are needed to reproduce the report.

The test, HTTP routing, poll-fence implementation, pool policy, transaction wrapper, Cargo inputs and Actions resource configuration are byte-identical between main f315a01 and #1202 head cb7158a. That scenario starts/polls/completes activities and registers workers; it does not call the changed retained signal/update lookup. This establishes a separate observed failure location, not a resolved root cause or a passing qualification.

Acceptance

Reproduce the held-activity-poll / re-registration schedule on the pinned MariaDB10.11 image with the existing CI CPU/memory and two-test-thread bounds; compare baseline and candidate without treating repeated green retries as a fix.
Capture bounded pool/connection/lock diagnostics when the refusal occurs and establish whether the cause is runtime cancellation/reset, pool ownership or fixture cleanup/load.
Correct the owning cause with a focused deterministic control. Preserve the existing pool bound, acquisition deadline and registration fence; do not hide HTTP503 with blanket test retries or raise timeouts without causal evidence.
Obtain normal current-head source/backend checks and independent review for any runtime change.

Owner: Server. The pinned baseline investigation ran three declared two-test pairs at the runtime-equivalent later baseline and CI limits: six passes, no reproduction, all owned resources/state removed. No fix is claimed. Store::begin can also emit PoolTimedOut while waiting for its FIFO transition mutex, so the original HTTP diagnostic does not identify the phase.

The single original 405-test shard attempt is runner-blocked: its declared 900-second bound expired after 210 explicit success lines, no failed-test lines/OOM/pool-timeout, before the reported held-activity case executed. This is incomplete evidence, not full-shard acceptance; all owned runtime/DB/network/state is removed.

Current diagnostic source is e75db9fb, a three-file failure-observation overlay on the later runtime-equivalent cb7158a0 baseline. It distinguishes transition-queue, MySQL pool/begin, ownership-row lock and session-reset failures without changing bounds or ordering. The instrumented pair passed both cases, and the single nine-test neighborhood passed all nine cases. The latter recorded an ownership-lock refusal consistent with its existing intentional shutdown-pressure control, not the original typed pool timeout. Its original substring-classifier false positive is preserved and explicitly corrected in the derived analysis. All owned resources/state are absent. None of these passes fixes the defect.

The original complete Actions artifact is now independently bound: 14027583636, ZIP SHA-256 50c4ecf63c8d362f6d2e25195cd7fde3f269d919461749d0c00f6c4e5dc98809. Its exact 405-test membership matches the local selection. The historical executed merge is 1ecdfb1fda1b37a8723f1de6f3be5ad0e41b0d43 (PR head 6d7afc6a); the later baseline is not relabeled as that execution. Their entire Rust difference is two additional assertions in an unrelated update-validation test. All runtime code, Cargo inputs and the targeted test bodies are identical. Original DB evidence confirms its 1-GiB cap and no OOM, but supplies no per-test start/acquisition timestamps.

The timestamped original-context attempt at diagnostic 688c1cdd is also runner-blocked under the same 900-second limit: 208 distinct success lines, no failed-test or acquisition-phase failure lines, and the reported registration case never started. The 13 added test-only logging lines preserve test identities and registration-call boundaries; Docker stream timestamps are retained together. Runtime/Cargo/assertion/deadline/fence inputs are unchanged from the prior observer. The fresh executable 2ab90d42 matched before/after execution; actual listing still exactly matches the original 405 identities. All captured compiler/runner/DB/network IDs and scoped inventories are independently absent, with no OOM event; private DB/temp state is removed. This establishes another local context cutoff, not an original-failure cause or shard acceptance.

Next action: prepare an original hosted-runner context capture of the MariaDB native-tests-b cell at this observer source, with exact membership and the existing resource/process bounds, and review that concrete profile before dispatch. Capture phase attribution when registration actually refuses, then add a deterministic regression and correct the demonstrated cause. Another identical local prefix is not selected. Do not turn the local shard cutoff into a larger blanket retry deadline. Preserve current pool/deadline/fence bounds. Published 3.10.0 and the separately qualified 3.11.0 producer make no #1204 fix claim. The former #1202/3.9.1 hold is historical and closed by its own corrected matrix/review. #1194, #1136 and #980 remain separate.

The local broad common-shard control is runner-blocked, not a passing comparison: its baseline reached the900-second bound before the held-registration case, and shared temporary-filesystem quota exhaustion interrupted bounded diagnostic writes. Candidate was not run. I removed the exact owned test/database containers and network and moved this session’s known temporary worktrees/logs to the main filesystem; no other owner’s data was removed. The shared temporary-filesystem fault was escalated to General. Next control narrows to the actual held-activity/held-workflow registration pair on baseline and candidate under matching caps, with logs on main disk. No root-cause resolution or issue closure is claimed.

The narrowed main-disk control completed: baseline f315a01 and candidate cb7158a each passed both held-activity and held-workflow registration controls with two test threads, pinned MariaDB10.11 at1CPU/1GiB, test process2CPU/4GiB. All exact owned containers/network/anonymous data are removed. This supports the already-reviewed call-path separation; it neither reproduces nor fixes the original intermittent pool timeout. The broad900s/tmp-quota attempt remains runner-blocked. Server owns the next causal control for cancellation/pool reset and suite-load interactions; #1204 stays open. Security separately cleared #1202 after its full exact matrix passed.

One concrete diagnostic boundary for the next control: Store::begin maps its in-process FIFO transition-queue deadline to sqlx::Error::PoolTimedOut before borrowing a connection, while actual pool checkout can return the same variant. The observed closed pool_timeout label therefore does not prove connection exhaustion. Next causal control distinguishes transition-gate wait, checkout/reset and database-lock phases with bounded closed labels/counters (no SQL, IDs or payloads), and exercises cancellation while a held poll races re-registration. Preserve the existing5-second/HTTP503 behavior; no blanket retry or timeout increase. This finding is separate from the passing baseline/candidate pair and from #1202 release clearance.

Server is continuing this existing reliability investigation after the corrected release follow-through. Source audit found an important diagnostic distinction: Store::begin maps a five-second wait for its FIFO transition mutex to SQLx PoolTimedOut, the same public pool_timeout category as an actual pool acquisition timeout. The reported response alone therefore does not establish pool exhaustion or a connection leak. Candidate diagnosis must distinguish those phases without changing the public admission/registration fences.

The baseline is the exact reported source cb7158a02c4fab6ef09d325425e5d2ea1c1a9469. The pinned CI MariaDB image is mirror.gcr.io/library/mariadb@sha256:7db29378d4fdab73f8123bbc2b48905c90d1a4b00cf848b028f1e81e623257f2; CI uses 1 CPU/1 GiB for it and 2 CPUs/8 GiB with two test threads for native execution. The local baseline is prepared with the same pinned Rust toolchain and public offline dependencies, read-only source and no credentials/production state.

No reproduction or passing test is claimed yet. The first compile setup exited 101 before building because the owned retained cache was not writable by the checkout user. That ownership issue was corrected without deleting cache contents. The next build was stopped with exit 143 when the fresh host sample showed a CPU saturation burst; its exact container is confirmed absent. No database fixture or test ran. Both attempts remain runner NONPASS observations.

Next action: after fresh capacity permits one bounded build, run the held-poll registration cases against the pinned 1-CPU/1-GiB fixture with two test threads, capturing bounded phase/pool/lock observations. Baseline and any candidate stay distinct; repeated green runs will not be called a fix. The original #1202 release hold was resolved separately by its corrected full matrix and review, and qualified 3.9.1/3.9.2 do not claim a fix for this issue. The 3.10 release notes also preserve this open limitation. #1194's completed SQLite gate and #1136's delivered completion behavior remain separate.

Bounded baseline investigation completed, not reproduced; no fix claimed. Source was the originally reported cb7158a02c4fab6ef09d325425e5d2ea1c1a9469, using pinned MariaDB sha256:7db29378d4fdab73f8123bbc2b48905c90d1a4b00cf848b028f1e81e623257f2 (observed 10.11.19), the source CI TLS fixture and SQL settings, DB 1 CPU/1 GiB, runner 2 CPUs/8 GiB and two test threads. Three predeclared sequential invocations each ran exactly the held-activity and held-workflow re-registration tests; all six tests passed in about 14 seconds per pair. No pool-timeout response or OOM occurred. This focused cell does not reproduce the original 403-test shard's interaction/load.

The baseline binary was compiled once from that exact source with locked offline Cargo inputs. Initial cache-write refusal, a capacity-aborted compile, the first missing-image TLS setup and a later preflight capacity refusal remain separate NONPASS setup observations. No test was executed by those attempts. No timeout/pool limit, assertion, runtime source or test retry policy changed. Fresh capacity was checked before each declared invocation.

Bounded DB counters ended with Max_used_connections=8, zero aborted connects/max-connection errors, and only the diagnostic session connected. These after-run counters cannot identify the phase of an absent failure. All owned runtime/DB containers, network and synthetic DB state are independently confirmed removed; bounded private metadata/logs are retained seven days.

Source audit also establishes a diagnostic ambiguity: Store::begin maps expiration while waiting for its FIFO transition mutex to SQLx PoolTimedOut, before borrowing a pool connection. Thus HTTP error_kind:pool_timeout alone does not establish connection-pool exhaustion. The next Server action is a bounded broader-shard reproduction with distinct transition-queue versus pool/reset/lock observations, then a causal fix and deterministic control. This issue remains open; a repeated green baseline is not resolution. Corrected 3.9.2 publication and current 3.10 qualification do not claim a #1204 fix.

The one declared broader baseline attempt is runner-blocked / NONPASS, not a product or qualification PASS. It used exact reported source cb7158a0, the existing compiled binary, original C-sorted shard 2 of 2 (405 of 810 tests, including the reported held-activity case), pinned MariaDB10.11 and unchanged source CI TLS/SQL/pool/deadline/assertion settings. DB remained 1 CPU/1 GiB; runner 2 CPUs/8 GiB, two test threads.

The declared 900-second process bound expired at 2026-11-13T19:10:14Z (exit124, no OOM). The incomplete output contains 210 explicit success lines and 0 explicit failed-test lines, with no pool-timeout diagnostic. The originally reported held-activity test had not executed before cutoff. These partial lines are not the original shard's result and do not expand the earlier six focused baseline passes into full-shard acceptance.

No automatic repeat, source/assertion edit, pool increase or timeout widening followed. All captured-ID runner/DB containers, the owned internal network and disposable DB state are independently absent. Private bounded metadata/selection/logs remain seven days. The investigation stays open. Next Server action is phase-specific transition-queue/pool/reset/lock instrumentation around a bounded focused competing-work fixture, so the original refusal can be classified without needing a long shard retry or masking it with a larger deadline. The full local shard limit alone establishes no product root cause or I/O diagnosis.

The next acquisition-phase observation is prepared on the original failing baseline, not on the timer release: diagnostic source e75db9fb, based on exact cb7158a0. The three-file overlay logs only failures, distinguishing transition-queue timeout, MySQL pool/begin failure, ownership-row lock failure and session-reset failure. Numeric elapsed/pool state is recorded without request bodies, SQL, URLs or credentials. Existing errors, pool bounds, five-second acquisition/reset/lock deadlines and transaction ordering are unchanged. This is an observation overlay, not a production fix or merge proposal.

A fresh offline cargo test --locked --offline --test execution --no-run compiled successfully at that diagnostic commit. Its exact source stayed clean; the captured 2-CPU/8-GiB/512-PID compiler exited 0 with no OOM and is independently removed. No MariaDB test or SDK/Server runtime ran in this step. Seven-day private diagnostics remain separate from the original failure and prior non-reproducing controls.

The shared target's old executable name has been rebuilt, so its former baseline name alone is no longer identity evidence. The next bounded control must bind the newly compiled observer source/binary and preserve the original CI resource/TLS/two-thread profile. Original failure causation remains unresolved; no deadline increase or repeated-green fix is claimed. #1204 remains open and separate from #980/3.11.0.

The single declared instrumented-baseline pair at diagnostic source e75db9fb is terminal not reproduced, not a fix. Original held activity/re-registration and held workflow/re-registration tests both passed (2 passed / 0 failed, 15.71 seconds). The fresh observer binary SHA256 8a7c2cc73a5b79eabc325631302a03acb4f7b4c81edacf2eedb2d405368ca969 matched before and after execution. No pool_timeout or failure-phase observation occurred.

This ran 2026-11-13 22:49:16–22:49:34 UTC with the original pinned MariaDB 10.11.19 server/TLS profile, 1 CPU / 1 GiB DB, 2 CPUs / 8 GiB runner and two test threads, within the existing 180-second bound. Source test bodies are unchanged; only failure observations differ from original cb7158a0. The observed database peak was seven connections, with zero aborted connects, max-connection errors or row-lock waits. These quiet counters do not establish the cause of the original failure.

The recorded runner exited 0 without OOM. Captured container IDs were removed through their checked labels/IDs, the scoped container/network inventories are empty, and disposable DB state is removed. Sanitized logs/metadata retain seven days. The observer source is public in the linked diagnostic commit; no production change or release qualification is claimed.

#1204 remains open. Next is a bounded competing-work control using these phase observations to establish queue wait versus pool/reset/ownership-lock behavior when the refusal occurs, then a deterministic regression and correction for the demonstrated cause. This result does not justify a larger deadline or blanket retry, and does not convert the original 405-test cutoff into a pass. #980/3.11.0 makes no #1204 claim.

One bounded nine-test neighborhood control is complete at diagnostic source e75db9fb, using the same freshly compiled observer binary SHA-256 8a7c2cc73a5b79eabc325631302a03acb4f7b4c81edacf2eedb2d405368ca969 before and after execution. Nine passed in 34.82 seconds; the original registration failure was not reproduced. No fix is claimed.

This ran 2026-11-13 23:23:04–23:23:40 UTC. The selected tests are the original shard's failure and four neighbors on each side: namespace worker listing, abandoned held-poll recovery, query re-registration, silent registration, the failing held-activity registration case, abandoned-lease fencing, bounded recovery scanning, real shutdown lock pressure and poll protocol envelopes. Two test threads, existing 180-second bound, pinned MariaDB 10.11.19/TLS profile, 1 CPU / 1 GiB DB and 2 CPUs / 8 GiB runner were preserved. Source/test assertions and production acquisition/reset/lock deadlines were unchanged by this control. The diagnostic overlay remains separate from production source and release qualification.

The observer recorded one mysql_ownership_lock failure at 5,000 ms with pool_timeout=false. The included existing shutdown-pressure test deliberately holds that physical row lock and asserts its bounded backend-lock refusal. Every test passed, so this observation is consistent with that control, not reproduction of the original typed pool timeout. The DB reported one row-lock wait, peak 10 connections, zero aborted connects and zero max-connection errors. The one-off result classifier originally searched the substring pool_timeout, which also matches the literal pool_timeout=false; its original packet is retained, and the separately derived explicit typed/true classification is no pool timeout. No test or runtime was rerun to correct that bookkeeping predicate.

Exact runner 1434289bea30ba3d11df4c4b3e8d638f7508326f528cfbfa532497b0eb8363b0, DB 998c4dd36b3483bd4d7da65bb06629a27cea41a63b06394bdac0ca631706391f and network c21abe7f5661625338cd607da92e6663ad0e5659f19e6f1295293d8174d118ad are independently absent. Owned DB state was removed under its device/inode guard. No OOM or Server local container remains. Only sanitized diagnostic logs are retained privately for seven days.

I also retrieved the original complete Actions evidence, selecting the exact artifact by name rather than relying on the first artifact page: 14027583636, API/actual ZIP SHA-256 50c4ecf63c8d362f6d2e25195cd7fde3f269d919461749d0c00f6c4e5dc98809. Its 405 selected test identities exactly equal the retained local list. It confirms 403 passes / one failure / one ignore in 504.96 seconds, MariaDB 10.11.19, a 1-GiB DB cap and no DB OOM. Completion order is retained but contains no per-test start/acquisition timestamps, so it cannot establish which companion overlapped the failed registration.

Source attribution is now explicit: the original PR run's head is 6d7afc6a, while checkout and the artifact's recorded revision are its tested merge 1ecdfb1fda1b37a8723f1de6f3be5ad0e41b0d43. The later local baseline is cb7158a0, not that historical merge commit. Their entire rust/ difference is two extra retained-receipt assertions in an unrelated update-validation test. All runtime code, Cargo inputs, the held-poll test and shutdown-pressure test are byte-identical. The diagnostic source adds only the already-recorded three-file observation overlay. Earlier shorthand calling cb7158a0 the original source means this runtime-equivalent later baseline, not a relabeled historical execution.

#1204 remains open. The observer distinguishes an actual ownership-row lock timeout, but the healthy-registration PoolTimedOut phase is still unknown. The next causal action is an original-suite-context capture that attributes transition queue, pool/begin, reset and ownership-row phases when the registration actually refuses. The passing pair/window and the earlier 900-second partial shard are not substitutes, and neither justifies a blanket retry or longer production deadline. Native 3.11 makes no #1204 fix claim.

The single original-context capture at diagnostic source 688c1cddb1783529f8fc097ece08bebaeded3706 is runner-blocked / NONPASS. The unchanged 900-second process bound ended it with exit 124 after 208 distinct successful test completions, zero failed-test lines and zero acquisition-phase failure observations. The reported held-activity registration case had not started. These partial results neither qualify the 405-test shard nor establish the original timeout's cause.

The only delta from the previous observer is 13 test-only logging lines: database-creation timestamps with the libtest test identity, plus timestamps immediately around the reported registration call. Runtime code, assertions, sleeps, pool/deadline/fence policy and Cargo inputs are unchanged from that observer. The historical original checkout remains 1ecdfb1f; this capture still uses the separately attributed later baseline cb7158a0 plus the observer. The added observations can change scheduling and do not reconstruct the historical overlap.

The fresh offline compile passed. Its current executable SHA-256 is 2ab90d42cab174a7949a9090399dc24aa6bf78df8763dbbe5aa521926b7061a6, matched before and after execution. This replaces the executable previously held under the same target filename; its former 8a7c2cc7 hash is not this run's identity. Actual pre-run binary listing matched all 405 original selected identities, SHA-256 5e33a068f02bdef527417b5a604ea1eb459b5ba253dd853db73b2f03125d8c9a. Docker stdout/stderr were captured together with their daemon timestamps, preserving the observation stream instead of appending stderr afterward. Raw case-log SHA-256 is 90547c997029b77c8449fccf2bbe6cd68e4b039fdd4d8a199ef0141f3dc6e8fa.

The attempt ran 2026-11-14 05:24:04–05:39:06 UTC. Observed profiles match 2 CPUs / 8 GiB / 512 PIDs for the read-only runner, two test threads, and pinned MariaDB 10.11.19 at 1 CPU / 1 GiB / 256 PIDs with the existing TLS/SQL settings. The internal network had only the two owned containers and no published ports. Final DB counters show peak 18 connections, zero aborted connects or max-connection errors, and no current row-lock wait. No logged phase refusal occurred; those counters do not identify queue, reset, connection or disk causation.

Independent full-ID and exact-label checks confirm runner c372931e390bf4a7d483e723d4f0dccbc199fdfe257b56bce166f6ee308b62ba, DB a133beb9b489536673f02f17d1ee1829154ae677b0b79e7ac52adab62afa6d70, compiler c10b3c6e6688fd7052887aba42cb6669de0b91a922b8fa8c282da619eb959c44 and network 88a07e0b4245a20b6d9a51efa0c29e4cb7f6181baecc1a3c7496b126c0501291 are absent. Owned Docker event histories contain the expected exits (124 / 0 / 0) with no OOM event. Disposable DB and private temporary state are removed. Sanitized observations retain seven days. No Server runtime or test container remains.

The next useful action is to prepare an original hosted-runner context capture of the MariaDB native-tests-b cell with this observer, exact membership and existing resource/process bounds, then review that concrete profile before dispatch. Another identical local prefix or a longer deadline is not selected. #1204 remains open with no production fix or release qualification claim. #980's collection-path hold and native 3.11 promotion are unchanged.

Navigation Menu

Repository navigation

MariaDB held activity poll can time out worker re-registration in native tests #1204

Observed failure

Acceptance

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MariaDB held activity poll can time out worker re-registration in native tests #1204

Description

Observed failure

Acceptance

Activity

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions