Skip to content

Flaky test: TestLinuxBindingConcurrentMarkerTransition — "binding file state changed before return" #833

Description

@HuggeK

TestLinuxBindingConcurrentMarkerTransition (go/internal/gatewayidentity/binding_test.go) fails intermittently in CI with:

--- FAIL: TestLinuxBindingConcurrentMarkerTransition (0.00s)
    binding_test.go:1274: binding file state changed before return
FAIL	github.com/srcfl/ftw/go/internal/gatewayidentity	0.074s

Two independent occurrences, same test, same message:

So the trigger is not any recent commit — the test races against itself.

What the test asserts

Eight goroutines each open the binding storage and perform the same
ReplaceExact("state", "pending" → "active") transition; the test requires
all eight to return success (idempotent CAS: losers must observe the
winner's value and treat the transition as already done). The failure message
comes from the loser path — a worker saw the marker change under it in a
window where the implementation reports an error instead of idempotent
success.

The window is small (fails well under 1 in 5 runs on CI, -count=5 locally on
Linux was not attempted here — the test skips on other platforms), which is
what makes it a recurring red herring: it lands on whichever PR happens to be
in CI, and the PR author debugs their own diff first.

Why this deserves a fix rather than a retry loop

The test encodes a real contract: concurrent first-boot marker transitions
must converge without error, because two processes racing the same
pending → active flip is exactly what happens when a supervisor restarts the
binary mid-transition. If ReplaceExact can report "state changed before
return" for a transition that in fact completed, the caller cannot distinguish
"someone else finished my work" from "my work was clobbered".

Happy to take a crack at the storage-side fix if useful — filing first since the
failure lands on unrelated PRs (found while working on #732).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions