Skip to content

test: pin the lapsed hop port hold on its socket, not the port number - #33884

Merged
bolichen97 merged 1 commit into
mainfrom
fix/32220-lease-lapse-bind-wait
Nov 9, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/32220-lease-lapse-bind-wait

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_the_port_is_free_again_once_the_lease_lapses stops failing unrelated PRs when another process briefly takes the port the hop port guard just gave back.

The test failed twice on Linux CI, on PRs that do not touch the hop port guard, always at the same line:

AssertionError: the port stayed unbindable after its lease lapsed
assert 'REFUSED 98' == 'BOUND'

Why it matters

A red backend shard blocks PR Readiness for whoever hits it and costs a rerun each time.

Not a goal

  • No production change: the guard already closes its hold the moment the lease lapses.
  • No change to how a port is picked for the hold, or to the tests that expect a refusal.

What changed (motivation → approach → change)

The test waited until the guard held no port, then asked a separate process to bind that port once. The freed port is an ordinary ephemeral port, so on a busy runner another process can be handed it in the gap before the probe runs. That process holds it for a moment, and the single probe reads EADDRINUSE (errno 98) although our socket is closed.

The test asserts on the guard's own socket instead of the port number, the way test_release_frees_it_for_a_legitimate_reconnect already does. It captures the hold's socket while the lease stands, lapses the lease through the manual_clock fixture, and requires that socket closed once the reaper drops the hold. The bind from another process stays as a containment check through _wait_for_bind_from_another_process, whose default bound is FALLBACK_TIMEOUT_SECS, so a brief competitor is waited out. The same one-shot probe on a just-freed port in test_a_second_process_is_refused_while_the_lease_stands (its "port was free to start" check) goes through the same wait.

Backwards compatibility

Compatible: test-only change.

Removes nothing: the lapse test checks the same release on the socket it owned, and the refusal assertions are unchanged.

Tests

test/test_hop_port_guard.py:

Test Locks in
test_the_port_is_free_again_once_the_lease_lapses a hold whose lease lapses is dropped by the reaper with its socket closed; another process can then bind the port
test_a_second_process_is_refused_while_the_lease_stands unchanged refusal checks; its "port was free to start" check waits out a brief competitor
test_release_frees_it_for_a_legitimate_reconnect unchanged; its containment wait shares the helper's FALLBACK_TIMEOUT_SECS bound

Proof: not provable -- test-only change

Determinism proof (testing-conventions D12):

  • Forced condition: a throwaway -p plugin (not committed) lets a second socket bind the just-freed port for 1 s, right after the guard releases it and right after _free_port() returns in the lease-stands test. On the parent 2caa69b7a6 both tests go red with the CI error (assert 'REFUSED 98' == 'BOUND', and port was not free to start); on this head both pass.
  • Negative control: a throwaway plugin whose release() drops the hold but closes its socket 2 s later makes the lapse test fail by name at the lapsed hold was dropped with its socket still open.
  • Repeats: the three tests this diff touches pass 20/20 at -n0. The whole file passes 16 of 17 -n0 runs; the 17th timed out (over 120 s, host load about 200) in the teardown of test_a_descriptor_limit_is_owed_rather_than_raised, which this diff does not touch. 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test/test_instances_port_allocator.py and test/test_instances_forwarder_hints.py.
  • Shuffled order of those three files (seed 29907): 51 passed.

Manual verification

N/A — test-only change; CI runs the shard.

Related Issues

Closes #32220

Sibling sweep

Swept: git grep -n '== "BOUND"' -- test/test_hop_port_guard.py
Sibling: test/test_hop_port_guard.py:151 -- changed here
Sibling: test/test_hop_port_guard.py:225 -- changed here (its containment wait takes the helper's default bound)
Sibling: test/test_hop_port_guard.py:386 -- not affected -- the port stays bound by the test's own socket for the whole check, so it is never a freed port
Sibling: test/test_hop_port_guard.py:156 -- not affected -- this and the other REFUSED probes target a port the guard holds, which no competitor can take

Pattern harvest

Rule candidate: review-prompt
Pattern: a test proves a socket was released by binding its freed port number once, racing every other process the kernel may hand that ephemeral port to

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

The lease-lapse test proved the release by binding the freed port once
from another process. The freed port is an ordinary ephemeral port, so
on a busy runner another process can be handed it before the probe
runs, and the probe reads EADDRINUSE although our socket is closed.

Assert on the socket the hold owned instead, as the release test does:
capture it while held, lapse the lease through an injected clock, and
require it closed once the reaper drops the hold. The external probe
stays as a containment check through the bounded wait the release test
uses, whose default becomes FALLBACK_TIMEOUT_SECS; the lease-stands
test's one-shot probe on a just-freed port uses the same wait.

Closes #32220
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 9, 2026 13:36
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Determinism evidence for this head (0f2fb3811e14), as testing-conventions D12 asks: the commands and both outputs.

Forced condition, a throwaway plugin loaded with PYTHONPATH=<dir> python -m pytest -n0 -p no:cacheprovider -p forced_competitor -q <the two node ids>:

"""Throwaway -p plugin, never committed: forces the race issue #32220 names.

Right after the port under test is freed, a short-lived competitor in this process
binds it (plain bind, no SO_REUSEADDR) for HOLD_SECS, the way another process handed
the freed ephemeral port would. A one-shot probe then reads REFUSED 98; a bounded
wait reads BOUND once the competitor goes.
"""
import socket
import threading

import pytest

HOLD_SECS = 1.0
FORCED = []


def _compete(port):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    try:
        s.bind(("127.0.0.1", port))
    except OSError:
        s.close()
        return
    FORCED.append(port)
    threading.Timer(HOLD_SECS, s.close).start()


@pytest.fixture(autouse=True)
def _forced_competitor(request, monkeypatch):
    from kiro_crew.instances import hop_port_guard as m

    name = request.node.name
    if name == "test_the_port_is_free_again_once_the_lease_lapses":
        orig = m.HopPortGuard.release

        def release(self, port):
            with self._lock:
                held = port in self._held
            orig(self, port)
            if held:
                _compete(port)

        monkeypatch.setattr(m.HopPortGuard, "release", release)
    elif name == "test_a_second_process_is_refused_while_the_lease_stands":
        mod = request.module
        orig_free = mod._free_port
        calls = []

        def free_port():
            port = orig_free()
            calls.append(port)
            if len(calls) == 1:
                _compete(port)
            return port

        monkeypatch.setattr(mod, "_free_port", free_port)
    yield


def pytest_terminal_summary(terminalreporter):
    terminalreporter.write_line(f"forced_competitor: competitor bound {len(FORCED)} port(s)")

On the parent 2caa69b7a6 (its version of the test file, run as a temporary copy test_hop_port_guard_parentcopy.py):

E       assert 'REFUSED 98' == 'BOUND'
E       assert 'REFUSED 98' == 'BOUND'
forced_competitor: competitor bound 2 port(s)
FAILED test/test_hop_port_guard_parentcopy.py::TestAnotherProcessCannotTakeAHeldPort::test_the_port_is_free_again_once_the_lease_lapses - AssertionError: the port stayed unbindable after its lease lapsed
FAILED test/test_hop_port_guard_parentcopy.py::TestAnotherProcessCannotTakeAHeldPort::test_a_second_process_is_refused_while_the_lease_stands - AssertionError: port was not free to start
================== 2 failed, 2 warnings in 112.74s (0:01:52) ===================

On this head:

forced_competitor: competitor bound 2 port(s)
======================== 2 passed, 2 warnings in 17.43s ========================

Negative control (-p late_close, the lapse test only), showing the socket assertion has teeth:

"""Throwaway -p plugin, never committed: negative control for the socket pin.

release() drops the hold from the held set but closes its socket 2 s later, off the
lock. The lapse test must fail by name at its socket assertion, not pass.
"""
import threading

import pytest


@pytest.fixture(autouse=True)
def _late_close(request, monkeypatch):
    from kiro_crew.instances import hop_port_guard as m

    if request.node.name == "test_the_port_is_free_again_once_the_lease_lapses":

        def release(self, port):
            with self._lock:
                entry = self._held.pop(port, None)
                if entry is None:
                    return
                try:
                    self._selector.unregister(entry[0])
                except (KeyError, ValueError):
                    pass
            threading.Timer(2.0, entry[0].close).start()

        monkeypatch.setattr(m.HopPortGuard, "release", release)
    yield
E       AssertionError: the lapsed hold was dropped with its socket still open
FAILED test/test_hop_port_guard.py::TestAnotherProcessCannotTakeAHeldPort::test_the_port_is_free_again_once_the_lease_lapses - AssertionError: the lapsed hold was dropped with its socket still open
======================== 1 failed, 2 warnings in 11.12s ========================

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0f2fb3811e1421b29ead7819ef9d1efce3f54251 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 0f2fb3811e1421b29ead7819ef9d1efce3f54251 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0f2fb3811e1421b29ead7819ef9d1efce3f54251 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change. It checks the guard's own socket instead of the port number, which removes the race with other processes that caused the flake. No design-level concerns.

[DESIGN-REVIEWED] 0f2fb38

Description read

[DESCRIPTION-READ] 8c9899a1a88f1ce04cc6a3ec5a36508f449c2e1c914090ed78061924bd814967

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0f2fb3811e1421b29ead7819ef9d1efce3f54251 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0f2fb38

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0f2fb3811e1421b29ead7819ef9d1efce3f54251 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0f2fb3811e1421b29ead7819ef9d1efce3f54251 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0f2fb38

Verdict parsed from the review's SHA-scoped output markers for commit 0f2fb3811e1421b29ead7819ef9d1efce3f54251.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0f2fb3811e1421b29ead7819ef9d1efce3f54251 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 9, 2026 13:47

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 0f2fb38. Test-only deflake for #32220: the lapse test now checks that the reaper closed the hold's own socket (test/test_hop_port_guard.py:180-200), and both one-shot BOUND probes use the bounded wait helper (test/test_hop_port_guard.py:150-151, :57-58). src/kiro_crew/instances/hop_port_guard.py:338-344 is unchanged behaviour. Two mutations each turn the named test red.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@bolichen97
bolichen97 merged commit 4d6d5f0 into main Nov 9, 2026
97 checks passed
@bolichen97
bolichen97 deleted the fix/32220-lease-lapse-bind-wait branch November 9, 2026 14:51
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_the_port_is_free_again_once_the_lease_lapses (test_hop_port_guard) gets EADDRINUSE on CI

2 participants