Skip to content

fix(shared): give each API client its own HTTP transport - #497

Merged
piekstra merged 1 commit into
mainfrom
piekstra/jtk-delete-test-flake
Oct 1, 2026
Merged

piekstra merged 1 commit into
mainfrom
piekstra/jtk-delete-test-flake

Conversation

@piekstra

Copy link
Copy Markdown
Member

Summary

Fixes the intermittent HTTP/1.x transport connection broken: http: CloseIdleConnections called failures in jtk tests. Seen on main in TestRunDelete_MultipleIssues, TestRunDelete (remote links), TestRunDelete_Force (fields), TestDeleteFieldContext and TestRunRemove_IDOnly (sprints).

Root cause: client.New built an http.Client with no Transport, so every client shared http.DefaultTransport. httptest.Server.Close calls http.DefaultTransport.CloseIdleConnections() on every test server shutdown. For a response without a body (a 204, or an empty 200), net/http puts the connection back in the idle pool in readLoop before it hands the response to the waiting roundTrip. If another parallel test closes its server in that window, the connection is closed with errCloseIdleConns, roundTrip sees the close before the response, and the request fails. Every failing test uses a bodiless response (DELETE 204, field trash, backlog move).

Changes:

  • shared/client: each Client gets its own http.DefaultTransport.Clone(), so other code closing the default pool cannot touch its connections. The clone keeps the default proxy, dial, TLS and HTTP/2 settings. This covers jtk and cfl, which both build on this client.
  • tools/jtk/internal/cmd/issues/delete_test.go: when runDelete fails, the test prints the per-issue errors it wrote to stderr. The returned error is only already reported (1 of 3 issue(s) failed), which hid the transport error in CI.

Tests

Reproduction: the flake did not reproduce at package level locally (issues.test -test.count=200 -test.cpu 1,2,8: 0 failures in 600 package runs), so I reproduced the mechanism directly with a temporary rig in shared/client (source below, not committed): 4 goroutines each issue DELETEs returning 204 against their own httptest server through client.New, while another goroutine calls http.DefaultTransport.(*http.Transport).CloseIdleConnections() every 2ms, the same call httptest.Server.Close makes. Command, 15s per run:

REPRO_SECS=15 REPRO_WORKERS=4 REPRO_BURNERS=0 REPRO_CLOSE_US=2000 GOMAXPROCS=4 go test -race -run TestReproCloseIdleRace -v ./client/
runs runs with a failure failed requests requests
before 20 6 8 7,846,683
after 20 0 0 8,835,001

Every failure before was the exact CI error: request failed: Delete "http://127.0.0.1:<port>/x": net/http: HTTP/1.x transport connection broken: http: CloseIdleConnections called.

  • New TestNew/owns_its_transport asserts clients do not share http.DefaultTransport or each other's transport.
  • go test -race ./... and golangci-lint run ./... pass in shared, tools/jtk and tools/cfl.
Reproduction rig
package client

import (
	"context"
	"net/http"
	"net/http/httptest"
	"os"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"
	"testing"
	"time"
)

func envInt(k string, d int) int {
	if v, err := strconv.Atoi(os.Getenv(k)); err == nil {
		return v
	}
	return d
}

// Temporary reproduction rig, not committed.
func TestReproCloseIdleRace(t *testing.T) {
	dur := time.Duration(envInt("REPRO_SECS", 20)) * time.Second
	closeEvery := time.Duration(envInt("REPRO_CLOSE_US", 2000)) * time.Microsecond
	stop := make(chan struct{})
	var bg sync.WaitGroup
	bg.Add(1)
	go func() {
		defer bg.Done()
		for {
			select {
			case <-stop:
				return
			case <-time.After(closeEvery):
			}
			// Same call httptest.Server.Close makes on every test server shutdown.
			http.DefaultTransport.(*http.Transport).CloseIdleConnections()
		}
	}()
	for b := 0; b < envInt("REPRO_BURNERS", 8); b++ {
		bg.Add(1)
		go func() {
			defer bg.Done()
			x := 0
			for {
				select {
				case <-stop:
					return
				default:
				}
				for i := 0; i < 10000; i++ {
					x += i
				}
			}
		}()
	}
	var fails, total atomic.Int64
	var wg sync.WaitGroup
	deadline := time.Now().Add(dur)
	for g := 0; g < envInt("REPRO_WORKERS", 4); g++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
				w.WriteHeader(http.StatusNoContent)
			}))
			srv.Start()
			defer srv.Close()
			c := New(srv.URL, "e", "t", nil)
			for time.Now().Before(deadline) {
				total.Add(1)
				if _, err := c.Delete(context.Background(), "/x"); err != nil {
					if fails.Add(1) == 1 {
						t.Logf("first failure: %v", err)
					}
					if !strings.Contains(err.Error(), "CloseIdleConnections called") {
						t.Errorf("other error: %v", err)
						return
					}
				}
			}
		}()
	}
	wg.Wait()
	close(stop)
	bg.Wait()
	t.Logf("CloseIdleConnections failures: %d of %d requests", fails.Load(), total.Load())
	if fails.Load() > 0 {
		t.Fail()
	}
}

client.New used http.DefaultTransport. httptest.Server.Close calls
CloseIdleConnections on http.DefaultTransport, so any parallel test that
shut down its server could close another test's connection. net/http
returns a connection to the idle pool before it hands a bodiless
response (such as a 204) to the waiting request; closing it in that
window fails the request with "HTTP/1.x transport connection broken:
http: CloseIdleConnections called". That is the intermittent failure in
the jtk delete, remote link, field, field context and sprint tests.

Each client now uses a clone of http.DefaultTransport, so other code
closing the default pool's idle connections cannot affect it.

The delete tests now print the per-issue errors from stderr when
runDelete fails, since its returned error only says "already reported".

@monit-reviewer monit-reviewer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 7eafdb1e9a35
Profile: claude-monit-reviewer - Posting as: monit-reviewer

Summary

Reviewer Findings
go:implementation-tests 0
architecture:solid-reviewer-agnostic 0

Reviewer Coverage

  • go:implementation-tests — complete (broad); skipped: none; constraints: none
  • architecture:solid-reviewer-agnostic — complete (broad); inspected 2 assigned files (3 inspected across reviewers): shared/client/client.go, shared/client/client_test.go; skipped: none; constraints: Only four production call sites construct client.New (jtk and cfl api packages), each once per command, so a per-Client transport does not multiply connection pools in practice. Ran go test ./client/ in shared (passes); did not run the full jtk/cfl suites or the PR's reproduction rig.
Inspected files (3)
  • shared/client/client.go
  • shared/client/client_test.go
  • tools/jtk/internal/cmd/issues/delete_test.go

0 PR discussion threads considered. 0 summarized; 0 resolved.


Completed in 1m 47s | ~$1.16 (est.) | claude-sonnet-5, claude-opus-5-5 | cr 0.10.318
Field Value
Model claude-sonnet-5, claude-opus-5-5
Reviewers go:implementation-tests, architecture:solid-reviewer-agnostic
Engine claude_cli · claude-sonnet-5, claude-opus-5-5
Reviewed by cr · monit-reviewer
Duration 1m 47s wall · 2m 01s compute
Cost ~$1.16 (est.)
Pricing basis anthropic-public-2026-09-23
Tokens 36 in / 6.2k out

Per-workstream usage

  • orchestrator-selection — claude-sonnet-5
    • In: 6
    • Out: 2.2k
    • Cache read: 89.9k
    • Cache create: 43.2k
    • Cost: ~$0.21 (est.)
    • Duration: 31s
  • go:implementation-tests — claude-sonnet-5
    • In: 12
    • Out: 1.7k
    • Cache read: 273.9k
    • Cache create: 43.4k
    • Cost: ~$0.25 (est.)
    • Duration: 44s
  • architecture:solid-reviewer-agnostic — claude-opus-5-5
    • In: 12
    • Out: 1.8k
    • Cache read: 217.7k
    • Cache create: 56.7k
    • Cost: ~$0.53 (est.)
    • Duration: 31s
  • orchestrator-rollup — claude-sonnet-5
    • In: 6
    • Out: 378
    • Cache read: 144.3k
    • Cache create: 33.8k
    • Cost: ~$0.17 (est.)
    • Duration: 14s

@piekstra
piekstra marked this pull request as ready for review October 1, 2026 12:26
@piekstra
piekstra merged commit a5a97cd into main Oct 1, 2026
16 checks passed
@piekstra
piekstra deleted the piekstra/jtk-delete-test-flake branch October 1, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants