fix: port-forward 429 handling, quick-deploy exit code, exec hints, live tests

CLAUDE.md

@@ -200,7 +200,7 @@ All API calls go to `/api/v1/*`. Key route groups:
 - `/api/v1/audit-logs` — list/export; `--action` and `--entity-type` are exact matches (`deploy`, `stack_instance`, …; v0.7.0+, see `auditFilterValuesHelp`). Older servers: HTTP method + last route segment (`create` / `deploy`)
 - `/api/v1/users` — list/delete/disable/enable/password (admin). `PUT /:id/role` `{"role"}` (k8s-stack-manager v0.8.0+, `user set-role <user> <role>`, user by ID or username via `resolveUserID`): response `id`, `old_role`, `new_role`, `changed`; 403 own role / caller not admin, 409 SSO user / last admin, 400 invalid role; a plain 404 (no route) maps to the v0.8.0 upgrade hint (`setRoleError`)
 - `/ws` — the server closes a socket with code 1008 and reason `session revoked` or `token expired`. `Client.Watch` returns a `WatchStream`; `Err()` gives a `*WSClosedError` ("Connection closed by the server: <reason>. Log in again."). `stack watch` exits non-zero on it; for `token expired` it calls `Client.RenewSession` (forced renewal, false without refresh token) and reconnects, at most once per 30 s (`stackWatchMinReconnect`), then with `--id` GETs each pending instance (`watchCatchUp`) so a terminal status in the gap counts. Ctrl-C during renewal or dial exits 0. Another connection loss: `--id` with pending instances exits non-zero ("connection lost before all instances reached a terminal status"), else a stderr note and exit 0. `StreamDeploymentLogs` returns the same `WSClosedError`; `--follow` (`followLogsCtx`, op deploy/stop/clean/rollback, "" for `stack logs -f`) uses the same `renewAfterClose`; it reads the instance after the first connect (the server sets deploying/stopping/cleaning before the 202; not for `logs -f`) and after each reconnect (`StreamOptions.AfterSubscribe`). A normal close (code 1000, stream status `unknown`) reads the instance (`followEndError`): a final status counts, else it fails like a lost connection. When it cannot follow to the end it says that the operation continues on the server and points to `stack watch --id` / `stack status` (never "run again"); errors name the operation ("rollback failed (status error): ..."). Terminal statuses live in `client.IsTerminalStatus` (running, stopped, draft, error, partial, failed) and `client.IsFailedStatus` (error, partial, failed → exit 1)
-- `GET /api/v1/stack-instances/:id/port-forward?service=&port=` — `stack port-forward` (omattsson/stackctl#178, k8s-stack-manager#722). WebSocket upgrade, auth only by header (`dialWSOpts` with `stream`: no `?token=` retry, a 401 gets one forced `RenewSession` and one more dial; no `Origin` header). Refusals are HTTP statuses with `{"error","code"}` (`WSHandshakeError.Code`, `StreamCode*`): `FeatureDisabled()` (404 `feature_disabled`), `NoRoute()` (404 without a code: older server; hint `client.PortForwardServerVersion`), `UpgradeDropped()` (2xx, or 400 without a code or `upgrade_required`: a proxy), `Fatal()` (400, 401, 403, 404, 409 stop the command; 429, 5xx and network errors close one connection; 3 handshake 429s in a row, with no successful upgrade between them (`portForwardMax429`, one counter for all tunnels), stop the command). After the upgrade: binary frames of at most 32 KiB both ways (server read limit 64 KiB), no half-close. `Client.Tunnel`: one reader and one writer goroutine, pings and the close through `WriteControl`; pings from their own goroutine every 30 s (a waiting ping cannot delay a cancel), read deadline 75 s moved by data, ping or pong; a local write deadline of 75 s; after a local EOF the tunnel copies server data to local until the server closes, a local write fails, or `DrainTimeout` (2 s) passes without server data, and only then sends close 1000; a server close 1000 does `CloseWrite` on the local socket. A 401 on the stream dial renews with the refused token (`renewSession(stale)`), so parallel 401s send one refresh. Close codes (`Close*` constants, `CloseCodeText`): 1001 shutdown, 1008 (`StreamClosedError.Fatal`, unwraps to `WSClosedError`; reason `session revoked` → login advice, `access revoked` (`CloseReasonAccessLost`) → stack stopped, deleted or cleaned, or access changed), 1011, 4001 time limit, 4002 idle, 4502 pod stream error. `--insecure`, and an `http://` API URL to a host that is not loopback (`plaintextRemoteURL`), are refused unless `--allow-insecure-stream` (`checkStreamTransport`, shared with `stack exec`; "insecure" is `effectiveInsecure()` in `root.go`, the same check as the TLS warning and `applyInsecureTLS`). At most 16 open tunnels (`portForwardConnLimit`); a further connection is closed at once. The non-loopback warning checks the bound address (`warnNonLoopback`). SIGTERM ends the command like Ctrl-C (`portForwardSignals`; `stack_portforward_signal_test.go` sends a real SIGTERM to a subprocess, not on Windows). A fatal tunnel error is not repeated in the per-connection close line.
+- `GET /api/v1/stack-instances/:id/port-forward?service=&port=` — `stack port-forward` (omattsson/stackctl#178, k8s-stack-manager#722). WebSocket upgrade, auth only by header (`dialWSOpts` with `stream`: no `?token=` retry, a 401 gets one forced `RenewSession` and one more dial; no `Origin` header). Refusals are HTTP statuses with `{"error","code"}` (`WSHandshakeError.Code`, `StreamCode*`): `FeatureDisabled()` (404 `feature_disabled`), `NoRoute()` (404 without a code: older server; hint `client.PortForwardServerVersion`), `UpgradeDropped()` (2xx, or 400 without a code or `upgrade_required`: a proxy), `Fatal()` (400, 401, 403, 404, 409 stop the command; 429, 5xx and network errors close one connection; a 429 `tunnel_limit` first gets 3 dial retries (`portForward429Delays`: 0.5, 1, 2 s, for tunnels that drain or close; the rate limiter's 429 is not retried); 3 connections in a row refused with 429, each while no other connection of the command dials or tunnels (`portForwarder.active`, decremented before the refused local connection closes; a refusal with others active resets `rateLimited`), stop the command (`portForwardMax429`), so a full server tunnel limit never closes working tunnels). After the upgrade: binary frames of at most 32 KiB both ways (server read limit 64 KiB), no half-close. `Client.Tunnel`: one reader and one writer goroutine, pings and the close through `WriteControl`; pings from their own goroutine every 30 s (a waiting ping cannot delay a cancel), read deadline 75 s moved by data, ping or pong; a local write deadline of 75 s; after a local EOF the tunnel copies server data to local until the server closes, a local write fails, or `DrainTimeout` (2 s) passes without server data, and only then sends close 1000; a server close 1000 does `CloseWrite` on the local socket. A 401 on the stream dial renews with the refused token (`renewSession(stale)`), so parallel 401s send one refresh. Close codes (`Close*` constants, `CloseCodeText`): 1001 shutdown, 1008 (`StreamClosedError.Fatal`, unwraps to `WSClosedError`; reason `session revoked` → login advice, `access revoked` (`CloseReasonAccessLost`) → stack stopped, deleted or cleaned, or access changed), 1011, 4001 time limit, 4002 idle, 4502 pod stream error. `--insecure`, and an `http://` API URL to a host that is not loopback (`plaintextRemoteURL`), are refused unless `--allow-insecure-stream` (`checkStreamTransport`, shared with `stack exec`; "insecure" is `effectiveInsecure()` in `root.go`, the same check as the TLS warning and `applyInsecureTLS`). At most 16 open tunnels (`portForwardConnLimit`); a further connection is closed at once. The non-loopback warning checks the bound address (`warnNonLoopback`). SIGTERM ends the command like Ctrl-C (`portForwardSignals`; `stack_portforward_signal_test.go` sends a real SIGTERM to a subprocess, not on Windows). A fatal tunnel error is not repeated in the per-connection close line.
 - `/api/v1/admin/orphaned-namespaces` — `orphaned list [--details]`; `DELETE /:ns?confirm=<ns>` is needed for a namespace without the label managed-by=k8s-stack-manager (409 without it; `orphaned delete --confirm`). `test/contract/path_contract_test.go` checks every `/api/v1` literal of `pkg/client` and `cmd` against the vendored swagger
 - `POST /api/v1/auth/logout-all` — `logout --all`: uses the session token of the context when a token file exists (also when an API key is set), else the API key; a 401 gets one forced `RenewSession` and one retry. On failure the command stops and keeps the token file; on success the normal logout follows.
 - `GET /api/v1/dashboard` — `dashboard`: covers all users; `failing_instances` has status error only, `expiring_soon` the next hour. For the role `user`, `clusters` has ID and name only (no health): the table hides clusters without `health_status`.

README.md

@@ -310,7 +310,7 @@ stackctl stack rollback my-app --target-log <log-id>
 `stack port-forward <name|id> <service> [LOCAL:]REMOTE` opens a local port that forwards to a service of the stack through the server (`GET /api/v1/stack-instances/:id/port-forward`). You need no kubeconfig. REMOTE is the service port. LOCAL must be 0 (any free port) or from 1025 to 65535. Without LOCAL, the local port is REMOTE when REMOTE is above 1024, else REMOTE + 10000; when that port is in use, stackctl tries the next 20 ports, then any free port. An explicit LOCAL port that is in use stops the command. `--address` sets the local addresses (default `127.0.0.1`, comma-separated). stdout gets `Forwarding from 127.0.0.1:3306 -> mysql:3306` (`-q`: only `host:port`); stderr gets one line when a connection opens and one when it closes. Each local connection opens its own tunnel. At most 16 tunnels are open at the same time: stackctl closes a further local connection at once and writes a line to stderr. When the local client closes its side, the tunnel still passes server data to it until the server closes, a local write fails, or 2 s pass without server data. A local client that reads no data for 75 s loses its connection. Ctrl-C or SIGTERM closes all connections and exits 0.
 
 - Errors that stop the command (exit 1): no access (401, 403), an unknown stack, service or port (404), the feature turned off on the server or on the cluster, a stack that is not deployed (409), and a close 1008 during a tunnel. Close 1008 with the reason `session revoked` means that the session ended: run `stackctl login`. The reason `access revoked` means that the stack was stopped, deleted or cleaned, or that your access to it changed. A 404 without an error code means that the server has no port-forward route: upgrade the server to k8s-stack-manager v0.15.0 or later. A status 200 without an upgrade means that a proxy dropped the WebSocket upgrade.
-- Errors that close only one connection: the rate limit or the tunnel limit (429), no ready pod (503), a failed pod dial (502), the server time limit (close 4001), the idle timeout (4002), a pod stream error (4502) and network errors. 3 refusals with 429 in a row, with no successful tunnel between them, stop the command (exit 1): wait and try again.
+- Errors that close only one connection: the rate limit or the tunnel limit (429; the tunnel limit, code `tunnel_limit`, first gets 3 dial retries within 3.5 s, while a tunnel that closes or drains frees its server slot), no ready pod (503), a failed pod dial (502), the server time limit (close 4001), the idle timeout (4002), a pod stream error (4502) and network errors. The open tunnels stay open. 3 connections in a row refused with 429 while no other connection of the command is open stop the command (exit 1): wait and try again.
 - The server can refuse API keys for port-forward: then run `stackctl login` and remove the API key from the context or the environment.
 - Security: anyone who can reach the local port uses your access to the stack. On a shared machine, other local users can connect to it. Web pages in your browser can reach a forwarded HTTP service through DNS rebinding: stop the command when you do not need it. An address that is not loopback opens the port to the network (stackctl writes a warning). `--insecure` is refused, because a man in the middle could read the tunnel data. An `http://` API URL is refused for the same reason, except for `localhost`, `127.0.0.1` and `[::1]`. `--allow-insecure-stream` accepts that risk.
 

cli/cmd/agent_guide.md

@@ -71,7 +71,8 @@ stackctl stack history my-app
 stackctl stack port-forward my-app mysql 0:3306 -q   # stdout: 127.0.0.1:<port>; exit 1 on 403/404/409
 # SIGTERM or Ctrl-C ends it with exit 0. At most 16 open connections; more are closed at once.
 # After a local EOF the tunnel passes server data until the server closes or 2 s pass without data.
-# 3 refusals with 429 in a row end it with exit 1: wait and try again.
+# A 429 closes only that connection (the server tunnel limit is retried 3 times first).
+# 3 refused connections in a row while no other connection is open end it with exit 1: wait and try again.
 # Needs k8s-stack-manager v0.15.0 or later (an older server: "upgrade the server to k8s-stack-manager v0.15.0 or later").
 # An http:// API URL to a host that is not loopback is refused (add --allow-insecure-stream to accept the risk).
 
@@ -109,7 +110,7 @@ Paged lists (`stack list`, `definition list`, `template list`) return one page a
 
 ## Long-running operations
 
-`stack deploy`, `stop`, `clean`, `rollback`, `template quick-deploy` and the `bulk` commands return when the server has accepted the operation; the work continues on the server (a `bulk` result `success` means that the operation started). To wait for the result:
+`stack deploy`, `stop`, `clean`, `rollback`, `template quick-deploy` and the `bulk` commands return when the server has accepted the operation; the work continues on the server (a `bulk` result `success` means that the operation started). When `template quick-deploy` creates the stack but its deploy cannot start (for example the namespace of an earlier stack with the same name is still terminating), it prints the stack (status `error`) and exits 1 with the reason: fix the cause, then run `stackctl stack deploy <id>`; do not run the quick deploy again (the stack exists). To wait for the result:
 
 - add `--follow` (`-f`, only `stack deploy`, `stop`, `clean` and `rollback`): stackctl streams the log and exits when the stack reaches a final status. With `-o json` the log goes to stderr and stdout gets the stack after the operation (also after a failed one);
 - or run `stackctl stack watch --id <id>[,<id>...]`, which exits when every listed stack reaches a final status;

cli/cmd/stack_exec_test.go

@@ -365,8 +365,8 @@ func TestStackExecCmd_ExitMapping(t *testing.T) {
 			handler: func(ws *websocket.Conn) { wsCloseAndDrain(ws, websocket.CloseGoingAway, "server shutdown") },
 			wantMsg: []string{"exec closed: the server is shutting down; the server did not start the command"}},
 		{name: "403 container not allowed", args: []string{"42", "web", "-c", "php", "--", "ls"}, wantCode: 255,
-			refuse:  jsonRefusal(http.StatusForbidden, "container_not_allowed", "Container does not allow exec"),
-			wantMsg: []string{"exec refused by the server (HTTP 403)", `Container "php" does not allow exec`}},
+			refuse:  jsonRefusal(http.StatusForbidden, "container_not_allowed", "The container is not allowed for exec"),
+			wantMsg: []string{`exec refused by the server (HTTP 403): The container is not allowed for exec: "php". Use -c with a container that allows exec, or ask an administrator which containers do.`}},
 		{name: "404 feature disabled", wantCode: 255,
 			refuse:  jsonRefusal(http.StatusNotFound, "feature_disabled", "This feature is disabled"),
 			wantMsg: []string{"The server has exec turned off"}},
@@ -377,8 +377,8 @@ func TestStackExecCmd_ExitMapping(t *testing.T) {
 			},
 			wantMsg: []string{"the server has no exec route (HTTP 404); upgrade the server to " + client.ExecServerVersion}},
 		{name: "503 no ready pod", wantCode: 255,
-			refuse:  jsonRefusal(http.StatusServiceUnavailable, "no_ready_pod", "No ready pod"),
-			wantMsg: []string{"HTTP 503", "no running and ready pod"}},
+			refuse:  jsonRefusal(http.StatusServiceUnavailable, "no_ready_pod", "No ready pod for the chart"),
+			wantMsg: []string{"exec refused by the server (HTTP 503): No ready pod for the chart. Check 'stackctl stack pods', then try again."}},
 		{name: "unknown stack name", args: []string{"nope", "web", "--", "ls"}, wantCode: 255,
 			handler: exitWith(0, "", ""),
 			wantMsg: []string{`no stack found with name "nope"`}},