Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 70 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,12 @@ jobs:
# not start until 397 s (6 m 37 s) into the step, leaving 83 s
# of the 480 s for the whole suite; the kill landed 96 s after
# the first test started. How many tests the current tree has is
# not measured -- the run never reached a total.
# not measured -- the run never reached a total. That head is
# not on develop: the PR was rebased on merge, and the commit
# carrying this same change there is `a774ac4d`. Both are named
# because the run really did run on `9c8d3673`;
# `git show <h> | git patch-id --stable` prints
# ad968515ab03ec2249ff050a284e1b90f3fe5ba0 for either one.
#
# The two compile figures are NOT a toolchain comparison and must
# not be read as one. 28,721 lines of test code and 17,144 lines of
Expand All @@ -206,13 +211,70 @@ jobs:
# Note which way that cuts on the build half: Sources grew by those
# 17,144 lines and the Build step still came back 240.40s -> 230.46s.
#
# 16 minutes (960 s) is 2.04x of 392 s + 78 s = 470 s -- this
# image's measured compile plus the OTHER image's measured run,
# because this image's run figure does not exist yet: the step was
# killed before the suite finished. The first green run on this
# image replaces that addend and this number with its own
# measurement.
timeout-minutes: 16
# The promise above -- that the first green run on this image
# replaces that addend and this number with its own measurement --
# is kept here, and the replacement does not keep the number.
# Withdrawn, quoted verbatim: "16 minutes (960 s) is 2.04x of
# 392 s + 78 s = 470 s -- this image's measured compile plus the
# OTHER image's measured run, because this image's run figure does
# not exist yet: the step was killed before the suite finished."
# What was wrong with it is not the borrowed addend but the
# stitching: it adds two figures from two different runs and bounds
# a step that neither of them is. A step budget bounds a step's
# wall clock, and that is a field pair of the Actions API, present
# on every run that reached this step:
#
# gh api repos/NoiXdev/macSCP/actions/runs/<id>/jobs --jq '.jobs[].steps[] | select(.name=="Unit-Tests") | "\(.conclusion) \((.completed_at|fromdateiso8601) - (.started_at|fromdateiso8601))s"'
#
# 38225383495 macos-15 329 s success
# 40933631584 macos-latest 493 s failure -- the 480 s budget's
# own kill plus the overhead of
# reporting it, not a suite duration
# 41045404989 macos-latest 581 s success (that RUN went red
# later, on the warning gate below;
# this step did not)
# 41047833837 macos-latest 517 s success
# 41048445836 macos-latest 640 s success
#
# 640 s is the measured maximum of a step that finished. The spread
# it sits in matters more than the maximum: 517 -> 640 s is a
# factor of 1.24 across two runs ONE commit apart on the same
# image, and the compile half alone moved 345.71s -> 492.86s, a
# factor of 1.43. That pair isolates the variance about as well as
# anything here can: the only difference between the two trees is
# 23 added lines of CLAUDE.md, which no target compiles. The runs
# had heads `b9ba9d27` and `6f1fe42a`, and the rebase on merge left
# both unreachable, so the command below names the develop commits
# carrying those same two changes instead -- measured 2026-10-31,
# it prints one file and 23 insertions for either pair:
#
# git diff --stat 6583396e 1ce3e9ff
#
# A budget has to clear that variance, not the mean.
#
# 20 minutes (1200 s) is 1.88x of 640 s. The two derivations this
# comment has carried landed at 2.09x and 2.04x, so this is the
# first one under 2 -- but neither of them states 2x as a
# requirement, so "the 2x floor" is a reading of them rather than a
# quote, and it is written here as a reading. 2x of 640 s is
# 1280 s, which makes 22 minutes (1320 s, 2.06x) the smallest
# whole-minute budget holding that reading. 20 is the maintainer's
# decision of 2026-10-31, taken while the best figure on offer was
# the 558.79 s the withdrawn stitching produced -- against which
# 1200 s read as 2.15x. The series above was measured after that
# decision, not before it, and 1.88x is what it leaves standing.
#
# Raising this does not crowd the JOB budget above it. Same API,
# one level up -- measured 2026-10-31:
#
# gh api repos/NoiXdev/macSCP/actions/runs/<id>/jobs --jq '.jobs[] | "\(.conclusion) \((.completed_at|fromdateiso8601) - (.started_at|fromdateiso8601))s"'
#
# gives 753 s and 937 s for the two green runs above, against the
# job's 35 minutes (2100 s). A step that ran the full 1200 s would
# put the worst of those at 937 - 640 + 1200 = 1497 s, 0.71x of the
# job budget, so the job still kills a hung step later than this
# step does -- which is the ordering the step budget exists for.
timeout-minutes: 20
run: |
set -o pipefail
swift test --event-stream-output-path "$RUNNER_TEMP/test-events.jsonl" \
Expand Down
5 changes: 4 additions & 1 deletion Tests/macSCPCoreTests/ConnectMainActorLivenessTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,10 @@ struct ConnectMainActorLivenessTests {
/// wording and the withdrawn condition. `bar` is derived FROM `ambient`,
/// so that test reduced to `ambient > 4 s`, and it let a control of one
/// second stand as adequate resolution for a three-second ceiling.
/// Run 41040277938 (2026-10-30, `c8041d19`) is where that broke: test 2
/// Run 41040277938 (2026-10-30, `c8041d19` -- the head that run
/// had; the PR was rebased on merge, so the same change sits on
/// develop as `bcd0bc61`, which `git patch-id --stable` gives the
/// same id for) is where that broke: test 2
/// printed `ambient=1.004113042 seconds gap=21.006908125 seconds
/// ceiling=3.012339126 seconds` and FAILED on the gap. Nothing was
/// blocking — the run's other 7974 tests passed. One sample of 1 s does
Expand Down
20 changes: 17 additions & 3 deletions Tests/macSCPCoreTests/ConnectionDiagnosticsJumpTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,23 @@ import Testing
/// **Hang bound, not a ceiling** (2026-09-28). Every case here races steps
/// against a budget through `DetachedProbe`, whose deadline is armed at
/// CREATION and so bounds how long a body waited for a thread as well as how
/// long it ran. `.timeLimit` is the net for a case that truly hangs, and it
/// is deliberately far larger than any step budget below, so no assertion in
/// this suite depends on it.
/// long it ran. `.timeLimit` is the net for a case that truly hangs, and no
/// assertion in this suite depends on it.
///
/// Until 2026-11-01 that sentence read "`.timeLimit` is the net for a case
/// that truly hangs, and it is deliberately far larger than any step budget
/// below, so no assertion in this suite depends on it". The withdrawn clause
/// was false of this file on the day it was written: the two roomy cases
/// below were already at `.seconds(600)` against this suite's 300 s bound,
/// `61850a9c` having set BOTH nine days earlier: `git blame` dates the
/// `stepTimeout:` line inside `theJumpsResolveNamesTheJumpsAddresses` and
/// the one inside `theJumpsSecretNeverReachesTheReport` to that commit, and
/// the "Hang bound, not a ceiling" heading above to `ae19a53a`. Named rather
/// than numbered
/// because the first draft of this sentence cited two line numbers and the
/// four lines it added moved both of them (CLAUDE.md, "prefer the symbol to
/// the line number"). What the sentence was about survives without the
/// withdrawn clause.
///
/// **Every case here takes that bound from the suite** (2026-10-05). Four
/// cases used to carry `@Test(.timeLimit(.minutes(1)))` of their own,
Expand Down
61 changes: 54 additions & 7 deletions Tests/macSCPCoreTests/ConnectionDiagnosticsTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,18 @@ import Testing
/// how long the body waited for a thread as well as how long it ran. On a
/// three-core CI runner a body that computes nothing can miss a small budget
/// entirely, and the step then settles as `notStarted`. `.timeLimit` is the
/// net for a case that truly hangs; it is deliberately far larger than any
/// step budget below, so no assertion in this suite depends on it.
/// net for a case that truly hangs, and no assertion in this suite depends
/// on it.
///
/// Until 2026-11-01 that sentence continued "it is deliberately far larger
/// than any step budget below, so no assertion in this suite depends on it",
/// and the clause quoted here is withdrawn: `roomy` is 600 s against this
/// suite's 300 s bound, so the budget is the larger of the two. The ordering
/// was never what made the claim true. A case asserting a dial's outcome
/// wants a budget its own deadline cannot win — see `roomy` — and a case
/// asserting the deadline sets a tight budget of its own and says so. Either
/// way the assertion is on the outcome, not on a clock, which is the
/// property the sentence was about.
@Suite("ConnectionDiagnostics", .timeLimit(.minutes(5)))
struct ConnectionDiagnosticsTests {
/// The secret the SSH dial cases hand the runner. Named rather than
Expand Down Expand Up @@ -70,6 +80,43 @@ struct ConnectionDiagnosticsTests {
/// the source text of what it checks.
private static let slashBearingSecret = "wJalrDiagnostics/TestSecret/Value"

/// The step budget for a case whose claim is the OUTCOME of a dial, not
/// how long the dial took to report it. Roomy for the reason the suite
/// note above gives: `DetachedProbe` arms its deadline at CREATION, so a
/// small budget bounds how long a body waited for a thread as much as how
/// long it ran, and on a starved runner the deadline settles the step
/// before the body says what it met.
///
/// 600 s, the figure `ResolveNamesTests.roomy` and the two roomy cases in
/// `ConnectionDiagnosticsJumpTests` already carry. One of those two,
/// `theJumpsSecretNeverReachesTheReport`, was moved off `.seconds(10)`
/// for exactly this failure on 2026-09-19 by `61850a9c`, whose diff is
/// one file, 10 insertions and 2 deletions — the other roomy case came
/// off `.seconds(30)` in the same commit — and which left the five
/// literals here standing. They fell 42 days later: CI run 41244948849
/// (2026-10-31, `e4e5d054`) read `.timedOut` in
/// `theSSHDialNeverPutsTheSecretInTheReport`, which fails that case's
/// anchor for the wrong reason. That run's own figures say why 10 s could
/// not hold — this command prints `8184 8307`:
///
/// gh run view 41244948849 --log-failed | grep -o 'ed after [0-9]*\.[0-9]* seconds' | sed -E 's/ed after ([0-9]+)\..*/\1/' | awk '{n++; if ($1>=10) k++} END {print k, n}'
///
/// 8184 of the 8307 cases carrying a duration line reported a wall clock
/// AT OR ABOVE the budget those five handed the runner — the pipeline's
/// comparison is `>= 10` on the integer seconds — against a run total of
/// 116.701 s; the case that went red reported 64.975 s itself. A budget
/// below almost every case's wall clock does not bound the step's work,
/// only whether its body was given a thread in time.
///
/// A roomy budget costs these five nothing, which is what makes it safe
/// here and unmeasured elsewhere: every target they use either answers
/// at once (the rig) or refuses at once (a closed loopback port, and the
/// skipped case never dials). None of them can park on this budget, so
/// raising it cannot lengthen a run. The same is NOT established for the
/// echo and trace cases, which send real probes — `docs/BACKLOG.md`,
/// "Wall-clock ceilings still in the tree", eighth shape.
private static let roomy: Duration = .seconds(600)

// MARK: - Resolve

@Test func resolveFindsALoopbackAddressForLocalhost() async {
Expand Down Expand Up @@ -1074,7 +1121,7 @@ struct ConnectionDiagnosticsTests {
descriptor: .descriptor(for: .ssh), values: values,
secrets: FixedSecretSource(value: secret), sessionID: UUID(), jump: nil,
throughput: DiagnosticThroughputSettings(),
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: .seconds(10)
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: Self.roomy
).run()

let dial = try #require(report.steps.first { $0.id == DiagnosticStepID.dial })
Expand All @@ -1101,7 +1148,7 @@ struct ConnectionDiagnosticsTests {
let report = await ConnectionDiagnostics(
descriptor: .descriptor(for: .ssh), values: values, secrets: nil, jump: nil,
throughput: DiagnosticThroughputSettings(),
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: .seconds(10)
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: Self.roomy
).run()

let dial = try #require(report.steps.first { $0.id == DiagnosticStepID.dial })
Expand Down Expand Up @@ -1341,7 +1388,7 @@ struct ConnectionDiagnosticsTests {
let report = await ConnectionDiagnostics(
descriptor: .descriptor(for: .s3), values: values, secrets: nil, jump: nil,
throughput: DiagnosticThroughputSettings(),
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: .seconds(10)
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: Self.roomy
).run()

let dial = try #require(report.steps.first { $0.id == DiagnosticStepID.dial })
Expand Down Expand Up @@ -1565,7 +1612,7 @@ struct ConnectionDiagnosticsTests {
let report = await ConnectionDiagnostics(
descriptor: .descriptor(for: .s3), values: values, secrets: nil, jump: nil,
throughput: DiagnosticThroughputSettings(),
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: .seconds(10)
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: Self.roomy
).run()

let dial = try #require(report.steps.first { $0.id == DiagnosticStepID.dial })
Expand All @@ -1584,7 +1631,7 @@ struct ConnectionDiagnosticsTests {
let report = await ConnectionDiagnostics(
descriptor: .descriptor(for: .webdav), values: values, secrets: nil, jump: nil,
throughput: DiagnosticThroughputSettings(),
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: .seconds(10)
internetSpeed: DiagnosticInternetSpeedSettings(service: .off), stepTimeout: Self.roomy
).run()

let dial = try #require(report.steps.first { $0.id == DiagnosticStepID.dial })
Expand Down
Loading
Loading