Skip to content

ci(unit-tests): cache the tshark .deb files and retry apt on a fallback mirror (#1432) - #1438

Merged
JarryShaw merged 1 commit into
mainfrom
ci/1432-apt-fetch-stall
Oct 8, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
ci/1432-apt-fetch-stall

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • ci — workflows or build tooling

Description of your pull request and other information

Closes #1432. Both apt sites (engine-tests, pypcap-parity) now plan, restore, seed, install, collect and save:

  • .deb cache (actions/cache, which works under contents: read). The key is ImageOS + arch + package set + a digest of apt-get install -s's Inst lines, so any new version makes a new key. restore-keys reuses an older set, apt checks each file's hash against the signed index, and only the installed versions get saved.
  • Fallback mirror on the retry. azure is dropped from /etc/apt/apt-mirrors.txt, because apt only fails over on an error. The partial download resumes.
  • The install loop, its 60s/180s bounds, the 10-minute cap and the ::warnings are unchanged. TestAptRetry pins the bounds.
  • The docs' unit-tests.yml:N citations move by +188 lines (pinned by TestDocCitationsResolve).

Checked in ubuntu:24.04. A cold run at 600 kB/s with a 50s bound failed attempt 1, switched to archive.ubuntu.com, fetched only 11.2 of the 24.4 MB left, and succeeded. A warm run printed Need to get 0 B (7s).

What to look for in this PR's CI: every job is a cold miss. Expect Plan … to print a key and Save … to store it. A rerun should show Seeded N .deb files and Need to get 0 B.

  • Only parity's Python 3.10 cell saves the cache (APT_CACHE_SAVE), so the matrix cells no longer race on one key.

@JarryShaw JarryShaw added ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 89.45% (unit tier, Python 3.14, bc76d3e5c, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18791 968 2352 869 90.90%
pcapkit/corekit 2038 89 634 25 94.76%
pcapkit/dumpkit 234 2 84 2 98.74%
pcapkit/foundation 2637 98 928 48 94.78%
pcapkit/interface 112 7 40 5 92.11%
pcapkit/protocols 16615 221 4474 185 98.04%
pcapkit/toolkit 539 74 168 3 84.58%
pcapkit/utilities 429 4 122 4 98.55%
pcapkit/vendor 4409 2342 1006 157 43.25%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Oct 8, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: NEEDS CHANGES at 650fa45df (Sonnet cross-review; the author ran on Opus). The code is sound, but it has no tests.

  • Works, re-checked by me: PyShark job 113530529150 logs Cache hit for: apt-debs-ubuntu24-X64-PyShark-5f14b79b393a0f71 and then Need to get 0 B/26.5 MB of archives, so the install finishes in 7 s instead of stalling past 180 s.
  • Reviewer confirmed:
    • The key (ImageOS, arch, APT_SET, sha256 of the sorted Inst lines) cannot give a stale exact hit.
    • restore-keys falls back only to the same image and package set.
    • apt rejects any seeded file whose hash does not match the signed index.
    • The fallback drops the azure line from apt-mirrors.txt only when another http mirror remains.
    • The worst case still fits the 10-minute step cap.
    • No new permissions, so the 4 contents: read callers are unaffected.
    • The renumbered doc citations resolve.
  • Needs: no test pins the new logic. A grep of tests/ for apt-mirrors, apt-debs and APT_SET finds nothing. Add tests for:
    • the mirror drop, including the azure-only and first-attempt-succeeded cases
    • the APT_PACKAGES ↔ install-list agreement
    • the cache key format
  • Optional: the two parity matrix cells race to save one key (Unable to reserve cache … another job may be creating this cache). It is harmless, but noisy.
  • Unexercised: the live slow-mirror retry path has not run in CI.

The worker is adding the tests.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 8, 2026
…ck mirror (#1432)

The apt install in engine-tests and pypcap-parity kept failing on a slow,
not dead, azure.archive.ubuntu.com: ~75 kB/s on 2026-10-08, so timeout 180
killed both attempts at the 23.7 MB libwireshark17t64, three times.

- Cache the downloaded .deb files with actions/cache, keyed on ImageOS,
  arch, the job's package set and a digest of `apt-get install -s`'s
  `Inst` lines, so a warm run downloads nothing and any new version is a
  new key. Only the installed versions are collected, and only one job
  per key saves (APT_CACHE_SAVE), so the parity cells do not race.
- Before the retry, drop azure from /etc/apt/apt-mirrors.txt, since apt
  only fails over on an error; the partial download resumes.
- The install step's update/install loop, its 60s/180s bounds, the
  10-minute cap and the ::warning annotations are unchanged.
- Add tests/project/test_workflow_apt_cache.py: mirror drop, package
  lists vs install operands, key format, one saver per key.
- Shift the docs' unit-tests.yml line citations by the added 188 lines.

tests/project: 420 tests OK. Both jobs' scripts dry-run in ubuntu:24.04.

Closes #1432
@JarryShaw
JarryShaw force-pushed the ci/1432-apt-fetch-stall branch from 650fa45 to bc76d3e Compare October 8, 2026 20:52
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 8, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict: GOOD TO GO at bc76d3e5c (round 2, Sonnet cross-review; the author ran on Opus).

  • Tests now pin the new logic. tests/project/test_workflow_apt_cache.py has 15 tests that run each step's real run: text, extracted from the workflow at test time and executed under bash -e with apt/sudo stubbed.

    • Re-derived by me: 15 OK on this head; 25 failures against main's unit-tests.yml.
    • tests/project: 420 OK.
  • Save race fixed. Only the Python 3.10 parity cell saves the cache (APT_CACHE_SAVE). The matrix uses quoted "3.10"/"3.11", so exactly one cell saves.

  • This head's CI. PyShark logs Cache hit for: apt-debs-ubuntu24-X64-PyShark-5f14b79b393a0f71 and Need to get 0 B/26.5 MB. Both parity cells hit the key the earlier head saved (0 B/27.9 MB). Every install step finished in ≤ 14 s.

  • Non-blocking, mutation gaps the reviewer found. These mutations went undetected:

    • removing the cache-hit != 'true' gate on collect/save
    • the seed if: condition
    • the collect step's installed-version filter
    • the seed cp
    • the plan step's apt-get update retry

    The seed and collect scripts never run under test. If any of these broke, the cache would quietly degrade, but the install itself would still be correct.

  • Unverified. The fallback-mirror retry has not yet fired in live CI.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 8, 2026
@JarryShaw
JarryShaw merged commit 16c87f9 into main Oct 8, 2026
40 checks passed
@JarryShaw
JarryShaw deleted the ci/1432-apt-fetch-stall branch October 8, 2026 22:36
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 8, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

ci(unit-tests): the tshark apt fetch of libwireshark17t64 still stalls past the 180s bound on both attempts

1 participant