Skip to content

ci(unit-tests): assign every test module a tier and run each tier where it belongs (#1958) - #1971

Merged
JarryShaw merged 1 commit into
mainfrom
ci/1958-test-tiers
Oct 20, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
ci/1958-test-tiers

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • ci — workflows or build tooling

Description of your pull request and other information

Phase 1 of #1958, implementing rulings A and B.

Every collected test id lands in exactly one tier, with none lost or duplicated. The tier lists are identical under 3.8, 3.10, 3.12 and 3.14.

Refs #1958

…re it belongs (#1958)

- tests/_tiers.py: module_tier() gives every test module one test tier,
  ALL or LATEST, from a module-level __tier__ constant read by AST, or
  else from the rule: LATEST when its run does not vary by interpreter
- Mark each real sweep LATEST and each hand-picked round trip ALL, and
  record a reasoned marker on each borderline module
- unit-tests.yml, experimental.yml: legs off latest ignore every LATEST
  module, from latest_tier_paths() at run time; integration off latest
  and pypcap-parity run fixture_tier_paths('ALL'); the *roundtrip* globs
  are gone. latest keeps ALL + LATEST, the pins stay in project-tests
- Tier guard, the #1732 and #1954 pins and _dependency_gates.py check
  the partition, the marker and the workflows' agreement with the rule
- workflows.rst: a Test Tiers section replaces the glob's description

Refs #1958
@JarryShaw JarryShaw added ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 20, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 20, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at d6fc7d507. A Sonnet review recollected the whole tree: 5511 ids, every one selected by exactly one job's tier. Every ALL id runs off latest and on latest. No module testing interpreter-specific behaviour (compat, enum, io, recursion limits) landed in LATEST. latest_tier_paths() is byte-identical on a real 3.8 and on 3.14, and every changed file parses as 3.8. A failed tier list fails the leg loudly; experimental warns. Doctoring the workflows or a marker trips the guards. It merges cleanly with main (including #1964's new module, which resolves to ALL).

One question for you, on ruling B. The four *_edge_roundtrip_unit modules and test_short_read_roundtrip_unit stay LATEST as tables of every #1202 edge case. Their rows are hand-written, though, so they could count as hand-picked and move to ALL. Which do you prefer?

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate needs: decision Waiting on the maintainer to decide — not blocked by other work and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 20, 2026
@JarryShaw JarryShaw moved this from In review to Needs decision in PyPCAPKit Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage: 90.07% (unit tier, Python 3.15, d6fc7d507, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 2 20 2 96.77%
pcapkit/const 19597 881 2618 867 92.11%
pcapkit/corekit 2504 70 792 36 96.54%
pcapkit/dumpkit 276 3 100 3 98.40%
pcapkit/foundation 2916 164 1016 34 93.59%
pcapkit/interface 138 7 52 5 93.68%
pcapkit/protocols 17411 224 4580 189 98.10%
pcapkit/toolkit 872 177 298 8 77.18%
pcapkit/utilities 522 2 158 4 99.12%
pcapkit/vendor 4431 2341 1018 156 43.64%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw merged commit dc906d6 into main Oct 20, 2026
60 checks passed
@JarryShaw
JarryShaw deleted the ci/1958-test-tiers branch October 20, 2026 01:57
@github-project-automation github-project-automation Bot moved this from Needs decision to Done in PyPCAPKit Oct 20, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

I merged without giving decision. What's your recommendation?

@JarryShaw

Copy link
Copy Markdown
Owner Author

My recommendation is to split them:

  • Move the four *_edge_roundtrip_unit tables to ALL (application, internet, link, transport). Their rows are hand-written edge cases that exercise parsing and building, which is ruling B's purpose: quality assurance on every interpreter. Together they take about 15 s on 3.15.
  • Keep test_short_read_roundtrip_unit on LATEST. It cuts each case at every offset, so it is a real sweep whose cost grows with its inputs (8.3 s).
  • Keep test_edge_roundtrip_deadline_1736_unit on LATEST. It tests the harness's deadline, not pcapkit.

If you agree, it is a four-marker follow-up.

@JarryShaw JarryShaw removed review: good-to-go Cross-review at the current head says ready; CI state is separate needs: decision Waiting on the maintainer to decide — not blocked by other work labels Oct 20, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

agreed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant