Skip to content

docs(conventions): a test-coverage convention - #1225

Merged
JarryShaw merged 1 commit into
mainfrom
docs/1203-test-coverage-convention
Oct 7, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
docs/1203-test-coverage-convention

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner
  • Searched for similar pull requests
  • Followed the coding style (make pylint, make mypy, make isort)
  • make test passes, and a test case covers the change
  • Added a changelog entry: N/A, added centrally

What is the purpose of your pull request?

  • docs — documentation only

Description of your pull request and other information

Closes #1203. Adds docs/source/contributing/conventions/test-coverage.rst (anchor test-coverage):

  • Round-trip invariants: the option and dispatch registry suites, a case for every code, known-failure tables checked both ways (regressions fixed, pre-existing defects recorded with file:line and an issue; test: round-trip coverage for every protocol code, dumper and lossless path #1202 extends coverage), the harness guards, and a spec check next to every cycle.
  • Raising coverage: the fail-without-fix proof, and how CI and a local run measure it.
  • Running the tests: one module per process, read the summary line, the tiers, reimport_once_per_class, util/run_unittest_leg.py.
  • Adding a protocol, option or dumper: what each must bring.

Linked from the conventions index and testing.rst. test-coverage is added to ANCHORS in tests/project/test_conventions_doc_claims.py; the page pins no figure. pytest tests/project -q: 389 passed, 1 skipped, 1314 subtests passed. Sphinx build of the worktree has no warnings on the changed pages; checked the rendered HTML for raw markup.

@JarryShaw JarryShaw added docs Pull requests that change documentation only (docs: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 88.91% (unit tier, Python 3.14, c960d1783, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18757 1034 2342 845 90.24%
pcapkit/corekit 1878 91 580 22 94.34%
pcapkit/dumpkit 142 0 42 0 100.00%
pcapkit/foundation 2442 102 852 39 94.38%
pcapkit/interface 112 7 40 5 92.11%
pcapkit/protocols 15777 190 4036 165 98.18%
pcapkit/toolkit 487 65 144 1 85.42%
pcapkit/utilities 429 4 122 4 98.55%
pcapkit/vendor 4409 2359 1006 158 42.84%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on 36b492353: NEEDS CHANGES (Sonnet reviewed; the author ran on Opus). Only accuracy nits. Everything the page names exists and behaves as described. The figures check out: 18 families, 324 cases, 5 SKIP entries and 35 expected failures. Every citation fits. The rendered page has no raw markup, and tests/project gives 389 passed.

Six statements are inaccurate:

  1. The unit tier is said to need only .[test], but CI installs .[test,DPKT,crypto,NGAP].
  2. The dumper byte-for-byte rule reads as current practice. It is not asserted yet; that is test: round-trip coverage for every protocol code, dumper and lossless path #1202.
  3. "A newly registered code that does not close is fixed rather than added" contradicts the suite's own instruction to add newly found defects to EXPECTED_FAILURES.
  4. The SKIP rationale doesn't fit EOL and NOP, which _make_tcp_options drops.
  5. KNOWN_DEGRADED entries have a different shape from the file:line-plus-issue rule.
  6. test_recorded_gaps_are_a_minority and CASE_TIMEOUT are not mentioned.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@JarryShaw
JarryShaw force-pushed the docs/1203-test-coverage-convention branch from 36b4923 to 498e4ac Compare October 7, 2026 18:00
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 7, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on 498e4ac00: NEEDS CHANGES (Sonnet, round 2; author Opus).

  • Must fix: test-coverage.rst:60-63 says a pre-existing defect "gets an entry and an issue, which is what the test's own failure message asks for". The message at test_option_roundtrip_unit.py:640-644 asks for the file:line, not an issue. The dispatch suite's message is the same. Suggested wording: "gets an entry naming the file:line that causes it, as the failure message asks, and an issue."
  • Confirmed: the other five round-1 fixes check out against the tree: the CI install line, the dumper target under test: round-trip coverage for every protocol code, dumper and lossless path #1202, the SKIP reasons, KNOWN_DEGRADED, and the guards. tests/project passes 389 tests, and the rendered page is clean.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
Adds docs/source/contributing/conventions/test-coverage.rst: round trips as
invariants with shrink-only known-failure tables, every change raising
coverage with a fail-without-fix proof, how to run the tests, and what a new
protocol, option or dumper must bring. Linked from the conventions index and
testing.rst; the anchor joins ANCHORS in test_conventions_doc_claims.py.

Closes #1203
@JarryShaw
JarryShaw force-pushed the docs/1203-test-coverage-convention branch from 498e4ac to c960d17 Compare October 7, 2026 18:15
@JarryShaw

Copy link
Copy Markdown
Owner Author

Verdict on c960d1783: GOOD TO GO. This head adds one change over 498e4ac00, which the round-2 Sonnet review checked in full: the reworded sentence round 2 asked for.

  • The new wording: a pre-existing defect "gets an entry naming the file:line that causes it, as the failure message asks, and an issue".
  • Checked against the source: the failure message at test_option_roundtrip_unit.py:649-656 asks exactly that.
  • Diff: test-coverage.rst, +2/−1.
  • Tests: tests/project passes, 389 passed.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 7, 2026
@JarryShaw
JarryShaw merged commit 4aca920 into main Oct 7, 2026
60 of 62 checks passed
@JarryShaw
JarryShaw deleted the docs/1203-test-coverage-convention branch October 7, 2026 19:05
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 7, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Pull requests that change documentation only (docs: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

docs(conventions): a test-coverage convention

1 participant