Skip to content

fix(foundation): absorb the pyshark crash a reaped tshark causes on Python 3.9–3.12 - #2642

Merged
JarryShaw merged 1 commit into
mainfrom
fix/2633-pyshark-reap
Nov 2, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/2633-pyshark-reap

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • fix — a bug fix

Description of your pull request and other information

On Python 3.9–3.12, extract(engine='pyshark') raised a spurious TSharkCrashException: … retcode: 255. The cause is CPython gh-87744: asyncio's kill() goes through Popen.poll(), which reaps an already-exited tshark, so asyncio reports 255. The owner chose option 1 on #2633, so PyShark.close() now absorbs the crash only under a strict version of that option's condition:

  • Python 3.9–3.12;
  • a TSharkCrashException, matched by class;
  • the capture has reached EOF;
  • asyncio reports 255 while the reaping Popen holds the real status, 0.

It then logs at DEBUG, clears the stale processes and finishes pyshark's close.

Requiring the Popen's 0 is stricter than "255 at EOF". pyshark sets EOF after a mid-stream crash too, and a truncated capture with a real exit 14 would otherwise be absorbed; it still raises. One residual case is documented: a genuine exit 255 that the watcher reaps first reads as 0. asyncio's own Unknown child process warning still prints, but the extraction succeeds.

Evidence:

  • A Sonnet cross-review reproduced the fix with real tshark: 3.12 went from 2/100 to 0/100 (14/20 to 0/20 with the kill widened), and 3.9 and 3.11 behave the same way.
  • Every condition was mutation-tested. The fix has no effect on 3.13.

Closes #2633

@JarryShaw JarryShaw added this to the PyPCAPKit v1.6 milestone Nov 2, 2026
@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) review: good-to-go Cross-review at the current head says ready; CI state is separate labels Nov 2, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at be29223b0. A Sonnet cross-review read CPython 3.8 to 3.12's Popen.send_signal/_internal_poll and asyncio's _process_exited, which confirmed that the reaping Popen keeps the real exit status. It reproduced the fix with real tshark on 3.9, 3.11 and 3.12. It killed every single-condition mutant, and showed that a real failure (exit 14 at EOF) still raises 30 out of 30 times. After the fix absorbs a crash, there is no hang, no leaked task and no change to the hooks.

@JarryShaw JarryShaw moved this to In review in PyPCAPKit Nov 2, 2026
@JarryShaw
JarryShaw force-pushed the fix/2633-pyshark-reap branch from be29223 to 06ae67a Compare November 2, 2026 07:21
@JarryShaw

Copy link
Copy Markdown
Owner Author

Fix round: be29223b0 → 06ae67a50, same base 18544c3c9.

  • Red Project tests: test_the_stubs_cover_every_import_of_them found no stub for pyshark.capture.capture. The new typings/pyshark/capture/capture.pyi declares TSharkCrashException, the one name the engine imports from it. With the stub removed, that test fails on exactly this case.

Verdict stays good to go.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Nov 2, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

NEEDS CHANGES at 06ae67a50: CI on Python 3.15 found a regression.

A fix round is in progress. CI on this head is cancelled until then.

@JarryShaw
JarryShaw force-pushed the fix/2633-pyshark-reap branch from 06ae67a to b3346a3 Compare November 2, 2026 08:20
@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Nov 2, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Fix round and rebase: 06ae67a50 → b3346a37c, now on 16ccc4373. The interdiff is one entry in the constants sweep test, and the gate itself is unchanged.

  • The sweep has no special case for version gates, so a sys.version_info comparison containing 9 always hits the shared constant RESOLUTION_PLACES. The entry goes in UNRELATED, next to PyPCAPFile's (3, 12) ceiling, following that precedent.
  • I checked it myself: the sweep passes on the new head (14 passed), and the worker showed it fails without the entry.

GOOD TO GO at b3346a37c.

@JarryShaw

Copy link
Copy Markdown
Owner Author

The red Python 3.15 leg is main's, not this PR's: its only failure is the re-export test, which #2641 fixes. The constants sweep passes. The verdict stays good to go.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Coverage: 94.08% -- tests failure, so this is partial (unit tier, Python 3.15, b3346a37c, Unit Tests run failure)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 20561 177 2832 158 98.53%
pcapkit/corekit 2697 74 854 37 96.65%
pcapkit/dumpkit 281 3 100 3 98.43%
pcapkit/foundation 3495 27 1216 32 98.75%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 21416 218 5650 196 98.46%
pcapkit/toolkit 961 28 342 10 96.32%
pcapkit/utilities 512 7 144 5 98.17%
pcapkit/vendor 5256 2260 1198 194 53.61%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Let's make no Python version gate on this behaviour.

@JarryShaw
JarryShaw force-pushed the fix/2633-pyshark-reap branch from b3346a3 to b4c2f2a Compare November 2, 2026 14:41
@JarryShaw

JarryShaw commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner Author

Rebased onto main (3e93eea1a, which includes #2641): b3346a37c → b4c2f2a76. The patch-id is identical (da18a7aba2e2) and there were no conflicts. The old head's only red was the re-export failure inherited from main, which #2641 fixed. The owner's comment above supersedes the verdict; see the next comment.

@JarryShaw

Copy link
Copy Markdown
Owner Author

The owner asks, paraphrased, for the reaped-tshark absorption to run on every Python version. _lost_tsharks loses its (3, 9) <= sys.version_info[:2] < (3, 13) gate, and its other checks decide on their own. The constants-sweep UNRELATED entry for that 9 goes with it, and the tests that were limited to 3.9–3.12 run everywhere.

I posted the rebase note above without reading this first. That note's "good to go" was wrong, and I've edited it. The label is review: needs-changes until the fix round lands, and CI on this head is cancelled.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Nov 2, 2026
@JarryShaw
JarryShaw force-pushed the fix/2633-pyshark-reap branch from b4c2f2a to fdfadc5 Compare November 2, 2026 14:52
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Nov 2, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Fix round: b4c2f2a76 → fdfadc5ed, same base 3e93eea1a. The absorption runs on every Python.

  • _lost_tsharks loses its version gate. The remaining checks decide the case: TSharkCrashException, a non-empty _running_processes, _eof_reached, and asyncio 255 against Popen 0. The lookups of private attributes now use getattr, so a missing attribute re-raises the original error.
  • The constants-sweep entry for the 9 is gone. The real-poll test reproduces the loss on any version by calling Popen.poll(). A new test checks that no Python version gates the absorption.

The label is review: running while a Sonnet re-review checks the round.

@JarryShaw

Copy link
Copy Markdown
Owner Author

NEEDS CHANGES at fdfadc5ed, from a Sonnet re-review.

  • Required: no test pins the getattr fallbacks that replaced the gate. Reverting any of them, or dropping the non-empty _running_processes guard, leaves every test passing. Tests where each missing attribute still re-raises the original crash are being added.
  • Confirmed: real pyshark and tshark runs absorb the lost exit status on 3.12 with no false absorption, and the module passes on 3.8 through 3.14. Re-adding the gate fails it.
  • Nits: a timed word in a comment, a debug log that always names gh-87744, and a stub comment narrower than pyshark.

CI on this head is cancelled until the fix round lands.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Nov 2, 2026
…o lost (#2633)

- A Popen.poll that reaps tshark before the event loop handles its SIGCHLD
  leaves asyncio's child watcher no child, so it reports 255 and pyshark's
  close raises TSharkCrashException. pyshark hits it at end of file: on
  CPython 3.9-3.12 asyncio's kill polls through Popen.send_signal (gh-87744).
- PyShark.close logs that crash at DEBUG instead, on every Python, only when
  the capture reached EOF and every listed tshark reports 255 to asyncio but
  0 to the Popen that reaped it. A real failure, a 255 the Popen also read,
  or any other error still raises. The capture is emptied for its __del__.
- Stub pyshark.capture.capture. New unit tests pair each absorbed crash with
  one that still raises, and lose real exit statuses to a real poll.

3.12, 4x100 natural + 40 widened extractions: 97 crashes before, 0 after.
@JarryShaw
JarryShaw force-pushed the fix/2633-pyshark-reap branch from fdfadc5 to c76ed35 Compare November 2, 2026 15:11
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Nov 2, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Fix round: fdfadc5ed → c76ed3515, rebased onto main (1e04dd0d4).

  • New tests: one where no tshark is listed, and one covering five process shapes: no _transport, a None transport, no get_extra_info, get_extra_info returning None, and a Popen with no returncode. Each must re-raise the exact crash, and each of the reviewer's mutants fails one of them.
  • Missing get_extra_info: a transport without it also re-raises the original crash.
  • Debug log: states the evidence (asyncio 255, Popen 0) rather than naming gh-87744.

The label is review: running while the same reviewer re-checks the round.

@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at c76ed3515, from the same Sonnet reviewer.

  • Mutants: every mutant the last round named now fails a test, as does one that re-raises a fresh copy instead of the original crash. With the gate re-added or the fix removed, the module fails. It passes on 3.8–3.14.
  • Real runs on 3.12, with pyshark 0.6 and tshark: main crashes in 5 of 200 extractions, this head in 0.
  • Not blocking: the capture-level _running_processes and _eof_reached lookups are not pinned against a missing attribute. pyshark 0.6 always sets both.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Nov 2, 2026
@JarryShaw
JarryShaw merged commit ce88220 into main Nov 2, 2026
61 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Nov 2, 2026
@JarryShaw
JarryShaw deleted the fix/2633-pyshark-reap branch November 2, 2026 15:31
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Nov 2, 2026
JarryShaw added a commit that referenced this pull request Nov 2, 2026
…hecksums and PyShark's lost tshark status into 1.6.0 (#2449)

- Breaking, in the HTTP entry: building HTTP from fields takes version= and
  from_data refuses a dict, where 1.5.x built HTTP/1.1 and took one; HTTPv1
  refuses preface, which 1.5.x wrote ahead of the message (#2651). The
  versioned-base entry and the conventions entry point at it.
- Folded into the checksum entry: IGMP and GRE verify and compute their
  checksums, verify_message, GRE's checksum_present, and a cut-short GRE
  frame reading INVALID (#2654). The contextvars entry drops in_fragment,
  private to the checksum module and in no release.
- Fixed: the PyShark engine absorbs the TSharkCrashException asyncio's lost
  tshark exit status raises, on every Python (#2642).
- No entry for the test-only #2592.
- CHANGELOG.md regenerated; util/changelog_md.py --check passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(foundation): the pyshark engine raises a spurious TSharkCrashException on Python 3.9–3.12

1 participant