Skip to content

fix(protocols): convert capture timestamps without the platform's fromtimestamp limits (#2099) - #2111

Merged
JarryShaw merged 1 commit into
mainfrom
fix/2099-platform-free-fromtimestamp
Oct 24, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/2099-platform-free-fromtimestamp

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • fix — bug fixes

Description of your pull request and other information

Closes #2099. Windows' gmtime_s covers only 1970-01-01 through 3000-12-31 UTC, and CPython turns its EINVAL into OSError. So on the Windows leg (job 120605026883), PCAP-NG timestamps of year 1 and year 9999 fell back to 1970 with invalid timestamp.

  • New helper: pcapkit.utilities.compat.fromtimestamp splits the float exactly as CPython does (math.modf, microseconds rounded half to even, carry), adds it to the epoch, and applies tz.fromutc. It never calls the C runtime, so it needs no version gate.
  • Users:
    • PCAPNG._read_timestamp: the bug itself.
    • Frame.read: for consistency, since its range never fails on Windows.
    • MH's _read_opt_timestamp and MesgIDOption.post_process: they had the same gap for NTP times before 1970 and PMIPv6 times after 3000.
  • Error classes: these match the stdlib on Linux: ValueError for NaN or a year outside 1–9999, OverflowError past time_t or out of range after the zone. Beyond a C int of seconds, glibc raises OSError while the helper raises ValueError. PCAP-NG, the only reader that can reach such values, catches both.
  • Linux unchanged: 200,049 values across 6 zones match the real fromtimestamp. The 504 that differ are that documented case.
  • Test: test_fromtimestamp_windows_range_2099_unit makes fromtimestamp raise OSError(22) outside 0–3000, as Windows does. Without the fix 17 checks fail, including the CI log's invalid timestamp: -62135510400. With it, 8 tests and 60 subtests pass, and rebuilds stay byte-exact. The PCAP, PCAP-NG, MH, dumpkit, toolkit, compat and tests/project files pass.

@JarryShaw JarryShaw added bug Issues reporting a defect (set by the bug report template; a default, not an assessment) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 73406d0e1 (Sonnet review).

  • Exactness: the helper matched datetime.fromtimestamp(ts, tz) on 80,180 cases across 5 zones (UTC, +5:45, −12, New York, London). The cases spanned year 1–9999, ±.5 µs boundaries, ints, bools, ±2⁶³, inf and NaN. Value, offset, fold and tzinfo identity all agree. The only mismatch is the documented OSError against ValueError at 1e17 s and beyond: 35 cases.
  • Call sites: all four already passed an aware tz. Frame.read catches ValueError on a uint32 input, and PCAP-NG catches all three error types, so nothing changes for callers.
  • MH: MesgIDOption.post_process is a real fix, since NTP seconds before 1970 give negative epochs that Windows rejects. _read_opt_timestamp is consistency only; its uint32 seconds stay before 2106. The reviewer called it harmless, so it stays.
  • Tests: half-up rounding planted as a mutation fails 8 checks, and base pcapkit fails the claimed 17. The PCAP-NG, header/frame, 1340 range, compat and all test_mh* files pass.
  • UNVERIFIED: Windows and 3.8. The arithmetic is version-independent (round is half-even).

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 91.54% (unit tier, Python 3.15, ebbd68a06, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 880 2618 866 92.12%
pcapkit/corekit 2580 75 814 37 96.46%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 2944 23 1028 25 98.79%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17418 219 4574 185 98.14%
pcapkit/toolkit 869 36 298 10 95.20%
pcapkit/utilities 498 7 144 5 98.13%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at 73406d0e1.

This closes #2099.

@JarryShaw

Copy link
Copy Markdown
Owner Author

pcapkit.utilities.compat is mainly for backward compatibility implementations, and re-exports. I don't think it's the right place for this next method.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Agreed, compat is for backports and re-exports. I'm moving fromtimestamp into a new pcapkit/utilities/timestamp.py. It isn't a backport: it is a conversion that has to work the same on every platform, and the PCAP, PCAP-NG and MH readers all need it, so a protocol module isn't the right home either. The name avoids shadowing the stdlib time and datetime.

compat keeps no re-export, as with other moved names. The docs entry moves to its own page. Say if you'd prefer another home and I'll move it there. I'll post the new head here when it's ready.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 23, 2026
…mtimestamp limits (#2099)

datetime.fromtimestamp(ts, tz) breaks the instant down with the C
library's gmtime(), and the Windows CRT's gmtime_s takes nothing before
1970 or after 3000, so in-range year-1 and year-9999 PCAP-NG timestamps
took the invalid-timestamp fallback and were dated 1970.

- new pcapkit.utilities.timestamp.fromtimestamp, not in compat, which
  holds backward-compatibility shims: split the float as CPython does
  (modf, half-even microseconds, carry), add to the epoch, tz.fromutc;
  ValueError/OverflowError as CPython on 64-bit Linux
- use it in PCAPNG._read_timestamp, Frame.read, MH._read_opt_timestamp
  and MesgIDOption.post_process (MH had the same Windows gap)
- document it on its own utilities page; the two tests that faked the module's datetime now fake
  the converter
- new test emulates Windows' range and pins exact datetimes for both
  extremes, if_tzone rendering, MH pre-1970/post-3000, and the fallback

Linux results identical to the stdlib on 1.2M fuzzed (value, tz) pairs.
@JarryShaw
JarryShaw force-pushed the fix/2099-platform-free-fromtimestamp branch from 73406d0 to ebbd68a Compare October 23, 2026 22:09
@JarryShaw

Copy link
Copy Markdown
Owner Author

Moved as requested, and rebased onto main (eef5aeabd). Head 73406d0e1 → ebbd68a06.

  • New home: fromtimestamp now lives in the new pcapkit/utilities/timestamp.py, documented on its own page, docs/source/pcapkit/utilities/timestamp.rst. The utilities index links it.
  • compat.py: identical to main again, with no re-export, so git grep finds no compat import of it.
  • Callers: PCAP-NG, Frame, MH and the MH schema import the helper from its new home.
  • Tests: they pass, and the 200,049 × 6-zone comparison with the stdlib still has 0 mismatches.

A short delta review is running on the new head.

@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 23, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at ebbd68a06 (Sonnet delta review of the move).

  • Same code: the fromtimestamp body in pcapkit/utilities/timestamp.py is byte-identical to the earlier compat version, and compat.py has no diff against main.
  • No re-export: git grep finds only direct imports from pcapkit.utilities.timestamp. utilities/__init__.py re-exports no submodule names, so leaving it out is consistent, and refactor!: remove every surviving re-export of a moved name or module #1744's no-re-export test passes.
  • Module and docs: the module has a .. module:: docstring, __all__, and stdlib-only imports, so there is no cycle. It has a toctree entry and its own page.
  • Tests: the 2099, pcapng, header/frame, refactor!: remove every surviving re-export of a moved name or module #1744, all_exports, isort, import-boundary, internal-refs and module-isolation tests pass. The docs page is checked by reading it, not by a Sphinx build.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 23, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at ebbd68a06.

  • Review: the Sonnet delta review of the move to pcapkit/utilities/timestamp.py is GOOD TO GO.
  • CI: every blocking check passed (Required checks passed SUCCESS). The 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: the protocols plain-unittest ordering leg (without tests/protocols/internet) passed on a scratch merge into main (82b901b91). It ran 1411 tests with 0 failures.

This closes #2099.

@JarryShaw
JarryShaw merged commit 0274b93 into main Oct 24, 2026
64 checks passed
@JarryShaw
JarryShaw deleted the fix/2099-platform-free-fromtimestamp branch October 24, 2026 00:19
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 24, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 24, 2026
JarryShaw added a commit that referenced this pull request Oct 24, 2026
Per the owner's ruling on #2136, pcapkit.utilities.timestamp becomes
pcapkit.utilities.datetime, reusing the stdlib name as utilities.logging
and utilities.warnings do. The old path was added by #2111 and is in no
release, so it is deleted with no re-export.

- move the module and its docs page, and update the toctree
- update the MH, PCAP frame and PCAP-NG importers and the tests
- test that the new module exposes fromtimestamp and totimestamp, imports
  the stdlib datetime, and that the old path no longer imports
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Issues reporting a defect (set by the bug report template; a default, not an assessment)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(protocols): PCAP-NG timestamps outside Windows' fromtimestamp range render as 1970

1 participant