Skip to content

fix(utilities): encode pre-1970 datetimes on GraalPy without its lost second (#2134) - #2136

Merged
JarryShaw merged 2 commits into
mainfrom
test/2134-graalpy-failures
Oct 24, 2026
Merged

JarryShaw merged 2 commits into
mainfrom
test/2134-graalpy-failures

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • fix — corrects a defect

Description of your pull request and other information

Closes #2134. Each failure was root-caused on real GraalPy 25.4.4, the build CI uses:

  1. test_collection_removes_the_temporary_file_of_an_unclosed_reader: the test assumed CPython's immediate finalisation. GraalPy's gc.collect() runs only the weakref callbacks already queued, so the finaliser can land after it returns. The test now uses the test: sort the remaining GraalPy-only failures into defects and gates #1963 collect_until / FINALISER_GRACE helpers, moved into tests/_support.py.
  2. test_errors_are_cpythons: a GraalPy difference. Its timezone.fromutc raises ValueError: year 0 is out of range where CPython raises OverflowError. The test now expects GraalPy's class on GraalPy; the library is unchanged.
  3. test_mesg_id_before_1970: a real wrong-bytes defect on GraalPy.
    • GraalPy's aware datetime.timestamp() truncates the whole seconds toward zero. So a pre-1970 instant with a fraction comes out one second late: 1969-12-31 23:59:59.5 UTC gives 0.5, not -0.5.
    • The new pcapkit.utilities.datetime.totimestamp(dt) uses (dt - epoch).total_seconds() on GraalPy for an aware dt, and dt.timestamp() everywhere else, so CPython output is byte-identical.
    • It is used by the MH, PCAP frame and PCAP-NG timestamp encoders.

Tests:

  • The new tests/utilities/test_totimestamp_graalpy_2134_unit.py reproduces GraalPy's timestamp() on any interpreter. With the call sites reverted, it fails 5 cases on CPython.
  • On GraalPy 25.4.4, the target files give 40 passed.
  • On CPython, tests/project + tests/utilities give 2038 passed, and the tier guard gives 133 passed.

The GraalPy citations point at graalpython's source at graal-25.4.4; no upstream issue has been filed yet.

@JarryShaw JarryShaw added bug Issues reporting a defect (set by the bug report template; a default, not an assessment) test Pull requests that add or correct tests (test: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 3cef9a032 (Sonnet review, run on real GraalPy 25.4.4 as well as CPython).

  • The GraalPy bug: confirmed. datetime(1969,12,31,23,59,59,500000,tzinfo=utc).timestamp() gives 0.5 on GraalPy and -0.5 on CPython. Over 40,000 random aware values, GraalPy's raw .timestamp() was wrong on 13,262 of them. The cause is TimestampNode's ChronoUnit.SECONDS.between in DateTimeBuiltins.java at graal-25.4.4, and no upstream issue exists.
  • totimestamp: exact. It is bit-identical to dt.timestamp() on CPython, and matches CPython's value on GraalPy for all 40,000 values. GraalPy's naive local-time path is already correct, so passing it through is right.
  • Call sites: the three encoders were the only datetime.timestamp() calls in pcapkit/. On GraalPy, a 1969-12-31 23:59:58.25 instant now encodes correctly in the frame header and the MH option.
  • Finaliser and ValueError gates: confirmed on GraalPy. fromutc / astimezone raise ValueError, while plain datetime - timedelta still raises OverflowError.
  • Tests: CPython gives 316 passed and GraalPy 173 passed. With the call sites reverted, the new file fails 6 tests.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 91.55% (unit tier, Python 3.15, 586b819db, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 880 2618 866 92.12%
pcapkit/corekit 2589 75 816 37 96.48%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 2971 23 1036 25 98.80%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17418 219 4574 185 98.14%
pcapkit/toolkit 869 36 298 10 95.20%
pcapkit/utilities 504 7 146 5 98.15%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at 3cef9a032.

  • Review: the Sonnet review is GOOD TO GO, run on real GraalPy 25.4.4 as well as CPython.
  • CI: every blocking check passed (Required checks passed SUCCESS), and the 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: each plain-unittest leg this touches passed on a scratch merge into main 8fdc2fb33:
Leg Tests Failures
utilities 388 0
protocols/internet 721 0
foundation 730 0
protocols (rest) 1411 0

This closes #2134. The GraalPy experimental leg on main should go clean after it lands.

@JarryShaw

Copy link
Copy Markdown
Owner Author

why not ..autofunction:: for from/totimestamp?

@JarryShaw

Copy link
Copy Markdown
Owner Author

No good reason. Every sibling page under docs/source/pcapkit/utilities/ uses .. autofunction::, and timestamp.rst's hand-written .. function:: entry, from #2111, just duplicates the docstring.

I'll switch both fromtimestamp and totimestamp to .. autofunction:: on this PR. The entry-only prose (the :raises: lines and the versionadded note) moves into each docstring, so the rendered page keeps everything. A fresh review follows the push.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 24, 2026
… second (#2134)

GraalPy's aware datetime.timestamp() counts whole seconds toward zero, so an
instant before 1970 with a fraction encoded a second late in the MH replay
protection option, the PCAP frame header and PCAP-NG blocks.

- pcapkit.utilities.timestamp.totimestamp computes (dt - epoch).total_seconds()
  for an aware dt on GraalPy, and is dt.timestamp() everywhere else; the three
  encoders call it. The module's rst page renders both functions with
  autofunction, their docstrings carrying the prose and versionadded notes.
- test_fromtimestamp_windows_range_2099_unit: GraalPy's timezone.fromutc raises
  ValueError, not OverflowError, for a result before year 1; the test accepts it
  there.
- test_io_buffer_save_tempfile_2098_unit: wait for the reader's finaliser with
  collect_until, since GraalPy can run it after gc.collect() returns; the helper
  moves from test_extraction_ownership to tests/_support.
- New tests/utilities/test_totimestamp_graalpy_2134_unit emulates GraalPy's
  timestamp() on any interpreter.

Target tests pass on CPython 3.14 and GraalPy 25.4.4; tier guard passes.
@JarryShaw
JarryShaw force-pushed the test/2134-graalpy-failures branch from 3cef9a0 to d5c0e7c Compare October 24, 2026 04:47
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at d5c0e7c53 (Sonnet delta review), rebased onto main ad71ef592.

  • Docs: timestamp.rst now uses .. autofunction:: for both fromtimestamp and totimestamp, laid out like chardet.rst.
  • Nothing lost: every piece of the old entries' prose (description, Raises:, and versionadded with :issue:) is now in the docstrings.
  • Scope: the range-diff against 3cef9a032 touches only timestamp.rst, the two docstrings and the commit message.
  • Checks: a full Sphinx build with the pinned docs requirements renders the page cleanly. The timestamp tests give 17 passed.

The ready note follows once CI finishes on this head.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at d5c0e7c53.

  • Review: GOOD TO GO, from the Sonnet review on real GraalPy and the delta review of the autofunction switch.
  • CI: every blocking check passed (Required checks passed SUCCESS). The 3.8–3.10 KTLO legs carry no warnings.
  • Ordering: the utilities (388/0), protocols/internet (721/0), foundation (730/0) and protocols (rest) (1411/0) legs passed at 3cef9a032. Since then only the timestamp.rst page and two docstrings changed, and the head was rebased onto ad71ef592.

This closes #2134.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Debating if we should rename the timestamp module as time or datetime to match the house convention of reusing stdlib names.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Recommendation: rename it to pcapkit/utilities/datetime.py, inside this PR.

  • datetime over time. Both functions convert between datetime.datetime objects and POSIX timestamps, mirroring datetime.fromtimestamp() and datetime.timestamp(). The stdlib time module never touches datetime objects.
  • House precedent. The convention already holds: utilities/logging.py, utilities/warnings.py, corekit/io.py, corekit/enum.py and others all reuse stdlib names and import their stdlib namesake absolutely.
  • No compatibility cost. The module arrived with fix(protocols): convert capture timestamps without the platform's fromtimestamp limits (#2099) #2111 today and is in no tag, not even 1.5.0b14. Per the moved-names ruling, a beta-only path is simply removed: no re-export and no versionremoved.
  • Why this PR. fix(utilities): encode pre-1970 datetimes on GraalPy without its lost second (#2134) #2136 already rewrites the module (it adds totimestamp) and its autofunction page. The rename touches 10 files: 3 protocol modules, the MH schema, the module itself, its docs/source/pcapkit/utilities/*.rst page and tests. A separate PR would conflict with this one.

Labelled needs: decision. Say "go" and I'll push the rename here, or "keep" to leave the name as is.

@JarryShaw JarryShaw added the needs: decision Waiting on the maintainer to decide — not blocked by other work label Oct 24, 2026
@JarryShaw JarryShaw moved this from In review to Needs decision in PyPCAPKit Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Take datetime then

@JarryShaw

Copy link
Copy Markdown
Owner Author

Taking datetime. A worker is moving pcapkit/utilities/timestamp.py to pcapkit/utilities/datetime.py on this branch. That covers every importer, the docs page and the tests, with no old-path re-export, plus a test pinning that pcapkit.utilities.timestamp no longer imports. The new head gets a fresh review before I post another ready note.

@JarryShaw JarryShaw removed the needs: decision Waiting on the maintainer to decide — not blocked by other work label Oct 24, 2026
@JarryShaw JarryShaw moved this from Needs decision to In review in PyPCAPKit Oct 24, 2026
Per the owner's ruling on #2136, pcapkit.utilities.timestamp becomes
pcapkit.utilities.datetime, reusing the stdlib name as utilities.logging
and utilities.warnings do. The old path was added by #2111 and is in no
release, so it is deleted with no re-export.

- move the module and its docs page, and update the toctree
- update the MH, PCAP frame and PCAP-NG importers and the tests
- test that the new module exposes fromtimestamp and totimestamp, imports
  the stdlib datetime, and that the old path no longer imports
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 24, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 586b819db (Sonnet review of the datetime rename).

  • No reference to the old path remains, and nothing was left at it: no shim, no versionremoved. The module is in no tag.
  • Inside the module, import datetime is the stdlib. m.datetime is datetime holds.
  • I read Sphinx 9.1's find_obj source: ~datetime.datetime can't match a pcapkit.utilities.datetime.* object, so it resolves via intersphinx. Not confirmed by a docs build.
  • The new test fails 3 of 3 on d5c0e7c53 and passes here. tests/project passes (1750 + 163). The import order in the four importers is alphabetical.
  • Non-blocking: the page title stays "Timestamp Conversion", and the layout test is LATEST where its sibling is ALL.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 24, 2026
@JarryShaw
JarryShaw merged commit 73e7799 into main Oct 24, 2026
64 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 24, 2026
@JarryShaw
JarryShaw deleted the test/2134-graalpy-failures branch October 24, 2026 13:33
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 24, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Issues reporting a defect (set by the bug report template; a default, not an assessment) test Pull requests that add or correct tests (test: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

test(graalpy): three failures on the GraalPy experimental leg after #2105 and #2111

1 participant