Skip to content

fix(dumpkit): drop redundant escape/date workarounds dictdumper 0.8.5 fixed - #2445

Merged
JarryShaw merged 1 commit into
mainfrom
worktree-agent-a1721672dad9275e1
Oct 30, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
worktree-agent-a1721672dad9275e1

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

Summary

Closes #2442

Test plan

  • Installed dictdumper==0.8.5 in a fresh venv inside this worktree; confirmed pcapkit.__file__ and dictdumper.__file__ both resolve inside the worktree before testing.
  • Read the installed dictdumper/json.py (encode_string()) and dictdumper/plist.py (encode_xml(), _format_date()) to confirm which defects 0.8.5 actually fixed before deleting any workaround -- and found one it did not fix (aware-datetime-to-UTC conversion), which stays as a smaller _append_date override.
  • coverage run -m pytest tests/dumpkit -- 76 passed, 1793 subtests passed (98% coverage on pcapkit/dumpkit/common.py, the three uncovered lines are pre-existing and unrelated).
  • Stashed the common.py fix and confirmed the rewritten test_common_unit.py assertions fail against the pre-fix code (proving they test the right thing), then restored it.
  • Ran everything that imports pcapkit.dumpkit directly (~24 files, including tests/integration/) -- all green except 4 pre-existing, unrelated failures in tests/project/test_docs_internal_refs_1940_unit.py, confirmed to fail identically on the pre-fix tree.
  • mypy, pylint (project's own flag set), isort clean on pcapkit/dumpkit/common.py; no new warnings on the changed test files.

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 30, 2026
@github-actions github-actions Bot added this to the PyPCAPKit v1.5 milestone Oct 30, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 30, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

NEEDS CHANGES (cross-review, Opus vs. authoring model).

Two required fixes, both small:

  1. common.py:586 — bare (GitHub issue #1448) in a docstring needs the :issue: role; fails test_docs_issue_pr_roles_1103_unit.py::test_rendered_text_uses_roles as written.
  2. The fix(dumpkit): JSON output corrupts characters above U+FFFF (dictdumper json.py:290-295) #1261 surrogate-pair fix lost coverage for JSON mapping keys: upstream's encode_string() still has the single-codepoint-escape defect for astral characters, so an emoji-bearing key now round-trips wrong. Needs a per-key surrogate-pair split on the JSON path, plus a regression test for keys (the existing test only covers values).

Non-blocking nit: common.py:334 cites "DictDumper#1152" — that's actually PyPCAPKit's own issue number; the real upstream issue is DictDumper#121.

Escaping fix, UTC conversion, and the untouched keep_none/tree_numbers workarounds all independently confirmed correct. Relayed both required fixes to the original agent.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 30, 2026
@JarryShaw
JarryShaw force-pushed the worktree-agent-a1721672dad9275e1 branch from 79ffcaa to 1cca7c8 Compare October 30, 2026 06:10
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 30, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Both required fixes applied (new head 1cca7c8b6): the :issue: role on the docstring, and a new _append_object override for JSON mapping keys above U+FFFF (split to a UTF-16 surrogate pair before handing to upstream's encode_string), with a regression test. The optional nit (wrong DictDumper issue number) fixed too. Verified directly: diff matches the report, and tests/dumpkit/ (77 passed/1793 subtests) reproduces exactly once tested under a scratch venv with dictdumper 0.8.5 — my first local run falsely failed because the shared repo venv still has the old 0.8.4.post6. Cross-review re-dispatched on the new head.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 30, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Review: GOOD TO GO at 1cca7c8b6 (Opus, different model from the author).

Both round-1 fixes independently re-derived as correct: the :issue: role satisfies the roles test, and the surrogate-pair key fix is sound against the full astral code-point range (0 mismatches vs. json.dumps semantics across all 1,048,576 code points) and writes no raw surrogate to the file under dictdumper 0.8.5. keep_none/tree_numbers confirmed byte-identical since main. tests/dumpkit/: 77 passed/1833 subtests. Integration: 21 passed/32 subtests.

Non-blocking: conda/requirements.txt and examples/benchmark/requirements-common.txt still pin dictdumper 0.8.4.post6 (the conda one matters for create-release.yml, which reads that file directly rather than waiting on the regen cron); top-level block names still go through unescaped encode_string; no test covers a non-str key beside an astral key.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 94.44% (unit tier, Python 3.15, 7e61bfb96, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19609 133 2620 120 98.84%
pcapkit/corekit 2605 74 818 37 96.52%
pcapkit/dumpkit 281 3 100 3 98.43%
pcapkit/foundation 3070 24 1072 26 98.79%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17593 193 4678 179 98.32%
pcapkit/toolkit 924 36 334 10 95.55%
pcapkit/utilities 502 7 144 5 98.14%
pcapkit/vendor 4449 1854 1020 167 55.11%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: good-to-go Cross-review at the current head says ready; CI state is separate labels Oct 30, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

New regression found by CI, not by either review round — the Python 3.15 leg is failing: tests/project/test_rfc_constants_one_home_1739_unit.py::TestNotReinlined::test_sweep flags pcapkit/dumpkit/common.py:328: codepoint > 65535 as an unexplained copy of a homed constant (0xFFFF/65535 coincidentally matches an existing RFC-pinned constant elsewhere, same class of false-positive LINUX_SLL's PR hit with its own 0xFFFF). Both review rounds ran narrow dumpkit/integration test selections and missed tests/project, so this slipped through GOOD TO GO. Needs a one-line NOT_A_COPY/UNRELATED registry entry in that test file, same pattern as the existing entries. Relaying the fix.

… fixed

dictdumper 0.8.5 shipped PR #123 (key+value escaping for plist/json,
fixing #772/DictDumper#125 and the json key-quoting defect DictDumper#121)
and PR #124 (whole-second <date> formatting, fixing #1448). pyproject.toml's
`dictdumper~=0.8.0` resolves to 0.8.5, so pcapkit.dumpkit.common's own
workarounds ran on top of the upstream fix and double-escaped/mis-dated
every plist/json report.

- Remove `escape_strings`/`escape_json_keys`/`escape_keys`/`escape_key()`
  and the escaping tail of `object_hook` -- the writer now escapes a
  string or mapping key itself, for both plist/xml and json.
- Shrink `_append_date` to just the UTC conversion dictdumper still does
  not do on its own; delegate the (now-fixed) whole-second formatting to
  `super()._append_date()`.
- Keep `keep_none` (#1257) and `tree_numbers` (#1266) untouched -- those
  defects are still open upstream.
- Bump `dictdumper~=0.8.0` to `>=0.8.5` in pyproject.toml and Pipfile.
- Add `_append_object`/`_surrogate_pairs` for json output: dictdumper's
  json `_append_object` escapes a mapping key through its module-level
  `encode_string()` directly, bypassing the overridable `_append_string`
  that already fixes #1261 for values, so a key holding an astral
  character still round-tripped wrong. Split such a key into its UTF-16
  surrogate pair before handing it to the upstream writer.
- Cite the upstream json key-escaping fix as DictDumper#121, the real
  upstream issue, not DictDumper#1152, which is this repo's own issue
  number.
- Use the `:issue:` Sphinx role, not a bare `#1448`, in `_append_date`'s
  docstring, per this repo's role-vs-plain-comment convention.
- Update tests/dumpkit/test_common_unit.py's escaping-behaviour unit
  tests for the new (non-escaping) object_hook contract, refresh stale
  prose in two integration test docstrings that described the removed
  workarounds as still-open upstream bugs, and add a regression test for
  a json mapping key carrying an astral character.

Verified against the installed dictdumper 0.8.5's actual source
(dictdumper/json.py's encode_string(), dictdumper/plist.py's encode_xml()
and _format_date()) before removing anything, and confirmed the new
test_common_unit.py assertions fail against the pre-fix code. Full
tests/dumpkit/ suite and everything importing pcapkit.dumpkit green
(77 passed / 1793 subtests); tests/project/test_docs_issue_pr_roles_1103_unit.py
green; mypy/pylint/isort clean on changed files.

Closes #2442
@JarryShaw
JarryShaw force-pushed the worktree-agent-a1721672dad9275e1 branch from 1cca7c8 to 7e61bfb Compare October 30, 2026 06:47
@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 30, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Fixed and verified directly (new head 7e61bfb96): added the NOT_A_COPY registry entry for common.py's codepoint > 65535, following the exact pattern of the existing adjacent entry for this same file. test_sweep passes (14 passed/141 subtests). Restoring GOOD TO GO — the earlier verdict's substance still stands, this was a review-coverage gap (narrow test selection missed tests/project), not a flaw in the actual fix.

@JarryShaw
JarryShaw deleted the worktree-agent-a1721672dad9275e1 branch October 30, 2026 13:44
JarryShaw added a commit that referenced this pull request Oct 30, 2026
Fixed a double-escaping regression: dictdumper 0.8.5 fixed the
mapping-key/XML-escaping defects pcapkit.dumpkit.common had its own
workarounds for, so the two fixes stacked (#2442, #2445).
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(dumpkit): dictdumper 0.8.5 fixed the defects common.py works around; now double-escaping

1 participant