Repository navigation
Description
On #1885 the owner asked for the repo's scripts to use requests and bs4[html5lib], and pexpect or psutil where they read a subprocess's output. #1885 moves util/conda_vendor.py --refresh-pins to requests; this covers the other scripts.
requests, replacing urllib.request:
util/ci_timing.py:urlopenat:437-446, Link-header pagination at:412.ci-timing.ymlruns it on a bare interpreter and installs nothing (:57), so that job needs setup-python and an install step.util/update_python_versions.py:fetch()at:179-191. Thedetectjob incron-python-versions.ymlinstalls only PyYAML (:61-62).examples/generators/pcapng.py:_downloadand its Retry-After backoff at:832-917. Every leg that runs it already installs plainrequeststhrough thetestextra, butunit-tests.yml:473-476says it needs only the standard library, so update that comment. Keep the offline fallback and the retry behaviour.
markdown-it-py, replacing the pipe-splitting in util/update_python_versions.py parse_images() (:245-268), which reads the runner-images README's image table. An escaped pipe or a pipe inside a code span would shift its columns; read the table through a GFM-table parser instead (added after the owner's follow-up on #1885). It runs in the same detect job as fetch().
lxml, replacing xml.etree.ElementTree in util/ci_timing.py parse_junit() and util/pyshark_encap_map.py's PDML read, for its strict parsing (the owner's preference on #1885). Use a parser that resolves no entities and has no network access. lxml has wheels for CPython 3.8–3.15 (free-threaded included) and PyPy 3.9–3.11, none for GraalPy: add it to the test extra with a marker excluding GraalPy, gate its tests there, and install it in ci-timing.yml.
psutil, replacing the ps -A -o pgid= -o rss= polling in util/vendor_test_gate.py:157-171. psutil is in no extra, and tests/project runs this script's tests on every leg, including KTLO and experimental. So either confirm that psutil installs on every one of those legs, or keep ps as the fallback.
The tests that stub urllib move with their scripts: tests/project/test_ci_timing.py and tests/project/test_pcapng_download_retry_1658_unit.py.
Out of scope:
util/project_status.pystays ongh. Its job runs onpull_request_target, holdsPROJECT_TOKEN, and installs nothing by design.bs4andpexpecthave nothing to replace: no script outsidepcapkit/vendorparses HTML, and none drives an interactive program.
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsDone
Activity
ci(util): fetch with requests and poll memory with psutil in the scri…
JarryShaw commentedon Oct 17, 2026
I suppose these should also go into the test extras and/or Pipfile dev deps?
JarryShaw commentedon Oct 17, 2026
They are in the
testextra already. #1896 adds all three there:psutil, on CPython only;markdown-it-py;lxml, everywhere except GraalPy.The Pipfile's
[dev-packages]and thedevextra don't list any of them yet, so #1896 will add all three to both, with the same markers.requestsis already in both.ci(util): fetch with requests and poll memory with psutil in the scri…
JarryShaw commentedon Oct 17, 2026
Correction to my reply above: #1896 adds them to the Pipfile's
[dev-packages]and thetestextra, but not to thedevextra.process.rstdefinesdevas what pylint, mypy and autodoc must resolve forpcapkit/. That is whyrequestsandbs4are there:pcapkit/vendorimports them. Only the scripts underutil/andexamples/import these three, and the toolchain does not analyse those.ci(util): fetch with requests and poll memory with psutil in the scri…
ci(util): fetch with requests and poll memory with psutil in the scri…