Predictable zip MD5s depend on which zlib build the interpreter links

[rtibbles]

❌ This issue is not open for contribution. Visit Contributing guidelines to learn about the contributing process and how to find suitable issues.

Target branch: main

Observed behavior

• create_predictable_zip output bytes depend on which zlib build the running interpreter links.
• That MD5 is the file identity Studio stores, so the bytes cannot be allowed to vary.
• zlib-ng emits different DEFLATE bytes than reference zlib: 39 of 40 real test fixtures differ.
• Divergence is size-dependent — inputs under roughly 256 bytes compress identically.
• Everything above that threshold diverges.
• No zlib-ng configuration reproduces reference zlib.
• A sweep of 495 combinations (level ×11, strategy ×5, memLevel ×9) over a 40-input corpus found no setting matching on all inputs.
• The best managed 10/40, all of them trivially small.
• zlib-ng is not self-consistent either: 2.2.5 and 2.3.3 disagree with each other.

Errors and logs

Fedora 44's system interpreter, which ships zlib-ng as the system libz:

$ /usr/bin/python3 -c "import zlib; print(zlib.ZLIB_RUNTIME_VERSION, getattr(zlib,'ZLIBNG_VERSION','absent'))"
1.3.1.zlib-ng 2.3.3
$ rpm -qf /lib64/libz.so.1
zlib-ng-compat-2.3.3-3.fc44.x86_64

The same 956-byte index.html zipped under each implementation:

zlib    3d2ab30ed1ad16cc417fe9f11211a747
zlib-ng 632bc86e46359cc425f55d9eaa58fa0d

Rebuilding every pinned case in tests/test_zip.py with zlib-ng as the compressor:

PASS (undetected!)  nested_text    220f0d36a5150d3912a0eebee2738d80
PASS (undetected!)  reversed       5f3c72e2f32c5b7919cd6c31e5f169cd
PASS (undetected!)  binaryFiles    18ba9ca5ba2ed25ada40111fcc055a82
PASS (undetected!)  nested_binary  0fdfc3bd5b661ae3cde677d542426386
PASS (undetected!)  simple_binary  461a08dc38d2b7dae48c2bc2e641b958
5/5 pinned fixtures still pass under zlib-ng

Every fixture member is 4–14 bytes, below the divergence threshold. The regression tests that exist to protect these hashes cannot detect the one thing that breaks them.

Expected behavior

• The same input content produces the same archive bytes on every supported interpreter and platform.
• The invariant is reference zlib raw DEFLATE at level 6, memLevel 8, default strategy.
• Studio's frontend independently produces those same bytes via pako ({level: 6, windowBits: -15, memLevel: 8, strategy: 0}).
• Every file already in Studio was hashed from them.

User-facing consequences

• Chefs on an affected interpreter produce archives whose MD5s differ from the ones Studio already holds for identical content.
• Deduplication misses, so identical content is stored twice.
• Every affected node's checksum changes.
• Learners re-download content that did not change.
• That cost lands hardest on the low-bandwidth, offline-first deployments Kolibri targets.
• Nothing errors at runtime, because neither side compares hashes during upload.
• The mismatch stays silent until someone compares hashes by hand.
• The same five pinned MD5s are asserted in tests/test_zip.py and in Studio's zipFile.spec.js.
• Neither test suite can detect a compressor change on either side of the contract.
• With Drop Python 3.9, add Python 3.14 #720's guard in place the failure becomes loud instead.
• ricecooker then refuses to process HTML5, ePub and KPUB content on any zlib-ng interpreter, including Fedora's system Python.
• The guard fires inside create_predictable_zip, so chefs fail mid-run after everything has downloaded.

Steps to reproduce

1. On an interpreter linked against zlib-ng — Fedora 40+ system Python, or CPython 3.14 on Windows — confirm the build: python -c "import zlib; print(zlib.ZLIB_RUNTIME_VERSION)" reports 1.3.1.zlib-ng.
2. Create a directory containing a single file larger than ~256 bytes, e.g. 30 repetitions of <p>Kolibri learning content</p>.
3. Run create_predictable_zip on that directory and take the MD5 of the result.
4. Repeat steps 2–3 on an interpreter linked against reference zlib — any uv-managed CPython, which statically bundles zlib 1.3.1.
5. Observe that the two MD5s differ.
6. Replace the file with an 11-byte one and observe that they now agree.
7. Note that uv run --group test pytest tests/test_zip.py passes on both interpreters throughout.

Without access to a zlib-ng interpreter, pip install zlib-ng and assign zipfile.zlib = zlib_ng.zlib_ng before building reproduces the same divergence.

Context

• ricecooker main, plus PR Drop Python 3.9, add Python 3.14 #720 (branch issue-719-621d1e).
• Verified affected: Fedora 44 system Python 3.14.6 against zlib-ng-compat 2.3.3.
• Reported in Drop Python 3.9, add Python 3.14 #720: CPython 3.14 on Windows links zlib-ng.
• Verified unaffected: uv-managed CPython 3.13.9 (python-build-standalone), which statically bundles reference zlib 1.3.1.
• That is why CI and local development have not caught this.
• The affected surface is a packaging trend, not one platform.
• Fedora has already switched its system libz.
• CPython's Windows 3.14 build is the same change arriving from a different direction.

Possible approaches

Not decided. These two are genuine alternatives.

Nothing off the shelf produces reference-zlib bytes, measured on a 28-input corpus of repo fixtures:

• gzip -6: 18/28 exact.
• Info-ZIP zip -6: 18/28 exact.
• zstd --format=gzip -6: 0/28.
• deflate wheel (libdeflate): 1/28.
• imagecodecs.zlib_encode: 1/28, because it links system libz.

So option 1 in either form means building madler zlib ourselves. Both forms mean never taking a deflate-output-changing update to it.

1. Carry our own reference zlib, so output stops depending on the interpreter's build. Distributed as:
   a. A shared library as package data, driven by ctypes.

   • Verified byte-exact against its linked library on 37/37 inputs, at 32.6 MB/s.
   • No C extension, so wheels vary by platform only.
   • Needs no compiler and no user action.
     b. A C extension wheel.
   • Fastest and most idiomatic.
   • Adds a cibuildwheel matrix over 3.10–3.14 × 3 OS × 2 arch.
   • sdist installs then need a compiler.

2. Stop deriving content identity from compressed bytes.

   • Hash sorted (path, sha256(member)) pairs, so the compressor stops mattering.
   • Ends the problem class.
   • Studio must accept a second identity alongside the file MD5.

Acceptance Criteria

• create_predictable_zip produces identical bytes for identical input on reference-zlib and zlib-ng interpreters.
• Those bytes match the MD5s currently pinned in tests/test_zip.py.
• Those bytes match what Studio's frontend produces via pako.
• A non-reference zlib build is detected by comparing compressed output against pinned vectors, not by reading version strings.
• Forks that do not self-identify as zlib-ng are caught.
• The pinned fixtures include at least one member over 1 KB.
• The pinned fixtures fail when the compressor is swapped for zlib-ng.
• CI covers an interpreter linked against zlib-ng.
• windows-latest, 3.14 is restored to the CI matrix, or this issue records why it stays excluded.
• HTML5, ePub and KPUB content processes successfully on Fedora system Python.

AI usage

Investigated with Claude Code. Every measurement here was produced by running it locally rather than taken from the model's knowledge.