Repository navigation
Release standalone native memory deterministically in the Python binding - #43
Merged
Merged
Conversation
The Python binding never owned databases returned by the standalone native factories, so their layers, per-row caches and file backing stayed allocated. Each truncate/fill/save/close cycle on a 15,498,593-row F32 table retained about 361 MiB, and each load/read/close cycle about 302 MiB. Numeric column lookup also built per-row feature and geometry caches through tryGetFeature(0). Factories now return owning proxies. load_xbuffer takes its buffer export in the same C call that constructs the database; the native release callback drops the export, and closes a shared mapping, after the layers are destroyed. Chunk arrays set their chunk as NumPy base in C, and numeric columns use the table address and field offset directly. Tables share a backing state with their cached accessors and views, so nothing refers back to a table; tables keep their engine alive and the engine registers them weakly. RecordEngine.close() releases engine references, and later writes that need the engine raise a clear error. Core validates standalone envelopes, table bounds and legacy string tables before constructing layers. With cyclic GC disabled, both cycles stay flat after the first. Validation on macOS ARM64: 403 Python tests on CPython 3.10 and 3.13 wheels, compileall, uv build, 45 native ASan/UBSan tests, Rust and TypeScript/WASM tests, both 117-symbol ABI checks, and subprocess tests for interpreter shutdown and cyclic collection. See docs/reports/0.2.4-native-memory.md.
On Linux CPython 3.12 and 3.13t, the module-destructor case reached ndarray.sum() after finalization had disabled imports, and NumPy's runtime import of its reduction helper raised ImportError. The mapped column itself was still readable. Verify the whole column with tobytes() and bytes.count, and report through a pre-bound os.write, so the destructor needs no imports.
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Python binding never owned databases returned by the standalone native factories, so their layers, per-row caches and file backing stayed allocated for the life of the process. With fastdb4py 0.2.3, each
truncate/fill/save/closecycle on a 15,498,593-row F32 table retained about 361 MiB, and each fileload/column copy/closecycle about 302 MiB. A long-running consumer that writes periodic frames and checkpoints ran out of memory this way. Numeric column lookup also built per-row feature and geometry caches throughtryGetFeature(0), andTableformed a reference cycle with its cached column accessor.Factories now return owning proxies.
load_xbuffertakes its buffer export in the same C call that constructs the database; the native release callback drops the export, and closes a shared mapping, after the layers are destroyed, so interpreter shutdown and cyclic collection cannot unmap memory that is still in use. Chunk arrays set their chunk as NumPy base in C, and numeric columns use the table address and field offset directly. Tables share a backing state with their cached accessors and views, so nothing refers back to a table; tables keep their engine alive and the engine registers them weakly.RecordEngine.close()releases engine references, and later writes that need the engine raiseRuntimeError: RecordEngine is closed.The contract is documented inpython/README.md. Core validates standalone envelopes, table bounds and legacy string tables before constructing layers. Unused borrowing interfaces are removed from the Python projection, andObjectEnginerelies on reference counting.With cyclic GC disabled, both cycles above stay flat after the first. Column access, string conversion and chunk arrays stay within a few percent of 0.2.3; cached
table.columnaccess is slightly faster. Files written by 0.1.12 and 0.2.3 decode identically.Validation on macOS ARM64: 403 Python tests against installed CPython 3.10 and 3.13 wheels, compileall,
uv build, 45 native ASan/UBSan tests, Rust and TypeScript/WASM tests, both 117-symbol ABI checks, and subprocess tests for daemon readers, module teardown and cyclic collection. An independent review reproduced the shutdown crashes of an earlier candidate; they return 0 here. The release_tools and clean-cut checks were replayed locally on this commit. Windows, Linux and macOS x86_64 run in CI. Details are indocs/reports/0.2.4-native-memory.md.Version metadata for 0.2.4 follows in a separate PR.