Release standalone native memory deterministically in the Python binding

docs/reports/0.2.4-native-memory.md

@@ -0,0 +1,128 @@
+# Standalone native memory ownership fix for 0.2.4
+
+## Root causes
+
+The Python projection did not own newly allocated databases returned by the
+native factories. Their destructors never released layers, cached features or
+file backing. Numeric column lookup also requested row zero through
+`tryGetFeature`, allocating a feature-pointer vector and a geometry-pointer
+vector for every row. Bulk columns now use the table base, field offset and
+row stride directly.
+
+`Table` and its cached column accessor retained each other. That cycle kept
+one complete backing allocation alive per read/write cycle until cyclic GC.
+Build-only references also retained native builders after finalization.
+
+## Lifetime design
+
+The buffer loader acquires an independent memoryview and `Py_buffer` in the
+same C call that constructs the native database. Its cookie holds that export
+and, on successful shared-memory loading, the mapping owner. The real native
+destructor first destroys its layers and features; its callback then releases
+the export and closes the mapping. Failed loads release the export and use one factory error;
+high-level shared-memory loaders close mappings they opened on failure.
+
+Factory results are owning SWIG proxies. Borrowed layers, features, references
+and chunks retain their parents. Chunk arrays set their actual chunk as base
+with `PyArray_SetBaseObject` in C; the owner argument is supplied internally by
+SWIG and cannot be overridden by a caller. Numeric column arrays retain their
+layer through the array interface. Raw arrays, slices, rows, tables and column
+accessors keep backing alive until their last reference is released.
+
+Tables strongly cache their column accessors and arrays. These projections
+share a backing-state object that never references a table, accessor or view.
+The state and write callbacks retain their RecordEngine, which maintains one
+registry of weak state and table references with direct lookup by name.
+Registrations survive while a state is retained, so string views still follow
+remaps after the table object is released. Write callbacks receive the
+backing state as an argument. Snapshot remapping updates shared state and
+replaces the table's accessor cache; retained checked accessors reject the old generation, including
+fields first accessed after remapping. There is no ownership cycle.
+
+Builders and push dispatch caches are discarded when their build phase ends.
+Mutable-build states contain native builders and row counts, without engine
+references; they are separate from the weak registry of public tables.
+`ObjectEngine` holds no duplicate backing reference and relies on reference
+counting.
+
+`RecordEngine.close()` is idempotent and drops engine-owned references. It does not
+invalidate a caller's `FdbViewOwner`; callers use `fdb.invalidate` when a lease
+ends. Engine-dependent writes after closure raise `RuntimeError` with
+`RecordEngine is closed.` Direct writable array access remains valid.
+Retained checked views still enforce owner validity. Buffer resize or
+mapping close raises `BufferError` while views pin its export. These contracts
+are documented in the [Python binding README](../../python/README.md).
+
+Unused unsafe borrowing interfaces are excluded from the Python projection.
+Invalid standalone envelope, descriptor, table and legacy string-table bounds
+are checked in Core before constructing layers. Optional column-section
+handling retains its existing behavior.
+
+## Memory measurements
+
+Measurements use macOS ARM64, CPython 3.10.17 and NumPy 2.2.6. The published
+0.2.3 wheel and a locally built 0.2.4 wheel run the same scripts. Each large case
+has 15,498,593 F32 rows, six complete write or read cycles, and cyclic GC
+disabled throughout. Closing and deleting the last reference releases native
+objects immediately; no `gc.collect()` is used in these measurements.
+
+| Path | Published 0.2.3 | 0.2.4 |
+| --- | ---: | ---: |
+| truncate, fill, save, close, delete | 361.18 MiB/cycle | 0.00 MiB/cycle |
+| file load, column copy, close, delete | 301.50 MiB/cycle | 0.00 MiB/cycle |
+
+Values are mean increments over cycles 2–6, measured using macOS physical
+footprint and rounded to 0.1 MiB by the script. All six 0.2.4 write samples
+are 257.5 MiB; all six read samples are 257.1 MiB. There is no observed
+post-warm-up growth at this resolution. First-cycle allocator
+warm-up is allowed; stable reuse is the criterion, rather than an immediate
+return to the import baseline. No cyclic collection or allocator relief occurs
+between samples.
+
+Automated memory regressions use 1,000,000 rows and ten cycles with GC disabled.
+After two warm-up cycles, all remaining samples must stay within 16 MiB above
+that baseline. The 0.2.3 native cache and backing leaks alone retain about
+16–20 MiB per cycle at this row count. Separate weak-reference and mapping assertions check
+immediate release independently of allocator RSS behavior.
+
+## Conversion timings
+
+The same benchmark takes the best of five runs using the same Python/NumPy
+versions and data. Timing is reported here; regression tests use deterministic
+cache-identity and weak-reference assertions rather than timing thresholds.
+
+| Operation | Published 0.2.3 | 0.2.4 |
+| --- | ---: | ---: |
+| `StringColumn.to_pylist`, 200,000 rows | 747.6 ms | 723.3 ms |
+| `chunk.as_array`, 200,000 calls | 71.7 ms | 70.1 ms |
+| `table.column.value`, 100,000 reads | 66.4 ms | 54.2 ms |
+| `table[i].value`, 100,000 reads | 321.4 ms | 318.8 ms |
+
+Column accessors and their arrays remain cached per table. Each measured path
+has latency at or below the published version in this run; repeated runs
+vary by a few percent in either direction.
+
+## Verification scope
+
+The regression suite covers caller-owned bytes, bytearray and mmap exports,
+shared mappings, invalid inputs, retained accessors/rows/columns, shared checked
+owners, and immediate builder release. Shutdown subprocesses cover daemon
+readers, module destructors and cyclic destructors; mapping closure is asserted
+on native destruction and on the cyclic case. Native ASan/UBSan tests revoke
+backing access with `PROT_NONE` before destruction and check release callbacks
+exactly once. Tests write file fixtures only in their assigned temporary or
+build directory.
+
+The source suite passes 403 Python tests on CPython 3.13; installed wheels
+each pass all 403 tests on CPython 3.10 and 3.13. Compileall and local
+sdist/wheel builds also pass, together with
+45 native ASan/UBSan tests, 21 Rust unit/integration tests plus one compile-fail
+doctest, 58 TypeScript/WASM tests and 117-symbol native and WASM ABI checks.
+The three shutdown scripts each return zero twice in both Python runtimes.
+The six retained-view/release scenarios and chained writes pass on CPython
+3.10. Release policy tests pass 105 cases and 98 subtests.
+Windows, Linux, macOS x86_64 and free-threaded Python have
+not been executed for this fix. Registry artifacts and the complete release
+matrix remain subject to the [publishing process](../releases/publishing.md).
+
+Raw logs, measurements and build archives are kept outside the source repository.

fastcarto/fastdb/include/fastdb.h

@@ -224,7 +224,7 @@ namespace wx
         void setTableBufferMaterialized(bool enabled);
         size_t tableBufferBytes();
         void setField(unsigned ix, const FastVectorDbFeatureRef* ref);
-        FastVectorDbFeatureRef* createFeatureRef(u32 ix=-1);
+        FastVectorDbFeatureRef* createFeatureRef(u32 ix=0xFFFFFFFF);
         void freeFeatureRef(FastVectorDbFeatureRef* ref);
         void addFeatureEnd();
         // List column build API
@@ -370,6 +370,9 @@ namespace wx
         const char*             getFieldDefn(unsigned ix, FieldTypeEnum &ft, double &vmin, double &vmax);
         size_t                  getFieldOffset(unsigned ix);
         size_t                  getFeatureByteSize();
+        // Borrowed base of the row table. Rows are separated by getFeatureByteSize().
+        // The address remains valid only while the database and its backing buffer live.
+        void*                   getTableDataAddress();
         void                    getExtent(double &minx, double &miny, double &maxx, double &maxy);
         u32                     getFeatureCount();
         void                    rewind();

fastcarto/fastdb/src/FastVectorDb.cpp

@@ -1,6 +1,7 @@
 #include "FastVectorDb_p.h"
 #include "FastVectorDbLayer_p.h"
 #include <cstring>
+#include <limits>
 #include <stdlib.h>
 
 #ifdef _WIN32
@@ -24,15 +25,61 @@ namespace wx
             std::memcpy(&value, source, sizeof(value));
             return value;
         }
+
+        bool valid_layer_bounds(const u8* layer, const layer_header_t& header)
+        {
+            const size_t tail = size_t(header.total_size) - sizeof(layer_header_t);
+            if (header.field_count > tail / sizeof(field_desc_ex_t))
+                return false;
+            const size_t descriptors = size_t(header.field_count) * sizeof(field_desc_ex_t);
+            const u8* data = layer + sizeof(layer_header_t) + descriptors;
+            const size_t size = tail - descriptors;
+            if (header.offset_table > size)
+                return false;
+            if (header.field_count) {
+                const auto last = load_unaligned<field_desc_ex_t>(
+                    layer + sizeof(layer_header_t) + descriptors - sizeof(field_desc_ex_t));
+                if (last.offset > std::numeric_limits<u64>::max() - last.size)
+                    return false;
+                const u64 stride = last.offset + last.size;
+                if (stride && header.feature_count > (size - header.offset_table) / stride)
+                    return false;
+                for (u16 i = 0; i < header.field_count; ++i) {
+                    const auto field = load_unaligned<field_desc_ex_t>(
+                        layer + sizeof(layer_header_t) + size_t(i) * sizeof(field_desc_ex_t));
+                    if (field.offset > stride || field.size > stride - field.offset)
+                        return false;
+                }
+            }
+            // Legacy string tables are read during construction, before the
+            // bounded optional column sections. Check their terminators here.
+            for (unsigned width : {1u, 2u}) {
+                u64 offset = width == 1 ? header.offset_strings : header.offset_wstrings;
+                if (offset > size || size - offset < sizeof(u32))
+                    return false;
+                const u32 count = load_unaligned<u32>(data + offset);
+                offset += sizeof(u32);
+                if (count > (size - offset) / width)
+                    return false;
+                for (u32 i = 0; i < count; ++i) {
+                    for (;;) {
+                        if (size - offset < width)
+                            return false;
+                        const bool terminated = width == 1 ? data[offset] == 0 :
+                            load_unaligned<u16>(data + offset) == 0;
+                        offset += width;
+                        if (terminated) break;
+                    }
+                }
+            }
+            return true;
+        }
     }
 
     FastVectorDb::Impl::Impl(void *pdata, size_t size, fnFreeDbBuffer fnFreeBuffer, void *cookie)
         : m_pdata(pdata), m_size(size), m_fnFreeBuffer(fnFreeBuffer), m_cookie(cookie)
     {
         u8 *ptr = (u8 *)pdata;
-        bool check_mask = strcmp((const char *)ptr, "FASTVectorDB0.1") == 0;
-        assert(check_mask);
-        m_mask_check_ok=check_mask;
         ptr += 16;
         const u32 count = load_unaligned<u32>(ptr);
         ptr += sizeof(u32);
@@ -113,17 +160,32 @@ namespace wx
 
     FastVectorDb *FastVectorDb::load(void *pdata, size_t size, fnFreeDbBuffer fnFreeBuffer, void *cookie)
     {
-        FastVectorDb::Impl *impl = new FastVectorDb::Impl(pdata, size, fnFreeBuffer, cookie);
-        if(impl->m_mask_check_ok)
-        {
-            auto db = new FastVectorDb(impl);
-            return db;
+        // Validate the standalone envelope once before reading its headers.
+        // The backing callback also releases invalid inputs.
+        bool valid = pdata && size >= 20 &&
+            std::memcmp(pdata, "FASTVectorDB0.1", sizeof("FASTVectorDB0.1")) == 0;
+        if (valid) {
+            const u8* ptr = static_cast<const u8*>(pdata) + 20;
+            size_t remaining = size - 20;
+            const u32 count = load_unaligned<u32>(static_cast<const u8*>(pdata) + 16);
+            for (u32 i = 0; i < count; ++i) {
+                if (remaining < sizeof(layer_header_t)) { valid = false; break; }
+                const auto header = load_unaligned<layer_header_t>(ptr);
+                if (header.total_size < sizeof(layer_header_t) || header.total_size > remaining ||
+                    !valid_layer_bounds(ptr, header)) {
+                    valid = false;
+                    break;
+                }
+                ptr += header.total_size;
+                remaining -= header.total_size;
+            }
         }
-        else
+        if (!valid)
         {
-            delete impl;
+            if (fnFreeBuffer) fnFreeBuffer(pdata, size, cookie);
             return nullptr;
         }
+        return new FastVectorDb(new Impl(pdata, size, fnFreeBuffer, cookie));
     }
 
     FastVectorDbFeature* FastVectorDb::tryGetFeature(wx::FastVectorDbFeatureRef* ref)

fastcarto/fastdb/src/FastVectorDbLayer.cpp

@@ -727,6 +727,10 @@ namespace wx
     {
         return m_table_line_size;
     }
+    void* FastVectorDbLayer::Impl::getTableDataAddress()
+    {
+        return m_table_data_ptr0;
+    }
     ///////////////////////////////////////////
 
     FastVectorDbLayer::FastVectorDbLayer(Impl *_impl) : impl(_impl) {}
@@ -842,6 +846,10 @@ namespace wx
     {
         return impl->getFeatureByteSize();
     }
+    void* FastVectorDbLayer::getTableDataAddress()
+    {
+        return impl->getTableDataAddress();
+    }
 
     // ---- List column read helpers ----
     chunk_data_t FastVectorDbLayer::Impl::getFieldAsListView_internal(u32 ifeature, u32 ix)

fastcarto/fastdb/src/FastVectorDbLayer_p.h

@@ -65,6 +65,7 @@ namespace wx
 
         size_t          getFieldOffset(unsigned ix);
         size_t          getFeatureByteSize();
+        void*           getTableDataAddress();
     private:
         void            move_next_geometry_ptr();
         size_t          get_geometry_like_size(const u8* pdata);

fastcarto/fastdb/src/FastVectorDb_p.h

@@ -20,7 +20,6 @@ namespace wx{
         size_t  m_size;
         fnFreeDbBuffer m_fnFreeBuffer;
         void* m_cookie;
-        bool    m_mask_check_ok;
         friend class FastVectorDb;
     };
 }