Feature/serialization

AGENTS.md

@@ -98,8 +98,11 @@ each implementation.
 2. **Non-owning indexes**: indexes use external immutable data where
    appropriate, commonly `std::span<const std::uint64_t>`. The caller keeps
    that data alive and stable for the index lifetime.
-3. **SIMD conditional compilation**: use `PIXIE_AVX512_SUPPORT` and
-   `PIXIE_AVX2_SUPPORT` guards with a scalar fallback.
+3. **SIMD primitive dispatch**: keep `PIXIE_AVX512_SUPPORT` and
+   `PIXIE_AVX2_SUPPORT` feature guards, SIMD implementations, and scalar
+   fallbacks inside the low-level primitive header that owns the operation.
+   Callers use one unconditional primitive API and must not branch on Pixie
+   SIMD feature macros.
 4. **Target domain**: optimize for bit sequences and indexes up to `2^64`
    bits.
 5. **Platform**: Linux/Unix is the current target platform. `MappedFile`
@@ -115,6 +118,20 @@ each implementation.
    invalidated by owner resize or destruction.
 8. **Optional adapters**: third-party implementations stay behind their build
    option and must not become a library runtime dependency.
+9. **Serialization and residency**: `BinaryReader` is a parse-time cursor over
+   contiguous virtual address space; a byte span does not imply that every page
+   is resident in RAM. A reader over `MappedFile` relies on normal OS demand
+   paging. Zero-copy deserializers retain views into the backing storage, not
+   the reader, so the backing owner must remain alive and immutable while
+   queries access those views directly.
+10. **Serialization output buffering**: `BinaryWriter` writes through an
+    explicit seekable sink and owns only a fixed-size staging buffer, or borrows
+    one supplied by the caller. `VectorOutputSink` is the explicit
+    whole-artifact-in-memory choice; `SpanOutputSink` and POSIX
+    `io::FileOutputSink` provide bounded-memory destinations. Call
+    `BinaryWriter::finish()` before consuming the sink. Current framing uses
+    backpatching, so non-seekable pipes, sockets, and compression streams would
+    require a future counting pass or format change.
 
 ### Why Header-Only?
 
@@ -227,8 +244,9 @@ ctest --preset release -L rank_select_tests
 The registered test executables are `bit_algorithms_unittests`,
 `rank_select_unittests`, `rank_select_tests`, `benchmark_tests`, `test_rmm`,
 `tree_tests`, `wavelet_tree_tests`, `storage_tests`,
-`excess_positions_tests`, `excess_record_lows_tests`, and `rmq_tests`. Run an
-executable directly only when debugging a focused Google Test filter.
+`serialization_tests`, `excess_positions_tests`, `excess_record_lows_tests`,
+and `rmq_tests`. Run an executable directly only when debugging a focused
+Google Test filter.
 
 ### Test Configuration via Environment Variables
 
@@ -327,13 +345,17 @@ The script configures and builds the `coverage` preset, deletes stale
 
 ### Modifying SIMD Code
 
-1. Keep an AVX-512 implementation, AVX2 implementation where useful, and a
+1. Keep feature detection and SIMD/scalar dispatch local to the low-level
+   primitive that owns the operation. Callers must use its stable API
+   unconditionally; do not leak `PIXIE_*_SUPPORT` guards into data-structure
+   code.
+2. Keep an AVX-512 implementation, AVX2 implementation where useful, and a
    scalar fallback behind the existing feature guards.
-2. Include `<immintrin.h>` only in translation units or headers that use SIMD
+3. Include `<immintrin.h>` only in translation units or headers that use SIMD
    intrinsics; do not make it a generic benchmark dependency.
-3. Validate the fallback with the `asan` preset or an isolated
+4. Validate the fallback with the `asan` preset or an isolated
    `DISABLE_AVX512=ON` build.
-4. Build the relevant benchmark preset before claiming a performance result.
+5. Build the relevant benchmark preset before claiming a performance result.
    Use `benchmarks-profile` for hardware counters when supported by the host.
 
 ### Adding Tests

CMakeLists.txt

@@ -138,6 +138,16 @@ target_compile_features(pixie INTERFACE cxx_std_20)
 target_include_directories(pixie INTERFACE
         $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>)
 
+# Keep Pixie's own targets warning-clean without imposing -Werror on consumers.
+function (pixie_enable_project_warnings target)
+    if (CMAKE_CXX_COMPILER_ID MATCHES "Clang|GNU")
+        target_compile_options(${target} PRIVATE
+                -Wall
+                -Wextra
+                -Werror)
+    endif ()
+endfunction ()
+
 if (PIXIE_DIAGNOSTICS)
     target_compile_definitions(pixie INTERFACE PIXIE_DIAGNOSTICS)
     target_link_libraries(pixie INTERFACE spdlog::spdlog_header_only)
@@ -232,6 +242,15 @@ if (PIXIE_TESTS)
             gtest_main
             ${PIXIE_DIAGNOSTICS_LIBS})
 
+    add_executable(serialization_tests
+            src/tests/serialization_tests.cpp)
+    target_include_directories(serialization_tests
+            PUBLIC include)
+    target_link_libraries(serialization_tests
+            gtest
+            gtest_main
+            ${PIXIE_DIAGNOSTICS_LIBS})
+
     add_executable(excess_positions_tests
             src/tests/excess_positions_tests.cpp)
     target_include_directories(excess_positions_tests
@@ -281,11 +300,13 @@ if (PIXIE_TESTS)
             tree_tests
             wavelet_tree_tests
             storage_tests
+            serialization_tests
             excess_positions_tests
             select512_experimental_tests
             excess_record_lows_tests
             rmq_tests)
     foreach (test_target IN LISTS PIXIE_TEST_TARGETS)
+        pixie_enable_project_warnings(${test_target})
         gtest_discover_tests(${test_target}
                 DISCOVERY_MODE PRE_TEST
                 TEST_PREFIX "${test_target}."
@@ -373,6 +394,15 @@ if (PIXIE_BENCHMARKS)
             benchmark_main
             ${PIXIE_DIAGNOSTICS_LIBS})
 
+    add_executable(serialization_benchmarks
+            src/benchmarks/serialization_benchmarks.cpp)
+    target_include_directories(serialization_benchmarks
+            PUBLIC include)
+    target_link_libraries(serialization_benchmarks
+            benchmark
+            benchmark_main
+            ${PIXIE_DIAGNOSTICS_LIBS})
+
     add_executable(bp_tree_benchmarks
             src/benchmarks/bp_tree_benchmarks.cpp)
     target_include_directories(bp_tree_benchmarks
@@ -425,6 +455,27 @@ if (PIXIE_BENCHMARKS)
             benchmark
             benchmark_main
             ${PIXIE_DIAGNOSTICS_LIBS})
+
+    set(PIXIE_BENCHMARK_TARGETS
+            rank_select_benchmarks
+            rmm_benchmarks
+            rmm_btree_benchmarks
+            rmq_benchmarks
+            louds_tree_benchmarks
+            wavelet_tree_benchmarks
+            serialization_benchmarks
+            bp_tree_benchmarks
+            dfuds_tree_benchmarks
+            alignment_comparison_benchmarks
+            excess_positions_benchmarks
+            select512_benchmarks)
+    if (PIXIE_THIRD_PARTY_BACKENDS)
+        list(APPEND PIXIE_BENCHMARK_TARGETS
+                rmm_sdsl_benchmarks)
+    endif ()
+    foreach (benchmark_target IN LISTS PIXIE_BENCHMARK_TARGETS)
+        pixie_enable_project_warnings(${benchmark_target})
+    endforeach ()
 endif ()
 
 # ---------------------------------------------------------------------------

agentic/local/cpp/skills/benchmarks/EXAMPLES.md

@@ -53,6 +53,37 @@ methodology, align tables visually in source, and exclude local JSON, failed
 probes, and before/after experiment history. Persist other results only for an
 explicitly experimental implementation that has a registered benchmark.
 
+## Serialization Benchmarking
+
+The serialization benchmark binary is:
+
+```bash
+./build/benchmarks/serialization_benchmarks
+```
+
+It separates primitive `BinaryReader` and `BinaryWriter` throughput, zero-copy
+byte-span traversal, fixed-span and growable-vector sinks, warm memory-mapped
+reads, page-cache file writes, framed records with backpatching, and end-to-end
+rank/select, RmM, RMQ, and wavelet-tree serialization. Structure rows report
+logical artifact throughput through `bytes_per_second`, plus `artifact_bytes`
+and `items` counters. Setup, source generation, mapping, and initial structure
+construction are outside the timed region.
+
+Use a pinned Release run and filter the subsystem being investigated. For
+example:
+
+```bash
+taskset -c 0 ./build/benchmarks/serialization_benchmarks \
+  --benchmark_filter='^(BM_Binary(Reader|Writer)|BM_(RankSelect|RmM|Rmq|WaveletTree))' \
+  --benchmark_report_aggregates_only=true \
+  --benchmark_display_aggregates_only=true
+```
+
+The mapped-reader rows warm every mapped page before timing. The file-writer
+rows use real time and close the file, but deliberately do not call `fsync()`;
+interpret them as page-cache/file-system throughput rather than durable-storage
+latency.
+
 ## RMQ Benchmark Tables
 
 The primary RMQ benchmark binary is usually:

include/pixie/bits.h

@@ -306,11 +306,11 @@ static inline uint64_t rank_512(const uint64_t* x, uint64_t count) {
 
 #else
 
-  uint64_t last_uint = count < 512 ? count >> 6 : 8;
+  size_t last_uint = count < 512 ? count >> 6 : 8;
 
   uint64_t pop_val = 0;
 
-  for (int i = 0; i < last_uint; i++) {
+  for (size_t i = 0; i < last_uint; i++) {
     pop_val += std::popcount(x[i]);
   }
 
@@ -412,7 +412,7 @@ template <bool Invert>
 static inline uint64_t select_512_scalar_impl(const uint64_t* x,
                                               uint64_t rank) {
   size_t word = 0;
-  int count;
+  uint64_t count;
   if constexpr (Invert) {
     count = std::popcount(~x[0]);
   } else {