Skip to content

design.md

Latest commit

 

History

History
108 lines (80 loc) · 7.88 KB

design.md

File metadata and controls

108 lines (80 loc) · 7.88 KB
Raw

libppswing: design

libppswing reads and writes PinPoint swing files (.ppsw, see specification/ppsw-format.md). It is C++17 with one dependency, vendored: zstd. It ships three things:

  • the library (ppswing::ppswing);
  • a CLI (ppsw);
  • a pure-Python reader (python/ppswing).

1. Why a separate library

PinPoint Studio, swinglab, and the Python analysis tools all read swing documents today, through QJsonDocument or json.load. A format with chunking, typed arrays and CRCs needs exactly one implementation per language, and those implementations must agree. A sister repository keeps the format's authority separate from the application, the same way libppcp does for the capture protocol. It is embedded the same way libwrist and libppcp are (FetchContent with a local-sibling override); that embed is step 5 of the roadmap and is not part of this repository.

2. API shape

namespace ppsw {
struct NdArray   { DType dtype; std::vector<uint64_t> shape; std::vector<uint8_t> data; // native LE, row-major, decoded
                   std::vector<uint8_t> nulls; bool narrowed; };
struct Table     { uint64_t rows; std::vector<std::string> keys; std::vector<Value> columns;
                   std::vector<std::optional<std::vector<uint8_t>>> present; std::vector<uint8_t> rownulls; };
struct Ref       { std::string name; };
class  Value     { /* null | bool | int64 | uint64 | double | string | Array | Object | NdArray | Table | Ref */ };

Value  parseJson(std::string_view text);                 // RFC 8259, int/double tokens preserved
std::string toJson(const Value &, JsonStyle = Compact);  // doubles in shortest round-trip form
Value  toPlain(const Value &);                           // NdArray/Table → arrays/objects (tree model)
bool   semanticEqual(const Value &, const Value &, std::string *firstDiffPath);

struct WriteOptions { size_t splitBytes = 64 * 1024; int zstdLevel = 9;
                      std::vector<std::pair<std::string, Precision>> hints; };
void   write(const std::string &path, const Value &plainTree, const WriteOptions & = {});

class Reader {
public:
    static Reader open(const std::string &path);          // header + TOC only
    const std::vector<TocEntry> &toc() const;
    Value root() const;                                    // root chunk, refs left as Ref
    Value chunk(const std::string &name) const;            // one chunk, refs left as Ref
    Value loadAll() const;                                 // refs resolved recursively
};
}

Errors are reported by exceptions (ppsw::Error, with a code and a message). The library is built with exceptions on; PinPoint Studio uses exceptions elsewhere.

The decoded representation keeps NdArray and Table. A future Qt adapter can therefore map an [N, 399] pose array straight into a typed buffer. toPlain() gives the JSON-model tree, for callers that want swing.json semantics.

3. Writer: encoding decisions

Encoding runs in one recursive pass over the plain tree. For each value:

  1. Array of numbers or bools, rectangular (spec §5.2): becomes an ndarray.
    • dtype:
      • all bool → bool;
      • all ints → the narrowest integer dtype holding the values;
      • otherwise float32 if every leaf round-trips exactly, else float64.
    • flags:
      • SHUFFLE always, when the width is > 1;
      • DELTA for integer arrays when the stored differences (from origin = the first non-null value) fit a strictly narrower signed dtype. A 4 s t_us column (values up to ~4e6, steps ~6600) goes from int32 to int16. Absolute-µs data (device.history, ~6e11) goes from int64 to int16 or int32;
      • COLMAJOR for rank ≥ 2.
    • Nulls go in the nulls bitmap (not for bool).
  2. Array of objects/null, with ≥ 2 object rows sharing at least one key, and no row larger than splitBytes: becomes a table. The row size is estimated cheaply (spec R1 #8). The /streams rows (~224 KB each, IMU samples) therefore stay a CBOR array, each stream its own chunk, so a reader wanting one alias does not decode all of them. Each column is the array of that key's values, encoded recursively by the same rules. So kp[399] over N frames becomes a [N, 399] float64 ndarray, t_us becomes a delta-coded int16 vector, and found becomes a bool vector.
  3. Anything else: plain CBOR.

Chunk splitting is bottom-up. After a member or element of an object or array is encoded, if its encoding exceeds splitBytes it is emitted as a chunk named by its JSON Pointer, and a ref goes in its place. Table columns are never split: a table is split whole (spec §5.4). The root is always a chunk.

Compression:

  • A chunk under 256 bytes is stored uncompressed.
  • Otherwise it gets zstd at zstdLevel, with the content size recorded.
  • Level 9 was measured within 2% of level 19 at about a tenth of the CPU time.

Atomic write: write <path>.tmp-<pid>, fsync, rename. On Windows, use MoveFileExW(MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH).

4. Precision hints: decision record

  • Default: lossless. Every corpus swing must round-trip exactly (spec §1.1).
  • display-f32 policy: narrows only the display-only pose tiers (synth, smoothed). They are about 60% of a swing's bytes and no reuse path reads them (spec §7 table).
  • Decision deferred to step 5. Whether the application writes with display-f32 is decided there, with the corpus sizes of both policies in hand (PinPointStudio/docs/implementation/swing_storage/ppsw_corpus.csv).

5. JSON import/export

The in-house RFC 8259 parser:

  • keeps integer tokens (no ./e) as int64 or uint64, and anything else as double via strtod under the "C" locale;
  • turns an integer token outside int64/uint64 into a double and counts it;
  • rejects duplicate keys, invalid UTF-8 and control characters;
  • decodes \u surrogate pairs.

Export writes doubles in the shortest form that round-trips: %.15g, then %.16g, then %.17g, checked by strtod_l. Integral doubles are written with a .0 so that the int/double type survives a JSON round trip. The .0 is appended only when the text has none of ., e, n, so 1e+20 stays valid.

⚠ Locale: Qt calls setlocale(LC_ALL, ""), so plain strtod/printf would read and write 0,5 under a German locale. Parsing uses strtod_l/_strtod_l with a "C" locale object. Printing fixes up the one character the locale can change, the decimal point.

6. zstd vendoring

zstd 1.5.7 is vendored as the official single-file amalgamation (third_party/zstd/zstd.c, generated by build/single_file_libs/combine.sh), BSD-3 licensed. It is compiled as a private static object with ZSTD_MULTITHREAD undefined and legacy formats off. Why vendored:

  • none of PinPoint Studio's platforms guarantees a system libzstd;
  • Qt's zstd feature is disabled in the build;
  • the sibling libraries' rule is "link nothing".

tests/purity.cmake asserts that the library links nothing else.

7. Python reader

python/ppswing is pure Python with zero required dependencies:

  • Decompression: the stdlib compression.zstd (Python 3.14), falling back to the optional zstandard package.
  • numpy is optional. With it, ndarrays decode to numpy.ndarray; without it, to nested lists via array/slicing.
  • load_swing(path) accepts a swing.json, a swing.ppsw, or a swing directory. It returns a tree semantically equal (spec §1.1) to what json.load returns. The one visible difference: Qt writes integral doubles as 37, so a real-valued column that happened to hold a 37 held a Python int there; from a .ppsw every element of such a column is a float. This is the migration seam for the ~26 scripts under PinPointStudio/tools/.
  • Reader exposes toc, root(), chunk(name) and load(plain=True).

The Python reader is written independently from the spec, not ported from the C++. The corpus check compares both decoders against json.load, so a bug shared by writer and reader in one language cannot pass unnoticed.