TUR-302: Cypher: order points and take their min, max and average

CLAUDE.md

@@ -80,10 +80,11 @@ These are structural properties of the graph model, not validation rules layered
 
 - **Every node has at least one label.** An empty label set is not representable; there is no "unlabeled node". An import file with a node whose label list is empty/null is malformed input, not a node to be created label-less.
 - **Every edge has exactly one edge type.** A null/empty edge type is not representable; an edge cannot exist without a type. A null `__type` in an import file is malformed input.
-- **A property value is a scalar, a list or a map.** The scalar types are `ValueType`'s Int64, UInt64, Double, Bool, String, Embedding, DateTime and Duration; `ValueType::List` is the list one, held by `TypedPropertyContainer<types::List>` over a `ListContainer` that owns its elements (see `docs/lists.md`). A stored list may be heterogeneous and may hold nested lists, so a LIST-typed column in an import file is a property to ingest, not unsupported input. `ValueType::Map` is the map one, held by `TypedPropertyContainer<types::Map>` over a `MapContainer` (see `docs/maps.md`): string keys, stored sorted, over values that may be scalars, lists or nested maps. A stored list may hold maps too, owned by its `ListContainer`'s own `MapContainer`.
-- **`ValueType` and `CellTag` are append-only.** Both are written to disk as their raw ordinals — the property tag in `PropertyTypeMapDumper`, and the cell tag of an element inside an `EncodedList` and of a value inside an `EncodedMap` — with no format version beside them. A new member goes immediately before the sentinel; inserting one anywhere else silently reinterprets every datapart already written. The same holds for `net::proto::ColumnInternalKind` on the wire.
+- **A property value is a scalar, a list or a map.** The scalar types are `ValueType`'s Int64, UInt64, Double, Bool, String, Embedding, DateTime, Duration and Point; `ValueType::List` is the list one, held by `TypedPropertyContainer<types::List>` over a `ListContainer` that owns its elements (see `docs/lists.md`). A stored list may be heterogeneous and may hold nested lists, so a LIST-typed column in an import file is a property to ingest, not unsupported input. `ValueType::Map` is the map one, held by `TypedPropertyContainer<types::Map>` over a `MapContainer` (see `docs/maps.md`): string keys, stored sorted, over values that may be scalars, lists or nested maps. A stored list may hold maps too, owned by its `ListContainer`'s own `MapContainer`.
+- **`ValueType` and `CellTag` are append-only.** Both are written to disk as their raw ordinals — the property tag in `PropertyTypeMapDumper`, and the cell tag of an element inside an `EncodedList` and of a value inside an `EncodedMap` — with no format version beside them. A new member goes immediately before the sentinel; inserting one anywhere else silently reinterprets every datapart already written. The same holds for `net::proto::ColumnInternalKind` on the wire, and for `PointCRS`, whose ordinal every stored point carries.
 - **`DateTime` is microseconds since the Unix epoch, UTC** (`storage/metadata/DateTime.h`). `datetime("...")` parses ISO-8601 and spends any offset on the instant, so `...T16:05:00+02:00` and `...T14:05:00Z` are one value and both render as `Z`. One int64 carries no zone; a zoned datetime would be a second type, not a widening of this one.
 - **`Duration` is a signed count of microseconds** (`storage/metadata/Duration.h`). It has no month component, because a month has no fixed length; a calendar duration would be a second type. `duration(<integer>)` reads the count as microseconds, and a duration renders as ISO-8601 hours, minutes and seconds (`PT25H1M1S`).
+- **A `Point` is x, y and z in one of four CRSs** (`storage/metadata/Point.h`): cartesian, cartesian-3d, wgs-84 and wgs-84-3d, Neo4j's. It is 32 bytes, so two equal points are equal in every byte: a 2D point holds a z of 0, no coordinate is -0.0, and the padding is zero. No coordinate is non-finite, a geographic latitude lies in [-90, 90] and its longitude in [-180, 180]. A point renders as the text `point({x: 1.0, y: 2.0, crs: 'cartesian'})`.
 
 ### Server (`server/`, `net/`)
 - Single TCP listener speaking TuringDB's binary protocol framed as HTTP/1.1 chunked transfer. The in-tree `net/http_parser/HTTPParser` reads the request (endpoints routed in `server/DBURIParser.h` / `server/Endpoints.h`), `server/TuringProtoServerProcessor` handles dispatch, and `net/turing_proto_server/TuringProtoWriter` emits one proto packet per HTTP chunk. There is no JSON protocol and no third-party web framework. Client side: `net/turing_proto_client/TuringClient` (C++), `python/turingdb/binary_client.py` (`BinaryClient`), `wasm/npm` (JS).

build_npm.sh

@@ -67,6 +67,7 @@ node decoder_smoke.mjs
 
 cd "$PACKAGE_DIR"
 node client_smoke.mjs
+node point_column_values_smoke.mjs
 node parameter_errors_smoke.mjs
 node split_error_smoke.mjs
 node payload_limit_smoke.mjs

common/StringUtils.cpp

@@ -1,5 +1,9 @@
 #include "StringUtils.h"
 
+#include <ctype.h>
+
+#include <algorithm>
+
 void StringUtils::splitString(std::string_view str,
                               char sep,
                               std::vector<std::string_view>& res) {
@@ -17,3 +21,11 @@ void StringUtils::splitString(std::string_view str,
         }
     }
 }
+
+bool StringUtils::equalsIgnoringCase(std::string_view lhs, std::string_view rhs) {
+    const auto sameLetter = [](char lhsCharacter, char rhsCharacter) {
+        return tolower(static_cast<unsigned char>(lhsCharacter)) == tolower(static_cast<unsigned char>(rhsCharacter));
+    };
+
+    return std::ranges::equal(lhs, rhs, sameLetter);
+}

common/StringUtils.h

@@ -10,4 +10,6 @@ class StringUtils {
     static void splitString(std::string_view str,
                             char sep,
                             std::vector<std::string_view>& res);
+
+    static bool equalsIgnoringCase(std::string_view lhs, std::string_view rhs);
 };

docs/maps.md

@@ -128,11 +128,11 @@ as `[u64 length][EncodedMap bytes]`.
 # Map projection
 
 `v {.name, .*, rank: v.age + 1, x}` builds a map out of `v`, a node, a relationship, a map, a
-datetime or a duration. `.key` holds `v.key`, null where `v` has no such property, and `.*` every
-property of `v`. On a datetime or a duration `.key` holds the component `v.key` reads:
-`WITH duration({hours: 1}) AS d RETURN d {.hours, x: 1}` is `{hours: 1, x: 1}`. A temporal has
-no properties, so `.*` and a name that is no component are rejected by the analyzer where `v` is
-typed a datetime or a duration, and raise on the row where a tagged cell holds one.
+datetime, a duration or a point. `.key` holds `v.key`, null where `v` has no such property, and
+`.*` every property of `v`. On a datetime, a duration or a point `.key` holds the component
+`v.key` reads: `WITH duration({hours: 1}) AS d RETURN d {.hours, x: 1}` is `{hours: 1, x: 1}`.
+These values have no properties, so `.*` and a name that is no component are rejected by the
+analyzer where `v` is typed one of them, and raise on the row where a tagged cell holds one.
 `key: expr` holds the expression, and `x` the variable `x` under the key `x`. A later entry
 shadows an earlier one under the same key, as in a map literal: `v {x: 1 / 0, x: 2}` is
 `{x: 2}`, and `v {.name, name: 1 / 0}` raises. An entry wins over the property `.*` reads under

docs/points.md

@@ -0,0 +1,124 @@
+We support the Cypher point type as a stored property value, following Neo4j's spatial values:
+https://neo4j.com/docs/cypher-manual/current/values-and-types/spatial/.
+
+# The value
+
+`ValueType::Point` is a scalar property type, held by `TypedPropertyContainer<types::Point>`.
+`db::Point` (`storage/metadata/Point.h`) is 32 trivially copyable bytes: `x`, `y` and `z` as
+doubles, then a one-byte `PointCRS` and 7 bytes of zero padding. The CRS is one of Neo4j's four:
+
+```
+PointCRS       name          SRID   dimension
+Cartesian      cartesian     7203   2
+Cartesian3D    cartesian-3d  9157   3
+WGS84          wgs-84        4326   2
+WGS84_3D       wgs-84-3d     4979   3
+```
+
+`PointCRS` is written to disk as its ordinal, inside every stored point, so it is append-only.
+
+The constructor makes two equal points equal in every byte, so a dump, a list encoding and a
+DISTINCT key can all compare raw bytes: a 2D point holds a `z` of 0, `-0.0` becomes `0.0`, and
+the padding is zero. It also rejects what no point holds: a non-finite coordinate, and a
+geographic latitude outside [-90, 90]. A geographic longitude outside [-180, 180] is wrapped into
+it, so `point({longitude: 190, latitude: 0})` has an x of -170.
+
+A point is a `CellTag::Point` inside a list or a map, written as its 32 bytes.
+
+# Building one
+
+`point(map)` reads the map's keys, ignoring case and ignoring any key it does not know:
+
+```
+point({x, y})                          cartesian
+point({x, y, z})                       cartesian-3d
+point({longitude, latitude})           wgs-84
+point({longitude, latitude, height})   wgs-84-3d; z may stand for height
+point({..., crs: 'wgs-84'})            the CRS by name, ignoring case
+point({..., srid: 4326})               the CRS by SRID
+```
+
+`x` and `y` win over `longitude` and `latitude` where a map holds both, and are read as longitude
+and latitude in a geographic CRS. A null map, or a null value under any key, gives null.
+
+It raises on a map naming both a `crs` and an `srid`, on an unknown CRS, on longitude and latitude
+in a cartesian CRS, on a number of coordinates the CRS does not have, and on a coordinate that is
+no number. Neo4j parses a numeric string coordinate; TuringDB does not.
+
+# Reading one
+
+`p.x`, `p.y` and `p.z` are its coordinates, and `p.longitude`, `p.latitude` and `p.height` the
+same ones in a geographic CRS; all six are doubles. `p.crs` is the CRS name and `p.srid` its SRID.
+`p.z` and `p.height` raise on a 2D point, and the three geographic names raise on a cartesian
+point. The CRS is a property of the value, not of the column, so a column of points mixing CRSs
+raises on the row that holds the wrong one.
+
+A point renders as the text Neo4j's `toString` writes, in `toString(p)` and in the shell:
+`point({x: 2.3522, y: 48.8566, crs: 'wgs-84'})`. The binary protocol carries it as its 32 bytes,
+under the `POINT` column kind.
+
+# Comparing
+
+Two points are equal when they share a CRS and every coordinate. Points have one total order, by
+SRID, then by x, y and z: ORDER BY, DISTINCT, grouping, `min()`, `max()` and `<`, `<=`, `>` and
+`>=` all follow it, so every wgs-84 point (4326) is less than every cartesian one (7203). Neo4j
+answers null for `<` between two different points. A point against a value of another type
+compares to null. Between types, ORDER BY sorts points after paths and before datetimes, as in
+Neo4j.
+
+# Aggregating
+
+`count`, `collect`, `min` and `max` take points, over a property, over the points a list holds,
+per group and with DISTINCT. `avg` of points is their centroid, coordinate by coordinate: a point
+of the CRS they share. Points of two CRSs raise, as points mixed with numbers do, and no point
+averages to null. A geographic centroid is the mean of the longitudes and of the latitudes, not a
+spherical mean, so points on both sides of the 180th meridian average to a longitude near 0.
+`sum` takes no point.
+
+# Distance and bounding boxes
+
+`point.distance(a, b)` is the Euclidean distance between two cartesian points. Between two
+wgs-84 points it is the great-circle distance in metres, by the haversine formula on a sphere of
+the IUGG mean Earth radius, 6,371,008.8 m. Neo4j uses 6,378,140 m, so its distances are 0.11%
+longer. Between two wgs-84-3d points the great circle is taken at the mean of the two heights and
+combined with the height difference, as Neo4j does. It is null for points in different CRSs.
+
+`point.withinBBox(p, lowerLeft, upperRight)` is whether `p` lies in the box, bounds included. In a
+geographic CRS a `lowerLeft` east of `upperRight` makes a box that crosses the 180th meridian, and
+a `lowerLeft` north of `upperRight` an empty one. It is null unless the three points share a CRS.
+
+# Importing
+
+`LOAD JSONL` reads a point out of a JSON object under a name a `WITH POINTS [...]` clause lists,
+by the rules `point()` reads a map with:
+
+```
+LOAD JSONL '/data/cities.jsonl' AS cities WITH POINTS [at]
+
+{"type": "node", "id": "0", "labels": ["City"],
+ "properties": {"name": "Paris", "at": {"longitude": 2.3522, "latitude": 48.8566}}}
+```
+
+A JSON null leaves the property unset. An object naming no point, or holding a null, raises.
+`WITH POINTS`, `WITH DATETIMES` and `WITH DURATIONS` fill one `TypedPropertySpec`, which maps
+each name to the type it is read as, and a name given two types is rejected.
+
+`LOAD PARQUET` reads points out of a geometry column with no clause of its own: a column
+Parquet's GEOMETRY or GEOGRAPHY logical type marks, or one GeoParquet's `geo` file metadata names
+with the WKB encoding (`io/parquet/ParquetGeometry.h`). Each value is a WKB point, little- or
+big-endian, ISO or EWKB: a 2D point or a 3D one, and an empty point reads as null. A list of
+geometries reads as a list of points.
+
+```
+CRS of the column                                 points
+OGC:CRS84, EPSG:4326, EPSG:4979, the default      wgs-84, or wgs-84-3d for a 3D point
+GeoParquet "crs": null                            cartesian, or cartesian-3d
+any other                                         rejected
+```
+
+A CRS given as PROJJSON is read by its `id`: GeoParquet's `crs` object, a `projjson:<key>` naming
+a key of the file metadata, or the PROJJSON text Arrow writes in the logical type itself.
+
+A geometry that is no point, a point with a measure, and bytes that are no WKB are rejected, a
+WKB type outside ISO's dimension codes 0 to 3 included. An EWKB SRID other than 0 has to name the
+column's CRS: 4326 or 4979 in a wgs-84 column, none in a cartesian one.

fuzz/cypher.dict

@@ -70,6 +70,8 @@
 "EMBEDDINGS"
 "AVAILABLE"
 "DATETIMES"
+"DURATIONS"
+"POINTS"
 
 # --- Boolean and null ---
 "TRUE"

import/common/CMakeLists.txt

@@ -1,8 +1,8 @@
 add_library(turing_db_import_common_s
   STATIC
-  DateTimeSpec.cpp
-  DurationSpec.cpp
   EmbeddingsSpec.cpp
+  TypedPropertySpec.cpp
 )
 
 target_include_directories(turing_db_import_common_s PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
+target_link_libraries(turing_db_import_common_s PUBLIC turing_db_storage_s)

import/common/JsonlImportSpecs.h

@@ -2,9 +2,8 @@
 
 #include <utility>
 
-#include "DateTimeSpec.h"
-#include "DurationSpec.h"
 #include "EmbeddingsSpec.h"
+#include "TypedPropertySpec.h"
 
 namespace db {
 
@@ -13,21 +12,16 @@ namespace db {
 // text or an instant - so the query names the properties that are neither by default.
 struct JsonlImportSpecs {
     EmbeddingsSpec _embeddings;
-    DateTimeSpec _dateTimes;
-    DurationSpec _durations;
+    TypedPropertySpec _types;
 
     // Takes the hints another clause of the same statement gave
     void absorb(JsonlImportSpecs&& other) {
         for (const auto& [name, dimension] : other._embeddings) {
             _embeddings.emplace(name, dimension);
         }
 
-        for (const std::string_view name : other._dateTimes) {
-            _dateTimes.emplace(name);
-        }
-
-        for (const std::string_view name : other._durations) {
-            _durations.emplace(name);
+        for (const auto& [name, type] : other._types) {
+            _types.emplace(name, type);
         }
     }
 };

import/common/TypedPropertySpec.cpp

@@ -0,0 +1,20 @@
+#include "TypedPropertySpec.h"
+
+using namespace db;
+
+ValueType TypedPropertySpec::getType(std::string_view propertyName) const {
+    const auto it = _types.find(propertyName);
+    if (it == _types.end()) {
+        return ValueType::Invalid;
+    }
+
+    return it->second;
+}
+
+void TypedPropertySpec::emplace(std::string_view propertyName, ValueType type) {
+    const auto [it, inserted] = _types.emplace(propertyName, type);
+
+    if (!inserted && it->second != type) {
+        _clashes.push_back(Clash {._name = propertyName, ._first = it->second, ._second = type});
+    }
+}

import/common/TypedPropertySpec.h

@@ -0,0 +1,43 @@
+#pragma once
+
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "metadata/PropertyType.h"
+
+namespace db {
+
+// The properties an import reads as a type its file format does not spell: a datetime is a
+// JSON string, a duration an integer count of microseconds, a point a JSON object. The query
+// names them, so the import does not guess from what a value looks like.
+class TypedPropertySpec {
+public:
+    // A name the query gave a second type. The first one stands.
+    struct Clash {
+        std::string_view _name;
+        ValueType _first {ValueType::Invalid};
+        ValueType _second {ValueType::Invalid};
+    };
+
+    using MapType = std::unordered_map<std::string_view, ValueType>;
+    using ConstIterator = MapType::const_iterator;
+
+    // ValueType::Invalid where the query names no type for @param propertyName
+    ValueType getType(std::string_view propertyName) const;
+
+    void emplace(std::string_view propertyName, ValueType type);
+
+    const std::vector<Clash>& getClashes() const { return _clashes; }
+
+    ConstIterator begin() const { return _types.begin(); }
+    ConstIterator end() const { return _types.end(); }
+
+    bool empty() const { return _types.empty(); }
+
+private:
+    MapType _types;
+    std::vector<Clash> _clashes;
+};
+
+}