diff --git a/claude-skill.mdx b/claude-skill.mdx index a41c7fe..1d46609 100644 --- a/claude-skill.mdx +++ b/claude-skill.mdx @@ -42,8 +42,8 @@ The skill is structured so Claude only loads the reference it needs: | Module | Covers | | --- | --- | | `startup` | Installation, starting the server, connecting (HTTP, authentication, embedded), loading a graph | -| `querying` | `MATCH`, `OPTIONAL MATCH`, paths, `WITH`, aggregation, `UNWIND`, subqueries, built-in functions, `LOAD CSV` | -| `writing` | `CREATE`, `MERGE`, `SET`, `DELETE`, the change and commit workflow | +| `querying` | `MATCH`, `OPTIONAL MATCH`, label expressions, paths, `WITH`, `NEXT`, `WHEN`, aggregation, `UNWIND`, maps, subqueries, built-in functions, parameters, `LOAD CSV` | +| `writing` | `CREATE`, `MERGE`, `SET`, `DELETE`, `FOREACH`, bulk edges by node ID, the change and commit workflow | | `importing` | CSV, Parquet, JSONL, GML and Neo4j imports | | `algorithms` | Shortest path, vector search, embeddings, GNN sampling | | `introspection` | Schema exploration, versioning and time travel, Python SDK reference | diff --git a/commands.mdx b/commands.mdx index 1a0cec1..c0551a5 100644 --- a/commands.mdx +++ b/commands.mdx @@ -1,7 +1,7 @@ --- title: "CLI Commands" sidebarTitle: "Commands" -description: "Reference for TuringDB CLI commands: start, UI, and stop." +description: "Reference for TuringDB CLI commands (start, UI and stop), the interactive shell, environment variables and the HTTP API." --- ## Start @@ -26,7 +26,9 @@ turingdb start [options] | `-ui` | Launch the built-in visualizer proxy | off | | `-ui-port ` | Visualizer proxy port | `8080` | | `-auth-on` | Require a bearer token on every request. The token is read from the `TURINGDB_AUTH_TOKEN` environment variable, and must be set when you use this flag | off | -| `-start-timeout ` | Time to wait for daemon readiness | `500` | +| `-no-vector-mmap` | Read HNSW [vector indexes](/vector-search) into memory at startup instead of memory-mapping their files | off | + +With `-demon`, `turingdb start` returns once the server is ready. If the server cannot start, it exits with an error that says why, for example `Could not start the server on 127.0.0.1:6666: bind: Address already in use`, or `TuringDB is already running` when another instance uses the same `-turing-dir`. ### Examples @@ -79,6 +81,8 @@ Without `-demon`, `turingdb start` serves HTTP **and** opens an interactive Cyph | `cd ` | Switch the active graph | | `checkout change-` | Work on a change. `checkout ` opens a past commit, and `checkout` returns to the latest commit of main | | `read ` | Run the queries in a file | +| `:param => ` | Set a [query parameter](/query/cypher_subset#query-parameters), with the value written as JSON: `:param label => "Person"`. `:param clear` removes them all | +| `SHOW PARAMETERS` | List the parameters set in the shell | | `help` | List the available commands | | `quit` / `exit` | Leave the shell | @@ -160,14 +164,17 @@ turingdb stop -turing-dir /data/myturing ## HTTP API -The Python SDK is a thin client over a single endpoint, so you can query a server from any language. With the default (JSON) server: +The Python SDK is a thin client over a single endpoint, so you can query a server from any language. With the default (JSON) server, send the query as a JSON object: ```bash -curl -s -X POST 'http://localhost:6666/query?graph=mygraph' -d "MATCH (n) RETURN count(n)" +curl -s -X POST 'http://localhost:6666/query?graph=mygraph' \ + -d '{"query": "MATCH (n:$label) RETURN count(n)", "parameters": {"label": "Person"}}' ``` -- The request body is the query text. The optional query-string parameters `change=` and `commit=` target a change or a past commit. +- The request body is a JSON object with a `query` string and an optional `parameters` object (see [query parameters](/query/cypher_subset#query-parameters)). +- The optional query-string parameters `change=` and `commit=` target a change or a past commit. The change ID is the decimal number that `CHANGE NEW` returns. - With `-auth-on`, send `Authorization: Bearer `. A missing or incorrect token returns HTTP `401` with an empty body. -- A successful response is `{"header": {"column_names": [...], "column_types": [...]}, "data": [chunk, ...], "time": }`, where each chunk is a list of columns. -- An error response is `{"error": "PARSE_ERROR", "error_details": "...", "time": }`. The code is one of `PARSE_ERROR`, `ANALYZE_ERROR`, `PLAN_ERROR`, `EXEC_ERROR`, `CHANGE_NOT_FOUND` or `COMMIT_NOT_FOUND`. +- A request, headers included, can be up to 1 MiB. A larger one returns HTTP `413` with the code `REQUEST_TOO_BIG`. +- A successful response is `{"header": {"column_names": [...], "column_types": [...]}, "data": [chunk, ...], "time": }`, where each chunk is a list of columns. Doubles are written with full precision. +- An error response is `{"error": "PARSE_ERROR", "error_details": "...", "time": }`. The code is one of `PARSE_ERROR`, `ANALYZE_ERROR`, `PLAN_ERROR`, `EXEC_ERROR`, `GRAPH_NOT_FOUND`, `CHANGE_NOT_FOUND`, `COMMIT_NOT_FOUND` or `REQUEST_TOO_BIG`, and the details name the graph, change or commit that was not found. - Queries on a non-HEAD commit need `LOAD COMMIT ''` first when you talk to the API directly. The CLI and the SDK do this for you. diff --git a/graph_dev/create_nodes_edges.mdx b/graph_dev/create_nodes_edges.mdx index 8ffc3a9..9cff65a 100644 --- a/graph_dev/create_nodes_edges.mdx +++ b/graph_dev/create_nodes_edges.mdx @@ -73,9 +73,9 @@ description: "Once you have a valid change, you can start adding new nodes and e ## Merge, delete and bulk loading -- Use `MERGE` instead of `CREATE` to avoid duplicates: `MERGE (n:Person {name: 'Jane'})`. Run `COMMIT` first when `MERGE` should find nodes created earlier in the same change. +- Use `MERGE` instead of `CREATE` to avoid duplicates: `MERGE (n:Person {name: 'Jane'})`. `MERGE` also finds nodes created earlier in the same change. - Remove data with `DELETE` (edges, or nodes without edges) and `DETACH DELETE` (a node and all its edges). - Every node needs at least one label, and every edge exactly one edge type. -- For more than a few thousand nodes or edges, build the graph with [`LOAD PARQUET`](/import_data/parquet) instead of Cypher writes. +- For more than a few thousand nodes or edges, build the graph with [`LOAD PARQUET`](/import_data/parquet) instead of Cypher writes. To add many edges between existing nodes, [`UNWIND` pairs of node IDs](/query/writing#edges-between-existing-nodes). See [Writing data](/query/writing) for the full reference. diff --git a/import_data/csv.mdx b/import_data/csv.mdx index 61df179..0e263fc 100644 --- a/import_data/csv.mdx +++ b/import_data/csv.mdx @@ -42,17 +42,17 @@ Read a row's fields as `row.col` or `row[]`. Without `WITH HEADER ## Combining with the rest of a query -`LOAD CSV` composes with `MATCH`, `WITH` / `WHERE`, aggregates, `ORDER BY` and `UNWIND`. For example, join the rows with nodes that are already in the graph: +`LOAD CSV` composes with `MATCH`, `OPTIONAL MATCH`, `WITH` / `WHERE`, aggregates, `ORDER BY`, `UNWIND` and subqueries, which can all read `row.x`. For example, join the rows with nodes that are already in the graph: ```jsx LOAD CSV "people.csv" WITH HEADERS AS row MATCH (p:Person {name: row.name}) RETURN p.name, row.city -``` - - To use `row.x` inside an `OPTIONAL MATCH` pattern that directly follows `LOAD CSV`, project the field first: `... AS row WITH row.name AS nm OPTIONAL MATCH (p:Person {name: nm}) ...`. - +LOAD CSV "people.csv" WITH HEADERS AS row +OPTIONAL MATCH (p:Person {name: row.name}) +RETURN row.name, p.age +``` ## Building a graph from CSV diff --git a/import_data/jsonl.mdx b/import_data/jsonl.mdx index 5a7c721..258d072 100644 --- a/import_data/jsonl.mdx +++ b/import_data/jsonl.mdx @@ -76,7 +76,7 @@ LOAD JSONL 'mydata.jsonl' AS mygraph WITH EMBEDDINGS [{"embedding", 384}, {"summ Each spec is `{"", }`. The dimension must be **greater than 1**, and every array under that property must have exactly that many numeric elements. This works for both node and relationship properties. Array properties not listed in `WITH EMBEDDINGS` are stored as Lists. - When an import fails, additional details are written to `/logs/turingdb.log`. + When an import fails, the error names the line of the file that could not be read, for example `JSONL error 'JSONL parse' at line 2: ...`. More details are written to `/logs/turingdb.log`. diff --git a/pythonsdk/reference.mdx b/pythonsdk/reference.mdx index 32ba583..c56c13d 100644 --- a/pythonsdk/reference.mdx +++ b/pythonsdk/reference.mdx @@ -22,7 +22,7 @@ or using `pip`: pip install turingdb ``` -The SDK targets TuringDB v3: pin `turingdb>=3.0` in your project (older projects may pin `1.x`, which uses the previous engine and a much smaller Cypher subset). The wheel supports Python 3.10 to 3.14 on Linux (x86_64, aarch64) and macOS 15 or later (arm64). It also contains the `turingdb` command-line tool, the `turing-parquet` import tool, and the in-process engine. +The SDK targets TuringDB v3: pin `turingdb>=3.0.2` in your project (older projects may pin `1.x`, which uses the previous engine and a much smaller Cypher subset). Upgrade the SDK and the server together: from 3.0.2 the JSON client sends each query as a JSON body, which is what a 3.0.2 server expects. The wheel supports Python 3.10 to 3.14 on Linux (x86_64, aarch64) and macOS 15 or later (arm64). It also contains the `turingdb` command-line tool, the `turing-parquet` import tool, and the in-process engine. ## Getting Started @@ -87,6 +87,7 @@ A server in binary mode serves the binary protocol only, so `curl`, the JSON cli - `host` accepts `"localhost"` together with `port=`, or a full URL such as `"http://localhost:6666"`. Use one of these two forms. - The query API is identical, and results come back as the same DataFrames. Embeddings are returned as numpy `float32` arrays instead of Python lists. +- A [`shortestPath`](/shortest-path) `path` cannot be read over the native backend yet. Run those queries through the JSON backend. - To list the graphs on disk, use the JSON client. On the native backend, `list_loaded_graphs()` lists the loaded graphs. - `token`, `reconnect()` and the S3 helpers work as on the JSON backend. @@ -159,17 +160,18 @@ client.load_graph("mygraph") If `raise_if_loaded=False`, it will silently continue if the graph is already loaded. -### `query(query: str) → pandas.DataFrame` +### `query(query: str, query_parameters: dict | None = None) → pandas.DataFrame` Runs a Cypher query on the current graph. ```python df = client.query("MATCH (n:Person) RETURN n.name, n.age") +df = client.query("MATCH (n:$label) RETURN n.name", {"label": "Person"}) ``` -Returns results as a `pandas.DataFrame`, with automatic typing and parsing. +Returns results as a `pandas.DataFrame`, with automatic typing and parsing. `query_parameters` fills `$name` [parameters](/query/cypher_subset#query-parameters), which stand for labels and edge types. -### `query_raw(query: str) → dict` +### `query_raw(query: str, query_parameters: dict | None = None) → dict` Like `query`, but returns the raw server response as a `dict` instead of a DataFrame. Useful for inspecting result metadata or column dtypes directly. @@ -223,7 +225,7 @@ client.set_commit("haax42") ### `set_change(change: int | str)` -Manually set the change ID to use (accepts int or hex string). +Manually set the change ID to use (accepts the int that `new_change()` returns, or the same number as a string). ```python client.set_change(3) @@ -292,15 +294,15 @@ All queries return a `pandas.DataFrame`, typed according to the schema: | `List`, `Map` | object (Python list, dict) | | `Embedding` | object (list over HTTP, numpy `float32` array on the native and embedded backends) | -Nodes and edges (`RETURN n`) come back as their internal integer IDs (`UInt64`), so project the properties you need. A named path comes back as a list of `{'type': 'node' | 'edge', 'id': ...}` dictionaries. See [Functions and data types](/query/functions#how-results-come-back) for the full table. +Nodes and edges (`RETURN n`) come back as their internal integer IDs (`UInt64`), so project the properties you need, or return `properties(n)` as a dictionary. A named path comes back as a list of `{'type': 'node' | 'edge', 'id': ...}` dictionaries. Doubles keep their full precision on every backend. See [Functions and data types](/query/functions#how-results-come-back) for the full table. - - Over the JSON backend, Doubles are serialised with 6 decimal places. Use the native or embedded backend if you need full precision. - + + Integer columns hold numpy integers, which an f-string prints as `np.int64(5)`. Convert IDs with `.tolist()` or `int()` before writing them into a query. + ## Error Handling -Query errors raise `TuringDBException`. The message starts with the error code (`PARSE_ERROR`, `ANALYZE_ERROR`, `PLAN_ERROR`, `EXEC_ERROR`, `CHANGE_NOT_FOUND`, `COMMIT_NOT_FOUND`, ...) and points at the offending part of the query. +Query errors raise `TuringDBException`. The message starts with the error code (`PARSE_ERROR`, `ANALYZE_ERROR`, `PLAN_ERROR`, `EXEC_ERROR`, `GRAPH_NOT_FOUND`, `CHANGE_NOT_FOUND`, `COMMIT_NOT_FOUND`, `REQUEST_TOO_BIG`, ...) and points at the offending part of the query, or names the graph, change or commit that was not found. ```python from turingdb import TuringDB, TuringDBException @@ -311,7 +313,7 @@ except TuringDBException as e: print("Query failed:", e) ``` -On the JSON backend, transport failures (the server is down, or HTTP 401 for a bad or missing token) surface as `httpx` exceptions instead. +On the JSON backend, transport failures (the server is down, or HTTP 401 for a bad or missing token) surface as `httpx` exceptions instead. HTTP errors that the server explains, such as `REQUEST_TOO_BIG`, still raise `TuringDBException`. ## Example Workflow diff --git a/query/aggregation.mdx b/query/aggregation.mdx index 0eb4121..ac770c8 100644 --- a/query/aggregation.mdx +++ b/query/aggregation.mdx @@ -1,7 +1,7 @@ --- title: "Results, Aggregation and Lists" sidebarTitle: "Results and Aggregation" -description: "Shape results with WITH, DISTINCT, ORDER BY, SKIP, LIMIT and UNION, then aggregate, unwind and transform lists." +description: "Shape results with WITH, DISTINCT, ORDER BY, SKIP, LIMIT and UNION, chain queries with NEXT and WHEN, then aggregate, unwind and transform lists." --- These examples use the [sample graph](/query/matching) from the matching page. @@ -97,6 +97,49 @@ RETURN c.name AS name `UNION` removes duplicates and `UNION ALL` keeps them. All branches must return the same column names, and each column must keep one value type across the branches. +## NEXT + +`NEXT` chains queries in one request: the columns returned by the query before it become the incoming rows of the query after it. + +```cypher +MATCH (n:Person) +RETURN n.name AS nm, n.age AS age +NEXT +RETURN nm +ORDER BY nm LIMIT 2 + +MATCH (n:Person) +RETURN n.city AS city, count(*) AS people +NEXT +RETURN max(people) AS biggestCity +``` + +- Every expression in a `RETURN` followed by `NEXT` needs an alias (`AS`). +- A later query sees what an earlier one wrote: `CREATE (x:Tag {v: 1}) RETURN x NEXT MATCH (t:Tag) RETURN count(t)`. + +## Conditional queries with WHEN + +`WHEN THEN [WHEN ... THEN ...] [ELSE ]` runs the first branch whose condition holds. Branches return the same columns, and they can also write. + +```cypher +WHEN true THEN RETURN 1 AS x ELSE RETURN 2 AS x + +MATCH (n:Person) +RETURN n.name AS nm, n.age AS age +NEXT +WHEN age > 30 THEN RETURN nm, 'senior' AS band +ELSE RETURN nm, 'junior' AS band + +MATCH (n:Person) +CALL (n) { + WHEN n.age > 30 THEN RETURN 'senior' AS band + ELSE RETURN 'junior' AS band +} +RETURN n.name, band +``` + +A `WHEN` can start a query, follow `NEXT`, or form the body of a [`CALL` subquery](/query/subqueries#call-subqueries), where it runs once per row. Several `WHEN` queries combine with `UNION`: `{ WHEN c1 THEN RETURN 1 AS x } UNION { WHEN c2 THEN RETURN 2 AS x }`. + ## Aggregation Grouping is implicit, as in Neo4j: the non-aggregated items in the `RETURN` are the grouping keys. @@ -130,6 +173,9 @@ ORDER BY c DESC MATCH (n) RETURN labels(n) AS l, count(*) AS c ORDER BY c DESC + +MATCH (n:Person) +RETURN stDev(n.age) AS sd, percentileCont(n.age, 0.5) AS median, percentileDisc(n.age, 0.9) AS p90 ``` | Aggregate | Notes | @@ -138,9 +184,9 @@ ORDER BY c DESC | `collect(x)`, `collect(DISTINCT x)` | Nulls are dropped | | `sum(x)` | | | `avg(x)` | Always a Double | -| `min(x)`, `max(x)` | Numbers, strings, booleans, `DateTime` and `Duration` | - -Statistical aggregates such as `stdev` and `percentileCont` are not yet available. Compute them in your application, for example in pandas. +| `min(x)`, `max(x)` | Numbers, strings, booleans, `DateTime`, `Duration`, lists and values of mixed types | +| `stDev(x)`, `stDevP(x)` | Sample and population standard deviation | +| `percentileCont(x, p)`, `percentileDisc(x, p)` | Interpolated and nearest-value percentile; `p` is a fraction, such as `0.5` for the median | **Rules:** @@ -187,7 +233,7 @@ RETURN t, count(*) `range(start, end[, step])` includes both bounds. Unwound nodes stay nodes, so `p.name` works after `UNWIND ns AS p`. You can also `UNWIND` a list of maps and read their keys: `UNWIND [{a: 1}, {a: 2}] AS m RETURN m.a`. - Write the list into the query text, for example `f"UNWIND {ids} AS x ..."` in Python. Query parameters such as `UNWIND $list` are not yet supported. + Write the list into the query text, for example `f"UNWIND {ids} AS x ..."` in Python. Value parameters such as `UNWIND $list` are not yet supported. ## Lists @@ -200,7 +246,7 @@ RETURN size([1, 2]), head([1, 2]), last([1, 2]), tail([1, 2]), reverse([1, 2, 3] MATCH (n:Person {name: 'Alice'}) RETURN n.tags[0], size(n.tags) -RETURN [1, 2] + [3], 'a' || 'b' +RETURN [1, 2] + [3], [1, 2] + 3, 'a' || 'b' ``` - Indexing and slicing work on literals, list properties and function results (`labels(n)[0]`). Negative indexes count from the end, and an out-of-range index returns `null`. @@ -237,3 +283,5 @@ RETURN n.name, ``` The simple form (`CASE n.city WHEN 'London' THEN ...`) and the searched form (`CASE WHEN condition THEN ...`) both work. TuringDB also accepts comparison tests in the simple form, such as `CASE n.age WHEN > 30 THEN 'old' WHEN IS NULL THEN '?' END`. + +Branches can return lists, maps, `DateTime` and `Duration` values, and they may mix types: `CASE WHEN n.age > 30 THEN n.tags ELSE [] END` and `coalesce(n.nickname, n.age)` both work. diff --git a/query/cheatsheet.mdx b/query/cheatsheet.mdx index 134d321..b44b4c3 100644 --- a/query/cheatsheet.mdx +++ b/query/cheatsheet.mdx @@ -26,6 +26,12 @@ RETURN a.name, e.since, b.name MATCH (a)-[r:KNOWS|WORKS_AT]->(b) RETURN a.name, type(r), b.name +MATCH (n:Person|Company) +RETURN n.name + +MATCH (n:!Person) +RETURN n.name + MATCH (a:Person)-[e]-(b) RETURN a.name, b.name @@ -40,7 +46,9 @@ RETURN n.name - `()` is a node, `[]` is an edge, `-` ignores direction. - Node labels use `:Label` and can be stacked: `(n:Person:Employee)`. - Edge types use `:EDGE_TYPE`: `-[:KNOWS]->`. An edge has exactly one type, and `[:A|B]` matches either. -- `RETURN n` gives the internal node ID. Return properties explicitly: `RETURN n.name`. +- Label expressions: `(n:A|B)`, `(n:!A)`, `(n:A&B)`, also on edge types and in `WHERE`. +- Labels and edge types can be [parameters](/query/cypher_subset#query-parameters): `client.query("MATCH (n:$label) RETURN n.name", {"label": "Person"})`. +- `RETURN n` gives the internal node ID. Return properties explicitly (`RETURN n.name`), or all of them with `properties(n)`. - Unknown labels and properties are forgiving: they match nothing or read as `null`. ## OPTIONAL MATCH @@ -95,7 +103,7 @@ RETURN p.name - Comparison: `= <> != < <= > >=`, null tests `IS NULL` / `IS NOT NULL`, boolean `AND OR XOR NOT`. - Strings and lists: `IN`, `STARTS WITH`, `ENDS WITH`, `CONTAINS`, concatenation with `+` or `||`. - Use `IS NOT NULL` to test for a property (`=~` and `exists(n.prop)` are not yet supported). -- For Doubles, use ranges (`n.score > 1.99 AND n.score < 2.01`) or convert with `toInteger`. +- Integers and Doubles compare by value (`n.age = 30.0`). Compare two Doubles with a range (`n.score > 1.99 AND n.score < 2.01`). ## Lookup by native node ID @@ -113,15 +121,22 @@ RETURN m.name UNWIND [0, 1, 2] AS x MATCH (n) WHERE n = x -RETURN id(n), n.name +RETURN x, n.name + +UNWIND [[0, 1], [2, 3]] AS p +MATCH (a) +WHERE a = p[0] +MATCH (b) +WHERE b = p[1] +RETURN a.name, b.name EXPLAIN MATCH (n) WHERE n = 0 RETURN n.name ``` -- Compare the **variable** (`n = 0`), not `id(n) = 0`. The first is a direct seek and the second scans every node. -- For batches, `UNWIND` a literal list of distinct IDs and return `id(n)` rather than `x`. +- `n = 0` and `id(n) = 0` both look the node up directly. +- For batches, `UNWIND` a list of IDs, from the query text, a variable, `collect()` or `range()`. Prefer it to `WHERE n IN [...]`, which checks every node. ## Joins and Cartesian products @@ -163,14 +178,17 @@ RETURN DISTINCT b.name MATCH (a:Person {name: 'Alice'}) ((x)-[:KNOWS]->(y) WHERE y.age > x.age)+ (b) RETURN b.name +MATCH (a:Person {name: 'Alice'})-[e:KNOWS]->+(b) +RETURN b.name, [r IN e | r.since] + MATCH p = (a:Person {name: 'Alice'})-[:KNOWS]->+(b) -RETURN [n IN nodes(p) | n.name], length(p) +RETURN [n IN nodes(p) | n.name], length(p), reduce(t = 0, r IN relationships(p) | t + r.since) ``` - Postfix `->*` is **0** or more hops. Bracket `-[:KNOWS*]->` is **1** or more. - Each edge is used at most once per path. Bound the length on large graphs. -## WITH, ORDER BY, SKIP, LIMIT, UNION +## WITH, ORDER BY, SKIP, LIMIT, UNION, NEXT, WHEN [Results and aggregation](/query/aggregation) @@ -192,9 +210,20 @@ RETURN p.name AS name UNION MATCH (c:Company) RETURN c.name AS name + +MATCH (n:Person) +RETURN n.city AS city, count(*) AS c +NEXT +RETURN max(c) AS biggest + +MATCH (n:Person) +RETURN n.name AS nm, n.age AS age +NEXT +WHEN age > 30 THEN RETURN nm, 'senior' AS band +ELSE RETURN nm, 'junior' AS band ``` -- Expressions in `WITH` must be aliased. +- Expressions in `WITH`, and in a `RETURN` followed by `NEXT`, must be aliased. - Nulls sort last ascending and first descending. ## Aggregation @@ -212,10 +241,12 @@ RETURN count(DISTINCT n.city) MATCH ()-[r]->() RETURN type(r) AS t, count(*) AS c ORDER BY c DESC + +MATCH (n:Person) +RETURN stDev(n.age), percentileCont(n.age, 0.5) AS median ``` - Grouping is implicit. To filter on an aggregate, compute it in a `WITH` and filter afterwards. -- `stdev` and `percentile*` aggregates are not yet available. ## UNWIND, lists, CASE @@ -259,6 +290,9 @@ RETURN p.name MATCH (p:Person) RETURN p.name, COUNT { (p)-[:KNOWS]->() } AS outDegree +MATCH (p:Person) +RETURN p.name, COLLECT { MATCH (p)-[:KNOWS]->(f) RETURN f.name } AS friends + MATCH (p:Person) CALL (p) { MATCH (p)-[:KNOWS]->(f) @@ -280,14 +314,14 @@ RETURN p.name, co | Category | Functions | | -------- | --------- | -| Entity | `id`, `labels`, `type`, `startNode`, `endNode` | +| Entity | `id`, `labels`, `type`, `startNode`, `endNode`, `keys`, `properties` | | Path and list | `nodes`, `relationships`, `length`, `size`, `head`, `last`, `tail`, `reverse`, `range`, `reduce` | -| Conversion | `toInteger`, `toFloat`, `toString`, `toBoolean`, `coalesce` | -| String | `toUpper`, `toLower`, `trim`, `ltrim`, `rtrim`, `reverse` | -| Math | `abs`, `sign`, `ceil`, `floor`, `round`, `sqrt`, `exp`, `log`, `log10`, `pi`, `e`, `sin`, `cos`, `tan` | +| Conversion | `toInteger`, `toFloat`, `toString`, `toBoolean`, their `...OrNull` forms, `coalesce` | +| String | `toUpper`, `toLower`, `trim`, `ltrim`, `rtrim`, `substring`, `left`, `right`, `split`, `replace`, `reverse`, `char_length`, `normalize` | +| Math | `abs`, `sign`, `ceil`, `floor`, `round`, `sqrt`, `exp`, `log`, `log10`, `pi`, `e`, `sin`, `cos`, `tan`, and more | | Temporal | `datetime`, `duration` | | Embedding | `cosine_similarity`, `euclidean_distance` | -| Aggregate | `count`, `collect`, `sum`, `avg`, `min`, `max` | +| Aggregate | `count`, `collect`, `sum`, `avg`, `min`, `max`, `stDev`, `stDevP`, `percentileCont`, `percentileDisc` | ```cypher MATCH (n)-[e]->(m) @@ -299,8 +333,9 @@ MATCH (n:Document) RETURN n.title, cosine_similarity(n.emb, (0.4, 0.3, 0.8, 0.1)) ``` -- Function names are case-sensitive, except `count` and `collect`. +- Function names are case-insensitive. - Use `type(e)` for an edge's type (it replaces `edgeType()`). +- Read map keys with `m.key` or `m[k]`, including maps stored as properties. ## Data types @@ -348,13 +383,22 @@ DETACH DELETE n MATCH (n:Person)-[e:KNOWS]->(m) DELETE e +FOREACH (i IN [1, 2, 3] | CREATE (:Num {v: i})) + +UNWIND [[0, 1], [2, 3]] AS p +MATCH (a) +WHERE a = p[0] +MATCH (b) +WHERE b = p[1] +CREATE (a)-[:KNOWS]->(b) + CHANGE SUBMIT ``` - Every node needs at least one label and every edge exactly one type. Labels are set at creation. -- Run `COMMIT` after creating entities that a later query must match. `CHANGE SUBMIT` commits for you. +- Run `COMMIT` after creating entities that a later query must `MATCH`. `MERGE` sees them without it, and `CHANGE SUBMIT` commits for you. - In the Python SDK, call `client.checkout()` after `CHANGE SUBMIT`. -- For more than a few thousand entities, use [`LOAD PARQUET`](/import_data/parquet). +- To build a new graph from more than a few thousand entities, use [`LOAD PARQUET`](/import_data/parquet). To add edges between existing nodes, `UNWIND` pairs of node IDs as above. ## Procedures @@ -403,4 +447,4 @@ See [Shortest path](/shortest-path) and [Vector search](/vector-search). | `CREATE VECTOR INDEX WITH DIMENSION METRIC ` | Create a vector index | | `SHOW VECTOR INDEXES` / `DELETE VECTOR INDEX ` | List / delete vector indexes | | `INSTALL ` / `SHOW EXTENSIONS` / `SHOW PROCEDURES` | Load a procedure extension / list extensions / list procedures | -| `EXPLAIN ` | Show the query plan without running it | +| `EXPLAIN ` / `EXPLAIN (db) ` | Show the query plan without running it / only the database plan | diff --git a/query/cypher_subset.mdx b/query/cypher_subset.mdx index 7f57456..4171773 100644 --- a/query/cypher_subset.mdx +++ b/query/cypher_subset.mdx @@ -4,7 +4,7 @@ sidebarTitle: "Overview" description: "TuringDB v3 runs Cypher on a new engine and supports most of openCypher: OPTIONAL MATCH, WITH, UNWIND, UNION, MERGE, variable-length paths, subqueries and more." --- -TuringDB v3 (`turingdb` 3.0 on PyPI) runs every query on a new MLIR-based engine and supports **most of openCypher**, extended with versioning, vector search and fast lookup by native node ID. +TuringDB v3 (`turingdb` 3.0.2 on PyPI) runs every query on a new MLIR-based engine and supports **most of openCypher**, extended with versioning, vector search and fast lookup by native node ID. If you know Cypher from Neo4j, you can write the queries you are used to. This page covers the basics and the handful of places where TuringDB differs. The rest of the language is documented in the pages that follow. @@ -13,11 +13,14 @@ If you know Cypher from Neo4j, you can write the queries you are used to. This p | Area | Supported | | ---- | --------- | | Reading | `MATCH`, `OPTIONAL MATCH`, `WHERE`, `WITH`, `RETURN [DISTINCT]`, `ORDER BY`, `SKIP`, `LIMIT`, `UNWIND`, `UNION [ALL]`, `LOAD CSV` | -| Patterns | Multiple labels, edge type alternatives (`[:A\|B]`), inline `WHERE`, variable-length paths (`*1..3`, `->+`, `{2,4}`), quantified path patterns, named paths, pattern predicates | -| Expressions | Arithmetic, string and list concatenation, `CASE`, list and pattern comprehensions, `reduce`, `all` / `any` / `none` / `single`, list slicing, maps | -| Aggregation | `count`, `sum`, `avg`, `min`, `max`, `collect`, `DISTINCT`, implicit grouping, aggregates inside expressions | -| Subqueries | `EXISTS { }`, `COUNT { }`, `CALL { }` and `OPTIONAL CALL { }` | -| Writing | `CREATE`, `MERGE` with `ON CREATE` / `ON MATCH`, `SET`, `REMOVE`, `DELETE`, `DETACH DELETE` | +| Composing queries | `NEXT` to chain queries, `WHEN ... THEN ... ELSE` conditional queries | +| Patterns | Multiple labels, label expressions (`:A\|B`, `:!A`, `:A&B`), edge type alternatives (`[:A\|B]`), inline `WHERE`, variable-length paths (`*1..3`, `->+`, `{2,4}`), quantified path patterns, named paths, pattern predicates | +| Expressions | Arithmetic, string and list concatenation, `CASE`, list and pattern comprehensions, `reduce`, `all` / `any` / `none` / `single`, list slicing, maps and map access | +| Functions | String (`substring`, `split`, `replace`, `left`, `right`, `normalize`, ...), math, conversion, `keys`, `properties` | +| Aggregation | `count`, `sum`, `avg`, `min`, `max`, `collect`, `stDev`, `stDevP`, `percentileCont`, `percentileDisc`, `DISTINCT`, implicit grouping, aggregates inside expressions | +| Subqueries | `EXISTS { }`, `COUNT { }`, `COLLECT { }`, `CALL { }`, `CALL (*) { }` and `OPTIONAL CALL { }` | +| Writing | `CREATE`, `MERGE` with `ON CREATE` / `ON MATCH`, `SET`, `REMOVE`, `DELETE`, `DETACH DELETE`, `FOREACH` | +| Parameters | Labels and edge types (`MATCH (n:$label)`) | | Types | String, Integer, Double, Boolean, List, Map, `DateTime`, `Duration`, Embedding | | TuringDB extras | Changes, commits and time travel, `shortestPath`, `VECTOR SEARCH`, `gnn.graphSAGE`, lookup by native node ID | @@ -61,25 +64,36 @@ df = client.query("MATCH (n:Person) RETURN n.name, n.age") # a pandas DataFram ``` - `EXPLAIN ` returns the compiled query plan without running the query. It is the easiest way to check that a lookup uses a constant node scan. + `EXPLAIN ` returns the compiled query plan without running the query, and `EXPLAIN (db) ` shows only the database plan. It is the easiest way to check how a lookup runs. +### Query parameters + +`client.query()` takes an optional dictionary of parameters. A `$name` parameter can stand for a **label or an edge type**, anywhere a label or type is written: + +```python +client.query("MATCH (n:$label) RETURN n.name", {"label": "Person"}) +client.query("MATCH (n:$labels) RETURN n.name", {"labels": ["Person", "Employee"]}) # every label in the list +client.query("MATCH (a)-[r:$type]->(b) RETURN a.name, b.name", {"type": "WORKS_AT"}) +client.query("CREATE (n:$label {name: 'Zoe'})", {"label": "Person"}) # inside a change +``` + +Put values directly in the query text for now: value parameters, such as `WHERE n.name = $name`, are not yet supported. In the [interactive shell](/commands#interactive-shell), set parameters with `:param`. + ## Coming from Neo4j TuringDB follows openCypher closely, so the queries you know carry over. These are the TuringDB specifics worth knowing: -- **`RETURN n` returns the node's internal ID**, an integer. Return the properties you need with `RETURN n.name, n.age`, or use [`db.getNodes`](/query/procedures#inspecting-nodes-and-edges) to see every property of a few nodes. -- **Values go in the query text.** Queries are plain strings, so put literal values directly in the text and escape quotes in your application. Query parameters such as `$name` are not yet supported. -- **One statement per request.** Send each statement as its own request (a single trailing `;` is fine). +- **`RETURN n` returns the node's internal ID**, an integer. Return the properties you need with `RETURN n.name, n.age`, or all of them with `RETURN properties(n)`. +- **Values go in the query text.** [Parameters](#query-parameters) name labels and edge types. Put literal values directly in the text and escape quotes in your application. +- **One statement per request.** Send each statement as its own request (a single trailing `;` is fine), or chain steps in one request with [`NEXT`](/query/aggregation#next). - **Labels are set at creation.** Every node has at least one label and every edge exactly one edge type, chosen when it is created. - **Property types are global.** A property name has a single value type across the whole graph: if `age` is an Integer, it is an Integer on every node. -- **Typed numeric equality.** Compare Integers with Integers, and match Doubles with a range (`n.score > 1.99 AND n.score < 2.01`) or convert with `toInteger`. `<`, `>` and arithmetic work across numeric types. +- **Comparing Doubles.** Integers and Doubles compare by value (`n.age = 30.0` matches 30). Match one Double against another with a range (`n.score > 1.99 AND n.score < 2.01`). - **Forgiving names.** An unknown label matches nothing and a misspelled property reads as `null`. Check names with [`CALL db.labels()` and `CALL db.propertyTypes()`](/query/procedures). -- **Label alternatives.** Write `WHERE n:A OR n:B` or `WHERE NOT n:A`. Edge type alternatives (`[:A|B]`) work directly in patterns. -- **String matching.** Use `STARTS WITH`, `ENDS WITH` and `CONTAINS`. Regular expressions (`=~`) are not yet supported. -- **Loops and uniqueness.** Use `UNWIND` for loops and `MERGE` to avoid duplicates. +- **String matching.** Use `STARTS WITH`, `ENDS WITH`, `CONTAINS` and the [string functions](/query/functions). Regular expressions (`=~`) are not yet supported. +- **Uniqueness.** Use `MERGE` to avoid duplicates. - **Shortest paths** use a TuringDB statement, see [Shortest path](/shortest-path). -- **Function names are case-sensitive** (`toInteger`, not `TOINTEGER`), except `count` and `collect`. Coming from TuringDB 1.x? The v3 engine replaces the previous one, and many earlier limits are gone: `OPTIONAL MATCH`, `WITH`, `DISTINCT`, `UNION`, `MERGE`, `DELETE`, aggregates in any position, `UNWIND` over any list, `LOAD CSV` followed by `MATCH`, and `CREATE ... SET` all work now. One function changed name: `edgeType(e)` is now `type(e)`. diff --git a/query/functions.mdx b/query/functions.mdx index cec5adf..8aa11f9 100644 --- a/query/functions.mdx +++ b/query/functions.mdx @@ -6,20 +6,20 @@ description: "Built-in functions, how values come back to the Python SDK, dates ## Built-in functions -Function names are **case-sensitive** (`toInteger`, not `TOINTEGER`), except `count` and `collect`. +Function names are case-insensitive, like keywords: `toInteger`, `TOINTEGER` and `ToInteger` are the same function. | Category | Functions | Notes | | -------- | --------- | ----- | -| Entity | `id(x)`, `labels(n)`, `type(r)`, `startNode(r)`, `endNode(r)` | `labels` returns a **list**. `type` returns a string. `startNode(r).name` works | +| Entity | `id(x)`, `labels(n)`, `type(r)`, `startNode(r)`, `endNode(r)`, `keys(x)`, `properties(x)` | `labels` and `keys` return **lists**. `properties` returns a map of a node's, an edge's or a map's properties. `type` returns a string. `startNode(r).name` works | | Path | `nodes(p)`, `relationships(p)`, `length(p)` | Lists of IDs. Read properties through a [comprehension](/query/aggregation#list-comprehensions) | | List | `size`, `length`, `head`, `last`, `tail`, `reverse`, `range(start, end[, step])`, `reduce` | `size`, `length` and `reverse` also work on strings. `range` bounds are inclusive | | Predicates | `all`, `any`, `none`, `single` `(x IN list WHERE ...)` | | -| Conversion | `toInteger`, `toFloat`, `toString`, `toBoolean`, `coalesce(a, b, ...)` | `toInteger` and `toFloat` accept strings, integers and doubles, and return `null` for a bad string. `toBoolean` accepts strings. `coalesce` arguments must share one type | -| String | `toUpper`, `toLower`, `trim`, `ltrim`, `rtrim`, `reverse` | Plus the operators `STARTS WITH`, `ENDS WITH`, `CONTAINS` and `\|\|` | +| Conversion | `toInteger`, `toFloat`, `toString`, `toBoolean`, `toIntegerOrNull`, `toFloatOrNull`, `toStringOrNull`, `toBooleanOrNull`, `coalesce(a, b, ...)` | `toInteger` accepts strings, numbers and booleans, and `toFloat` strings and numbers. Both return `null` for a bad string. `toBoolean` accepts strings and integers. `toString` formats `DateTime` and `Duration` values. The `OrNull` forms return `null` for any value they cannot convert. `coalesce` arguments may mix types | +| String | `toUpper`, `toLower`, `trim`, `ltrim`, `rtrim`, `substring(s, start[, length])`, `left(s, n)`, `right(s, n)`, `split(s, delimiter)`, `replace(s, search, replacement)`, `reverse`, `char_length` (or `character_length`), `normalize(s[, NFC\|NFD\|NFKC\|NFKD])` | Positions start at 0. Plus the operators `STARTS WITH`, `ENDS WITH`, `CONTAINS`, `IS NORMALIZED` and `\|\|` | | Math | `abs`, `sign`, `ceil`, `floor`, `round`, `sqrt`, `exp`, `log`, `log10`, `pi()`, `e()` | Trigonometry: `sin`, `cos`, `tan`, `cot`, `asin`, `acos`, `atan`, `degrees`, `radians`, `haversin` | | Temporal | `datetime(isoString)`, `datetime(epochSeconds)`, `datetime()`, `duration({days: 2, hours: 3})`, `duration(micros)` | See [Dates and durations](#dates-and-durations) | | Embedding | `cosine_similarity(a, b)`, `euclidean_distance(a, b)` | Arguments must be Embeddings, not lists. See [Vector search](/vector-search) | -| Aggregate | `count`, `collect`, `sum`, `avg`, `min`, `max` | See [Aggregation](/query/aggregation#aggregation) | +| Aggregate | `count`, `collect`, `sum`, `avg`, `min`, `max`, `stDev`, `stDevP`, `percentileCont`, `percentileDisc` | See [Aggregation](/query/aggregation#aggregation) | ```cypher MATCH (n)-[e]->(m) @@ -41,13 +41,24 @@ RETURN abs(-3), round(2.5), sqrt(16.0), pi() MATCH (n:Person) RETURN n.name, coalesce(n.nickname, n.name) + +RETURN substring('hello', 1, 3), left('hello', 2), split('a,b,c', ','), replace('aXbX', 'X', '-') + +MATCH (n:Person) +WHERE toLower(n.name) STARTS WITH 'a' +RETURN n.name + +MATCH (n:Person {name: 'Alice'}) +RETURN keys(n), properties(n) + +RETURN toIntegerOrNull('x'), toString(datetime('2024-03-15T10:30:00Z')) ``` In v3, `type(e)` returns an edge's type as a string. It replaces `edgeType()` from earlier versions. -The function library keeps growing. Some functions, such as `substring`, `split`, `rand()`, `keys` and `properties`, are not yet available. For now, filter strings with `STARTS WITH`, `ENDS WITH` and `CONTAINS`, and do any other post-processing in your application. +The function library keeps growing. `rand()`, `date()`, `timestamp()`, `elementId()`, `atan2()`, `isNaN()` and the two-argument `round(x, digits)` are not yet available, so compute those in your application. ## Arithmetic in `RETURN` and `SET` @@ -65,7 +76,7 @@ WHERE r.since > 0 RETURN r.since / 10 * 100 ``` -`+ - * / % ^` are supported. Integer `/` truncates, and `^` returns a Double. `+` and `||` concatenate strings and lists, and `'x' + 1` gives `'x1'`. +`+ - * / % ^` are supported. Integer `/` truncates, and `^` returns a Double. `+` and `||` concatenate strings and lists, `'x' + 1` gives `'x1'`, and `[1, 2] + 3` gives `[1, 2, 3]`. ## Dates and durations @@ -109,7 +120,7 @@ RETURN duration({hours: 1}) > duration({minutes: 30}) AS longer ## Maps -Map literals can be built, returned, compared with `=`, and stored as node or edge properties: +Maps can be built, returned, compared, sorted and stored as node or edge properties, and their keys can be read anywhere in a query: ```cypher RETURN {name: 'Alice', tags: ['a', 'b']} AS m @@ -117,15 +128,24 @@ RETURN {name: 'Alice', tags: ['a', 'b']} AS m WITH {a: 1, b: 'x'} AS m RETURN m.a, m.b +WITH {a: 1, b: 2} AS m +UNWIND keys(m) AS k +RETURN k, m[k] + UNWIND [{a: 1}, {a: 2}] AS m RETURN m.a -``` -You can read a key from a map **value** that is in scope, for example one bound by `WITH` or `UNWIND`, with `m.key`. +WITH [{a: 1}, {a: 2}] AS ms +RETURN ms[1].a, [m IN ms | m.a] - - To read individual values from a map stored as a property, return the property as a whole or store the values as separate properties. Dereferencing a stored map inside a query (`n.meta.level`, `m['key']`) and map projection (`n {.name, .age}`) are not yet supported. - +MATCH (n:Person {name: 'Alice'}) +RETURN properties(n).name, keys(n) +``` + +- `m.key` reads a key written in the query, and `m[k]` a key held in a variable. A map stored as a property reads the same way: `n.meta.level`. +- Write a fixed key as `m.key`: the bracket form with a literal (`m['key']`) is reported with that hint. +- Map projection (`n {.name, .age}`) is not yet supported. Build the map you need (`{name: n.name, age: n.age}`) or use `properties(n)`. +- Values read from a map can hold any type. To store one as a property, convert it first, for example `toString(row.name)` or `toInteger(row.pop)` (see [Bulk writes](/query/writing#bulk-writes)). ## How results come back @@ -143,12 +163,10 @@ You can read a key from a map **value** that is in scope, for example one bound | Embedding | `Embedding` | object (list over HTTP, numpy `float32` array over the native and embedded backends) | | DateTime | `DateTime` | `datetime64[us, UTC]` | | Duration | `Duration` | `timedelta64[us]` | -| Node or edge (`RETURN n`) | `UInt64` | `UInt64` (internal ID) | +| Node or edge (`RETURN n`) | `UInt64` | `UInt64` (internal ID, `null` when an `OPTIONAL MATCH` found nothing) | | Named path (`RETURN p`) | `EntityList` | object (list of `{type, id}` dicts) | - - Over the default HTTP/JSON backend, Doubles are serialised with **6 decimal places** (`3.14159265` becomes `3.141593`). If you need full precision, scale the value in the query, or use the [native or embedded backend](/pythonsdk/reference#client-backends). - +Doubles keep their full precision on every backend (`0.1 + 0.2` comes back as `0.30000000000000004`). `NaN` and infinities come back as the floats `nan` and `inf`, and `NaN` sorts after every number. ## Embeddings diff --git a/query/matching.mdx b/query/matching.mdx index 6caaf37..da71321 100644 --- a/query/matching.mdx +++ b/query/matching.mdx @@ -1,7 +1,7 @@ --- title: "Matching Patterns" sidebarTitle: "Matching" -description: "Read data with MATCH, OPTIONAL MATCH and WHERE, join patterns, and look up nodes quickly by native node ID." +description: "Read data with MATCH, OPTIONAL MATCH and WHERE, combine labels with label expressions, join patterns, and look up nodes quickly by native node ID." --- Read queries run directly against the current graph, with no change to open first. @@ -63,7 +63,7 @@ RETURN b.name ``` - `(n:Person:Employee)` matches nodes that have **both** labels. -- `(a)-[e]-(b)` (or `(a)--(b)`) ignores direction. +- `(a)-[e]-(b)` (or `(a)--(b)`, `(a)<-[e]->(b)`, `(a)<-->(b)`) ignores direction. - `[:KNOWS|WORKS_AT]` matches either edge type. An edge has exactly one type, so each edge pattern takes a single type or a list of alternatives. - `type(r)` returns the edge type as a string. - Several `MATCH` clauses in a row are allowed, and a later one can reuse variables from an earlier one. @@ -73,6 +73,29 @@ RETURN b.name `RETURN n` returns the node's internal ID, not its properties. Return the properties you need (`RETURN n.name, n.age`). This is different from Neo4j, where `RETURN n` returns the whole node. +### Label expressions + +Label expressions combine labels with `|` (or), `&` (and) and `!` (not), on nodes and on edge types: + +```cypher +MATCH (n:Person|Company) +RETURN n.name + +MATCH (n:!Person) +RETURN n.name + +MATCH (n:Person&Employee) +RETURN n.name + +MATCH (n:(Person&!Employee)|Company) +RETURN n.name + +MATCH (a)-[r:!KNOWS]->(b) +RETURN type(r), count(*) +``` + +The same expressions work as tests in `WHERE`: `WHERE n:Person|Company`. A label or an edge type can also come from a [query parameter](/query/cypher_subset#query-parameters): `MATCH (n:$label)`. + ### Filtering inside the pattern A `WHERE` can sit inside a node or edge pattern. The variable must be named: @@ -112,7 +135,7 @@ RETURN a.name, count(b) The second query returns `0` for people who know nobody. A `WHERE` attached to an `OPTIONAL MATCH` belongs to the optional part, so rows are kept, as in Neo4j. -Return a property (`c.name`) or `id(c)` from an optional node, rather than the bare variable. +An unmatched node or edge returned on its own (`RETURN c`) is `null`, like its properties. ## WHERE and operators @@ -161,7 +184,7 @@ WHERE NOT (p)-[:KNOWS]->() RETURN p.name ``` -- **Label and type tests:** `n:Person` tests a node label, `e:KNOWS` tests an edge type. Combine them with `AND`, `OR` and `NOT`. +- **Label and type tests:** `n:Person` tests a node label, `e:KNOWS` tests an edge type. Combine them with `AND`, `OR` and `NOT`, or write a [label expression](#label-expressions) such as `n:Person|Company`. - **Pattern predicates:** `WHERE (p)-[:LIKES]->()` keeps the nodes for which the pattern exists. For richer conditions see [`EXISTS` subqueries](/query/subqueries). | Operators | | @@ -169,9 +192,9 @@ RETURN p.name | Comparison | `=`, `<>`, `!=`, `<`, `<=`, `>`, `>=`, and chained comparisons such as `1 < x < 3` | | Null tests | `IS NULL`, `IS NOT NULL` | | Boolean | `AND`, `OR`, `XOR`, `NOT`, with three-valued logic for `null` as in Neo4j | -| Membership and strings | `IN`, `STARTS WITH`, `ENDS WITH`, `CONTAINS` | +| Membership and strings | `IN`, `STARTS WITH`, `ENDS WITH`, `CONTAINS`, `IS [NFC\|NFD\|NFKC\|NFKD] NORMALIZED` | | Arithmetic | `+`, `-`, `*`, `/`, `%`, `^`. Integer `/` truncates and `^` returns a Double | -| Concatenation | `+` and `\|\|` join strings and lists | +| Concatenation | `+` and `\|\|` join strings and lists, and `list + x` appends an element | Property constraints in a pattern, `MATCH (n {name: 'Matt', age: 20})`, are exactly equivalent to the same equality tests in `WHERE`. @@ -181,26 +204,27 @@ Property constraints in a pattern, `MATCH (n {name: 'Matt', age: 20})`, are exac ### Comparing numbers -Equality compares numbers of the same kind. An Integer compared with a Double using `=`, or two Doubles compared with `=`, is reported when the query is analysed: - -```text -MATCH (n:Person) WHERE n.age = 30.0 RETURN n.name -ANALYZE_ERROR: Operands are not valid or compatible types -``` - -Use range comparisons for Doubles, or convert the value: +Integers and Doubles compare by value, so an Integer `30` equals the Double `30.0`: ```cypher MATCH (n:Person) -WHERE n.age > 29.5 AND n.age < 30.5 +WHERE n.age = 30.0 RETURN n.name MATCH (n:Person) -WHERE n.age = toInteger(30.0) +WHERE n.age IN [30.0, 35] RETURN n.name ``` -Mixed-type `<`, `>` and arithmetic work as expected. Division by zero returns an execution error, and Int64 arithmetic wraps around on overflow. +To match one Double against another, use a range. `=` between two Doubles is reported when the query is analysed, because of rounding: + +```cypher +MATCH (n:Person) +WHERE n.age * 1.1 > 32.9 AND n.age * 1.1 < 33.1 +RETURN n.name +``` + +Mixed-type `<`, `>` and arithmetic work as expected, and an ordering between values with no common order, such as `1 < 'a'`, is `null`. Division by zero returns an execution error, for Doubles too, and Int64 arithmetic wraps around on overflow. ## Joins and Cartesian products @@ -240,7 +264,7 @@ RETURN a.name, i.name, c.name ## Look up nodes by native node ID -Every node has a **native node ID**, the integer you get from `RETURN n` or `id(n)`. When you already know which nodes you want, **compare the node variable itself with the ID**, written as a literal. The planner turns this into a constant scan of exactly those nodes: +Every node has a **native node ID**, the integer you get from `RETURN n` or `id(n)`. When you already know which nodes you want, compare the node with its ID, `WHERE n = 123` or `WHERE id(n) = 123`. TuringDB then looks those nodes up directly instead of scanning the graph: ```cypher MATCH (n) @@ -248,16 +272,23 @@ WHERE n = 0 RETURN n.name MATCH (n)-[:KNOWS]->(m) -WHERE n = 0 +WHERE id(n) = 0 RETURN m.name UNWIND [0, 1, 2] AS x MATCH (n) WHERE n = x -RETURN id(n), n.name +RETURN x, n.name + +UNWIND [[0, 1], [2, 3]] AS p +MATCH (a) +WHERE a = p[0] +MATCH (b) +WHERE b = p[1] +RETURN a.name, b.name ``` -`EXPLAIN` shows whether it worked. The plan should contain `const_scan_nodes` rather than `scan_nodes`: +`EXPLAIN` shows the lookup: the plan contains `const_scan_nodes` for IDs written in the query, or `fetch_nodes` for IDs read from a row, rather than `scan_nodes`: ```cypher EXPLAIN MATCH (n) @@ -265,16 +296,9 @@ WHERE n = 0 RETURN n.name ``` -On large graphs this is orders of magnitude faster than looking nodes up again through a key property such as `n.uuid`. The rules that keep it fast: - -- **Compare the variable, not the function.** `WHERE n = 123` is a direct seek. `WHERE id(n) = 123` scans every node. Use `id(n)` only to *return* the ID. -- **Use `UNWIND` for batches, not `IN`.** `WHERE n IN [...]` and `WHERE id(n) IN [...]` scan every node. The `UNWIND [] AS x MATCH (n) WHERE n = x` form is rewritten at plan time into one constant scan, which costs roughly a microsecond per node. -- **The batch form needs an exact shape:** - - the list is a **literal written in the query text** (build it in Python: `f"UNWIND {ids} AS x ..."`), not a variable, `collect(...)`, `range(...)` or a list property; - - its elements are **distinct** integers, so deduplicate first; - - `x` is used **only** in `n = x`. Return `id(n)` rather than `x`. -- **Fetch IDs once and reuse them.** Look nodes up with whatever filter you need (`MATCH (n:Person {name: 'Alice'}) RETURN id(n)`), keep the IDs in your application, and address the nodes by ID from then on. -- **Keep IDs constant.** Use literal IDs rather than IDs computed per row, and match one node per `MATCH`. Forms such as `WHERE a = p[0] AND b = p[1]` are not rewritten into a constant scan, so they run as a Cartesian product of two full node scans, which is expensive on large graphs. To connect two known nodes, put both IDs in as literals, one pair per query: `MATCH (a) WHERE a = 5 MATCH (b) WHERE b = 7`. For bulk edges, see [Parquet import](/import_data/parquet). +- **Batches.** `UNWIND` a list of IDs and compare each one: `UNWIND AS x MATCH (n) WHERE n = x`. The list can be written in the query (`f"UNWIND {ids} AS x ..."` in Python), held in a variable, or built with `collect()` or `range()`, and each ID costs a few microseconds. Use this form rather than `WHERE n IN [...]`, which checks every node. +- **Several IDs per row.** Two IDs from the same row work too, which is the fast way to [create many edges between existing nodes](/query/writing#edges-between-existing-nodes). +- **Fetch IDs once and reuse them.** Look nodes up with whatever filter you need (`MATCH (n:Person {name: 'Alice'}) RETURN id(n)`), keep the IDs in your application, and address the nodes by ID from then on. Looking each row's node up through a key property such as `n.uuid` reads every node with that label instead, and is about 100 times slower on a large graph. - **Vector search results are already nodes.** `VECTOR SEARCH ... YIELD ids` binds `ids` as nodes, so use `ids.title` or `MATCH (ids)-[...]->(...)` directly. See [Vector search](/vector-search). **Stability.** The native IDs of existing nodes are stable across queries, change submits and server restarts, so it is safe to cache them. Two exceptions: diff --git a/query/paths.mdx b/query/paths.mdx index 6505a3d..dcddc35 100644 --- a/query/paths.mdx +++ b/query/paths.mdx @@ -36,13 +36,16 @@ RETURN b.name MATCH (a {name: 'Alice'})-[e:KNOWS]->+(b) RETURN b.name, size(e) + +MATCH (a {name: 'Alice'})-[e:KNOWS]->+(b) +RETURN b.name, [r IN e | r.since] AS sinces ``` - **Edge type alternatives** (`[:KNOWS|WORKS_AT]`) work on variable-length hops. - **A predicate or property map applies to every hop.** `-[e:KNOWS WHERE e.since > 2012]->+` only follows edges that satisfy it, and `-[:KNOWS*1..3 {since: 2010}]->` requires every edge on the path to have that property value. - **Trail semantics.** Each edge is used at most once within a path, while nodes may repeat. Each distinct path is its own row, so use `DISTINCT` or an aggregate to collapse them. - **One quantifier per edge.** Choose either `[*1..3]` or `{1,3}` for an edge. -- **The edge variable of a variable-length hop is a list of edge IDs.** `size(e)`, `e[0]`, `e[-1]` and `UNWIND e` work on it. To read edge properties along a path, name the path and use `relationships(p)` (below). +- **The edge variable of a variable-length hop is a list of edges.** `size(e)`, `e[0]`, `e[-1]`, `UNWIND e` and comprehensions such as `[r IN e | r.since]` work on it. Variable-length matching **enumerates** paths. On dense graphs the number of paths can grow very quickly, so always bound the length (`*1..6` or `{1,6}`). For weighted shortest paths on large graphs, use the [`shortestPath` statement](/shortest-path). @@ -50,7 +53,7 @@ RETURN b.name, size(e) ## Quantified path patterns -A quantified path pattern repeats a one-hop sub-pattern, with its own `WHERE`, a number of times: +A quantified path pattern repeats a sub-pattern, with its own `WHERE`, a number of times: ```cypher MATCH (a:Person {name: 'Alice'}) ((x)-[r:KNOWS]->(y) WHERE r.since > 2009){1,3} (b) @@ -58,9 +61,15 @@ RETURN b.name MATCH (a:Person {name: 'Alice'}) ((x)-[:KNOWS]->(y) WHERE y.age > x.age)+ (b) RETURN b.name + +MATCH (a:Person {name: 'Alice'}) ((x)-[:KNOWS]->(y)-[:KNOWS]->(z)){1,2} (b) +RETURN b.name + +MATCH (a:Person {name: 'Alice'}) ((x)-[r:KNOWS]->(y)){1,3} (b) +RETURN b.name, [n IN y | n.name] AS via, size(r) AS hops ``` -The second query follows `KNOWS` edges only while ages strictly increase. Inside the parentheses only a **single hop** of the form `(node)-[edge]->(node) [WHERE ...]` is allowed, and the inner variables (`x`, `r`) become lists. +The second query follows `KNOWS` edges only while ages strictly increase, and the third repeats a two-hop sub-pattern. The inner variables (`x`, `y`, `r`) are **group variables**: each becomes a list with one element per repetition, which you can use in comprehensions, `size()` and `UNWIND`. ## Named paths @@ -77,7 +86,7 @@ ORDER BY hops - `nodes(p)` and `relationships(p)` return **lists of IDs**. Read properties through a [list comprehension](/query/aggregation#list-comprehensions). - `length(p)` is the number of hops. - A bare `RETURN p` comes back as a list of `{"type": "node" | "edge", "id": ...}` dictionaries. -- `count(p)` counts paths. +- `count(p)` counts paths, and `RETURN DISTINCT p` groups on whole paths. - `OPTIONAL MATCH p = ... RETURN p IS NULL` works. ## Finding routes @@ -98,19 +107,15 @@ MATCH (a:Person {name: 'Alice'})-[:KNOWS]->+(b) RETURN count(DISTINCT b) ``` -To sum a property along a route, build a list with a comprehension and aggregate over its elements: +To sum a property along a route, fold the path's edges with `reduce`: ```cypher MATCH p = (a:Person {name: 'Alice'})-[:KNOWS*1..6]->(b:Person {name: 'Dave'}) -WITH [n IN nodes(p) | n.name] AS route, [r IN relationships(p) | r.since] AS years -UNWIND years AS y -WITH route, sum(y) AS total -RETURN route, total +RETURN [n IN nodes(p) | n.name] AS route, + reduce(total = 0, r IN relationships(p) | total + r.since) AS total ORDER BY total LIMIT 1 ``` - - To sum a property along a route, compute the list with a comprehension first, as above. - +To aggregate over the edges of every matched path at once, `UNWIND relationships(p) AS r` and aggregate `r.since`. For a true weighted shortest path, see [Shortest path](/shortest-path). diff --git a/query/procedures.mdx b/query/procedures.mdx index 095b5ee..f300ead 100644 --- a/query/procedures.mdx +++ b/query/procedures.mdx @@ -64,11 +64,19 @@ RETURN `commit`, nodeCount ### Inspecting nodes and edges -`RETURN n` gives a node's internal ID. To see every property of a few nodes, use `db.getNodes`, which returns the properties as a JSON string: +`RETURN n` gives a node's internal ID. To see every property of some nodes, return `properties(n)`, which comes back as a dictionary, or `keys(n)` for just the property names: ```cypher -CALL db.getNodes([0, 1, 2]) YIELD id, labels, properties -RETURN id, labels, properties +MATCH (n:Person) +RETURN id(n), labels(n), properties(n) +LIMIT 5 +``` + +The procedures `db.getNodes` and `db.listNodes` also return the properties, as a JSON string, together with edge counts: + +```cypher +CALL db.getNodes([0, 1, 2]) YIELD id, labels, inEdgeCount, outEdgeCount, properties +RETURN id, labels, inEdgeCount, outEdgeCount, properties CALL db.listNodes(['Person'], {name: 'ali'}, 0, 10) YIELD id, labels, properties ``` @@ -108,12 +116,12 @@ Outside of Cypher, a set of commands controls the engine. Send them through `cli | `CREATE GRAPH ` | Create an empty graph | | `LOAD GRAPH ` | Load an on-disk graph. Its files must be in the TuringDB directory (`-turing-dir`, `graphs` subdirectory) | | `LIST GRAPH` | List the graphs currently **loaded** in the server | -| `LIST AVAILABLE GRAPHS` | List the graphs on disk, with `isLoaded` and `isLoading` | +| `LIST AVAILABLE GRAPHS` | List the graphs on disk and in memory, with `isLoaded` and `isLoading` | | `LOAD JSONL 'f.jsonl' AS ` | Build a graph from a [JSONL](/import_data/jsonl) file in the `data` subdirectory | | `LOAD GML 'f.gml' AS ` | Build a graph from a [GML](/import_data/gml) file in the `data` subdirectory | | `LOAD PARQUET '' AS ` | Build a graph from Parquet files, see [Parquet import](/import_data/parquet) | | `LOAD CSV '' ... AS row` | Stream rows from a CSV file, see [CSV import](/import_data/csv) | -| `EXPLAIN ` | Show the compiled plan without running the query | +| `EXPLAIN ` | Show the compiled plan without running the query. `EXPLAIN (db) ` shows only the database plan | Dropping and unloading graphs from a running server is not yet available. @@ -125,7 +133,7 @@ Dropping and unloading graphs from a running server is not yet available. | `CHANGE SUBMIT` | Commit pending writes and merge the current change into main | | `CHANGE DELETE` | Discard the current change | | `CHANGE LIST` | List the open changes on the current graph | -| `COMMIT` | Seal the pending writes inside a change. Needed before a later query can `MATCH` or `MERGE` newly created entities | +| `COMMIT` | Seal the pending writes inside a change. Needed before a later query can `MATCH` newly created entities | | `LOAD COMMIT ''` | Load a past commit for querying. Only needed when sending queries directly to the REST API. The CLI `checkout` and the Python SDK do it for you | | `MERGE_DATAPARTS` | Compact all [DataParts](/concepts/dataparts) of the graph into one. Run it on main with no change open on any graph | diff --git a/query/subqueries.mdx b/query/subqueries.mdx index 998daf0..9fb95d5 100644 --- a/query/subqueries.mdx +++ b/query/subqueries.mdx @@ -1,7 +1,7 @@ --- title: "Subqueries" sidebarTitle: "Subqueries" -description: "Use EXISTS, COUNT and CALL subqueries to filter, count and compute per-row results in TuringDB." +description: "Use EXISTS, COUNT, COLLECT and CALL subqueries to filter, count, collect and compute per-row results in TuringDB." --- These examples use the [sample graph](/query/matching) from the matching page. @@ -47,6 +47,19 @@ WHERE COUNT { RETURN p.name ``` +## COLLECT + +`COLLECT { ... }` returns the single column of its body as a list, for each row: + +```cypher +MATCH (p:Person) +RETURN p.name, COLLECT { MATCH (p)-[:KNOWS]->(f) RETURN f.name } AS friends + +MATCH (p:Person) +WHERE size(COLLECT { MATCH (p)-[:KNOWS]-(f) RETURN f.name }) >= 2 +RETURN p.name +``` + ## CALL subqueries A `CALL { ... }` subquery runs once per incoming row. It is the way to compute a per-row top-k or a per-row aggregate. Import the outer variables either in the `CALL (p)` header or with a `WITH p` as the first clause of the body: @@ -68,6 +81,13 @@ CALL { } RETURN p.name, deg +MATCH (p:Person) +CALL (*) { + MATCH (p)-[:KNOWS]->(f) + RETURN count(f) AS deg +} +RETURN p.name, deg + MATCH (p:Person) OPTIONAL CALL (p) { MATCH (p)-[:WORKS_AT]->(c) @@ -86,10 +106,11 @@ RETURN nm ORDER BY nm ``` +- **Importing variables.** `CALL (p)` imports the listed variables, and `CALL (*)` imports every variable in scope. - **Dropped rows.** A non-optional `CALL` drops the outer rows for which its body returns nothing. Use `OPTIONAL CALL` to keep them with `null` values. -- **Imported variables.** To return an imported variable from a `CALL` body, give it an alias: `RETURN p AS me`. -- A `CALL` body can contain `UNION`, aggregation, `ORDER BY` and `LIMIT`. Write-only subqueries also work inside a [change](/query/writing). +- **Returning an imported variable.** Give it an alias in the `CALL` body: `RETURN p AS me`. +- A `CALL` body can contain `UNION`, aggregation, `ORDER BY` and `LIMIT`, or be a [conditional query](/query/aggregation#conditional-queries-with-when). Write-only subqueries also work inside a [change](/query/writing). - `COLLECT { ... }`, `CALL (*) { }`, `CALL { } IN TRANSACTIONS` and `OPTIONAL CALL` on a procedure are not yet available. To collect values, use `collect()` in a `CALL` body or a [pattern comprehension](/query/aggregation#list-comprehensions). + `CALL { } IN TRANSACTIONS`, `OPTIONAL CALL` on a procedure, and `WHEN` inside an `EXISTS`, `COUNT` or `COLLECT` body are not yet available. diff --git a/query/writing.mdx b/query/writing.mdx index 842eef9..7bf7926 100644 --- a/query/writing.mdx +++ b/query/writing.mdx @@ -1,10 +1,10 @@ --- title: "Writing Data" sidebarTitle: "Writing Data" -description: "CREATE, MERGE, SET, REMOVE and DELETE in TuringDB, and the change workflow that every write goes through." +description: "CREATE, MERGE, SET, REMOVE, DELETE and FOREACH in TuringDB, bulk edges by node ID, and the change workflow that every write goes through." --- -All writes (`CREATE`, `MERGE`, `SET`, `REMOVE`, `DELETE`) must run **inside a change**. A change is TuringDB's equivalent of a git branch: you write in an isolated copy and then submit it to the main graph. A write sent outside a change returns a clear message such as `Cannot perform CREATE outside of a write transaction.` +All writes (`CREATE`, `MERGE`, `SET`, `REMOVE`, `DELETE`, `FOREACH`) must run **inside a change**. A change is TuringDB's equivalent of a git branch: you write in an isolated copy and then submit it to the main graph. A write sent outside a change returns a clear message such as `Cannot perform CREATE outside of a write transaction.` ## The change workflow @@ -31,12 +31,12 @@ See [Create a change](/graph_dev/create_change), [Submit a change](/graph_dev/su | An earlier query in the same change did... | Visible to later queries before `COMMIT`? | | ------------------------------------------ | ----------------------------------------- | -| `CREATE` nodes or edges | **After `COMMIT`.** Run `COMMIT` so that a later `MATCH`, `MATCH ... CREATE` or `MERGE` sees the new entities | +| `CREATE` or `MERGE` nodes or edges | **To `MERGE`, yes. To `MATCH`, after `COMMIT`.** Run `COMMIT` so that a later `MATCH` or `MATCH ... CREATE` sees the new entities | | `SET`, `REMOVE` or `DELETE` on entities that already existed | Yes, immediately | | anything, followed by `COMMIT` | Yes | -| writes earlier in the **same query** | Yes | +| writes earlier in the **same query**, including earlier `NEXT` steps | Yes | -**Rule of thumb: run `COMMIT` after any query that creates entities that a later query must `MATCH` or `MERGE`.** You can use several `COMMIT`s in one change. `CHANGE SUBMIT` commits whatever is still pending, so no `COMMIT` is needed right before it. +**Rule of thumb: run `COMMIT` after any query that creates entities that a later query must `MATCH`.** You can use several `COMMIT`s in one change. `CHANGE SUBMIT` commits whatever is still pending, so no `COMMIT` is needed right before it. ```python client.new_change() @@ -89,7 +89,7 @@ CREATE (a)-[:KNOWS {since: 2021}]->(b) RETURN a.name, b.name ``` -If you know their [native node IDs](/query/matching#look-up-nodes-by-native-node-id), address them directly. That is the fastest option, but the IDs must be **constants** written in the query, one pair per query: +If you know their [native node IDs](/query/matching#look-up-nodes-by-native-node-id), address them directly. That is the fastest option: ```cypher MATCH (a) @@ -99,18 +99,16 @@ WHERE b = 1 CREATE (a)-[:KNOWS {since: 2021}]->(b) ``` +To create many edges at once, put the ID pairs in one `UNWIND` (see [Edges between existing nodes](#edges-between-existing-nodes)). + **Rules:** -- **Labels and edge types.** Create nodes with a label, `CREATE (n:Label)`, and give each edge a single type, `[:TYPE]`. +- **Labels and edge types.** Create nodes with a label, `CREATE (n:Label)`, and give each edge a single type, `[:TYPE]`. Several labels are joined with `:` or `&`, and a label or edge type can come from a [query parameter](/query/cypher_subset#query-parameters): `CREATE (n:$label)`. - **Labels are set at creation** and cannot be changed afterwards. - **Property types are global.** Each property name has **one value type across the whole graph** (see `CALL db.propertyTypes()`). If `age` is an Integer, `CREATE (:X {age: 'old'})` and `SET n.age = 2.5` are both reported as type incompatibilities, and values are not widened from Integer to Double automatically. -- **Nulls.** Leave a key out of `CREATE` when it has no value, rather than writing `{name: null}`. +- **Nulls.** A key set to `null` in `CREATE` is skipped, so `{name: null}` simply leaves `name` unset. - **Write patterns use single hops.** Variable-length edges are for reading, and a matched edge variable is not reused in `CREATE`. -- **Parallel edges and self-loops are allowed.** An undirected `CREATE (a)-[:R]-(b)` is accepted and creates `a->b`. - - - Match the two endpoints of an edge with separate `MATCH` clauses and constant IDs. Taking both from one `UNWIND` row (`a = p[0] AND b = p[1]`) plans as a Cartesian product of two full node scans, which is expensive on large graphs. For thousands of edges, use [`LOAD PARQUET`](/import_data/parquet). - +- **Edges are directed.** Write the arrow, `CREATE (a)-[:R]->(b)`. Parallel edges and self-loops are allowed. ## MERGE @@ -140,9 +138,9 @@ RETURN t.k `MERGE` deduplicates within one query, so the last example creates two `Key` nodes, not three. - - Run `COMMIT` before a `MERGE` that should find nodes created by an earlier query in the same change, otherwise `MERGE` creates them again. `MERGE` is the way to avoid duplicates. - +- `MERGE` also finds the nodes and edges that earlier queries of the same change wrote, before any `COMMIT`, so running the same `MERGE` twice creates one node. +- A `MERGE` key must have a value. A key that is `null` when the query runs is reported (`MERGE cannot match on property 'k', which is null`), so filter nulls out first with `WHERE k IS NOT NULL`. +- `MERGE` is the way to avoid duplicates. ## SET and REMOVE @@ -232,13 +230,68 @@ DELETE e, n - Use `DETACH DELETE` for a node that still has edges. A plain `DELETE` reports `Cannot delete a node with relationships; use DETACH DELETE`. - To delete a path, delete its nodes or edges. +## FOREACH + +`FOREACH` runs write clauses once for each element of a list, without producing rows: + +```cypher +FOREACH (i IN [1, 2, 3] | CREATE (:Num {v: i})) + +FOREACH (nm IN ['x', 'y', 'x'] | MERGE (:Tag {name: nm})) + +MATCH (n:Person) +WITH collect(n) AS people +FOREACH (p IN people | SET p.reviewed = true) + +FOREACH (i IN [1, 2] | FOREACH (j IN [10, 20] | CREATE (:Cell {v: i * j}))) +``` + +The second example creates two `Tag` nodes, since `MERGE` deduplicates. `FOREACH` blocks nest. + +### Conditional writes + +The branches of a [conditional query](/query/aggregation#conditional-queries-with-when) can write: + +```cypher +MATCH (n:Person) +CALL (n) { + WHEN n.age > 30 THEN CREATE (:Senior {name: n.name}) +} +RETURN count(*) +``` + ## Bulk writes - **Beyond a few thousand entities, use [`LOAD PARQUET`](/import_data/parquet) rather than Cypher writes.** Above roughly 5,000 nodes or 1,000 edges, write your data to Parquet and load it into a new graph. It loads 100,000 nodes and 200,000 edges in about 0.2 seconds, while Cypher-driven loads that `MATCH` both endpoints of every edge take minutes at a few thousand edges. Use Cypher writes for small, incremental changes. + **Building a new graph?** Beyond a few thousand entities, use [`LOAD PARQUET`](/import_data/parquet). It loads 100,000 nodes and 200,000 edges in about 0.2 seconds. + + **Adding edges between nodes that already exist?** Send their [native node IDs](/query/matching#look-up-nodes-by-native-node-id) in pairs, as below: about 10,000 edges in 45 ms. -For small volumes, inline the data with `UNWIND`: +### Edges between existing nodes + +Read each node's ID once together with your own key, map your edge list to ID pairs in your application, and create the edges in batches of about 10,000 pairs: + +```python +ids = client.query("MATCH (n:Person) RETURN n.pid AS pid, id(n) AS id") +key_to_id = dict(zip(ids["pid"].tolist(), ids["id"].tolist())) # .tolist() gives plain ints +pairs = [[key_to_id[src], key_to_id[dst]] for src, dst in edge_list] + +client.new_change() +for i in range(0, len(pairs), 10_000): + batch = pairs[i:i + 10_000] + client.query(f"UNWIND {batch} AS p " + "MATCH (a) WHERE a = p[0] MATCH (b) WHERE b = p[1] " + "CREATE (a)-[:KNOWS]->(b)") +client.query("CHANGE SUBMIT") +client.checkout() +``` + +Edge properties ride along in each row: `UNWIND [[12, 34, 2021], ...] AS p ... CREATE (a)-[:KNOWS {since: p[2]}]->(b)`. Use IDs rather than matching both endpoints by a key property (`MATCH (a:Person {pid: p[0]})`), which reads every `Person` node for each row. + +### Small volumes + +For a few thousand rows, inline the data with `UNWIND`: ```cypher UNWIND range(1, 1000) AS i @@ -251,12 +304,14 @@ UNWIND [['Brest', 'FR-29'], ['Rennes', 'FR-35']] AS row CREATE (:City {name: row[0], code: row[1]}) ``` -For rows with mixed types, such as `['Brest', 139000]`, use parallel lists: +Rows can also be maps, or lists of mixed types. Their values can hold any type, so convert each one as you store it: ```cypher -WITH ['Brest', 'Rennes'] AS names, [139000, 220000] AS pops -UNWIND range(0, size(names) - 1) AS i -CREATE (:City {name: names[i], pop: pops[i]}) +UNWIND [{name: 'Brest', pop: 139000}, {name: 'Rennes', pop: 220000}] AS row +CREATE (:City {name: toString(row.name), pop: toInteger(row.pop)}) + +UNWIND [['Brest', 139000], ['Rennes', 220000]] AS row +CREATE (:City {name: toString(row[0]), pop: toInteger(row[1])}) ``` `LOAD CSV` can drive writes too. Create the nodes, `COMMIT`, then create the edges (see [CSV import](/import_data/csv)). Alternatively `MERGE` both endpoints in a single pass, which deduplicates across the rows of one query so no `COMMIT` is needed: @@ -268,9 +323,9 @@ LOAD CSV 'knows.csv' WITH HEADERS AS row CREATE (a)-[:KNOWS {since: toInteger(row.since)}]->(b) ``` -Use `UNWIND` to loop over a list. +Use `UNWIND` or [`FOREACH`](#foreach) to loop over a list. -**Building the query text in Python.** Queries are plain text, so escape quotes in your application: +**Building the query text in Python.** Values go in the query text, so escape quotes in your application: ```python def lit(v): @@ -284,4 +339,4 @@ names = ["Alice", "O'Brien"] client.query(f"UNWIND [{', '.join(lit(n) for n in names)}] AS nm MERGE (:Person {{name: nm}})") ``` -Keep each query to a few thousand rows at most. +Keep each request under 1 MiB, the server's limit. A larger one is refused with `REQUEST_TOO_BIG`. diff --git a/shortest-path.mdx b/shortest-path.mdx index c47037f..d10db8e 100644 --- a/shortest-path.mdx +++ b/shortest-path.mdx @@ -175,10 +175,10 @@ shortestPath(a, b, distance, dist, path) RETURN dist, path """).iloc[0] -node_ids = sorted({int(i) for i in row["path"][0::2]}) -names = client.query(f"UNWIND {node_ids} AS x MATCH (n) WHERE n = x RETURN id(n) AS id, n.name AS name") +node_ids = [int(i) for i in row["path"][0::2]] # the stations, from source to target +names = client.query(f"UNWIND {node_ids} AS x MATCH (n) WHERE n = x RETURN x AS id, n.name AS name") ``` -The `UNWIND` list is a literal of distinct IDs and `x` is not returned, so this is a constant scan (see [lookup by node ID](/query/matching#look-up-nodes-by-native-node-id)). +Each station is looked up directly by its ID (see [lookup by node ID](/query/matching#look-up-nodes-by-native-node-id)). For large graphs prefer `shortestPath` over variable-length paths: variable-length matching enumerates every path. diff --git a/troubles/troubles.mdx b/troubles/troubles.mdx index 3a2cf45..f60b16a 100644 --- a/troubles/troubles.mdx +++ b/troubles/troubles.mdx @@ -40,6 +40,21 @@ See [client backends](/pythonsdk/reference#client-backends) for details. --- +## `The request body must be a JSON object` + +The server expects each query as a JSON body, `{"query": "...", "parameters": {...}}`. Two common causes: + +- **An older SDK against a 3.0.2 server.** Upgrade the Python SDK to the same version as the server: `pip install 'turingdb>=3.0.2'`. +- **A raw HTTP request with the query as plain text.** Send JSON instead, see the [HTTP API](/commands#http-api). + +--- + +## `REQUEST_TOO_BIG` + +A request can be up to 1 MiB. Split a large `UNWIND` into batches, for example 10,000 node-ID pairs per request, or load bulk data from a file with [`LOAD PARQUET`](/import_data/parquet) or [`LOAD JSONL`](/import_data/jsonl). + +--- + ## `401 Unauthorized` The server was started with `-auth-on` and your client did not send a valid token. Pass `token="..."` to `TuringDB(...)`, or set the `TURINGDB_AUTH_TOKEN` environment variable. The JSON client raises this as an `httpx.HTTPStatusError`. @@ -60,13 +75,13 @@ All writes (`CREATE`, `MERGE`, `SET`, `REMOVE`, `DELETE`) must run inside a chan ## A `MATCH` right after a `CREATE` returns no rows -Entities created earlier in a change are not visible to later queries until you run `COMMIT`. Run `COMMIT` after creating nodes and before matching them to create edges or to `MERGE`. See [Writing data](/query/writing#when-you-need-commit-inside-a-change). +Entities created earlier in a change are visible to a later `MATCH` once you run `COMMIT`. Run `COMMIT` after creating nodes and before matching them to create edges. A `MERGE` finds them without it. See [Writing data](/query/writing#when-you-need-commit-inside-a-change). --- ## `Function 'X' does not exist` -Function names are case-sensitive (`toInteger`, not `TOINTEGER`), and some functions are not yet available in TuringDB. In v3, `type()` replaces `edgeType()`. See [Functions and data types](/query/functions). +A few functions are not yet available in TuringDB, such as `rand()`, `date()`, `timestamp()` and `elementId()`. In v3, `type()` replaces `edgeType()`. See [Functions and data types](/query/functions) for the full list. --- diff --git a/vector-search.mdx b/vector-search.mdx index b276125..f2924ff 100644 --- a/vector-search.mdx +++ b/vector-search.mdx @@ -51,9 +51,11 @@ CREATE VECTOR INDEX WITH DIMENSION METRIC [TYPE ] The metrics return raw scores. `COSINE` scores are the **inner product** of the stored and query vectors, so L2-normalise both to get true cosine similarity. `EUCLID` scores are the **squared** L2 distance. +HNSW indexes are memory-mapped from disk rather than read into memory when the server starts. Start the server with [`-no-vector-mmap`](/commands#options) to read them into memory instead. + ## Load embeddings -Once the index exists, load your pre-computed embeddings from a header-less CSV file. Each line is `id,v1,v2,...,vd`, where `id` is normally the native TuringDB node ID (`id(n)`). +Once the index exists, load your pre-computed embeddings from a header-less CSV file. Each line is `id,v1,v2,...,vd`, where `id` is normally the native TuringDB node ID (`id(n)`). Loading a file again with IDs that are already in the index replaces their vectors. The file path is relative to your TuringDB data directory (`~/.turing/data` by default).