Rewrite upsertGraph() and insertGraph() (#1268, #1378, #1694, #853)

doc/guide/validation.md

@@ -75,3 +75,25 @@ try {
   }
 }
 ```
+
+## Write-only properties
+
+`insertGraph()` and `upsertGraph()` validate their models once the references and keys of the graph are resolved, after the models are created. A property that a setter consumes when the model is created, e.g. a `password` that a setter turns into a hash, is gone by then, and fails a `required` rule or skips its other rules. Keep such properties on the model and transform them in the `$beforeInsert` and `$beforeUpdate` hooks, which run after the validation in all queries:
+
+```js
+class User extends Model {
+  static jsonSchema = {
+    type: 'object',
+    required: ['password'],
+    properties: {
+      password: { type: 'string', minLength: 8 }
+    },
+  };
+
+  async $beforeInsert(context) {
+    await super.$beforeInsert(context);
+    this.passwordHash = await hash(this.password);
+    delete this.password;
+  }
+}
+```

doc/release-notes/changelog.md

@@ -54,9 +54,13 @@
 - `$fetchGraph()`, `fetchGraph()` and `$relatedQuery()` throw an error when owner models loaded from the database are missing a join property, e.g. an id left out of a partial `select()`, instead of warning and finding no related models for them. Owner models built in memory, e.g. with `fromJson()`, find no related models for a missing join property without a warning. This includes the foreign key of a `BelongsToOneRelation`, which 3.x didn't report unless it was part of the id. Models returned by `insert()` aren't loaded from the database, also with `returning()`. `$clone()` keeps whether a model was loaded from the database. See [migration](/release-notes/migration.md#missing-and-array-join-properties-throw) [#1832](https://github.com/Vincit/objection.js/issues/1832)
 - `withGraphFetched()` and `$fetchGraph()` throw an error when related models come back without their join property, e.g. snake_case relation mappings with `knexSnakeCaseMappers()`, instead of warning and dropping them. See [migration](/release-notes/migration.md#missing-and-array-join-properties-throw) [#2258](https://github.com/Vincit/objection.js/issues/2258)
 - Fetching a relation for owner models whose join property holds an array, e.g. a JSON array of keys, throws an error instead of warning and finding no related models. `$relatedQuery()` keeps working for them. See [migration](/release-notes/migration.md#missing-and-array-join-properties-throw) [#114](https://github.com/ditojs/objection/issues/114)
-- `upsertGraph()` relates the rows that move to existing owners of a `HasManyRelation` or `HasOneRelation` before it deletes the removed rows, so rows moved out of a deleted parent, up a level or to a sibling are kept, where `ON DELETE CASCADE` deleted them and foreign keys without `ON DELETE` action failed. The update hooks of these relates run before the delete hooks. A relate into a `HasOneRelation` still waits for the removal of the current model, or for the relate that moves it to another owner, also a new one or one whose slot is freed first, but not into the subgraph of a related model, which is upserted last. A row related from another owner that takes the unique key of a row that the graph removes in its new place, e.g. with `unique(ownerId, name)`, fails with a `UniqueViolationError`, where 3.x removed the other row first. See [migration](/release-notes/migration.md#upsertgraph-relates-moved-rows-before-it-deletes) [#214](https://github.com/ditojs/objection/issues/214)
+- `upsertGraph()` relates the rows that move to existing owners of a `HasManyRelation` or `HasOneRelation` before it deletes the removed rows, so rows moved out of a deleted parent, up a level or to a sibling are kept, where `ON DELETE CASCADE` deleted them and foreign keys without `ON DELETE` action failed. The update hooks of these relates run before the delete hooks. Rows that wait for the keys or references of models that the graph inserts are moved without them, and patched again once these are known, so a unique key at the new owner that only these values satisfy fails. A relate into a `HasOneRelation` still waits for the removal of the current model, or for the relate that moves it to another owner, also a new one or one whose slot is freed first. A row related from another owner that takes the unique key of a row that the graph removes in its new place, e.g. with `unique(ownerId, name)`, fails with a `UniqueViolationError`, where 3.x removed the other row first. See [migration](/release-notes/migration.md#upsertgraph-relates-moved-rows-before-it-deletes) [#214](https://github.com/ditojs/objection/issues/214)
 - `findById()` and `findByIds()` add their id condition in its own parentheses, like `$query()` and the `*ById()` writes, so `orWhere()` can't widen it: `findById(1).orWhere(b)` is `id = 1 and (b)`, where 3.x built `id = 1 or b`, and `where(a).orWhere(b).findById(1)` is `(a or b) and id = 1`. See [migration](/release-notes/migration.md#findbyid-and-findbyids-keep-their-id-condition-in-its-own-group) [#271](https://github.com/ditojs/objection/issues/271)
 - On PostgreSQL drivers (`pg`, `pgnative`, `cockroachdb`), JSON attributes keep the values that the driver returns, so a `json` / `jsonb` column holding a string that looks like JSON, e.g. `'123'` or `'{"a":1}'`, is no longer parsed a second time, and JSON in `text` columns is no longer parsed. Set `jsonColumnFormat = 'text'` on models that keep JSON in `text` columns on PostgreSQL. Other drivers parse JSON attributes as in 3.x. See [migration](/release-notes/migration.md#json-in-postgresql-text-columns-is-no-longer-parsed) [#205](https://github.com/ditojs/objection/issues/205)
+- `upsertGraph()` upserts the models below related models in the same graph, instead of in nested `upsertGraph()` calls after all other writes: their rows are fetched together, their writes run in the same steps as the rest of the graph, and `#ref` can point between them. A related model with relations whose row doesn't exist throws a `NotFoundError` before anything is written, where 3.x wrote the rest of the graph first. See [migration](/release-notes/migration.md#upsertgraph-upserts-below-related-models-in-the-same-graph)
+- `insertGraph()` and `upsertGraph()` write in fewer queries: rows are removed with one query per relation, with all owners as `items` in the static hooks, instead of one query per owner, and the changed rows of models without update hooks are patched with one query per table. See [migration](/release-notes/migration.md#insertgraph-and-upsertgraph-write-in-fewer-queries)
+- Custom relation classes that extend `Relation` directly declare where their keys are stored with a `keySide` getter (`'owner'`, `'related'` or `'join'`) to be used in `insertGraph()` and `upsertGraph()`. Relations that extend the built-in relation classes inherit it
+- `insertGraph()` and `upsertGraph()` never return the `#id`, `#ref`, `#dbRef`, `#unrelate` and `#delete` markers, where 3.x returned them on models that the graph ignored or removed, and graph options with values of the wrong type throw also when no model of the graph reads them
 
 ### Features
 
@@ -75,6 +79,12 @@
 - The error of `upsertGraph()` queries with other query builder calls names `$query()`, which adds such a call too [#1168](https://github.com/Vincit/objection.js/issues/1168)
 - `upsertGraphAndFetch()` loads the relations of a `withGraphFetched()` on its query with their modifiers, named modifiers, aliases and options into the fetched models, like `insertGraphAndFetch()`, instead of ignoring the modifiers and aliases [#839](https://github.com/Vincit/objection.js/issues/839)
 - On SQLite, `insert()` stores arrays in columns that aren't `jsonAttributes` as JSON, like objects, instead of `'[object Object]'`
+- `upsertGraph()` keeps the join table extras of each join row when it relates the same model twice, e.g. a person as director and producer of a movie, instead of writing the last extras to both [#1268](https://github.com/Vincit/objection.js/issues/1268)
+- `insertGraph()` and `upsertGraph()` resolve a `#ref{}` in a foreign key that the relation sets too, instead of failing with `replace is not a function` [#1378](https://github.com/Vincit/objection.js/issues/1378)
+- `insertGraph()` and `upsertGraph()` insert several models without properties on PostgreSQL, instead of failing with "The query is empty" [#1694](https://github.com/Vincit/objection.js/issues/1694)
+- `insertGraph()` and `upsertGraph()` fetch the keys that the database generates for inserted rows, like a uuid that isn't the id, when relations join through them, instead of writing join rows and foreign keys without them [#853](https://github.com/Vincit/objection.js/issues/853)
+- `upsertGraph()` gives new models the keys of their existing parents that the graph leaves out, e.g. a `code` column that the relation joins through, instead of inserting them with NULL
+- `insertGraph()` and `upsertGraph()` stay under the number of values that a query can bind on SQL Server, PostgreSQL and SQLite when they insert, patch and remove many rows, and fetch many roots or related models
 - A many-to-many `patch()` / `update()` with join table extras and `limit()` on MySQL and SQL Server can order by columns of the join table [#161](https://github.com/ditojs/objection/issues/161)
 - `cancelQuery()` with a single row in the `beforeInsert` hook of an `insert()` merges it into the inserted model, like an array of rows, instead of ignoring it [#66](https://github.com/ditojs/objection/issues/66)
 - `insertGraph()` and `upsertGraph()` reject only after the queries they already started have finished, so a transaction is rolled back after them. Before, rows of these queries could be committed after the rollback on SQLite. [#335](https://github.com/ditojs/objection/pull/335)
@@ -91,6 +101,7 @@
 
 ### Other
 
+- `insertGraph()` and `upsertGraph()` are rewritten, with a third less code. Graphs write in far fewer queries, e.g. a 500 row insert in 6 queries instead of 500 on SQLite, and an upsert that changes and removes 100 rows in 7 instead of 80 to 127
 - The tests run with vitest instead of mocha
 - TypeScript 7 for development
 - sqlite3 6 for development and the examples

doc/release-notes/migration.md

@@ -21,6 +21,8 @@ This section lists the breaking changes in objection 4.0.
 - [Casts, json values and json methods on MySQL and SQLite](#casts-json-values-and-json-methods-on-mysql-and-sqlite)
 - [Missing and array join properties throw](#missing-and-array-join-properties-throw)
 - [upsertGraph() relates moved rows before it deletes](#upsertgraph-relates-moved-rows-before-it-deletes)
+- [upsertGraph() upserts below related models in the same graph](#upsertgraph-upserts-below-related-models-in-the-same-graph)
+- [insertGraph() and upsertGraph() write in fewer queries](#insertgraph-and-upsertgraph-write-in-fewer-queries)
 - [typescript: The typings are ES module declarations](#typescript-the-typings-are-es-module-declarations)
 - [Static hooks get the result in the shape that the query returns](#static-hooks-get-the-result-in-the-shape-that-the-query-returns)
 - [withGraphFetched() assigns what the relation query returns](#withgraphfetched-assigns-what-the-relation-query-returns)
@@ -174,10 +176,22 @@ Fetching relations with `withGraphFetched()`, `$fetchGraph()`, `fetchGraph()` or
 
 ## upsertGraph() relates moved rows before it deletes
 
-`upsertGraph()` relates the rows that move to existing owners of a `HasManyRelation` or `HasOneRelation` first, before it deletes or unrelates the rows that the graph removes. 3.x deleted first, so a row that moved out of a deleted parent was deleted with it by `ON DELETE CASCADE`, and failed with a foreign key without `ON DELETE` action. The update hooks of these relates now run before the delete hooks of the same call.
+`upsertGraph()` relates the rows that move to existing owners of a `HasManyRelation` or `HasOneRelation` first, before it deletes or unrelates the rows that the graph removes. 3.x deleted first, so a row that moved out of a deleted parent was deleted with it by `ON DELETE CASCADE`, and failed with a foreign key without `ON DELETE` action. The update hooks of these relates now run before the delete hooks of the same call. Rows that wait for the keys or references of models that the graph inserts are moved without them, and patched again once these are known, which runs their update hooks a second time, as the nested `upsertGraph()` calls of 3.x did. Their first write keeps the old values of the props that wait, e.g. a name from a `#ref{}` to an inserted model, so a unique key at the new owner that only the new value satisfies fails with a `UniqueViolationError`.
 
 A row related from another owner can no longer take the unique key of a row that the graph removes in its new place, e.g. with `unique(ownerId, name)`: the relate fails with a `UniqueViolationError`. Delete the conflicting row with a separate query in the same transaction first. Replacing such a row by a new one, or renaming a kept row to its key, still works, as do relates into a `HasOneRelation` with a unique foreign key.
 
+## upsertGraph() upserts below related models in the same graph
+
+3.x upserted the models below a related model with a nested `upsertGraph()` call for each owner and relation, after all other writes of the graph. 4.0 fetches the rows of the related models with their relations together with the rest of the graph, and writes them in the same steps, so their deletes, inserts and patches run with the others of their kind instead of last. The options keep matching the full relation paths.
+
+A related model with relations whose row doesn't exist throws a `NotFoundError` saying that the model doesn't exist, before anything is written. 3.x wrote the rest of the graph first and then failed in the nested call, or with a foreign key error.
+
+## insertGraph() and upsertGraph() write in fewer queries
+
+Rows that the graph removes are deleted or unrelated with one query per relation instead of one per owner, so the static `beforeDelete` and `afterDelete` hooks (or `beforeUpdate` and `afterUpdate` for has-many unrelates) run once with all owners as `items`. Join rows are still removed per owner.
+
+The changed rows of a model class without `beforeUpdate`, `afterUpdate`, `$beforeUpdate` and `$afterUpdate` hooks are patched with one `UPDATE` per table that picks the values of each row with `CASE`, unless the upsert uses `returning('*')`, the values include json or raw values, or the rows are patched through a many-to-many or belongs-to-one relation. Custom query builders see that one patch query instead of one per row. Models with update hooks are patched one by one, as before.
+
 ## typescript: The typings are ES module declarations
 
 The typings no longer use `export =`. They declare the named exports of the ES module they describe. Apart from the default import, ES module and bundler projects need no changes.