Rewrite upsertGraph() and insertGraph() (#1268, #1378, #1694, #853)

[lehni]

What

A rewrite of the graph code: 22 files and 3,359 lines become 6 files and 2,390 lines, 28% less, with the same behaviour apart from the changes listed below.

• GraphNode: the graph model. The markers (#id, #ref, #dbRef, #unrelate, #delete) are read once when the graph is built, and #ref / #ref{} are linked there.
• GraphLoader: fetches the current graph in rounds, and pairs each row with its model.
  • First round: the roots with their relations.
  • Then, level by level: the rows of the models that the graph relates and gives relations, or looks up for insertMissing. A #ref to an existing model loads that model's row.
  • The models below related models are upserted in the same graph, instead of in nested upsertGraph() calls after all other writes.
• GraphPlan: the rules.
  • Each model gets an intent from its data (insert, relate, patch, remove), and then the options that disable it apply, which replaces the shouldX() / shouldXIgnoreDisable() pairs.
  • The current rows get delete, unrelate or keep.
  • Pruning.
  • The decisions are cached once loading is done.
• GraphWriter: the writes, in the order of upsertGraph() deletes rows that the same graph relates to another owner #214.
  • Moves of related rows, then removals, inserts by dependency level with cycle breaking, join rows, patches, and removals of BelongsToOne rows.
  • Rows related to existing owners of HasMany and HasOne relations are moved first with their patch. Rows that wait for the keys or references of inserted models are moved without them and patched again with the other patches. A relate into a HasOne slot waits for the move of the row in it.
  • Inserts and join rows go in batches wherever the database allows them. On SQLite, the rows of a batch have the same columns, as knex would fill missing ones with NULL instead of their defaults, and models with beforeInsert hooks, which may still change the columns, are inserted one by one.
  • Rows are removed with one query per relation. Join rows are still removed per owner.
  • Models without update hooks are patched with one UPDATE … CASE per table. Models with hooks are still patched one by one.
  • Batches, id lists and bulk patches stay under the number of values that a query can bind on SQL Server, PostgreSQL and SQLite. The relation queries of the fetch below the first level are left to withGraphFetched().
  • A write that finds no row throws a NotFoundError only if the row is gone. Rows that a trigger skips, or that MySQL doesn't count as changed, aren't errors.
• Relation#keySide ('owner', 'related' or 'join') tells the graph where a relation keeps its keys, instead of the graph checking relation classes. Owners of relations that aren't one-to-one hold arrays of keys, which is how BelongsToManyRelation from Add BelongsToManyRelation for owners that hold an array of keys (#2667) #319 takes part.
• GraphOptions only parses and matches the options. Options with values of the wrong type always throw.
• The internal tests of GraphInsert go through insertGraph() and upsertGraph().

Fixes

• Vincit#1268: relating the same model twice with different join table extras keeps the extras of each join row.
• Vincit#1378: #ref{} in a foreign key that the relation sets too no longer fails with replace is not a function.
• Vincit#1694: several models without properties can be inserted on PostgreSQL.
• Vincit#853: keys that the database generates, like a uuid that isn't the id, are fetched after inserts when relations join through them.
• New models get the keys of their existing parents that the graph leaves out, e.g. a code column that the relation joins through, instead of being inserted with NULL.
• A #ref with relations, related through a ManyToManyRelation to a model that the graph patches elsewhere, no longer fails with a NotFoundError.
• Vincit#1059 is documented: insertGraph() and upsertGraph() validate their models once the references and keys are resolved, so write-only input like passwords belongs in $beforeInsert / $beforeUpdate hooks, not in setters.

Speed

Median ms, with queries in parentheses. The no-op row is about CPU.

	SQLite	PostgreSQL	MySQL
insert 500 rows	196 (500) → 7 (6)	5 (6) → 5 (6)	162 (500) → 138 (401)
upsert that changes, moves and deletes 100 rows	47 (127) → 6 (7)	11 (80) → 5 (7)	40 (127) → 20 (54)
relate 20 models with subtrees	30 (84) → 4 (7)	9 (65) → 5 (7)	32 (84) → 6 (7)
no-op upsert of 2,200 models	15 → 12	13 → 12	14 → 12

Breaking

These are in the changelog and the migration guide.

• Related models with relations: the models below them are written in the same steps as the rest of the graph. A related model with relations whose row doesn't exist throws a NotFoundError before anything is written.
• Removals: one query per relation, so the static delete hooks (or update hooks for has-many unrelates) get all owners as items at once.
• Patches: models without update hooks are patched in one query per table. Custom query builders see that query.
• Markers: results never contain #id, #ref, #dbRef, #unrelate or #delete.
• Options: values of the wrong type always throw.
• Custom relations: classes that extend Relation directly need a keySide getter to be used in graphs.

Reasoning

• The bulk CASE patches and the batched removals only apply where nothing observes single rows: no update hooks, no returning('*'), plain values, no join table extras. Everything else is written one row at a time, as before.
• Relation#keySide describes where the keys are stored, not which class a relation is. That keeps the graph code free of relation types, and it would let other relation classes take part.