Serialize arrays in SQLite inserts, and return ids from batch inserts only when needed

doc/api/model/static-methods.md

@@ -73,8 +73,8 @@ console.log(sylvester.fullName());
 // --> 'Sylvester Stallone'.
 
 // Batch insert. On MySQL, this only works for models that
-// have their identifiers already, as MySQL doesn't return
-// the identifiers of _all_ inserted rows.
+// have their identifiers already, or have no `idColumn`, as
+// MySQL doesn't return the identifiers of _all_ inserted rows.
 const inserted = await Person.query().insert([
   { firstName: 'Arnold', lastName: 'Schwarzenegger' },
   { firstName: 'Sylvester', lastName: 'Stallone' }

doc/api/query-builder/mutate-methods.md

@@ -18,11 +18,17 @@ properties - see [this recipe](/recipes/returning-tricks.html) for some examples
 `returning(['only', 'some', 'props'])` note that the result object will still contain the input properies
 **plus** the properties listed in `returning`. On other databases you can use the [insertAndFetch](/api/query-builder/mutate-methods.html#insertandfetch) method.
 
-Batch inserts work on Postgres, SQL Server and SQLite (3.35 or newer), which return
-the identifiers of _all_ inserted rows. On MySQL, they only work if all models have
-their identifiers already, as MySQL only returns the id of the first inserted row.
-If you need batch inserts of new rows on MySQL, you can use knex directly through
-[knexQuery](/api/model/static-methods.html#static-knexquery).
+Batch inserts work on Postgres, SQL Server and SQLite, which return the identifiers
+of _all_ inserted rows. On MySQL, they only work if all models have their identifiers
+already or the model has no `idColumn`, as MySQL only returns the id of the first
+inserted row. If you need batch inserts of new rows on MySQL, you can use knex
+directly through [knexQuery](/api/model/static-methods.html#static-knexquery).
+
+On SQLite, batch inserts of new rows need SQLite 3.35 or newer for `returning`, and
+fail with a syntax error near "returning" on older versions. Columns that some rows
+of a batch leave out get NULL instead of their defaults with knex's
+`useNullAsDefault` option, and make knex throw without it. Batches of more than 500
+rows or 32766 values fail.
 
 `insert` writes whole values, so it rejects [FieldExpressions](/api/types/#type-fieldexpression) like `'meta:a.b'` that write parts of a json column with a [ValidationError](/api/types/#class-validationerror). Use [patch](/api/query-builder/mutate-methods.html#patch) for them. Keys with a `:` that the schema declares in `properties` are normal properties and columns, not field expressions.
 
@@ -630,10 +636,11 @@ the case of a many-to-many relation, creates a join row to the join table. In
 the case of a belongs-to-many relation, adds the keys to the arrays of the
 owners.
 
-Multiple items can be related by giving an array of identifiers: for has-many
-and belongs-to-many relations on all databases, for many-to-many relations only
-on Postgres and SQL Server, as it needs a batch insert. Belongs-to-one relations
-take one identifier.
+Multiple items can be related by giving an array of identifiers: for has-many,
+belongs-to-many and many-to-many relations on all databases. On MySQL, the join
+rows of many-to-many relations need to have their ids, which they have unless a
+`through.modelClass` is used whose `idColumn` isn't made of the join columns. Belongs-to-one relations take
+one identifier.
 
 The return value of the query is the number of affected items. On all databases
 except MySQL you can chain [returning('\*')](/api/query-builder/find-methods.html#returning)
@@ -723,7 +730,7 @@ const numRelatedRows = await person
 console.log('movie 50 is now related to person 123 through `movies` relation');
 ```
 
-Relate multiple (only works with postgres)
+Relate multiple
 
 ```js
 const numRelatedRows = await Person.relatedQuery('movies')

doc/release-notes/changelog.md

@@ -56,7 +56,7 @@
 
 ### Features
 
-- `insert()` of several models works on SQLite (3.35 or newer) through `returning`, and on all databases when all models have their ids, e.g. join rows. `insertGraph()` and `upsertGraph()` insert in batches there too
+- `insert()` of several models works on SQLite (3.35 or newer, up to 500 rows) through `returning`, and on all databases when all models have their ids or the model has no `idColumn`, e.g. join rows
 - `BelongsToManyRelation`, a relation where the owner holds an array of the related models' keys, e.g. `posts.tagIds = [1, 4, 7]` for `post.tags`, in a JSON column on all databases or a native array column on PostgreSQL. It supports finding the related models with `$relatedQuery()`, `relatedQuery()`, `withGraphFetched()`, `withGraphJoined()` and `joinRelated()`. `insert()`, `delete()`, `relate()` and `unrelate()` through the relation rewrite the arrays, `insertGraph()` and `upsertGraph()` write them, and `$setRelated()` and `$appendRelated()` keep them in sync. See [relations](/guide/relations.md) [#2667](https://github.com/Vincit/objection.js/issues/2667)
 - `Model.jsonColumnFormat` says how the database returns JSON columns: `'auto'` (default) leaves them as they are on PostgreSQL drivers and parses JSON text on others, `'text'` parses them strictly and throws for invalid JSON, `'parsed'` never parses them [#205](https://github.com/ditojs/objection/issues/205)
 - JSON field expressions work on MySQL and SQLite: in `ref()`, e.g. `ref('column:a.b').castInt()` in `select()`, `where()`, `orderBy()`, joins and relation mappings, as keys in `patch()` and `merge()`, e.g. `patch({ 'column:a.b': 1 })`, and in the `whereJson*()` methods, except for `whereJsonSupersetOf()` and `whereJsonSubsetOf()` on SQLite. objection writes the SQL of each database, `JSON_EXTRACT()` and `JSON_SET()` on MySQL and `json_extract()` and `json_set()` on SQLite, and casts to their types. `ref().castJson()` works on both too. See [JSON queries](/recipes/json-queries.md) [#276](https://github.com/Vincit/objection.js/issues/276)
@@ -68,6 +68,7 @@
 
 - The error of `upsertGraph()` queries with other query builder calls names `$query()`, which adds such a call too [#1168](https://github.com/Vincit/objection.js/issues/1168)
 - `upsertGraphAndFetch()` loads the relations of a `withGraphFetched()` on its query with their modifiers, named modifiers, aliases and options into the fetched models, like `insertGraphAndFetch()`, instead of ignoring the modifiers and aliases [#839](https://github.com/Vincit/objection.js/issues/839)
+- On SQLite, `insert()` stores arrays in columns that aren't `jsonAttributes` as JSON, like objects, instead of `'[object Object]'`
 - A many-to-many `patch()` / `update()` with join table extras and `limit()` on MySQL and SQL Server can order by columns of the join table [#161](https://github.com/ditojs/objection/issues/161)
 - `cancelQuery()` with a single row in the `beforeInsert` hook of an `insert()` merges it into the inserted model, like an array of rows, instead of ignoring it [#66](https://github.com/ditojs/objection/issues/66)
 

lib/queryBuilder/operations/InsertOperation.js

@@ -27,20 +27,23 @@ export class InsertOperation extends QueryBuilderOperation {
   }
 
   async onBefore2(builder, result) {
+    await callBeforeInsert(builder, this.models);
+
     // Without ids to return, rows can be inserted in batches on all databases.
+    // The hooks may give the models their ids, or cancel the query.
     if (
       this.models.length > 1 &&
       !hasBatchInsert(builder.knex()) &&
-      !this.models.every((model) => model.$hasId())
+      !builder.isExplicitlyResolvedOrRejected() &&
+      this._hasIdsToReturn(builder)
     ) {
       throw new Error(
-        'batch insert only works with Postgresql, SQL Server and SQLite, or with models that have their ids',
+        'batch insert only works with Postgresql, SQL Server and SQLite, or with models that have their ids or no idColumn',
       );
-    } else {
-      await callBeforeInsert(builder, this.models);
-      this.models.forEach(removeOverriddenValues);
-      return result;
     }
+
+    this.models.forEach(removeOverriddenValues);
+    return result;
   }
 
   onBuildKnex(knexBuilder, builder) {
@@ -55,7 +58,10 @@ export class InsertOperation extends QueryBuilderOperation {
         // conflict columns are returned too, so that the returned rows can
         // be matched to the models when some of the rows are ignored.
         knexBuilder = knexBuilder.returning(defaultReturning);
-      } else if (isSqlite(knex) && (builder.has('onConflict') || this.models.length > 1)) {
+      } else if (
+        isSqlite(knex) &&
+        (builder.has('onConflict') || (this.models.length > 1 && this._hasIdsToReturn(builder)))
+      ) {
         // Without a `returning` clause, SQLite returns the id of the last
         // inserted row, which belongs to a previous insert if the row was
         // ignored, and only one id for batch inserts. Use `returning` to find
@@ -65,9 +71,9 @@ export class InsertOperation extends QueryBuilderOperation {
     }
 
     const modelClass = builder.modelClass();
-    // knex inserts several rows into SQLite with `select ... union all`, which
-    // doesn't serialize objects and arrays like single rows do.
-    const shouldSerialize = isSqlite(knex) && this.models.length > 1;
+    // knex leaves arrays unserialized in SQLite inserts, and also objects when
+    // it inserts several rows with `select ... union all`.
+    const shouldSerialize = isSqlite(knex);
 
     return knexBuilder.insert(
       this.models.map((model) => {
@@ -78,6 +84,13 @@ export class InsertOperation extends QueryBuilderOperation {
     );
   }
 
+  _hasIdsToReturn(builder) {
+    return (
+      builder.modelClass().getIdColumnArray().length > 0 &&
+      !this.models.every((model) => model.$hasId())
+    );
+  }
+
   onAfter1(builder, ret) {
     if (!Array.isArray(ret) || !ret.length || ret === this.models) {
       // Early exit if there is nothing to do.

tests/integration/insert.js

@@ -209,7 +209,8 @@ export default (session) => {
           });
       });
 
-      if (session.isPostgres()) {
+      // MySQL doesn't return the ids of several inserted rows.
+      if (!session.isMySql()) {
         it('should accept an array', () => {
           let models = [
             Model1.fromJson({ model1Prop1: 'hello 3' }),
@@ -282,6 +283,42 @@ export default (session) => {
           });
       });
 
+      it('should accept a json array of models that get their ids in $beforeInsert', async () => {
+        let nextId = 200;
+
+        class IdModel extends Model1 {
+          $beforeInsert(context) {
+            super.$beforeInsert(context);
+            this.id = nextId++;
+          }
+        }
+
+        const inserted = await IdModel.query().insert([
+          { model1Prop1: 'hello 200' },
+          { model1Prop1: 'hello 201' },
+        ]);
+
+        expect(inserted.map((it) => it.id)).toEqual([200, 201]);
+        expect(await session.knex(Model1.getTableName()).whereIn('id', [200, 201])).toHaveLength(2);
+      });
+
+      it('should accept a json array of models whose insert a hook cancels', async () => {
+        class CancelModel extends Model1 {
+          static beforeInsert({ cancelQuery }) {
+            cancelQuery([]);
+          }
+        }
+
+        await CancelModel.query().insert([
+          { model1Prop1: 'cancelled 1' },
+          { model1Prop1: 'cancelled 2' },
+        ]);
+
+        expect(
+          await session.knex(Model1.getTableName()).where('model1Prop1', 'like', 'cancelled%'),
+        ).toEqual([]);
+      });
+
       it('should accept a json array of models with ids', async () => {
         const inserted = await Model1.query().insert([
           { id: 100, model1Prop1: 'hello 100' },
@@ -906,7 +943,8 @@ export default (session) => {
             });
         });
 
-        if (session.isPostgres()) {
+        // MySQL doesn't return the ids of several inserted rows.
+        if (!session.isMySql()) {
           it('should accept an array', () => {
             let inserted = null;
 
@@ -1134,7 +1172,8 @@ export default (session) => {
             });
         });
 
-        if (session.isPostgres()) {
+        // MySQL doesn't return the ids of several inserted rows.
+        if (!session.isMySql()) {
           it('should accept an array', () => {
             let inserted = null;
 
@@ -1557,7 +1596,8 @@ export default (session) => {
             });
         });
 
-        if (session.isPostgres()) {
+        // MySQL doesn't return the ids of several inserted rows.
+        if (!session.isMySql()) {
           it('should accept an array', () => {
             return Model1.relatedQuery('model1Relation2')
               .for(1)

tests/integration/misc/issue205.js

@@ -60,7 +60,7 @@ export default (session) => {
       return knex('issue205Item').delete();
     });
 
-    // Not all databases support batch inserts:
+    // MySQL doesn't return the ids of several inserted rows:
     const insertItems = async (items) => {
       for (const item of items) {
         await Item.query().insert(item);

tests/integration/misc/issue2305.js

@@ -70,8 +70,7 @@ export default (session) => {
       expect(await rows()).toEqual([{ name: 'a', runs: 1 }]);
     });
 
-    // Only Postgres supports batch inserts.
-    it.skipIf(!session.isPostgres())('should insert multiple models', async () => {
+    it('should insert multiple models', async () => {
       const jobs = await JobExecution.query().insert([
         { name: 'a', runs: 1 },
         { name: 'b', runs: 2 },

tests/integration/misc/sqliteBatchInsert.js

@@ -0,0 +1,101 @@
+import { describe, it, expect, beforeAll, beforeEach, afterAll } from 'vitest';
+import { Model } from 'objection';
+
+export default (session) => {
+  describe.skipIf(!session.isSqlite())('insert() of several models on SQLite', () => {
+    const { knex } = session;
+
+    class Item extends Model {
+      static get tableName() {
+        return 'batch_insert_item';
+      }
+    }
+
+    class Pair extends Model {
+      static get tableName() {
+        return 'batch_insert_pair';
+      }
+
+      static get idColumn() {
+        return ['a', 'b'];
+      }
+    }
+
+    const dropTables = () =>
+      knex.schema.dropTableIfExists('batch_insert_item').dropTableIfExists('batch_insert_pair');
+
+    beforeAll(async () => {
+      await dropTables();
+      await knex.schema
+        .createTable('batch_insert_item', (table) => {
+          table.increments('id');
+          table.string('name').unique();
+          table.text('data');
+        })
+        .createTable('batch_insert_pair', (table) => {
+          table.integer('a');
+          table.integer('b');
+          table.string('name');
+          table.primary(['a', 'b']);
+        });
+    });
+
+    beforeEach(async () => {
+      await knex('batch_insert_item').delete();
+      await knex('batch_insert_pair').delete();
+    });
+
+    afterAll(dropTables);
+
+    it('should store objects and arrays as JSON, in one row and in several', async () => {
+      await Item.query(knex).insert({ name: 'a', data: [1, 2] });
+      await Item.query(knex).insert([
+        { name: 'b', data: { b: 1 } },
+        { name: 'c', data: [3] },
+      ]);
+
+      expect(await knex('batch_insert_item').select('name', 'data').orderBy('name')).toEqual([
+        { name: 'a', data: '[1,2]' },
+        { name: 'b', data: '{"b":1}' },
+        { name: 'c', data: '[3]' },
+      ]);
+    });
+
+    it('should return the ids of the inserted rows with onConflict().ignore()', async () => {
+      await knex('batch_insert_item').insert({ name: 'a' });
+
+      const [a, b] = await Item.query(knex)
+        .insert([{ name: 'a' }, { name: 'b' }])
+        .onConflict('name')
+        .ignore();
+
+      expect(a.id).toBeUndefined();
+      expect(b.id).toBe((await knex('batch_insert_item').where('name', 'b').first()).id);
+    });
+
+    it('should insert models with composite ids', async () => {
+      const pairs = await Pair.query(knex).insert([
+        { a: 1, b: 2, name: 'x' },
+        { a: 2, b: 1, name: 'y' },
+      ]);
+
+      expect(pairs.map((pair) => pair.$id())).toEqual([
+        [1, 2],
+        [2, 1],
+      ]);
+      expect(await knex('batch_insert_pair').orderBy('a')).toHaveLength(2);
+    });
+
+    it('should not return anything for models with ids', () => {
+      const sql = Item.query(knex)
+        .insert([
+          { id: 1, name: 'a' },
+          { id: 2, name: 'b' },
+        ])
+        .toKnexQuery()
+        .toString();
+
+      expect(sql).not.toMatch(/returning/);
+    });
+  });
+};